Title: A Label is Worth A Thousand Images in Dataset Distillation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dataset distillation

Data condensation

Synthetic data generation

Data-efficient learning
Abstract: Data *quality* is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good” training data. However, a major challenge in achieving this goal is the observation that distillation approaches, which rely on sophisticated but mostly disparate methods to generate synthetic data, have little in common with each other. In this work, we highlight a largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance of state-of-the-art distillation methods is not the specific techniques used to generate synthetic data but rather the use of soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain *structured information* to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establish an empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available at https://github.com/sunnytqin/no-distillation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the effect of synthetic image soft labels on training performance, and show that the success of DD methods is attributed to the use of informative labels. The authors showed that the structured information in soft labels is important, and there is a tradeoff between knowledge and data. Generally, the paper conducted extensive ablations to analyze the synthetic soft labels.\n\nSTRENGTHS:\n1. The paper studies the interesting and novel aspect of the role of synthetic labels in DD.\n2. Thorough experiments are done and the comparisons are fair and meaningful.\n3. The experimental results and analysis provide important and interesting insights that are beneficial for the DD community.\n\nWEAKNESSES:\nI do not observe major weaknesses of the paper. However, I do have some minor concerns that I would like to discuss with the authors:\n\n1. Observing Figure 4, it is interesting that when swapping the top-1 label with the last one, the relative performance can still be even preserved to relatively 20% and even 50% (IPC=1). The top-1 label should be the correct prediction, and swapping makes all images wrongly labeled. I wonder why there is still 20%~50% relative performance when all the training data are wrongly labeled.\n2. For the experiments corresponding to Figure 7 (treatment 2), why no re-normalization? This results in probabilities that do not sum up to 1 and may cause unexpected consequences during training. \n3. It seems that there are two types of labels used for analysis. One type is directly downloaded synthetic dataset (image-label pairs) in section 4. Another type is generating soft-labels via ensembling in section 5. Does the second guarantee that the ensembled soft-labels are correct (same one-hot encodings as the original pairs)? Also, using early-stopping experts have drawbacks that it does not provide correct label information (e.g. Figure 3). To what extent (how early) may the experts be useful for label generation?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes, limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the importance of soft labels for dataset distillation methods, conducting detailed ablation experiments on the role of labels under various settings. It deeply explores the impact of labels on learning, providing an in-depth analysis and study of the intrinsic properties of labels. The work also provides empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establishes an empirical Pareto frontier for data-efficient learning.\n\nSTRENGTHS:\n1. This paper clearly demonstrates the crucial role that soft labels play in the effectiveness of data distillation methods. This aspect has never been carefully studied in previous dataset distillation work; it was usually considered an additional trick and did not receive much attention. This work points out new directions for data-efficient learning.\n2. The designed experiments are interesting and comprehensive, presenting what constitutes good soft labels, the importance of knowledge (soft labels) in learning, and how to effectively obtain higher-quality soft labels.\n\nWEAKNESSES:\n1. According to Table 7, the expert model (epoch) used to produce soft labels in the soft label baseline appears to be carefully selected. Does the choice of epoch introduce additional costs? Additionally, do the other mentioned dataset distillation methods also use the best soft labels from these epochs to ensure a fair comparison?\n2. MTT and SRe2L are not the SOTA methods currently. I would like to know if more advanced methods like DATM [1] and G-VBSM [2] still heavily rely on soft labels and whether their performance still can not surpass the soft label baseline.\nMinor: line 185 In Figure 3 left.\n\n[1] Towards lossless dataset distillation via difficulty-aligned trajectory matching. ICLR 2024\n\n[2] Generalized large-scale data condensation via various backbone and statistical matching. CVPR 2024\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces soft probabilistic labels to the dataset distillation task. Specifically, it finds that the labels should consider structured information and perform unequally. Experiments on diverse datasets demonstrate its effectiveness.\n\nSTRENGTHS:\n1) This paper proposes the introduction of soft labeling in dataset distillation and provides an interesting analysis of the intrinsic properties of data distillation, rather than focusing on improving the base module of specific methods.\n2) The method is well-supported, and the results are reliable.\n\nWEAKNESSES:\n1) The motivation needs to be stated more clearly. Why are label-level methods regarded as superior to image-level methods？\n2) In the introduction, the author introduces several techniques such as 'expert' models, Pareto frontier, and knowledge-data scaling laws but lacks detailed explanation. \n3) Section 3.2 requires reorganization to enhance readability.\n4) The structure information needs to be expressed more clearly and intuitively.\n\nQUESTIONS:\n1) In the Method section, could different soft-labeling strategies (excluding cutmix) influence the entire pipeline?\n2) Does early stopping result in varied performances across different baselines and datasets? How do the authors address this issue?\n3) In the experiment, the author uses the swap test to validate the structure information and claims “Top labels contain structured information and the non-top contain unstructured noise”. Why not just remove the unstructured noise?\n\nLIMITATIONS:\n1) The writing requires improvement as the author attempts to cover too information without adequately establishing connections and justifying the necessity of the key technologies.\n2) The tables and figures need reorganization to enhance clarity. Reviewer currently struggles to understand their intended messages.\n3) Some typos need to be corrected, such as those in the representation of Figure 4.\n4) The limitations mentioned in the Conclusion are not clearly stated.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyses the role of soft labels used in dataset distillation. Experiments with different ablation studies show that the performance of soft labels based data distillation approaches is primarily attributed to the use of soft labels. Secondly, the authors study the various types of soft labels and their effect on model performance. Additionally an empirical scaling law is provided to characterize the relation between effectiveness of soft labels and image per class in distilled dataset.\n\nSTRENGTHS:\nThe paper is written clearly with well-presented motivation and is easy to follow\n\nExtensive analysis and ablations are presented providing a better understanding of role of labels in dataset distillation\n \nThe paper focuses on a largely overlooked aspect of dataset distillation methods: the degree of contribution of soft labels.\n\nWEAKNESSES:\nThe paper could benefit from a theoretical analysis of why soft labels are so effective. Additionally, the generalizability of the findings to data distillation in other modalities could be insightful.\n\nMinor comments: \n\n1. The description in line 222 appears inconsistent with results in figure 4. IPC=1 appears more robust to swapping of top labels compared to IPC=10\n\n2. A few writing and grammar errors exist in related work, line numbers 87-90. Also, 'generation' should be 'general' in line  167\n\n3. In Figure 2, it might be better to use the same dataset to highlight the dependence on expert accuracy and label entropy.\n\nQUESTIONS:\nI am concerned about the use of experts at tuned epoch for comparison of soft label baseline with previous methods. Are the expert epochs also tuned for previous approaches? \n\nCan the authors provide insight as to why structured information is more beneficial in soft labels in the case of dataset distillation while the opposite is true for knowledge distillation as mentioned in related work?\n\n It is unclear how past approaches in figure 1 right  (which as I understood to originally use soft labels) were adapted to hard labels. Is it done by performing argmax on the soft labels?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}