A Label is Worth A Thousand Images in Dataset Distillation
Authors: Tian Qin
Zhiwei Deng
David Alvarez-Melis
Venue: NeurIPS 2024 poster (2024)
Keywords: Dataset distillation
Data condensation
Synthetic data generation
Data-efficient learning
Primary area: machine_vision
TL;DR: Informative soft labels is the key behind the success of dataset distillation methods.

Abstract: Data *quality* is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good” training data. However, a major challenge in achieving this goal is the observation that distillation approaches, which rely on sophisticated but mostly disparate methods to generate synthetic data, have little in common with each other. In this work, we highlight a largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance of state-of-the-art distillation methods is not the specific techniques used to generate synthetic data but rather the use of soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain *structured information* to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establish an empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available at https://github.com/sunnytqin/no-distillation.

A Label is Worth a Thousand Images in Dataset Distillation
Tian Qin Harvard University Cambridge, MA
Zhiwei Deng Google DeepMind Mountain View, CA zhi zhiweidengggoogle.ccm
David Alvarez-Melis Harvard University & MSR Cambridge, MA damβseas. harvard, edu
tqinêg. harvard edu
Abstract
Data quality is crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good" training data. However, major challenge in achieving this goal the observation that distillation approaches, which rely on sophisticated but mostly disparate methods generate synthetic data, have little in common with each other. In this work, we highlight largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance is not the specific techniques used to generate synthetic data but rather the use soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain structured information to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-dee in the distilled dataset and establish empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available https:///gii wwwgghttu.... http://gggttuuu..mmm.uuuuta www.t/////onnnnnnnnnn no-tisiillllleee
state-of-the- art distillation methods
Introduction
Data central to the success of modern machine learning models, and there is increasing evidence that data quality trumps quantity in many settings. For example, in the context of large language models (LLLM), Abdin al. [T), Gunasekar et [TT), and Hughes (14) show that training L.MMs in "data-optimal" regime allows for significant reductions in model size without sacrificing performance and capabilities. In other words, when trained on high-quality data, small models can rival the performance of their much larger counterparts. Despite its obvious importance, there is no clear answer what characteristics define "good data." Data distillation offers one approach to answering this question. First introduced by Wang al. (29). the goal of dataset distillation is to condense a dataset into a smaller (synthetic) counterpart, such that training on this distilled dataset achieves performance comparable training on the original dataset. By studying the distillation process, can thus investigate what information is preserved the data (features, symmetries, and information), which in turn may shed light fundamental characteristics that make for "good" training data.
The success of dataset distillation raises two key questions. First, what aspects of the training data best facilitate "data-efficient" learning? Second, what distillation procedure better captures the aspects of the dataset relevant for prediction? So far, dataset distillation work has focused almost exclusively
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Dore
Dataset Distillation for ImageNet-1K
Data
Image
Label
Student Accuracy (%6
60
SReiL
SRc2L soft label
Random
w/ epoch tuning
Ro BP
FReeo
SPTT
Hard label
Solt label
40
(TESLA)
SRoat
20
MTT
50
100
200
Image/Class (IPC)
Figure 1: Soft labels are crucial for dataset distillation Left: Synthetic images by different distillation methods. Right: Student test Accuracy comparison between different distillation methods and random baseline from training data (both with hard labels or with soft labels). For soft/hard label generation details, see Appendix A.2
on the second question, with a strong emphasis on improving ways to generate synthetic data for each class (23). Figure (left) shows examples of the synthetic images generated by various state-of-the-nn
(SOTA) data distillation methods. Our first key observation is that, despite very different generation strategies resulting in perceptually distinct synthetic images, virtually all leading methods, especially those able to scale to large datasets such as ImageNet-1K (or its downsized version) (66 9 33 39) )
soft (i.e., probabilistic) labels for the datasets they distill. Furthermore, Cui et al. and Yin et al.
33 directly use soft labels generated by pretrained experts.
Based on this observation, we take a step back and revisit the first key motivating question above. Specifically, we ask: what is the relative importance of features (e.g., images) and labels for distilla- tion-and and therefore, data-efficient training? To answer this question, we design a series of ablation experiments aimed at studying the role of labels in dataset distillation, including simple but effective baseline that pairs randomly sampled training images (i.e., not synthetic, learned ones) and with soft labels. Surprisingly, the use of soft labels, and (ii) the simple soft label baseline achieves performance comparable to SOTA dataset distillation methods (Figure [ right) This result alone has important implications for practitioners and researchers alike. For practitioners, it calls into question spending compute or research efforts on generating synthetic images, given their relatively limited contribution overall distillation performance. For researchers, it suggests revisiting common assumptions about existing data distillation approaches and considering alternatives.
observe that (i) the main factor explaining the success of these methods is
After demonstrating the importance of soft labels, and given the limited analysis of their role in data distillation, we design further experiments to study in detail what makes them so effective for learning. We focus on the setting where soft labels are generated by an "expert" model, e.g., by labeling an image with the class probabilities predicted by the model, as most SOTA distillation methods do. We find that when learning data-limited setting, student model trained soft-labeled data achieves its best performance by learning from an carly-stopped expert, and its success relies on leveraging semantic structure contained in these soft labels. We also present an empirical knowledge-data scaling law along with Pareto frontier that showcase how (optimal) knowledge can be used to reduce dataset size. Next, we argue that the soft label baseline is special case of Knowledge Distillation (KD) [13 where the role of soft labels has been better appreciated. Our final set of experiments seek to turn these findings into better methods for generating soft labels. To this end, we first show that expert ensembles can improve learning value of soft labels. We also show that directly optimizing labels with data distillation methods (without training any experts) recovers the same information labels, indicating that expert knowledge might be the necessary way for dataset distillation.
Taken together, the results of our in-depth analysis highlight the importance of labels in dataset distillation, challenging conventional wisdom. Our contributions can be summarized as follows:
Ra- BPTT 9 and FRePo 39 only scale to IPC=1, 2 on downsized ImageNet- IK. TESLA 6 is an MTT variant with memory- wfficncceeeffiinnnttttoe implementation and uses soft labels.

---

We conduct series of ablations. including simple-but-powee baseline, to show that the success of existing data distillation methods is driven not by synthetic data generation strategies but by the use of informative labels.
We show that structured semantic information (e.g., distributional similarity across related classes) is key to good soft labels and that different distillation budgets imply different optimal soft label show how to effectively modulate these profiles by using expert
structure profiles. Furthermore, labelers trained with early stopping, a strategy that turns out to recover soft labels obtained with existing data distillation methods as a particular case.
We establish an empirical data- satta-knowlddgee scaling law to quantify how knowledge can be used to reduce dataset size, and we also establish a Pareto frontier for data-efficient learning.
2 Related Work
Dataset distillation.
Data distillation methods have been primarily developed for image classification
tasks. Existing methods can be organized into three main categories: (1) meta-model matching, (2)
distribution matching, and (3) trajectory matching (23). Meta-model matching methods approach
the problem by solving the original bi-level optimization formulation of Wang al. (29]. To tackle
the computation challenge, various methods have been proposed to either approximate the objective Distribution Matching (DM) seeks minimize
220 or improve the optimization process 9
15 39
the statistical distance in the output (logit) space between original and distilled dataset samples Yin al. [33] propose to
(36). Further refinements to the method include [16 28 37 39
match network statistics (batchnorm) along with logits and then assigning soft labels to synthetic images. Follow-up works [12] 32] brings further improvements to the soft label assignment strategy. Matching training trajectories (TT) was proposed by Cazenavette et al. [5), suggesting that matching long-range training dynamies can serve as a proxy objective to the expensive bi-level formulation.
Further improvements and variations include [6] 8). In this work, we aim to uncover common factors responsible for the success of these methods (i.e., the use of soft labels). We also draw connection between data distillation and knowledge distillation. Concurrent work (30) also discovers that models learned from distilled data behaves similar to an early stopped expert.
Distillation with soft labels. Almost all existing distillation methods able to scale to ImageNet- KK (downsized or original) leverage soft labels. Cui et al. (6) and Yin et al. (33) decouple image learning from label assignment and directly use pre-trained experts to assign soft labels. Feng et al. (9]
and Zhou et al. [39] learn the distilled images and soft labels simultaneously. In the very early years of data distillation research, the importance of soft labels was highlighted by Bohdal et al. (3) and Sucholutsky and Schonlau (26). although this was done in very limited experimental setting (MNIST), limiting its influence in further work that tackled larger and more complex classification tasks more representative of modern machine learning. Despite the increasing prevalence of soft labels in sfate-of-the-an dataset distillation methods, little work has been done in a controlled setting to understand how data and labels each contribute to the quality of the distilled data. This work extensively addresses this question.
Knowledge Distillation. The goal of knowledge distillation (KD) is to transfer knowledge learned by large teacher model to a small student model [2] 10 13.. While the KD objective may seem at
odds with the dataset distillation objective, many dataset distillation methods [32 33 are directly
inspired by data-free KD methods
[25] 31.. In fact, the use of soft labels has been extensively
studied in the context of KD [3
19 27 Generally, soft labels function both as supervisory signals
and regularizers. Yuan al. [34] and Zhou al. (38) further argue that under the KD setting the
regularization effect from soft labels is equally -if not more- important than sharing knowledge.
In this work we uncover deeper connections between the two fields. We show that one way to achieve dataset distillation is to incorporate expert knowledge into soft labels (i.e., KD). We further show that, unlike the conclusions drawn KD, the knowledge-sharing aspect of soft labels is the dominant effect under the data distillation setting.
3
Soft Labels are Crucial for Distillation
3.1 Background on dataset distillation and a simple soft label baseline
The goal of dataset distillation to create condensed version of a larger dataset that retains the essential information needed for training model. Formally, we denote the original dataset as
3

---

Table 1: Benchmark SOTA methods against CutMix baseline and soft label baseline ImageNet-IK. SRe2L is the only method that can scale to ImageNet-1K. I. Both soft label-based baselines with random training images can already achieve comparable performances.
Labeling Strategy
Hard label
Soft label baseline
CutMiX augmented soft labels
Labeling Expert
Nome
R8
ResNet50
ResNet18
Image Generation
Random E Drrain
Random E Derain
Random E Derain
Random E Dreain
SRe"l.
Eval Model
ResNet18
ResNet18
ResNer50
ResNer18
ResNer50
ResNer18
ResNet18
IPC=1
0.6 (o.1
6.6 (04)
6.8 ma))
6.7 (44)
7.3 (0.2)
1.6 (..
(.9) (0.5)
10
3.9 (0.)
23.9 (0.4)
24.0 (0.4)
22,9 (u
23.9 (0.5)
25.8 (07)
21.3 ((
50
20.4
47.9
53.1 (0.5)
43,0 (u:
53.2 (0))
54.3 m6)
46.8 (02)
100
28.2 (0.3)
53.4 (0.5)
57.6 (0.4)
53.5 (.5))
58.3 (..5)
54.7 m2)
52.8 (n.3)
200
37.8 (0:5)
56.8 (14)
62.3 (0.0)
56.5 (0s)
62.7 (0.4)
57.7 (.)))
57.0 (0.4)
Full
ResNet18 69.8%
ResNet50 76.1%
Diarget { y:) where z;'s 's are input images and y, are labels. Similarly, we can denote the distilled dataset D Dayn { }. and to achieve dataset size reduction [D syn
<< [Ptarget|+
Dataset distillation problem can be formulated as a bi-level optimization problem 22
argmin C(0Psom; Dtarget)
s.t.
Dapn = argmin C(O; D Daya)
(1)
Pogn
By convention, the size of the distilled data set is often quantified in terms of images per class (IPC). The goal of this work is to study the role of ÿ in a controlled setting while fixing T.
Our first set of experiments comparing the performance of popular distillation methods with hard vs. soft labels (Figure[] show that soft labels are crucial for the performance of those distillation methods. On the other hand, the use of pretrained experts during distillation is also common. These two observations suggest an ablation study to evaluate the importance of soft labels in the context of dataset distillation. To this end, we introduce a simple baseline that "distills" datasets by selecting real (i.e., not synthetic) samples and soft-labeling them. Specifically, we randomly sample images from the training data and use pretrained experts to generate labels for them. The only hyper-parament is thus which expert to use. To generate diverse experts, we save checkpoints at each epoch and vary which checkpoint is used based on the data budget.
3.2 Benchmarking distillation methods against the soft label baseline
From the many dataset distillation methods that have been proposed, we select the top- cop-performing one from each of the three categories summarized in Section as a representative: Ra-BPTT (9) (bi-level optimization objective), MTT (4) (trajectory matching objective). and SR2 (3 (distribution matching objective). Among these, L(33L(33) is the only one that generalizes to ImageNet- A.33 for downsized comparison). L first generates images with distribution matching objective using pretrained expert. The same expert is then used to generate soft labels for these synthetic images. Specifically, LRe'L proposes generating 300 soft labels per prototype, where each soft label corresponds to unique augmentation to the image. We establish an additional baseline, which we denote as "CutMix augmented soft labels" since SRe"L directly derives the labelling method from from Shen and Xing (25). In this additional baseline, we remove the synthetic image generation process and replace with random images sampled from the training data. We then employ the same CutMix augmented label generation. Table IT shows that randomly sampled training images paired with soft labels yield performance comparable to distilled synthetic ones. Moreover, the storage cost of the 300 soft labels needed for SRe"L is at least x larger than that of the soft label baseline [22]. We also observe that when the labels are generated by teacher model with different architecture, the student still performs well. In other words, the soft label baseline meets the cross- arosttrrch generalizability requirement for dataset distillation.
without the need downsize it (see Appendix A.3
In addition to ImageNet, we benchmark the performance of these methods on smaller datasets, where they generally tend to perform better. We compare the soft label baseline against other methods on TinylmageNet, 100, and CIFAR-10, as shown in Table For these datasets, existing methods sometimes significantly outperform the baseline at very low data budgets (i.e., high compression rates). However, at higher data budgets, the soft label baseline again yields better or comparable performances. These results show that current methods still face issues scaling to larger data budgets.

---

Table 2: Benchmark SOTA methods against soft label baseline ("Sl baseline") on TinyImageNet, CIFAR-100 and CIFAR-10. For smaller datasets at small data-budget, SOTA distillation methods outperform the baseline. Scaling those methods to large data- data-budget remains a challenge.
Dataset
TiuyImageNe
CIEAR-100
CIEAR-10
Distill Method Ra-BPTT
MTT
DM
SI Baseline Ra BaTBPTT MTT
DM
SI Baseline
Ra-BPTT
MTT
DM
SI Baseline
IPC=1
20.1 (03) 8.8 m.) 3,9 8.2
7.6
35.3 may 24.3 ma) 11,4 03)
1..0
53.2 (07) 46.3 (53) 26.0 as
21.1 0.5
10
23.2
12.9 me
27.2
47.5 mm 40.1 m.
29.7. 03
34.3
69.4 (94)
65.3 (m 48.9 me
46.3 mx)
50
28.0 m3
35.6 eon
50.6 m0) 47.7 (2) 43.6 (m
47.1 m-
75.3 (k))
71.6 (aa) m
m1))
100
38.2 62
51.3 m.5
Full
ConvNet(F) 38.5%
ConvNet(F) 56.4%
ConnNer(F) 81.5%
ImageNet- 1K Soft Label Baseline ResNet18
TinylmageNet Soft Label Baseline ConvNetD4
5.0
IPC=1
Label Entropy
Student test Acc (%)
1PC=2
Avg soft label entropy
Student test acc (9)
Expert test acc (%)
IPC-5
20.
IPC-10
37
32
4.0
35
10.
IPC=100
5.
3.0
33
Expert
33
Student
15
20
25
30
35
40
45
50
2.5
2.0
L.5
1.0
0.5
Expert Lest Acc (%)
Avg softlabel entropy
Figure 2: Expert test accuracy v.s. student test accuracy v.s. soft label entropy. The quality of soft labels (measured by student accuracy) depends expert accuracy (leff) and label entropy (right).
Importantly, the soft label baseline is robust to random image selection and expert training seeds (Ap- pendix (..).. Using simple image selection strategy (cross-entropy) could improve the performance of the soft label baseline (Appendix B). However, the goal of establishing this baseline is not to achieve the best performance but to showcase the importance of soft labels for dataset distillation. In the next section, aim to understand why soft labels are important for learning from limited data.
4 Good Soft Labels Require Structured Information
Despite its embarrassing simplicity, the soft label baseline yields strong performances, suggesting that labels are an
-prrhaps the most important- aspect of data-efficient learning. Therefore, we next seek to understand why soft labels are so effective for learning. In Section 4.1 we start with a generation observation that expert epoch determines the optimal soft label. In Section 4.2 we develop understanding that the structured information encoded in soft labels is what allows models to learn from so little data. Finally, in Section 4.3 we establish an empirical knowledge-data scaling law and an extreme version of distillation learning with no (feature) data.
4.1 General observations from the soft label baseline
In establishing the baseline, we observe that the optimal expert to generate soft labels depends on the (distilled) data budget. Specifically, for smaller data budgets. it is often better to use labels generated by an carly-stopped expert, which we will refer to "expert at epoch xr." " As an example, Figure 2 visualizes how using expert at different epochs impact the student test accuracy for ImageNet-1 K.
As we train the expert for more epochs, two things in soft labels can change: the information being conveyed in softmax values (i.c., the softmax distribution itself), and the amount of information being conveyed (i.e., the entropy). In Figure we visualize the expert soft labels for randomly selected image from ImageNtt German Shepherd, with the corresponding label shown in red) at 4 different epochs. At epoch 0, the soft labels are randomly initialized. The expert at epoch 1 (optimal for IPC=1) considers "Bison" as top guess but only assigns 2% probability. At epoch 20, the top two guesses become "Brown bear" and "Bison." Finally, at epoch 50 (roughly optimal for IPC=10), the expert correctly assigns "German Shepherd" with the maximum likelihood. This anecdotal example shows that the information being conveyed change over the course of expert
5

---

TinylmageNet IPC=10
TinylmageNet, i-th Label Swap Test
(Expert Epoch, Temp) Grid
100
84
Relative perfomance (%)
70
2
274
26
16A
26-2
26:2
26.0
78.6
28.7
247
29
80
66
27.0
28.3
150
23.4
25.4
25.2
23.4
29.1
21
28 g8
550
23.4
23.1
29.5
215
58
Expert
27
54
23.3 32.0
30.2
25.0
20.0
29.6
21.6
40
50
26.
280
26-7
36.5
28.8
76.0
26.7
70.6
20.9 2016
IPC=50
PCC110
46
26.5
28.4
38.7
33.5
26.1
20.4
30.3
23.0
20
IPC=1
Swap i-th label (log2-scale)
32
38
1.0
20
28
296
2..
24.5
2.8
3.7
28.
4.1
2611 21
4.6
5.0
24
Temperature
Figure 3: Importance of i-th label by per- forming label swapping test. Swap the i-th label (sorted by softmax value) with the last la- bel. Top labels contain structured information and the non-top labels contain noise.
Figure 4: (Expert Epoch, Temp) grid search
Tiny ImageNet IPC=10. Temperature smoothing does not fully resolve the issue that later epoch experts yield sub-optimal labels for a given data budget.
training. Besides changes in the per-class information, the entropy labels also drop over the course of training. Eventually, the labels sharpen and converge to hard labels, meaning less information being conveyed. As a first attempt isolate these two factors (information, entropy), we look at experts that have been trained until convergence. Figure 2 (right) shows that the expert starts to converge and even slightly overfits. The test accuracy stabilizes around 38%, but the entropy of the labels continues to drop drastically. Despite similar performance by the expert, the later epoch experts generate labels that are significantly worse for the student model than the earlier epoch experts. the next section, further disentangle the contributions of these two sources of variation (information and entropy) to the quality of soft labels.
4.2 Structured information in soft labels and its importance for distillation
We first distinguish two components in soft la- bels structured information and unstructured noise. We define structured information as soft- max values that the student models learn from they contain information such as semantic simi- larities between classes (see Appendix C.I for visualization of information in soft labels). Un-
Soft labels for a German shepherd sample
0.012
Expert at Epoch
0.006
M m
Softmax Probabliity
Expert at Epoch 11
structured noise, on the other hand, contains no information but may provide regularization dur-
0.000
ing student training. The contribution of these
Expert at Epoch 20
0.080
two effects has only been studied in KD settings 34 (88. In data distillation, we aim quantify the percentage of softmax values that provide learnable information and the percentage that contains only noise. We also want to determine whether the optimal structured information is unique to each data budget.
0.400
Expert at Epoch 50
0.200
0.000
Class
Figure 5: Soft labels generated by expert at dif- ferent epochs. The structured information in soft labels changes over the course of training.
Label swapping test. To quantify the percent- age of softmax values that contain structured information, we design "i-th label swapping test." " This experiment relies on the assumption that if we sort labels by their softmax values in descending order, the last label likely contains no useful information. relative performance compared to unswapped labels. Refer to Appendix C.2 for a detailed experiment procedure. ( i-th label swapping test results on TinylmageNet. For all IPC budgets, swapping top labels significantly hurts the performance, indicating when learning with few data,
the experiment, we swap i-th label with the last label, and measure the

---

TinylmageNet Data-Knowledge Scaling Law
TinylmageNet Data- eota-Knowledge Scaling Law
14
(Expert at epoch 11)
Pareto Front
40
Student Accuracy (96)
12
Student Accuracy (96)
35
10
30
25
Soft label Keeping Top-K
15
Expert at epoch
<
Epoch
10
fpoch
Data
129
Epoch
Fittod
200
5
Epoch 60
Fitedd
K- 20
Handibal
Epoch 90
12 20
40
10 20
100 200
Dataset Size (IPC), Log10-scales
Dataset Size (IPC), Log 10-scaed
Figure 6: Data-Knowledge Scaling Law for TinyImageNet. Left: Trading knowledge (amount of information in soft labels) with data using one expert. Right: Establishing the Pareto-optimal front for data-efficient learning.
students rely more on structured information (knowledge) during learning. For 1 IPC, the top label does not contain as much information compared to higher IPC. It is likely because the top-1 label generated by an early-epoch expert, which itself has poor performance, does not contain much useful information.
Effect of temperature and epoch in soft labels. So far we have only observed that for given data budget, there is an optimal expert epoch for label generation (Figure 66) This observation does not imply that the structured information generated by this expert is indeed optimal. Specifically, we could not eliminate the possibility that labels generated by the expert at later epoch may contain better information but too sharp for the student to learn from (i.e., suboptimal entropy). To further disentangle the two factors (information and entropy) and understand the impact of label entropy on label quality, we use solution from KD: varying the temperature of the softmax value (13) to control label entropy. Specifically, we perform an extensive grid search on (Expert epoch, Temperature) under TinylmageNet IPC=10 setting, results shown in Figure 4 Increasing the temperature does further improve the soft label baseline performance, but the optimal epoch in the temperature=1 setting remains optimal even when temperature smoothing is used. From these two experiments, we conclude that students leverage structured information soft labels during learning, and the optimal information (expert) is unique to each data budget.
4.3 Trading data with knowledge
We have established that the smaller the data budget, the more heavily the student model relies on structured information present in the soft labels. In this section, we further explore the trade- off between data and knowledge two settings. First, we explore a scaling law to quantify how knowledge can reduce dataset size. Second, we further the idea of "learning from limited data" to "learning from data" for given class.
Scaling law. The data-knowledge scaling law experiment quantitatively measures how knowledge can reduce dataset size and the optimal knowledge for each dataset size. For the first experiment, we use an expert trained on TinylmageNet for 7 epochs (with test accuracy of 1 11%) as the teacher model. As in the baseline setting, the data (images) are randomly selected from the original training set. establish the empirical scaling law, we vary dataset sizes (measured by IPC) and the amount of knowledge in the soft labels. To control the amount of knowledge in the soft labels, we retain different top-k values (sorted by softmax probabilities in descending order) and zero out the rest without re-normalization. We compare the katalknowledge scaling law against the standard training scaling law (i.e., using hard labels from data).
The results of the knowledge the teacher model, the student needs around 10 IPC if full knowledge is present (k=200)) But the IPC value quadruples to 40 if we reduce & to 32. Moreover, learning from this expert becomes suboptimal as the dataset size increases, and students trained on hard labels start to outperform. This finding supports the observation from Section 4.2 with an increased data budget, the student benefits more from learning from a model at later epoch. In Appendix C.3 we repeat the knowledge-date trade-off
trade-ofi off experiment are shown in Figure To fully recover
7

---

TinylmageNet. Remove class images v.s. class label
20
20
20
IPC1
IPC10
IPC50
Control
Image Class
1.5
Remove Cluss
15
Remove Elass label
10
10
10
30
40
30
Student Accuracy (%) for Removed class
Figure 7: Zero-shot learning in the absence of knowledge v.s. data. The student model can achieve good performances when data is absent but much worse performances when knowledge is absent.
experiment using expert trained for later epochs. By combining experts at different epochs with different data budgets, we can establish a Pareto-optimal front for data-efficient learning, as shown in Figure Figure&rright and curve fit below. The use of expert knowledge provides constant 6x data size reduction since 'D) num class x IPC:
IPC
0.077
Hard Label: Student Test Acc
0.8
29.9
6.04 IPC
0.077
Soft Label: Student Test Acc =
0.8
29.9
Learning from (almost) no data. We have used the empirical knowledge-dee scaling law experi- ment to establish that one can directly trade knowledge with data. This observation inspires to push this idea to the extreme. For selected class, in which of the following two scenarios would the student perform better? The student learns in the complete absence of data from this class. 2) The student learns with data from this class but in the absence of knowledge.
For a selected class i, we design zero-shot experiment with a control and two treatment groups:
Control: The student model is trained with data from all classes and full soft label knowledge. Treatment - Remove image: We remove all training images from class i while keeping the softmax values corresponding to class in the rest of the training data.
Treatment 2 Remove label: We keep images from class and the full soft label for those images. For all other images in the training data that don't 't belong to class i, we zero-out softmax values corresponding class without re-normalization.
Figure shows the experimental results IinyImageNet under three IPC settings (classes sorted by control group performance). Compared to the control group, the performance of the remove-data treatment suffers only a little.This good performance implies that the student can perform zero-shot classification on the removed class without having seen any images from the class. In contrast, the student's performance suffers drastically when labels are removed. For IPC=1, the student model fails to learn anything at the remove-label setting (0% test accuracy for removed classes), and the performance of the remove-label treatment only starts to catch up IPC=50. This substantial performance difference between the two treatment groups suggests that the information conveyed in labels is more important than the data.
5
Soft Labels are Not Created Equal
So far have established that one way to achieve dataset distillation is by trading knowledge for data, specifically using knowledge from pretrained experts. In this section, we continue to work under the setting where images are randomly sampled from the training data, but we expand on different ways to obtain "knowledge." In Section 5.1 we show that an expert ensemble can consistently improve
soft label quality. In Section 5.2 distillation methods.
we discuss the possibility of obtaining soft labels through data
8

---

Table 3: Comparison of label generation strategies TinyImageNet. Ensemble provides
consistent performance improvement. Distillation method (BPTT) can adopted to learn labels.
Label Souree
Data
Expert
Distilled
Labeling Method
Hard
Single
Ensemble
BPTT
MTT
IPC=I
(.1)
(0.))
8.7 mm
13.5 (..)
2.2 m.
10
7.3 (.2)
27.2 (04)
30.1 (z2)
25.0
10.3 (2))
50
20.7
35.6 (04)
39.3
22.2 (05)
5.1 Expert ensemble
common strategy to boost predictive performance machine learning is to use ensemble predictors. By combining predictions from various models, ensemble methods effectively capture more comprehensive representation of the underlying data distribution. Therefore, we hypothesize that averaging soft label predictions from multiple experts might improve the quality of the labels. We train 10 ConvNet experts with different random seeds on TinylmageNet. labels, average the logits from all the experts for each of the randomly selected training images before passing them through the softmax calculation. For a fair comparison, we use the expert epoch previously determined in the soft label baseline. T shows that soft labels derived from expert ensembles consistently improve student model accuracy compared to single-expert labels.
generate ensemble soft
5.2 Learning soft labels through data distillation methods
Instead using pretrained experts to generate soft labels, natural alternative is learn labels through dataset distillation methods. The motivation is twofold: First, we aim to explore different approaches hopes of finding better ways to obtain labels. More importantly, we have demonstrated that using the knowledge-dee dowleege-distillatioo distillation framework (i.e., using pretrained experts) is sufficient mechanism to obtain informative labels. We now want understand whether the information generated by experts is the only (or necessary) solution.
Distribution matching-basee based distillation methods, one of the three predominant categories of success- ful distillation methods identified above, cannot be easily adapted to learn labels because they aim to generate synthetic images that minimize cross entropy on teacher model. This leaves two other possible families of methods: training trajectory matching and bi-level optimization. For both meth- ods, adopt the same training objective but freeze the images and learn the labels only.For For the training trajectory matching objective, we adopt MTT [4] ) and for the bi-level optimization objective, we adopt cruncated-BPTT [9] (also known as BPTT). We experiment with these two adaptations on Tiny Imageee,,,, with results shown in Table The MTT adaptation fails to meaningfully improve on the hard label baseline- Appendix D.I for detailed description of the MTT objective and discussion on why it fails. On the other hand, the BPTT adaptation yields meaningful results, even slightly improving ensemble labels in the IPC=1 setting.
To understand whether expert knowledge is nec- essary to obtain useful soft labels, we compare BPTT-learned labels with ensemble- mnsemble-generated labels. The motivation behind comparing BPTT labels is not due to their performance, but because BPTT addresses dataset distillation di- rectly from problem formulation (Eqn. D By directly solving the bi-level optimization prob- lem, no experts are trained during the distillation process. Moreover, it eliminates concerns about whether proxy distillation objective, such as training trajectory matching, may introduce bias into the distillation outcome. See Appendix [D.2 for detailed description of fruncated-BPPT algorithm and our adaptation to learn labels.
200 Normalized JSD for BPTT v.s. Ensemble labels
2k
1.0
PC-10
Norma JSD
Per
25
50
0
23
50
Expert Epoch
Expert Epoch
Figure 8: learned labels and ensemble-expert labels on TinyImageNet. Despite not training any experts, distillation method (BPTT) recovers the same la- bels as those generated by early stopped experts.
Normalized JSD between BPTT-
We fix the randomly sampled training images and generate soft labels using an expert ensem-

---

ble and using BPTT. We use ensemble labels instead of single expert labels to reduce bias. We use the -hasnnnS Distance (JSD) to quantify the distance between the softmax distributions. For each image, we also perform a min-max normalization across all expert epochs to identify the epoch at which the BPTT-generated label for that image is most similar to the ensemble-generann See Appendix [.2 for a detailed description of the comparison methodology. Figure X shows the normalized JSD for all images under the TinyImageNet IPC=1 and IPC=10 settings. For both data budgets, the minimizing epoch for the normalized JSD across all images is roughly the same (Epoch 13 for IPC=1, 25 for I0C=10). Conveniently, epoch 13 is fairly close to the optimal early-stop epoch for IPC=1 (optimal stop epoch 11). In Appendix D.2] we also compare raw JSD values to further ensure that the distributions are sufficiently close to each other on an absolute scale.
labels.
The two observations above provide strong evidence that BPTT recovers the same labels generated by the optimal carly-stopped expert, without explicitly training any. Note that by restricting to only learning labels, we limit the scope of the dataset distillation problem. However, in this restricted setting, the dataset distillation solution converges to the knowledge distillation solution (i.e., labels from an optimal early-stopped expert), indicating that expert knowledge is necessary to obtain informative labels. A future direction is to explore how this conclusion might change when images are learned simultaneously with labels.
Discussion and Conclusions
This paper highlights the crucial role of soft labels in dataset distillation. Through a series of ablation experiments and simple soft label baseline, we have shown that the use of soft labels is the main factor behind the success of state-of-the-an dataset distillation methods. We also demonstrated that structured information in soft labels is erucial for data-efficienn learning. We established an empirical knowledge-date scaling law to characterize the effectiveness of using knowledge to reduce dataset size and an empirical Pareto frontier for data-efficient learning. Additionally, showed that when only learning labels, data distillation converges to the knowledge distillation solution.
Implications for dataset distillation Our findings have several implications. First, they suggest rethinking existing data distillation methods that emphasize on synthetic image generation techniques, while under emphasize the importance of informative labels. Shifting the focus of distillation research to account for both of these factors is likely to lead to novel, potentially better, approaches. For example, in this work we only considered the setting where images are randomly selected from the training data, but future work may explore simultancous image/label synthesis methods to improve distillation performance. Future research may also investigate whether data distillation can be achieved without using expert knowledge, either implicitly explicitly, in a model-agnostic task-agnostic way. For example, future research may investigate how to synthesize images or labels by summarizing characteristics of the dataset, such as symmetries and other invariance priors.
Limitations and future work We have emphasized the importance of soft labels by implementing a simple soft label baseline. While this approach performs well on ImageNet-IK., advanced methods such as [12 24 continue to push the state of the art. This suggests that, while labels are crucial, suc- cessful dataset distillation requires leveraging both images and labels to achieve optimal compression. Future work should explore how best to optimize both components during distillation and examine how each uniquely influences student model learning. Additionally, investigating what other types of information, beyond expert knowledge, can be distilled for effective compression remains open research question.
On the label generation front, we have explored strategies based on existing methodologies, including pretrained experts and Ra-BPTT. Future research could further investigate more effective ways to generate informative labels that enhance model performance.
Lastly, like much of the dataset distillation research, our focus has primarily been image classifica- tion tasks. Although dataset distillation has demonstrated strong results in this area, extending these methods to other tasks and data modalities remains under-explored. While are generalizable, a limitation of this work is the narrow range of tasks evaluated. Future studies should explore how distillation techniques perform across broader set of domains and modalities.
believe our conclusions
10

---

Acknowledgments
Tian Qin and David Alvarez-Melis were partially supported by the Kempner Institute, the Aramont Fellowship Fund, and the FAS Dean's Competitive Fund for Promising Scholarship. We would also like to thank Clara Mohri and Eran Malach for their insightful discussions and helpful feedback throughout the development of this work. Their input has been invaluable in shaping the direction of our research.
References
[1] M. Abdin et al. "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone". arXiv CL] (Apr. 2024).
[2] J. Ba and R. Caruana. "Do deep nets really need to deep?" Adv. Neural Inf. Process. Syst. (Dec. 2013), pp. 2654-2662.
[3] O. Bohdal, Y. Yang, and T. Hospedales. "Flexible Dataset Distillation: Learn Labels Instead of Images". arXiv [cs.LG] (June 2020).
[4] G. Cazenavette et al. "Dataset Distillation by Matching Training Trajectories". arXiv [cs. CV] (Mar. 2022).
[5] G. Cazenavette et al. "Generalizing dataset distillation via deep generative prior". arXiv [cs.CV) (May 2023), pp. 3739-3748.
[6] J. Cui, Wang, S. Si, and C.-J. Hsich. "Scaling Up Dataset Distillation ImageNet- IK with Constant Memory". arXiv [cs, fcs.CV] (Nov. 2022).
[7] Z. Deng and O. Russakovsky. "Remember the past: Distilling datasets into addressable memories for neural networks". aurivv [cs.LG] (June 2022). Ed. by S. Koyejo et al., pp. 34391- 34404.
[8] J. Du et al. "Minimizing the Accumulated Trajectory Error Improve Dataset Distillation". arXiv [cs.LG] (Nov. 2022).
[9] Y. Feng, R. Vedantam, and J. Kempe. "Embarassingly Simple Dataset Distillation". arXiv
[cs.I.G] (Nov. 2023).
[10] J. Gou, B. Yu, S. J. Maybank, and D. Tao. "Knowledge Distillation: Survey". Int. J. Comput.
Vis. 129.6 (June 2021). pp. 1789-1819. "Textbooks Are All You Need". arXiv CLS (June 2023).
[11] S. Gunasekar
[12] Z. Guo et al. "Towards lossless Dataset Distillation via difficulty-alignedl trajectory matching".
arXiv [cs. CVJ (Oct. 2023).
[13] G. Hinton, O. Vinyals, and J. Dean. "Distilling the Knowledge in Neural Network". arXiv [LL (Mar. 2015).
[14] A. Hughes. Phi-2: The surprising power of small language models. https /// www microsoft com / en us rewearchhccch/ blog /hi 2 the surprising power of small- Saall-language-aae
Accessed: 2024-5-21.
[15] J.-H. Kim et al. "Dataset Condensation via Efficient Synthetic-Data Parameterization" arXiv [cs.LG] (May 2022).
[16] H. B. Lee, D. B. Lee, and S. J. Hwang. "Dataset Condensation with Latent Space Knowledge Factorization and Sharing". arXiv [cs (Aug. 2022).
[17] T. Maintainers and Contributors. Torch TorchVision: Vision: PyTorch's 's Computer Vision librury. 2016.
[18] G. A. Miller. "WordNet: lexical database for English". Commun. ACM 38.11 (Nov. 1995),
pp. 39-41.
[19] Müller, Kornblith, and G. E. Hinton. "When does label smoothing help?" Adv. Neural Inf. Process. Syst. abs/1996.022 026699 (June 2019).
[20] T. Nguyen, Z. Chen, and J. Lee. "Dataset Meta-Learning from Kernel Ridge-Regression"
arXiv [cs.LG] (Oct. 2020).
[21] A. Paszke et al. TPrCh::: An Imperative Style, High-Performance Deep Learning Library".
arXiv [cs.LG] (Dec. 2019).
[22] T. Qin, Z. Deng, and D. Alvarez-Melis. "Distributional Dataset Distillation with Subtask Decomposition" arXiv Lcs.LG] (Mar. 2024).
[23] N. Sachdeva and J. McAuley, "Data Distillation: A Survey". arXiv [cs.LG] (Jan. 2023).

---

[24] S. Shao et al. "Generalized large-scale data condensation via various backbone and Statistical Matching". arXiv /cs. CVJ (Nov. 2023).
[25] Shen and E. Xing. "A Fast Knowledge Distillation Framework for Visual Recognition".
arXiv [es.CV] (Dec. 2021).
[26] 1. Sucholutsky and M. Schonlau. "Soft-Label Dataset Distillation and Text Dataset Distillation".
2021 International Joint Conference on Neural Networks (IJCN)). IEEE, July 2021, pp. 1-8.
[27] Y. Tian, D. Krishnan, and P. Isola. "Contrastive Representation Distillation".. arXiv [cs.LG] (Oct. 2019).
[28] K. Wang et al. "CAFE: Learning to Condense dataset by Aligning FFatures". arXiv [cs. CV] (Mar. 2022), pp. 12196-12205.
[29] T. Wang, J.-Y. Zhu, Toralla,, and A. Efros. "Dataset Distillation".. arXiv [cs.LG] (Nov.
2018).
[30] W. Yang. Y. Zhu, Z. Deng, and O. Russakovsky. "What is Dataset Distillation Learning? arXiv [cs.LG] (June 2024).
[31] H. Yin et al. "Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion". arXiv
[cs.IG) (Dec. 2019).
[32] Z. Yin and Z. Shen. "Dataset Distillation Large Data Era". arXiv [cs.CV] (Nov. 2023).
[33] Z. Yin, E. Xing. and Z. Shen. "Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective". arXiv [cs. CVs (June 2023).
[34] L. Yuan al. "Revisiting Knowledge Distillation via label smoothing regularization" Proc.
IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (June 2020), pp. 3902-3910.
[35] B. Zhao and H. Bilen. "Dataset Condensation with Differentiable Siamese Augmentation". Proceedings of the 38th International Conference on Machine Learning. Ed. by M. Meila and Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 12674-12685.
[36] B. Zhao and H. Bilen. "Dataset Condensation with Distribution Matching". 2023 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV). IEBE, Jan. 2023, pp. 6514 6523.
[37]
Zhao, R. Mopuri, and Bilen. "Dataset Condensation with Gradient Matching". arXiv (cx. CV) (June 2020).
[38] H. Zhou et al. "Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Variance Tradeoff' Perspective". arXiv [cs (Feb. 2021).
[39] Y. Zhou, E. Nezhadarya, and J. Ba. "Dataset Distillation using Neural Feature Regression". arXiv [C) (June 2022).
12

---

A Soft label baseline
A.1 Experimental Details
For datasets including downsized ImageNet-1K, TinyImageNet, CIFAR-100, and CIFAR- 10 we employ the standard ConvNet architecture that is used by data distillation methods we benchmark against 4 36). We use 3 convolutional blocks for low resolution datasets (res:32 32, CIFAR-10, CIFAR- and 4 convolutional blocks for medium resolution datasets (res: 64 x 64, TinyImageNer, downsized ImageNet-IK3 For the original size ImageNet-IK, we use the PyTorch 21] official implementation of R 8 and ResNet50 to standardize comparison against the SOTA we benchmark against.
Models
methods 33
Expert training We follow standard training recipe to train experts on downsized ImageNet- IK, TinylmageNet, CIFAR-10, and CIFAR-100. This standard training recipe involves an SGD optimizer and a simple step learning rate schedule, and is used by distillation methods that require experts training in the distillation pipeline 4 6). Note that due to the simplicity of the training recipe and model architecture, the expert performance on the full dataset does not reach SOTA image classification performances. The expert training procedure is chosen to facilitate fair comparisons in the context of dataset distillation. For ImageNet- K SOTA methods (33) we benchmark against leverage Py ToTchh pretrained ResNet18 as their teacher model, therefore we adopt the PyTorch official training recipe [17] for expert training.
Evaluation To train student models on distilled dataset, we only allow learning rate tuning and we train student models until convergence. We train five student models with different random seeds and report mean and variance. For other methods we reported in Table 4 and 2 we report values from the original paper. For methods that we report performance without using soft labels (Figure I right), we use results reported in the original paper for MTT (T and FRePo [3)] For RT-BPIT () and SR (33). since they do not report results without soft labels, we acquire the distilled datasets and replace soft labels with hard labels. We perform the same evaluating procedure as above.
Compute All experiments are conducted on NVIDIA A100 SXM4 40GB or NVIDIA H100 80GB HBM3. To train small datasets, experiments typically require less than 5 GPU hours and for large datasets, experiments require at most 100 GPU hours.
A.2 Label generation and epoch tuning for existing methods
We provide further details regarding how labels are generated in FigureDrightt We compare our soft label baselines to four existing methods, each with their own soft label generation strategies proposed by the original authors. We apply the original soft label generation strategy used by each method. Overall, some of these methods already include epoch tuning (MTT/TESLA), while others do not LS) and for some, the concept of epoch tuning is not well-defined (Ra-BPTT and FRePo). In addition, we also compare the performance of the four proposed methods with or without soft labels. We clarify how hard labels and soft labels are obtained for each of the methods:
L The original method uses labels generated from fully trained expert without epoch tuning. In our reported results for SRe21,, we include a version of the soft label generation method that directly replicates the original method (detailed in Section 3.2). We also include version of the soft label generation method that is identical to the soft label baseline, which includes epoch tuning (detailed in Scction 3.1. To obtain the hard label counterpart, since SRe"L initializes its distilled images directly from training images, we use the original label of those images as the hard label.
MTT (TESLA) In the original method, the expert used to generate labels is already epoch-tuned, and the epoch also impacts the image generation pipeline. Therefore, we replicate the soft label generation strategy in the original method. To obtain the hard label counterpart, since MTT/TESLA initialize their distilled images directly from training images, we use the original label of those images as the hard label.
Ra-BPTT The original method generates labels along with images using a bi-level optimization objective, so no experts are trained during the process. Because labels are not generated by experts, epoch tuning is not applicable. We use the soft labels generated by the original method. In addition, we have experimented with using pre-trained experts to generate labels for Ra-BPTT generated images but observed that experts trained on real images perform poorly on images synthesized by Ra-BPTT. This is likely because the generated images are too distr--dittiiuution for experts trained on real training data. Thus, the original labels generated by Ra- BPTT should be considered optimal
13

---

for this method. Since Ra-BPTT images are initialized with random Gaussian noise, we use the argmax as hard labels.
FRePo Similar to Ra-BPTT, FRePo is based on Back-Propagation in Time (BPTT), and no experts are trained during the distillation process. Like Ra-BPTT, labels are learned during the distillation process along with images. We use the hard label results reported in the original paper.
A.3 Downsized ImageNet-IK results
There are three other methods that can scale to ImageNet-IK but a downsized version (reducing image resolution from 256 x 256 64 64) and only at small IPC values. We compare their performances against the soft label baseline in Table Among those methods, only Ra-BPTT can slightly outperform the soft label baseline, albeit by small margin. Scaling these methods to larger data-budget remains a challenge, and as a result, we truncate Table to IPC=2.
Table 4: Benchmark SOTA methods against the soft label baseline ("SI Baseline") on (downsized) ImageNet-IK. Other distillation methods can only scale downsized (64 x 64) ImageNet- IK and they do not significantly outperform the baseline.
ImageNet-1K (Downsized)
Distillation Method Ra-BPTT
FRePo
DM
SI Baseline
IPC=1
10.77 (0.4) 7.5 (0.5)
1.5 (0.1)
8.0 (.0
12.05 (0.5) 9.7 (0.2)
1.7 (0.1)
9.6 (0.3)
A.4 Variance from expert training seed and random image seed
The soft label baseline involves randomly selecting images from training data and use pretrained single expert generate soft labels for the selected images. We use 6 different random seeds for image selection to confirm that the soft label baseline is not sensitive to random image selection, shown in Table 5 Additionally, we confirm that the soft label baseline is no sensitive to experts training seeds. Specifically, we train six different experts using the same hyperparameters and only varying random seeds. In expert seed experiment, we fix the randomly selected images for the distilled dataset. Results shown in Table As above, student test accuracy is reported as the average of five runs with different random seeds.
Table 5; Image selection with different random seeds on IinylmageNet. Our soft label baseline is not sensitive random image selections.
IPC
Seed 1
Sced 2
Seed
Seed 4
Soed 5
Seed 6
7.7 (0.2)
7.5 (0.2)
7.3
7.0 (0:2)
7.2 (0.2)
7.5 (0.1)
10
26.8 (1))
26.8 ma)
27.3 (0))
26.6 (1))
26.9 (0))
27.0 (01)
50
36.1 (0.)
a5.
35.6 (04)
36.0 a.))
35.8 (03)
35.6 (..)
Table 6: Image selection with different expert training seeds on Tiny Our soft label baseline is not sensitive to experts with different training seeds
IPC
Seed I
Seed 2
Seed
Seed 4
Seed 5
Seed 6
7.6 (.:)
7.5 (..))
8.0 (0:)
7.5 (0:))
7.9 (0.1)
7.6 (0.1)
10
26.4 (04)
26.8
26.4 (03)
26.6 (02)
26.4 (u))
26.8
50
35.7 a.
35.9
36.0 (1))
35.7 (44)
35.7 (03)
36.2
A.5 Hyperparameters for expert training and soft label generation
We include the optimal expert epochs to reproduce results reported in Table Table For smaller IPC values, we train experts with a reduced learning rate so that when we save expert checkpoints, we get expert accuracy at more granular scale. Note that using smaller learning rate for expert training for implementation simplicity. Alternatively, one could use partial epoch checkpoints to achieve the same outcome.
14

---

Table 7: Hyperparameter list to reproduce soft label baseline results in Table 1 and Table 2
Dataser
Expert Archirecture
IPC
Expert I.R
Student LR
Expert Epoch
Expert Test Accuracy (%)
ImageNet
ResNer18
5 10-1
1 10
17.3
10
5 x 10-4
5 10
43
35.8
50
x 10
5 10
30
60,9
100
1 1x
5 10
62.8
200
10-1
2 10
62.8
ImageNet
ResNer50
5 10-4
1 10
15.4
5 10-4
5 10
31,0
50
1 10
10
66.6
1 10
5 10
68.6
200
1
X 10
73.3
TaageeageNet
ConNNetD4
X 10-2
1 10
19.2
I x 10-2
1 10
35.0
50
X 10-2
1x 10
38.1
100
1 10"2
1x 10
102
38.8
CIFAR-100
ConvNetD3
1
X 10"2
1x 10
13
33.1
10
X 10
1 10
36
47.6
50
X 10
1 10
125
52.7
100
X 10
1 10
125
52.7
CIFAR-10
ConNNeDD3
5 10
1 10
20
56,0
10
5 10
1 10
44
65.1
50
x 10
1 10
128
74.9
100
x 10
1 10
164
76.5
B Image selection using cross entropy
The soft label baseline involves the most naive way of selecting images from training data: random selection. We additionally show that using cross ross-entropy (CE) as a selection criteria further improves the data quality. For each class, we first use the optimal expert compute cross-entres cross-entropy values for each images in the original training set. We then divide the training images into 10 quantiles according to the CE values, where corresponds to "easiest" sample with lowest CE. We perform the CE-selection on TinylmageNet with IPC=1, 10, 50 settings, and report results in Figure 9 Using the "easiest" samples provide a small but consistent improvement (#x 1%) to random selection, while using the "hardest" samples hurts the performance much more.
JinylmageNet Image selection w/ Cross-Entropy
compared to random baseline
Random Selection Bas eine
IPC=1
Student Accuracy 3 (%)
1 IPC=50
-1
Cross Entropy Quantile
Figure 9: Selecting images with cross-entropy criteria further improves the soft label baseline performances. We select training images based cross-entropy scores using epoch-tuned experts, and report the relative student accuracy change when compared to random selection.
C
Additional Analysis Results
C.1 Soft label visualization
We visualize the soft labels for TinylmageNet training set generated by ConvNet expert at epoch 50 in Figure 10 For every image in the original training set, we use the expert to generate soft labels
15

---

and aggregate soft labels by the predicted class (i.e., argmax) with simple averaging. The diagonal (probability for the predicted class) is zeroed out so can see the structures non-top softmax values. Qualitatively, the clusters present correspond to WordNet ontology (18), some of which we hand annotate with their common ancestors WordNet.
TinylmageNet Expert at Epoch 50 Training data soft labels
Animal
16
14
Artifact
12
Average Softmax probability (%)
Top guess class (argmax)
10
Food
2
Non-top guess class (Average entropy=2.7)
Figure 10: Visualizing softmax probabilities for each class Tiny ImageNet Soft labels are generated by pre-trained experts, and they exhibit structures related to semantic similarity.
C.2 i-th label swapping test The experiment procedure is the following:
Use the optimal early-stopped expert to assign soft labels for given set of randomly samples training images. ii
Sort softmax probabilities in descending order
At each iteration i E [1, [C'], swap the i-th label with the last label.
iv
Train student model on the swapped label, and compute
Student Model Accuracy on swapped i-th label
Relative Performance
Student Model Accuracy on unswapped label
C.3 Data-knowledge Scaling Law
In Figure we repeat the same knwwlede scaling law Section 4.3 The comparison Figure 6 and Figure T1 shows that fully learn the expert model at later epoch, the student models need more data. example, to fully learn the expert model at epoch (e.g.. Figure (), with full knowledge, the student model needs less than 20 IPC. In contrast, fully learn the expert model
16

---

at epoch 46 (e.g. Figure I the student model needs almost 100 IPC. By combining scaling laws for different experts and for different data budgets, we establish the Pareto frontier for fata-efficient learning shown in Figure right.
TinylmageNet Data- Knowledge Scaling Law
(Expert epoch 16)
35
Student Accuracy (%)
30
25
20
15.
label Keeping Top-K
10
Hardiael
20
40
100
Dataset Size (IPC), LoglD-scaled
Figure 11: Empirical Pata-Knowledge Scaling Law with expert at epoch 46. We repeat the scaling law experiment in Section 11 using later-epoch expert. The comparison between two experiments shows that more data needed for the student to fully recover a later-epoch expert.
D Details on label learning with distillation methods
D.1 Learn label with MTT
MTT (4) distills data by a trajectory matching objective. Given network, the method first trains many experts on the full dataset, and saves the entire training trajectory (namely, the expert model parameters at each epoch). To learn the distilled data, student model is initialized from a random checkpoint of a random expert, and then the student model is trained for several iterations on the distilled dataset. The goal is to match the model parameters (after being trained on the distilled dataset) with the model parameters trained the full dataset a future epoch. We adopt Algorithm in the original paper (4) but simply freeze images after initializing them with random training data. Labels are initialized as hard labels according to the image class. During distillation, only labels are learned. We perform an extensive hyperparameter search sensitive parameters such as: N: number of steps the student model trains on the distilled data, M: the future expert epoch we compare student model parameters T: the maximum expert epoch initialize student model parameters with. Best results are reported in Table
TableB]show that MTT-adaptation only provides marginal improvements to the hard label baseline. When only images are distilled, the trajectory matching objective is suitable proxy to the bi-level dataset distillation objective. Nevertheless, the training dynamics student learning with soft labels could be quite different from the training dynamics of experts learning with hard labels. In other words, when soft labels are used train student models, is not guaranteed that matching experts' trajectory remains suitable proxy for dataset distillation's 's objective.
D.2 Learn label with BPTT
We Muncated-BPTT (7.9 solve the bi-level optimization problem detailed in Eqn. [ See Algorithm [ for details. Instead of learning both images and labels, we initialize images from randomly selected training data and freeze them during distillation and learn label only. The BPTT algorithm has relative high sensitivity to hyperparameters such as T: total number of unrolling steps, M: truncated window size. Additionally, one common pitfall to BPTT is exploding gradients. To combat the optimization challenge. we adopt the algorithm and the training recipe from Feng et al. [9]. We perform an extensive hyperparameter search and report the best results in Table
The comparison methodology is the following. To compare labels learned by BPTT and those generated from experts, we use the same data (1P)) to generate labels with either methods. We denote ensertee labels as {u/}. where / indicates the early-stopped epoch. We denote the BPTT-learred labels as BPTT). We use Jensen-Shee -hannon Distance (JSD) to quantify the distance between the soft labels generated by two methods.
17

---

Algorithm Learn soft label with BPTT
Require:
Target dataset Dtarget. T: total number of unrolling steps. M: truncated window size. or student network learning rate. Or2: distilled data(label) learning rate. L: loss functiont(ross-entrooyy for classification)
Randomly sample images from {ar H D Initialize soft label with hard label. Distilled dataset: Dayn faz, Vs Vim
while Not converged do
Sample batch of data drarget Dtarget Randomly initialize student network parameter Oo
for 0
do
if If n = T M then
Start accumulating gradients
end if' Sample mini-batch of distilled data dayn Dayn Perform gradient update on student network parameters a.l1 On 011 V V.(ddyn's fn) end for
Compute loss target data L(dearget. (N) Perform gradient update on soft label { + {ui) - 02 4 (( On) end while
For each image 24, we compute JSD(), vt) for all l. We perform min-max normalization across epochs:
uy BPTT
v ming (y BPTT y y
Normalized JSD( BPTT v//)
max; JSD BPTT
min; JSD( (y.
RPTT
In addition to the normalized JSD) shown in Figure Figure 12 shows the raw JSD comparisons between BPTT learned labels and ensemble-cxpert tagged labels. build a baseline reference, we compute JSD values between soft labels generate by four single experts (only differ by training seeds) on the same set of images. For IPC=1, the JSD between BPTT learned labels and ensemble labels have slightly higher JSDs than the expert pairwise distance. For IPC=10, the JSD values fall into the same distribution. The raw JSD comparison further suggests that BPTT generated labels recovers a softmax distribution that is very similar to ensemble labels.
Similar to standard distillation, BPTT might suffer from the same scaling problem. In other words, it yields suboptimal performance when distilling on larger IPCs. We suspect that optimization might be the cause behind BPTT learned label has worse performance than the best ensemble-expert labels at IPC=10.
Export
5D Expert spoch 25
Duert 1/ Zopert
Coar spocy 1RT1 SharEs
700
600
500
200
100
22
0.0
Figure 12: JSD between BPTT-learned labels and expert-ensemble labels JSDs between BPTT and expert-ensemble labels fall into the same distribution as JSDs between labels generated by two random experts.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes)
Justification: The contributions and scope summarized in abstract can be found in Section 3, Section 4 and Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Section discuss limitations. In each of section 3, and 5, we also discuss specific limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
19

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include any theoretical result this paper.
Guidelines:
The answer NA means that the paper does not include thcoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix A includes experiment details reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
20

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis-
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code and data for all experiments are available online.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce cc/ hooola/cgiiiee//ddddllllssssssso
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //npps.cc/pibbl//ggiiieee/////// wwwpg//dddeeeeeeebiiiieeeeeoeeee Poliicyy))) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Appendix A includes experiment details to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are reported and the source of errobars are detailed in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
22

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative emror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A details the compute needed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp////aarrrr....///////////d
Question:
Answer: [Yes]
Justification:
We conform with NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Question:
Answer: [NA]
Justification:
No immediate societal impact.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
24

---

Answer: [Yes]
Justification: Code, data and models are cited when applicable.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, thcoee thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/coodemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "gDQdMv52Xh", "reviewer": "Reviewer_pAYe", "summary": "This paper analyses the role of soft labels used in dataset distillation. Experiments with different ablation studies show that the performance of soft labels based data distillation approaches is primarily attributed to the use of soft labels. Secondly, the authors study the various types of soft labels and their effect on model performance. Additionally an empirical scaling law is provided to characterize the relation between effectiveness of soft labels and image per class in distilled dataset.", "questions": "I am concerned about the use of experts at tuned epoch for comparison of soft label baseline with previous methods. Are the expert epochs also tuned for previous approaches? \n\nCan the authors provide insight as to why structured information is more beneficial in soft labels in the case of dataset distillation while the opposite is true for knowledge distillation as mentioned in related work?\n\n It is unclear how past approaches in figure 1 right  (which as I understood to originally use soft labels) were adapted to hard labels. Is it done by performing argmax on the soft labels?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is written clearly with well-presented motivation and is easy to follow\n\nExtensive analysis and ablations are presented providing a better understanding of role of labels in dataset distillation\n \nThe paper focuses on a largely overlooked aspect of dataset distillation methods: the degree of contribution of soft labels.", "weaknesses": "The paper could benefit from a theoretical analysis of why soft labels are so effective. Additionally, the generalizability of the findings to data distillation in other modalities could be insightful.\n\nMinor comments: \n\n1. The description in line 222 appears inconsistent with results in figure 4. IPC=1 appears more robust to swapping of top labels compared to IPC=10\n\n2. A few writing and grammar errors exist in related work, line numbers 87-90. Also, 'generation' should be 'general' in line  167\n\n3. In Figure 2, it might be better to use the same dataset to highlight the dependence on expert accuracy and label entropy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! We are glad that you found our work interesting. We respond to their specific comments below:\n\n__Response to weakness__\n\n> *The paper could benefit from a theoretical analysis of why soft labels are effective. The generalizability of the findings to data distillation in other modalities could be insightful.*\n\nSoft labels are effective because compared to their hard label counterparts, they contain probability for each class for a given image. Those logits not only convey the class information for the given image but also contain other information such as class similarities. While it can be challenging to study the theoretical framework for a large ConvNet on an image classification task, we believe one can demonstrate the effectiveness for smaller convex optimization problems, such as simple regression. Specifically, one can show how providing logits can reduce sample complexity to learn the same classifier. We believe formalizing this form from a learning theory persepective is an exciting avenue for future work.  \n\nTo date, almost all dataset distillation work has focused exclusively on image classification tasks. To our knowledge, the only work extending this line of research beyond image classification is [1], where the authors explored dataset distillation for vision-language models. We believe that the conclusions drawn from our work, particularly the importance of labels, should apply to other vision tasks and classification tasks in other modalities. As future dataset distillation research begins to explore other modalities, we hope the soft label baseline we established will serve as a strong starting point for future work. We acknowledge that our work could benefit from additional results in other modalities, and we are currently considering including language modeling tasks for the camera-ready version.\n\n> *The description in line 222 appears inconsistent with results in figure 4. IPC=1 appears more robust to swapping of top labels compared to IPC=10*\n\nThank you for pointing out the mistake. Indeed, the description in line 222 is not entirely accurate. If we only consider $i=1$ in the \"Swap i-th label\" test, IPC=1 is more robust to swapping top labels compared to IPC=10. However, when we consider $2≤i≤32$ (swapping top-k labels for  $k>1$), IPC=1 relies more on top-k labels than IPC=10, and thus is less robust to swapping top-k (k>1) labels than IPC=10. We will ensure this clarification is made in the camera-ready version.\n\n__Response to questions__\n> *I am concerned about the use of experts at tuned epoch for comparison of soft label baseline with previous methods. Are the expert epochs also tuned for previous approaches?*\n\nThank you for the question! We address this concern in the global response with new plots included in the PDF attachment. \n\n> *Can the authors provide insight as to why structured information is more beneficial in soft labels in the case of dataset distillation while the opposite is true for knowledge distillation as mentioned in related work?*\n\nYou have pointed out a very interesting contradiction! Our understanding is that the key difference between data distillation and knowledge distillation lies in data budgets. In knowledge distillation, the student network has full access to the entire training dataset, along with soft labels generated by the expert (teacher). Conversely, in the data distillation setting, we impose a strict limitation on the size of the data. As a result, the student network must rely more heavily on the labels generated by the teacher. Therefore, it relies more on structured information in data-poor settings.\n\n> It is unclear how past approaches in figure 1 right (which as I understood to originally use soft labels) were adapted to hard labels. Is it done by performing argmax on the soft labels?\n\nFor SRe2L, MTT, and FRePo, the distilled images are initialized with real images from the training data. To obtain hard labels, we use the label based on the “intialization” image. Specifically, in MTT (TESLA) and FRePo, the original works include comparisons of hard versus soft labels, and they both chose to use hard labels in the same way (i.e., hard labels based on the initialization image). We maintained the choice made by the original authors for consistency and reproducibility. However, we suspect that the outcome would be the same if we used the argmax of the soft labels to obtain hard labels, as each distilled image is supposed to be representative of its respective class.\n\nFor Ra-BPTT, where images are initialized with random Gaussian noise, we use the argmax of the soft labels as the hard labels.\n\n[1] Vision-Language Dataset Distillation https://arxiv.org/abs/2308.07545", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5FFznHyMF", "reviewer": "Reviewer_UKPz", "summary": "This paper introduces soft probabilistic labels to the dataset distillation task. Specifically, it finds that the labels should consider structured information and perform unequally. Experiments on diverse datasets demonstrate its effectiveness.", "questions": "1) In the Method section, could different soft-labeling strategies (excluding cutmix) influence the entire pipeline?\n2) Does early stopping result in varied performances across different baselines and datasets? How do the authors address this issue?\n3) In the experiment, the author uses the swap test to validate the structure information and claims “Top labels contain structured information and the non-top contain unstructured noise”. Why not just remove the unstructured noise?", "limitations": "1) The writing requires improvement as the author attempts to cover too information without adequately establishing connections and justifying the necessity of the key technologies.\n2) The tables and figures need reorganization to enhance clarity. Reviewer currently struggles to understand their intended messages.\n3) Some typos need to be corrected, such as those in the representation of Figure 4.\n4) The limitations mentioned in the Conclusion are not clearly stated.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) This paper proposes the introduction of soft labeling in dataset distillation and provides an interesting analysis of the intrinsic properties of data distillation, rather than focusing on improving the base module of specific methods.\n2) The method is well-supported, and the results are reliable.", "weaknesses": "1) The motivation needs to be stated more clearly. Why are label-level methods regarded as superior to image-level methods？\n2) In the introduction, the author introduces several techniques such as 'expert' models, Pareto frontier, and knowledge-data scaling laws but lacks detailed explanation. \n3) Section 3.2 requires reorganization to enhance readability.\n4) The structure information needs to be expressed more clearly and intuitively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! We respond to their specific comments below:\n\n__Response to Weakness__\n > *The motivation needs to be stated more clearly. Why are label-level methods regarded as superior to image-level methods?*\n\nTo our best understanding, the dataset distillation community does not consider label-level methods superior to image-level methods. In fact, it appears that more work has been dedicated to image-level methods, often treating labels as a minor additional “trick” to boost performance.\n\nThe main motivation behind our study is to emphasize the importance of soft probabilistic labels in the dataset distillation task. We aim to correct this misconception and argue that labels are, in fact, quite central to data distillation. Overall, we do not believe that label-level methods are necessarily superior to image-level methods. We hope our work inspires future research to explore methods that effectively utilize both images and labels to achieve the best outcomes.\n\n> *In the introduction, the author introduces several techniques such as 'expert' models, Pareto frontier, and knowledge-data scaling laws but lacks detailed explanation.*\n\nThank you for the constructive feedback! Our introduction will indeed be clearer with a more detailed explanation of key terms when we first introduce them. We will incorporate your suggestions in the camera-ready version for better readability. We also included a detailed explanation in Official comment. \n\n> *The structure information needs to be expressed more clearly and intuitively.*\n\nWe define structured information as the softmax values in logits that contain semantic information, such as class or feature similarities. These are also referred to as “dark knowledge” [1] or “supervisory signals” [2] by the knowledge distillation community. For example, a soft label for a goldfish image may assign a 70% likelihood to the “goldfish” class and a 10% likelihood to the “orange” class. The 10% likelihood assigned to “orange” is not noise; it indicates that the two classes are similar, likely due to color. We consider these logits to contain “structured noise.” We expect this pattern (goldfish-orange) to appear consistently in many goldfish images.\n\nConversely, if the model randomly assigns a 2% likelihood to the “pizza” class due to spurious features or other noise, we do not expect all goldfish images to have a consistent pattern of being slightly identified as pizza. We consider this as “unstructured noise.”\n\n\n__Response to Questions__\n\n> *In the Method section, could different soft-labeling strategies (excluding cutmix) influence the entire pipeline?*\n\nYes, different labeling strategies can impact the distilled outcome. In this paper, we employ a simple labeling strategy, which involves using experts trained at different epochs as the labelers to generate labels. Specifically, our soft label baseline uses this expert-based soft labeling strategy on randomly selected training images. We demonstrate that this simple baseline achieves performance comparable to state-of-the-art distillation results.\n\nIn Section 5, we explore a few alternative labeling strategies and show how these strategies impact the distillation pipeline and outcomes, still within the setting where images are randomly selected from the training data. We hope that future work will further explore and develop labeling strategies to improve the dataset distillation pipeline.\n\n> *Does early stopping result in varied performances across different baselines and datasets? How do the authors address this issue?*\n\nBased on the dataset and the data budget (IPC), the optimal early stopping epoch varies, as shown in Figure 6 (right). In this figure, we vary both the expert epoch and the data budget. For larger data budgets (IPCs), it is more effective to use a later epoch to generate labels. From this observation, we understand that the optimal information stored in soft labels can differ significantly depending on the data budget.\n\nWe believe it would be an interesting follow-up for future work to study how one can predict the optimal epoch given a data budget for different datasets. In this work, we aim to shed light on the fact that early stopping plays an important role in determining the quality of labels. For instance, in Figure 2 (left), we demonstrate this effect on ImageNet-1K.\n\n> *In the experiment, the author uses the swap test to validate the structure information and claims “Top labels contain structured information and the non-top contain unstructured noise”. Why not just remove the unstructured noise?*\n\nIndeed, we completely agree that this is a valid suggestion. By removing unstructured noise, we might further improve the performance of our soft label baseline. In our analysis, we also identified additional modifications to our soft label baseline for boosting performance, such as ensembling (Section 5.1) and temperature smoothing (Section 4.2). Practitioners could benefit from these additional enhancements, potentially including the removal of unstructured noise.\n\nIn our reported soft label baseline, we did not include any of these add-ons because we aimed to keep our baseline as simple as possible. Our goal was to demonstrate that soft labels are crucial for dataset distillation.\n\n\n__Response to Limitations__\n\n> The limitations mentioned in the Conclusion are not clearly stated. \n\nThank you for pointing out that our work could benefit from a more in-depth exploration of limitations. We appreciate this valuable feedback. Here is a brief version of what we intend to add to our conclusion to further address these limitations:\n\n> Writing and Presentation\n\nWe will carefully revise writing and presentation for the camera ready version. \n\n[1] Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531\n\n[2] Rethinking soft labels for knowledge distillation: A Bias-Variance Tradeoff Perspective https://arxiv.org/pdf/2102.00650", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHVheECA5H", "reviewer": "Reviewer_hrBt", "summary": "This paper investigates the importance of soft labels for dataset distillation methods, conducting detailed ablation experiments on the role of labels under various settings. It deeply explores the impact of labels on learning, providing an in-depth analysis and study of the intrinsic properties of labels. The work also provides empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establishes an empirical Pareto frontier for data-efficient learning.", "questions": "Please refer to weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper clearly demonstrates the crucial role that soft labels play in the effectiveness of data distillation methods. This aspect has never been carefully studied in previous dataset distillation work; it was usually considered an additional trick and did not receive much attention. This work points out new directions for data-efficient learning.\n2. The designed experiments are interesting and comprehensive, presenting what constitutes good soft labels, the importance of knowledge (soft labels) in learning, and how to effectively obtain higher-quality soft labels.", "weaknesses": "1. According to Table 7, the expert model (epoch) used to produce soft labels in the soft label baseline appears to be carefully selected. Does the choice of epoch introduce additional costs? Additionally, do the other mentioned dataset distillation methods also use the best soft labels from these epochs to ensure a fair comparison?\n2. MTT and SRe2L are not the SOTA methods currently. I would like to know if more advanced methods like DATM [1] and G-VBSM [2] still heavily rely on soft labels and whether their performance still can not surpass the soft label baseline.\nMinor: line 185 In Figure 3 left.\n\n[1] Towards lossless dataset distillation via difficulty-aligned trajectory matching. ICLR 2024\n\n[2] Generalized large-scale data condensation via various backbone and statistical matching. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive comments and thoughtful insights. We respond to their specific comments below:\n\n__Response to Weakness__\n\n> *According to Table 7, the expert model (epoch) used to produce soft labels in the soft label baseline appears to be carefully selected. Does the choice of epoch introduce additional costs?*\n\nYes, tuning the expert epoch is used for our soft label baseline, and it does come at a cost. Other SOTA methods spend the majority of compute on generating images. For example, Ra-BPTT requires more than 100 GPU hours to generate distilled images for the results shown in Figure 1. In contrast, the soft label baseline simply uses randomly selected training images, incurring no costs on image generation or selection.\n\nFor methods like FrePo and SRe2L, generating the best images also requires extensive hyper-parameter tuning in their distillation pipelines. Therefore, we do not believe the additional cost incurred for epoch tuning will be the bottleneck.\n\nTable 7 might give the impression that the epoch needs to be “carefully” selected. However, we hope these hyperparameters will ensure easy reproducibility. In Figure 6 (right), we showcase how to choose the “optimal expert epoch” by establishing a Pareto frontier. From this figure, our understanding is that we can establish a robust Pareto front with only five expert epochs, covering the optimal expert for IPC values ranging from 1 to 200.\n\n\n> *Additionally, do the other mentioned dataset distillation methods also use the best soft labels from these epochs to ensure a fair comparison?*\n\nThank you for the question! We address this concern in the global response with new plots included in the PDF attachment. \n\n> *MTT and SRe2L are not the SOTA methods currently. I would like to know if more advanced methods like DATM [1] and G-VBSM [2] still heavily rely on soft labels and whether their performance still can not surpass the soft label baseline.*\n\nThe field has been advancing rapidly. DATM introduces innovation to MTT based on “difficulty-level” trajectory matching. They claim that by using late trajectories, they improve performance for larger synthetic sets (i.e., large IPC) and achieve lossless distillation (i.e., matching distilled data to the quality of training data). Section 4.3 in DATM shows that they also heavily rely on soft labels. For TinyImageNet, they achieve 39.7% “lossless” test performance with IPC=50. Our soft label baseline achieves 35.6%. While the soft label baseline underperforms compared to DATM, the comparison still reflects the great importance of labels.\n\nSimilarly, reading Section 4.1in G-VBSM, our understanding is that G-VBSM also heavily relies on soft labels. Several improvements they leverage, including ensemble techniques, align with our observations. Since G-VBSM builds on SRe2L and the soft label baseline performs on par with SRe2L, G-VBSM is able to achieve further improvements.\n\nWe do not aim to outperform all SOTA methods using such a simple baseline, instead we hope to bring some understanding of what information is being distilled that leads to data-efficient learning and correct the misconception that labels are less important compared to images in dataset distillation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PyTLx6FlHF", "reviewer": "Reviewer_Zv9o", "summary": "The paper studies the effect of synthetic image soft labels on training performance, and show that the success of DD methods is attributed to the use of informative labels. The authors showed that the structured information in soft labels is important, and there is a tradeoff between knowledge and data. Generally, the paper conducted extensive ablations to analyze the synthetic soft labels.", "questions": "See Weaknesses.", "limitations": "Yes, limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper studies the interesting and novel aspect of the role of synthetic labels in DD.\n2. Thorough experiments are done and the comparisons are fair and meaningful.\n3. The experimental results and analysis provide important and interesting insights that are beneficial for the DD community.", "weaknesses": "I do not observe major weaknesses of the paper. However, I do have some minor concerns that I would like to discuss with the authors:\n\n1. Observing Figure 4, it is interesting that when swapping the top-1 label with the last one, the relative performance can still be even preserved to relatively 20% and even 50% (IPC=1). The top-1 label should be the correct prediction, and swapping makes all images wrongly labeled. I wonder why there is still 20%~50% relative performance when all the training data are wrongly labeled.\n2. For the experiments corresponding to Figure 7 (treatment 2), why no re-normalization? This results in probabilities that do not sum up to 1 and may cause unexpected consequences during training. \n3. It seems that there are two types of labels used for analysis. One type is directly downloaded synthetic dataset (image-label pairs) in section 4. Another type is generating soft-labels via ensembling in section 5. Does the second guarantee that the ensembled soft-labels are correct (same one-hot encodings as the original pairs)? Also, using early-stopping experts have drawbacks that it does not provide correct label information (e.g. Figure 3). To what extent (how early) may the experts be useful for label generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s constructive comments and are glad they found our work interesting. We respond to their specific comments below:\n\n__Response to Weakness__\n> *Figure 4, regarding performance gain when data are wrongly labeled*\n\nThe argmax certainly contains a lot of information. However, our intuition regarding why replacing such an important logit with a wrong one lies in the fact that not all useful information is stored in the top-1 prediction. For example, for IPC=1, the labels are generated with an expert who has only trained for 7 epochs on the real data. As shown in Figure 3, panel 2, the argmax at epoch 7 only has a 2% probability. In fact, the top 5 predictions all share roughly the same likelihood. These top logits contain semantic information such as class feature similarities (see Figure 10 in the Appendix). Therefore, even with the top-1 label being incorrect, the rest of the logits still contain useful and correct information for the model to learn from.\n\n> *For the experiments corresponding to Figure 7 (treatment 2), why no re-normalization?*\n\nIt is a perfectly valid concern regarding re-normalization. During our experiment design, we carefully weighed the pros and cons of using normalization for label treatment. The rationale behind not using re-normalization is two-fold.\n\nFirst, as shown in Figure 2 (right), label entropy significantly impacts final model performance. By using re-normalization, we would alter the label entropy of the treatment group. Second, for each training input, re-normalization only impacts the gradient with a constant scalar shift. Therefore, we believe that whether we perform re-normalization should not matter.\n\nTo eliminate the possibility that using un-normalized logits causes downstream effects, we re-ran the treatment 2 group with re-normalization and compared it against the original results (no re-normalization). The results, shown in the attached PDF, confirm that re-normalization does not make a statistically significant difference to the experimental outcomes.\n\n\n> *It seems that there are two types of labels used for analysis. One type is directly downloaded synthetic dataset (image-label pairs) in section 4. Another type is generating soft-labels via ensembling in section 5. Does the second guarantee that the ensembled soft-labels are correct (same one-hot encodings as the original pairs)?*\n\nIn Section 3, for the SOTA methods we compare our soft label baselines to, we obtain soft labels directly from the synthetic dataset (i.e., downloaded). In our soft label baseline, the images are randomly sampled from the training data, and labels are generated by experts with epoch tuning.\n\nIn Section 5, we further demonstrate that ensembling can bring additional improvements to the soft label baseline. We do not impose the restriction that ensemble soft labels are correct. In other words, no restrictions are imposed to ensure that the argmax of the ensemble labels corresponds to the “correct” class for that particular image.\n\nTo our understanding, correctness is less important in the dataset distillation setting, which we discuss further below.\n\n\n> *Also, using early-stopping experts have drawbacks that it does not provide correct label information (e.g. Figure 3). To what extent (how early) may the experts be useful for label generation?*\n\nYou have pointed out a very counterintuitive phenomenon! Indeed, early experts generate labels that are “incorrect.” Here, we can loosely define “incorrect labels” as those whose argmax does not lead to a correct classification. In Figure 6 (right), we used experts trained until various stages (from early to late) as the labelers for different data budgets (i.e., different IPC values). We observe that on TinyImageNet, experts as early as Epoch 11 could be useful for label generation under small data budgets (IPC=1). As we increase the data budgets, it becomes more optimal to use later experts.\n\nOur intuition regarding why incorrect labels are more optimal, especially under low data budget regimes, is that when the student model is learning with a limited amount of data, it benefits from mimicking behaviors from a less well-trained teacher, learning only simple features and functions. For example, the student network may learn features/functions that can distinguish between broad categories like dogs (corresponding to many classes in TinyImageNet) and vehicles (also corresponding to many classes in TinyImageNet). Such a crude classifier is the best the model can achieve given the data limit. In other words, with so little data, the student network learns a simpler function because it does not have enough data to learn a complex decision boundary to distinguish between a golden retriever and a German Shepherd. In these “data-limited regimes,” a less well-trained teacher provides better guidance for the student model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for the detailed and thoughtful response! We are glad that reviewers have found our work interersting and scientifically sound. We have carefully read through all the comments and we believe that all your feedback will bring improvements to our work! 

We address some common questions below.

__Soft label generation with epoch tuning__

Among all the reviews, we noticed a common theme regarding experiment details for Figure 1, where we compare our soft label baseline with SOTA methods. Specifically, there are questions regarding how soft labels are generated for SOTA methods. We provide further details below, and the new experiment results can be found in the attached PDF.

We have compared our soft label baselines to four existing methods, each with their own soft label generation strategies proposed by the original authors. In Figure 1, we applied the original soft label generation strategy used by each method. Overall, some of these methods already include epoch tuning (MTT/TESLA), while others do not (SRe2L), and for some, the concept of epoch tuning is not well-defined (Ra-BPTT and FRePo).

To address your concern, we have also included a version of Figure 1 where we apply the exact same label generation strategy used for the soft label baseline (expert label generation with epoch tuning) on SRe2L. Further analysis is provided in the supplementary PDF.

We clarify how soft labels are obtained for each of the methods below:

* SRe2L: The original method uses labels generated from a fully trained expert without epoch tuning. Therefore, in our reported results for SRe2L, the soft labels are not epoch-tuned.

* MTT (TESLA): In the original method, the expert used to generate labels is already epoch-tuned, and the epoch also impacts the image generation pipeline.


* Ra-BPTT: The original method generates labels along with images using a bi-level optimization objective (Eq 1), so no experts are trained during the process. As labels are not generated by experts, epoch tuning is not applicable. We experimented with using pre-trained experts to generate labels for Ra-BPTT generated images but observed that experts trained on real images perform poorly on Ra-BPTT generated data. This is likely because the generated images are too out-of-distribution for experts trained on real training data. Thus, the original labels generated by Ra-BPTT should be considered optimal for this method.


* FRePo: Similar to Ra-BPTT, FRePo is based on Back-Propagation in Time (BPTT), and no experts are trained during the distillation process. Like Ra-BPTT, labels are learned during the distillation process along with images.

We included a revised version of Figure 1 in the PDF. Our additional results show that (1) soft label strategy used by the original method is more effective for the given method than epoch tuning (2) our soft label baseline remains competitive. Please refer to the supplementary PDF for further details.

__Figure 7 revision__
In addition, we included a revised version of Figure 7 (data-efficient experiment with zero shot learning). In this new figure, we address the concerns regarding the softmax re-normalization raised by reviewer Zv9o.Please refer to the supplementary PDF for further details.

Meta-review: The manuscript has been reviewed by four reviewers. All of the reviewers, after rebuttal, gave positive ratings (including borderline positive).

Essentially, the reviewers found the study well-motivated, clearly written, and sufficiently verified by experiments.

The AC agrees with the consensus, and recommends the manuscript to be accepted.

Congrats!
Decision: Accept (poster)