Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?
Authors: Lingao Xiao
Yang He
Venue: NeurIPS 2024 poster (2024)
Keywords: Dataset Condensation
Dataset Distillation
Primary area: other

Abstract: In ImageNet-condensation, the storage for auxiliary soft labels exceeds that of the condensed dataset by over 30 times.
However, are large-scale soft labels necessary for large-scale dataset distillation?
In this paper, we first discover that the high within-class similarity in condensed datasets necessitates the use of large-scale soft labels.
This high within-class similarity can be attributed to the fact that previous methods use samples from different classes to construct a single batch for batch normalization (BN) matching.
To reduce the within-class similarity, we introduce class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes.
As a result, we can increase within-class diversity and reduce the size of required soft labels.
A key benefit of improved image diversity is that soft label compression can be achieved through simple random pruning, eliminating the need for complex rule-based strategies. Experiments validate our discoveries.
For example, when condensing ImageNet-1K to 200 images per class, our approach compresses the required soft labels from 113 GB to 2.8 GB (40$\times$ compression) with a 2.6\% performance gain.
Code is available at: https://github.com/he-y/soft-label-pruning-for-dataset-distillation

Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?
Lingao Xiao and Yang He
CCAR, Agency for Science, Technology and Research, Singapore IHPP, Agency for Science, Technology and Research, Singapore
'National University of Singapore xiao alao_lingaouu.om mus edu, he_yangScfar a-star edu. sg
Abstract
In ImageNet--ondeesstionn, the storage for auxiliary soft labels exceeds that of the condensed dataset by over 30 times. However, are large-scale saft labels necessary for large-scale dataset distillation? In this paper, we first discover that the high within-class similarity condensed datasets necessitates the use of large-scale soft labels. This high within-class similarity can be attributed to the fact that previous methods use samples from different classes to construct a single batch for batch normalization (BN) matching. To reduce the within-class similarity, we introduce class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes. As a result, we can increase within-class diversity and reduce the size of required soft labels. A key benefit improved image diversity is that soft label compression simple random pruning, climinating the need for complex rule-based strategies. Experiments validate discoveries. For example, when condensing ImageNet- K to 200 images per class, our approach compresses the required soft labels from 113 GB 2.8 GB (40x compression) with 2.6% performance gain. Code is available at:
be achieved through
Introduction
5M61L (NautPP
C-V5MM (CVPH23))
KOeU (CVPNE24)
COA
Dur
We are pacing into the era of ImageNet-level condensation, and the previous works [I]
Guns483
2. 5] fail in scaling up to large-scale
Gans deb
datasets due extensive memory constraint. Until recently, Yin et al. (6] decouple the tra- ditional distillation scheme into three phases. First, teacher model is pretrained with full datasets (squeeze phase). Second, images are synthesized by matching the Batch Nor- malization (BN) statistics from the teacher and student models (recover phase). Third, auxiliary data such as soft labels are pre- generated from different image augmenta- tions create abundant supervision for post- training (relabel phase).
Our--68
50
Accurscy
Daru-8 58
SRAOL-00
IPC-50
IPC-200
ImageNet-1K
30
20
5AM2L-13
Image Label (Base)
0.790
2,900
20:300
115.300
10
0
20
40
60
80
100
120
Total Storage (GB, Image Label)
Figure 1: The relationship between performance and total storage of auxiliary information needed. Our method achieves SOTA performance with fewer soft labels than images.
However, the auxiliary data are 30x larger than the distilled data in ImageNet-IK. To
Cooresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(b) Ours (IPC=10). Figure 2: Visual comparison between SRe² and the proposed method. The classes are hammer
(a) SRe²L 6
shark (top), pineapple (middle), and pomegranate (bottom). Our method is more visually diverse.
attain correct and effective supervision, the exact augmentations and soft labels of every training epoch are stored 66 7
9 [0]. The required soft label storage the colored circles in Fig. !
In this paper, we consider whether large-scale soft labels are necessary, and what causes the excessive requirement of these labels? answer these questions, we provide an analysis of the distilled images using SRe21. [7]. and we find that within-class diversity is at stake as shown in Fig, 2
To be more precise, we analyze the similarity using Feature Cosine Similarity and Maximum Mean Discrepancy in Sec. 3.2 The high similarity of images within the same class requires extensive data augmentation to provide different supervision.
To address this issue, we propose Label Pruning for Large-scale Distillation (LPLD). Specifically, we modified the algorithms by batching images within the same class, leveraging the fact that different classes are naturally independent. Furthermore, we introduce class-wise supervision to align our changes. In addition, we have explored different label pruning metrics and found that simple random pruning was performed on par with carefully selected labels. To further increase diversity, we improve the label pool by introducing randomness in finer granularity (i.e., batch-level). Our method effectively distills the images while requiring less label storage compared to image storage, as shown in Fig.
The key contributions of this work are: (1) To the best of our knowledge, it is the first work to introduce label pruning to large-scale dataset distillation. (2) We discover that high within-class diversity necessitates large-scale soft labels. (3) We re-batch images and introduce class-wise supervision to improve data diversity, allowing random label pruning to effective with an improved label pool. (4) Our LPLD method achieves SOTA performance using lot less label storage, and it is validated with extensive experiments on various networks (e.g., ResNet, EfficientNet, MobileNet, and Swin-V2) and datasets (e.g., Tiny-ImageNet, ImageNet- IK, and Imag
2 Related Works
Dataset Distillation. DD 1 first introduces dataset distillation, which aims to learn a synthetic dataset that is equally effective but much smaller in size. The matching objectives include performance
matching I 12
13
[4], gradient matching 16 17), distribution or feature matching (5] 2
18), trajectory matching 3 19 20), representative matching 21 22], loss-curvature matching 23
and Batch-Norm matching 6 7 [0]
Dataset Distillation of Large-Scale Datasets. Large-scale datasets scale up in terms of image size and the number of total images, incurring affordable memory consumption for most of the well- designed matching objectives targeted for small datasets. MTT k is able to condense Ting-Ima (ImageNet-11 K subsets with images downsampled to 64 x 64 and 200 classes). IDC (24) conducts experiments on ImageNet- 10, which contains an image size of 224 x 224 but has only 10 classes. TESLA 20 manages to condense the full ImageNet-Iee dataset by exactly computing the unrolled

---

gradient with constant memory or complexity. SR [6] decouples the bilevel optimization into three phases: 1) squeezing, 2) recovering, and 3) relabeling. The proposed framework surpasses TESLA 20 by a noticeable margin. CDA 7 improves the recovering phase by introducing curriculum learning. RDED g replaces the recovering phase with an optimization- approach by concatenating selected image patches. SC-DD [TO] uses self-suuervised models as recovery models. Existing methods 77 88 [0] place high emphasis on improving the recovering phase; however, the problem of the relabeling phase is overlooked: large amount storage is required for the relabeling phase.
Label Compression. The problem of excessive storage seems to be fixed if the teacher model generates soft labels immediately used by the student model on the fly. However, when considering the actual use case of distilled datasets (i.e.. Neural Architecture Search), using pre- pre-generated labels enjoys speeding up training and reduced memory cost, More importantly, the generated labels can be repeatedly used. FKD 22 employs label quantization to store only the top-k logits. In contrast, our method retains full logits, offering an orthogonal approach to quantization. comparison to FKI) is provided in Appendix[D.3 Unlike FerKD [26), which removes some unreliable soft labels, our strategy targets higher pruning ratios.
Comparison with G- G-VB (9]. In one recent work, G-VBSM also mentioned re-batching the im- ages within classes; however, the motivation is that having a single image in a class insufficient [9]. It re-designed the loss by introducing a model pool, matching additional statistics from convolutional layers, and updating the statistics synthetic images using exponential moving averages (EMA). Additionally, an ensemble of models is involved in both the data synthesis and relabel phase, requiring a total of N forward propagation from N different models, where N experiments. On the other hand, we aim to improve the within-class data diversity for reducing soft label storage. Furthermore, to account for the re-batching operation, we introduce class-wise supervision while all G-VBSM statisties remain global.
used for ImageNet-
Method
3.1 Preliminaries
The conventional Batch Normalization (BN) transformation is defined as follows:
x -z
+ 3,
(I)
Vo² +c
where y and B are parameters learned during training, 11 and 0² are the mean and variance of the input features, and is a small constant to prevent division by zero. Additionally, the rumning mean and running variance are maintained during network training and subsequently utilized as μ (mean) and o² (variance) during the inference phase, given that the true mean and variance of the test data are not available.
The matching object of SRe2L 7 follows DeepInversion 27 which optimizes synthetic datasets by matching the models' waser-wise BN statistics:
LB +
Σ
(#)
E (ML 1 +
Σ |of (x) - E (of 17)
2 Σ (x) - BN RM
+ Σ
o - BN
where the BN's running mean BNRM RM and rumning variance BN are used to approximate the expected mean E (14 (T) and expected variance E (a? IT) of the original dataset 'T, repsectively. The BN loss matches BN for layers 1, and (*) and (क) are the mean and variance of the synthetic images
The BN loss term is used as a regularization term applied to the classification loss Leg. Therefore, the matching objective is:
arg min e ta LBN (x)
Los
where f the model pretrained on the original dataset T. The symbol a is small factor controlling the regularization strength of BN loss.
3

---

Mos mesemmen Disreng Berrogeoy
SRe2L (fug: 10.09)
Table 1: The cosine similarity between image features. The
(Aug: 8.35)
Durs (Awg: 4.69)
similarities are the average of IK class on the synthetic ImageNet- IK dataset. Features extracted using pretrained ResNet-18.
ue
SRePL
CDA
Ours
Full Dataset
50
0.841 0.023
0.816 # 0.026
0.796 + 0.029
+ 0.015
200
0.839
0.011
0.813 1 0.013
0.793 1 0.015
Figure 3: MMD visualization.
3.2
Diversity Analysis on Synthetic Dataset
3.2.1 Similarity within Synthetic Dataset: Feature Cosine Similarity
critical aspect of image diversity is how similar different the images are within the same class. To quantify this, we utilize the feature cosine similarity measure defined above. Lower cosine similarity values between images within the same class indicate greater diversity, as the images are less similar to one another. This relationship is formally stated as follows:
The lower feature cosine similarity of images indicates higher diversity because the images less similar to another.
Proposition 1.
Feature Cosine similarity can be formally put as:
cos similarity = f( f(x')) [
Eis f(ee(a) f(ø' VE (Be,a) E f(''
where Te and E'c are two images from the same class c, f(-) are the features extracted from a pretrained model, and is the feature dimension.
3.2.2 Similarity between Synthetic and Original Dataset: Maximum Mean Discrepancy
The similarity between images is not the only determinant of diversity since images can be dissimilar to each other yet representative of the original dataset. Therefore, further validate the diversity of our synthetic dataset, we consider an additional metrie: the Maximum Mean Discrepancy (MMD) between synthetic datasets and original datasets. This measure helps evaluate how well the synthetic data represents the original data distribution. The following proposition clarifies the relationship between MMD and dataset diversity:
Proposition 2. A lower MMD) suggests that the synthetic dataset captures a broader range of features similar the original dataset, indicating greater diversity.
The empirical approximation of MMD can be formally defined as 28 29
MMD (PT, Ps) = RT.T Ks.s 2kT,S
where Æx.y i
KC (f (21)) with { X, for ~ Y. and S
denote real and synthetic datasets, respectively; 1C is the reproducing kernel (e.g., Gaussian kernel); P is the feature (embedding) distribution, and f()) is the feature representation extracted by model 0, where F(T) ~ F(S) Ps-
3.3
Label Pruning for Large-scale Distillation (LPLDD)
3.3.1 Diverse Sample Generation via Class-wise Supervision
The previous objective function follows Eq. 3 it uses a subset of classes B, to match the BN statistics of the entire dataset, and images in the same class are independently generated, causing an low image diversity within classes. However, inspired by He el al. [30]. images in the same class should work
collaboratively, and images that are optimized individually (see Baseline work (3)) do not lead to the optimal performance when IPC (Images Per Class) gets larger.

---

Trainable Layer (e.g., Conv)
Stats. (mean/ / variance)
I layer BN BN-Matching Loss
update
update
IPC
1111
Batch Stats.
re- brthhig
[+1
Batch Stats. Class
BN Layer
BN Layer
Global Stats.
Class
Batch Stats.
Batch Stats.
m00
BN Layer
BN Layer
Global
Stats.
SReaL / CDA/ / SC-DD
Synthetic Data
Ours
Figure 4: Illustration of existing methods (left, grey) and the proposed method (right, blue). Existing methods (i.e., SRe²L, CDA) independently generate along the IPC (Image-Per-Chal Per-Class) dimension, causing high similarity between images of the same class. The proposed method allows images of the same class to collaborate, leaving different classes naturally independent. addition, synthetic images updated under class-wise supervision. The classification loss is omitted for simplicity.
Step 1: Re-batching Images within Class. Subsequently, to obtain collaborative effect among different images of the same class, we sample images from the same class and provide the images with wlass-wise supervision 24 Fig. fillustrates the changes.
Step 2: Introducing Class-wise Supervision. However, the running mean and variance approximate the original dataset's is expected mean and variance in global aspect. The matching objective becomes sub-optimal in class-wise matching situation. To this end, we propose to track BN statisties for each class separately. Since we only track the running mean and variance, the extra storage is marginal even when up to IK classes in ImageNet-IK (see Appendix E.2 and E.4).
Step 3: Class-wit Objective Function. The new class-wise objective function is modified from Eq. 3] which has two loss functions. First, we compute the classification loss (i.e., the Cross-Entropy Loss) with BN layers using global statistics to ensure effective supervision. Second, compute BN loss by matching class-wise wise BN statistics. The modified parts are highlighted in blue color, and the objective function is formally put as,
Cross-Entrope Loss with Global BN Statistics
Te.s - BNRM global
arg min
Σ Ve, log
softmax
0r
BNRY global +e
(6)
Σ
BNRM RM
+ [Pli(t
BN Le
Batch Norm Loss with Class-wise BN Statistics
We want emphasize that even though global statistics of the dataset are still taken into account. The output logits for calculating CE loss produced using global statisties. This is because altering y and without fine-tuning ? and could lead to a decline in model performance, resulting in less effective supervision.
adjusting the BN loss with class-wise statistics, the
Theoretical Number of Updates for Stable Class-wise BN Statistics. Traditional BN layers do not compute class-wise statistics; therefore, we need to either keep track of the class-wise statistics while training model from scratch compute these statistics using pretrained model. We prefer the latter as the former requires extensive computing resources. To understand how many BN statistics updates are needed, first look at the update rules of BN rumning statistics for class
BNRM +- (1 c) BN le
RM
+ +.)))))))
RN +- (1-c) BN 1.
+ +c (xc))
5

---

Image
Epoch
Augmentation (e.g, CutMix)
4th
7th
100le
Ist
4th
TOWh
Ist
Ist
4th
7th
10th
2nd
5th
8th
11th
2nd
3th
11th
3th
1ith
2nd
3th
8nh
11th
2nd
3th
Rrh
11th
3rd
6th
91h
172hh
3rd
6dh
12th
3rd
12th
3rd
6d0
999
12th
3rd
Generated
Epoch-level Pruning
Bateh-level Pruning
Training Epoch
Training Epoch
Random Pruning for Soft Label Pool
Random Sampling for Training
Figure 5: Illustration of two random processes in label pruning with improved label pool. First, we need smaller soft label pool due the storage budget. We can conduct pruning at two levels: (1) epoch-level and (2) batch-level. Batch-level pruning can provide more diverse label pool since augmentations (e-g., Mixup or CutMix) are different across batches. The illustrated pruning ratio 25%; the crossed-out labels denote the pruned labels, and the remaining form the label pool. Second, we randomly sample soft labels for model training.
where is the momentum, Since the momentum factor for the current batch statistics usually set to small value (i.e., 0.1), we can theoretically compute existing running statistics that can be statistically significant after how many updates, assuming all other factors are fixed.
Since the running statistics are computed per class, we provide the theoretical number of updates required stabilize all class statisties (see Appendix for the proof):
21
In
2
max
(8)
82 min(
8) min(
Cherooff Bound
Convergence
where is the number updates needed, 9. is the probability that class appears in batch, T is probability threshold, & the momentum parameter in Batch Normalization, is the acceptable relative deviation (where 0 <1), C' is constant, and is the desired convergence tolerance for the BN statistics. How Eq. g guides our experiment design is detailed Appendix E.3
3.3.2 Random Label Pruning with Improved Label Pool
Excelling in Both Similarity Measures. By adopting the changes provided in Sec. 3.3.1 synthetic dataset is more diverse and representative than the existing methods. First, our dataset exhibits smaller feature cosine similarity within classes compared to datasets produced by existing methods, as shown in Table This indicates that our synthetic images are less similar to each other and, thus, more diverse. Second, our dataset exhibits significantly lower MMD shown in Fig compared to datasets produced by existing methods. This suggests that our synthetic dataset better captures the feature distribution of the original dataset. After obtaining diverse dataset, the next move is to address superfluous soft labels.
Random Label Pruning. Different from dataset pruning metrics, which many wield training dynamics [31 32), label pruning is inherently different since the labels in different epochs are independently generated or evaluated. Subsequently, these methods do not directly apply, and modify these metrics to determine which epochs contain the most useful augmentations and soft labels. Through empirical study, we find that using soft labels carefully pruned from different metrics no better than simple random pruning. As a result, we can discard complex rule- based pruning metrics, attaining both simplicity and efficiency. After obtaining the soft label pool, have to decide which labels will used. Following the previous random pruning scheme, we randomly sample the labels for model training in order ensure diversity and avoid any prior knowledge.
Improved Label Pool. Considering that random selection may the most efficient choice, we rethink the diversity of the label pool, as labels at the epoch-level are not the finest elements.

---

Table Tiny-ImageNer label pruning results. The standard deviation attained from three different runs. ddenotes the reported results.
(a) Comparison between SOTA methods.
1x
10x
20%
30%
40x
R8
SRe*1.
Ours
SRR²t
SRe"1.
CDA
Ours
SRe"1.
[RRe
IPC50
41.11
48.71
+8.8
40.3
45.0
16.7+0.0
39.0
41.2
44.3-0.55
34.6
35.8
40.2
29.8
M0.9
38.4
IPC100
49.71
53.2
53.610.3
48.3
50.7
$2.2
46.5
48.0
50.6.0.2
43.0
44.2
47.61..2
39.4
40.0
46.1 2..1
(b) Experiments larger networks.
10x
20x
30x
40
ResNer-50
RosNet-101
ResNet-50
ResNet-101
ResNer-50
ResNet-101
ResNet-50
ResNet-101
ResNet-50
ResNet-101
JPC50
19.0
49.7.10.9
4
48.71.055
47.2.20.6
26.4
43.4
43.0
42.3.1.2
42.1
ICC100
55.3 +0.2
+5.4
+4.0+0.3
54.1+...
52.7+0.
53.7+004
+1.0
51.1 +0.2
50.3+0.3
48.7 +1.7
Table 3: ImageNet-1k IK label pruning result. Our method consistently shows better performance under various pruning ratios. The validation model is ResNet-18. denotes the reported results.
1x
10x
20x
30
40
R8sNet-18
SR
CDA
Ours
SRcAL
CDA
Ours
SRc
CDA
Oors
SRc
CDA
Ours
SRc
CDA
Ours
JPC10
20.1
33.3
34.6+0.9
18,9
28.4
32.7
16.0
21,9
28.6-0.4
14.1
14.2
23.1+
11.4
13.2
20.2
JPC20
33.6
44.0
47.2
31.1
39.7
44.7
29.2
34.1
41.0-0.3
24.5
27.5
35,9
to.3
21.7
24.0
33.0=0.6
JPC50
46,8
55.+0.3
44.1
50.3
54.400.2
41.5
46.1
518-0.2
37.2
41.8
48.6
35.5
38.0
46.7=0.8
1PC
58.01
59.410.
58.80.00
27.4
46.7
55.2
54.0.0.8
JPC200
$7.0
63.3
62.61..3
62.4
61.7.0.7
60.1
09.6
The augmentations such as CutMix and Mixup are performed at the batch level, where the same augmentations are applied to images within the same batch and are different across batches. Therefore, we improve the label pool by allowing batches in different epochs to form new epoch. 'he improved label pool breaks the fixed batch orders and the fixed combination of augmentations within an epoch, allowing more diverse training process while reusing the labels. Our label pruning method is illustrated in Fig.
4 Experiments
4.1
Experiment Settings
Dataset details can be found in Appendix B and detailed settings are provided in Appendix C Computing resources used for experiments can be found in Appendix E.5
Dataset. Our experiment results are evaluated on Tiny-ImageNet B33. ImageNet-IK [34 and
ImageNet-2IK-P [35] We follow the data pre-processing procedure SRe"L 6 and CDA
Squeeze. We modify the pretrained model by adding class-wise BN running mean and running variance; since they not involved in computing the BN statistics, they do not affect performance. As mentioned in Sec. 3.3. we compute class-wise BN statistics by training for one epoch with model parameters kept frozen.
The batch size for the recovery phase the same as the IPC. Besides, we adhere the original setting in L.
Recover. We perform data synthesis following Eq. 6
Relabel. We use pretrained ResNet18 36) for all experiments as the relabel model except otherwise stated. For Tiny-ImageNo and ImageNet- K, we use Pytorch pretrained model. For ImageNet-21K-P. we use Timm pretrained model.
Validate. For validation, we adhere to the hyperparameter settings of CDA 7
Pruning Setting, For label pruning, we exclude the last batch (usually with an incomplete batch size) of each epoch from the label pool. There are two random processes: (1) Random candidate selection from all batches. (2) Random reuse of candidate labels.
7

---

Table 5: Comparison between different pruning metrics, Results are obtained from ImageNet- 1
IPC10 and validated using ResNet-
(b) Calibration labe! pool.
(a) Random pruning vs. Pruning metrics at x0
Easy
x0
30
50x
x
IPC10
correct
diff _siged cut, aet_ratio confidence
-90%
25.5
22.8
17.4
10.3
22.5
Hard
19.6
18.9
19.2
19.5
19.0
-00%
-30%
28.3
22.6
17.1
8.7
Easy
19.5
-30%
27.8
21.5
7.9
Uniform
20.0
18.5
20.1
19.7
19.2
-90%
27.7
22.1
16.3
8.6
Random
20.2
17.6
4.2 Primary Result
Iing-emageNet. Table 2a presents comparison between the label pruning outcomes on Tiny- ImageNet for our approach, SRe²L [6], and the subsequent work, CDA [7]. Our method not only consistently surpasses SRe*L across identical pruning ratios but also achieves comparable results to SRe*L while using 40x fewer labels. When compared to CDA, our method exhibits closely matched performance, yet demonstrates superior accuracy preservation. For instance, at a x0 label reduction, our method secures notable 7.5% increase in accuracy over CDA, even though the improvement stands a mere 0. 1 the benchmark. Table (able/2bpproee provides the pruning results on ResNe150 and R0sNet101. Although there are consistent improvements observed when compared to ResNet18, scaling to large networks does not necessarily bring improvements.
ImageNNet-1K, Table compares the ImageNet- IK pruning results with SOTA methods on ResNet18. Our method outperforms other SOTA methods at various pruning ratios and different IPCs. More importantly, our method consistently exceeds the unpruned version of SRe with 30x less storage. Such result is not impressive at first glance; however, when considering the actual storage, the storage is reduced from 29G to 0.87G. In addition, we notice the performance at 10x (or 90%) pruning ratio degrades slightly, especially for large IPCs. For example, merely 0.2% performance degradation on IPC200 using ResNet18. Pruning results of larger IPCs can be found in Appendix D.2
4.3 Analysis
Ablation Study, Table presents the ab- lation study of the proposed method. Row I is the implementation of SRe*L under CDA's hyperparameter settings. Row 2 is simply re-ordering the loops, and the per- formance at x is improved; nevertheless, when considering the extreme pruning ra- tio (i.e.. 100x), ), it falls short of the existing method. Row computes class-wise BN running statistics in the "squeeze" phase, and these class- blass-wise statistics are used as supervision the "recover" phase. A steady improvement is observed. Row 4 allows pre-generated labels to be sampled at batch level from different epochs, further boosting the performance. Refer to Appendix D.I for an expanded version of ablation.
Table 4: Ablation study of the proposed method. denotes using class-wise matching. CS denotes suing class-wise supervision. ILP denotes using an improved label pool. (IPC50, ResNet18, 18, ImageNet-IK3
46.4
41.1
34.8
54.7
51.9
48.5
42.9
37.7
22.6
55.3
53.2
49.9
45.7
39.7
29.1
Label Pruning Metries. From Table 5a we empirically find that using different metrics explained in Appendix E. is no better than random pruning. addition, as mentioned in FerKD (25), calibrating the searching space by discarding portion of easy hard images can be beneficial. We conduct a similar experiment to perform random pruning on a calibrated label pool, and the metric for determining easy or hard images is "conf idence". However, shown in Table 5b no such range can consistently outperform the non-calibrated ones (last row). An interesting observation that the label pruning law at large pruning ratio seems to coincide partially with data pruning, where removing hard labels becomes beneficial B7].
Generalization. Table 6a shows the performance under large compression rates. Smaller IPC datasets suffer more from label pruning since it requires more augmentation and soft label pairs to boost data diversity. Furthermore, label pruning results ResNet50 are provided in Table 6b

---

Table 6: Additional ImageNet- K label pruning results.
(a) Large pruning rate.
(h) Label pruning results on ResNet-50.
(c) Cross-architecture result. IPC50.
ResNet- 18
50x
100x
RexNet- 50
1x
10x
20x
30x
50x
100x
Model
Size
Full Acc
1x
10x
30x
IPC10
17.6
IPC10
37.7
27.5
11.0
ReaNet-18
55.44
54.45
48.62
IPC20
30.0
17.9
IPC20
54.4
52.3
48.9
45.4
39.5
24.0
ResNet-50
56
25.6M
76.13
62.24
61.22
56.24
IPC50
43.1
33.7
IPC50
62.2
61.2
58.8
56.2
52.3
44.7
EthoicatNet-B0
33
5.3M
77.69
55.51
54,69
52.10
IPC100
52.0
44.7
IPC100
65.7
65.1
63.9
62.0
59.8
54.2
MobileNet-V2 V2
39
3.5M
71,88
49.12
49.26
45.80
IPC200
57.7
52.6
IPC200
67.8
67.1
66.7
63.4
64.1
60.1
Swin V2: Tiny
40
28.4M
82.07
40.59
37.35
29.54
Table Label pruning result on ImageNet-2IK: using ResNet- 18. I denotes image storage. L denotes label storage. denotes reported results.
Table Label pruning for optimization-free method. "Ours" uses improved label pool.
1x SR
10x
40x
ResNet-18
RDED
Ours
RDED
Ours
RDED
Ours
RDED
Ours
IPC
L
CDA
Ours
L
Ours
L
Ours
IPC10
37.9
39.1
32.5
35.7
25.4
30.8
24.0
29.1
IPC20
45.8
48.1
41.2
44.3
36.2
39,5
32,9
38.4
IPC10 3G
643G
18.51
22.6'
25.4
65G
24.1
16G
21.3
IPC50
48.8
48.7
JPC20
5G
12850
20.5t
26.4"
30.3
1296
313
320
29.4
[PC100
57.3
57.8
55.3
57.1
55.2
55.3
51.4
54.2
Not only scaling to large networks of the same family (i.e., ResNet) but Table also demonstrates the generalization capability the proposed method across different network architectures. An analogous trend is evident in the context of label pruning: comparable performance is achieved with 10x fewer labels. This reinforces the statement that the necessity for extensive augmentations and labels can be significantly reduced if the dataset exhibits sufficient diversity.
Large Dataset. ImageNNt-21I-P2 has 10,450 classes, significantly increasing the disk storage as each soft label stores a probability of 10,450 classes. The IPC20 dataset leads to a 1.2 TB (i.e., 1285 GB) label storage, making the existing framework less practical. However, with the help our method, can surpass L 6 by a large margin despite using 40x less storage. For example, we attain an 8.9% accuracy improvement on IPC20 with label storage reduced from 1285 GB to 32 GB.
Pruning for Optimization-Fre Approach. RDED g is an optimization-free approach during the "recover" phase. However, extensive labels still required for post-evaluation. To prune labels, consistent improvements are observed using the improved label pool, as shown in Table
Comparison with G-VBSM [9]. Compared to G-VBSM [9. which uses an ensemble of 4 models to recover and relabel, our method outperforms it various pruning ratios with only single model (see 9)) Furthermore, the techniques used for G- G-VBSM apply to method. By adopting label generation with ensemble and a loss function of "MSE+0.1 X GT 29,, our method can be fur- ther improved by large margin on IPC10 of ImageNet-IK, using ResNet18. Implementation details can be found in Appendix C..4
Table 9: Compare with G- "Ours+" uses en-
VBSM ( semble and MSE+GT loss.
IPCI0
G-VBSM
Ours+
1x
31.4
35.7
39.0
10x
28.4
32.7
20x
26.5
28.6
34.8
30x
22.5
23.1
Visualization. Fig, 2b visualizes method on three classes. More
40x
18.8
20.2
27.9
visualizations are provided in Appendix
Conclusion
To answer the question "whether large-scale soft labels are necessary for large-scale dataset dis- tillation?", we conduct diversity analysis synthetic datasets. The high within-class similarity is observed and necessitates large-scale soft labels. Our LPLD method re-batches images within classes and introduces class-wise BN supervision during the image synthesis phase to address this issue. These changes improve data diversity, that simple random label pruning can perform on par with complex rule- based pruning metrics. Additionally, we randomly conduct pruning an improved label pool. Finally, LPLD is validated by extensive experiments, serving strong baseline that takes into account actual storage. Limitations and future works are provided in Appendix E.6 The ethics statement and broader impacts can be found in Appendix E.7

---

Acknowledgement
This work was supported in part by A Career Development Fund (CDF) under C2333 12004, in part by the National Research Foundation, Singapore, and the Maritime and Port Authority of Singapore/ Singapore Maritime Institute under the Maritime Transformation Programme (Maritime Al Research Programme Grant number SMI-2022-MTP-O00 The computational work for this article was partially performed on resources of the National Supercomputing Centre (NSCC), Singapore hhtpps//wwe)) www. nscc.ggi sg..
References
[1] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 1811. 2018.
[2] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinehao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog.. pages 11196-12205, 2022.
[3] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, 2022.
[4] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In Proc. Int. Conf. Learn. Represent., 2021.
[5] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proc. IEEE Winter Conf. Appl. Comput. Vis., pages 6514-6523, 2023.
[6] Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In Proc. Adv. Neural Inform. Process. Syst., 2023.
[7] Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. arXiv preprint arXiv:2311.1883 18838, 2023.
[8] Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. arXiv preprint arXiv:2312.03526. 2023.
[9] Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2024.
[10] Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation:
A good compression is all you need. arXiv preprint arXiv:2404.07976, 2024.
[11] Timothy Nguyen. Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge-regression. In Proc. Int. Conf. Learn. Represent., 2021.
[12] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Proc. Adv. Neural Inform. Process. Syst., pages 5186-5198, 2021.
[13] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In Proc. Adv. Neural Inform. Process. Syst., 2022.
[14] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Proc. Adv. Neural Inform. Process. Syst., 2022.
[15] Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint 2tt.:2200000311 2022.
[16] Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. Proc. Int. Conf. Mach. Learn., pages 12352-12364, 2022.
10

---

[17] Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexi- fied implicit gradients. In Proc. Int. Conf. Mach. Learn., 2023.
[18] Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 7856-7865, 2023.
[19] Jiawei Du, Yidi Jiang, Vincent TF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[20] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- Ik with constant memory. In Proc. Int. Conf. Mach. Learn., pages 6565-6590. PMLR, 2023.
[21] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302. 14416, 2023.
[22] Murad Tukan, Alaa Maalouf, and Margarita Osadchy. Dataset distillation meets provable subset selection. arXiv preprint wxxvv2230.000086 2023.
[23] Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. Loss- curvature matching for dataset selection and condensation. International Conference on Artificial Intelligence and Statistics, pages 8606-8625 8628, 2023.
[24] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameteri- zation. In Proc. Int. Conf. Mach. Learn., 2022.
[25] Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In Proc. Eur. Conf. Comput. Vis., pages 673-690, 2022.
[26] Zhiqiang Shen. Ferkd: Surgical label adaptation for efficient distillation. In Proc. Int. Conf. Comput. Vis., pages 1666-1675, 2023.
[27] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 8715-8724 2020.
[28] Hansong Zhang, Shikun Li, Pengju Wang, Dan Zeng, and Shiming Ge. M3d: Dataset conden- sation by minimizing maximum mean discrepancy. Proc. AAAI Conf. Artif. Intell., pages 9314 9322, 2024.
[29] Tian Qin, Zhiwei Deng, and David Alvarez-Melis. Distributional dataset distillation with subtask decomposition. arXiv preprint arXiv:2403.00999 2024.
[30] Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. In Proc. Int. Conf. Learn. Represent. 2024.
[31] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In Proc. Ini. Conf. Learn. Represent., 2019.
[32] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In Proc. Adv. Neural Inform. Process. Syst., pages 20596-20607, 2021.
[33] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 248-255 Icee, 2009.
[35] Tal Ridnik, Emanuel Ben-BBruchh, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 2021.

---

[36]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recog. pages 770-778, 2016.
[37] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Proc. Adv. Neural Inform. Process. Syst., pages 19523- 19536, 2022.
[38]
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMLR, 2019.
[39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov. and Liang-Chicd Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 4510-4520, 2018.
[40] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang. Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22009-12019 2022.
[41] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 9729-97388 9738, 2020.
[42] Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout, arXiv preprint arXiv: 1oXiv:/708.0455: 04552, 2017.
[43] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In Proc. Int. Conf. Mach. Learn., pages 10096- 10106. PMLR, 2021.
[44] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proc. IEEE Conf. Comput. Vis. Puttern Recog., pages 6848-6856, 2018.
[45] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. In Proc. Adv. Neural Inform. Process. Syst., pages 17044-17056, 2020.
12

---

A Proof
We aim determine lower bound on the number of batches (updates) n required to ensure that the Batch Normalization (BN) statistics for each class in the ImageNet dataset converge within specified tolerance т, with high probability. The dataset has a varying number images per class, affecting the probability of each class appearing in a batch during sampling.
A.1 Preliminary Analysis
Defining Class Probabilities: Let Pa denote the probability that randomly selected image from the dataset belongs to class
Number of images in class e
Pc Total number of images in the dataset"
Due to the unequal distribution images across classes, Pe varies among classes.
Probability of Class Appearance in a Batch: When sampling a batch of size B, the probability that class does not appear the batch is (1 Pe) Therefore, the probability that class appears in the batch is:
= 1- p.)
This represents the likelihood that at least one image from class c included in a given batch. Number of Batches: Let n be the total number of batches sampled during training.
We assume that batches sampled independently with replacement from the dataset. Under this assumption, each batch is an independent trial where class e appears with probability qe. Therefore, the number of batches M where class appears follows binomial distribution:
M (inomial qc).
Remark: In practice, batches are often sampled without replacement within an epoch, introducing
dependency between batches. However, for large datasets where the total number of images N is significantly larger than the batch size B and the number of batches n, the dependence becomes negligible. In such cases, the binomial distribution serves reasonable approximation.
The expected value M is:
E [ nge-
A.2 Chernoff Bound
To ensure that M is not significantly less than its expected value E[M]. we apply the Chernoff bound:
8
(M s (1 -1- ()- exp
where 5 E (0, 1) represents the acceptable relative deviation from the expected value. This bound provides way to quantify the probability that random variable deviates from its expected value, which is crucial for making high-confidence guarantees.
To ensure the probability that M is less than (1 - 8[E [[]MMn is at most Tj, we set:
82
exp
< Ti.
2
Solving for n:
13

---

-21n(T1)
2
82
To ensure this condition holds for all classes, we use the minimum value of ge:
2 (Ti
82 min(ge)
A.3 BN Convergence
BN Statistics Update: Batch Normalization updates its running statistics using an exponential moving average. The update rule for the BN statistics of class at iteration & + 1 is:
BN (1
BBN + =BN f+1
where is the momentum parameter, and BN: [++ for class c.
is the BN statistics estimated from the current batch
Since BN statistics for class e are updated only when class appears in the batch, we consider only the updates corresponding to those batches. After M such updates:
BN = (1- BN + Σ ( 11-8 8) M-*B
where BN is the BN statistics estimated in the k-th batch containing class
Assuming BN an unbiased estimator of the true BN statistics fe when class e appears, the expected value is:
B[BN+M (1 BB + Me [1 (1 11
Convergence Within Tolerance: To ensure that the BN statistics converge within a tolerance r of the true statistics Pe:
E[BN - Me ST. T.
Since:
ElBN (1 (1)"" |BN 1e
and assuming |BN - Me| I for some constant C, we have:
(1 CM
Taking natural logarithms:
M In - -) + In(C) s In(7).
Using the approximation In(1 8) - for small s:
-M In() S In()).
Solving for M:
14

---

In
M 2
Mo
Origin of My: Here, Mo is derived from the BN convergence requirement that ensures:
Mo BN
Me| S T.
represents the minimum number of updates required for the BN statistics of class c to converge within the desired tolerance
A.4 Combining Bounds
Event Definitions:
Let E be the event that class c appears in sufficient batches (as guaranteed by the Chernoff bound).
* Let E2 be the event that the BN statistics for class converge within the desired tolerance
Target Probability: We aim ensure that both events simultaneously with high probability:
P(E n E2) 2 T.
Union Bound Application: For any events, the probability of their intersection satisfies:
P(E n E2)
P(E 0 Eg)
1 - P(E U En
1 - P(E P(E2).
Error Probability Allocation: For simplicity, we allocate the total acceptable failure probability T equally between the two events:
R
P(E
Elat
(allocated to Chernoff bound),
P(E2)
(allocated to BN convergence).
Chernoff Bound Analysis: For event Er, we require that the probability of class appearing in fewer than the expected number of batches is at most :
T
P (1 8)ngg) S
2
Applying the Chernoff bound:
8 8nge
T
cxp
s
2
Solving for n::
82 nde
T
< ln
2
2 In
T
n
>
82
BN Convergence Requirement: For event Ez, we require that the number of batches M where class appears is sufficient for BN convergence:
ln
M Z Mo
E
15

---

To ensure that this condition holds when event Ej occurs. we use the fact that, with probability at least 1 T. we have:
M 2 (1 5)nqe.
Therefore, to guarantee M 2 Mo, we require:
In
(1 11-gg 2 Mo
Solving for m:
In
6)89.
Final Combined Bound: To ensure that both conditions hold for all classes, we use min(g.):
- T
max
8 min
8) min (ge)
Chernoff
Convergenes
where 5 represents the acceptable relative deviation from the expected number of batches, is the momentum parameter in BN updates, T denotes the acceptable total failure probability (T +2), is the convergence threshold for BN statistics, C' represents an upper bound on BNN Me| at initialization, and min(1.) represents the minimum probability that class appears in batch.
This bound ensures:
With probability least T each class appears in at least (1 8)ngg, batches (event Ej occurs).
With probability at least 2. the BN statistics for each class e converge within tolerance T (event E2 occurs).
By the union bound, both events Ej and Es occur simultaneously with probability at least T.
B Dataset Details
We perform experiments on the following three datasets:
Tiny- Imageett 33 is the subset of ImageNet- 1K containing 500 images per class a total of 200 classes, and spatial sizes of images downsampled to 64 x 64.
ImageNet- 1K [34] contains 1,000 classes and 1,281,167 images in total. The image sizes are resized to 224 x 224.
ImageNet-21K-- K-P 35 is the pruned version of ImageNet-21K,, containing 10,450 classes and 11,060,223 images in total. Images are sized to 224 224 resolution.
C Hyperparameter Settings
C.1 ImageNet-1K
Table 10: Squeezing and class- wise BN statistics of ImageNet- I IK.
Info
Detail
Total Images
1,281,167
Batch Size
256
BN Updates
5005
Source
16

---

Table 11: Data Synthesis of ImageNet-Ik IK.
Config
Value
Detail
Iteration
4,000
Optimizer
Adam
81. 32 = 0.95
Image LR
0.25
Batch Size
IPC-dependent Random
e.g., 50 for IPC50
Initialization
BN Loss (a)
0.01
Table 12: Relabel and Validation of ImageNet- IK.
Config
Value
Detail
Epochs
300
Optimizer
Adam
Model LR
0.001
Batch Size
128
Scheduler
Conccalnn
EMA Rate
Not Used
RandomResizedCrop RandomHorizontalHHi]
scale ratio = (0.08, 1.0)
Augmentation
probability 0.5
CutMix
We Pytorch pretrained ResNet- 18 [36]. with Top-1 accuracy of 69.76%, both the recovery and relabeling model. Class-wise BN statistics are computed using a modified version of the training script of the source provided in Table [O The recovery, data synthesis, phase is provided in
TT] which follows CDA 77 except by changing the batch size to an IPC-dependent size. Relabel and validation processes share the same setting as provided in Table 12
C.2 Imag-Im
Table 13: Squeezing and class-wise BN statistics of Imagene.
Info
Detail
Total Images
100,000
Batch Size
256
BN Updates
391
Source
Table 14: Data Synthesis of Tiny-ImageNet.
Config
Value
Detail
Iteration
4,000
Optimizer
Adam
B1,32 (0.5, 0.9)
Image LR
0.1
Batch Size
IPC-dependent Random
e.g., 50 for IPC50
Initialization
BN Loss (a)
0.05
17

---

Table 15: Relabel and Validation of Tiny-emageNet.
Config
Value
Detail
Epochs
100
Optimizer
SGD
p= 0.9, = 0.0001
Model LR
0.2
Batch Size
64
Warm-up Scheduler
Linear
epoch = 5, 6 0.01
Scheduler
CosineAnnealing Not Used
EMA Rate
RandomResizzedrrop
scale ratio = (0.08, 1.0)
Augmentation
RandomtHorizomallFiii
probability = 0.5
Following I. and CDA 77. we use modified version of 18 41 for Iiny-Im We
modify the training script from Table [3 to compute class-wise BN statistics. The pretrained model has Top-1 accuracy of 59.47%, and the model is used for data synthesis and relabel/validation as shown in Table 14 and Table T5 respectively. Note that for the validation phase, a warm-up epochs is added with different learning rate scheduler (i.e., linear).
C.3 1mageNet-21K-P
Table 16: Squeezing and class-wise BN statistics of Imagenet-2IK-P.
Info
Detail
Total Images Batch Size
11,060,223
1,024
BN Updates
10,801
Source
https//gghhubbcomm/Allbbaaalllll
Table 17: Data Synthesis of ImageNet-21I-4
Config
Value
Detail
Iteration
2,000 Adam
Optimizer
B1.32 = (0.5, 0.9)
Image LR
0.05
Batch Size
IPC-dependent Random
e.g., 20 for IPC20
Initialization
BN Loss (a)
0.25
Table 18: Relabel and Validation of 2
Config
Value
Detail
Epochs
300
Optimizer
AdamW
decay = 0.01
Model LR
0.002
Batch Size
32
Scheduler
ConealAnnealingg
Label Smoothing
0.2
EMA Rate
Not Used
RandomResizedCCrp CutOut
scale ratio = (0.08, 1.0)
Augmentation
Following CDA 7. we use ResNet-18 18 trained for 80 epochs initialized with well-trained ImageNet- 1K weight 355 Class-wise BN statistics are computed using modified version of the training script of the source provided in Table [6] The pretrained ResNet- 18 on ImageNet-211 K-P has Top-1 accuracy of 38. 1%, and the model is used for data synthesis and relabel/validation as shown in Table 17 and Table 18 respectively. Note that CutMix used in ImageNet- IK is replaced with CutOut 42 and relatively large label smooth 0.2 is used during the ImageNet-21K-P pretraining phase. We incorporate the same changes to the relabel/validation phase of the synthetic dataset.
18

---

C.4 Implementation of Baselines
RDED. RDED g has several different changes to the SRe²L settings. (1) The batch size is adjusted according to the IPC size (i.e., 100 for IPC10 and 200 for IPC50). (2) It uses additional augmentation (i.e., ShufflePatch to shuffle the position of patches). Such augmentations are considered additional storage since the exact order patch shuffling needs to be stored. (3) Weaker augmentation (i.e., larger lower bound for the random area of the resized crop). (4) A smoothed learning rate scheduler. We adhere to all the changes for experiments regarding RDED.
G-VBSM. In Table 9 we adopt the several techniques used for G-VBSM 9 (1) Soft labels are generated with ensemble of models. Specifically, EfficientNet-Bl B0 [3], ShuffleNetV2-0.5 44 (2) Logit Normalization is used to keep the same label storage. (3) A different MSE+yxGT loss replaces KL divergence, where 7 0.1.
ResNet18 36. MobileNetV2 [39].
D
Additional Experiments
D.1
Ablation
Table 19: Ablation study of the proposed method. C denotes using class-wise matching. CS de- notes suing class-wise supervision. ILP denotes using an improved label pool. (IPC50, ResNet18, ImageNet-1K).
+
+CS
+ILP
1x
10x
20x
30x
50x
100x
52.0
49,4
46.4
41.1
34.8
25.4
54.7
51.9
48.5
42.9
37.7
22.6
v
54.7
52.9
49.9
46.2
40.7
27.0
55.3
53.2
49.9
45.7
39.7
29.1
v
55.4
54.4
51.8
48.6
43.1
33.7
Table 19 presents an expanded version of Table The row highlighted in grey outlines the ablation study class-wise supervision, demonstrating that the ILP component (Improved Label Pool) enhances performance independently class-wist wise supervision.
D.2 Scaling on Large IPCs
Table 20:
Experiment on scalability of large IPCs. denotes the total storage of images and labels, and storage is measured in GB. The validation model is ResNer18.
1x
T
30x
T
40x
IPC300
65.3
178
62.6
10
61.9
IPC400
67.4
237
65.2
13
64.6
12
ImageNet-1K IK
69.8
138
Table 20 demonstrates that method exhibits commendable scalability across large IPCs. We observe non-m marginal enhancements when deploying even larger IPCs, such as IPC300 and IPC400. Moreover, our approach achieves nearly identical accuracy levels specifically, 65.3% 65.2% when comparing the use of IPC300 at 1 x with IPC400 at 30x less labels. Compared to the full ImageNet- 1K dataset, our method preserves large portion of the accuracy with 10x less storage. This performance is achieved despite the vastly different storage requirements of 178G and 13G, respectively, indicating higher flexibility of IPC choice with fixed storage budget.
D.3 Comparison with Fast Knowledge Distillation
25
The label quantization technique mentioned in Fast Knowledge Distillation (FKD) 25
is orthogonal to the proposed method for several reasons. Firstly, as demonstrated in Table 21 there are six components related to soft labels. FKD only compresses the prediction logits (component 6), while the our method addresses all six components.
19

---

Secondly, even for the overlapping storage component (component 6: prediction logits), the compres- sion targets differ between FKD and our method, shown Table The total stored prediction log- its can be approximated by the formula: number_o sa__off_oonoe condensed images number_ of augmentations dimension _ogiiiot___ooitsss....oo FKD's s label quantization focuses on compressing the dimension of logits, whereas the proposed label pruning method focuses compressing the number_oo of augmentations.
Table 21: Different storage components between FKD and the proposed method. FKD, originally for model distillation, requires storage only for components 1, 2, and 6. Adapting to dataset distillation requires additional storage for components 4, and 5.
Components of Storage
FKD
Proposed Method
1. coordinates of crops
X
2. flip status
X
3. index of cutmix images
X
4. strength of cutmix
X
5. coordinates of cutmix bounding box
X
6. prediction logits
Table 22: Breakdown explanation for component (prediction logits) storage between FKD's label quantization and the proposed label pruning. The number condensed images is computed by N IPC number_ of _lassess FKD's compression target is dimension of logits, while the proposed method's target is number_ of augmentatiins..
Method
Number of
Dimension of
Total Storage for Prediction Logits
Augmentations per Image
Logits per Augmentation
Baseline (no compression)
1,000
300 1000 N x 300 x 10
Label Quantization (FKD) 300
10
Label Pruning (Proposed) 3
1,000
N x 3 1000
Although FKD's approach is orthogonal our method, comparative analysis was conducted to better understand their relative performance, Table presents a detailed comparison between FKD's two label quantization strategies (Marginal Smoothing and Marginal Re-Norm) and the proposed method. It is important to note that FKD only compresses component 6, with the compression rate related to
K. Components 1-5 remain uncompressed (1 rate) FKD, Additionally, FKD's quantized logits store both values and indices, so their actual storage is doubled, and their compression rate is halved.
This analysis has yielded two key observations. First, our method demonstrates higher accuracy at comparable compression rates. For IPC10, our method achieves 32.70% accuracy 10x compres- sion, while FKD only reaches 18. 10% at 8.2x compression. Second, our method exhibits better compression similar accuracy levels. On IPC10, our method attains 20.20% accuracy 40x compression, whereas FKD achieves 19.04% at just 4.5x compression.
Table 23: Comparison between FKD's 's two label quantization strategies (Marginal Smoothing and Marginal Re-Norm) and ours.
Method
Compression Rate of Component
Full Compression Rate
Accuracy (%) IPC10
Component 1-5
Baseline (no compression)
1x
1x
1x
34.60
FKD (Smoothing, K=100)
1x
(10/2)=5x
4.5 x
18.70
FKD (Smoothing. K=50)
1x
(20/2)=10x
8.2
15.53
FKD (Smoothing, K=10)
(100/2)=50x
23.0
9,20
FKD (Re-Norm, K=100)
1x
(10/2)=5x
4.5x
19.04
FKD (Re-Norm, K=50)
1X
(20/2)=10x
8.2x
18.10
FKD (Re-Norm, K=10)
1x
(100/2)=50x
23.0
15.52
Ours (10x)
10x
10x
10x
32.70
Ours (20x)
20x
20x
20x
28.60
Ours (40x)
40x
40x
40x
20.20
20

---

E Additional Information
E.1 Label Pruning Metries
We determine labels according to the statistics of the auxiliary information:
1. correct: the number of correctly classified images
31
2. diff: the absolute difference between the Top-2 outputs
3. signed_diie _nnef the signed difference between Top-2 output 45
4. cut_ratio: the cut-mix ratio
5. confidence: the value of the largest output 266
These metrics serve for the baselines compared to random label pruning in Table After knowing the metric, knowing which data type prune (i.e., "easy", "hard", or "uniform") is important. Additionally, FerKD [26] argues the reliability of generated soft labels and proposes to use neither easy nor too hard samples.
E.2 Image and Label Storage
Table 24: Image and label storage. 1l denotes image storage. L denotes label storage. "Ratio" is label-to-image ratio.
-mageImteeNet (MB)
ImageNet- IK (GB)
Storage
L
Ratio
Storage
II
L
Ratio
IPC10
0.15
5.67
37.0
IPC50
21
449
21.4
IPC20
0.30
11.33
37.6
IPC100
40
898
22.5
IPC50
0.75
28.33
37.9
ImageNet-2IK-P (GB)
IPC100
1.49
56.66
38.0
IPC200
2.98
113.33
38.0
Storage
IL
Ratio
IPC300
4.76
172.63
36.3
IPC10
643
214.3
IPC400
6.33
229.80
36.3
IPC20
1285
257.1
Table. 24 shows that stored labels are more than 10x, 30x, and 200x sized of the image storage, depending on the number of classes of the dataset.
E.3
Theoretical Analysis on the Number of Updates
Our experiments grounded in careful analysis of the number of updates required for stable Batch Normalization (BN) statistics. We begin by examining the derived bound from Eq.
T
2 In
2 max
82 min(ge)
8) min( (g.)
Chernolf
Convergence
To evaluate this bound, we substitute the following values:
T 0.05 (acceptable total failure probability, corresponding 95% confidence)
* 5 = 0.2 (acceptable relative deviation from the expected number of batches)
Σ = 0.1 (momentum parameter in BN)
732
min(p.) 1, 281, 167
0.0005711 (ratio of the least number of images in class total
images)
255 (batch size)
min(g.) 1 (1 min(p.)) (minimum probability that any class appears in batch)
21

---

First, we compute min(
min(ge)
(1 min (p=))
(1 0.0005711) 256
(0.9994229))28 256
-25xx0.0005711
(since min(pe) is small)
0.1462
0.8639 0.1361.
Thus, min( R 0.1361.
Next, compute the two parts of the bound separately.
From Chernoff Bound Term: Given that we allocate the total failure probability T eequall between the two events, we have T 0.025.
T
2 ln
n z
8 min
2 l00..225
(0.2)² x 0,1361
-2 x (-3.6889)
(since ln(0.025) -3.6889)
0.04 0.1361
7.3778
0.005444
1, 355.2.
From BN Convergence Term: We need to specify C' and r. Let's assume:
C 1 (an upper bound on BN Pe| at initialization, as the running mean is typically initialized to zero)
r 0.01 (desired convergence tolerance)
Compute the numerator:
in
In(100) 4.6052.
0.01
Now, compute the denominator:
(1 - d) min(g.) = (1 0.2) x 0.1 x 0.1361 = 0.8 x 0.1 x 0.1361 = 0.010888.
Compute the second part:
In
(1 8) min (go)
4.6052
0.010888
423.08.
Final Bound:
n 2 max (1, 355.2, 423.08)
= 1,555.2 ~ 1, 3,6
(rounding up the nearest whole number).
22

---

This theoretical result indicates that approximately 356 batches are needed for stable BN statistics with the specified parameters.
Practical Implications: This observation leads to a key insight: pretrained models have already undergone sufficient updates to achieve stable BN statisties. Specifically, in the context of ImageNet- IK:
Updates per epoch = 1,281, 167
Re 5, 005 updates >> 1, 356
256
Since one epoch consists of approximately 5, 005 updates, which is substantially more than the theoretical requirement of 356 batches, we can confirm that a single epoch of training is sufficient for the BN statistics of each class converge within the desired tolerance with high probability.
E,4
Class-wise Statisties Storage
Table 25: Additional storage required for wlsse-wise statistics. The model ResNet- 18, and storage measured in MB.
Tiny-Imagels
ImageNet-IK
ImageNet-21K-P
Original
43.06
44.66
247.20
Class Stats
50.41
81.30
445.87
Diff.
7.35
36.64
198.67
The additional storage allocation for class-specific statistics is detailed in Table 25 It is observed that this storage requirement escalates with an increase in the number of classes. However, this one-time necessity during the recovery phase and becomes redundant once the synthetic data generation is completed.
E.5 Computing Resources
Experiments are performed on 4 A100 80G GPU cards. We notice that for Tiny- ImageNet,, there a slight performance drop when multiple GPU cards are used with DataParalles in PyTorch. Therefore, use 4 GPU cards for ImageNet-1 K and ImageNet-21K-P experiments and 1 GPU card for all Tmag- experiments.
E.6 Limitation and Future Work
We recognize that there are several limitations and potential areas for further investigation. Firstly, while our work significantly reduces the required storage, the process for generating the soft labels is still necessary,
randomly select from this label space. Secondly, reducing the required labels may not directly lead to reduced training speed, as the total training epochs remain the same in order achieve the best performance. Future work is warranted reduce label storage well the required training budget simultaneously.
E.7
Ethics Statement and Broader Impacts
Our research study focuses on dataset distillation, which aims to preserve data privacy and reduce computing costs by generating small synthetic datasets that have no direct connection to real datasets. However, this approach does not usually generate datasets with the same level of accuracy as the full datasets.
In addition, our work in reducing the size of soft labels and enhancing image diversity can have positive impact on the field by making large-scale dataset distillation more efficient, thereby reducing storage and computational requirements. This efficiency can facilitate broader access to advanced machine learning techniques, potentially fostering innovation across diverse sectors.
23

---

Visualization
In this section, present visualizations of the datasets used in experiments. Duc to the different matching objectives, datasets of different IPCs should have distinct images. Therefore, we provide the visualization of different IPCs. Figure ( shows randomly sampled images from ImageNet- 1 various IPC. Figure deppicts the Lin--mmgeNet dataset with IPC50 and IPC Figure provides visualizations of ImageNet-21K-4 IPC10 and IPC20.
F.1 ImageNet-IK
(a) IPC10
(b) IPC550
(c) IPC100
(d) IPC200
Figure 6: Visualization of ImageNet-1K IK. Images are randomly sampled.

---

F.2
Tiny-Iman MaageNete
(a) IPCS0
(b) IPC100
Figure 7: Visualization of Tiny-ImageNet. Images are randomly sampled.
F.3
ImageNet-2IK-P
(a) IPC10
(b) IPC20
Figure 8: Visualization of ImageNet-2118 21PP- Images are randomly sampled.
25

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [TODO]
26

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllccguideeee http://oeeuuiiiiiooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
28

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
29

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htatps///eerriss..cppbbb../iiiit
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses such risks.
30

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
31

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "V8N4ms7q1M", "reviewer": "Reviewer_S5vb", "summary": "This paper discovers that the conventional method generates images with high similarity. To solve this, the authors introduce class-wise supervision during the image-synthesizing process by batching the samples within classes. Thanks to the increase in diversity, the soft labels can be pruned to reduce the storage size. Extensive experiments are performed on ImageNet with different compression ratios of labels to validate the effectiveness of the method.", "questions": "See weaknesses", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Improving the diversity in condensed images is crucial for obtaining high performance.\n\n2. Label pruning is necessary to reduce the storage size for soft labels.", "weaknesses": "1. There are confusing sentences. The authors mention that ”The high similarity of images within the same class requires extensive data augmentation to provide different supervision” and then “To address this issue, we propose Label Pruning for Large-scale Distillation”. How does label pruning address the issue? I see that increasing the diversity of synthetic samples does improve the performance and label pruning is proposed to reduce the storage size. These are not highly related.\n2. The authors should compare the performance of the proposed Label pruning with other compression methods (e.g., Marginal Smoothing/Re-Norm with Top-K, using different K when targeting at 10x/20x…) proposed in FKD. Without comparing these methods, it is unclear whether the proposed approach is better than previous pruning methods.\n\n\nFKD: A Fast Knowledge Distillation Framework for Visual Recognition, Zhiqiang Shen et al", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for raising these questions and concerns. We want to address them one by one.\n\n> 1. There are confusing sentences. The authors mention that ”The high similarity of images within the same class requires extensive data augmentation to provide different supervision” and then “To address this issue, we propose Label Pruning for Large-scale Distillation”. How does label pruning address the issue? I see that increasing the diversity of synthetic samples does improve the performance and label pruning is proposed to reduce the storage size. These are not highly related.\n\nThank you for your insightful question. \n\n```\nPrevious:      High Similarity -> More Augmentation -> More Label with\n                                                       High Similarity\n\nNaive Pruning: High Similarity -> More Augmentation -> More Label with ------> Fewer Label with\n                                                       High Similarity         High Similarity\n\nOurs LPLD:     Low Similarity  -> Less Augmentation -> Fewer Label with\n                                                       Low Similarity\n```\n\nWe draw the above diagram to clarify the relationship between image similarity and our proposed Label Pruning for Large-scale Distillation (LPLD) method.\n\nFirst, it's important to distinguish between naive pruning and our LPLD pruning:\n\n1. **Naive pruning**: This method directly prunes soft labels generated by previous approaches like SRe2L and CDA, starting with **high-similarity** images. It does not address the issue of image diversity.\n2. **Our LPLD** pruning: We begin with **low-similarity** images and then apply label pruning. This two-step process addresses both image diversity and storage efficiency.\n\nWhen the reviewer mentions the lack of **direct relation** between pruning and image similarity, we believe this observation is most applicable to naive pruning methods, not our LPLD. Our approach, however, takes a fundamentally different direction:\n\n1. We first enhance image diversity by creating **low-similarity** images. This step creates a prerequisite for effective **pruning**.\n2. We then apply label pruning to this diverse set of images, which allows for more effective compression while maintaining performance.\n\nTo illustrate the effectiveness of **connecting image similarity and pruning**, we conducted a comparison between naive pruning (**NOT related** to similarity) and our LPLD (**related** to similarity) for ImageNet-1K at IPC50, as shown in **Table A**. The results demonstrate that our method outperforms naive pruning, especially at higher compression rates. \n \n**Table A**: Comparison between naive pruning on previous methods and our LPLD pruning for ImageNet-1K at IPC50. \n\n| Compression Rate | 0x | 10x | 20x | 30x | 40x |\n|---|---|---|---|---|---|\n| SRe2L | 41.10% | 40.30% | 39.00% | 34.60% | 29.80% |\n| CDA | 48.70% | 45.00% | 41.20% | 35.80% | 30.90% |\n| Ours | **48.80%** | **46.70%** | **44.30%** | **40.20**% | **38.40%** |\n\nIn summary, our LPLD method addresses the issue of high image similarity within classes by creating diverse images first and then applying label pruning. This approach improves both the diversity of synthetic samples and reduces storage size, addressing both concerns at the same time.\n\n> 2. The authors should compare the performance of the proposed Label pruning with other compression methods (e.g., Marginal Smoothing/Re-Norm with Top-K, using different K when targeting at 10x/20x…) proposed in FKD. Without comparing these methods, it is unclear whether the proposed approach is better than previous pruning methods.\n\nThank you for your question.\nAlthough FKD's approach is orthogonal to our method (reasons for the orthogonality are listed in another response comment), we compared different target components in **Table B** and conducted a comparative analysis presented in **Table C**. Table C follows the definition of the components in Table B. Please note that FKD only compresses component 6, with the compression rate related to hyper-parameter $K$. Components 1-5 remain uncompressed (1x rate). We achieve better performance:\n\n1. Higher Accuracy at Comparable Compression Rates: For IPC10, our method achieves 32.70% accuracy at 10x compression, while FKD only reaches 18.10% at 8.2x compression.\n\n2. Better Compression at Similar Accuracy Levels: On IPC10, our method attains 20.20% accuracy at 40x compression, whereas FKD achieves 19.04% at just 4.5x compression.\n\nWe hope this response addresses your concern.\n\n**Table B**: Different target components between FKD and ours. FKD, originally for **model** distillation,  requires storage only for components 1, 2, 6. Adapting it to **dataset** distillation requires additional storage for components 3, 4, 5.\n|  | FKD | Ours |\n|---|---|---|\n| 1. coordinates of crops | ❌ | ✔ |\n| 2. flip status | ❌ | ✔ |\n| 3. index of cutmix images | ❌ | ✔ |\n| 4. strength of cutmix | ❌ | ✔ |\n| 5. coordinates of cutmix bounding box | ❌ | ✔ |\n| 6. prediction logits | ✔ | ✔ |\n\n**Table C**: Comparison between FKD's two label quantization strategies (Marginal Smoothing and Marginal Re-Norm) and ours.  FKD’s quantized logits store both values and indices, so their actual storage is doubled, and their compression rate is halved. \n\n| Method | Compression rate of component 1-5 | Compression rate of component 6 | Full compression rate | Accuracy (%) on IPC10 |\n|---|---|---|---|---|\n| Baseline (no compression) | 1x | 1x | **1x** | 34.60 |\n| FKD (Smoothing, K=100) | 1x | (10/2)=5x | **4.5x** | 18.70 |\n| FKD (Smoothing, K=50) | 1x | (20/2)=10x | **8.2x** | 15.53 |\n| FKD (Smoothing, K=10) | 1x | (100/2)=50x | **23.0x** | 9.20 |\n| FKD (Re-Norm, K=100) | 1x | (10/2)=5x | **4.5x** |  19.04 |\n| FKD (Re-Norm, K=50) | 1x | (20/2)=10x | **8.2x** | 18.10 |\n| FKD (Re-Norm, K=10) | 1x | (100/2)=50x | **23.0x** | 15.52 |\n| Ours (10x) | 10x | 10x | **10x** | **32.70** |\n| Ours (20x) | 20x | 20x | **20x** | 28.60 |\n| Ours (40x) | 40x | 40x | **40x** | 20.20 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SEhW5luLDf", "reviewer": "Reviewer_8MXG", "summary": "The paper focuses on reducing the size of soft-label storage in large-scale dataset condensation. The authors discussed why the labels are large-scaled and then proposed to prune the labels by increasing the diversity of synthetic images. Extensive experiments are conducted to validate the effectiveness of the proposed method.", "questions": "Please see Weaknesses.", "limitations": "Yes, the limitations are addressed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles the problem of large storage space consumed by soft-labels, and discover that this is attributed to the lack of image diversity within classes.\n2. Extensive experiments demonstrate the effectiveness of the proposed method, achieving comparable or even better performance than previous methods with reduced label storage.", "weaknesses": "The motivation of the work is well-grounded. I don’t observe major drawbacks but do have some minor concerns towards its technical contributions.\n\n1. The proposed method seems to be a class-wise version of SRe^2L, with a soft label-reuse mechanism during training. Thus the technical contributions seem to be weak.\n2. How is Equation (8) used in the proposed method? Or is it just a theoretical result?\n3. I have doubts on the computational cost (memory and time) of the proposed method (not storage), doing class-wise optimization may increase the distillation time, especially on ImageNet.\n4. How is the random label pruning done? E.g. Is the same number of soft-labels removed for each epoch?\n5. Proposition 2 is quite confusing, is “higher MMD” a typo?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for bring up all these values feedbacks.\n\n> 1. The proposed method seems to be a class-wise version of SRe^2L, with a soft label-reuse mechanism during training. Thus the technical contributions seem to be weak.\n\nThank you for your important question.\nOur approach is not merely a variant of existing methods but a comprehensive redesign that addresses key limitations. We'd like to elaborate on our key contributions and the novelty of our work:\n\n1. **Novel Motivation and Insight**. We recognized the critical importance of image diversity in large-scale distillation tasks. This insight led us to rethink the fundamental approach to distillation tasks.\n2. **Unified Framework for Diversity and Pruning**. We uniquely combine image diversity enhancement and label pruning synergistically.\n3. **Simplicity**. The strength of our approach lies in its simplicity. We demonstrate that **random** **pruning** within our class-wise framework is sufficient, eliminating the need for complex pruning metrics.\n4. **Effectiveness**.  Our method enhances performance with full labels. Additionally, our method can maintain comparable results at a 40x compression ratio, demonstrating its robustness across various data scenarios. This shows that rethinking the problem from a class-wise angle can lead to significant improvements.\n\n> 2. How is Equation (8) used in the proposed method? Or is it just a theoretical result?\n\nThanks for your thoughtful question. Our experiments are grounded in a careful analysis of the number of updates required for stable BN statistics. \n\nFirst, for Equation 8: \n\n$$n \\geq \\max \\left(\\frac{-2 \\ln (T)}{\\delta^2 \\min \\left(p_c\\right)}, \\frac{\\ln (T)}{(1-\\delta) \\ln (1-\\varepsilon) \\cdot \\min \\left(p_c\\right)}\\right)$$\n\nLet’s substitute the values and compute the two parts one by one,\n\n1. $T=0.05$ (95% confidence)\n2. $\\delta=0.2$ (moderate sigma to handling class imbalance)\n3. $\\epsilon=0.1$ (default value in BN)\n4. $p_c=(732/1,281,167)= 0.0005711$  = least number of images in a class / total images\n\n$$n \\geq \\max \\left( \\frac{-2 \\times \\ln(0.05)}{0.2^2 \\times 0.0005711},  \\frac{\\ln(0.05)}{(1 - 0.2) \\ln(1 - 0.1) \\cdot 0.0005711}\\right) = \\max (262234, 62234) = 262,234$$\n\nNext, let us elaborate on how we use the theoretical result to guide the experiment design:\n\n1. $n \\geq 262,234$ means that the theoretical number of updates needed for stable BN statistics is $262,234$.\n2. In ResNet training on ImageNet-1K, the standard setting [a] uses a batch size of 256 and trains for 90 epochs. The total number of updates is $(1,281,167/256) \\times 90 = 450,411$. This significantly exceeds our theoretical requirement of 262,234 updates.\n3. This observation gave us a key insight: pretrained models have already undergone sufficient updates to achieve stable BN statistics. Therefore, we can recompute class-wise BN statistics using a pretrained model for only **one epoch** (5,005 updates).\n\n[a] He, Kaiming, et al. \"Deep residual learning for image recognition.\" *CVPR*. 2016.\n\n> 3. I have doubts on the computational cost (memory and time) of the proposed method (not storage), doing class-wise optimization may increase the distillation time, especially on ImageNet.\n\nThank you for bringing up your concern. We acknowledge that our method can indeed increase computational costs in some cases. However, the impact varies depending on the Images-Per-Class (IPC) setting. Based on our experiments in **Table A**, we have:\n- **IPC 50**:  19.95% slower than SRe2L.\n- **IPC 100**: 7.28% slower than SRe2L.\n- **IPC 200**: 4.83% **faster** than SRe2L.\n\nOn average, the computational time increase is 7.47%. This suggests that while there is indeed a computational cost increase in some scenarios, the impact is not uniform and our method can even be more efficient in certain configurations.\n\nWe believe this trade-off is exceptionally favorable: a minimal **7.47%** increase in processing time unlocks a dramatic **40x** reduction in data volume. Such powerful compression capability far outweighs the marginal computational cost, making our approach particularly valuable for large-scale applications where data efficiency is paramount.\n\n**Table** **A**: Real computation cost of SRe2L and our method on ImageNet-1K. The time spent on inner loops is averaged from 3 loops.  The optimization process contains two loops: the outer loop and the inner loop.  Total time = (Time for Inner Loop * Outer Loop Numbers).\n\n|  | IPC 50 | IPC 100 | IPC 200 |\n|---|---|---|---|\n| SRe2L - Breakdown | 37m5s $\\times$ 50 | 37m5s $\\times$ 100 | 37m5s $\\times$ 200 |\n| SRe2L - Total | 30h55m | 61h49m | 123h37m |\n| Ours - Breakdown | 2m13s$\\times$ 100 | 3m59s $\\times$ 1000 | 7m4s $\\times$ 1000 |\n| Ours - Total | 37h5m | 66h19m | 117h39m |\n\n> 4. How is the random label pruning done? E.g. Is the same number of soft-labels removed for each epoch?\n\nThank you for this important question.\nYes,  the number of soft labels removed for each epoch is the **same**. The detailed process for random label pruning is shown in Figure 5. There are two different levels for random pruning:\n\n1. First-level random pruning for the soft label pool: **9 samples** from **12 samples**. \n2. Second-level random pruning for training: **3 samples** from **9 samples**. \n\nFor first-level random pruning, we introduce two different schemes:\n\n1. Epoch-level random pruning: if 1st sample is removed, 2nd 3rd must be removed.\n2. Batch-level random pruning: even if 1st is removed, 2nd 3rd can be randomly removed or kept.\n\nIn our paper, for first-level random pruning, we stick to batch-level random pruning as it introduces more randomness. For second-level random pruning, we use normal randomness.\n\n\n> 5. Proposition 2 is quite confusing, is “higher MMD” a typo?\n\nThank you so much for bringing out this typo. We have corrected this error in the original manuscript. Also, we have conducted a thorough review of the entire document to ensure no similar errors exist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2olNT9D5W", "reviewer": "Reviewer_FdYh", "summary": "This paper addresses the challenge of large-scale dataset distillation, specifically focusing on reducing the storage requirements for auxiliary soft labels in ImageNet-level condensation. The authors propose Label Pruning for Large-scale Distillation (LPLD), which aims to achieve state-of-the-art performance while significantly reducing the storage needed for soft labels. The main contributions include:\n1. Identifying that high within-class similarity in condensed datasets necessitates large-scale soft labels.\n2. Introducing class-wise supervision during image synthesis to increase within-class diversity.\n3. Demonstrating that simple random pruning of soft labels is effective when data diversity is improved.\n4. Achieving SOTA performance with significantly reduced label storage, validated across various networks and datasets.", "questions": "1. How sensitive is the method to the choice of hyperparameters, particularly in the label pruning process? Is there a way to automatically determine the optimal pruning rate? \n\n2. Have you explored the potential of combining your class-wise supervision approach with other dataset distillation techniques? Could this lead to further improvements? \n\n3. How does the performance of your method scale with the number of classes? Is there a point where the benefits of class-wise supervision diminish? \n\n4. Have you investigated the impact of your method on the training time of the distilled dataset compared to previous approaches?\n\n5. In Table 6(c), why the performance on Swin-V2-Tiny is the worst even if the architecture has the largest size (28.4M)?", "limitations": "Yes. Limitations are provided in Appendix E.5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses an often overlooked problem in dataset distillation: the large storage requirements for auxiliary data. Their class-wise supervision approach offers a novel solution to this issue. \n\n2. The analysis is thorough, using feature cosine similarity and MMD to demonstrate increased diversity in synthetic data. Experiments across Tiny-ImageNet, ImageNet-1K, and ImageNet-21K show the method's effectiveness on various datasets. \n\n3. The paper is well-organized and written. The authors clearly explain their motivations, provide detailed methodology, and offer comprehensive analysis of their results. Their figures and tables effectively illustrate key points. \n\n4. By significantly reducing storage needs (40x compression of soft labels) while maintaining or improving performance, this work makes dataset distillation more practical for large-scale applications.", "weaknesses": "Limited theoretical analysis: While the paper provides comprehensive empirical evidence, a more rigorous theoretical analysis could strengthen it. I know this is a common challenge in dataset distillation research. However, the detailed empirical study presented here lays valuable groundwork for potential future theoretical analyses in this direction, particularly regarding the relationship between class-wise supervision and increased diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your questions and feedbacks. We want to address them one by one.\n\n> 1. How sensitive is the method to the choice of hyperparameters, particularly in the label pruning process? Is there a way to automatically determine the optimal pruning rate?\n\nThank you for bringing up this point. Regarding the label pruning process, we adopt two random processes: **(1) random soft label pruning** and **(2) random resampling for training**, as shown in Figure 5. The pruning ratio is the only hyperparameter introduced in this process.\n\nIt's important to note that there is no universally optimal pruning rate in this scenario, as the ideal rate depends on the trade-offs between accuracy and storage. From our observations, datasets with higher Images-Per-Class (IPCs) tend to be more robust to label pruning. This indicates that such datasets can better maintain accuracy even when labels are pruned, offering more flexibility in balancing the trade-offs.\n\n\n\n> 2. Have you explored the potential of combining your class-wise supervision approach with other dataset distillation techniques? Could this lead to further improvements?\n\nThank you for the question. We explored applying our method to CDA, and it shows improvements over the pruning results, making CDA more robust to label pruning, as shown in **Table A**.\n\n**Table A**: Directly applying our method on CDA. ImageNet-1K, IPC50. \n\n|             | 0x   | 10x  | 20x  | 40x  |\n| ----------- | ---- | ---- | ---- | ---- |\n| CDA         | 53.5 | 50.3 | 46.1 | 38.0 |\n| Ours on CDA | 55.2 | 53.4 | 51.0 | 46.0 |\n\n\nAdditionally, we believe that initializing with images from RDED [a] could eventually be beneficial. RDED enhances image diversity through an optimization-free approach, and we can further boost image diversity through class-wise supervision.\n\n\n> 3. How does the performance of your method scale with the number of classes? Is there a point where the benefits of class-wise supervision diminish?\n\nThank you for the question. We do not observe a diminishing performance with an increasing number of classes. **Table B** provides the results of the ImageNet-21K-P dataset (10,450 classes). Despite the increasing challenges of the dataset itself, our method outperforms SRe2L and even CDA by a noticeable margin at 0x baseline. Notably, for IPC20 under a 40x pruning ratio, the performance suffers only a 0.9% drop on ImageNet-21K-P. \n\n**Table B**: Performance of different methods on ImageNet-21K-P. The dataset contains 10,450 classes. This is a simplified version of Table 7 from our paper. \n\n| ImageNet-21K-P | SRe2L (1x) | CDA (1x) | Ours (1x) | Ours (40x) |\n| -------------- | ---------- | -------- | --------- | ---------- |\n| IPC10          | 18.5       | 22.6     | 25.4      | 21.3       |\n| IPC20          | 20.5       | 26.4     | 30.3      | 29.4       |\n\n\n\n> 4. Have you investigated the impact of your method on the training time of the distilled dataset compared to previous approaches?\n\nThank you for the insightful question; this is worth exploring. Currently, we have not changed the total training epochs (e.g., 300 epochs for ImageNet-1K), so there is no apparent difference in training the distilled dataset. The only difference is that SRe2L uses the full set of labels while we reuse a subset of the labels. The reuse of labels has almost no impact on the training time.\n\n\n> 5. In Table 6(c), why the performance on Swin-V2-Tiny is the worst even if the architecture has the largest size (28.4M)?\n\nTable 6(c) presents the results of cross-architecture performance. Specifically, synthetic data are recovered using ResNet18 and evaluated on other networks. Poor performance on Swin-V2-Tiny may be due to its different network structure (i.e., transformer-based). \n\n1. Transformer-based networks may adapt less effectively to CNNs, especially since all the information is obtained from ResNet. \n2. Additionally, transformer-based networks are well-known for being data-hungry, and distilled datasets contain even less data. Therefore, Swin-V2-Tiny may exhibit the poorest performance despite having the largest model size.\n\nWe note that a similar pattern is observed in RDED [a], as shown in **Table C**.\n\n**Table C**: Cross-architecture performance. Data is obtained from Table 5 of RDED [a]. \n\n| Recover: ResNet-18 | Model Size | SRe2L      | RDED       |\n| ------------------ | ---------- | ---------- | ---------- |\n| ResNet-18          | 11.7M      | 21.7 ± 0.6 | 42.3 ± 0.6 |\n| EfficientNet-B0    | 5.3M       | 25.2 ± 0.2 | 42.8 ± 0.5 |\n| MobileNet-V2       | 3.5M       | 19.7 ± 0.1 | 34.4 ± 0.2 |\n| Swin-V2-Tiny       | 28.4M      | 9.6 ± 0.3  | 17.8 ± 0.1 |\n\n\nI hope the responses address your concerns. Thank you!\n\n[a] Sun, Peng, et al. \"On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm.\" *CVPR, 2024.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received all positive recommendations.  Reviewers acknowledged the motivation of this paper,  the thorough analysis, and the significant results. Most questions were well addressed during the rebuttal and discussion. The AC agrees with the acceptance recommendation.
Decision: Accept (poster)