GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost
Authors: Xinyi Shang
Peng Sun
Tao Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset Distillation
Soft Label
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose an effective plug-and-play approach to consistently enhances the SOTA dataset distillation methods without incurring additional computational costs.

Abstract: Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models.
    In this paper, we introduce a novel perspective by emphasizing the full utilization of labels.
    We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization.
    This finding highlights the necessity of a universal loss function for training models on synthetic datasets.
    Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. 
    Extensive experiments indicate that GIFT consistently enhances state-of-the-art dataset distillation methods across various dataset scales without incurring additional computational costs.
    Importantly, GIFT significantly enhances cross-optimizer generalization, an area previously overlooked.
    For instance, on ImageNet-1K with IPC = 10, GIFT enhances the state-of-the-art method RDED by 30.8% in cross-optimizer generalization. Our code is available at https://github.com/LINs-lab/GIFT.

Published as a conference paper at ICLR 2025
GIFT: UNLOCKING FULL POTENTIAL OF LABELS IN DISTILLED DATASET AT ZERR-ZERO COST
Xinyi Shang! Peng Sun Tao Lin
University College London 7Zhejaang University WWettlake University xinyi. shang. 230uc1 ac. uk, sunpengewest ea, Jintso@westLake..
ABSTRACT
Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models. In this paper, we introduce novel perspective by emphasizing the full utilization of labels. We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. This finding highlights the necessity of universal loss function for training models on synthetic datasets. Building on these insights, we introduce an extremely simple yet surprisingly effective hlug-and-plas approach, GIFT, which encompasses soft label refinement and a cosine similarity- loss function to efficiently leverage full label information. Extensive experiments indicate that GIFT consistently enhances state-of-the-art dataset distillation methods across various dataset scales, without incurring additional computational costs. Importantly, GIFT significantly enhances cross-optimizer generalization, an area previously overlooked. For instance, on ImageNet- IK with IPC 10, GIFT enhances the state-of-the ofrtte--rttttttt art method RDED by 30.8% in cross-optimizer generalization
INTRODUCTION
Dataset distillation (DD) (Wang et al., 2018) has demonstrated its potential significantly reduce data size while maintaining comparable model performance (Cazenavette et al., 2022; 2023; Zhao et al., 2020; Zhao & Bilen, 2023; Zhao et al., 2023). Most existing DD methods focus on optimizing images (Wang et al., 2018; Zhao & Bilen, 2022; Cazenavette et al., 2022; Kim et al., 2022; Liu ct al., 2022), but recent studies (Yin et al., 2023; Sun et al., 2024; Shao et al., 2024; Guo et al., 2024) have highlighted the substantial benefits of soft labels. These studies utilize labels assigned by pre-trained models (also called teacher models), yielding siginificant enhancement in stability during the synthesizing process and considerable performance. Moreover, notable study by Qin et al. (2024) examines the role of soft labels in depth, emphasizing their importance.
To fully explore the utilization of soft labels in the state-of-the-art dataset distillation methods, we conduct a comprehensive comparison of loss functions for soft label in synthesic datasets of IPC 10 2 via the SOTA dataset distillation methods on Tiny-ImageNet and large-scale ImageNet- IK 1. As shown in Figure 1. different dataset distillation methods employ distinct loss functions, and performance varies significantly across loss functions. In particular, simply replacing KL divergence loss (Hinton et al., 2015) with Soft cross cross-entropy loss (Bridle, 1989) for SR results in significant performance drop of 13.3% on Tiny- ImageNet. This substantial gap indicates that models trained on synthetic datasets are highly sensitive to the choice of loss function.
Furthermore, we observe a notable performance degradation in these loss functions when applied across different optimizers. For example, when utilizing distilled data produced by RDED and
"qqual contribution with the first author
Corresponding author.
Our code is available https:/github..om///nssaaa
We also have provided Pytorch implementation
code in Appendix F.
Detailed experiments are elaborated in Section 3.
'The synthetic dataset of DATM for ImageNet- 1 not provided, comparing with DATM impossible.

---

Published as a conference paper at ICLR 2025
KJ.
Sont CE
IISEICT:
GarT(Ourd)
Sont CX
44.0
30-
(13.8)
(114))
24.3
16.9
20.5
20-3
Accuracy
36+
11.
15.4
17.4
17.3
15-
14.3
21-5
27.5
12.5
246
10-
11.7
221
7M
IKD
G-VISM
SRr2L.
ROED
G-VIRSM
DD Methods
DD Methods
(a) Tiny-Imm ImageNet
(b) ImageNet-IK
Figure 1: Top-1 accuracy various synthetic datasets via the SOTA dataset distillation methods across loss functions I'nn--mageNNet and ImageNet-1KK when IPC =10. value means the results of the loss function used by the distillation method itself (e.g.. SRe²L (Yin et al., 2023) uses KL divergence (Hinton al., 2015)). value means the results of our GIFT, and (f) denotes improvements over the dataset distillation methods. It is obvious that our method GIFT significantly enhances the dataset distillation methods.
training with the KI. divergence loss, altering the optimizer from Adam to Adam results in performance decrease from 47.5% to 17.8% (as shown in Table 8) Hence, it is crucial to propose a universal and effective loss function that is robust across various scenarios.
Moreover, one challenge of soft labels is that they are inherently suboptimal, as the performance of the teacher model itself may be limited. For instance, a teacher model trained Image Ik on ConvNet has only 43.6% test accuracy. Despite its worse performance, such model is frequently used in Sun al. (2024); Yin al. (2023) to assign labels. Furthermore, in practical scenarios, is common for the teacher model to have a smaller architecture than the student model due to cross- architecture challenge (Sun et al., 2024), which further limits the performance of the student model when only relying on an inferior teacher model. By contrast, hard labels are accurate and provide reliable supervision information. Nonetheless, directly utilizing hard labels through cross-entropy loss is not effective, as demonstrated in previous studies (Cui et al., 2023) and our experiments in Section 5.6. Therefore, is crucial to effectively integrate hard label information to mitigate the limitations associated with soft labels.
Based on the above two findings, we propose an extremely simple yet effective plug-and houg-and-plas approach called Gaining Improvement from Full Labels Near-zero CosT (GIFT) effectively utilize both hard and soft labels. It first refines soft labels by incorporating an additional smoothing label obtained through hard label smoothing (Szegedy al., 2016). This simple module offers two significant advantages: firstly, it can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns incorrect label of the highest value; secondly, soft labels mainly contain intra-class information, which can hinder class separation to some extent (Zhang et al., 2015). Therefore, relatively sharp label can enhance the dispersion between classes, thereby improving generalization ability, as demonstrated in Section 5.6. After obtaining the refined labels, we find that they are uniformly distributed and are approximately orthogonal to each other, as elaborated in verify theoretically that simply using cosine similarity the loss function
Section 4. Therefore, achieves optimal performance. As shown in Figure 1 (red bars), our method GIFT consistently and significantly enhances the state-of-the-ar dataset distillation methods across various scale datasets.
In summary, our contributions are fourfold:
(a) To the best of our knowledge, this paper is the first to provide a comprehensive comparison of loss functions for label utilization in dataset distillation. Our study reveals the intriguing fact that models trained on synthetic datasets are highly sensitive to the choice of loss function.
(b) We propose GIFT, simple and universal label utilization algorithm including label refinement and a cosine similarity-basee loss function. GIFT is built on top of the off-the-shelf dataset distillation methods and requires no extra information, thus raising no additional cost. Moreover, provide theoretical analysis to support the proposed use of cosine similarity.
*DDD employs AdamW (Loshchilov, 2017) for evaluation.
2

---

Published as a conference paper at ICLR 2025
(c) We identify critical issue that has been overlooked in prior research: cross-optimizer generaliza- tion, as defined in Section 3.1. We reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers detailed in Section 5.5. In contrast, GIFT significantly enhances dataset distillation methods in cross-optimizer generaliza- tion. We conduct both empirical and theoretical analyses of this challenge in Appendix E.
(d) Experiments demonstrate that GIFT significantly improves performance over the state-of-te -hhthh- art dataset distillation methods across varying scales and resolutions datasets, particularly for large-scale dataset distillation tasks. Furthermore, GIFT significantly enhances dataset distilla- tion methods in cross-architecture, cross-optimizer generalization and proves advantageous in applications such as continual learning.
RELATED WORK
In dataset distillation, the majority of existing methods fix the labels one-hot format (Wang et al., 2018; Zhao Bilen, 2022: Cazenavette et al., 2022; Liu et al., 2022; Zhao & Bilen, 2023), with a primary focus optimizing synthetic images. Recent research highlights the significant benefits of utilizzig soft labels to enhance model performance (Sun et al., 2024; Guo et al., 2024; Yin et al., 2023). Methods for obtaining soft labels be broadly classified into two categories.
Optimization-based Soft Labels. The first type involves learning labels, with several studies (Bohdal et al., 2020; Nguyen et al., 2020; Sucholutsky & Schonlau, 2021; Zhou et al., 2022; Guo al., 2024) finding that learning labels significantly improve performance. Recent work (Guo et al., 2024) highlights that optimizing labels can enhance training stability and improve performance.
Teacher model-based soft labels. The subsequent works directly obtain soft labels. Inspired by knowledge distillation (Hinton et al., 2015), TESLA (Cui et al., introduces soft label assignment strategy, directly generating soft labels by leveraging pre-trained teacher models trained on real datasets. These soft labels provide rich intra-class information, thereby improving distillation performance, Following this trend, the state-of-art methods (Yin et al., 2023; Sun et al., 2024; Shao al., 2024) also utilize soft labels predicted by teacher models, achieving significant improvements.
MOTIVATION
3.1 PRELIMINARY
Dataset Distillation. Given large dataset D {xx,,,))) where x, E Rd represents the input sample and yi E {l, .......)) C} denotes hard label, the objective of dataset distillation is to generate synthetic dataset = ((x,, ;5)} such that a model trained on S performs comparably to one trained D. We explore how to fully utilize both soft labels and hard labels in given datasets. Therefore, we re-define the synthetic dataset S as S = { Vj. U1)1, U where X, denotes the synthetic images, and yj and Ú, denote the corresponding hard labels and soft labels, respectively.
y:)
Cross-Optimize Generalization. In deep learning, models use various network architectures and optimization algorithms. Optimizers like SGD and Adam have unique properties that affect model performance and generalization. Therefore, evaluating distilled dataset's performance aross optimizers is essential to ensure its robustness across different training strategies.
Definition 1 (Cross-optimizer Generalization). It refers to the capability of distilled datasets to maintain robust and consistent performance across different optimization algorithms.
3.2 ARE Loss FUNCTIONS PULLING THE STRINGS IN SYNTHETIC DATASET PERFORMANCE?
Why do we need answer this question? Labels in dataset distillation are commonly and typically utilized through a variety of established loss functions, such as cross-entropy (CE) (Bridle, 1989), Kellback-Leibles (KL) divergence (Hinton et al., 2015), soft cross-entropy (Bridle, 1989) or mean squared error (MSE) (Nielsen, 2015). Specifically, the current trae--of-the-r dataset distillation methods L (Yin et al., 2023) and RDED (Sun et al., 2024) employ KL divergence, DATM (Guo et al., 2024) uses soft cross-entropy and G- G-VBSM (Shao al., 2024) simultaneously utilizes MSE and CE. However, different distillation methods employ varying loss functions to train models on synthetic datasets, yet there is notable lack of comprehensive comparison among these loss
3

---

Published as a conference paper at ICLR 2025
functions. Hence, we investigate the performance of four offte--f-hh-arr dataset distillation methods under three commonly used loss functions.
Experiment settings. Our experiments span both small-scale Tiny-Image-y and large-scale ImageNet-IK, Note that the symthetic dataset for DATM on ImageNet- is unavailable, prevent- ing comparative analysis this dataset. We conduct evaluations these synthetic datasets with IPC E (1, 10, 50}. Additional visualizations for Tiny-ImageNer and ImageNet- K at IPC = 1 and IPC = 50 are provided in Appendix D. Notably, our empirical study is conducted from the end-user perspective: we treat the distillation of the synthetic dataset as black box and apply different loss functions during the evaluation of the synthetic datasets.
Results and Analysis. The results, visualized in Figure 1, reveal that even for the same synthetic dataset, the evaluation model performance varies significantly under different loss functions. For example, the performance of L decreases by 13.2% on Tiny-ImageNet when switching from KL loss to soft CE loss but improves by 0.7% when using MSE+CE loss. These findings illustrate that the performance of models trained on synthetic datasets highly sensitive to the choice of the loss function, highlighting the necessity for a unified and effective loss function in dataset distillation.
METHOD
Motivated by these findings, we propose an extremely simple but surprisingly effective plug-and-play approach called Gaining Improvement from Full Labels at Near-zero CosT (GIFT) effectively utilize both hard and soft labels. GIFT includes two key modules: label refinement and a cosine similarity-based loss function. The former aims refine soft labels by incorporating an additional smoothing label derived from hard label smoothing (Szegedy et al., 2016), while the latter is theoreti- cally validated to achieve optimal performance simply using cosine similarity as the loss function. PyTorch implementation of our method is provided in Appendix F.
Label Refinement. As discussed in Section 1, soft labels generated by teacher models are inher- ently suboptimal due to two primary reasons. Firstly, the performance of the teacher model is limited, particularly on complex datasets such as ImageNet- Secondly, soft labels predominantly provide intra-class information, thereby limiting class dispersion.
An intuitive method to address these shortcomings is to integrate with hard labels. On the one hand, hard labels offer accurate and reliable supervision, which can rectify erroneous information provided by soft labels. On the other hand, they can assist in inter-class dispersion. Therefore, refine soft labels by weighing them with smoothed hard labels, thereby solving the two limitations of soft labels.
The refined soft label is defined as Újj < Te, images, Vj is the smoothed label obtained via label smoothing technique for hard label, and iū is the soft label. In our experiments, 7 = 0.1 is validated to be optimal through experiments depicted in Figure 2 in Section 5.6, and Figure 8 and Figure 9 in Appendix D.
Vi
+ (1 7) R;
T where j is the j-th synthetic
Mutual information bounded loss function. Prior study (Sun al., 2024) points that representa- tion learning from any samples X to targets Y is based on maximizing their mutual information. They propose distill the dataset by maximizing (x, Y). where ly denotes the V-information (Xu et al., 2020), which has demonstrated superior performance. However, we observe that most prior studies, including (Sun et al., 2024), have not investigated training models using Iyx(X Y). To address this gap, we explore the application of (X, Y) by deriving an upper bound for the Y-information, as presented in Theorem 1 (Detailed proof is provided in Appendix A).
Theorem 1
The Y- Y-information (X, Y) is upper bounded by a function involving the cosine similarity between the positive pair (X4, Vs). the expected cosine similarity between the anchor x, and negative samples yj, and the number of negative samples K. Specifically,
exp(F((o(x).)))
LISaNCE - log Eyer (xp(f(@o6xx))))
(1)
1 (B|
(
2e (x(xx))
T E vsl ) log(K),
100 (x)))
|1 (x), [B]|
4

---

Published as a conference paper at ICLR 2025
where denotes the temperature parameter, LIstONCE (Oord al., 2018) serves a proxy for Iv(X, Y) (Sun et al., 2024), and represents the similarity function.
Moreover, the targets Y in the synthetic dataset pre- pre-generated using pre-trained models (Sun et al., 2024; Yin et al., 2023). These targets can be considered high-dimensional vectors that are approximately orthogonal to each other (Ma al., 2022; Yu et al., 2023; Awasthi et al., 2024). Consequently, the term ElØ)(x:) yy] approaches zero, allowing (1) to be simplified as:
t (xi) y/
1ls -() + +log(NN,
LINGNCE I
(2)
(x,
where is a small positive term approaching zero.
To minimize this upper bound (2), our proposed loss function for training on distilled data S can defined
[
Φo
x;) y1
C = B(x,
(3)
|
(x)) 1
where úJ is the refined soft label discussed above and we replace the original training loss with (3).
EXPERIMENTS
In this section, evaluate the superiority of our proposed GIFT across various datasets and archi- tectures. First, we demonstrate the superior improvements of GIFT over the state-of-the-art dataset distillation in Section 5.2 and knowledge distillation methods in Section 5.3 Subsequently, val- idate that GIFT achieves near-zero computational cost relative to baseline approaches (Section 5.4). Additionally, we show that GIFT significantly improves cross-architecture and cross-optimizer generalization ( (Section 5.5), ). To further understand the contributions of individual components, we perform detailed ablation studies Section 5.6). Finally, we demonstrate the effectiveness of GIFT in enhancing continual learning performance, detailed in Appendix D. For additional experimental details and comprehensive results, refer Appendix C and Appendix D.
5.1 EXPERIMENT SETUP
Datasets and Networks. We conduct experiments both large-scale and small-scale datasets, including the full 224 x 224 ImageNet-Ik (Deng et al., 2009), Tiny-ImageNet (Le & Yang. 2015) and CIFAR- 1000 (Krizhevsky al., 2009). Following previous dataset distillation studies (Yin et al., 2023; Cazenavette et al., 2022; Zhao et al., 2023; Cui et al., 2023; Guo et al., 2024), we employ ConvNet (Guo et al., 2024) and ResNet-18 (He et al., 2016) our backbone architectures across all datasets. Specifically, Conv-3 is employed for CIFAR- 1R1100 10/100, while Conv-4 is used for Tiny- ImageNet and ImageNet-IK. IK For cross-acchitecture experiments, we additionally utilize large-scale networks, including ResNet-101 101 (He al., 2016) and Swin-V22 V2-Tiny (Liu al., 2021) and small-scale networks, such as EfficientNet-Bi (Tan Le, 2019), and MobilcNet-V2 (Sandler et al., 2018) to verify the generalizability of our approach.
Baselines. We benchmark our method GIFT against state-of-the-art dataset distillation methods. We categorize current state-of-the-aa methods based on two key factors: scalability to ImageNet- IK and the utilization of soft labels. These categorizations Given that our primary focus is on enhancing the use of soft labels in dataset distillation, we restrict comparisons methods that leverage soft labels, including SRe21. (Yin et al., 2023), RDED (Sun et al., 2024), DATM (Guo et al., 2024), G-VBSM (Shao et al., 2024), and CDA (Yin & Shen, 2023) Additionally, DATM only provides synthetic datasets on ConvNet and does not provide the ImageNet- Ik synthetic dataset. CDA provides higher IPC synthetic datasets of Tiny-ImageNet and ImageNet- 1k, distilled using ResNet architectures, so we mainly compare with CDA in Section 5.2.
summarized in Table 13 in Appendix C.
Knowledge distillation (Hinton et al., 2015) is a straightforward approach that utilizes both hard and soft label information. Thus, we also compare our method GIFT with state-of-the-an knowledge distillation methods, including KD (Hinton et al., 2015), WSLD (Zhou et al., 2021), DKD (Zhao et al., 2022), and NKD (Yang al., 2023). Further details on these methods are provided in Appendix C.
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with the state-of-the-art methods of dataset distillation CIFAR-100 and Tiny- ImageNet. In this table, "_" are absent due to scalability.
CIRARIOO
JtageNe
Network
Method
10
50
10
20
SRcaL
136 04
53.7 + 0.5
523
121 + 04
34.5 + 04
46.3 0.1
-RaP
554.:0
RDRD
221
47.5
557.004
179
414
47.2 0.1
CowvNet
DATM
ROCD + Ours
03
50.5 (.35
36.1 1 0.2
57. 57.91.0221 (1:2))
43.0::02
1911 L L2)
26.5 1 0.2
48.0 022 (166
$4.2 0.5
63.3 LD
DAIM - Ours
57.8 0,3 L.)
43.6 (7.06)
27.5 4:02 1.0)
34.8 +: 0. 0.00
G-VBSM
147
40.9 =
0.
84 04
14.5 0.5
47.0
G-VBSM
0.21
57.2
89 (.))
369 0.
47.8 012
127
SRc*I, - Ours
127 1.2
44.3 + 0.3 1.6)
56.6 036 0.8
14.2 04 (.5)
44.2 = 0.3 (1 0.7)
$4.5 0.2 0.63
RDED
4.7
528 1: 0.2
64.4 0.1:::0.1
15.1 1 0.3
48.2 1: 0.4
$1.6 1 0.3
ResNe: 18
ROCD Ours
3.0102()03)
$4,0 1 L.3
47.5 0.
65.31.0222
159 (3
26.7
49. 1 0.1 (1)
4 0.5
58,1 0.
DATM
(1(00.)))
0. (1.04)
20.0 (1
G-VRSM
134
48.5
62.0 =
88
39.
$2.8 0.2
G-BBSM - Ours
137 03 0.3)
49.2 1. 0.2 0.7)
625 1. 0. (1 0.5
93 03 0.5)
40.5 1 02 (1 0.6)
53.1 1 (f
Table 2: Comparison with the state-of-the-art methods of dataset distillation on ImageNet-1K. In the table, (1) means the improvements over these methods.
ImegoNes-IK
ConvNet
RosNat-18
Mechod
10
30
100
10
50
100
SR
12.5 1:5::00 0.3
35,4 +: 1.0
40.1 1:
31,5 4
1: 0,1
54.3 0.2
SRcPL Ours
14.2 0.6 (1 1.7)
38.1 +: 0.4 2.7)
41.5 + 0.2 1.4)
31.9 4
0.2
(1 0.4)
20.1
0.2 0.6)
54.8 + 0.1 0.5)
RDED
20.1 1: 0.4
38,5 1: 0,2
41.8 1: 0.2
41.4 1: 0.4
55.5 1:5::000 0.2
58.8 :1: 0.1
RDED Ours
24.0 1100::00 0.8 3.9)
39.5 4 0.1 (1 1.0)
4..5: 0.7)
43.2 1
0.1
L8)
36.5
39.3:1 (1
37.3
40.1
57.3
G G-VDSM Ours
21.3 2: (12 1.7)
39.1 1: 0.3 (1 1.8)
42.1 1: 0.3 2.01
37.9
1 0.5
(1 L2)
13.1::0::2 0.8)
57.6 ::
(1 0.3)
Implementation details of GIFT. Our method does not involve any distilling datasets process. We obtain all synthetic datasets directly from the source data provided by the authors 5. Notably, distilled data is generalized using both ConvNet and ResNet-18. We replace the loss function during evaluation. Thus, our method is plug-and-play approach that can be easily integrated into existing dataset distillation pipelines without additional dataset synthesis or modification.
For the data augmentation of synthetic datasets, only synthetic datasets generated via DATM (Guo et al., 2024) are processed using ZCA whitening, as these datasets were initially distilled through ZCA whitening. Other distilled datasets processed using DSA (Zhao & Bilen, 2021), as detailed in Table 14. All experiments are conducted using an NVIDIA RTX 4090 GPU.
Hyperparameter Settings. We provide detailed hyperparameter configurations for our synthetic dataset evaluation in Appendix C. Following recent works (Yin et al., 2023; Shao et al., 2024), the
evaluation on all datasets uses the parameters outlined in Table 15. We set the coefficient of label smoothing a = 0.1 and the weight hyper-parameter 7 = 0.1 for all methods across various synthetic datasets, as 7 = 0.1 validated to be optimal through experiments depicted in Section 5.6 and Section 8 in Appendix D.
5.2 CAN GIFT IMPROVE PERFORMANCE OF DATASET DISTILLATION?
Small-Scale Dataset Comparison. In Table 1, we present the test accuracy on CIFAR-100 and Tiny-ImageNet datasets before and after applying our GIFT algorithm. Notably, DATM does not provide synthetic datasets when IPC =1, nor does it provide synthetic datasets for ResNet-18. Consequently, we used ConvNet synthetic data to train ResNet- 18. It is evident that applying GIFT increases performance for all baseline methods. Specifically, the SRe²L method exhibits the most significant improvement, with an accuracy gain of up to 4.3% on CIFAR- 100 when IPC =10. This particularly noteworthy as GIFT requires no additional information or cost. The considerable accuracy gains can be achieved simply by replacing the loss function with our proposed approach.
SRc2L: https://gtthub.com//// www..oo///.A.ALa//eeeeeea.
RDED: http::/gtttubbcom//iiillllttt
DATM: https:/ggyaftemmattgghhubbbbbubu VBSM Caataset_Condensatie
G- GBV::
CDA: https//ggituu..om// http://hiacooollllAaaaaaaaeeeeaa

---

Published as a conference paper at ICLR 2025
Table 3: Comparison with CDA under higher IPC small-scale Tiny- ImagemmaeNNet and large-scale ImageNet- IK on ResNet-18. In the table, (f) means the improvements over CDA.
Tiy ImageNet
ImageNet
Method
50
100
200
49,5
53.5
CDA Ours
54.5 + 0.3 5.)
56.6 + 02 (1 (.))
54.8 = 0.2 1.1
59.0 + 02 0.8)
63.9 + 0.1 (1 0.5)
Table 4: Comparison with RDED Large-seale Netwrok. In the table, (1) means the improvements over RDED.
Tiny ImageNett
ImageNet-JK K
Method
10
50
10
RDED
47.14 03
55.1 0.3
12.3 0:0.
58.6 0.1
RDED Ouss
48.6 + (2 15)
56.2 + 022
43.5 022
59.4 0.2
Table 5: Comparison with the knowledge distillation methods on the synthetic dataset via RDED (Sun
al., 2024) using ConvNet. In this table, bold means the hest result, underlined means the second best, and (1) denotes improvements over the second best baseline.
CIFARIOO
Tay ImageNet
ImageNer.1k IK
10
10
50
100
49.73
43.6
45.2
519
+
40.7
0.1
27.5 + 0.4
2.1
49.0
56.7
: 0.2
47.2
20
31.1-+0.11
NKD
16.3
03
363
37.5 17.81.0
4111
19:5
0.1
27.9
2..91.0.
10.6 0.22
GIFT (ours)
50.6 1: 0.3
1.6)
$7.9 1:
0.2
1.20
44,9 = 0.211
0.211
3.0)
48.3 =
0.1
L.)
24.0 1: 0.8 3.4)
39.5 0.0.1 (.4)
42.81 0.1 6.01
Large-Scale Dataset Comparison. In the large-scale ImageNet- k dataset, as reported in Table 2, our proposed method GIFT consistently improves all baseline methods across all TPC values in 10, 50, 100, using both ConyNet and ResNet- 18 as evaluation models. Specifically, compared with the current state-of-the-ar methods RDED and G-VBSM, GIFT achieves significant performance gains of 1,8%and 1.2% on ResNet-18 when IPC =10, respectively. The substantial performance improvements obtained by GIFT demonstrate its capability effectively scale to large-scale datasets.
Comparison under Higher IPC. Given that only CDA (Yin & Shen, 2023) provides higher IPC synthetic dataset distilled, our comparison primarily centers on CDA. The results in Table 3 demonstrate that our GIFT method substantially improves the performance of CDA. Furthermore, also achieves significant enhancements at higher TPC.
Comparison on Large-scale Network. In addition conventional networks like ResNet-18, employ the state-of-the-a dataset distillation method, RDED, to generate distilled data for Tiny- ImageNet and ImageNet- 1K using Swin Transformer (Liu et al., 2021). The results, as shown in Table 4, indicate notable enhancements, verifying the effectiveness and promise of our method.
5.3 CAN KNOWLEDGE DISTILLATION WORK?
A straightforward approach to combining soft labels and hard labels is knowledge distillation (Hinton al., 2015), which transfers knowledge from a teacher model to a student model using hard labels (cross-entropy loss) and soft labels provided by strong teacher model (KL divergence loss). In Table 5, we compare our proposed method, GIFT, with the state-of-the-e knowledge distillation techniques across synthetic datasets distilled via RDED (Sun et al., 2024) 6.
can be observed that GIFT outperforms all knowledge distillation methods. We attribute the failure of knowledge distillation methods to the extremely small size of synthetic datasets, which significantly hampers the performance of knowledge distillation methods, as corroborated by (Stanton al., 2021). Therefore, knowledge distillation is not well-suited for our problem, further highlighting the effectiveness of GIFT.
'DDED is the current state-of-the-art method as shown in Table 1. conduct the experiment on its synthetic datasets.
7

---

Published as a conference paper at ICLR 2025
Table 6: Training Time (s) and memory (GB) costs on three synthetic datasets using ResNet-18 when IPC=10. (+) denotes additional cost over these methods.
CIFAR-100
Tin--ImigeNet
ImageNNs-11 IK
Method
Training Time
Memary
Training Time
Memory
Training Time
Memory
SR
180.22
0.69
1249:66
2,01
2011.24
2.32
S Ours
181.36 1.14)
0.69
1275.75 26.09)
2,01
2078.59 (a 67.35)
2:32
RDED
171.07
0.69
1272.28
2.01
2066.35
2.32
RDED Ouns
175.17 (+220)
0.69
1298.73 (+ 26.45)
2,01
2121.61 (+ 58.26)
2.32
DATM
132.51
1.36
1018:31
12.55
DATM + Oura
179.06 (4.666(4 6.55)
1.36
1039.65
34)
12.55
G-VBSM
183,90
0.69
1256.35
2,01
2074.72
2.32
G-VBSM
3,98)
1282.68
26.26)
2,01
2129.98 55.26)
2.32
Table 7: Top-1 accuracy cross-acchitecture generalization Tiny-ImageNet. We the synthetic datasets distilled (D) on ConyNet and ResNet-18 on Tiny-ImageNet when IPC=10. Then, evaluations (E) performed across both small-scale and large-scale architectures. (1) denotes improvements over these methods.
Small Scale Architecture
Large Scale Architecture
D/F
ConvNet
ResNet-18
EfiiieeNNa-00
MobileNa--2
RecNes-101
Swin- V2-Tin
SR
3455 0.4
43.04
43.04.1.11 1.0.1
11.2
14,0 1. 0.
0.55
10.3 1 0.4
SRc Ours
37.52003 3.0)
44.2 03 1166)
1 (.D
14.4 1.4003
106 + 0,3
112 (1
ComoNet
0.3
02
90.3
50.2
28.2
268
Olus
440002 2.6)
47.2= (1
31.4 (f
31.3 01
308 18
28. (4
DAIM
36.1 = 0.2
27.3
02
18.0 03
14.7 02
15.3 + 08
4.1 +
DATM + Ours
37.8 0.3 1.2)
29.0 = 02
18.4 0. 0.4
17.5 4 +.0. (1.8)
1684 0.5 (1)
16. 0.3 12:)
VBSM
34,5 0.5
0.3
13.7 1 1.4
15,0 15.0.100.
7.7 0.06
10.4 0.4
G-VBSM
0.6
155 (1
2.8)
SRe-L
19.2 = 0.1
43.5 = 0.1
11.611
11.91003
80:1 0.2
L Ours
19.4 (2.0)
44.2 0.30 0.7)
122 0221:0220 (.)
12.3 0.0...111 0.4)
9.0 1 0,5 (1 0.31
8.8.1 (f 0.8)
ResNet
0.3
24.1
RDED Ours
29.9 (. 0.7)
492=0.1
02 LD)
24.1 0: 0.6)
03
204 0.3 0.8)
G-VRSM
16.0 = 0.3
39.9 = 04
888 ===00 0.1
11.5004
6.3 1.1
6.500.3
G. VBSM & Ous
16.5 044
40.=0.2(006)
10.1 02 E3)
11.8 (33)
9.2 (1.9)
8.24 0.3 1.7)
5.4 CAN GIFT ACHIEVE NEAR-ZERO COST?
We perform experiments to evaluate memory and training time costs using ResNet-18 1 across datasets with varying scales and resolutions, as presented Table 6. Our method demonstrates no additional peak memory usage and incurs negligible computational overhead. This efficiency is due to its emphasis on label refinement and the implementation of a general and simple loss function during the evaluation phases. Importantly, despite the negligible additional cost, GIFT yields significant performance improvements across datasets of varying scales and resolutions.
5.5 CAN GIFT IMPROVE GENERALIZATION?
Cross-Architecture Generalization. To validate the enhancement of generalization capability by our GIFT, it is necessary to assess its effectiveness across various neural architectures not encountered during the dataset synthesis phase. We evaluate performance on both small and large-scale model architectures. Table presents the performance before and after applying our GIFT to dataset distillation methods. The results indicate that GIFT enhances the cross-architecture generulization of all dataset distillation methods across diverse architectures. Notably, our method shows significant improvements when generalizing from small networks to larger networks. For instance, GIFT yields performance gains of 2.6% and 7.8% for RDED and G-VBSM, respectively, when synthesizing data using ConvNet while training model on ResNet-
The success of our method is attributed to its stability. In crss-architecture scenarios, soft labels may not be sufficient due to architectural differences (Vyas et al., 2020). However, our cosine similarity approach inherently includes a normalization operation, mitigating the negative impact of label fluctuation. These results are promising, indicating that our method, which does not incur additional computational costs, is well-suited for applications involving large-scale models.
Cruss-optimizer Generalization. Similar to cross-ares arshtteccee generation, it is crucial to estimate the cross-optimizer generalization of dataset distillation methods. Different optimizers, such as
8

---

Published as a conference paper at ICLR 2025
Table 8: Top-1 accuracy (%) ) cross-optimization generalization Tiny- -mageee and CIFARI00 when IPC =10. We evaluate the performance synthetic datasets across various optimizers.
Ditaset
CIFARIO0
Tmn--mageNNe
Optimiaer
SGD
Adam
Adamn
SGD
Adam
AdamoW
SR
1.5 d:
8.7 d
34,5 4
0.6 0:00
3.1 d: 0.1
33.7 0,5
SRt Ouurs
43.0 + (f 42.0)
44.7 + 0.6 37.5)
0.0+
43.8 + 0.5 (f 43.2)
45.2 + 0.2 42.3)
37.5:t
3.0)
RDED
199 +: 0.0
17.8 + 0.1
47.5 + 0.3
0.6 + 0.0
4.5 + 0,2
41.4 + 0.3
RDED Ours
53.4 1441::022 51.5)
53.7 1: 0.4 35,9)
46.6 1: (f
66.5 1: 0.2 42.0)
44.0 0.2 (16).6)
37.3-
36.7
36.1
27.8
DATM + Ours
38.54 0.2 (1 122)
40.0:: 0.2 (.33)
17.8 (1
30.1 4: 1.99
29.1 4: 0.1 (1 1.3)
27.3 0.2 L.))
G VISM
24.2 1.8
41.2 2: 0.4
10.9 1 0.4
41.4 1:
36.5 1: 0.4
34.5 1: 0.5
G-VBSM Ours
51.3
022
45.3
0.7
Table 9: Comparsion with loss functions employed in dataset distillation. The experiment is conducted on synthetic datasets distilled via RDED (Sun et al., 2024). In this table, bold means the best result, underlined means the second best, and (1) denotes improvements over the second best baseline.
Tay-InsgeNat
ImngeNas-IK
10
10
Hand Label
CB
1:
408 1
14.2
26.9 1 06
2 0.1
17.5 0.1
21.5 0.1
KL J5
15.7
47.2 0.1
38.5 0.2
41. 0.2
Label
0:03
4: 0.2
47.3 022
38.6 4: 0.3
0:33
MSE Co
00
52.1
0.4
170
303
37.1:000
Soft Label
Handg
Sunl C
MSE CE
Kl-+ CBs
30. +
68.3 044
47.4 2:4+:00
56.2 4 0.0
16.3
0:4
41.7 + 0.5
326 003
41.62:03
43.2 02
47.1 0.1
6682:0
20.5 + 0.4
152
20:32 02
29.1 2..1
35.3 2
35.22:03
$4.7 1 0.6
40.5 : 022
Wi 022
GEFT (surs)
50.6 0.3
= 012
44.0 + 0.2/1 2.30
1.05
0.8 3.3)
:: 01
42.5%t 0.1
SGD and Adam, exhibit distinct characteristics that influence model performance and generalization. Therefore, in practical applications, it is necessary to choose the most appropriate optimizer according to the training conditions. In this experiment, we report the accuracy before and after applying our GIFT to baseline methods across multiple optimizers not seen during dataset distillation, as shown in Table 8. The results clearly indicate that GIFT enhances the generalization ability of all baseline methods. More results ImageNet-Iie IK can be found in Table 16 in Appendix D.
It is notable that LRe2L and RDED perform poorly in this cross-optimizer challenge. However, with our method, performance increased by 42.6% and 51.5% using SGD on CIFAR-100. We present an emperical and theoretical analysis of cross-optimizer generalization Appendix E.
5.6 ABLATION STUDY
Is Our Loss Function the Best? To validate the superiority of GIFT, we compare with existing loss functions, utilizing both hard and soft labels. The results, presented in Table 9, clearly demonstrate that GIFT consistently outperforms other loss fanctions and their combinations.
For hard label utilization, the cross-entropy (CE) loss exhibits subpar performance, mainly due to the limited information content in typically small synthetic datasets. For soft label utilization, Jensen-Shannon (JS) divergence loss marginally outperforms the KL divergence, consistent with observations in (Kim et al., 2021). In summary, existing loss functions in synthetic datasets fail to fully exploit the potential of all labels. In contrast, our method leverages both hard and soft labels simultancously. thereby maximizing label utilization potential and improving performance.
Are Both Modules of GIFT Necessary? We conduct an ablation study to assess the necessity of the label refinement and cosine similarity loss function on the small-scale dataset in Table 10 and the large-scale dataset ImageNet- 1K in Table 17 in Appendix D. We compare the complete method with variants lacking either the teacher label refinement or the cosine similarity loss function. In the absence of both modules, the method is trained using its native loss function.
is obvious that when only one module is employed, the cosine similarity loss function significantly enhances performance due to its direct label utilization. Label refinement consistently enhances per- formance, regardless of the presence of cosine similarity loss function, demonstrating its effectiveness. Thus, both modules essential for enhancement, consistent with our analysis in Section 4.

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study of label refinement (Refine) and cosine similarity loss function (Loss) CIFAR 100 and Tiny-ImageNNt when =10 This evaluation is conducted on both optimization based (SRe"L LL (Yin et al., 2023)) synthetic datasets and non-optimization based (RDED (Sun et al., 2024)) synthetic datasets.
GIFT
CIFARIOU
Thy ImageNet
+
33.7 +
$2.3
12.1
34.5 0.4
+1
SR
0.5
12.5
35.1 (.
0.3
0.1
12.7
(4
15.1 0.3
38.0 0.5 4.3)
55.4
0.
13.1
02 1.0
31.5
0.30
47.1 ( 0.8)
47.5
17. 0.
0.3
0.3 0.5)
0.
56.3
1822
0.
41.9
11.9+0..1100
(1 Q3)
23.8
0.2
1.71
69.5 1:
0.200 2.0)
$7.0 1:
0.1
13)
185
03
0.60
42.9
0.2 1.5)
47.5
0.01 0.3)
24.7 +
0.3
2.6)
50.6 +
0.3
3.1)
$7.9 = 0.2
2.20
19.1
0.3
1.2)
44.0
44.0:000
0.211 2.6)
48.3 1.13
Moreover, to verfiy the efficacy of label refinement, we compare the label accuracy before and after refinement. The results, shown in Figure 10 in Appendix D demonstrate that it leads to significant performance improvements, highlighting the critical role of the proposed label refinement.
Influence of Hyper-parameterry ? We examine the impact of the weight hyperparameter y, defined in Section 4. As shown in Figure 2, GIFT achieves optimal performance when ? 0.1 across various datasets. This consistency attributed the fact that the soft labels are generated by pre-trained models. Specifically, both RDED and SRe2L utilize the same pre-trained model.
Notably, for values of y greater than 0.1, a significant performance decline is observed across all methods as y increases. A plausible explanation is that larger values of y diminish the intra-class information content in soft labels. This observation aligns with our findings in Table 9, where training with exclusively hard labels via CE results in poor performance. To verify that 7 = 0.1 is also optimal for different settings, we conduct experiments on different network architectures and augmentation, as shown in Figure 8 and Figure 9 in Appendix D.
Can GIFT Enhance utilization of Hard and Smoothed Labels? To evaluate the efficacy of GIFT across various data rypes, we conduct experiments on both distilled and randomly selected datasets, employing hard and smoothed labels. The results are presented in Table 18 and Table 19 Appendix D. Obviously, GIFT consistently enhances label utilization across various label types.
SRo-1
RDED
C-VB
SRc
RDED
G-VBSM
SRc
RDED
G. GBVB
Accuracy (%)
Accuraac
30-
Accuracy
30
30-
20
30
(a) CIFAR 100
(b) Imageeet
(c) ImageNet-Ii IK
Figure 2: Top-1 I accuracy (%) the state-of-the-art dataset distillation methods on various synthetic datasets
when IPC= =10 on ResNet-18 18 with different 7.
CONCLUSION
This work introduces a novel perspective on dataset distillation by emphasizing the full utilization of synthetic labels. We first conduct a comprehensive comparison of existing loss functions used for soft labels in the field of dataset distillation. Our findings reveal that models trained on synthetic datasets exhibit significant sensitivity to the choice of loss function. Building on these observations, we propose a simple yet effective plug-and-play method, GIFT, which fully leverages synthetic labels without requiring additional information. The method incorporates label refinement and introduces a cosine similarity- based loss function. Furthermore, we provide a theoretical analysis to substantiate the use of cosine similarity. Experimental results across various scales and resolutions of image datasets demonstrate that GIFT consistently achieves superior performance compared to of-t-of-the-rtt dataset distillation methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We thank anonymous reviewers for their precious comments and feedback. This work was supported in part by the National Science and Technology Major Project (No. 1022000115101), Research Center for Industries of the Future (RCIF) at Westlake University, Westlake Education Foundation, and Westlake University Center for High-performance Computing.
REFERENCES
Pranjal Awasthi, Nishanth Dikkala, Pritish Kamath, and Raghu Meka. Learning neural networks with sparse activations. In The Thirty Seventh Annual Conference on Learning Theory, pp. 406-425. PMLR, 2024.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint 2oXiv:2006.085722 2020.
John Bridle. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. In Advances in neural information processing systems, volume 2, 1989.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4759, 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci A Efros, and Jun- Vun-Yan Zhu. General- izing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7739-3748, 2023.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 6565-6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Ziyao Guo, Kai Wang. George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via allggnd trajectory matching. In International Conference on Learning Representations, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recagnition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha. and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102-11118 PMLR, 2022.
Tachycon Kim, Jaehoon Oh, Nik Kim, Sangwook Cho, and Se- Young Yun. Comparing kullback-leibler divergence and mean squared error loss knowledge distillation. arXiv preprint arXiv:2/05.089/9, 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 4rXiv:1412.69980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
11

---

Published as a conference paper at ICLR 2025
Songhua Liu, Kai Wang. Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in neural information processing systems, volume 35, pp. 1100-1113, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
I Loshchilov, Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
Yi Ma, Doris Tsao, and Heung-Yeung Shum. On the principles of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, 23 (9):1298-1323, 2022.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-leaming from kernel ridge- regression. arXiv preprint hat::2011.000500 2020.
Michael A Nielsen. Neural networks and deep learning, volume 25. Determination press San Francisco, CA, USA, 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /80700380 2018.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 524-540, 2020.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. A label is worth thousand images in dataset distillation. arXiv preprint arXiv:2406. 10485, 2024.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In CVPR, 2024.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34: 6906-6919, 2021.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference on Neural Networks (LJCNN), pp. 1-8, 2021.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: efficient dataset distillation paradigm. In CVPR, 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14. PMLR, 2019.
Nidhi Vyas, S. Saxena, and Voice. Learning soft labels via meta learning. ArXiv, abs/2009.09496. 2020.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint /11. 2018.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint 1068 2020.
12

---

Published as a conference paper at ICLR 2025
Zhendong Yang, Ailing Zeng, Zhe Li. Tianke Zhang, Chun Yuan, and Yu Li. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
17185-17194, 2023.
Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. 2023.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. arXiv preprint arXiv:2306. 1092, 2023.
Yaodong Yu. Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 36:9422-9453 2023.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng, Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3903-3911, 2020.
Zhao Zhang, Mingbo Zhao, and T. Chow. Graph based constrained semi-supervised learning framework via label propagation over adaptive neighborhood. IEEE Transactions on Knowledge and Data Engineering.
2222362223766 2015.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685 PMLR, 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.077 200.:0113,, 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.0520 05929, 2020.
Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953-11962, 2022.
Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 7856-7865. 2023.
Helong Zhou, Liangchen Song. Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2 120.2200200000 102.00050.000.00 2021.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. Advances in Neural Information Processing Systems, 35:9813-999 33-9977, 2022.
13

---

Published as a conference paper at ICLR 2025
A
PROOF OF THEOREM
Proof. The InfoNCE loss is defined as:
LINONCE = E log exp(sim(2 2z. 3/)/T)
(4)
==1 exp(sim(zz) 11)
where sim( W) represents the cosine similarity between 24 and y/:
2i 1/i
sim (2 Us)
(5)
2| |1
Substituting the expression for cosine similarity into the InfoNCE loss:
exp
LIsANCEE
E
log
N j=1
54 [T>>|||l>>>>>
exp
Zt - Di
24 yy
-E
log exp
log
exp
| |
j=1
(6)
24 ys
23 . Uo
-EE
|| |B/l|T
log
exp
|| 112|||1u|17
=-E
Z4 yi
]
IE
log Σ exp
24 Yj
||= |/
||: 1
Applying Jensen's inequality to the logarithm:
( ||g [vvl
LInfANCE S -EE
+ log
E
Σ
exp
24 Vj | M/ ||
Assuming the negative samples B/ are drawn from similar distribution, we approximate the denomi- nator:
2, yo
El y5 |1 Bill, [1TT
exp
||
||
= N exp
(8)
j=1
Substituting this approximation into the upper bound:
LIntONCE 5
E
2, y/s
E[2, V3l
+ log(NN
2 [1v..
2 [E [1u,,111
B RELATED WORK
B. KNOWLEDGE DISTILLATION
A straightforward method to simultaneously utilize soft and hard labels is knowledge distillation (Hinton et al., 2015), which transfers knowledge from a large teacher model small student model. In this training process, the student model is supervised by hard labels and soft labels from the teacher's output. Many following works aim to enhance the use of soft labels for more effective knowledge transfer. (Yuan al., 2020) investigated the regularization property of soft labels and introduced a teacher-free distillation approach. WSLD (Zhou et al., 2021) analyzes soft labels and distributes different weights for them from perspective of bias- vias-nariance trade-off. DKD (Zhao et al., 2022) decouples the logit and assigns different weights for the target and non-target classes.
Despite the promising potential of knowledge distillation in transferring knowledge from teacher to student models using soft labels, its application to our problem yields limited improvement. A detailed analysis and comparison of these limitations provided in Section 5.3.
14

---

Published as a conference paper at ICLR 2025
C EXPERIMENT DETAILS
Datasets. As described Section 5.1, , we evaluate the state-of-the-art dataset distillation methods and our proposed GIFT on both small-scale and large-scale datasets. More Information about datasets utilized are listed in Table 11.
Table 11: Details about datasets
Dataset
Num of Classes
IPC of Trainset
IPC of Testset
CIFAR-10
10
5000
1000
CIFAR-100
100
500
100
Tiny-ImageNet
200
500
50
ImageNet-IK
1000
732 1300
50
Models. The experiment utilized a plethora of pre-trained models, and we provided the accuracy of these pre-trained models in the Table 12. The results are provided for reference only.
To elucidate the rationale behind our method selection for comparison, we categorize current state-of-the-ar methods based two key factors: scalability ImageNet-IK and the utilization of soft labels. These categorizations are summarized in Table 13.
Baselines.
Given that our primary focus on enhancing the of soft labels in dataset distillation, we restrict our comparisons to methods that involve soft labels:
TESLA (Cui et al., 2023) marks the first distillation approach that extends to the full ImageNet- K, circumventing the extensive memory demands associated with MTT-deriiee methods through constant memory footprint. However, TESLA does not provide public synthesic datasets, so we are able to conduct it.
SR (Yin et al., 2023) and RDED (Sun et al., 2024): both of them use soft labels assigned by a teacher model and use them via KI. divergence.
DATM (Guo al., 2024): initial soft labels assigned by multiple teacher models and then optimized based on trajectory matching. Finally, this method employs soft cross-entropy loss for soft labels.
G-VBSM (Shao et al., 2024): soft labels assigned by multiple teacher models and then used via MSE-CE loss function.
CDA (Yin & Shen, 2023): soft labels are assigned by a teacher model and are used via soft cross-entropy loss.
Knowledge distillation (Hinton et al., 2015) is straightforward method utilize labels, especially for soft labels. Therefore, we also compare our method GIFT with the state-of-the-arr knowledge distillation methods that focus soft labels utilization:
Table 12: Accuracy of pretrained models.
Dataset
Model
Size
Accuracy
resnet18 modified
32 x 32
93.86
CIFAR-10
ConvNet-3
x 32
82.24
resnet18 I ConvNet-3
32 x 32
72.27
CIFAR-100
x 32
61.27
resnet18 modified ConvNet-4
x 64
61.98
TmageNn
64 x 64
49.73
resnet18
224 x 224
69.31
ImageNet-1k
ConvNet-4
64 x 64
43.6
15

---

Published as a conference paper at ICLR 2025
Table 13: Categorize methods based their utilization of soft labels and their scalability to ImageNet-IK.
G-VBSM
SRe2L
SeqMarch
DREAM
DM
DSA
Solt Label Scale ImigeNe-1K
KD (Hinton et al., 2015): is the first method transfer knowledge using both hard and soft labels from the teacher's outpur.
WSLD (Zhou et al., 2021): ) it analyzes soft labels and then distributes different weights for them from a perspective of bias-variance trade-off.
DKD (Zhao et al., 2022): it decouples the logits and assigns different weights for the target and non-target classes.
NKD (Yang et al., 2023): it finds the sum of the two non-target logits is different, preventing logits' distributions from being identical. Therefore, it normalizes the non-target logits to equalize their sum.
Evaluating main results. For both dataset distillation and performance evaluation, employ identical neural network architectures. Consistent with previous studies (Cazenavette et al., 2022; Cui et al., 2023; Zhao al., 2023), Conv-4 for Tiny- Jin-IImaeNNet (with the exception of DREAM, which utilizes Conv-3) 3) and ImageNet-IK, Conv-5 for ImageNet-10, and Conv-6 for ImageNet- 100 distillation. In line with (Cazenavette et al., 2022: Cui et al., 2023), MTT and TESLA apply reduced resolution for distilling 224 x 224 images. According (Yin al., 2023), for retrieving and evaluating distilled datasets, S and GIFT adopt ResNet-18.
Conv-3 for CIFAR-10 10 and CIFAR- 100 distillation tasks,
Evaluating the distilled dataset. Consistent with recent works (Yin et al., 2023; Shao et al., 2024), the evaluation the distilled dataset follows the parameters outlined in Table 15. Furthermore, implement Diffcrentiable Siamese Augmentation (DSA) as described by (Zhao & Bilen, 2021) to enhance images during both the distillation and evaluation phases of our experiments.
Differentiable Siamese Augmentation (DSA). We use DSA (Differentiable Siamese Augmen- tation) as a tool for image augmentation. For the sake of clarity, we delineate the DSA operations utilized in Table 14, alongside their respective transforms and probabilities.
Table 14: Differentiable Siamese (DgmentationnDSA)) and ratios
DSA
Transform
Ratio
Brightness=1.0
Color
Color Jitter
Saturation=2.0
Contrast=0.5
Crop
Random Crop
Crop Pad=0.125
Cutout
Random Cutout
Cutout=0.5
Flip
Random Horizontal Flip
Flip=0.5
Scale
Random Scale
Scale= 1.2
Rotate
Random Rotation
Rotate= 15.0
D EXPERIMENT RESULTS
Application:
Continual Learning Following prior studies (Zhao & Bilen, 2023; Kim et al., 2022; Yin et al., 2023) that leverage synthetic datasets in continual learning to assess the quality of synthetic data, employ the GDumb framework (Prabhu al., 2020) for our continual learning setup. This framework sequentially stores prior training data in memory and utilizes both new and stored data for model training.
We conduct insssiinrremetall learning on Tmay-Ima with an IPC =10 using ResNet-15 Figure 3 illustrates both 5-step and 10-step class-incremental learning strategies, partitioning the 200
16

---

Published as a conference paper at ICLR 2025
Table 13: Evaluation Hyperparameter setting
Conlig
Value
Explanation
300 for ImageNet- Ik,
Epochs
300/1000
1000 for default
Optimizer
AdamW
NA NA
Learning Rate
0.001
10 for 0 Num Images 10,
50 for 10 < Num of Images < 500.
Batch Size
10/50/100/200
100 for 500 < Num of Images v 20000,
200 for 20000 < Num of Images
milestones=|2 x epochs // 3, x epochs // 6] gamma=0.2
Scheduler
MultiStepLR
Augmentation
DSA strategy
color, crop, cutout, flip, scale, rotate
Table 16: Top-1 accuracy cross-pttimization generalization ImageNet-1IK when IPC=10.We evaluate the performance synthetic datasets across various optimizers.
ImageNat-IK
Dotaset
ConvNet
ResNet
Method
SGD
Adam
Adama
SGD
Adam
Adam
SRe LL
0.1 1:
0.1 = 0.0
12.5 + 0.1
0.1 1.0
0.1 = 0,0
31.5 0,3
02
36.6 0.2
14
24.5 0.2
RDED
0.1 0.0
=:
20.1 0.4
0.1 0.0:::00
0.0
41.4 0.4
RDED + Ours
26.7 0.66: 26.5)
30,9 = 0.7 (1 26.5)
(1:3::2
45.8 1 048:004(100 30.8)
29.1 + 0.3 29.0)
43.2 0.12 L.8)
VBSM
0.8
25.0 +
22.6 226-4-05
4: 0.2
27.0 0.2
0.2
VBSM
(2
1..
classes into either 5 10 learning steps, corresponding to 40 and 20 classes per step, respectively. evident that our results substantially improve upon the baseline method RDED
50
50
(96)
Accurecy (96
48
48
44
RDED (Or))
RDED CIPTOOUr)
42
42
120
Number of classes
Number of classse
(a) 5-step
(b) 10-step
Figure 3: 5-step and 10-step class-incremental learning on Tiny Tiay-emageNet on ResNee--1.
Comprehensive Comparison Between Different Loss Functions. Our experiments span two datasets, including Imag-lma,,Net, and ImageNet- IK on IPC E {1, 10, 50} using ConvNet (Guo et al., 2024). Note that the synthetic dataset for DATM on ImageNet- is unavailable, precluding comparisons this dataset. We evaluate IPC E (1, 10, 50). The results, visualized in Figure 4,
Figure 6, Figure 5 and Figure 7, reveal that the performance of models trained on synthetic datasets highly sensitive to the choice of the loss function, highlighting the necessity for a unified and effective loss function in dataset distillation.
Influence of Hyper-parameter on ConvNet. We also examined the impact of the hyper- parameter using ConvNet, and the results shown in Figure 8. Similar to the findings with ResNet, GIFT achieves optimal performance when 0.1.
RDDDD is the current state-of-the-art method as shown in Table 1. conduct the experiment on its
synthetic datasets.
17

---

Published as a conference paper at ICLR 2025
KL
Soft CE
MSE+CE
GIFT(Ourr)
25
Accuracy (%)
(1122)
19.1
17.9
17.3
(11.0)
15.
12.2
13.1
(10.5)
9.8
8.4. 8.9
5.3
5
4.1
0
SRe
RDED DD Methods
G-VBSM
Figure 4: Top-1 accuracy on various synthetic datasets via the SOTA dataset distillation methods across loss functions on Imng- when IPC=1.
Soft
MSE+CE
GIrr(Ourr)
25
Accuracy (%)
(%)
20-
10-
(10.5)
(+2.0)
(11.8)
4.4
5.5
4.7
5.3
6.0
5.5
3.5
3.5
2.5
2.6
S
RDED
G-VBSM
DD Methods
Figure S:
Top-1 accuracy various synthetic datasets via the SOTA dataset distillation methods loss functions on ImageNet-IK when IPC=1.
Ablation study on ImageNet-IK. We also conducted an ablation study to assess the necessity of the teacher label refinement and cosine similarity loss function on the large-scale dataset in Table 17 This evaluation was performed on both optimization based (SRe²L) and non-optimization-one based (RDED) synthetic datasets. It obvious both modules are essential for performance enhancement, consistent with our analysis in Section 4.
Efficacy of Label Refinement. The necessity of incorporating hard labels into the soft labels generated by teacher models arises from the inherent limitations the performance of these teacher models. Specifically, the test accuracies of teacher models are only 61.27%, 49.73%, and 43.6% for CIFAR-100, ConvNet architectures in dataset distillation. Consequently, the accuracy of soft labels is constrained by the suboptimal nature of the teacher models. To mitigate potential inaccuracies in these soft labels, integrate hard labels to enhance reliability.
and ImageNet-IKG respectively, when trained on commonly used
18

---

Published as a conference paper at ICLR 2025
KT.
Soft CE
MSE+CE
(Our(
45
46.7
(10.8)
47.2
47.5
(11.)
(10.8)
47.8
44.1
44.1
Accuracy(
30-
(10.6)
34.5 34.2 34.0
34,8
20-
SRc²L
DATM
RDED
G-VBSM
DD Methods
Figure 6: Top-1 accuracy on various synthetic datasets via the SOTA dataset distillation methods across loss functions on Inn-ImageNet when IPC=50.
Soft CE
MSE+CE
GIFT(Ourr)
45
(11.0)
11)8)
(12.7)
38.1
38.5
37.5
39.5
38.5
37.3
39.1
Accuracy(" (%)
35.4
35.9
35
31.7
30.5
30-
25
20
16.7
10
S
RDED DD Methods
G-VBSM
Figure 7: Top-1 accuracy various synthetic datasets via the SOTA dataset distillation methods across loss functions ImageNet-1 K when IPC=50.
SR.2L
SR.PL
SR.PL
58
Acuuracy 1%6)
58
(%))
40
38
38
38
0.1
6.7
89
0.1
0.5
6.1
8.7
89
CIFAR- 100
(b) ImageNet
ImageNet-1K
Figure 8: Top-1 accuracy for the SOTA dataset distillation methods on various synthetic datasets when IPC =10 on ConvNet with different
To verify the efficacy of the label refinement, we record the label accuracy before and after refinement across each training epoch for three datasets. The parameter 7, controlling the integration ratio, is set
19

---

Published as a conference paper at ICLR 2025
SR
G-VBSM
SRe
G-VBSM
SR
C-VBSM
Acceracy (%)
Accuracy (%6
(%4
10
10-
0.1
0.3
0.5
0.7
89
9.3
0.3
0.5
0.7
0.
9.1
0.3
0.5
0.7
89
V
(a) CIFAR- -00
(b) Tiny- ImageNet
(c) ImageNet- IK
Figure 9: Top-1 accuracy for the SOTA dataset distillation methods on various synthetic datasets with different data augmentation techniques when IPC =10 on ResNet-18 18 with different 7.
wel kodao
Spech
Spech
Spech
(a) C1F
(b) Ting-ImageNet
ImageNet-11K
Figure 10: Top-1 accuracy for the SOTA dataset distillation methods on various synthetic datasets when PCC =10 on R8sNet-18.
Table 17: Ablation study label refinement (Refine) cosine similarity loss function (Loss) on ImageNet- IK. In the table, (1) means the improvements over these methods.
ImogeNet-1K
ConwNet
ResNes. 18
Method
10
30
100
10
50
100
SR
12.5 03
35.4 = 1.0
40.1 +: 0.4
31.5
49.5 5:
54.3 0.2
L+ Reline
13.3 0. 0.8)
36.3 :: 0.4 (f 0.9)
40.5 1: 0.4 040
318 03
49.7
03 02)
51.5 1: 0.2)
SRe Loss
13.1 0.6)
3688 : (1 (.2)
40.7 (3 06)
02.7122(002)
49.8 0.2 0.3)
54.5 0.2 (1
SRe Reine + Loss
14.2 0.210.((11
38.1 0.4 2.7)
41.5 02 1.4
0.1
50.1 (2
51.8 1 (.
RDED
20.1 04
38.5 1: 0.2
413
41.4004
05
$8.8
RDED Reline
21.2 11
38.7 = 022
4233 (..(
423
03 0.)
033 0.4)
RDED
11. 1.6)
39.3
5..9 02
0.1
RDED Reline
240
39.3 0.1
0.
422001(111)
56.5 00 1.))
59.3 (f..))
G-VBSM
0.6
37.3
411 04
367002
42.3
$7.3
G G-VBSM+ Refine
23.3 1.3100000 (" 0.7)
37.8 = 0.3 0.)
(1 (1
409
37.2
$2.7 0.2 0.4)
51.5
0.2 0.2)
G-VBSM Loss
23 1.2)
38.3
04 LO)
418 1.7
37,5 05 0.)
52.8 022
57 (fO.
G-VRSM
24.102
421103
079
50.102
07.601
to 0.1. As depicted in Figure 10, refining soft labels results in significant performance improvements of 37.1% 1%, 40.95%, and 71.39% for CIFAR-100, ImageNn, and ImageNet-1K, 1K,, respectively, highlighting the critical role of the proposed label refinement.
Pross-Optimizzer on ImageNet- In this experiment, we report the accuracy before and after applying GIFT to baseline methods across multiple optimizers not seen during dataset distillation on as shown in Table 16. The results clearly indicate that GIFT enhances the generalization ability of all baseline methods. This leads to more stable gradients, especially in scenarios with small dataset sizes.
GIFT Enhance Utilization of Hard and Smoothed Labels To evaluate the efficacy of GIFT across various data types, we conduct experiments on both distilled and randomly selected datasets, employing hard and smoothed labels, with IPC =1C=10. (1) For the distilled dataset, we use the state-of- the-art art dataset distillation method, RDED, which uses soft labels generated by teacher models. In the experiments, we maintain the distilled images and replace soft labels with hard and smoothed
20

---

Published as a conference paper at ICLR 2025
labels for model training. (2) For random dataset, randomly select 10 images for each class from the original dataset. The results for two types of data are presented Table 18 and Table 19. is evident that applying method to label utilization consistently improves performance for both the distilled and the randomly selected data.
Table 18: Evaluation of Loss Functions Across Hard and Smoothed Labels on the Distilled Data Generated via RDED with IPC =10.
Label Type
Loss Function
CIFARI00
Imageet
ImageNet-IK
Hard
CE
21.6 + 0.2
13.5 + 0.3
8.3 + 9.8 0.2 (1 1.5)
Ours
22.8 + 0.2 (t 1.2)
14.8 0.3 (f 1.3)
Smoothed
SoftCE
21.9 + 0.2
13.8 + 0.2
8.5 +
KI.
21.7 + 0.3
13.3 + 0.3
8.0 + 0.3 8.8 + 0.3
MSE
22.1 + 0.1
14.1 + 0.1
Ours
23.1 + 0.2 (f 1.0)
15.2 = 0.3 (f L.)
10.2 +: 0.2 (1 1.4)
Table 19: Evaluation of Loss Functions Across Hard and Smoothed Labels on the Randomly Selected Data with IPC=10.
Label Type
Loss Function
CIFARI00
Tiny- ImageNet
ImageNet-IK
Hard
CE
18.9 + 0.3
9.4 0.1
4.0 0.3 5.1 + 0.2 (1 1.1)
Ours
20.3 0.1 (t 1.4)
11.6 0.1 (f 2.2)
Smoothed
SofCCE
19.0 + 0.3
9.8 + 0.3
4.5 + 0.2
KL
17.9 + 0.2
8.5 8.5 0.3
3,9 + 0.1
MSE
19.2 + 0.1
9.7 9.2 0,2
4.6 0.3 5.6 (1 1.0)
Ours
20.4 + 0.2 (1 1.2)
11.8 + (f
KL.
Our GIPT
Training Loos
0.3
0.2
0.1
0.0
Epoch
Figure Comparison of Training Loss Between KL Our GIFT Across Training Epochs
Wi wooc- 47.6)
$443
9i and Olick 47.00)
8811
(a) SGD
(b) Adam
(c) Adam W
Figure 12: Gradient Norms Across Various Optimizers and Hyperparameters.
21

---

Published as a conference paper at ICLR 2025
E EMPERICAL AND THEORETICAL ANALYSIS OF CROSS-OPTIMIZA GENERALIZATION
We begin by analyzing the three optimizers, namely:
SGD (Robbins & Monro, 1951) directly computes gradients from loss values, significantly impacting the update of model parameters.
Adam (Kingma Ba, 2014) includes weight decay in the gradient computation, meaning that #when the primary gradient signal (from the loss) is small, weight decay overshadow
it, leading to ineffective updates toward minimizing the loss.
Adam W (Loshchilov, 2017) separates the concerns optimization and regularization. By applying weight decay independently, it ensures that the optimization process remains focused on minimizing the loss function, while regularization acts as a controlled adjustment to the parameter magnitudes.
Optimizers inherently exhibit diverse characteristics, resulting in distinct training dynamics when applied distilled datasets generated by various methods. Specifically, these distilled datasets, trained with varying loss functions, exhibit distinct loss values. We reveal that the performance of the optimizers is highly influenced by these loss values, as demonstrated by the subsequent empirical evidence and theoretical analysis.
E.1 EMPERICAL ANALYSIS
When employing the KL divergence loss function, the RDED generally exhibits low loss values, as depicted Figure 11. Notably, our GIFT framework does not achieve small loss values.
To examine the impact of loss values on optimizer performance, we conduct experiments utilizing RDED- generated distilled data. We train models using three distinct optimizers, each with distinct hyperparameter configurations, and compute the gradient norm for each. Specifically, we varied the learning rate for SGD optimizer and modified the weight_deeca) parameter for both the Adam and AdamW optimizers.
The gradient norms of these models, presented Figure 12, demonstrate that when loss values are small, the training dynamics exhibit heightened sensitivity to the optimizer choice. Therefore, the performances different optimizers are highly influenced by the loss values. Notably, our GIFT framework can not obtain small loss values, achieving robust performance across various optimizers.
E.2
THEORETICAL ANALYSIS
E.2.1 STOCHASTIC GRADIENT DESCENT (SGD)
Stochastic Gradient Descent (SGD) is foundational optimization algorithm widely used for training machine learning models, particularly neural networks. SGD iteratively updates model parameters to minimize the loss function by moving in the direction of the negative gradient of the loss with respect to the parameters.
SGD Update Rules. Define the following:
Og: Parameters at time step f..
ge VoL(0.-1): Gradient of the loss function with respect to parameters 0 at time step 1.
ny: Learning rate.
The basic SGD update rule is:
0: = 0 179t
Sensitivity to Loss Values in SGD. The gradient 9t = V ((L(6,11) indicates the direction and
magnitude of change needed minimize the loss function. Larger loss function values typically
22

---

Published as a conference paper at ICLR 2025
result in larger gradients. Therefore, in SGD, both the loss values and the learning rate 1) directly affect the model updates.
E.2.2 ADAM OPTIMIZER
Adam (Adaptive Moment Estimation) is an optimization algorithm that combines the advantages of Iwo extensions of stochastic gradient descent: Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation (RMSProp). Adam maintains per-parameter learning rates adapted based on the first and second moments of the gradients.
Adam Update Rules.
Define the following:
Og: Parameters at time step t.
gr VaL(O Gradient of the loss function L with respect to parameters 0 at time step 1-
my: First moment estimate (exponentially decaying average of past gradients).
Vt: Second moment estimate (exponentially decaying average of past squared gradients).
B1, 32: Decay rates for the first and second moments, respectively.
E: Small constant to prevent division by zero.
ty. Learning rate.
The update rules follows:
me 31me -1)g
Ut 320/-1
+ -B22
m4
ri 1 B1
(Bias-corrected first moment)
0 1 82
(Bias-corrected second moment)
me
0. 0,
6o,- +
Weight Decay in Adam. In the original Adam implementation, weight decay is typically incorpo- rated by adding an L2 regularization term directly to the loss function:
(0 L(
X
10113
The gradient of the modified loss function with respect to f is:
VeL'(0) VoL(0) + A0
Consequently, the gradient used in the Adam update rule is augmented with the weight decay term:
g: VpL(O,-1) A0,-1
Substituting this into the Adam update equations, the parameter update rule becomes:
0, 0,
02 +
Here, m, and i incorporate the additional gradient component 20--1-
E.2.3 ADAMW OPTIMIZER
AdamW is modification of the Adam optimizer that decouples weight decay from the gradient- based update. AdamW addresses the intertwined nature weight decay and gradient updates in the original Adam, leading improved generalization performance and more stable training dynamics.
23

---

Published as a conference paper at ICLR 2025
Adam W Update Rules. The primary distinction in AdamW lies in how weight decay is applied. Instead of incorporating weight decay into the gradient computation, AdamW applies directly to the parameters after the standard Adam update. The update rule is expressed as:
my
0. 0z-1
1720,-1
V0 +
Alternatvell, this can be broken down into two sequential steps:
me
0, = 0,-1 "0, +e
8. = 8 - -1200-1
In this formulation:
The first step performs the standard Adam gradient-basee update. The second step applies
weight decay independently of the gradient computation.
Sensitivity to Loss Values in AdamW. By decoupling weight decay from the gradient- based update, AdamW mitigates the issue of weight decay dominating parameter updates when the loss LO) small. In AdamW, the gradient-based update remains primarily responsible for minimizing the loss, while weight decay independently enforces regularization. This separation ensures that even when (G) is minimal, the optimizer can continue adjust parameters based on the loss gradient without being overly constrained by the weight decay term.
When 1(0 is small, VoI(Ot-1) 0. Adam W, the update rule is:
th 11A0,-1
0, 0,-1 1)
Vūz
+
The gradient-based update -" Weett remains tied to the loss gradient, allowing continued optimiza- tion of L(#). Simultaneously, the weight decay term -7pA0,- independently controls the magnitude of 0 without influencing the direction dictated by the loss gradient. This ensures that weight decay does not overshadow the gradient-bar bassd updates, enabling effective model training even when the loss is minimal.
E.2.4 WIIY EXCESSIVE WEIGHT DECAY IN ADAM IMPEDES UPDATES WHEN LOSs IS SMALL?
Adam's Update Mechanism Under Small Loss. decay integrated into the gradient:
Consider the Adam update rule with weight
m 02-1 = n Vo, +
where 9: = VVL((,)) + X0, Assume that the loss (())) is sufficiently small, implying VoL(O)) = 0. Thus 9r A0, Therefore, the update rule is expressed as:
X0,
Rs (1
nTA
041 0g
7)
Vot
/uz +
Here, the parameter 0, is scaled by factor less than (assuming n/ (vT + 0), leading to reduction in 8, If large, the scaling factor can significantly less than 1, causing Úg diminish rapidly. This aggressive shrinking overshadows the limited gradient from the loss function, effectively halting meaningful updates aimed at minimizing T.))..
Adam W's Update Mechanism Under Small Loss. In AdamW, the update rule is:
m
0zt1 02
n2,,,
Even when T.(O.) small, the gradient-based update term -1 Vo.te remains focused on minimizing the loss, while the weight decay term -11A0, independently enforces regularization.
This separation ensures that:
24

---

Published as a conference paper at ICLR 2025
The optimization process remains primarily influenced by the loss gradient.
Weight decay controls the magnitude of the parameters without dictating the direction of updates.
Thus, AdamW allows the model to continue optimizing L(O:) effectively, even when the loss is already minimal, while maintaining controlled regularization through weight decay.
F PYTORCH IMPLEMENTATION CODE
# num_class number of classes
# output tensor of model outputs
# loft_label. : Tensor, shape=(bsz,
# hard_label : Tensor, (bsz, 1]
# alpha smoothing parameter for label smoothing
def CLFT(numcclass output soft_label, hard_ alpha):
# apply label smoothing to hard label smooth_label = label_smootthn Label_smoothingghhrd_laaass label,, num_class, alpha)
# refine soft label
soft_label = F. normalizeessftt_lbeee dim=1)
smooth_ label (smoroth dim=1)
refined_soft_lane
weight mmothh_abeel (1 weight) soft_ label
# calculate the coisen similariy loss F. f.ooiin_simmlarriyyyooooe refiind__sftt_abells dim=1)
return loss
Figure 13: Pytorch Implementation Code
25


Reviews:
[{"review_id": "1dLmbdivhx", "reviewer": "Reviewer_vaeM", "summary": "This paper conducts a comprehensive comparison of various loss functions for soft label utilization in DD and find that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. Building on these insights, the authors introduce GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to use full label information. Extensive experiments demonstrate the effectiveness of GIFT.", "questions": "- Can you explain theoretically why different optimizers have such a big impact on DD?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is interesting to find that models trained on synthetic datasets show high sensitivity to the choice of loss function for soft label utilization.\n- The authors reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers.\n- They propose GIFT and prove its effectiveness through experiments.\n- The experiment is well designed.", "weaknesses": "- In Table 1, the results of ResNet-18 on CIFAR-100 are not very significant, which may not reflect the effect of the proposed algorithm.\n- The current synthetic datasets on Tiny-ImageNet and ImageNet-1K are only generated using Conv-4. Can we experiment with structures such as transformer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TxZ7cqmJxH", "reviewer": "Reviewer_FuA8", "summary": "This paper introduces a novel method for dataset distillation that fully leverages soft labels generated by pre-trained teacher models. The authors highlight the critical role of selecting the right loss function, showing that the model's performance on synthetic datasets is highly dependent on this choice. They propose **GIFT**, a simple yet effective method that refines soft labels and uses a cosine similarity-based loss function to boost performance across various tasks. The approach also enhances generalization across optimizers and architectures. Extensive experiments demonstrate that **GIFT** significantly improves performance without adding computational costs, making it scalable even for large datasets like ImageNet-1K.", "questions": "- Could you share the results of your label utilization method on a randomly selected data subset?  \n- How does GIFT perform with higher IPC values, such as 100 or 200?  \n- Is it possible to achieve lossless dataset distillation using GIFT on any dataset?  \n- It would be interesting to see how the Jensen-Shannon divergence loss function performs within the GIFT framework.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "[S1] The paper dives into a crucial part of dataset distillation, specifically how to choose effective loss functions for label utilization in DD frameworks.  \n[S2] The writing is clear and easy to follow.  \n[S3] The use of cosine similarity is backed by solid theoretical reasoning.  \n[S4] The ablation studies are detailed, and the method shows promising results in continual learning.  \n[S5] GIFT consistently improves state-of-the-art dataset distillation methods across various datasets without adding any extra computational cost.", "weaknesses": "[W1] From a technical standpoint, the contribution of the paper feels somewhat limited. Both the hard-label refinement approach and the cosine-similarity loss function have already been explored in many knowledge distillation studies. Additionally, the paper lacks convincing evidence or justification that these techniques are specifically suited for the synthetic datasets used in dataset distillation.\n\n[W2] Regarding the experimental results, the improvements brought by the proposed label utilization algorithm to existing DD methods are relatively modest. However, since the approach incurs no additional computational cost and offers general performance boosts, the results are still reasonable and acceptable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oMzwvaRQn1", "reviewer": "Reviewer_izfT", "summary": "This work focuses on the problem of soft label utilization in dataset distillation. Currently, multiple different loss functions are employed to use the soft labels generated by pre-trained models. The choice of loss function can have large influence on the training effects for different distilled datasets. Therefore, the authors propose to refine the generated soft labels, and adopt a unified cosine similarity-based loss function to leverage the soft label information. The experiments suggest that the proposed method uniformly improve the validation performance on multipld benchmarks.", "questions": "1. How would cosine similarity perform if it is applied directly on hard labels / smoothed labels? Will it still outperform the other compared loss functions?\n2. The performance improvement is generally higher on small datasets / poor architectures compared with the other end. Can the authors explain the potential reason?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and the designed model has correspondence to the motivation. \n2. The experiments results show uniformly performance improvement over multiple distilled datasets and multiple benchmarks.\n3. It is valuable to look into the validation for dataset distillation.", "weaknesses": "1. The authors claim that GIFT can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns an incorrect label of the highest value. However, there is no qualitative or quantitative result supporting it. Since the $\\gamma$ parameter is only set as 0.1, it remains unclear how it can correct potential soft label error. Can the authors summarize how much the soft labels change after the refinement?\n2. The authors introduce a new evaluation scenario named cross-optimizer generalization. But it is not defined until the end of the introduction section. It is recommended to raise the issue of poor cross-optimizer generalitzation earlier. Does it serve as one of the motivations, or is it simply empirical results after the method is designed?\n3. The authors show in Table 6 the cross-architecture generalization results on Tiny-ImageNet. In many cases the performance gain of large models is on par or lower than that on small models. Is the improvement benefited simply from better training effects of cosine similarity or better generalization?\n4. Although the proposed method uniformly improves the validation results with soft labels on different distilled datasets, the soft label requires enormous storage space. While it is feasible to use currently available pre-trained models for popular dataset distillation benchmarks, it is not practical for actual use on new application scenarios. Thus, the overall practicality slightly degrades the value of this work for the dataset distillation society.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGKSROtlgd", "reviewer": "Reviewer_twQ2", "summary": "This paper proposes a simple but effective plug-and-play dataset distillation method to leverage both hard and soft labels at near-zero cost. \nIt identifies that existing dataset distillation methods are highly sensitive to different loss functions, highlighting the need for a universal loss function. Based on this insight, GIFT introduces a label refinement process that incorporates hard label smoothing, thereby correcting any erroneous signals from the teacher model and enhancing inter-class separation. Additionally, the authors use a cosine similarity-based loss function to fully exploit label information during training.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Unlike previous methods, this paper shifts focus from the distillation process itself to the model training phase using synthetic data, presenting a novel perspective for dataset distillation. The paper proposes a universal loss function that enhances the performance of most dataset distillation methods. \n2. Currently, most state-of-the-art methods incorporate soft labels into distillation, complicating unified evaluation when training models with synthetic data. The approach in this paper could potentially help establish a standardized evaluation benchmark for dataset distillation products.\n3. Extensive experimental results demonstrate that GIFT significantly improves state-of-the-art dataset distillation methods on large-scale datasets, such as ImageNet-1K, while also enhancing cross-optimizer and cross-architecture generalization. GIFT’s plug-and-play nature makes it versatile across various datasets and network architectures.\n4. The presentation of the paper is great. It provides a detailed discussion and clearly showcases the motivation and methodology.", "weaknesses": "1. Although the method offers a novel perspective, its lack of improvement in the distillation process itself somewhat limits the contribution of the paper. \n2. I suggest the authors consider incorporating GIFT into [1], i.e., replacing the loss with GIFT's loss in [1], or adding a loss term in other trajectory-matching methods (as long as they can access a well-trained model during training), I am interested to see if this combination could further improve the effectiveness of dataset distillation.\n\n\n[1] Can pre-trained models assist in dataset distillation? arXiv:2310.03295", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)