GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost

## Abstract

Abstract Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models.
In this paper, we introduce a novel perspective by emphasizing the full utilization of labels.
We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization.
This finding highlights the necessity of a universal loss function for training models on synthetic datasets.
Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT , which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. Extensive experiments demonstrate that GIFT consistently enhances the state-of-the-art dataset distillation methods across various scales datasets without incurring additional computational costs.
For instance, on ImageNet-1K with IPC = 10, GIFT improves the SOTA method RDED by 3.9% and 1.8% on ConvNet and ResNet-18, respectively. Code: https://github.com/LINs-lab/GIFT . 1 1 1 We have provided Pytorch implementation code in Appendix E

## 1 Introduction

Dataset distillation (DD) [] has demonstrated its potential to significantly reduce data size while maintaining comparable model performance [, , , , , ].
Most existing DD methods focus on optimizing images [, , , , , , , ], but recent studies [, , , ] have highlighted the substantial benefits of soft labels. These studies utilize labels assigned by pre-trained models (also called teacher models), yielding siginificant enhancement in stability during the synthesizing process and considerable performance. The success can be attributed to their rich intra-class information, which is particularly valuable for training on extremely small datasets [].

To fully explore the utilization of soft labels in current state-of-the-art dataset distillation methods, we conduct a comprehensive comparison of loss functions for soft label utilization in synthesic datasets via the SOTA dataset distillation methods on Tiny-ImageNet and ImageNet-1K (^2^22The synthetic dataset of DATM for ImageNet-1k is not provided, so comparing with DATM is not possible on this dataset)on $\texttt{IPC}\in\{1,10,50\}$ (^3^33Detailed experiments are elaborated in Section 3. The results of Tiny-ImageNet and ImageNet-1K when IPC = 1 and IPC = 50 are shown in Appendix D).
As illustrated in Figure 1, different dataset distillation methods employ distinct loss functions, and performance varies significantly across these loss functions.
In particular, simply replacing KL divergence loss [] with Soft cross-entropy loss [] for SRe^2L results in a significant performance drop of 13.3% on Tiny-ImageNet.
This substantial gap indicates that models trained on synthetic datasets are highly sensitive to the choice of loss function.
*Hence, it is crucial to propose a universal and effective loss function that is robust across various synthetic datasets.*

Moreover, one challenge of soft labels is that they are inherently suboptimal, as the performance of the teacher model itself may be limited.
For instance, a teacher model trained on ImageNet-1k on ConvNet has only 43.6% accuracy, a common model used in [, ] to assign labels.
Furthermore, in practical scenarios, it is common for the teacher model to have a smaller architecture than the student model due to cross-architecture challenge [].
This discrepancy further limits the performance of the student model when only relying on an inferior teacher model.
By contrast, hard labels are accurate and provide reliable supervision information.
However, directly utilizing hard labels through cross-entropy loss is not effective, as demonstrated in previous studies [] and our experiments in Section 5.6.
*Therefore, it is imperative to effectively integrate hard label information to mitigate the limitations associated with soft labels.*

Based on the above two findings and considerations, we propose an extremely simple yet effective plug-and-play approach called $\underline{\textnormal{G}}$aining $\underline{\textnormal{I}}$mprovement from $\underline{\textnormal{F}}$ull Labels at Near-zero Cos$\underline{\textnormal{T}}$ (GIFT) to effectively utilize both hard and soft labels.
GIFT includes two key components: label refinement and a cosine similarity-based loss function.
It first refines soft labels by incorporating an additional smoothing label obtained through hard label smoothing [].
This simple module offers two significant advantages: firstly, it can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns an incorrect label of the highest value;secondly, soft labels mainly contain intra-class information, which can hinder class separation to some extent [].
Therefore, a relatively sharp label can enhance the dispersion between classes, thereby improving generalization ability, as demonstrated in Section 5.6.
After obtaining the refined labels, we find that they are uniformly distributed in the target space and are approximately orthogonal to each other, as demonstrated in Section 4.
Therefore, we verify theoretically that simply using cosine similarity as the loss function achieves optimal performance.
As shown in Figure 1 (red bars), our method GIFT consistently and significantly enhances the state-of-the-art dataset distillation methods across various scale datasets.

In summary, our contributions are fourfold:

- 1.
To the best of our knowledge, this paper is the first to provide a comprehensive comparison of loss functions for label utilization in dataset distillation.
- 2.
Our study reveals the intriguing fact that models trained on synthetic datasets are highly sensitive to the choice of loss function.
- 3.
We propose GIFT, a simple and universal label utilization algorithm for dataset distillation.
GIFT is built on top of the off-the-shelf dataset distillation methods and requires no extra information, thus raising no additional cost.
- 4.
Experiments verify that GIFT significantly improves performance over the state-of-the-art dataset distillation methods across varying scales and resolutions.
It also enhances cross-architecture and cross-optimizer generalization.

Figure: (a) Tiny-ImageNet
Refer to caption: x1.png

## 2 Related Work

In the field of dataset distillation, the majority of existing methods fix the labels as a one-hot format [, , , , , , , ], with a primary focus on optimizing synthetic images. Recent research highlights the significant benefits of utilizing soft labels to enhance model performance [, , ]. These benefits are particularly pronounced when training models on extremely small synthetic datasets, due to the enriched intra-class information soft labels provide. Methods for obtaining soft labels can be broadly classified into two categories.

#### Optimization-based Soft Labels.

The first type involves learning labels, with several studies [, , , , , ] finding that making labels learnable can significantly improve performance and efficiency. Recent work [] further highlights that optimizing labels can enhance training stability and improve performance.

#### Teacher model-based soft labels.

The subsequent trajectory of works seeks to directly obtain soft labels.
Inspired by knowledge distillation [], TESLA []introduces a soft label assignment strategy, directly generating soft labels by leveraging pre-trained teacher models trained on real datasets.
These soft labels provide rich intra-class information, thereby improving distillation performance.
Following this trend, the state-of-art methods [, , ] also utilize soft labels predicted by teacher models, achieving significant improvements.

## 3 Motivation

### 3.1 Preliminary

Given a large dataset $\mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N}$, where $\mathbf{x}_{i}\in\mathbb{R}^{d}$ represents the input sample and $y_{i}\in\{1,\ldots,C\}$ denotes the corresponding hard label, the objective of dataset distillation is to generate a synthetic dataset $\mathcal{S}=\{(\tilde{\mathbf{x}}_{j},\tilde{y}_{j})\}_{j=1}^{M}$, such that a model trained on $\mathcal{S}$ performs comparably to one trained on $\mathcal{D}$.
In this paper, we explore how to fully utilize both soft labels and hard labels in given datasets.
Therefore, we re-define the synthetic dataset $\mathcal{S}$ as $\mathcal{S}=\{(\tilde{\mathbf{x}}_{j},y_{j},\tilde{y}_{j})\}_{j=1}^{M}$, where $\tilde{\mathbf{x}}_{j}$ denotes the synthetic images, and $y_{j}$ and $\tilde{y}_{j}$ denote the corresponding hard labels and soft labels, respectively.

### 3.2 Are Loss Functions Pulling the Strings in Synthetic Dataset Performance?

Why do we need to answer this question?
Labels in dataset distillation are commonly and typically utilized through a variety of established loss functions, such as cross-entropy (CE) [], Kullback-Leibler (KL) divergence [], soft cross-entropy [] or mean squared error (MSE) [].
Specifically, SRe^2L [] and RDED [] employ KL divergence, DATM [] uses soft cross-entropy, and G-VBSM [] simultaneously utilizes MSE and CE.
However, different distillation methods employ varying loss functions to train models on synthetic datasets, yet there is a notable lack of comprehensive comparison among these loss functions. Hence, we investigate the performance of four state-of-the-art dataset distillation methods under three commonly used loss functions.

Experiment settings.
Our experiments span small-scale dataset Tiny-ImageNet and large-scale dataset ImageNet-1K on $\texttt{IPC}\in\{1,10,50\}$ using ConvNet [].
Note that the synthetic dataset for DATM on ImageNet-1K is unavailable, precluding comparisons on this dataset.
We evaluate at $\texttt{IPC}\in\{1,10,50\}$. Additional visualizations for Tiny-ImageNet and ImageNet-1K at $\texttt{IPC}=1$ and $IPC=50$ are provided in Appendix D.
Notably, our empirical study is conducted from the end-user perspective: we treat the distillation of the synthetic dataset as a black box and apply different loss functions during the evaluation of the synthetic datasets.

Analysis.
The results, visualized in Figure 1, reveal that even for the same synthetic dataset, the evaluation model performance varies significantly under different loss functions.
For example, the performance of SRe^2L decreases by 13.2% on Tiny-ImageNet when switching from KL loss to soft CE loss but improves by 0.7% when using MSE+CE loss.
These findings illustrate that *the performance of models trained on synthetic datasets is highly sensitive to the choice of the loss function*, highlighting the necessity for a unified and effective loss function in dataset distillation.

## 4 Method

Motivated by the aforementioned findings and considerations, we propose an extremely simple but surprisingly effective plug-and-play approach called $\underline{\textnormal{G}}$aining $\underline{\textnormal{I}}$mprovement from $\underline{\textnormal{F}}$ull Labels at Near-zero Cos$\underline{\textnormal{T}}$ (GIFT) to effectively utilize both hard and soft labels.

GIFT includes two key modules, namely, label refinement and a cosine similarity-based loss function.
The former aims to refine soft labels by incorporating an additional smoothing label derived from hard label smoothing [], while the latter is theoretically validated to achieve optimal performance simply using cosine similarity as the loss function.
A PyTorch implementation of our method is provided in Appendix E.

#### Label Refinement.

As discussed in Section 1, soft labels generated by teacher models are inherently suboptimal due to two primary reasons.
Firstly, the performance of the teacher model is limited, particularly on complex datasets such as ImageNet-1K. Secondly, soft labels predominantly provide intra-class information, thereby limiting class dispersion.

An intuitive method to address these shortcomings is to integrate with hard labels.
On the one hand, hard labels offer accurate and reliable supervision, which can rectify erroneous information provided by soft labels.
On the other hand, they can assist in inter-class dispersion.
Therefore, we refine soft labels by weighing them with smoothed hard labels, thereby solving the two limitations of soft labels.

The refined soft label is defined as $\tilde{y}_{j}\leftarrow\gamma\cdot\frac{y_{i}}{\|y_{j}\|}+(1-\gamma)\cdot\frac
{\tilde{y}_{j}}{\|\tilde{y}_{j}\|}$, where $j$ is the j-th synthetic images, $y_{j}$ is the smoothed label obtained via label smoothing technique for hard label, and $\tilde{y}_{j}$ is the soft label.
In our experiments, $\gamma=0.1$ is validated to be optimal through experiments depicted in Figure 3 and Figure 8 in Appendix D.

#### Mutual information bounded loss function.

Prior study [] has posited that representation learning from any samples $X$ to targets $Y$ is based on maximizing their mutual information.
They propose to distill the dataset by maximizing $I_{\mathcal{V}}(X,Y)$, where $I_{\mathcal{V}}$ denotes the $\mathcal{V}$-information [], which has demonstrated superior performance.
However, we observe that most prior studies, including [], have not investigated training models using $I_{\mathcal{V}}(X,Y)$.
To address this gap, we explore the application of $I_{\mathcal{V}}(X,Y)$ by deriving an upper bound for the $\mathcal{V}$-information, as presented in Theorem 1 (see Appendix A for the proof).

###### Theorem 1

The $\mathcal{V}$-information $I_{\mathcal{V}}(X,Y)$ is upper bounded by a function involving the cosine similarity between the positive pair $(\mathbf{x}_{i},y_{i})$, the expected cosine similarity between the anchor $\mathbf{x}_{i}$ and negative samples $y_{j}$, and the number of negative samples $K$. Specifically,

$$ $\displaystyle\mathcal{L}_{\textup{InfoNCE}}$ $\displaystyle=-\mathbb{E}\left[\log\frac{\exp(f(\boldsymbol{\phi}_{\boldsymbol {\theta}}(\mathbf{x}),y))}{\sum_{y^{\prime}\in\mathcal{Y}}\exp(f(\boldsymbol{ \phi}_{\boldsymbol{\theta}}(\mathbf{x}),y^{\prime}))}\right]$ (1) $\displaystyle\leq-\frac{1}{\tau}\left(\mathbb{E}\left[\left(\frac{\boldsymbol{ \phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\cdot y_{i}}{\|\boldsymbol{\phi}_{ \boldsymbol{\theta}}(\mathbf{x}_{i})\|\|y_{i}\|}\right)\right]-\frac{\mathbb{E }[\boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\cdot y_{j}]}{\| \boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x})_{i}\|\mathbb{E}[\|y_{j}\|] }\right)+\log(K)\,,$ $$

where $\tau$ denotes the temperature parameter, $\mathcal{L}_{\textup{InfoNCE}}$ [] serves as a proxy for $I_{\mathcal{V}}(X,Y)$ [], and $f$ represents the similarity function.

Moreover, the targets $Y$ in the distilled dataset are pre-generated using pre-trained models [, ]. These targets can be considered high-dimensional vectors that are approximately orthogonal to each other.
Consequently, the term ${\mathbb{E}}[\boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\cdot y_{j}]$ approaches zero as $\mathbb{E}\left[\frac{\boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})
\cdot y_{i}}{\|\boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\|\|y_{i
}\|}\right]$ approaches one, allowing (1) to be simplified as:

$$ $\mathcal{L}_{\text{InfoNCE}}\leq-\frac{1}{\tau}\left(\mathbb{E}\left[\frac{ \boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\cdot y_{i}}{\| \boldsymbol{\phi}_{\boldsymbol{\theta}}(\mathbf{x}_{i})\|\|y_{i}\|}\right]- \epsilon\right)+\log(N)\,,$ (2) $$

where $\epsilon$ is a small positive term approaching zero.

To minimize this upper bound (2), our proposed loss function for training on distilled data $\mathcal{S}$ can defined as:

$$ $\mathcal{L}=\mathbb{E}_{(\tilde{\mathbf{x}}_{i},\tilde{y}_{i})\sim\mathcal{S}} \left[1-\frac{\boldsymbol{\phi}_{\boldsymbol{\theta}}(\tilde{\mathbf{x}}_{i}) \cdot\tilde{y}_{i}}{\|\boldsymbol{\phi}_{\boldsymbol{\theta}}(\tilde{\mathbf{x }}_{i})\|\|\tilde{y}_{i}\|}\right]\,,$ (3) $$

where $\tilde{y}_{j}$ is the refined soft label discussed above and we replace the original training loss with (3).

###### Theorem 1

## 5 Experiments

In this section, we evaluate the superiority of our proposed GIFT across various datasets and neural architectures.
First, we demonstrate the superior improvements of GIFT over the state-of-the-art dataset distillation in Section 5.2 and knowledge distillation methods in Section 5.3. Next, we verify that our proposed GIFT significantly enhances cross-architecture and cross-optimizer generalization in Section 5.4, and improves continual learning in Section 5.5.
Furthermore, we conduct extensive ablation experiments to investigate the effect of each component of our method in Section 5.6.
More details about experiments details and results can be found in Appendix C and Appendix D.

### 5.1 Experiment Setup

#### Datasets and Networks.

We conduct experiments on both large-scale and small-scale datasets, including the full $224\times 224$ ImageNet-1k [], Tiny-ImageNet [] and CIFAR-100 [].
Following previous dataset distillation studies [, , , , ], we employ ConvNet [] and ResNet-18 [] as our backbone architectures across all datasets.
Specifically, Conv-3 is employed for CIFAR-10/100, while Conv-4 is used for Tiny-ImageNet and ImageNet-1K.
For cross-architecture experiments, we additionally utilize large-scale networks, including ResNet-101 [] and Swin-V2-Tiny [] and small-scale networks, such as EfficientNet-B0 [], and MobileNet-V2 [] to verify the generalizability of our approach.

#### Baselines.

We benchmark our method GIFT against state-of-the-art dataset distillation methods that leverage soft labels and scale to large high-resolution datasets, including SRe^2L [], RDED [], DATM [], and G-VBSM [].
We do not compare with TESLA [] due to the unavailability of its synthetic dataset and its inability to surpass methods like RDED []. Additionally, DATM only provides synthetic datasets on ConvNet and does not provide the ImageNet-1k synthetic dataset.

Knowledge distillation [] is a straightforward approach that utilizes both hard and soft label information. Thus, we also compare our method GIFT with state-of-the-art knowledge distillation methods, including KD [], WSLD [], DKD [], and NKD []. Further details on these methods are provided in Appendix C.

#### Implementation details of GIFT .

Our method does not involve any distilling datasets process. We obtain all synthetic datasets directly from the source data provided by the authors (^4^44 $\star$ SRe^2L: [https://github.com/VILA-Lab/SRe2L](https://github.com/VILA-Lab/SRe2L) $\star$ RDED: [https://github.com/LINs-lab/RDED](https://github.com/LINs-lab/RDED) $\star$ DATM: [https://gzyaftermath.github.io/DATM/](https://gzyaftermath.github.io/DATM/) $\star$ G-VBSM: [https://github.com/shaoshitong/G_VBSM_Dataset_Condensation](https://github.com/shaoshitong/G_VBSM_Dataset_Condensation))and simply replace the loss function. Thus, our method is a plug-and-play approach that can be easily integrated into existing dataset distillation pipelines without necessitating additional dataset synthesis or modification. Ahe experiments are conducted using an NVIDIA RTX 4090 GPU.

#### Hyperparameter Settings.

We provide detailed hyperparameter configurations for our distilled dataset evaluation in Appendix C. Following recent works [, ], the evaluation on all datasets uses the parameters outlined in Table 11. We set the coefficient of label smoothing $\alpha=0.1$ and the weight hyper-parameter $\gamma=0.1$ for all methods across various synthetic datasets, as $\gamma=0.1$ is validated to be optimal through experiments depicted in Section 5.6

**Table 1: Comparison with the state-of-the-art methods of dataset distillation on CIFAR-100 and Tiny-ImageNet.In the table, “-” are absent due to scalability problems.**
|  |  | CIFAR100 | Tiny-ImageNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Network | Method | 1 | 10 | 50 | 1 | 10 | 50 |
|  | SRe^2L | 13.6 $\pm$ 0.4 | 33.7 $\pm$ 0.5 | 52.3 $\pm$ 0.2 | 12.1 $\pm$ 0.4 | 34.5 $\pm$ 0.4 | 46.3 $\pm$ 0.1 |
|  | SRe^2L + Ours | 15.1 $\pm$ 0.3 ($\uparrow$ 1.5) | 38.0 $\pm$ 0.5 ($\uparrow$ 4.3) | 55.4 $\pm$ 0.1 ($\uparrow$ 3.1) | 13.1 $\pm$ 0.2 ($\uparrow$ 1.0) | 37.5 $\pm$ 0.3 ($\uparrow$ 3.0) | 47.1 $\pm$ 0.1 ($\uparrow$ 0.8) |
|  | RDED | 22.1 $\pm$ 0.3 | 47.5 $\pm$ 0.3 | 55.7 $\pm$ 0.4 | 17.9 $\pm$ 0.3 | 41.4 $\pm$ 0.3 | 47.2 $\pm$ 0.1 |
|  | RDED + Ours | 24.7 $\pm$ 0.3 ($\uparrow$ 2.5) | 50.6 $\pm$ 0.3 ($\uparrow$ 2.5) | 57.9 $\pm$ 0.2 ($\uparrow$ 2.2) | 19.1 $\pm$ 0.3 ($\uparrow$ 1.2) | 44.0 $\pm$ 0.2 ($\uparrow$ 2.6) | 48.3 $\pm$ 0.1 ($\uparrow$ 1.1) |
|  | DATM | - | 36.1 $\pm$ 0.2 | 43.0 $\pm$ 0.2 | - | 26.5 $\pm$ 0.2 | 34.2 $\pm$ 0.5 |
|  | DATM + Ours | - | 37.8 $\pm$ 0.3 ($\uparrow$ 1.7) | 43.6 $\pm$ 0.3 ($\uparrow$ 0.6) | - | 27.5 $\pm$ 0.2 ($\uparrow$ 1.0) | 34.8 $\pm$ 0.6 ($\uparrow$ 0.6) |
|  | G-VBSM | 14.7 $\pm$ 0.5 | 40.9 $\pm$ 0.4 | 54.7 $\pm$ 0.3 | 8.4 $\pm$ 0.4 | 34.5 $\pm$ 0.5 | 47.0 $\pm$ 0.3 |
| ConvNet | G-VBSM + Ours | 16.0 $\pm$ 0.2 ($\uparrow$ 1.3) | 44.6 $\pm$ 0.2 ($\uparrow$ 3.7) | 57.2 $\pm$ 0.1 ($\uparrow$ 2.5) | 8.9 $\pm$ 0.3 ($\uparrow$ 0.5) | 36.9 $\pm$ 0.7 ($\uparrow$ 2.4) | 47.8 $\pm$ 0.2 ($\uparrow$ 0.8) |
|  | SRe^2L | 11.5 $\pm$ 0.5 | 42.7 $\pm$ 0.2 | 57.8 $\pm$ 0.6 | 12.7 $\pm$ 0.3 | 43.5 $\pm$ 0.1 | 53.9 $\pm$ 0.0 |
|  | SRe^2L + Ours | 12.7 $\pm$ 0.4 ($\uparrow$ 1.2) | 44.3 $\pm$ 0.3 ($\uparrow$ 1.6) | 58.6 $\pm$ 0.3 ($\uparrow$ 0.8) | 14.2 $\pm$ 0.3 ($\uparrow$ 1.5) | 44.2 $\pm$ 0.3 ($\uparrow$ 0.7) | 54.5$\pm$ 0.2 ($\uparrow$ 0.6) |
|  | RDED | 4.7 $\pm$ 0.1 | 52.8 $\pm$ 0.2 | 64.4 $\pm$ 0.1 | 15.1 $\pm$ 0.3 | 48.2 $\pm$ 0.4 | 57.6 $\pm$ 0.3 |
|  | RDED + Ours | 5.0 $\pm$ 0.2 ($\uparrow$ 0.3) | 54.0 $\pm$ 0.3 ($\uparrow$ 1.2) | 65.3 $\pm$ 0.2 ($\uparrow$ 0.7) | 15.9 $\pm$ 0.3 ($\uparrow$ 0.8) | 49.2 $\pm$ 0.1 ($\uparrow$ 1.0) | 58.1 $\pm$ 0.1 ($\uparrow$ 0.5) |
|  | DATM | - | 25.8 $\pm$ 1.0 | 47.5 $\pm$ 0.4 | - | 26.7 $\pm$ 0.2 | 41.9 $\pm$ 0.3 |
|  | DATM + Ours | - | 26.3 $\pm$ 0.4 ($\uparrow$ 0.5) | 47.9 $\pm$ 0.3 ($\uparrow$ 0.4) | - | 29.0 $\pm$ 0.5 ($\uparrow$ 2.3) | 42.4 $\pm$ 0.2 ($\uparrow$ 0.4) |
|  | G-VBSM | 13.4 $\pm$ 0.3 | 48.5 $\pm$ 0.5 | 62.0 $\pm$ 0.2 | 8.8 $\pm$ 0.3 | 39.9 $\pm$ 0.4 | 52.8 $\pm$ 0.2 |
| ResNet-18 | G-VBSM + Ours | 13.7 $\pm$ 0.3 ($\uparrow$ 0.3) | 49.2 $\pm$ 0.2 ($\uparrow$ 0.7) | 62.5 $\pm$ 0.3 ($\uparrow$ 0.5) | 9.3 $\pm$ 0.3 ($\uparrow$ 0.5) | 40.5 $\pm$ 0.2 ($\uparrow$ 0.6) | 53.1 $\pm$ 0.1 ($\uparrow$ 0.3) |

**Table 2: Comparison with the state-of-the-art methods of dataset distillation on ImageNet-1K. In the table, ($\uparrow$) means the improvements over these methods.**
|  |  | ImageNet-1K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ConvNet | ResNet-18 |  |  |  |  |
|  | Method | 10 | 50 | 100 | 10 | 50 | 100 |
|  | SRe^2L | 12.5 $\pm$ 0.3 | 35.4 $\pm$ 1.0 | 40.1 $\pm$ 0.4 | 31.5 $\pm$ 0.3 | 49.5 $\pm$ 0.1 | 54.3 $\pm$ 0.2 |
|  | SRe^2L + Ours | 14.2 $\pm$ 0.6 ($\uparrow$ 1.7) | 38.1 $\pm$ 0.4 ($\uparrow$ 2.7) | 41.5 $\pm$ 0.2 ($\uparrow$ 1.4) | 31.9 $\pm$ 0.2 ($\uparrow$ 0.4) | 50.1 $\pm$ 0.2 ($\uparrow$ 0.6) | 54.8 $\pm$ 0.1 ($\uparrow$ 0.5) |
|  | RDED | 20.1 $\pm$ 0.4 | 38.5 $\pm$ 0.2 | 41.8 $\pm$ 0.2 | 41.4 $\pm$ 0.4 | 55.5 $\pm$ 0.2 | 58.8 $\pm$ 0.1 |
|  | RDED + Ours | 24.0 $\pm$ 0.8 ($\uparrow$ 3.9) | 39.5 $\pm$ 0.1 ($\uparrow$ 1.0) | 42.5$\pm$ 0.1 ($\uparrow$ 0.7) | 43.2 $\pm$ 0.1 ($\uparrow$ 1.8) | 56.5 $\pm$ 0.1 ($\uparrow$ 1.0) | 59.3$\pm$ 0.1 ($\uparrow$ 0.5) |
|  | G-VBSM | 22.6 $\pm$ 0.5 | 37.3 $\pm$ 0.3 | 40.1 $\pm$ 0.4 | 36.7 $\pm$ 0.2 | 52.3 $\pm$ 0.1 | 57.3 $\pm$ 0.1 |
|  | G-VBSM + Ours | 24.3 $\pm$ 0.2 ($\uparrow$ 1.7) | 39.1 $\pm$ 0.3 ($\uparrow$ 1.8) | 42.1 $\pm$ 0.3 ($\uparrow$ 2.0) | 37.9 $\pm$ 0.5 ($\uparrow$ 1.2) | 53.1 $\pm$ 0.2 ($\uparrow$ 0.8) | 57.6 $\pm$ 0.1 ($\uparrow$ 0.3) |

### 5.2 Can GIFT Improve Performance of Dataset Distillation?

#### Small-Scale Dataset Comparison

In Table 1, we present the test accuracy on CIFAR-100 and Tiny-ImageNet datasets before and after applying our GIFT algorithm. Notably, DATM does not provide synthetic datasets when IPC =1, nor does it provide synthetic datasets for ResNet-18. Consequently, we used ConvNet synthetic data to train ResNet-18. It is evident that applying GIFT increases performance for all baseline methods. Specifically, the SRe^2L method exhibits the most significant improvement, with an accuracy gain of up to 4.3% on CIFAR-100 when IPC =10. This is particularly noteworthy as GIFT requires no additional information or cost. The considerable accuracy gains can be achieved simply by replacing the loss function with our proposed approach.

#### Large-Scale Dataset Comparison

In the large-scale full 224 $\times$ 224 ImageNet-1k dataset, as shown in Table 2, our proposed method consistently improves all baseline methods across all IPC values in 10, 50, 100, using both ConvNet and ResNet-18 as evaluation models. Specifically, compared with the current state-of-the-art methods RDED and G-VBSM, GIFT achieves significant performance gains of 1.8%and 1.2% on ResNet-18 when IPC =10, respectively. The substantial performance improvements obtained by GIFT demonstrate its capability to effectively scale to large datasets.

### 5.3 Can Knowledge Distillation work?

A straightforward approach to combining soft labels and hard labels is knowledge distillation [], which transfers knowledge from a teacher model to a student model using hard labels (cross-entropy loss) and soft labels provided by a strong teacher model (KL divergence loss).
In Table 3, we compare our proposed method, GIFT, with the state-of-the-art knowledge distillation techniques across synthetic datasets distilled via RDED [] (^5^55RDED is the current state-of-the-art method as shown in Table 1, so we conduct the experiment on its synthetic datasets.)

It can be observed that GIFT outperforms all knowledge distillation methods.
We attribute the failure of knowledge distillation methods to the extremely small size of distilled datasets, which significantly hampers the performance of knowledge distillation methods, as corroborated by [].
Therefore, knowledge distillation is not well-suited for our problem, further highlighting the effectiveness of GIFT.

**Table 3: Comparison with the knowledge distillation methods on the synthetic dataset via RDED [] using ConvNet. In this table, bold means the best result, underlinedmeans the second best, and ($\uparrow$) denotes improvements over the second best baseline.**
|  | CIFAR100 | Tiny-ImageNet | ImageNet-1K |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | 10 | 50 | 10 | 50 | 10 | 50 | 100 |
| Teacher | 61.27 | 61.27 | 49.73 | 49.73 | 43.6 | 43.6 | 43.6 |
| KD | 43.2 $\pm$ 0.1 | 51.9 $\pm$ 0.4 | 33.3 $\pm$ 0.4 | 40.7 $\pm$ 0.2 | 16.7 $\pm$ 0.1 | 24.3 $\pm$ 0.2 | 27.5 $\pm$ 0.4 |
| WSLD | 38.6 $\pm$ 0.3 | 49.5 $\pm$ 0.5 | 27.1 $\pm$ 0.3 | 37.5 $\pm$ 0.2 | 13.4 $\pm$ 0.1 | 22.8 $\pm$ 0.1 | 26.2 $\pm$ 0.0 |
| DKD | 49.0 $\pm$ 0.2 | 56.7 $\pm$ 0.2 | 40.9 $\pm$ 0.2 | 47.2 $\pm$ 0.1 | 20.5 $\pm$ 0.1 | 33.1 $\pm$ 0.1 | 36.5 $\pm$ 0.1 |
| NKD | 46.3 $\pm$ 0.3 | 54.3 $\pm$ 0.1 | 37.6 $\pm$ 0.4 | 44.1 $\pm$ 0.2 | 19.84 $\pm$ 0.1 | 27.9 $\pm$ 0.2 | 30.6 $\pm$ 0.2 |
| GIFT (ours) | 50.6 $\pm$ 0.3 ($\uparrow$ 1.6) | 57.9 $\pm$ 0.2 ($\uparrow$ 1.2) | 44.0 $\pm$ 0.2($\uparrow$ 3.1) | 48.3 $\pm$ 0.1 ($\uparrow$ 1.1) | 24.0 $\pm$ 0.8($\uparrow$ 3.4) | 39.5 $\pm$ 0.1 ($\uparrow$ 6.4) | 42.5$\pm$ 0.1 ($\uparrow$ 6.0) |

### 5.4 Can GIFT Improve Generalization?

#### Cross-Architecture Generalization

To validate the enhancement of generalization capability by our GIFT, it is necessary to assess its effectiveness across various neural architectures not encountered during the dataset synthesis phase.
We evaluate performance on both small and large-scale model architectures. Table 4presents the performance before and after applying our GIFT to dataset distillation methods. The results indicate that GIFT enhances the cross-architecture generalization of all dataset distillation methods across diverse architectures. Notably, our method shows significant improvements when generalizing from small networks to larger networks. For instance, GIFT yields performance gains of 2.6% and 7.8% for RDED and G-VBSM, respectively, when synthesizing data using ConvNet while training model on ResNet-101.

The success of our method is attributed to its stability. In cross-architecture scenarios, soft labels may not be sufficient due to architectural differences []. However, our cosine similarity approach inherently includes a normalization operation, mitigating the negative impact of label fluctuation. These results are promising, indicating that our method, which does not incur additional computational costs, is well-suited for applications involving large-scale models.

**Table 4: Top-1 accuracy on cross-architecture generalization on Tiny-ImageNet. We use the synthetic datasets distilled (D) on ConvNet and ResNet-18 on Tiny-ImageNet when IPC=10. Then, evaluations (E) are performed across both small-scale and large-scale architectures. ($\uparrow$) denotes improvements over these methods.**
|  |  | Small-Scale Architecture | Large-Scale Architecture |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| D/E | ConvNet | ResNet-18 | EfficientNet-B0 | MobileNet-V2 | ResNet-101 | Swin-V2-Tiny |  |
| ConvNet | SRe^2L | 34.5 $\pm$ 0.4 | 43.04 $\pm$ 0.1 | 11.2 $\pm$ 1.1 | 14.0 $\pm$ 0.6 | 9.9 $\pm$ 0.5 | 10.3 $\pm$ 0.4 |
| SRe^2L + Ours | 37.5$\pm$0.3 ($\uparrow$ 3.0) | 44.2 $\pm$ 0.3 ($\uparrow$ 1.16) | 15.3 $\pm$ 1.0 ($\uparrow$ 4.1) | 14.4 $\pm$ 0.3 ($\uparrow$ 0.4) | 10.6 $\pm$ 0.3 ($\uparrow$ 0.7) | 11.2 $\pm$ 0.2 ($\uparrow$ 0.9) |  |
| RDED | 41.4 $\pm$ 0.3 | 46.5 $\pm$ 0.2 | 30.3 $\pm$ 1.4 | 30.2 $\pm$ 0.2 | 28.2 $\pm$ 1.9 | 26.8 $\pm$ 0.6 |  |
| RDED + Ours | 44.0$\pm$0.2 ($\uparrow$ 2.6) | 47.2 $\pm$ 0.1 ($\uparrow$ 0.7) | 31.4 $\pm$ 0.9 ($\uparrow$ 1.1) | 31.3 $\pm$ 0.2 ($\uparrow$ 1.1) | 30.8 $\pm$ 1.8 ($\uparrow$ 2.6) | 28.7 $\pm$ 0.4 ($\uparrow$ 1.9) |  |
| DATM | 36.1 $\pm$ 0.2 | 27.3 $\pm$ 0.2 | 18.0 $\pm$ 0.3 | 14.7 $\pm$ 0.2 | 15.3 $\pm$ 0.8 | 4.1 $\pm$ 3.1 |  |
| DATM + Ours | 37.8 $\pm$ 0.3 ($\uparrow$ 1.7) | 29.0 $\pm$ 0.2 ($\uparrow$ 1.7) | 18.4$\pm$ 0.5 ($\uparrow$ 0.4) | 17.5 $\pm$ 0.1 ($\uparrow$ 2.8) | 16.8$\pm$ 0.5 ($\uparrow$ 1.5) | 16.3 $\pm$ 0.3 ($\uparrow$ 12.2) |  |
|  | G-VBSM | 34.5 $\pm$ 0.5 | 42.2 $\pm$ 0.3 | 13.7 $\pm$ 1.4 | 15.0 $\pm$ 0.4 | 7.7 $\pm$ 0.6 | 10.4 $\pm$ 0.4 |
|  | G-VBSM + Ours | 36.9$\pm$0.7 ($\uparrow$ 2.4) | 42.8 $\pm$ 0.2 ($\uparrow$ 0.6) | 16.3 $\pm$ 1.0 ($\uparrow$ 2.6) | 15.8$\pm$ 0.4 ($\uparrow$ 0.8) | 15.5 $\pm$ 0.3 ($\uparrow$ 7.8) | 13.2$\pm$ 0.1 ($\uparrow$ 2.8) |
| ResNet-18 | SRe^2L | 19.2 $\pm$ 0.1 | 43.5 $\pm$ 0.1 | 11.6 $\pm$ 0.4 | 11.9 $\pm$ 0.3 | 8.7 $\pm$ 1.0 | 8.0 $\pm$ 0.2 |
| SRe^2L + Ours | 19.4 $\pm$ 0.2 ($\uparrow$ 0.2) | 44.2$\pm$0.3($\uparrow$ 0.7) | 12.2 $\pm$ 0.2 ($\uparrow$ 0.6) | 12.3 $\pm$ 0.1 ($\uparrow$ 0.4) | 9.0 $\pm$ 0.5 ($\uparrow$ 0.3) | 8.8$\pm$ 0.3 ($\uparrow$ 0.8) |  |
| RDED | 29.2 $\pm$ 0.3 | 48.2 $\pm$ 0.4 | 24.1 $\pm$ 0.7 | 23.5 $\pm$ 0.3 | 21.8 $\pm$ 0.3 | 19.6 $\pm$ 0.4 |  |
| RDED + Ours | 29.9 $\pm$ 0.1 ($\uparrow$ 0.7) | 49.2$\pm$0.1 ($\uparrow$ 1.0) | 25.2 $\pm$ 0.2 ($\uparrow$ 1.1) | 24.1 $\pm$ 0.3 ($\uparrow$ 0.6) | 23.5$\pm$ 0.3 ($\uparrow$ 1.7) | 20.4$\pm$ 0.3 ($\uparrow$ 0.8) |  |
| G-VBSM | 16.0 $\pm$ 0.3 | 39.9 $\pm$ 0.4 | 8.8 $\pm$ 0.1 | 11.5 $\pm$ 0.4 | 6.3 $\pm$ 1.1 | 6.5 $\pm$ 0.3 |  |
| G-VBSM + Ours | 16.5 $\pm$ 0.4 ( $\uparrow$ 0.5) | 40.5$\pm$0.2 ($\uparrow$ 0.6) | 10.1$\pm$ 0.2 ($\uparrow$ 1.3) | 11.8 $\pm$ 0.3 ($\uparrow$ 0.3) | 9.2 $\pm$ 0.6 ($\uparrow$ 2.9) | 8.2$\pm$ 0.3 ($\uparrow$ 1.7) |  |

**Table 5: Top-1 accuracy on cross-optimization generalization on Tiny-ImageNet and CIFAR100 when IPC=10.We evaluate the performance of synthetic datasets across various optimizers. ($\uparrow$) denotes improvements over these methods.**
| Dataset | CIFAR100 | Tiny-ImageNet |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Optimizer | SGD | Adam | AdamW | SGD | Adam | AdamW |
| SRe^2L | 1.5 $\pm$ 0.1 | 8.7 $\pm$ 0.1 | 34.5 $\pm$ 0.4 | 0.6 $\pm$ 0.0 | 3.1 $\pm$ 0.1 | 33.7 $\pm$ 0.5 |
| SRe^2L + Ours | 43.0 $\pm$ 0.8 ($\uparrow$ 42.6) | 44.7 $\pm$ 0.6 ($\uparrow$ 37.5) | 38.0 $\pm$ 0.5 ($\uparrow$ 3.5) | 43.8 $\pm$ 0.5 ($\uparrow$ 43.2) | 45.2 $\pm$ 0.2 ($\uparrow$ 42.1) | 37.5$\pm$ 0.3 ($\uparrow$ 3.0) |
| RDED | 1.9 $\pm$ 0.0 | 17.8 $\pm$ 0.1 | 47.5 $\pm$ 0.3 | 0.6 $\pm$ 0.0 | 4.5 $\pm$ 0.2 | 41.4 $\pm$ 0.3 |
| RDED + Ours | 53.4 $\pm$ 0.2 ($\uparrow$ 51.5) | 53.7 $\pm$ 0.4 ($\uparrow$ 35.9) | 50.6 $\pm$ 0.3<br>($\uparrow$ 2.6) | 46.6 $\pm$ 0.3 ($\uparrow$ 46.0) | 46.5 $\pm$ 0.2 ($\uparrow$ 42.0) | 44.0 $\pm$ 0.2 ($\uparrow$ 2.6) |
| DATM | 37.3 $\pm$ 0.3 | 36.7 $\pm$ 0.1 | 36.1 $\pm$ 0.2 | 28.2 $\pm$ 0.1 | 27.8 $\pm$ 0.1 | 26.5 $\pm$ 0.2 |
| DATM + Ours | 38.5$\pm$ 0.2 ($\uparrow$ 1.2) | 40.0$\pm$ 0.2 ($\uparrow$ 3.3) | 37.8 $\pm$ 0.3 ($\uparrow$ 1.7) | 30.1 $\pm$ 0.3 ($\uparrow$ 1.9) | 29.1 $\pm$ 0.1 ($\uparrow$ 1.3) | 27.5 $\pm$ 0.2 ($\uparrow$ 1.0) |
| G-VBSM | 44.2 $\pm$ 1.8 | 41.2 $\pm$ 0.4 | 40.9 $\pm$ 0.4 | 41.4 $\pm$ 0.4 | 36.5 $\pm$ 0.4 | 34.5 $\pm$ 0.5 |
| G-VBSM + Ours | 49.8 $\pm$ 0.4 ($\uparrow$ 5.6) | 51.3 $\pm$ 0.6 ($\uparrow$ 10.1) | 44.6 $\pm$ 0.2 ($\uparrow$ 3.7) | 44.5 $\pm$ 0.1 ($\uparrow$ 3.1) | 45.3 $\pm$ 0.1 ($\uparrow$ 8.8) | 36.9 $\pm$ 0.7 ($\uparrow$ 2.4) |

#### Cross-optimizer Generalization.

Similar to cross-architecture generation, it is crucial to estimate the cross-optimizer generalization of dataset distillation methods. Different optimizers, such as SGD and Adam, exhibit distinct characteristics that influence model performance and generalization.
Therefore, in practical applications, it is necessary to choose the most appropriate optimizer according to the training conditions. In this experiment, we report the accuracy before and after applying our GIFT to baseline methods across multiple optimizers not seen during dataset distillation, as shown in Table 5. The results clearly indicate that GIFT enhances the generalization ability of all baseline methods. More results on ImageNet-1K can be found in Table 12 in Appendix D.

It is notable that SRe2L and RDED perform poorly in this cross-optimizer challenge. However, with our method, performance increased by 42.6% and 51.5% using SGD on CIFAR-100. A major reason for their failure is that the KL divergence loss function they use leads to high gradient variance and instability in extremely small dataset training []. In contrast, AdamW maintains optimization stability across optimizers by mitigating overfitting in small datasets through explicit weight decay, enhancing generalization and preventing excessive parameter growth.
Our method involves cosine similarity, which computes the angle between vectors rather than their magnitude, making it less sensitive to the scale of the input data.
This leads to more stable gradients, especially in scenarios with small dataset sizes.

### 5.5 Application: Continual Learning

Building upon many prior studies [, , , ] that leverage synthetic datasets in continual learning to assess the quality of synthetic data, we employ the GDumb framework [] for our continual learning setup. This framework sequentially stores prior training data in memory and utilizes both new and stored data to train the model.

We conduct class-incremental learning on Tiny-ImageNet with an IPC =10 using ResNet-18. Figure 2 illustrates both 5-step and 10-step class-incremental learning strategies, partitioning the 200 classes into either 5 or 10 learning steps, corresponding to 40 and 20 classes per step, respectively. It is evident that our results substantially improve upon the baseline method RDED (^6^66RDED is the current state-of-the-art method as shown in Table 1, so we conduct the experiment on its synthetic datasets.).

Figure: (a) 5-step
Refer to caption: x3.png

### 5.6 Ablation Study

#### Is Our Loss Function the Best?

To validate the efficacy and superiority of GIFT, we conducted comparative experiments with existing loss functions on synthetic datasets, utilizing both hard and soft labels. The results, presented in Table 6, clearly demonstrate that GIFT consistently outperforms all other loss functions and their combinations.

For hard label utilization, the cross-entropy (CE) loss exhibits subpar performance, mainly due to the limited information content in typically small synthetic datasets. For soft label utilization, MSE marginally outperforms the KL divergence, consistent with observations in []. In summary, existing loss functions in synthetic datasets fail to fully exploit the potential of all labels. In contrast, our method leverages both hard and soft labels simultaneously, thereby maximizing label utilization potential and improving performance.

**Table 6: Comparsion with loss functions employed in dataset distillation. The experiment is conducted on synthetic datasets distilled via RDED []. In this table, boldmeans the best result, underlinedmeans the second best, and ($\uparrow$)denotes improvements over the second best baseline.**
|  |  | CIFAR100 | Tiny-ImageNet | ImageNet-1K |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 10 | 50 | 10 | 50 | 10 | 50 | 100 |
| Hard Label | CE | 26.6 $\pm$ 0.4 | 40.8 $\pm$ 0.1 | 14.2 $\pm$ 0.3 | 26.9 $\pm$ 0.6 | 9.1 $\pm$ 0.1 | 17.5 $\pm$ 0.1 | 21.5 $\pm$ 0.1 |
| Soft Label | KL | 47.5 $\pm$ 0.3 | 55.7 $\pm$ 0.4 | 41.4 $\pm$ 0.3 | 47.2 $\pm$ 0.1 | 20.1 $\pm$ 0.4 | 38.5 $\pm$ 0.2 | 41.8 $\pm$ 0.2 |
| MSE | 47.6 $\pm$ 0.4 | 55.9 $\pm$ 0.1 | 41.6 $\pm$ 0.2 | 47.3 $\pm$ 0.0 | 20.7 $\pm$ 0.4 | 38.8 $\pm$ 0.4 | 41.9 $\pm$ 0.1 |  |
| Soft CE | 40.8 $\pm$ 0.3 | 52.1 $\pm$ 0.3 | 33.4 $\pm$ 0.2 | 44.1 $\pm$ 0.4 | 17.0 $\pm$ 0.3 | 30.5 $\pm$ 0.8 | 37.2 $\pm$ 0.6 |  |
| Hard&<br>Soft Label | KL + CE | 48.2 $\pm$ 0.4 | 56.3 $\pm$ 0.5 | 41.6$\pm$ 0.3 | 46.8$\pm$ 0.5 | 20.3$\pm$ 0.2 | 35.2$\pm$ 0.2 | 39.0 $\pm$ 0.2 |
| MSE + CE | 47.4 $\pm$ 0.2 | 56.2 $\pm$ 0.0 | 41.7 $\pm$ 0.5 | 47.1 $\pm$ 0.1 | 20.5 $\pm$ 0.4 | 38.3 $\pm$ 0.3 | 40.5 $\pm$ 0.2 |  |
| Soft CE + CE | 39.7 $\pm$ 0.3 | 51.1 $\pm$ 0.4 | 32.6 $\pm$ 0.5 | 43.2 $\pm$ 0.2 | 15.2 $\pm$ 0.4 | 29.1 $\pm$ 0.8 | 34.7 $\pm$ 0.6 |  |
| GIFT (ours) | 49.5 $\pm$ 0.2($\uparrow$ 1.3) | 57.0 $\pm$ 0.1($\uparrow$ 0.7) | 42.9 $\pm$ 0.2($\uparrow$ 1.2) | 47.5 $\pm$ 0.1($\uparrow$ 0.2) | 21.7 $\pm$ 0.6($\uparrow$ 1.0) | 39.5 $\pm$ 0.1($\uparrow$ 0.7) | 42.5$\pm$ 0.1($\uparrow$ 0.6) |  |

#### Are Both Modules of GIFT Necessary?

We conducted an ablation study to assess the necessity of the teacher label refinement and cosine similarity loss function on the small-scale dataset in Table 7 and the large-scale dataset in Table 13 in Appendix D.
This evaluation was performed on both optimization-based (SRe^2L) and non-optimization-based (RDED) synthetic datasets. We compared the complete method with variants lacking either the teacher label refinement or the cosine similarity loss function. In the absence of both modules, the method was trained using its native loss function.

It is obvious that in scenarios where only one module was employed, the cosine similarity loss function significantly enhanced performance due to its direct label utilization.
Teacher label refinement consistently enhances performance, regardless of the presence of cosine similarity loss function, demonstrating its effectiveness.
Thus, both modules are essential for performance enhancement, consistent with our analysis in Section 4.

**Table 7: Ablation study of label refinement (Refine) and cosine similarity loss function (Loss) on CIFAR 100 and Tiny-ImageNet when IPC=10. This evaluation was conducted on both optimization-based (SRe^2L []) synthetic datasets and non-optimization-based (RDED []) synthetic datasets using ConvNet.**
|  | GIFT | CIFAR100 | Tiny-ImageNet |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DD Method | Refine | Loss | 1 | 10 | 50 | 1 | 10 | 50 |
| SRe^2L | ✗ | ✗ | 13.6 $\pm$ 0.4 | 33.7 $\pm$ 0.5 | 52.3 $\pm$ 0.2 | 12.1 $\pm$ 0.4 | 34.5 $\pm$ 0.4 | 46.3 $\pm$ 0.1 |
| ✓ | ✗ | 14.2 $\pm$ 0.4 ($\uparrow$ 0.6) | 34.5 $\pm$ 0.5 ($\uparrow$ 0.8) | 52.8 $\pm$ 0.5 ($\uparrow$ 0.5) | 12.5 $\pm$ 0.4 ($\uparrow$ 0.4) | 35.1 $\pm$ 0.4 ($\uparrow$ 0.6) | 46.6 $\pm$ 0.3 ($\uparrow$ 0.3) |  |
| ✗ | ✓ | 14.7 $\pm$ 0.4 ($\uparrow$ 1.1) | 37.3 $\pm$ 0.4 ($\uparrow$ 3.6) | 54.6 $\pm$ 0.1 ($\uparrow$ 2.3) | 12.7 $\pm$ 0.4 ($\uparrow$ 0.6) | 36.9 $\pm$ 0.3 ($\uparrow$ 2.4) | 46.9 $\pm$ 0.1 ($\uparrow$ 0.6) |  |
| ✓ | ✓ | 15.1 $\pm$ 0.3 ($\uparrow$ 1.5) | 38.0 $\pm$ 0.5 ($\uparrow$ 4.3) | 55.4 $\pm$ 0.1 ($\uparrow$ 3.1) | 13.1 $\pm$ 0.2 ($\uparrow$ 1.0) | 37.5 $\pm$ 0.3 ($\uparrow$ 3.0) | 47.1 $\pm$ 0.1 ($\uparrow$ 0.8) |  |
| RDED | ✗ | ✗ | 22.1 $\pm$ 0.3 | 47.5 $\pm$ 0.3 | 55.7 $\pm$ 0.4 | 17.9 $\pm$ 0.3 | 41.4 $\pm$ 0.3 | 47.2 $\pm$ 0.1 |
| ✓ | ✗ | 22.9 $\pm$ 0.3 ($\uparrow$ 0.8) | 48.0 $\pm$ 0.3 ($\uparrow$ 0.5) | 56.3 $\pm$ 0.1 ($\uparrow$ 0.6) | 18.2 $\pm$ 0.3 ($\uparrow$ 0.3) | 41.9 $\pm$0.4 ($\uparrow$ 0.5) | 48.1 $\pm$ 0.2 ($\uparrow$ 0.9) |  |
| ✗ | ✓ | 23.8 $\pm$ 0.2 ($\uparrow$ 1.7) | 49.5 $\pm$ 0.2 ($\uparrow$ 2.0) | 57.0 $\pm$ 0.1 ($\uparrow$ 1.3) | 18.5 $\pm$ 0.3 ($\uparrow$ 0.6) | 42.9 $\pm$ 0.2 ($\uparrow$ 1.5) | 47.5 $\pm$ 0.1 ($\uparrow$ 0.3) |  |
|  | ✓ | ✓ | 24.7 $\pm$ 0.3 ($\uparrow$ 2.6) | 50.6 $\pm$ 0.3 ($\uparrow$ 3.1) | 57.9 $\pm$ 0.2 ($\uparrow$ 2.2) | 19.1 $\pm$ 0.3 ($\uparrow$ 1.2) | 44.0 $\pm$ 0.2 ($\uparrow$ 2.6) | 48.3 $\pm$ 0.1 ($\uparrow$ 1.1) |

#### Influence of Hyper-parameter γ 𝛾 \gamma italic_γ .

We examined the impact of the weight hyperparameter $\gamma$, defined in Section 4. As shown in Figure 3, GIFT achieves optimal performance when $\gamma=0.1$. Notably, for values of $\gamma$ greater than 0.1, a significant performance decline is observed across all methods as $\gamma$ increases.
A plausible explanation is that larger values of $\gamma$ diminish the intra-class information content in soft labels. This observation aligns with our findings in Table 6, where training with exclusively hard labels via CE results in poor performance.
To verify that $\gamma=0.1$ is also optimal for different network architectures, we conducted experiments on the same three datasets using ConNet. The results, shown in Figure 8 in Appendix D, support the same conclusion.

Figure: (a) CIFAR-100
Refer to caption: x5.png

## 6 Conclusion and Limitation

This work introduces a novel perspective on dataset distillation by emphasizing the full utilization of synthetic labels. We first conduct a comprehensive comparison of existing loss functions used for soft labels in dataset distillation. Our findings reveal that models trained on synthetic datasets exhibit significant sensitivity to the choice of loss function. Based on these insights, we propose a simple yet effective plug-and-play method, GIFT, designed to fully exploit synthetic labels. Experimental results across various scales and resolutions of image datasets demonstrate that GIFT consistently outperforms state-of-the-art dataset distillation algorithms. In this work, we mainly focus on the image classification task. However, we believe the extensions of this work to other tasks such as text data [] and architectures like LSTM [].

## Appendix A Proof of Theorem 1

###### Proof.

The InfoNCE loss is defined as:

$$ $\mathcal{L}_{\text{InfoNCE}}=-\mathbb{E}\left[\log\frac{\exp(\text{sim}(z_{i}, y_{i})/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(z_{i},y_{j})/\tau)}\right]$ (4) $$

where $\text{sim}(z_{i},y_{i})$ represents the cosine similarity between $x_{i}$ and $y_{i}$:

$$ $\text{sim}(z_{i},y_{i})=\frac{z_{i}\cdot y_{i}}{\|z_{i}\|\|y_{i}\|}$ (5) $$

Substituting the expression for cosine similarity into the InfoNCE loss:

$$ $\displaystyle\mathcal{L}_{\text{InfoNCE}}$ $\displaystyle=-\mathbb{E}\left[\log\frac{\exp\left(\frac{z_{i}\cdot y_{i}}{\|z _{i}\|\|y_{i}\|\tau}\right)}{\sum_{j=1}^{N}\exp\left(\frac{z_{i}\cdot y_{j}}{ \|z_{i}\|\|y_{j}\|\tau}\right)}\right]$ (6) $\displaystyle=-\mathbb{E}\left[\log{\exp\left(\frac{z_{i}\cdot y_{i}}{\|z_{i} \|\|y_{i}\|\tau}\right)}-\log{\sum_{j=1}^{N}\exp\left(\frac{z_{i}\cdot y_{j}}{ \|z_{i}\|\|y_{j}\|\tau}\right)}\right]$ $\displaystyle=-\mathbb{E}\left[\left(\frac{z_{i}\cdot y_{i}}{\|z_{i}\|\|y_{i} \|\tau}\right)-\log{\sum_{j=1}^{N}\exp\left(\frac{z_{i}\cdot y_{j}}{\|z_{i}\| \|y_{j}\|\tau}\right)}\right]$ $\displaystyle=-\mathbb{E}\left[\left(\frac{z_{i}\cdot y_{i}}{\|z_{i}\|\|y_{i} \|\tau}\right)\right]+\mathbb{E}\left[\log{\sum_{j=1}^{N}\exp\left(\frac{z_{i} \cdot y_{j}}{\|z_{i}\|\|y_{j}\|\tau}\right)}\right]$ $$

Applying Jensen’s inequality to the logarithm:

$$ $\mathcal{L}_{\text{InfoNCE}}\leq-\mathbb{E}\left[\left(\frac{z_{i}\cdot y_{i}} {\|z_{i}\|\|y_{i}\|\tau}\right)\right]+\log\left(\mathbb{E}\left[\sum_{j=1}^{N }\exp\left(\frac{z_{i}\cdot y_{j}}{\|z_{i}\|\|y_{j}\|\tau}\right)\right]\right)$ (7) $$

Assuming the negative samples $y_{j}$ are drawn from a similar distribution, we approximate the denominator:

$$ $\sum_{j=1}^{N}\exp\left(\frac{z_{i}\cdot y_{j}}{\|z_{i}\|\|y_{j}\|\tau}\right) \approx N\exp\left(\frac{\mathbb{E}[z_{i}\cdot y_{j}]}{\|z_{i}\|\mathbb{E}[\|y _{j}\|]\tau}\right)$ (8) $$

Substituting this approximation into the upper bound:

$$ $\boxed{\mathcal{L}_{\text{InfoNCE}}\leq-\frac{1}{\tau}\left(\mathbb{E}\left[ \left(\frac{z_{i}\cdot y_{i}}{\|z_{i}\|\|y_{i}\|}\right)\right]-\frac{\mathbb{ E}[z_{i}\cdot y_{j}]}{\|z_{i}\|\mathbb{E}[\|y_{j}\|]}\right)+\log(N)}$ (9) $$

∎

###### Proof.

## Appendix B Related Work

### B.1 Labels in Dataset Distillation

In the field of dataset distillation, the majority of existing methods fix the labels as a one-hot format [, , , , , , , ], with a primary focus on optimizing synthetic images.
The potential of soft labels has been demonstrated in recent research, and the methods for obtaining soft labels can be broadly categorized into two types.

#### Optimization-based Soft Labels.

The first type involves learning labels, with several studies [, , , , , ] finding that making labels learnable can significantly improve performance and efficiency.
LD [] first introduces label distillation and reveals that even only learning labels without learning images can achieve satisfactory performance with high efficiency.
Recent work [] further highlights that optimizing labels can enhance training stability and improve performance.

#### Teacher model-based soft labels.

The subsequent trajectory of works seek to directly obtain soft labels.
Inspired by knowledge distillation [], TESLA []introduces a soft label assignment strategy, directly generating soft labels by leveraging pre-trained teacher models trained on real datasets.
These soft labels provide rich intra-class information, thereby improving distillation performance.
Following this trend, the state-of-art methods [, , ] also utilize soft labels predicted by teacher models, achieving significant improvements.

### B.2 Knowledge Distillation

A straightforward method to simultaneously utilize soft and hard labels is knowledge distillation [], which transfers knowledge from a large teacher model to a small student model. In this training process, the student model is supervised by hard labels and soft labels from the teacher’s output. Many following works aim to enhance the use of soft labels for more effective knowledge transfer. [] investigated the regularization property of soft labels and introduced a teacher-free distillation approach.
WSLD [] analyzes soft labels and distributes different weights for them from a perspective of bias-variance trade-off. DKD [] decouples the logit and assigns different weights for the target and non-target classes.

Despite the promising potential of knowledge distillation in transferring knowledge from teacher to student models using soft labels, its application to our problem yields limited improvement. A detailed analysis and comparison of these limitations are provided in

## Appendix C Experiment Details

#### Datasets.

As described in Section 5.1, we evaluate the state-of-the-art dataset distillation methods and our proposed GIFT on both small-scale and large-scale datasets. More Information about datasets utilized are listed in Table 8.

**Table 8: Details about the datasets**
| Dataset | Num of Classes | IPC of Trainset | IPC of Testset |
| --- | --- | --- | --- |
| CIFAR-10 | 10 | 5000 | 1000 |
| CIFAR-100 | 100 | 500 | 100 |
| Tiny-ImageNet | 200 | 500 | 50 |
| ImageNet-1k | 1000 | 732 - 1300 | 50 |

#### Models.

The experiment utilized a plethora of pre-trained models, and we provided the accuracy of these pre-trained models in the Table 9. The results are provided for reference only.

**Table 9: Accuracy of pretrained models.**
| Dataset | Model | Size | Accuracy |
| --- | --- | --- | --- |
| CIFAR-10 | resnet18_modified | 32 $\times$ 32 | 93.86 |
| ConvNet-3 | 32 $\times$ 32 | 82.24 |  |
| CIFAR-100 | resnet18_modified | 32 $\times$ 32 | 72.27 |
| ConvNet-3 | 32 $\times$ 32 | 61.27 |  |
| Tiny-ImageNet | resnet18_modified | 64 $\times$ 64 | 61.98 |
| ConvNet-4 | 64 $\times$ 64 | 49.73 |  |
| ImageNet-1k | resnet18 | 224 $\times$ 224 | 69.31 |
| ConvNet-4 | 64 $\times$ 64 | 43.6 |  |

#### Baselines.

We first compare our method GIFT with the state-of-the-art dataset distillation methods, that involve soft labels and can scale to large high-resolution datasets:

- •
TESLA [] marks the first distillation approach that *extends to the full ImageNet-1K*, circumventing the extensive memory demands associated with MTT-derived methods through a constant memory footprint. However, TESLA does not provide public synthesic datasets, so we are not able to conduct on it.
- •
SRe^2L [] and RDED []: both of them use soft labels assigned by a teacher model and use them via KL divergence.
- •
DATM []: initial soft labels assigned by multiple teacher models and then optimized based on trajectory matching. Finally, this method employs soft cross-entropy loss for soft labels.
- •
G-VBSM []: soft labels are assigned by multiple teacher models and then used via MSE-CE loss function.

Knowledge distillation [] is a straightforward method to utilize labels, especially for soft labels.
Therefore, we also compare our method GIFT with the state-of-the-art knowledge distillation methods that focus on soft labels utilization:

- •
KD []: it is the first method to transfer knowledge using both hard and soft labels from the teacher’s output.
- •
WSLD []: it analyzes soft labels and then distributes different weights for them from a perspective of bias-variance trade-off.
- •
DKD []: it decouples the logits and assigns different weights for the target and non-target classes.
- •
NKD []: it finds the sum of the two non-target logits is different, preventing logits’ distributions from being identical. Therefore, it normalizes the non-target logits to equalize their sum.

#### Evaluating main results.

For both dataset distillation and performance evaluation, we employ identical neural network architectures.
Consistent with previous studies [, , ], we use Conv-3 for CIFAR-10 and CIFAR-100 distillation tasks, Conv-4 for Tiny-ImageNet (with the exception of DREAM, which utilizes Conv-3) and ImageNet-1K, Conv-5 for ImageNet-10, and Conv-6 for ImageNet-100 distillation. In line with [, ], MTT and TESLA apply a reduced resolution for distilling $224\times 224$ images. According to [], for retrieving and evaluating distilled datasets, SRe^2L and GIFT adopt ResNet-18.

#### Evaluating the distilled dataset.

Consistent with recent works [, ], the evaluation on the distilled dataset follows the parameters outlined in Table 11.
Furthermore, we implement Differentiable Siamese Augmentation (DSA) as described by [] to enhance images during both the distillation and evaluation phases of our experiments.

#### Differentiable Siamese Augmentation (DSA).

We use DSA (Differentiable Siamese Augmentation) as a tool for image augmentation. For the sake of clarity, we delineate the DSA operations utilized in Table 10, alongside their respective transforms and probabilities.

**Table 10: Differentiable Siamese Augmentation(DSA) and ratios**
| DSA | Transform | Ratio |
| --- | --- | --- |
| Color | Color Jitter | Brightness=1.0<br>Saturation=2.0<br>Contrast=0.5 |
| Crop | Random Crop | Crop Pad=0.125 |
| Cutout | Random Cutout | Cutout=0.5 |
| Flip | Random Horizontal Flip | Flip=0.5 |
| Scale | Random Scale | Scale=1.2 |
| Rotate | Random Rotation | Rotate=15.0 |

**Table 11: Evaluation Hyperparameter setting**
| Config | Value | Explanation |
| --- | --- | --- |
| Epochs | 300/1000 | 300 for ImageNet-1k,<br>1000 for default |
| Optimizer | AdamW | NA |
| Learning Rate | 0.001 | NA |
| Batch Size | 10/50/100/200 | 10 for 0 $\textless$ Num of Images $\leq$ 10,<br>50 for 10 $\textless$ Num of Images $\leq$ 500,<br>100 for 500 $\textless$ Num of Images $\leq$ 20000,<br>200 for 20000 $\textless$ Num of Images |
| Scheduler | MultiStepLR | milestones=[2 $\times$ epochs // 3, 5 $\times$ epochs // 6]<br>gamma=0.2 |
| Augmentation | DSA strategy | color, crop, cutout, flip, scale, rotate |

## Appendix D Experiment Results

#### Comprehensive Comparison Between Different Loss Functions

Our experiments span two datasets, including Tiny-ImageNet, and ImageNet-1K on $\texttt{IPC}\in\{1,10,50\}$ using ConvNet [].
Note that the synthetic dataset for DATM on ImageNet-1K is unavailable, precluding comparisons on this dataset.
We evaluate at $\texttt{IPC}\in\{1,10,50\}$.
The results, visualized in Figure 4, Figure 6, Figure 5 and Figure 7, reveal that *the performance of models trained on synthetic datasets is highly sensitive to the choice of the loss function*, highlighting the necessity for a unified and effective loss function in dataset distillation.

Figure: Figure 4: Top-1 accuracy on various synthetic datasets via the SOTA dataset distillation methods across loss functions on Tiny-ImageNet when IPC=1.
Refer to caption: x8.png

Figure: Figure 5: Top-1 accuracy on various synthetic datasets via the SOTA dataset distillation methods across loss functions on ImageNet-1K when IPC=1.
Refer to caption: x9.png

Figure: Figure 6: Top-1 accuracy on various synthetic datasets via the SOTA dataset distillation methods across loss functions on ImageNet-1K when IPC=50.
Refer to caption: x10.png

Figure: Figure 7: Pytorch Implementation Code
Refer to caption: x11.png

**Table 12: Top-1 accuracy on cross-optimization generalization on ImageNet-1K when IPC=10.We evaluate the performance of synthetic datasets across various optimizers.**
| ImageNet-1K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | ConvNet | ResNet |  |  |  |  |
| Method | SGD | Adam | AdamW | SGD | Adam | AdamW |
| SRe^2L | 0.1 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 12.5 $\pm$ 0.1 | 0.1 $\pm$ 0. | 0.1 $\pm$ 0.0 | 31.5 $\pm$ 0.3 |
| SRe^2L + Ours | 18.2 $\pm$ 0.2 ($\uparrow$ 18.1) | 26.6 $\pm$ 0.2 ($\uparrow$ 26.5) | 14.2 $\pm$ 0.6 ($\uparrow$ 1.7) | 36.1 $\pm$ 0.1 ($\uparrow$ 36.0) | 24.5 $\pm$ 0.2 ($\uparrow$ 24.4) | 31.9 $\pm$ 0.2 ($\uparrow$ 0.4) |
| RDED | 0.1 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 20.1 $\pm$ 0.4 | 0.1 $\pm$ 0.0 | 0.1 $\pm$ 0.0 | 41.4 $\pm$ 0.4 |
| RDED + Ours | 26.7 $\pm$ 0.6 ($\uparrow$ 26.5) | 30.9 $\pm$ 0.7 ($\uparrow$ 26.5) | 24.0 $\pm$ 0.8 ($\uparrow$ 3.2) | 45.8 $\pm$ 0.4 ($\uparrow$ 30.8) | 29.1 $\pm$ 0.3 ($\uparrow$ 29.0) | 43.2 $\pm$ 0.1 ($\uparrow$ 1.8) |
| G-VBSM | 20.4 $\pm$ 0.8 | 25.0 $\pm$ 0.4 | 22.6 $\pm$ 0.5 | 38.7 $\pm$ 0.2 | 27.0 $\pm$ 0.2 | 36.7 $\pm$ 0.2 |
| G-VBSM + Ours | 27.4 $\pm$ 0.8 ($\uparrow$ 7.0) | 29.8 $\pm$ 0.5 ($\uparrow$ 4.8) | 24.3 $\pm$ 0.2 ($\uparrow$ 1.7) | 41.8 $\pm$ 0.1 ($\uparrow$ 3.1) | 27.8 $\pm$ 0.8 ($\uparrow$ 0.8) | 37.9 $\pm$ 0.5 ($\uparrow$ 1.2) |

#### Influence of Hyper-parameter γ 𝛾 \gamma italic_γ on ConvNet.

We also examined the impact of the hyper-parameter $\gamma$ using ConvNet, and the results are shown in Figure 8. Similar to the findings with ResNet, GIFT achieves optimal performance when $\gamma=0.1$.

Figure: (a) CIFAR-100
Refer to caption: x12.png

#### Ablation study on ImageNet-1K

We also conducted an ablation study to assess the necessity of the teacher label refinement and cosine similarity loss function on the large-scale dataset in Table 13.
This evaluation was performed on both optimization-based (SRe^2L) and non-optimization-based (RDED) synthetic datasets.
It is obvious both modules are essential for performance enhancement, consistent with our analysis in Section 4.

#### Cross-Optimizaer on ImageNet-1K

In this experiment, we report the accuracy before and after applying our GIFT to baseline methods across multiple optimizers not seen during dataset distillation onImageNet-1K , as shown in Table 12. The results clearly indicate that GIFT enhances the generalization ability of all baseline methods. This leads to more stable gradients, especially in scenarios with small dataset sizes.

**Table 13: Ablation study of label refinement (Refine) and cosine similarity loss function (Loss) on ImageNet-1K. In the table, ($\uparrow$) means the improvements over these methods.**
|  |  | ImageNet-1K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | ConvNet | ResNet-18 |  |  |  |  |
|  | Method | 10 | 50 | 100 | 10 | 50 | 100 |
|  | SRe^2L | 12.5 $\pm$ 0.3 | 35.4 $\pm$ 1.0 | 40.1 $\pm$ 0.4 | 31.5 $\pm$ 0.3 | 49.5 $\pm$ 0.1 | 54.3 $\pm$ 0.2 |
|  | SRe^2L + Loss | 13.1 $\pm$ 0.6 ($\uparrow$ 0.6) | 36.8 $\pm$ 0.3 ($\uparrow$ 1.2) | 40.7 $\pm$ 0.3 ($\uparrow$ 0.6) | 31.7 $\pm$ 0.2 ($\uparrow$ 0.2) | 49.8 $\pm$ 0.2 ($\uparrow$ 0.3) | 54.5 $\pm$ 0.2 ($\uparrow$ 0.2) |
|  | SRe^2L + Refine + Loss | 14.2 $\pm$ 0.6 ($\uparrow$ 1.7) | 38.1 $\pm$ 0.4 ($\uparrow$ 2.7) | 41.5 $\pm$ 0.2 ($\uparrow$ 1.4) | 31.9 $\pm$ 0.1 ($\uparrow$ 0.4) | 50.1 $\pm$ 0.2 ($\uparrow$ 0.6) | 54.8 $\pm$ 0.1 ($\uparrow$ 0.5) |
|  | RDED | 20.1 $\pm$ 0.4 | 38.5 $\pm$ 0.2 | 41.8 $\pm$ 0.2 | 41.4 $\pm$ 0.4 | 55.5 $\pm$ 0.2 | 58.8 $\pm$ 0.1 |
|  | RDED + Loss | 21.7 $\pm$ 0.6 ($\uparrow$ 1.6) | 39.3 $\pm$ 0.1 ($\uparrow$ 0.8) | 42.1 $\pm$ 0.1 ($\uparrow$ 0.3) | 42.0 $\pm$ 0.2 ($\uparrow$ 0.6) | 55.9 $\pm$ 0.1 ($\uparrow$ 0.4) | 59.1 $\pm$ 0.1 ($\uparrow$ 0.3) |
|  | RDED + Refine + Loss | 24.0 $\pm$ 0.8 ($\uparrow$ 3.9) | 39.5 $\pm$ 0.1 ($\uparrow$ 1.0) | 42.5$\pm$ 0.1 ($\uparrow$ 0.7) | 43.2 $\pm$ 0.1 ($\uparrow$ 1.8) | 56.5 $\pm$ 0.2 ($\uparrow$ 1.0) | 59.3$\pm$ 0.1 ($\uparrow$ 0.5) |
|  | G-VBSM | 22.6 $\pm$ 0.5 | 37.3 $\pm$ 0.3 | 40.1 $\pm$ 0.4 | 36.7 $\pm$ 0.2 | 52.3 $\pm$ 0.1 | 57.3 $\pm$ 0.1 |
|  | G-VBSM + Loss | 23.8 $\pm$ 0.2 ($\uparrow$ 1.2) | 38.3 $\pm$ 0.4 ($\uparrow$ 1.0) | 41.8 $\pm$ 0.3 ($\uparrow$ 1.7) | 37.5 $\pm$ 0.5 ($\uparrow$ 0.8) | 52.8 $\pm$ 0.2 ($\uparrow$ 0.5) | 57.4 $\pm$ 0.1 ($\uparrow$ 0.1) |
|  | G-VBSM + Refine + Loss | 24.3 $\pm$ 0.2 ($\uparrow$ 1.7) | 39.1 $\pm$ 0.3 ($\uparrow$ 1.8) | 42.1 $\pm$ 0.3 ($\uparrow$ 2.0) | 37.9 $\pm$ 0.5 ($\uparrow$ 1.2) | 53.1 $\pm$ 0.2 ($\uparrow$ 0.8) | 57.6 $\pm$ 0.1 ($\uparrow$ 0.3) |

## Appendix E Pytorch Implementation Code

Figure: Figure 9: Pytorch Implementation Code
Refer to caption: extracted/2405.14736v1/figures/torch_code.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you explain theoretically why different optimizers have such a big impact on DD?
soundness: 3
strengths: - It is interesting to find that models trained on synthetic datasets show high sensitivity to the choice of loss function for soft label utilization.
- The authors reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers.
- They propose GIFT and prove its effectiveness through experiments.
- The experiment is well designed.
summary: This paper conducts a comprehensive comparison of various loss functions for soft label utilization in DD and find that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. Building on these insights, the authors introduce GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to use full label information. Extensive experiments demonstrate the effectiveness of GIFT.
weaknesses: - In Table 1, the results of ResNet-18 on CIFAR-100 are not very significant, which may not reflect the effect of the proposed algorithm.
- The current synthetic datasets on Tiny-ImageNet and ImageNet-1K are only generated using Conv-4. Can we experiment with structures such as transformer?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - Could you share the results of your label utilization method on a randomly selected data subset?  
- How does GIFT perform with higher IPC values, such as 100 or 200?  
- Is it possible to achieve lossless dataset distillation using GIFT on any dataset?  
- It would be interesting to see how the Jensen-Shannon divergence loss function performs within the GIFT framework.
soundness: 3
strengths: [S1] The paper dives into a crucial part of dataset distillation, specifically how to choose effective loss functions for label utilization in DD frameworks.  
[S2] The writing is clear and easy to follow.  
[S3] The use of cosine similarity is backed by solid theoretical reasoning.  
[S4] The ablation studies are detailed, and the method shows promising results in continual learning.  
[S5] GIFT consistently improves state-of-the-art dataset distillation methods across various datasets without adding any extra computational cost.
summary: This paper introduces a novel method for dataset distillation that fully leverages soft labels generated by pre-trained teacher models. The authors highlight the critical role of selecting the right loss function, showing that the model's performance on synthetic datasets is highly dependent on this choice. They propose **GIFT**, a simple yet effective method that refines soft labels and uses a cosine similarity-based loss function to boost performance across various tasks. The approach also enhances generalization across optimizers and architectures. Extensive experiments demonstrate that **GIFT** significantly improves performance without adding computational costs, making it scalable even for large datasets like ImageNet-1K.
weaknesses: [W1] From a technical standpoint, the contribution of the paper feels somewhat limited. Both the hard-label refinement approach and the cosine-similarity loss function have already been explored in many knowledge distillation studies. Additionally, the paper lacks convincing evidence or justification that these techniques are specifically suited for the synthetic datasets used in dataset distillation.

[W2] Regarding the experimental results, the improvements brought by the proposed label utilization algorithm to existing DD methods are relatively modest. However, since the approach incurs no additional computational cost and offers general performance boosts, the results are still reasonable and acceptable.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How would cosine similarity perform if it is applied directly on hard labels / smoothed labels? Will it still outperform the other compared loss functions?
2. The performance improvement is generally higher on small datasets / poor architectures compared with the other end. Can the authors explain the potential reason?
soundness: 3
strengths: 1. The motivation is clear and the designed model has correspondence to the motivation. 
2. The experiments results show uniformly performance improvement over multiple distilled datasets and multiple benchmarks.
3. It is valuable to look into the validation for dataset distillation.
summary: This work focuses on the problem of soft label utilization in dataset distillation. Currently, multiple different loss functions are employed to use the soft labels generated by pre-trained models. The choice of loss function can have large influence on the training effects for different distilled datasets. Therefore, the authors propose to refine the generated soft labels, and adopt a unified cosine similarity-based loss function to leverage the soft label information. The experiments suggest that the proposed method uniformly improve the validation performance on multipld benchmarks.
weaknesses: 1. The authors claim that GIFT can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns an incorrect label of the highest value. However, there is no qualitative or quantitative result supporting it. Since the $\gamma$ parameter is only set as 0.1, it remains unclear how it can correct potential soft label error. Can the authors summarize how much the soft labels change after the refinement?
2. The authors introduce a new evaluation scenario named cross-optimizer generalization. But it is not defined until the end of the introduction section. It is recommended to raise the issue of poor cross-optimizer generalitzation earlier. Does it serve as one of the motivations, or is it simply empirical results after the method is designed?
3. The authors show in Table 6 the cross-architecture generalization results on Tiny-ImageNet. In many cases the performance gain of large models is on par or lower than that on small models. Is the improvement benefited simply from better training effects of cosine similarity or better generalization?
4. Although the proposed method uniformly improves the validation results with soft labels on different distilled datasets, the soft label requires enormous storage space. While it is feasible to use currently available pre-trained models for popular dataset distillation benchmarks, it is not practical for actual use on new application scenarios. Thus, the overall practicality slightly degrades the value of this work for the dataset distillation society.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to weaknesses.
soundness: 3
strengths: 1. Unlike previous methods, this paper shifts focus from the distillation process itself to the model training phase using synthetic data, presenting a novel perspective for dataset distillation. The paper proposes a universal loss function that enhances the performance of most dataset distillation methods. 
2. Currently, most state-of-the-art methods incorporate soft labels into distillation, complicating unified evaluation when training models with synthetic data. The approach in this paper could potentially help establish a standardized evaluation benchmark for dataset distillation products.
3. Extensive experimental results demonstrate that GIFT significantly improves state-of-the-art dataset distillation methods on large-scale datasets, such as ImageNet-1K, while also enhancing cross-optimizer and cross-architecture generalization. GIFT’s plug-and-play nature makes it versatile across various datasets and network architectures.
4. The presentation of the paper is great. It provides a detailed discussion and clearly showcases the motivation and methodology.
summary: This paper proposes a simple but effective plug-and-play dataset distillation method to leverage both hard and soft labels at near-zero cost. 
It identifies that existing dataset distillation methods are highly sensitive to different loss functions, highlighting the need for a universal loss function. Based on this insight, GIFT introduces a label refinement process that incorporates hard label smoothing, thereby correcting any erroneous signals from the teacher model and enhancing inter-class separation. Additionally, the authors use a cosine similarity-based loss function to fully exploit label information during training.
weaknesses: 1. Although the method offers a novel perspective, its lack of improvement in the distillation process itself somewhat limits the contribution of the paper. 
2. I suggest the authors consider incorporating GIFT into [1], i.e., replacing the loss with GIFT's loss in [1], or adding a loss term in other trajectory-matching methods (as long as they can access a well-trained model during training), I am interested to see if this combination could further improve the effectiveness of dataset distillation.


[1] Can pre-trained models assist in dataset distillation? arXiv:2310.03295

Meta-review: This work comprehensive evaluates several loss functions for dataset distillation and reveals the high sensitivity of model trained on the
synthetic dataset to the choice of loss function. leveraging this finding, the authors introduce a new method called GIFT, which is a simple yet effective method for dataset differentiation with soft label refinement and a cosine similarity-based loss function. Overall, I believe the proposed GIFT method is novel, the experimental results are solid and demonstrate that GIFT significantly enhances performance without incurring additional computational costs. Therefore, I would like to suggest accept to this manuscript.

Decision: Accept (Poster)