GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost

Authors: Xinyi Shang, Peng Sun, Tao Lin
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Dataset Distillation, Soft Label

Abstract:
Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models.
    In this paper, we introduce a novel perspective by emphasizing the full utilization of labels.
    We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization.
    This finding highlights the necessity of a universal loss function for training models on synthetic datasets.
    Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. 
    Extensive experiments indicate that GIFT consistently enhances state-of-the-art dataset distillation methods across various dataset scales without incurring additional computational costs.
    Importantly, GIFT significantly enhances cross-optimizer generalization, an area previously overlooked.
    For instance, on ImageNet-1K with IPC = 10, GIFT enhances the state-of-the-art method RDED by 30.8% in cross-optimizer generalization. Our code is available at https://github.com/LINs-lab/GIFT.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730019895486

thread_type:
official_review

<thread_object>
creator: reviewer_twq2

<content>
summary:
This paper proposes a simple but effective plug-and-play dataset distillation method to leverage both hard and soft labels at near-zero cost. 
It identifies that existing dataset distillation methods are highly sensitive to different loss functions, highlighting the need for a universal loss function. Based on this insight, GIFT introduces a label refinement process that incorporates hard label smoothing, thereby correcting any erroneous signals from the teacher model and enhancing inter-class separation. Additionally, the authors use a cosine similarity-based loss function to fully exploit label information during training.

strengths:
1. Unlike previous methods, this paper shifts focus from the distillation process itself to the model training phase using synthetic data, presenting a novel perspective for dataset distillation. The paper proposes a universal loss function that enhances the performance of most dataset distillation methods. 
2. Currently, most state-of-the-art methods incorporate soft labels into distillation, complicating unified evaluation when training models with synthetic data. The approach in this paper could potentially help establish a standardized evaluation benchmark for dataset distillation products.
3. Extensive experimental results demonstrate that GIFT significantly improves state-of-the-art dataset distillation methods on large-scale datasets, such as ImageNet-1K, while also enhancing cross-optimizer and cross-architecture generalization. GIFT’s plug-and-play nature makes it versatile across various datasets and network architectures.
4. The presentation of the paper is great. It provides a detailed discussion and clearly showcases the motivation and methodology.

weaknesses:
1. Although the method offers a novel perspective, its lack of improvement in the distillation process itself somewhat limits the contribution of the paper. 
2. I suggest the authors consider incorporating GIFT into [1], i.e., replacing the loss with GIFT's loss in [1], or adding a loss term in other trajectory-matching methods (as long as they can access a well-trained model during training), I am interested to see if this combination could further improve the effectiveness of dataset distillation.


[1] Can pre-trained models assist in dataset distillation? arXiv:2310.03295

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730227840740

thread_type:
official_review

<thread_object>
creator: reviewer_izft

<content>
summary:
This work focuses on the problem of soft label utilization in dataset distillation. Currently, multiple different loss functions are employed to use the soft labels generated by pre-trained models. The choice of loss function can have large influence on the training effects for different distilled datasets. Therefore, the authors propose to refine the generated soft labels, and adopt a unified cosine similarity-based loss function to leverage the soft label information. The experiments suggest that the proposed method uniformly improve the validation performance on multipld benchmarks.

strengths:
1. The motivation is clear and the designed model has correspondence to the motivation. 
2. The experiments results show uniformly performance improvement over multiple distilled datasets and multiple benchmarks.
3. It is valuable to look into the validation for dataset distillation.

weaknesses:
1. The authors claim that GIFT can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns an incorrect label of the highest value. However, there is no qualitative or quantitative result supporting it. Since the $\gamma$ parameter is only set as 0.1, it remains unclear how it can correct potential soft label error. Can the authors summarize how much the soft labels change after the refinement?
2. The authors introduce a new evaluation scenario named cross-optimizer generalization. But it is not defined until the end of the introduction section. It is recommended to raise the issue of poor cross-optimizer generalitzation earlier. Does it serve as one of the motivations, or is it simply empirical results after the method is designed?
3. The authors show in Table 6 the cross-architecture generalization results on Tiny-ImageNet. In many cases the performance gain of large models is on par or lower than that on small models. Is the improvement benefited simply from better training effects of cosine similarity or better generalization?
4. Although the proposed method uniformly improves the validation results with soft labels on different distilled datasets, the soft label requires enormous storage space. While it is feasible to use currently available pre-trained models for popular dataset distillation benchmarks, it is not practical for actual use on new application scenarios. Thus, the overall practicality slightly degrades the value of this work for the dataset distillation society.

questions:
1. How would cosine similarity perform if it is applied directly on hard labels / smoothed labels? Will it still outperform the other compared loss functions?
2. The performance improvement is generally higher on small datasets / poor architectures compared with the other end. Can the authors explain the potential reason?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730245152266

thread_type:
official_review

<thread_object>
creator: reviewer_fua8

<content>
summary:
This paper introduces a novel method for dataset distillation that fully leverages soft labels generated by pre-trained teacher models. The authors highlight the critical role of selecting the right loss function, showing that the model's performance on synthetic datasets is highly dependent on this choice. They propose **GIFT**, a simple yet effective method that refines soft labels and uses a cosine similarity-based loss function to boost performance across various tasks. The approach also enhances generalization across optimizers and architectures. Extensive experiments demonstrate that **GIFT** significantly improves performance without adding computational costs, making it scalable even for large datasets like ImageNet-1K.

strengths:
[S1] The paper dives into a crucial part of dataset distillation, specifically how to choose effective loss functions for label utilization in DD frameworks.  
[S2] The writing is clear and easy to follow.  
[S3] The use of cosine similarity is backed by solid theoretical reasoning.  
[S4] The ablation studies are detailed, and the method shows promising results in continual learning.  
[S5] GIFT consistently improves state-of-the-art dataset distillation methods across various datasets without adding any extra computational cost.

weaknesses:
[W1] From a technical standpoint, the contribution of the paper feels somewhat limited. Both the hard-label refinement approach and the cosine-similarity loss function have already been explored in many knowledge distillation studies. Additionally, the paper lacks convincing evidence or justification that these techniques are specifically suited for the synthetic datasets used in dataset distillation.

[W2] Regarding the experimental results, the improvements brought by the proposed label utilization algorithm to existing DD methods are relatively modest. However, since the approach incurs no additional computational cost and offers general performance boosts, the results are still reasonable and acceptable.

questions:
- Could you share the results of your label utilization method on a randomly selected data subset?  
- How does GIFT perform with higher IPC values, such as 100 or 200?  
- Is it possible to achieve lossless dataset distillation using GIFT on any dataset?  
- It would be interesting to see how the Jensen-Shannon divergence loss function performs within the GIFT framework.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554132095

thread_type:
official_review

<thread_object>
creator: reviewer_vaem

<content>
summary:
This paper conducts a comprehensive comparison of various loss functions for soft label utilization in DD and find that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. Building on these insights, the authors introduce GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to use full label information. Extensive experiments demonstrate the effectiveness of GIFT.

strengths:
- It is interesting to find that models trained on synthetic datasets show high sensitivity to the choice of loss function for soft label utilization.
- The authors reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers.
- They propose GIFT and prove its effectiveness through experiments.
- The experiment is well designed.

weaknesses:
- In Table 1, the results of ResNet-18 on CIFAR-100 are not very significant, which may not reflect the effect of the proposed algorithm.
- The current synthetic datasets on Tiny-ImageNet and ImageNet-1K are only generated using Conv-4. Can we experiment with structures such as transformer?

questions:
- Can you explain theoretically why different optimizers have such a big impact on DD?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119647595

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank the reviewers for their insightful feedback. We are delighted that the reviewers acknowledged that our ***presentation*** is excellent and easy to follow (Reviewers FuA8, twQ2), ***motivation*** is clear and novel (Reviewers FuA8, twQ2,izfT), ***findings*** are crucial and interesting (Reviewers vaeM and FuA8), the newly ***revealed challenge*** is important (Reviewer vaeM), ***method*** has solid theoretical foundation without any extra cost (Reviewer FuA8), ***experiments*** are extensive and well-designed (Reviewers vaeM and twQ2), the ***improvements*** are significant and consistent (All Reviewers vaeM, FuA8, izfT, twQ2).

Furthermore, our contributions to diving into a ***crucial part*** of dataset distillation, establishing a ***standardized evaluation benchmark*** (Reviewer twQ2), offering a ***new perspective*** (Reviewer twQ2), and providing ***valuable insights into the validation*** for dataset distillation (Reviewer izfT) are acknowledged.

### Summary of Contribution and Novelty

Our work stands out through the following key contributions and innovations:

1. **Novel Perspective and Insightful Findings:**
    - To our knowledge, this paper is the **first t**o explore how to choose effective loss functions for label utilization in DD frameworks, which is recognized by Reviewers vaeM and FuA8.
    - Our study reveals the intriguing fact that models trained on synthetic datasets are ***highly sensitive*** to the choice of the loss function, which has been acknowledged as crucial and interesting by Reviewer vaeM and FuA8.
2. **Identification of a Key Yet Overlooked Challenge:**
    - We reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers, a critical but overlooked challenge highlighted by Reviewer vaeM.
    - Our proposed method notably improves cross-optimizer generalization, which has been emphasized and recognized by Reviewer twQ2 and vaeM.
3. **Method with a Solid Theoretical Foundation and No Additional Costs:**
    - We propose GIFT, a simple and universal label utilization algorithm including label refinement and a cosine similarity-based loss function.
    - We provide a ***solid theoretical analysis*** supporting the proposed use of cosine similarity, as acknowledged by Reviewer FuA8.
    - GIFT builds upon existing dataset distillation methods without requiring additional information, thus raising **no extra cost**, as highlighted by Reviewer twQ2.
4. **Well-designed Experimental Validation:**
    - Our experiments encompass datasets of ***varying scales and resolutions***, from small-scale, low-resolution datasets (e.g., CIFAR-10/100) to large-scale, high-resolution datasets (e.g., ImageNet-1K). The effectiveness and practical benefits of our method are recognized by all reviewers.
    - Beyond state-of-the-art experiments, our ***strong ablation studies and applications*** in continual learning have shown promising results, as highlighted by Reviewer **FuA8.**
    - GIFT significantly enhances dataset distillation methods in both ***cross-architecture and cross-optimizer generalization.***
        - We ***empirically and theoretically*** analyze why optimizers exert a significant influence on the performance of various distilled datasets, as elaborated in Appendix E on pages 21-25 in the revised manuscript.
5. **Potential Impact:**
    - This paper shifts focus from the distillation process itself to the model training phase using synthetic data, presenting ***a novel perspective*** for dataset distillation (Reviewer twQ2).
    - The approach in this paper could potentially help ***establish a standardized evaluation benchmark*** for dataset distillation products, as recognized by Reviewer twQ2.
    - It provides ***valuable*** insights into the validation of dataset distillation, as noted by Reviewer izfT.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work comprehensive evaluates several loss functions for dataset distillation and reveals the high sensitivity of model trained on the
synthetic dataset to the choice of loss function. leveraging this finding, the authors introduce a new method called GIFT, which is a simple yet effective method for dataset differentiation with soft label refinement and a cosine similarity-based loss function. Overall, I believe the proposed GIFT method is novel, the experimental results are solid and demonstrate that GIFT significantly enhances performance without incurring additional computational costs. Therefore, I would like to suggest accept to this manuscript.