Breaking Class Barriers: Efficient Dataset Distillation via Inter-Class Feature Compensator

Authors: Xin Zhang, Jiawei Du, Ping Liu, Joey Tianyi Zhou
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Dataset distillation, inter-class feature compensator (INFER)

Abstract:
Dataset distillation has emerged as a technique aiming to condense informative features from large, natural datasets into a compact and synthetic form. While recent advancements have refined this technique, its performance is bottlenecked by the prevailing class-specific synthesis paradigm. Under this paradigm, synthetic data is optimized exclusively for a pre-assigned one-hot label, creating an implicit class barrier in feature condensation. This leads to inefficient utilization of the distillation budget and oversight of inter-class feature distributions, which ultimately limits the effectiveness and efficiency, as demonstrated in our analysis.
To overcome these constraints, this paper presents the Inter-class Feature Compensator (INFER), an innovative distillation approach that transcends the class-specific data-label framework widely utilized in current dataset distillation methods. Specifically, INFER leverages a Universal Feature Compensator (UFC) to enhance feature integration across classes, enabling the generation of multiple additional synthetic instances from a single UFC input. This significantly improves the efficiency of the distillation budget.
Moreover, INFER enriches inter-class interactions during the distillation, thereby enhancing the effectiveness and generalizability of the distilled data. By allowing for the linear interpolation of labels similar to those in the original dataset, INFER meticulously optimizes the synthetic data and dramatically reduces the size of soft labels in the synthetic dataset to almost zero, establishing a new benchmark for efficiency and effectiveness in dataset distillation. In practice, INFER demonstrates state-of-the-art performance across benchmark datasets. For instance, in the $\texttt{ipc} = 50$ setting on ImageNet-1k with the same compression level, it outperforms SRe2L by 34.5\% using ResNet18. Codes are available at https://github.com/zhangxin-xd/UFC.

Decision: accept (poster)
Reviewer scores: 3, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730210142498

thread_type:
official_review

<thread_object>
creator: reviewer_ahjp

<content>
summary:
The authors highlight that the “one label per instance” paradigm reduces intra-class diversity and amplifies inter-class differences within the distilled dataset. These limitations hinder the efficient use of the distillation budget and lead to poorly defined decision boundaries. To overcome these challenges, the authors propose a novel “one instance for all classes” approach, incorporating a Universal Feature Compensator (UFC). This method enhances class overlaps and ambiguities, improving representation quality and achieving state-of-the-art performance across a variety of datasets and architectures.

strengths:
- The analysis of the two limitations in class-specific dataset distillation is insightful.
- The proposed Inter-class Feature Compensator is novel and improves the utilization of the distillation budget by leveraging inter-class features.
- To assess the size of the final distilled dataset, the authors introduce a compression ratio, in addition to the commonly used IPC metric, providing a fairer comparison with methods that utilize soft labels.
- The paper is well-written and well-structured.
- It offers extensive validation experiments across multiple datasets and architectures, reinforcing the reliability of the proposed approach.

weaknesses:
- [1] explores inter-class and inter-class relations in dataset distillation, introducing a class centralization constraint to tighten within-class clustering and improve class discrimination. While both methods report significant performance improvements, the conclusions appear to contrast with those of this paper. Could the authors elaborate on this discrepancy?
- [2] also employs multiple backbone architectures for dataset distillation. It would be helpful if the authors discuss how their approach aligns with or differs from this.
- Since the optimizable parameter is not the entire distillation budget but only the M compensators, will this speed up the distillation or synthesis phase?
- How did the authors accurately calculate the compression ratio in Table 2? Additionally, how did the author estimate the size of soft labels?
- Does the compensator consume the IPC budget in the experiments? If yes, according to the design of compensator, the IPC values should not be integers.
- Including comparisons with recent state-of-the-art dataset distillation methods extended from SRe2L [3, 4] would provide a more comprehensive evaluation.
- There are some typos; for example, in line 401, “Our INFER also employ the soft labels” should be corrected to “Our INFER also employs the soft labels.”
[1] Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. CVPR 2024.
[2] Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching. CVPR 2024
[3] Dataset Distillation in Large Data Era
[4] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm. CVPR 2024

questions:
please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730210357817

thread_type:
official_review

<thread_object>
creator: reviewer_enyk

<content>
summary:
This paper presents a novel dataset distillation (DD) method, INFER (Inter-Class Feature Compensator), which explores the potential of leveraging inter-class features to significantly enhance distillation performance. In addition, it addresses a critical challenge faced by decoupled DD methods (e.g., SRe2L), where soft labels require nearly 30 times more storage than images. To overcome this limitation, the proposed method introduces a static labelling strategy that not only retains competitive performance but also dramatically reduces storage requirements, making it a more efficient and scalable solution.

strengths:
a. The authors present an innovative analysis of existing class-specific dataset distillation methods. This work distinguishes itself from prior research by proposing the Universal Feature Compensator (UFC), which enhances inter-class feature integration, resulting in a more efficient dataset distillation framework. Relevant related works are appropriately cited. b. The submission is technically sound, with thorough evaluations and comparisons. c. The paper is clearly written and well-organized, making the content easy to follow and understand. d. This work addresses two key limitations of class-specific dataset distillation methods and achieves SOTA performance. Moreover, the proposed static labelling strategy reduces storage requirements by 99.3%.

weaknesses:
a. The compensator is denoted by $\mu$ in Fig. 2, but by $u$ in Fig. 3. Maintaining consistent notation across the figures would improve clarity and help avoid potential confusion.
b. Some results fall short of SOTA benchmarks, and it would be helpful if the authors provided a discussion of potential reasons for this discrepancy.
c. The current evaluation only focuses on same-architecture distillation performance for ConvNet and ResNet18 on CIFAR and tiny-ImageNet. A broader cross-architecture evaluation would offer better insights into generalization. 
d. While the static labelling approach effectively reduces storage requirements and, in some cases, even outperforms the dynamic method, the paper does not explore whether these benefits extend to other decoupled DD methods, such as SRe2L.

questions:
e. I have a question regarding the IPC used in the experiments, which refers to the number of images per class. If the authors apply the augmentation approach described in Equation 2, the final number of distilled images per class becomes IPC \times \left(1 + \frac{M}{{class number}}\right) . This seems to differ slightly from the IPC used in other comparison methods. While this difference might have minimal impact on large datasets with many classes, it could be more influential for smaller datasets. Have the authors accounted for this adjustment to ensure a fair comparison?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730446448054

thread_type:
official_review

<thread_object>
creator: reviewer_hymx

<content>
summary:
In general, dataset distillation methods employ a class-specific synthesis paradigm, each synthetic instance has one label. This work suggests two limitations that these "one label per instance" approaches do not utilize the budget efficiently and fails to adequately reflect inter-class features. To overcome these limitations, this work introduces a new approach, Inter-class Feature Compensator (INFER), for “one instance for all classes” framework by utilizing Universal Feature Compensator (UFC). Experimental results show the effectiveness of the proposed framework.

strengths:
1. This manuscript is well written and easy to follow along with their reasoning.
2. The idea of UFC is simple and easy to adapt.
3. The motivations of this work, inefficient budget utilization and oversight of inter-class features, are straightforward and novel. The proposed method aligns with these motivations.

weaknesses:
1. The calculation of utilized budget of INFER is unclear. Some components in INFER seem to be consist of several vectors which have same size with original natural instance. However, in the manuscript, there is explicit description of utilized budget of INFER. Therefore, it is essential to clearly specify how the budget used by INFER is calculated and how it is set to ensure a fair comparison with prior studies.

2. There is insufficient baseline used for performance comparison. Although many dataset distillation methods are recently proposed to enhance the performance and budget efficiency, this work only compares INFER to some (possibly outdated) studies. Additionally, comparing INFER’s performance with recent methods raised doubts about its effectiveness. For example, DATM [1], which also address the feature redundancy as the manuscript said (Line 238-240), achieves higher performance than INFER in many settings: 47.2% (IPC=10), 57.5% (IPC=100) for CIFAR-100 and 31.1% (IPC=10). 39.7% (IPC=50) for Tiny-ImageNet.

3. One of the component in INFER, $P^k$, is consist of real instances. Storing real instances can be a drawback in tasks where privacy issues are crucial (e.g., continual learning). Continual learning is one of the primary applications of dataset distillation.

[1] Towards lossless dataset distillation via difficulty-aligned trajectory matching

questions:
1. How to calculate the utilized budget of INFER? According to Algorithm 1, it seems that $P^k$, $U^k$, and $Y^k$ are stored in budget after training. $P^k$ consists of instances equal to the number of classes $C$ (Line 263), and $U^k$ consists of $M$ instances (Line 269). Since there are a total of $K=\text{IPC}$ (Line 261) sets of $P^k$ and $U^k$, the total number of instances in $P$ and $U$ amounts to $\text{IPC} \times (C + M)$, which already exceeds the budget of $\text{IPC}×C$ used by conventional methods. If my understanding is incorrect, please clarify the budget utilized by INFER.

2. Is there a reason for learning $U^k$ independently for each $k$? Learning $U^k$ independently for each $k$ aims to capture the inter-class features of $P^k$. This approach seems to capture only local inter-class features, which may be suboptimal. Introducing a single $U$ could potentially be beneficial for obtaining global inter-class features. Are there experimental results using a single $U$?

3. In Algorithm 1, all $u_j \in U^k$ are initialized with zero vector and independently optimized with same ingredient. This suggests a risk that all instances of $u_j$ may converge to the same optimal value. I am curious about how different values of the optimized $u_j$ actually are. Additionally, I am interested in the rationale for initializing with a zero vector rather than a random noise vector, as well as the resulting performance when using the latter.

4. In general, data factorization methods generate more instance than IPC under same budget. Given this characteristic, feature duplication may present a greater limitation in data factorization methods. Does feature duplication occur in data factorization as well? If it does, is there an improvement in performance when combining INFER with data factorization methodologies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730656647099

thread_type:
official_review

<thread_object>
creator: reviewer_rt9a

<content>
summary:
This paper first identifies two limitations of the current dataset distillation methods: inefficient utilization of the distillation budget and oversight of inter-class features. Based on this, this paper proposes to leverage a Universal Feature Compensator to enhance feature integration across classes.
Experiments demonstrate that the proposed INFER improves the storage efficiency and effectiveness of dataset distillation.

strengths:
1. It analyses the limitations of the current optimization-based dataset distillation methods.
2. The proposed method can significantly reduce the storage.
3. The paper is well-written.

weaknesses:
1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.

2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when 
IPC=50 using ResNet-101. Why is this the case?

3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?

4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. 

[1] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
1. The authors claim that the existing dataset distillation methods focus on optimizing synthetic data exclusively for a pre-assigned one-hot label. However, RDED [1] introduces a non-optimization method that changes this paradigm. It is recommended that the authors analyze and compare their method with RDED.

2. In Table 1, lines 416-417, the authors claim that static labels perform better on small networks, while dynamic labels excel on large networks. However, in Table 2, static labels sometimes outperform dynamic labels on large networks and datasets, particularly when 
IPC=50 using ResNet-101. Why is this the case?

3. Additionally, in Table 4, the performance comparison between static and dynamic labels is inconsistent. Sometimes static labels are superior, other times dynamic labels are. What factors determine the choice between static and dynamic labels in real-world applications?

4. Analyzing computational cost is crucial in dataset distillation. It would be beneficial to compare both computation cost and peak memory usage against existing methods. 

[1] Sun, Peng, et al. "On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733197041752

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of our responses during the discussion period

comment:
Dear AC and reviewers,

We would like to sincerely thank you for dedicating your valuable time and effort to providing insightful feedback and suggestions that have greatly contributed to enhancing the quality of our work.

We have carefully addressed all comments and suggestions raised by the Reviewers. The additional experiments, analyses, and discussions will be incorporated into the revised manuscript and appendix.

Below, we provide a summary of the key points discussed during the rebuttal:


- Comparison with SOTAs.
> We conducted a comprehensive comparison with SOTA methods, including RDED, CDA, LPLD, DATM, IID, and G-VBSM, and highlighted our approach’s notable advantages in both efficiency and effectiveness under fair evaluation settings. (**RT9a, hymx, enyK, aHjp**)

- Additional experiments.
> We have conducted extensive experiments to validate the cross-architecture generalization capability of our approach (**enyK**).
>  We also performed ablation experiments to analyze the detailed implementation of the initialization and optimization processes for the cross-feature compensator (**hymx**).
>  We further validated the strong efficiency of our method by applying our static labeling strategy to other SOTA methods, including SRe2L, CDA, and RDED (**RT9a, hymx, aHjp**).

- Computational cost.
> We provided a detailed comparison of computational costs with other SOTA methods. (**RT9a, aHjp**)

- Budget allocation.
> We analyzed the utilization of the distillation budget and detailed the adjustment of $P$ to ensure fairness in comparisons. (**hymx, enyK, aHjp**)

- Compression ratio calculation.
> We provided a detailed explanation of the compression ratio calculation and demonstrate the advantages of our method in compressing soft labels. (**aHjp**)

- Batch size ablation study.
> We conducted experiments to verify that our method, INFER, outperforms the baseline methods SRe2L, CDA, G-VBSM, and LPLD using the same batch sizes of 32 and 128, as requested by **RT9a**.

- Further discussion.
> We also discussed potential applications in privacy preservation, the feasibility of combining with factorization methods, current limitations, and future trends in dataset distillation. (**hymx**)


As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. Your feedback would be greatly appreciated, and we would be delighted to engage in further discussions if needed.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a new framework for dataset distillation that can better utilise the distillation budget and consider inter-class feature distributions. The idea is to move from the traditional class-specific data-label framework and develop a one-instance-for-all-class approach. The key components consist of the universal feature compensator and the enhancement of synthetic data with inter-class augmentation. Experimental study is conducted on benchmark datasets to verify the efficacy of the proposed method. 

Reviewers comment that the proposed method can significantly reduce the storage, the paper is well written, the work is technically sound with thorough evaluations, and the proposed inter-class feature compensator is novel, and so on. Meanwhile, the reviewers raise concerns related to the comparison with more methods, the evaluation of computational cost, the fairness of experimental study, the privacy issue related to the use of real instances, the effectiveness under high compression ratios, the cross-architecture evaluation, the generality of the proposed idea, and the clarification of details. The authors provide a rebuttal of high quality. The responses to the comments of three reviewers address most or all of the concerns and lead to the increase of score. The authors’ responses during the multiple rounds of discussion with Reviewer RT9a are effective. During the further discussion between reviewers, AC highlights two concerns for them to check, including 1) the fairness of the experimental comparison (raised primarily by Reviewer RT9a), and 2) the effectiveness and consistency of the proposed idea under high compression ratios (raised primarily by Reviewer hymx). Reviewer aHjp replies that the two concerns have been well addressed during the rebuttal period. The final ratings are 3 (Reviewer RT9a), 5 (Reviewer hymx), 6 (Reviewer enyK) and 8 (Reviewer aHjp) .   

AC has carefully reviewed the submission, along with the reviews, the rebuttals, the discussions, and the message provided by the authors. AC concurs with the reviewers regarding the novelty and efficiency of the proposed method and are satisfied with the authors' rebuttal in addressing the major concerns. Taking all the factors into account, AC recommends to accept this work and suggests the authors to further strengthen this work by properly incorporating the responses in the rebuttal.