Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information

Authors: Xinhao Zhong, Bin Chen, Hao Fang, Xulin Gu, Shu-Tao Xia, EN-HUI YANG
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: dataset distillation, conditional mutual information

Abstract:
Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks on large datasets, by creating a smaller synthetic dataset that has similar performance to that of the full real dataset. However, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metrics measuring feature similarity, e,g., distribution matching (DM). In this work, we introduce conditional mutual information (CMI) to assess the class-aware complexity of a dataset and propose a novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the class-aware complexity of the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as a general regularization method to existing DD methods and improve the performance and training efficiency.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730534500687

thread_type:
official_review

<thread_object>
creator: reviewer_zh5i

<content>
summary:
This paper proposes a plug-and-play method, termed CMI, designed to enhance existing DD techniques by minimizing conditional mutual information. By applying CMI, the distilled data is concentrated more effectively around the center of each class, thereby improving generalizability.

strengths:
- The logic is clear.
- The experiments are comprehensive.
- The review of related works is thorough.
- The proposed method is theoretically sound.

weaknesses:
- The core ideas, methodology, and formulations in this paper draw substantially from the approach proposed in [1].
- If  \hat{Z}  contains excessive confusing or uninformative information related to  S , then  H(S | \hat{Z}, Y)  will not be reduced; rather, it could remain the same or even increase. This is because conditional entropy reflects the remaining uncertainty in  S  after observing both  \hat{Z}  and  Y . When  \hat{Z}  is noisy or irrelevant for predicting  S , it does not help in reducing this uncertainty.
- Line 213 states that “minimizing the class-aware CMI reduces the uncertainty brought to  \hat{Z}  conditioned on  S ,” which should be  “minimizing the class-aware CMI reduces the uncertainty brought to S conditioned on  \hat{Z}”.
- The authors’ derivation of Equation 6 lacks an explicit explanation, making it challenging to fully understand the transition from previous formulations.
- Works like [2] and [3], which also target improvements in dataset distillation, are not adequately considered. 
- Equation 3 requires summing over all synthetic instances within class  y , how the authors adapt this approach to instance-based distillation methods like SRe2L. 

[1] Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information

[2]TOWARDS LOSSLESS DATASET DISTILLATION VIA DIFFICULTY-ALIGNED TRAJECTORY MATCHING

[3]Prioritize Alignment in Dataset Distillation

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607817383

thread_type:
official_review

<thread_object>
creator: reviewer_i6mf

<content>
summary:
This paper introduces a novel regularization method for dataset distillation (DD) by minimizing both the distillation loss and Conditional Mutual Information (CMI) of synthetic datasets. It uses an efficient CMI estimation method to measure class-aware properties and combines CMI with existing DD techniques. Experiments show that the proposed CMI-enhanced loss significantly outperforms state-of-the-art methods, improving performance by up to 5.5%. The method can be used as a plug-and-play module for all existing DD methods with diverse optimization objectives.

strengths:
The strengths of this paper lie in its comprehensive experimentation across diverse datasets and network architectures, which effectively demonstrates the versatility and robustness of the proposed method. Furthermore, the method's ability to be integrated as a plug-and-play module into existing dataset distillation techniques, regardless of their optimization objectives, showcases its innovation and flexibility, making it a significant contribution to the field.

weaknesses:
The paper lacks a clear discussion of the limitations of the proposed method. Furthermore, the authors should consider using more intuitive explanations, visual aids, and pseudocode to help readers better understand the technical details of the method.

questions:
Can you discuss any potential limitations of your proposed method and suggest directions for future work to address these limitations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722258100

thread_type:
official_review

<thread_object>
creator: reviewer_kma8

<content>
summary:
This paper proposes a Conditional Mutual Information (CMI) method as a plug-and-play loss function to enhance the performance of dataset distillation methods. Experiments conducted on multiple baseline methods demonstrate the effectiveness of the CMI loss.

strengths:
1.	The proposed CMI method is a relatively simple yet effective approach that is plug-and-play in nature. It has demonstrated its effectiveness across multiple baseline methods.
2.	The motivation behind the method proposed in the paper is solid and is supported by a certain theoretical foundation.
3.	The experiments in the paper are comprehensive, conducted across various scales of datasets.

weaknesses:
1.	There are now newer and more powerful methods available, such as "Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching" (ICLR 2024). The authors could consider experimenting with their proposed method on these methods.
2.	The description of the method in the paper could be clearer, particularly regarding the explanation of the formula symbols, to better emphasize the key points of the approach. Currently, it appears somewhat ambiguous.
3.	In my view, using mutual information or KL divergence is not a particularly novel approach, as it has been employed in many works across various fields.

questions:
Please kindly refer to the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730730857513

thread_type:
official_review

<thread_object>
creator: reviewer_omqb

<content>
summary:
This paper proposes a new approach for dataset distillation by introducing a class-aware conditional mutual information (CMI) metric to address challenges in creating compact, representative synthetic datasets. Traditional dataset distillation methods often compress feature similarity without considering class-specific complexity, making it hard for models to generalize across different classes. This work leverages CMI as a regularization constraint, optimizes synthetic datasets and improves training efficiency as well as model performance.

strengths:
1.	The idea of using CMI in dataset distillation to address the inherent class-aware complexity issue is interesting.
2.	The experiments are conducted based on multiple datasets and various model architectures, providing solid evidence for the method's effectiveness. 
3.	The proposed method CMI is a versatile, "plug-and-play" regularization component that can be applied to numerous dataset distillation methods, such as DSA, MTT, and IDC. This flexibility allows the approach to generalize across different scenarios and highlights its robustness.
4.	By controlling the complexity of the synthetic dataset, the CMI-enhanced loss achieves faster convergence and reduces the number of required training iterations, which is particularly beneficial for large-scale datasets and resource-intensive models.

weaknesses:
1.	While the paper demonstrates the CMI constraint’s benefits clearly, this method also introduces additional computation overhead, especially when dealing with high-resolution datasets. Although the authors briefly mention several strategies for mitigating this cost (e.g., reducing CMI calculations frequency), a more thorough discussion on balancing cost and performance might strengthen the practical feasibility.
2.	Although empirical evidence is strong, the theoretical basis for CMI as a regularization term could be expanded. Specifically, further details on how CMI inherently captures class complexity or why it is preferable over alternative complexity measures would provide deeper insight.
3.	While the experiments on Tiny-ImageNet and ImageNet-1K are promising, it remains unclear how the proposed method scales with even larger datasets or more complex models, such as those used in real-world applications with hundreds of classes. Additional experiments in such contexts would further show the robustness of this method.

questions:
1.	The paper is well-organized and clearly presents the methodology, results, and analyses. Figures and tables are effectively used to convey improvements and insights. However, further explanation of certain key terms, such as "empirical CMI," might enhance accessibility for readers unfamiliar with the topic.
2.	The ablation studies conducted to assess the influence of the weighting parameter on the CMI constraint are informative. Still, a broader exploration of other hyperparameters affecting CMI estimation, such as the dimensionality of feature space and network depth, could reveal potential optimizations.
3.	The potential of CMI for real-world applications, such as federated learning or privacy-preserving tasks, is not discussed. Given the emphasis on dataset distillation's applications in these areas, an exploration of how CMI might support these domains would align well with the broader goals of dataset distillation research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732158428428

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely express our gratitude for dedicating your valuable time to providing insightful suggestions that can enhance our paper. Your praise regarding our methodology, experiments, and contributions have greatly encouraged and motivated us. Our detailed responses to all of your conerns are presented below.

Encouragingly, reviewers praise the insight of using CMI in dataset distillation ( `R#oMQb` , `R#kma8` ), solid theoretical foundation ( `R#kma8` , `R#zh5i` ), robustness ( `R#oMQb` , `R#kma8` ) and comprehensive experiments ( `R#oMQb` , `R#kma8`, `R#i6mf`, `R#zh5i` ) of our work. To better address the reviewers' concerns, we provide point-by-point responses to each reviewer below. If there are any other questions, please don't hesitate to discuss them with us.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732358694762

thread_type:
official_comment

<thread_object>
creator: area_chair_cqbd

<content>
title:
Please review author response

comment:
Dear reviewer,

Could you review the author response and let them know if you are satisfied with it or if you have any additional questions?

Kind regards,

Your AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732542261617

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thanks all the reviewers once again for dedicating your valuable time to reviewing our paper and providing constructive comments!

As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. Your feedback would be greatly appreciated, and we would be delighted to engage in further discussions if needed.

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733141091912

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

Thanks again for providing the constructive review. We have tried our best to address your concerns point by point so that you can have a better understanding of our work.

However, since the discussion period is going to end in one day, we really want to hear from you and see if we have addressed your concerns. Please participate in the discussion because your feedback is important for us to improve our work. If all your queries have been addressed, we kindly ask you to consider raising your rating. If you still have other questions, we are more than willing to explain in detail.

Looking forward to your reply.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work focuses on the complexity of the synthetic dataset in dataset distillation. It proposes to apply conditional mutual information (CMI) to assess this complexity and uses this criterion as a regulariser of the existing dataset distillation methods. Experimental study shows that the proposed regulariser can be used as a plug-and-play component to work with various distillation losses to achieve improved performance and faster convergence. 

Reviewers comment that the proposed method is interesting, versatile and effective; the experiments are comprehensive and solid; this work is innovative and makes significant contribution; the review is thorough; and it is technically sound. Meanwhile, the reviewers raise the issues related to the extra computational overhead, the theoretical basis of using CMI as a regulariser, the scalability with respect to dataset size and model complexity, experimenting with new methods, the novelty of this work, the discussion of limitation, the difference from an existing work on CMI, and the lack of explanation.  

The authors provide a response to each of the raised issues by further clarification, additional experiments, and pseudo code. The rebuttal is overall detailed and effective. Reviewer oMQb explicitly replies that the rebuttal addresses most of the concerns they raised. However, Reviewer Zh5i still has concerns related to the difference of this work from an existing work on CMI and therefore the adequacy of the novelty. The final ratings are 6, 6, 6 and 3 (Reviewer Zh5i). 

AC carefully reviews the submission, the comments, the rebuttals, the discussion, and the message from the authors. AC generally concurs with the reviewers' comments on this work. In addition, although this work adopts the CMI criterion from the literature and therefore does not have significant contribution in this regard, it does provide new insights on the complexity of the synthetic dataset and innovatively utilises CMI to consider this information in the optimisation of dataset distillation. Also, experimental study verifies the effectiveness and the generality of this work. In the follow-up discussion between reviewers and AC, Reviewer Zh5i indicates that they will not oppose acceptance if the majority of reviewers strongly advocate for it.  

Taking all the factors into account, AC would like to recommend this work for acceptance. Meanwhile, the authors are strongly suggested to further improve the clarity of this work, provide more explanation, and explicitly list the differences from the work raised in the comments of Reviewer Zh5i.