Going Beyond Feature Similarity: Effective Dataset distillation based on Class-aware Conditional Mutual Information
Authors: Xinhao Zhong
Bin Chen
Hao Fang
Xulin Gu
Shu-Tao Xia
EN-HUI YANG
Venue: ICLR 2025 Poster (2025)
Keywords: dataset distillation
conditional mutual information
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: introducing conditional mutual information to enhance the performance and the efficiency of dataset distillation

Abstract: Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks on large datasets, by creating a smaller synthetic dataset that has similar performance to that of the full real dataset. However, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metrics measuring feature similarity, e,g., distribution matching (DM). In this work, we introduce conditional mutual information (CMI) to assess the class-aware complexity of a dataset and propose a novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the class-aware complexity of the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as a general regularization method to existing DD methods and improve the performance and training efficiency.

Published as a conference paper at ICLR 2025
GOING BEYOND FEATURE SIMILARITY: EFFECTIVE DATASET DISTILLATION BASED ON CLASS- CON-
DITIONAL MUTUAL INFORMATION
Xinhao Zhong' Bin Chen 2.2* Hao Fang* Xulin Gu Shu-Tao Xia
En-Hui Yang*
1 Harbin Institute of Technology, Shenzhen Peng Cheng Laboratory Tsinghua Shenzhen International Graduate School, Tsinghua University xh0212136gmail com, chenbin2021ehit.com edu. en, 21011 210110720estu.i hit edu. en, chyang@uwatorloo.com ca, xiast@sz tsinghua edu. en;
UUniversity of Waterloo
tsinghua edu.cn
ABSTRACT
Dataset distillation (DD) aims to minimize the time and memory consumption needed for training deep neural networks large datasets, by creating smaller synthetic dataset that has similar performance to that of the full real dataset. How- ever, current dataset distillation methods often result in synthetic datasets that are excessively difficult for networks to learn from, due to the compression of a substantial amount of information from the original data through metries measuring feature similarity, c.g., distribution matching (DM), In this work, we introduce conditional mutual information (CMI) assess the class-aware complexity of a dataset and propose novel method by minimizing CMI. Specifically, we minimize the distillation loss while constraining the Hass-awaree complexity the synthetic dataset by minimizing its empirical CMI from the feature space of pre-trained networks, simultaneously. Conducting on a thorough set of experiments, we show that our method can serve as general regularization method to existing DD meth- ods and improve the performance and training efficiency. Our code is available at https: / giggiittuun thub www www.:hwwwwoooommgnonceeececeeeee
INTRODUCTION
Dataset distillation (DD) (Wang et al. 2018 Nguyen et al. 2021a b Zhou et al. 2022) has received tremendous attention from both academia and industry in recent years highly effective data compression technique, and has been deployed in different fields, ranging from continual learning (Gu et al.) 2024b), federated learning (Wang et al. 2024b) to 2022). Dataset distillation aims to generate a smaller synthetic dataset from a large real dataset, which encapsulates higher level of concentration of task-specific information than the large real counterpart and which the deep neural networks (DNNs) trained on can attain performance comparable to that of those trained large real datasets, but with benefits of significant savings in both training time and memory consumption.
(Dong et al.)
Existing dataset distillation methods optimize the smaller synthetic dataset through stochastic gra- dient descent by matching gradient (Zhao et al. (02I). feature (Zhao & Bilen 202 or statistic information Yin al. (2024), aiming to compress task-relevant information sensitive the backbone model into the generated images. Building on this, some recent methods propose using feature clustering (Deng et al. 2024), maximizing mutual information (Shang et al., 2024), or progressive optimization (Chen et al. 20244 to enhance performance. However, these plug-and-play methods focus on improving the alignment of the synthetic dataset with the real dataset in specific informa- tion space, e.g., distribution matching, while neglecting the properties of different classes inherent in the synthetic dataset. This limitation restricts their applicability to specific distillation methods and results in relatively modest performance improvements.
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
A promising group of methods (Paul et al. 2021 Zheng et al. 2022) in sew-shot learning, simpler datasets are often more beneficial for model training. This suggests that while synthetic datasets aim to distillate all the information from the real dataset during optimization, the complexity of distilled information can make it more challenging for models to learn, and the overly complex samples often introduce biases during model training. This limitation has been further supported by recent studies (He et al., [2024] Yang et al. (024)) which show harder samples can not support the entire training and often lead to performance drops and fluctuations.
dataset evaluation indicate that
Feature Space
Probability Space
Feature Space
Probability Space
(a) Synthetic dataset w.o. CMI constraint.
(b) Synthetic datasct CMI constraint.
Figure 1: Visualization of the synthetic dataset generated by DM with (a) high CMI value, and (b)
low CMI value.
To achieve a more granular measurement of the information involved in the dataset distillation process, especially regarding the varying difficulty levels across different classes, this paper introduces conditional mutual information (CMI) from information theory (Yang et al. [023 Zhuang et al. (0255) as a class- aware complexity metric for measuring grainge condensed information. Specifically, given a pre-trained neural network fø- (-) parameterized by 0", we define the CMI L(S; Ý (Y), where S, Y and Y are three random variables representing the input synthetic dataset, the ground truth label, and the output of fe- (-), respectively. Building on this, (S; Y YY) quantifies the amount of information about S contained in Y given the class condition Y, indirectly measuring the elass-aware complexity of the synthetic dataset by demonstrating how the output of pre-trained network influenced by the synthetic dataset. Through empirical verification, we demonstrate that minimizing CMI in the feature space of fe- (-) invariably leads to the distilled data becoming more focused around the center of each class, thereby enhancing the generalization of the distilled data across different network architectures as shown in Figure
Based this intuition, propose novel regularization method to existing DD methods by simultaneously minimizing the distillation loss of the synthetic datasets and its CMI. First, we employ an efficient CMI estimation method to measuring the class-aware inherent properties of the synthetic dataset. Under the guidance of this metric, we treat CMI as a universal regularization method and combine it with existing dataset distillation techniques with diverse optimization objectives. Experiments demonstrate that the CMI enhanced losses significantly outperform various state-of- the-art methods in most cases. Even huge improvements of over 5% can be obtained under some conditions.
In summary, the contributions of the paper are as follows:
We provide an insight into the properties of different classes inherent in the synthetic dataset and point out that the generalization of the distilled data can be improved by optimizing the class-aware complexity of synthetic dataset quantified via CMI empirically and theoretically.
Building on this perspective, we propose the CMI enhanced loss that simultaneously mini- mizes the distillation loss and CMI of the synthetic dataset in the feature space. This enables the class-aware complexity of synthetic dataset could be efficiently reduced, while distilled data becoming more focused around their class centers.
Experimental results show that our method can effectively improve the performance of existing dataset distillation methods by up 5.5%. Importantly, our method can be deployed as an plug-and module for all the existing DD methods with different optimization objectives.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Dataset Distillation (DD) (Wang et al. 2018) aims to generate a synthetic dataset that, when used for training, can achieve performance similar to training on the full real dataset. To achieve this, DD adopted a meta-learning process comprising two nested loops. The method minimizes the loss function of the synthetic dataset in the outer loop, using model trained on the synthetic dataset in the inner loop. To address the issuc of unrolled computational graphs, recent studies propose matching proxy information. DC (Zhao et al.) (02T) and DCC (Lee et al. 2022) minimize the distance between the gradients of the synthetic and original data on the network being trained with synthetic data. DM
(Zhao & Bilen 2021a) and CAFE Wang al. (2022) matche the extracted features between the
synthetic dataset and the real dataset. MTT (Cazenavette et al. 2022) and TESLA (Cui et al. 2023) match the training trajectories of the network parameters obtained from training on the complete and synthetic datasets, respectively. SReºI. YYin et al., 2024) successfully scales up dataset distillation to larger datasets and deeper network architectures by utilizing model inversion loss and matching the
statisties natch-normalization layers.
Subsequently, various techniques have been proposed to enhance the performance of the synthetic dataset. DSA (Zhao & Bilen 202 applied differentiable siamese augmentations to both the real and synthetic data while training IDC (Kim et al.) 2022) proposed a multi formulation framework to generate more augmented examples under the same memory budget, successfully achieved more efficient utilization of the limited pixel space by reducing the resolution of generated images. Since then, incorporating differentiable data augmentation and multi formulation have been adopted by almost all subsequent studies. DREAM (Liu et al. 2023) utilized clustering to select more representative original images. MIM4DD (Shang et al.
2024 proposed leveraging a contrastive
learning framework to maximize the mutual information between the synthetic dataset and the real dataset. Several methods have been improved based on the characteristics of the matching objectives. PDD (Chen et al. 2024) and SeqMatch (Du et al. 20224) propose multi-stage distillation based on the characteristics of network parameter variations during the stochastic gradient descent process. IID (Deng al. 2024) further aligned the inter-class and intra-class relationships when using feature
matching.
Existing methods primarily focus on achieving more precise estimation and utilization of surrogate information or compressing additional beneficial information into the synthetic dataset, while neglect- ing the impact of synthetic dataset complexity. In contrast, we show that properly constraining the complexity of the synthetic dataset yields superior performance.
3 NOTATION AND PRELIMINARIES
3.1 NOTATION
For a positive integer K. we denote [K] - [K. Let P([C)) be the set of all possible probability distributions over the C-dimensional probability simplex indexed by [C]. For any two probability distributions PI, P2 E P((CI), the cross-entropy between Pi and P2 is defined as HPP,, Pb) Σ (i) log (2))). and the Hellback-Leibler (KL) divergence between P1 and (1(0) Pa is defined as D(Pi)|P)) = Σ (i) log Pz()
For random variable X, let Px denote its probability distribution, and [-] represent the expected value with respect to X. For two random variables X and Y, let P(x,y) denote the joint distribution of X and Y. The mutual information between the random variables X and Y is defined as [(X; Y) H(X)= H(X Y), where H(X) is the entropy of X. The conditional mutual information of X and Z given a third random variable Y is defined as [[x; Z Y) = H(X (Y)-H(X Y,Y).
Given deep neural network fo(-) parameterized by 0. we can view it as mapping from E Rd to Px. When there is no ambiguity, we denote Px as Px.s. and Px(y) as P(y x, 0) for any y E [C]. Let Ý denote the output predicted by the DNN, with the probability Px (X) in response to the input
x. Specifically, for any input x E IRd, we have P(ý 9 X = x) (x) P(ù 0) Since
y + X y forms Markov chain, we can infer that Y and ÿ are independent conditioned on X.
3

---

Published as a conference paper at ICLR 2025
3.2 DATASET DISTILLATION
Given large-scale dataset T = [xx, );)) dataset distillation aims to generate a synthetic dataset S ((s y:) ):)}]=1 that retains much class-relevant information as possible, where |S] < 17. The key motivation for dataset distillation is to create an informative S that allows models to achieve performance within an acceptable deviation from those trained on T. This can be formulated as:
sup
, (x), y) - 1(09. (x), v)1 S
(x.w)ET
where U(:,)) represents the loss function, o the parameter of the neural network o trained on T. and similar definition applies to As:
OT arg m (Xp)TT (I(0e(x),
To transform this metric into a computable optimization method, previous optimization- ap- proaches utilize various d(-) to extract informative guidance from and S in a specific information space while alternately optimizing S, formulated as:
S arg min M(o(S),
where .(...) denotes various matching metrics, such neural network gradients (Zhao et al. 2021 extracted features (Zhao & Bilen 2021a), and training trajectories (Cazenavette et al. 2022).
METHODOLOGY
4.1
CLARS-AWARE CMI AS A VALID MEASURE OF SYNTHETIC DATASET
Prior data evaluation studies (Paul et al. 2021 Zheng et al. 2022) indicate that when [S] is small, the model's ability to learn complex representations is constrained. To enable the model to effectively capture dominant patterns across different classes in a limited dataset, the samples in S must represent the most prevalent patterns. By focusing on learning from these representative samples, the model can capture essential discriminative information across classes. Additionally, easier samples of certain classes in S tend to yield lower loss function values, facilitating faster convergence and enhancing performance within limited number of training iterations. Therefore, it becomes essential to introduce measure for the class-aware complexity of the synthetic dataset S itself.
Recognizing that feature representation contains more effective information than probabilistic repre- sentation Y (Gou al. 2021). we impose constraints on the synthetic dataset in the feature space. To be specific, for a pre- trained network so- (.) trained on the original large-scale dataset T, we can decompose it as fø- (-) l0z (.) o he; (-) (0" {0], ())), where he; (-) s +> z denotes feature extractor that maps an input sample E S C Rd into an M-dimensional feature vector z E 2 C RM (e.g.. the 512-dimensions penultimate features of ResNet-18), and log (-) z ÿ is parametric classifier that takes as input and produces a class prediction ÿ.
For given input s S, the output feature z is a deterministic feature vector. We apply the softmax function to the feature vector z = ( 2e,,,?,...,, 2d) for an input sample E S, which maps it to one-dimensional random variable 2, whose probability distribution Pa [2) is random point in the probability simplex P([M]) indexed by [M], where
exp(2*)
Pal = P(2 =i (|s)
for any i E [M] = (1,2,...... M}.
M E/=n exp(zi)
Based on the above analysis, we can see that Y + S + 2 forms a Markov chain. To quantify the class-aware complexity of the synthetic dataset, we introduce the class-aware conditional mutual information (CMI) I(S; 2 Y) as a novel measure of the complexity of the synthetic dataset and further explain its validity below.
Note that the non- non-linear relationship between the input S and the output 2 can be quantified by the conditional mutual information T(S; 2 Y). This can also be expressed as T(S; 2 Y) H(S)
4

---

Published as a conference paper at ICLR 2025
Y)- 2,Y) Y), representing the difference between the uncertainty of S given both 2 and Y and that of S given Y. When a relatively diverse and large dataset (e.g., 7 Px) is used as the input to fo- the corresponding output follows a more certain probability distribution produced by fg-(-). (-). In contrast, since S is more challenging for randomly initialized networks learn, its output 2 often contains excessive confused information related it, leading significant reduction in (S) Y. Thus, minimizing the Mlassaware CMI value constraints the uncertainty brought to 2 with S as the imput of fe-(-), preventing S from overly complicating the prediction process of for (-). Additionally, minimizing CMI leads reduction in H(S Y) implicitly, which represents the complexity of 5 conditioned on Y alone.
4.2 ESTIMATING THE CLASS-AWARE CMI FOR SYNTHETIC DATASET
Given Y = y and y E [C]. the input S is conditionally distributed according (-ly and then mapped into Ps e P[[M]). The conditional distribution P' i.., the centroid of this cluster, is exactly the average of input distribution Ps with respect to Ps|r (-ly), which is formulated as:
12 PsPs = y]-
(5)
Similar to the calculation in Ye et al. 2024). we employ the Kullback-Leebler (KI.) divergence [[Ps] 2/)) quantify the distance between Ps and the conditional distribution 2zw""
Then derive the class-aware conditional mutual information as follows:
P( =
I(S; 2 Y =)
Σ PS
=s
Σ P(2 = x
(6)
PgW 12 = Y =v)
SES
[[~
Ps|
Esiy
Ps In
y-
P2 (2 = y m)
Esjy (P((S1122))
Y 1Y=
(8)
Averaging I(; 2ly) with respect to the distribution (v) of Y, we can obtain the conditional
mutual information between 8 and given Y as follow:
CMI(S) e I(S; Y) Y)
Σ (S) 1)).
VE(C)
In practice, the joint distribution P(s, v) of (S may be unknown. To compute the CMI(S) this case, we approximate P(s, y) by the empirical distribution of synthetic dataset S, (s1 v), (S2, v), v).... (Sn, y)} for any y E [C]. Then the CMIemp can be calculated as:
CMlemp (S)
Σ SEE,,
KL (10mp)))mp))
(10)
S
WE C1
where
Pemp
Pay. for y E [C].
Syy s,ES,
4.3 DATASET DISTILLATION WITH CMI ENHANCED LOss
According to the above calculation of CMI, we propose the CMI enhanced Loss C. Overall, includes two parts:
= CoD + A CMIemp
(11)
The first term LDD represents any loss function in previous DD methods, e.g., DM, DSA and MTT, > is a weighting hyperparameter.
The proposed CMI enhanced loss is universal since it provides plug-and and-ala solution minimize the class cansscooniiionaa complexity of the synthetic dataset, making adaptable to all previous dataset distillation methods that focus better aligning the synthetic dataset with the real one just based on
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparative analysis of dataset distillation methods. A: the improvement magnitude of
CMI as a plugin to the base distillation methods. Ratio (%) the proportion of condensed images relative the number entire training set. Whole Dataset: the accuracy of training on the entire
original dataset. The best results are highlighted.
Method
SVHN
CIFARIO
CIFAR100
50
50
Ratio (%)
0.14
0.7
0.2
1
10
Random
35.1+4.1
70,9+-09
26.0+1.2
43.4+100
14.6+0.5
30.0+0.4
Herding Welling 2009
50.5=3.3
72.6:0.8
31.620.7
40.410.6
17.3+0.3
33.7
K-Center Seher & Savarese
2017
1..0=1.3
20.14:.4
14.7=10.9
27.0
17.3+0.2
30.5:0.3
Forgetting Toneva 2019
16.8=1.2
27.21:.5
23.32110
23.321.1
15.1
30.520.4
MTT (Cazenavette et al.
2022
79.9:2.11
87.740.3
65.3-204
716:2022
39,720044
47.7
MIM4DD (Shang et 2024
66.400.2
714=0.3
41,5+002
SeqMatch (Du
80.210.6
88,510.2
66.210.6
74.400.5
41,9:0.5
5121003
MTT+CMI
80.8=0.2
88,8+0011
66,7=20.3
72.4=0.3
41,9:0,4
48.8+0.2
(0.91)
(1.11)
(1.41)
(0.81)
(2,21)
(1.11)
DM (Zhao Bilen 2023
72.8+0.3
82.6++0.5
48.9+
63.0+044
29.7+0.3
43.6+0.4
IID- IID-DM (Deng al. 2024
75.7=0.3
85,3+:0.2
55.1=0.1
65.1=0.2
32.2+00.5
43,6+0.3
DM+CMI
77.9=0.4
84,9+10.4
52,9=0.3
65,8=0.3
32,5+0.4
44,9
(5.11)
(2.31)
(4.01)
(2.81)
(2.81)
(1.31)
IDM (Zha al. 2023
$1.0=0.1
84.1:10.1
58.6+0.1
67.5:0.1
45.140.11
50.0:0.2
IID IID-IDM (Deng et al. 2024
82.1=2033
85.1+:0,5
59,9=0.2
69.0=1.33
45.7+0.44
$1.3+0.4
IM+CMT
8..3+0.2
88,9+-0.2
62.2+0.3
7,.3+0.2
47,2+0.4
51,9+0.3
(3.31)
(4.81)
(3.61)
(3.81)
(2.11)
(1.91)
DSA (Zhao al. 2021 DMA
79.2:10.5
84.4:10.4
52,1
60.6:1055
32,340.3
42,810.4
80.5+0.2
85.5+-0.3
54.7+0.4
6..1+0.1
35,0+0.4
45.9+
(1.31)
(1.11)
(2.61)
(5.5t)
(2.71)
(3.11)
IDC (Kim et al. 2022
87.5:10.3
90.11:0.1
67.510.5
74,5:1.11
45.11:044
DREAM (La el al. 2023
87.9+0.4
90.5++00.1
69.4+0.4
74.8+0.1
46.8+-0.7
52,6+0.4
PDD (Chen 2024
67.9-0.2
76.5-1044
53.1104
IDC+CMI
88.5
92.2-+0.1
70.0=0.3
76.6=0.2
46.6+-0.3
5..8
(1.07)
(2.11)
(2.51)
(2.11)
(1.51)
Whole Dataset
95,4-10.2
84.8
56.2410.3
feature similarity. In addition, minimizing CMI complexity constraint poses certain challenges, as the optimization target is general synthetic dataset. To address this issue, we compute CMI the feature space rather than the probability space demonstrated in (Ye et al. 2024), thereby establishing a stronger constraint. Addistionally, during the optimization process, we randomly sample pre-trained networks with varying initializations compute CMI, resulting in dataset-specific optimization process that is independent of the network parameters.
5 EXPERIMENTS
In this section, we conduct comprehensive experiments to evaluate our proposed method across various datasets and network architectures. We begin by detailing the implementation, followed by an analysis of the performance improvements achieved with different distillation techniques. Finally, assess the effectiveness of the the CMI enhanced loss through series of ablation studies.
5.1 EXPERIMENTAL SETTINGS
Datasets. We conduct our experiments on five standard datasets with varying scales and resolutions:
SVHN(Sermanet et al. 2012) CIFAR-10/11 (Krizhevsky) 2009), Tiny-mmageNet (Le & Yang 2015) and InageNett1K((emmm around the world. CIFAR- 10 and CIFAR- 100 consist of 50, 000 training images, with 10 and 100 classes, respectively. The image size for CIFAR is 32 32. For larger datasets. -mageNet contains 100, 000 training images from 200 categories, with the image size of 64 x 64. ImageNet-1K consists of over 200, 000 training images with various resolutions from 1, 000 classes.
al. 2009). SVHN contains over 600, 000 images of house numbers from
Baselines. We consider ofate-of-the-an (SOTA) Blug-and-yg all-play techniques for enhancing dataset distillation performance baselines, each with different optimization objectives:

---

Published as a conference paper at ICLR 2025
MIM4DD (Shang et al.
2024 introduces contrastive learning framework to estimate and
maximize the mutual information between the synthetic dataset and the real dataset.
SeqMatch (Du et al. 2024) implements dynamic adjustment of the optimization process by splitting the synthetic dataset in gradient matching methods.
IID (Deng et al. 2024) better aligns the intra-class and inter-class relationships between the original dataset and the synthetic dataset in feature matching methods.
DREAM (Liu et al. 2023) universally accelerates dataset distillation by selecting represen- tative original images through clustering.
PDD (Chen et al., 2024) generates considerably larger synthetic dataset by aligning the parameter variations throughout the gradient matching process.
Architectures. Our experimental settings follow that of Cazenavette et al. (2022) and SRe Yin eet al. (2024): we employ a ConvNet for distillation, with three convolutional blocks for CIFAR-I0 and CIFAR-100 and four convolutional blocks for Tiny-ImageNet respectively, each containing 228eeeree convolutional layer, instance normalization layer (Ulyanov al. 2016), ReLU activation function (Nair & Hinton 2010) and an average pooling layer. For larger datasets (i.e., ImageNet-1K), we employ ResNer18 (He et al. 2016) as the backbone of SRe²1. To compute CMI, employ multiple pre-trained ResNet18 trained on each corresponding dataset. AlexNet, VGGII and R8 are used to assess the cross- cooss-acchitectuee performance.
Evaluation. Following previous works, we generate synthetic datasets with IPC=1, 10, 50, and 100, then train multiple randomly initialized networks to evaluate their performance. All the training configurations follow the corresponding base distillation methods. We compute the Top-1 accuracy on the test set of the real dataset, repeating each experiment five times to calculate the mean.
Table 2: Synthetic dataset performance($6 under Table 3: Performance of the CMI enhanced loss IPC=1. CMI can only be applied methods that on higher resolution datasets, f: the reported error utilize fortulfoomlltion introduced by IDC.
range is reproduced by us.
Method
Imagena
CIFARIO
CIFAR100
IPC
IDM Zhao et al 2023
65.3=0.3
45.62077
20.1=0.3
MTT Cazenavette et
2022
8.8.0.33
23.2-10.2
28.0:1033
IID-IDM Dong & 2024
66.3+0.11
47.1=0.1
24.6=0.1
MT WTTFMM
24.1-0.3
288++0.3
IDM-CMI
66,420.3
47.5=0.2
24.7=0.5
(1.91)
(0.81)
(1.1f)
(1.91)
(4.61)
IDM Zhao et al.
2023
9.820.2
21.9:2022
26.2:0.3
IDC Kim al. 2022
68.510.9
50.6:1.44
IDMFUMI
10.410.3
23.7:0.33
27.710.1
DREAM Liu 2023
69.820.8
51.1=033
29.5=003
A
(0.61)
(1.81)
(1.51)
IDC+CMI
70.0=0.5
(1.51)
51.-003
(1.01)
301=00.1
IDC Kim DDCCMI
2023
9.54-..3
10.4=0.3
24.540.4"
25.7-003
29.0:0.2"
30.1:0.1
(0.91)
(1.21)
(1.1f)
5.2 PERFORMANCE IMPROVEMENT
We demonstrate the effectiveness of our proposed method by applying the CMI-enhanced loss to previous optimization- based distillation methods on CIFAR-10/100, Tiny-ImageNet, and ImageNet- IK. Our approach is compared with state-of-the-us baselines for various distillation methods on low-resolution datasets, with IPC set to 10 and 50 for all benchmarks, as using only one distilled image per class makes empirical computation of CMI infeasible. Additionally, we evaluate methods that employ calti-ffrmulltion techniques, such as IDM. As shown in Table and Table our method consistently yields significant performance improvements across different datasets and compression ratios, with modest gains in trajectory matching and substantial gains in feature and gradient matching methods. Notably, CMI + DSA surpasses DSA by 5.5% on CIFAR-10 with IPC=50.
For higher-resolution datasets, conducted ex- periments on representative optimization- basse dis- tillation methods with various objectives Tiny- ImageNet. As shown T noticeable improve- ments were achieved under different settings. For ImageNet-1K. IK, we applied the CMI enhanced loss to "R which decouples the distillation process by using pre-trained model and optimizes the synthetic
Table 4: Performance improvement with
SRe2L. on ImageNet-1K:
Mathod
ImageNet-IK
IPC
10
50
100
SRe"t Yin
2024
21.3.00.6
46,8.10.2
$2.8.0.3
LRe'L
24.2:20.3
49.1:0.1
$4.610.2
(2.91)
(2.31)
(1.87)
7

---

Published as a conference paper at ICLR 2025
Table Cross-architecture performance on CI- Table 6: Performance obtained simultancously ap- FARIO 10 under IPC=10 with differnent distilla- plying previous techniques and CMI enhance loss on tion methods.
different distillation methods.
Method
AlexNet
VGG11
ResNer18
Method
SVHN
CIFARIO
CIFARI00
MTT Zhao et
2025
34.2=2.6
50.3+0088
46.4+0.66
MTT Cazenavetic 2022
79.9=0.1
65.3+004
39.740.4
MIT+CMI
349-11
51,3:0.6
47.7=0.77
SeqMiien [7 at A124
80.2.:0.6
66.21:066
4199:0.5
(0.71)
(1.01)
(1.31)
Mat
81.2-0.4
67.3+-0.5
43.1+0.3
ZtMohaoetal
2023
4.88+1.1
IDM Zhao et 2025
81.020.1
$8.620.11
45.110.11
IMM-CM
54,0=04
57,0+2.2
53.7=0.2
Deng
59.91022
45.7100.
(9:4))
(9.21)
(9.11)
IDDCMM
85.1
63.0:0.3
47,910.33
IDC Kim
2022
63.5-0.4*
64.44-0."
61,24-0.2
IDC
Kim at al 2022
87,5=0.3
67.5+0.5
45.120.4
DM
64.4_0.3
67.410.2
66.0.00.4
DREAM Lure 30 2025
87.9=0.4
69.4+0.4
46 86.77
(0.91)
(3.01)
(4.81)
DREAM=CMI
88.2=0.1
71.5:0.1
47.910.3
dataset through cross-entropy loss and batch normalization statistics. This enables it to handle large datasets and achieve reliable performance. However, SRe²L faces the same challenge of excessive dataset complexity other optimization- biiiizatto---beedddd distillation methods. Following the original settings, we validate the effectiveness of our proposed method with the results shown in Table 4 for larger synthetic datasets (e.g., IPC=)00).
5.3 CROSS-ARCHITEECTII GENERALIZATION
We evaluated the performance of architectures different from the backbone used to distill CIFAR- 10. Following the settings in previous works, our experiments included widely used models such as AlexNet, VGGI1, and ResNet18, with consistent optimizers and hyperparameters. Following the same pipeline used to evaluate the same-architecture performance, each experiment is repeated five times and the mean values are reported as shown in Table 5
We can see that the considerable archttrchite generalization performance improvement achieved by method, especially when using IDM and IDC as the basic distillation method. By leveraging pre-trained models with diverse architectures the distillation phase, reducing the CMI value effectively constrains the complexity of the synthetic dataset. The results indicate that our synthetic dataset is robust to changes in network architectures.
5.4 COMBINATION WITH DIFFERENT TECHNIQUES
Unlike existing dataset distillation techniques, our method achieves stable performance improvements by reducing the complexity of the synthetic dataset itself. Furthermore, Our approach can serve as unified technique added existing methods, and experiments have demonstrated its generality. T demonstraasss the effectiveness of combining existing techniques with CMI enhanced loss on different basic distillation methods on CIFAR-I0 with IPC=10. can be observed that different opti- mization directions enable our method to be integrated with other techniques, leading to considerable performance improvements.
62-
61
60
DK
674
59-
(DC (M)
19N
63
20
150
200
100
59
s10
10t0
2000
1000
900
(a) IDC
(b) IDM
Figure 3: Accuracy curve w. and u.o. CMI constraint.
Figure 2: Ablation study on the weighting parameter X
5.5
ABLATION STUDIES AND ANALYSIS
We conducted additional experiments to verify the effectiveness CMI enhanced loss. Unless otherwise specified, the experiments were conducted CIFAR-10 with IPC=10.
8

---

Published as a conference paper at ICLR 2025
Sample distribution. As defined in Equation (5). the centroid of the DNN's outputs from specific class Y = y, where y E [C], is the average of Px with respect to the conditional distribution (ixpr((-Ie y). Referring Section 4.2 fixing the label Y y. the CMI(f, Y y) measures the distance between the centroid Q" and the output Px for class y. Thus, CMI(f: Y y) indicates how concentrated the output distributions Px are around the centroid Q" for class y.
To more intuitively demonstrate how the strategy of minimizing CMI values affects the complexity of the synthetic dataset, we visualize the t-SNE graphs of the synthetic dataset in both feature space and probability space using pre-trained ConvNet. As shown in Figure even when evaluating the dataset using different network architectures, the clusters for the synthetic dataset with lower CMI values are more concentrated. Furthermore, reducing CMI values in the feature space also effectively concentrates the dataset clusters around the class center in probability space.
Evaluation of hyper-parameter A. We conducted sensitivity analysis focusing on the weighting parameter À in Section 4.3 The results shown in Figure indicate that the CMI value is too small compared to LDD in practical applications, requiring larger À achieve the desired constraint effect. To achieve this, the value of À must be adjusted according to LDD. However, a too large À often shifts the optimization objective towards clustering the synthetic dataset around the centroids, leading to degraded performance. Varying À between 500 and 5000 resulted in only 1 1.5% performance difference on IDM, indicating moderate sensitivity.
Training Efficiency. In addition to the stable performance improvements shown in Table I we visualized the accuracy curve during training by applying the CMI enhanced loss with different distillation methods. As shown in Figure 3 unlike previous methods that typically require exten- sive training iterations to converge, our proposed method achieves comparable performance with significantly fewer iterations (c.g., one-fifth and one-tenth of the epochs required on IDC and IDM respectively) and further improve the performance with additional iterations, more visualizations can be seen in Appendix A.3
CMI Constraint Performance under different Set- tings. We explore multiple settings for computing the empirical CMI value to more comprehensively assess the effectiveness of CMI enhanced loss: (1)
Table 7: Ablation study on different settings of computing CMI value. PT indicates pre- trained or not.
Pre-trained: we investigate whether it is necessary pre-train the surrogate model for computing CMI value on the corresponding dataset; (2) Architecture: we compute CMI using surrogate models with differ- ent network architectures; (3) Space: whether com- puting CMI in probability space or feature space leads to better constraint effects.
Arch
Space
PT
ConvNet
ResNet18
67.7++0.1
67.0++0.4
Probability
68.1+-0.2
0..
67.3++0.2
67.9+00.4
Feature
68.7+10.3
70.0
Table 7 illustrates the results of computing and minimizing the CMI value under different settings. We observe that selecting an effective surrogate network architecture for CMI calculation and pre-training the model is essential, as allows the model to generate more accurate class centroids for computing CMI values. Additionally, better results are achieved by computing CMI in feature space rather than in probability space. We hypothesize that computing CMI in probability space makes the constraint more dependent on the certain model architecture. Note that to align with the baseline, all settings related to pre-training are consistent with IID (Deng et al. 2024).
5.6 VISUALIZATIONS
We compare the distillation results with and without the pro- posed CMI enhanced loss in Figure Unlike the t-SNE results Figure the constraint has minimal impact on the generated images, with differences nearly imperceptible to the human eye. This intriguing phenomenon suggests that CMI enhances performance without making substantial changes to the pixel space, offering a method that differs from existing approaches by having minimal impact on the images. Additional visu- alizations comparison of the distilled datasets with different distillation methods are provided in Appendix A.8
Figure 4: Visualization comparison between DSA (Left column) and DSA with CMI constraint (Right column).

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
In this paper, we present a novel conditional mutual information (CMI) enhanced loss for dataset distillation by analyzing and reducing the class-aware complexity of synthetic datasets. The proposed method computes and minimizes the empirical conditional mutual information of a pre-trained model, effectively addressing the challenges faced by previous dataset distillation approaches. Extensive experiments demonstrate the effectiveness of the CMI constraint in enhancing the performance of existing dataset distillation methods across various benchmark datasets. This novel approach emphasizes the intrinsic properties of the dataset itself, contributing dataset distillation.
advanced methodology for
Acknowledgement. This work is supported in part by the National Natural Science Foundation of China under grant 62177 12171248, 62301 189, Peng Cheng Laboratory (PCL2023A08), and Shenzhen Sci- ence and Technology Program under Grant KJZDD002449031005000004 JCY JCYJ202208 18101012025, GXWD2022081
1 11129300000
REFERENCES
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Jun-Yan Zhu. General- izing dataset distillation via deep generative prior. arXiv preprint 1xXiv:2305.01666 2023.
Xuxi Chen, Yu Yang. Zhangyang Wang, and Baharan Mirzasoleiman. Data distillation can be like vodka: Distilling more times for better quality. In The Twelfth International Conference on Learning Representations, 2024.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1 with constant memory. In International Conference Machine Learning, pp. 6565-6590. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang. Hongguang Zhang, Kuihua Huang. Jing Huo, and Yang Gao. Exploiting inter-sample and inter-feature relations in dataset distillation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17057-17066, 2024.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference Machine Learning, pp. 1378-5399) PMLR, 2022.
Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation.
Advances in Neural Information Processing Systems, 36, 2024.
Jiawei Du, Juncheng Hu, Wenxin Huang, Joey Tianyi Zhou, et al. Diversity-driver synthesis: Enhancing dataset distillation through directed weight adjustment. Advances in neural information processing systems, 37:1 19443-1 19465, 2025.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision. 129(6):179--888) 2021.
Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15793-15803, 2024a.
Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memory- restrictd online continual learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024b.
10

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Yang He, Lingao Xiao, and Joey Tianyi Zhou. You only condense once: Two rules for pruning condensed datasets. Advances Neural Information Processing Systems, 36, 2024.
Jang- Hyuu Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102 1118. PMLR, 2022.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung. Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In International Conference on Machine Learning, pp. 12352-12364 PMLR, 2022.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17314-17324 2023.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge- regression. International Conference on Learning Representations, 2021a. URL https
n wow.:odd=dl-prrraea0 [ttpepnser..aaeeeee///o/ooieeeep
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5199 2021b.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 10596-206077 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: core-set approach. arXiv preprint 1eXiv:100.0004833 2017.
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 21st international conference pattern recognition (ICPR2012), pp. 3288-3291. IEEE, 2012.
Yuzhang Shang, Zhihang Yuan, and Yan Yan. Mim4dd: Mutual information maximization for dataset distillation. Advances in Neural Information Processing Systems, 36, 2024.
Duo Su, Junjie Hou, Weizhi Gao, Yingjie Tian, and Bowen Tang. D 4: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5801111 5818,, 2024.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL htttps://openrev///
net fhet/fose forum (attrummid=3.0 xm mcKm
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 160.00067700022 2016.
11

---

Published as a conference paper at ICLR 2025
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12196-12205, 2022.
Shaobo Wang. Yantai Yang. Qilong Wang. Kaixin Li, Linfeng Zhang, and Junchi Yan. Not all samples should be utilized equally: Towards understanding and improving dataset distillation. arXiv preprint arXiv:2408. /2483, 2024a.
Tongzhou Wang, Jun- Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 1rXiv:8811.1090 2018.
Yuan Wang, Huazhu Fu, Renuga Kanagavelu, Qingsong Wei, Yong Liu, and Rick Siow Mong Goh. An aggregation-free federated learning for tackling data heterogeneity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 26233-26242 2024b.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference Machine Learning, pp. 1121-1128, 2009.
En-Hui Yang, Shayan Mohajer Hamidi, Linfeng Ye, Renhao Tan, and Beverly Yang. Conditional mutual information constrained deep learning classification. arXiv preprint arXiv:2309.09/23 2023.
William Yang, Ye Zhu, Zhiwei Deng, and Olga Russakovsky, What is dataset distillation learning? arXiv preprint arXiv:2406.0444 04284, 2024.
Linfeng Ye, Shayan Mohajer Hamidi, Renhao Tan, and En-Hui Yang. Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information. arXiv preprint arXiv:2401.08732, 2024.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from new perspective. Advances in Neural Information Processing Systems, 36, 2024.
Zhao and Hakan Bilen. Dataset condensation with distribution matching. arXiv preprint 10.04211 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp. 22674-1265.. PMLR, 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1(2):3, 2021.
Ganlong Zhao, Guanbin Li. Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7856-7865, 2023.
Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centrii coreset selection for high pruning rates. arXiv preprint arXiv:2210.15880 2022.
Xinhao Zhong, Hao Fang, Bin Chen, Xulin Gu, Tao Dai, Meikang Qiu, and Shu- Xia. Hierarchical features matter: A deep exploration of gan priors for improved dataset distillation. arXiv preprint arXiv:2406.05704, 2024a.
Xinhao Zhong, Shuoyang Sun, Xulin Gu, Zhaoyang Xu, Yaowei Wang, Jianlong Wu, and Bin Chen. Efficient dataset distillation via diffusion driven patch selection for improved generalization. arXiv preprint arXiv:24/2.09955 2024b.
12

---

Published as a conference paper at ICLR 2025
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. /tt/itttt////peeee.eee...o. forrrr forum?
[d=2clwrA2t.f1
Chenyang Zhu, Kai Li, Yue Ma, Chunming He, and Li Xiu. Multibooth: Towards generating all your concepts in image from text. arXiv preprint arXiv:2404.14239, 2024a.
Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, and Xiu Li. Instantswap: Fast customized concept swapping across sharp shape differences. arXiv preprint arKiv:2412.01 197, 2024b.
Tianqu Zhuang, Hongyao Yu, Yixiang Qiu, Hao Fang, Bin Chen, and Shu-Tao Xia. Stealthy shield defense: A conditional mutual information- based approach against black-box model inversion attacks. In International Conference on Learning Representations (ICLR), 2025.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 COMPARISON WITH RELATED WORKS
Several recent works also focus optimizing the dataset distillation process from the perspec- tive of the synthetic dataset itself. IID (Deng
Table 8: Performance comparison with SDC under differment settings.
et al. 2024) proposes using the L2 norm to aggregate the features of the synthetic dataset, thereby constraining its classification boundary. SDC (Wang et al., 2024a) suggests reducing the complexity of the synthetic dataset using
Dasaset
SVHN
TinyImageNet
Method
DSA
MTT
ICC
50
SDC
(Wan et
20224) (2024)]
79.4210.4
85.340
28.0.0..2
+CMI
80.5
85.5.0.3
24.1.10.3
28.810.3
GraDN-Score (Paul et al., 2021) constraints. However, IID restricts its method to distribution matching, and the L2 norm constraints fail to provide stable improvements due to the problem of dimensionality explosion. Notably, the performance comparison Table T is based on IID, which additionally optimizes the variance matching between the synthetic and original datasets.
Although SDC shares a similar starting point with our approach, it merely uses the gradient of the distillation loss constraint and restricts its method to gradient matching. This approach is constrained by the proxy backbone, leading to weaker constraint effects and limited performance improvements due to the optimization objective with the distillation loss. A comparison of experimental results is shown in Table g
A.2 REGULARIZATION PROPRIETY
Here, we analyze the trends of dataset distillation loss LDD during training as shown in Section 4.3 Lpn refers to any dataset distillation loss. By incorporating CMI as a regularization term during dataset distillation, we successfully constrained the complexity of the synthetic dataset. As shown in Figure 5 we present the LDD curves throughout the training process. It can be observed constraint decreased the main
that adding CMI function losses, demonstrating that our proposed method effectively reduces the complexity of the synthetic dataset.
100
10.0
300
2100
010
601
750
1006
Egarh
Figure 5: The LIDC curves while train- ing on CIFAR10 under IPC=10.
28
CMC
sia 108
130
400
100
135
2.5
050
(a) The accuracy curve of adding CMI constraint to DM.
(b) The accuracy curve of adding CMI constraint to DSA.
(c) The accuracy curve of adding CMI constraint to MTT.
Figure 6: Applying CMI constraint brings stable performance and efficiency improvements with different optimization objectives.
A.3 EFFICIENT TRAINING PROCESS
By deploying the CMI constraint, we not only achieve performance improvements but also sig- nificant computational acceleration. As shown in Figure 6 we present the accuracy curves after incorporating the CMI constraint DSA (Zhao & Bilen 202 DM (Zhao & Bilen 2021a), and MTT (Cazenavette et al. 2022). It can be seen that our method achieves considerable computational acceleration with the same initialization settings. These results indicate that actively constraining the
14

---

Published as a conference paper at ICLR 2025
complexity of the synthetic dataset during training benefits both new network training and reduces gradient bias, leading to faster convergence.
Algorithm Disentangled computation of applying CMI constraint
Input: C: distillation loss; fe=: pre-trained classifier; T: real dataset; o(-): proxy information extrac-
tor; M: distance metric; K: training iteration number O: class number;
1: L =
2: Randomly initialize S
3: for i <- to K - 1 do
4: Randomly sample fo-
5: C L +- M( ((T))
6: for j to 0
do
7:
CMllmpa (S) Exes, KL (Pa Qemp)
[= C + X C.llemp.e 9: end for
8:
S SGD(S; L)
II: end for
Output: Synthetic dataset S
A.4 DISENTANGLED CMI VALUE COMPUTATION
Simultancously computing and minimizing the CMI value for the entire synthetic dataset during distillation can lead to significant memory consumption as IPC and the number of categories increase.
To address this issue, propose decomposing the CMI computation shown in Section 4.2 by category and embedding it into the original distillation process, as shown in Algorithm T Since the computation of the empirical CMI value is calegory-independent, the decoupled calculation can equivalently represent the original equation without any loss of fidelity, This approach greatly alleviates the memory overhead while achieving consistent results.
A.5 ADDITIONAL TRAINING COST
We further analyze the extra time consumption caused by applying CMI constraint in Table on
Table 9: Additional time consumption (s) per itera- tion of adding CMI constraint.
a single NVIDIA GeForce RTX 3090. For IDM, adding the CMI constraint nearly doubles the time consumption; however, considering that need only one-tenth one-quarter of the distillation iterations to achieve the original per- formance, using the CMI constraint can save up to 50% of the time. In contrast, for IDC, the additional time cost introduced by the CMI constraint is negligible compared to the method itself. Moreover, since applying CMI enhanced loss can reduce the number of distillation iterations to one-fifth of the iterations to achieve the original performance, deploying CMI is justified. Notably, we calculate and optimize CMI each iteration. However, to save computational resources, we could reduce CMI calculations to once every five iterations while maintaining performance, thus speeding up distillation.
Method
CIFARIO
CIFAR100
IPC
10
50
10
50
IDM (Zhao al. 2023
0.5
0.6
5.3
7.1
IDM+CMI
1.0
1.2
10.2
11.2
IDC Kim al. 2022
18.2
22.4
190.5
215.7
IDC+CMI
22.6
24.2
222.5
236.8
A.6 ABLATION STUDY ON MODEL ARCHITECTURES
To further investigate the impact of teacher model architecture, we conduct more comprehensive ablation experiments. Under the CIFARIO with IPC=10, utilized simply pre-trained ConyNet, AlexNet, ResNet18, and VGG11 as proxy models to compute CMI for both DM and DSA.
We sequentially report the following metrics: the depth (i.e., number of convolution layer), the dimensionality M of the feature space, and the accuracy of the simply pre-trained proxy model; the start CMI value of the synthetic dataset, the CMI value at the end of the optimization process and
15

---

Published as a conference paper at ICLR 2025
Table 10: Ablation study of model architecture on Table 11: Ablation study of model architecture on DM.
DSA.
Method
ConvNet
AlexNet
VGGI1
ResNet18
Method
ConvNet
AlexNet
VGGII
Re
Depth
Dopth
17
Dimension
2048
4096
512
512
Dimension
2048
4096
512
512
Acc (5)
80.2
79.4
Model (%)
79.5
Start value
0,1045
199388
0.0021
0.0016
Start value
0.1045
193388
0.0021
0.0016
End value w.o. CMI
0.0866
1.617
0.0059
0.0066
End value CMI
0.0825
1,8755
0.0076
0.0051
End value w. CMT
0,0326
0.6642
0.0006
0.0004
End value w. CMI
0.0455
0.8875
0.0005
0.0004
Performance (%)
51.2
50.8
52.4
52.9
Performance (8)
53.2
53.4
54.8
54.7
w. CMI constraint of the synthetic dataset, and the accuracy of the model trained using the synthetic dataset. As shown in table below, it can be observed that the dimensionality of the feature space has a greater impact on CMI values than network depth, and CMI effectively reflects the degree of clustering in 2 under the same dimensional conditions. Nonetheless, the proposed CMI constraint effectively regulates the complexity of the synthetic dataset, demonstrating the strong generalization capability of our method.
A.7 LIMITATION AND FUTURE WORK
IPC=1. Minimizing CMI is equal to reduce the Kullback-Leibler (KL) divergence D(-II-) between Ps and 22 hence, when the IPC is 1, the CMI value is O.
Proxy Model. Computing CMI requires choosing appropriate proxy model architecture to provide Ž when S is used the input, and selecting an architecture often relies heuristic methods. Besides, similar to IID (Deng et al.) 2024), constraining the synthetic datasets with CMI needs multiple pre-trained teacher networks, which leads substantial consumption overhead.
Here, we suggest a future direction to better utilize the proxy model; mixing the proxy model could potentially provide a more informative CMI constraint. And One can try perturb single model to obtain multiple teacher models which is similar to what DWA (Du et al. 2025) did.
Recent years, some dataset distillation methods have introduced GANs as a parameterization tech- nique (Cazenavette et al. 20233 Zhong et al. 2024a). Moreover, with the successful application of diffusion models various fields (Zhu et al.) 2024b a), some dataset distillation methods have begun using diffusion models to directly generate images (Gu et al. 2024a] (Su et al., [024] [hhon et al. 2024b). In future work, we will explore the application of CMI to these types dataset distillation methods.
A.8 DISTILLED DATASET VISUALIZATION
To demonstrate the effectiveness of our method, we present the t-SNE visualization of the synthetic dataset with the CMI constraint on DSA, shown in Figure 7 It be observed that the CMI constraint provides strong regularization effects in both feature space and probability space, resulting in clearer class boundaries. Additionally, we visualize the effects of synthetic datasets produced using different datasets and distillation methods, As shown in Figures g to II the CMI constraint not only significantly enhances performance but also introduces minimal perceptual distortion.
Feature Space
Probability Space
Feature Space
Probability Space
Synthetic dataset CMI constraint.
(b) Synthetic dataset w. CMI constraint.
Figure 7: Visualization of the synthetic dataset generated by DSA with (a) high CMI value, and (b)
low CMI value.
16

---

Published as a conference paper at ICLR 2025
28
(a) IDC
(b) IDC+CMI
Figure 8: Visualization of the synthetic dataset generated by IDC on SVHN.
(a) DSA
(b) DSA+CMI
Figure 9: Visualization of the synthetic dataset generated by DSA on CIFAR1O.
17

---

Published as a conference paper at ICLR 2025
(a) IDM
(b) IDM+CMI
Figure 10: Partial visualization of the synthetic dataset generated by IDM on CIFAR100.
DM
(b) DM+CMI
Figure 11: Partial visualization of the synthetic dataset generated by DM on TinylmageNet.
18


Reviews:
[{"review_id": "FWcDWGRgeg", "reviewer": "Reviewer_oMQb", "summary": "This paper proposes a new approach for dataset distillation by introducing a class-aware conditional mutual information (CMI) metric to address challenges in creating compact, representative synthetic datasets. Traditional dataset distillation methods often compress feature similarity without considering class-specific complexity, making it hard for models to generalize across different classes. This work leverages CMI as a regularization constraint, optimizes synthetic datasets and improves training efficiency as well as model performance.", "questions": "1.\tThe paper is well-organized and clearly presents the methodology, results, and analyses. Figures and tables are effectively used to convey improvements and insights. However, further explanation of certain key terms, such as \"empirical CMI,\" might enhance accessibility for readers unfamiliar with the topic.\n2.\tThe ablation studies conducted to assess the influence of the weighting parameter on the CMI constraint are informative. Still, a broader exploration of other hyperparameters affecting CMI estimation, such as the dimensionality of feature space and network depth, could reveal potential optimizations.\n3.\tThe potential of CMI for real-world applications, such as federated learning or privacy-preserving tasks, is not discussed. Given the emphasis on dataset distillation's applications in these areas, an exploration of how CMI might support these domains would align well with the broader goals of dataset distillation research.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of using CMI in dataset distillation to address the inherent class-aware complexity issue is interesting.\n2.\tThe experiments are conducted based on multiple datasets and various model architectures, providing solid evidence for the method's effectiveness. \n3.\tThe proposed method CMI is a versatile, \"plug-and-play\" regularization component that can be applied to numerous dataset distillation methods, such as DSA, MTT, and IDC. This flexibility allows the approach to generalize across different scenarios and highlights its robustness.\n4.\tBy controlling the complexity of the synthetic dataset, the CMI-enhanced loss achieves faster convergence and reduces the number of required training iterations, which is particularly beneficial for large-scale datasets and resource-intensive models.", "weaknesses": "1.\tWhile the paper demonstrates the CMI constraint’s benefits clearly, this method also introduces additional computation overhead, especially when dealing with high-resolution datasets. Although the authors briefly mention several strategies for mitigating this cost (e.g., reducing CMI calculations frequency), a more thorough discussion on balancing cost and performance might strengthen the practical feasibility.\n2.\tAlthough empirical evidence is strong, the theoretical basis for CMI as a regularization term could be expanded. Specifically, further details on how CMI inherently captures class complexity or why it is preferable over alternative complexity measures would provide deeper insight.\n3.\tWhile the experiments on Tiny-ImageNet and ImageNet-1K are promising, it remains unclear how the proposed method scales with even larger datasets or more complex models, such as those used in real-world applications with hundreds of classes. Additional experiments in such contexts would further show the robustness of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phgCzg8sHh", "reviewer": "Reviewer_kma8", "summary": "This paper proposes a Conditional Mutual Information (CMI) method as a plug-and-play loss function to enhance the performance of dataset distillation methods. Experiments conducted on multiple baseline methods demonstrate the effectiveness of the CMI loss.", "questions": "Please kindly refer to the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed CMI method is a relatively simple yet effective approach that is plug-and-play in nature. It has demonstrated its effectiveness across multiple baseline methods.\n2.\tThe motivation behind the method proposed in the paper is solid and is supported by a certain theoretical foundation.\n3.\tThe experiments in the paper are comprehensive, conducted across various scales of datasets.", "weaknesses": "1.\tThere are now newer and more powerful methods available, such as \"Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching\" (ICLR 2024). The authors could consider experimenting with their proposed method on these methods.\n2.\tThe description of the method in the paper could be clearer, particularly regarding the explanation of the formula symbols, to better emphasize the key points of the approach. Currently, it appears somewhat ambiguous.\n3.\tIn my view, using mutual information or KL divergence is not a particularly novel approach, as it has been employed in many works across various fields.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vaq2qudvVq", "reviewer": "Reviewer_i6mf", "summary": "This paper introduces a novel regularization method for dataset distillation (DD) by minimizing both the distillation loss and Conditional Mutual Information (CMI) of synthetic datasets. It uses an efficient CMI estimation method to measure class-aware properties and combines CMI with existing DD techniques. Experiments show that the proposed CMI-enhanced loss significantly outperforms state-of-the-art methods, improving performance by up to 5.5%. The method can be used as a plug-and-play module for all existing DD methods with diverse optimization objectives.", "questions": "Can you discuss any potential limitations of your proposed method and suggest directions for future work to address these limitations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The strengths of this paper lie in its comprehensive experimentation across diverse datasets and network architectures, which effectively demonstrates the versatility and robustness of the proposed method. Furthermore, the method's ability to be integrated as a plug-and-play module into existing dataset distillation techniques, regardless of their optimization objectives, showcases its innovation and flexibility, making it a significant contribution to the field.", "weaknesses": "The paper lacks a clear discussion of the limitations of the proposed method. Furthermore, the authors should consider using more intuitive explanations, visual aids, and pseudocode to help readers better understand the technical details of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "osFoVxV0DZ", "reviewer": "Reviewer_Zh5i", "summary": "This paper proposes a plug-and-play method, termed CMI, designed to enhance existing DD techniques by minimizing conditional mutual information. By applying CMI, the distilled data is concentrated more effectively around the center of each class, thereby improving generalizability.", "questions": "see weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The logic is clear.\n- The experiments are comprehensive.\n- The review of related works is thorough.\n- The proposed method is theoretically sound.", "weaknesses": "- The core ideas, methodology, and formulations in this paper draw substantially from the approach proposed in [1].\n- If  \\hat{Z}  contains excessive confusing or uninformative information related to  S , then  H(S | \\hat{Z}, Y)  will not be reduced; rather, it could remain the same or even increase. This is because conditional entropy reflects the remaining uncertainty in  S  after observing both  \\hat{Z}  and  Y . When  \\hat{Z}  is noisy or irrelevant for predicting  S , it does not help in reducing this uncertainty.\n- Line 213 states that “minimizing the class-aware CMI reduces the uncertainty brought to  \\hat{Z}  conditioned on  S ,” which should be  “minimizing the class-aware CMI reduces the uncertainty brought to S conditioned on  \\hat{Z}”.\n- The authors’ derivation of Equation 6 lacks an explicit explanation, making it challenging to fully understand the transition from previous formulations.\n- Works like [2] and [3], which also target improvements in dataset distillation, are not adequately considered. \n- Equation 3 requires summing over all synthetic instances within class  y , how the authors adapt this approach to instance-based distillation methods like SRe2L. \n\n[1] Bayes conditional distribution estimation for knowledge distillation based on conditional mutual information\n\n[2]TOWARDS LOSSLESS DATASET DISTILLATION VIA DIFFICULTY-ALIGNED TRAJECTORY MATCHING\n\n[3]Prioritize Alignment in Dataset Distillation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)