MIM4DD: Mutual Information Maximization for Dataset Distillation
Authors: Yuzhang Shang
Zhihang Yuan
Yan Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset Distillation

Abstract: Dataset distillation (DD) aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model. State-of-the-art (SoTA) methods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig.1, Left), such as gradients and training trajectories. DD is essentially a compression problem that emphasizes on maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement, but are never considered by previous works. Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets who share the same labels as positive pairs, and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As a result, the targeted MI can be transformed into a lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that MIM4DD can be implemented as an add-on module to existing SoTA DD methods.

MIM4DD: Mutual Information Maximization for Dataset Distillation
Yuzhang Shang', Zhihang Yuan², Yan Yan'-
Department of Computer Science, Illinois Institute of Technology
Huomo Al
yshang40hawk, iit..iit,,, zhihang yuanehuuunanooooo ai, yyan348iit.edu
Abstract
Dataset distillation (DD) aims to synthesize small dataset whose test performance is comparable to full dataset using the same model. State-of-the-ant art (SoTA) meth- ods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig. 1, Left), such gradients and training trajectories. DD is essentially compression problem that emphasizes maximizing the preservation of information contained the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Thus, introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIMADD numerically maximizing the MI via newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets that share the same labels as positive pairs and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As result, the targeted MI can be transformed into lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that MIM4DD can implemented as an add-on module to existing SoTA DD methods.
1 Introduction
Real
Deep learning has remarkably successful perfor- mance in computer vision tasks [25, 6]. but most deep-learning- methods require enormous amounts of data followed by extensive training sources. For example, in order to train a CLIP model [31] in a self-supervised manner [8], high- resolution dataset with more than 10 million images collected for training, which consumes tens of hundreds of GPU hours.
Dataset
Dataset
Motching
Mutual Information
Lown
Loss
Synthetic
Synthetic
Dataset
Dutaset
Divtribution
Farweddrrssgitee
Rakwwrd Propagation
Figure (Left) General framework of previous SoTA DD methods: matching heuristic indica- tors extracted by networks from real and syn- thetic datasets; (Right) the motivation of MIM4DD: maximizing the mutual information hetween datasets optimizing the synthetic dataset. In the era of deep learning, in contrast to
A straightforward solution to eliminate the reliance on training DNNs on data to construct small train- ing sets. Before the deep learning era. coreset or subset selection is the most prevalent paradigm, in which can obtain subset of salient data points to represent the original dataset interest [1, 12, 34].
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

previously mentioned selection based methods, Dataset Distillation (DD), also known as Dataset Condensation [41, 46, 40, 5], offers a revolutionary paradigm to synthesize small dataset using gradient descent algorithms [33], as shown in Fig.1 (Left). In DD, the key question is how to define metrics to measure the distance between synthetic and real datasets. Only by optimizing a well designed distance metric can gradient descent be properly applied to update synthetic data. To answer this question, researchers design several distance algorithms. For example, Zhao et al. [46] measure distance between the batch gradients of the synthetic samples and original ones; Wang et al. [40] use the similarity between the feature maps extracted by networks on different datasets; Cazenavette et al. [5] resort to the MSE between the training trajectories of networks on real and synthetic datasets.
Dataset distillation task is essentially a compression problem with strong emphasis on maximizing the preservation of information contained in the data [41, 19, 11]. Admittedly, previous heuristic- designed distance metries have achieved promising performance. However, we argue that previous works neglect the well-defined distributional metric in information theory, which is necessary for success measurement. Specifically, if we define a dataset's samples as variables, is imperative that the high-level distributional properties of these variables from synthetic and real datasets (e.g., correlations and dependencies between two datasets) should be captured and utilized to guide the update of synthetic datasets. Motivated by this notion, we introduce Mutual Information (MI), metric in information theory for dataset distillation. In detail, MI quantifies the information amount shared by the real and synthetic datasets. In contrast to the aforementioned works focusing on aligning the indicators extracted by different neural networks, MI can naturally capture non-linear statistical dependencies between variables and be used measure of true dependence, which is important information in data compression [4, 19,39, 36].
Based on MI metric, we propose novel method, termed Mutual Information Maximization for Dataset Distillation (MIM4DD). In particular, we first formulate DD as a problem of MI maximization between two datasets. Then, we derive numerically feasible lower bound and maximize this lower bound via contrastive learning [15, 18, 8]. Finally, we design a highly effective optimization strategy for the dataset distillation task using contrastive estimation for MI maximization. In this way, contrastive learning theoretically leads to the targeted MI maximization and also contributes to the inter-class decorrelation of synthetic samples. In other words, MIK4DD is built upon contrastive learning framework to synthesize the small dataset, where samples from the synthetic dataset are pulled closer to the counterparts with the identical label in the real dataset, and pulled further away from the ones with different labels in the contrastive space. To the best of our knowledge, it is the first work aiming MIM of the datasets over the DD task within contrastive learning framework.
Overall, the contributions of this paper are three-fold: (i) To distill information from a large real dataset to small synthetic dataset under well-defined metric, we formulate the DD task an MI maximization problem. To the best of our knowledge, this is the first work to introduce MI into the literature of DD. (ii) To maximize the targeted MI, we derive numerically feasible lower bound and maximize it via contrastive learning. In this way, the heterogeneity in the generated images gets enhanced, which further improves the performance of the DD method. (iii) Experimental results show that our method outperforms existing SoTA methods. Importantly, our method can be implemented as plug- plug-and- module for existing methods.
2 Methodology
In this section, demonstrate the methodology of Mutual Information Maximization for Dataset Distillation (MIMADD). Firstly, we briefly revisit the general framework in previous SOTA DD methods, and we illustrate the MI background. Secondly, we model the DD problem within the literature on mutual information maximization (MIM). Then, we convert the targeted but numerical inaccessible MIM goal into a learnable object optimization problem, and solve the problem via a newly designed MIMADD loss. Finally, we discuss the potential insights of MIMADD. Note that we only elaborate on the key derivations in this section due to the space limitation; detailed discussions and technical theorems can be found in the supplemental material.
2.1 Preliminaries
Dataset Distillation. In short, the goal of dataset distillation (also dubbed dataset condensation) is M to synthesize small training set, D: Deyn {xx y' Jj=1 such that models trained on this synthetic
2

---

dataset can have comparable performance as models (with the same architecture) trained on the large real set, D Dreal eat. y' 1==1. in which M < N. In this way, after obtaining the targeted synthetic data, the training process of DNNs can be largely accelerated. In this premise, we review some representative DD methods [43, 5, 40], which generally propose to enforce the behaviors of models trained on real and synthetic datasets to be similar. The core idea can be formalized in a form of an optimization problem:
D syn = arg min Ex.yn Deyn Ex, real Dist(f(Xreaul) f 1 7))), Daye
(2.1)
in which H and y are the learned parameters of networks trained on the real dataset D Dreat and the synthetic dataset Dayn: respectively; and the distance function Dist(:;-) is specifically developed to measure the similarity of two networks across datasets. With aligning the networks, one can optimize the synthetic data. Note that designing the distance functions aligning the networks trained on different datasets to optimize the synthetic dataset is the core of previous DD methods, e.g., CAFE [40] uses the MSE distance of feature maps, and MTT [5] calculates the difference of training trajectories as distance functions.
Although those heuristically designed distance functions lead to promising results, argue that well-defined metrics in information theory for measuring the amount of shared information between variables have never been considered, as DD is essentially a compression problem with different emphasis on the information contained in the data.
Mutual Information and Contrastive Learning. Mutual information quantifies the amount of information obtained about one random variable by observing the other random variable. It is dimensionless quantity with (generally) units of bits, and can be considered as the reduction in uncertainty about one random variable given knowledge of another. High mutual information indicates large reduction in uncertainty and vice versa [23]. Strictly, for two discrete variables X and Y, their mutual information (MI) can be defined as [23]:
I(X, Y) = E Pxy log (xy
(2.2)
Px(x)P((U)
where Pxy (z, 3) is the joint distribution, Px(x)) Ey, Ey Pxy((,)))) y) and Pr = E. (x,1)
are the marginals of X and Y, respectively.
In the content of DD, we would like them to share as much information as possible. Theoretically, considering the samples in real and synthetic datasets as two random variables, the MI between those two variables should be maximized. Recently, contrastive learning has been proven an effective approach to maximize MI. Many methods based on contrastive loss for self-supervised learning are proposed, such as [18), [29]. [42]. These methods are theoretically based on NCE [15] and InfoNCE [18]. Essentially, the key concept of contrastive learning is to pull representations in positive pairs close and push representations in negative pairs apart in a contrastive space. Thus the major obstacle for modeling problems in contrastive way define the negative and positive pairs. In this work, we also resort to a contrastive learning framework for maximizing our targeted MI. Meanwhile, we illustrate how we formulate DD as MI maximization problem, and how we solve this targeted problem within the contrastive learning framework.
2.2 MIM4DD: Mutual Information Maximization for Dataset Distillation
In this section, we first formalize the idea of maximizing the MI between the distributions of real and Estimation (NCE). synthesized data, via constructing contrastive task based on information from real data Dreat Specifically, we derive novel MIMADD loss to distill task-specific to synthetic data Dayn. where NCE is introduced to avoid the direct MI computation by estimating it with its lower bound in 2q 14. From the perspective of NCF, straight-forwarde synthesized samples from the same class be pulled close, and samples from different classes can be pushed away, which corresponds to the core idea of contrastive learning.
the real and
Problem Formulation: Ideally, for variable Xreat representing the samples in real data and X syn in the synthetic data, we desire to maximize the MI between Xreat and Xayn in terms of X i.e.,
Targeted MI: X syn = arg max [ (eeass Xoyn):
(2.3)
Xaya
3

---

In this way, synthetic data syn and fixed real data D* real can share maximal information. However, directly approaching this goal is unattainable since the distributions of datasets themselves are absurd to estimate [15, 18]. To encounter this obstacle, let us include the networks trained on real and synthetic datasets. We define those networks the form of K --yyyer Multi-Layer Perceptrons (MLPss. For simplification, we discard the bias term of those MLPs. Then the network f(x) can be denoted as:
f(w),.. W x) (WK W .y........ W))
(2.4)
where x is the input sample and W Rd - R (k: -,0==1,... I..,,K K) stands for the weight matrix connecting the (k - 1) and the k-th layer, with da and dy representing the sizes of the input and output of the k-th network layer, respectively. The of.) function performs element-wise activation
operation the input feature maps. Based on those predefined notions, the sectional MLP f*(x) with the front & layers of the fxx can be represented as:
I (wW
W -W* a (x)
(2.5)
The k-th layer's feature maps are A syn and A (k {1,... ,K),, where A syn (a ... syn and A Afcal = (a a reat) can be considered as a series of variables. Specifi- cally, the feature map can be obtained by:
= ( j E {1,--.,,, M). Jade Asya
kife, = ('(fxa))) i (1,
real
(2.6)
Here, we show the relationship of MI in data X level and in feature maps A level, i.e. the relationship between [(Xyeal Xxyn) and I(A [Ayyall Asya). We utilize the in-variance of MI to understand it. The property can be illustrated as follows:
Theorem 1 (In-variance of Mutual Information): Mutual information is invariant under the reparametrization of the marginal variables. If X' =F(X) and Y' G(Y) are homeomorphisms (i.e., F.) and G(-) are smooth uniquely invertible maps), then
I(X,Y) = X(X'
(2.7)
Since each layer's mapping W Rd 1 - Rd (k = I.... K) can be considered as the smooth uniquely invertible maps Theorem 1. Combining this theorem with the definition of MI in Eq. we observe that the MI in the targeted data level is equivalent to MI in the feature maps level, i.e.
I(Xyeal: X xym)) I(A A p (I ......,).
(2.8)
More details and the proof of this theorem are in Supplemental Materials and [21].
Facilitated by this property, we are able to access the calculation of MI between real and synthetic datasets via the feature maps of networks trained on those datasets. In other words, Theorem helps transfer the targeted MI maximization in Eq.2.3 to a reachable MI in Eq.2.8 in the dataset distillation literature, i.e.
K Σ
Accessible MI: arg max
T(At. Aggn)):
(2.9)
Xuyn
in which A (X(xyy)) and A teal fM (X
Apart from the theoretical derivation, intuitively, the corresponding variables (a Feat) should share more information, i.e., MI of the same layer's output feature maps I(a a a (A: e {1....,K3 .K) should be maximized to enforce them mutually dependent. This motivation has also been testified from the perspective of KD [17, 13] by CRD [39] and WCoRD [7]. In those methods, the penultimate layer's feature maps of teacher and student are aligned in a contrastive learning man- ner to enhance the heterogeneity of representations, which can also be explained via MI maximization. However, MIM4DD is different from those methods (details is in Related Work).
To optimize the accessible Mutual Information (MI) as defined in Eq,2.9, we incorporate a contrastive learning framework into our targeted Dataset Distillation (DD) task. The fundamental principle of contrastive learning involves comparing varying perspectives of the data, typically under different data augmentations, to compute similarity scores [29, 18, 2, 16, 8). This framework suitable for our case, since the activations from real and synthetic datasets can be seen two different views.

---

Synthetic
Roal Data
Pull
Representations
Network
Network
Contrastive Space
Forward Pass
Backward Pass
Figure 2: Feeding images from two datasets into two corresponding neural networks, and we obtain the three pairs of representations for each layer. We embed the representations into a contrastive space, then learn from the pair correlation with the contrastive learning task in Eq. 2.16. In this way. the heterogeneity in the generated images be enhanced. Theoretically. Eq. 2.9, which is equivalent the targeted MIM Eq. 2.3.
realize the formulated mutual information maximization (MIM)
(Definition: Positive and Negative Samples.) Let's denote each sample from the real and synthetic and {xaja (Ye, E -1,...,,M respectively. Here, c, and Co E {1,-... ,CC represent the class labels of xTea, and Xaya. respectively. We feed two batches of samples from two datasets to two different neural networks and obtain N MM pairs of k-th layer's activations (asyn* areal) ), which can further be utilized for the contrastive learning task. We define pair containing two activations corresponding the same labels as the positive pair, i.e., if o, Ca. (asyn" a
datasets (xteat) (je, E (1,.... .N)
k.jer )+ and vice versa. Consequently, there is C C== t N M positive pairs, and
thus (1 - )) ) ..... .M negative pairs. The key concept of contrastive learning is to discriminate kife, k.jor)
whether given pair of activation (asgn areal
positive or negative. In other words, it involves kite, k.Jer) in which D is scalar variable indicating whether estimate the distribution P(D) asgn .aaeal
C2 = C, or C # cr. Specifically, = 1 when C2 = Gr, and = 0 when c, * G,. However, we can not directly reach the distribution P(D k.s., (earr) [15], and thus we introduce its corresponding variational form:
q(D
k.Jor k. A asyn
(2.10)
K.dey) can be treated as a binary classifier, which can classify a given pair (asyn a into positive or negative. Importantly, q(D a kife, k.der) k,de, k.jor)
Intuitively, ( K.Jor areal
a asya can be estimated by some mature statistical methods, such as NCE [15] and InfoNCE [18].
Using the Bayes rule, the posterior probability of two activations from the positive pair can be formalized as:
kião, kidor 1D D==1) ((aggn areal
qDD = 1
kides asya a
(2.11)
hife. Rider ((axyjn aral 1) + q(aajn kiis Rider
=
araal
= 0)
The probability of activations from negative pair is q(D = 0 asijn k:jorr 1 q(D =
kifo, a ). To simplify the NCE derivative, several works [15, 42, 39] build assumption about
asyn'
the dependence of the variables, we also use the assumption that the activations from positive pairs
are dependent and the ones from negative pairs are independent, i.e. q(a kiie, D 0) P(asys P(a Hence, the above equation P(ange kife, feler) and g(asgn kise, Rejer
Kite. a
kjc,
1)
areal
can be simplified as:
P(asin
kiiee K.der) kidey P(a (CC
Ee q(D = asyn
(2.12)
K.des K.ee + P(a P(asyn
Performing logarithm Eq.2.12 and arranging the terms,
achieve
Kibo, P(a a )
log q(D =
kjjor) < log
log
(2.13)
P(a P(a
Kohon KJJor)
5

---

Taking expectation of P(asyn" ksiey Asfer) on both sides, and combining Eq.2.2, we can transfer the MI into:
Accossible MI Eq. 2.9
optimized lower bound
[(a areat)
2 log((
Eytes Auior ID=1) log q = 1 asgn a K.Joy
(2.14)
P(aaye
where I(a syn: a reat) is the MI between the real and synthetic data distributions. Instead of directly maximizing the MI, maximizing the lower bound in the Eq.2.14 is practical solution.
However, even (DD 1| a tifn,, a a..e..) ) still hard estimate. Thus, tackled by many contrastive learning works [29, 18, 2, 39, 36, 37], we introduce discriminator network d(.,) -) with parameter Φ
(i.e., d(aauyx R.fe, rreal o)). Basically, the discriminator d can map a a to [0, 1] (i.e., distinguish
given two samples a afeat belonging to positive or negative pair). Specifically, the discriminator function is designed follows:
kiies k.Jor exp( < Io (asyri d(
), Iob (aa(aa,c (areal)
d(ayy Areal
3/C,
(2.15)
in which ()(-) is the embedding function for mapping the activations into the contrastive space and C' = exp( Salaivn keice ).g (og
wou >> + 1, and r is a temperature parameter that controls the concentration
level of the distribution [17. 39].
Loss Function. We define the contrastive loss function CICC between the k-th layer's activations a and a as: C LNCE=
syn
E
koles Noden (D=1) "(aays i
llogh(ang: k.c. kJJ +(C-1)-B
paa krics "uar 1D log(1
kife. - h(asgn
a Helar ))
(2.16)
In the view of contrastive learning, the first term the above loss function about positive pairs is op- timized for capturing more intra-class correlations and the second term of negative pairs is for inter- class decorrelation. Because we construct the pairs instance- wisely, the number of negative pairs can be the size of the entire real dataset, e.g. 50K for CIFAR [22]. By incorporating hand-crafted, addi- tional contrastive pairs for the proxy optimization problem in Eq.2.16, the representational quality of generated images can be further enhanced as demonstrated by numerous contrastive learning methods[8, 29, 18, 2].
Figure
Ablation Studies. Effect in LMIMADD (Eq.2.17) on CIFARIO (Left) and CIFARIOO (Right). X = 0 is the baseline, MTT [5].
Finally, by incorporating the set of NCE loss for various layers {Ckco) (k = ,,,,,,,,,, we can then formulate MIM4DD loss LMIMADD as:
CiNCE 3K-1-k
LMIMADD
+ LDD.
(2.17)
in which LpD can be any loss functions in previous DD methods [40, 43, 5], À is used to control the NCE loss, and is scalar greater than 1. In practice, we use MTT [5] or BPTT [11] as default Lnn-
2.3 Discussion on MIM4DD
In addition to the theoretical formulation, we are going to provide more straightforward mani- festation of MIMADD. As illustrated in Fig.2, by embedding the activations to the contrastive space and constructing proper negative-and- -nd-pssiiveee embedding pairs, we can heterogeneously distill the relevant information from the real dataset to learnable synthetic dataset, and enhance the heterogeneity of the synthetic dataset within the contrastive learning framework as shown in Fig.5. Specifically, networks on different datasets first learn self-supervised information benefited from a large number of the designed pairs [18, 39]. Then, via the backward pass in Fig.2, the synthetic data updated

---

w.r.t. this contrastive loss. While the actual number of negative samples in practice can be huge, e.g., 50K for synthesizing 10 pictures per-class replace CIFARIO [22] despite only drawing two samples in Fig.2.
Furthermore, we give direct explanation of why optimizing MIM4DD in Eq.2.17 can lead to the largeted MIM between synthetic dataset and real dataset in Eq.2.3. Firstly, optimizing the formulated contrastive loss in Eq.2.1 is equivalent to maximizing the derived accessible MI between activations in Eq.2.9, as the inequality in Eq.2.14 (Eq.2.10-2.13 can derive Eq.2. 14). Secondly, based on Theorem I (Eq.A.1) and the property of the networks itself (Eq.2.4 and Eq.2.5). maximizing the accessible MI 2..2.9 equals to maximizing the targeted MI in 2.8,. which is our goal, to minimize the MI between synthetic dataset and real dataset. Since real data are fixed and synthetic data are learnable, the optimized synthetic dataset can contain as much information as the real dataset under the metric of MI.
3 Experiments
In this section, we conduct comprehensive experiments evaluate our proposed method MIMADD on four different datasets for DD task. We first describe the implementation details of MIMADD, and then compare our method with several SoTA DD methods to demonstrate superiority of our proposed method. Finally, we validate the effectiveness of MI module (connected with Eq.2. 16 and Eq.2.17) by a series of ablation studies.
3.1 Datasets and Implementation Details
Datasets. We use MNIST [24], SVHN [35], and CIFAR10/100 datasets conduct our experiments. MNIST [24] is dataset for handwritten digits recognition that is widely used for validating image recognition models. contains 60,000 training images and 10,000 testing images with the size of 28 x 28. CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of 32 x 32 from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing. More details of the datasets can be found in Supplemental Materials.
Implementation Details. the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using a three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of Conv-InstNorm ReL.U-AvgPool." Each convolutional layer has 128 channels, and AvgPool represents 2 x 2 average pooling operation with stride 2. The synthetic images' initial learning rate 0.1, which is halved at the 1,800th and 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and À in Eq.2.17 is 0.8. The effect of À is explored in 3.c.3.3.
3.2 Comparison with SoTA
We compare MIM4DD with a series of state-of-the-an (SoTA) dataset distillation methods, including
Dataset Distillation (DD) [41]. LD [3], Dataset Condensation (DC) [46], DC with differentiable
siamese augmentation (DSA) [43], DC with distribution matching (DM) [45], CAFE [40], FRePo [47], TESLA [10], BPTT [11], and MTT [5]. We report the performances of our method and comparisons on three datasets in Table Taking into account the overall performance of MIMADD across these mainstream dataset distillation benchmarks, it's apparent that our method consistently outperforms existing SoTAs. For instance, in the setting of generating 10 images per class, our method delivers top-tier results across all datasets. Additionally, when we synthesize 10 images per class using CIFARIOF 100 as a real-world dataset, our method surpasses MTT by a margin of 1.4%. Importantly, our method can serve as an effective plug-and-ple module for existing state-of-the-art DD methods.
3.3 Ablation Studies
7

---

Table 1: Dataset distillation methods comparisons. The settings the as previous SoTAs, BPTT [11],
MTT [5], and DREAM [27]. Importantly, MIMADD can work as add-on module SoTA methods.
MNIST
CIFAR10 IPC-10
CIFARI00
IPC-1
IPC-10
IPC-50
IPC-1
IPC-50
IPC-1
IPC-10
Full Set
99.6
84.8 + 0.1
56.2 + 0.3
DD [41]
79.5 8.1
36.8 1.2
LD [3]
6.2
0.3
25.7 0.7
0.4
42.5
11.5
CAFE [40]
0.3
972
9,2
30.3 1.1
0.6
05.
1.30
0.2
DM [44]
0.6
0.1
0..6
0.80
4.6
63.0
11
2.37
DSA [43]
8.6
97.8
0.1
0.7
5.5
60.6
16.8
0.3
DC [46]
91.7
9.2
9.2
0.5
0.5
53.9
0.
2,3
DCC [26]
3.8
0.5
61.6
13.3
30.6
DSAC [26]
3.70
5.55
64.2004
1.36
33.
FRePo [47]
0.5
0.1
9.1
4.5
59.6
63.6#0.2
2..8
312
FRePo-w [47]
0.
0.1
0.0
0.7
65.5
71.7
28.7
4.2
MTT [5]
9.94
9.1
0.1
0.8
6,.3
71.6
0.3
40.1 10.10.4
TESLA [10]
0.8
66.4
72.6=0.7
2.
0.3
MTT
0.94
9.1
98.5
4.
6,73
71,6
2.33
40,1 0.4
MIMADD
0.6
0.2
4.2
66.4
7.34
2.3
0.2
A
(0.61)
(0.87)
(0.41)
(1.31)
(1.11)
(0.21)
(0.81)
(1.41)
BPTT
94.7
98.9 0.1
99.2
49.1
6..4
21.3 0.
3.5
MIMADD A
95,8
98.9 0.1
9..2
0.3
6..4
72,9
2..0
0.6
(L.If)
(0.0-)
(0.0-)
(2.71)
(4.01)
(2.4f)
(3.71)
(3.8t)
DREAM [27]
0..1
6.44
7.18
29.5 0.3
0.7
MIMADD
5..9
70,8 0.1
74.7
31.1 0.4
4.3
(0.87)
(1.41)
(0.11)
(0.61)
(0.6f)
We conduct a series of ablation studies of MIM4DD in CIFAR-10 and CIFAR100. By adjusting the coeffi- cient in the loss function CMIMADD (Eq.2.16 and Eq.2.17), investigate the effect of MIMADD loss in synthesizing distilled datasets. The results are shown in Fig.3. We observe the trend that with À in- creasing, the performance improves, which validates the effectiveness of our designed method. However, when the ratio of LNCF in overall loss is greater than a threshold, the performance of student net- works drops, which means the main task, dataset distillation is overlooked in the optimization.
Table 2: Hyperparameters selection w. 10 Imgs/Cls CIFAR10.
Hyper- pppee-parreeter
Accuracy
critic function w.o network
64.9 + 0.2
critic function w. fc-layer
65.9 + 0.4
critic function w. 2 fc-layer
65.3 + 0.4
8 = 1.0 in LMIMADD
63.8 + 0.6
2.0 in LMIMADD
66.0 + 0.5
8 = 0.5 in LMIMADD
62.8 + 0.6
MIM4DD as an Add-on Module. We apply the MIM4DD framework to state-of-the-art Dataset Distillation (DD) techniques, including MTT [5] and BPTT [11]. The results presented in Table It is observed that MIM4DD effectively enhances the performance of all the tested methods, providing substantial evidence that our approach can be utilized
add-on module to existing techniques.
Hyper-parameters and Relative Module Selection. addition to ablation studies, we conduct experiments to select the important hyper-parameters and modules. We investigate one hyper- parameter, the co-efficient to adjust the weight of layer in overall loss in Eq.2. 17, and one module, the architecture of embedding network in 2q.2.15. The results are in Table 2.
3.4 Regularization Propriety
355
1.5 W//Ahwa
W Myymmmmm
Here, we analyze the trends of dataset distillation loss, LDD in 2.77.17 during training. As we presented in Sec. 2.2, the term LDD Eq.2.17 can be any dataset dis- tillation loss. By controling the co-efficient À in Eq.2.17. 17, we can separately study the DD loss LDD. Their training curves are presented in Fig.4. When turning on NCE loss (assigning À = 0.8), we observe that the LDD drops using designed NCE loss (Left), while the corresponding testing performance improves (see Table 1).
2hp
Step
Figure 4: LDD curves while training w.o.
(left) and w. (right) LNCE.

---

CIFAR100), Combining this phenomenon and analysis on the effect of À in Fig. 3, we conclude that our designed MIMADD module can act as a regularization term.
3.5 CKA analysis
We formulate DD in a view of variable distribution, and thus we analyze distributional information flow within layers of neural networks. Centered kernel alignment (CKA) [9, 20, 32] is able to compare activations within or across networks quantitatively. Specifically, for network fed by m samples, CKA algorithm takes X Rmxpi and Y RmXP2 as inputs which are output activations of two layers (with p1 and P2 neurons respectively). Letting K & XX and L & YY HSIC(K.L)
denote the Gram matrices for the two layers CKA computes: CKA(K, =
KHHSSCCLL where HSIC is the Hilbert-Schmmn independence criterion [14]. Given the centering matrix H _11 1 and the centered Gram matrices K' НKK and L' HLH, HSIC vee( (ee the similarity between these centered Gram matrices. Therefore, we employ Centered Kernel Alignment (CKA) to delve into the information interplay between real and synthetic data. Specifically, we input two datasets into two correspondingly trained networks and examine the similarity between the networks' feature maps using CKA. The findings are depicted in Fig.5. The CKA heatmap reveals that the dataset distilled via MIMADD shares more information with the real dataset compared to the MTT [5]. This is because, in DD formulation, the output similar- ity is supposed to be highly related to the data similarity (Theorem 1, E).
I,
[T -1
MIT
MRA4DD
Layen' Datser
Layen Convlett Carass
Figure 5: CKA analyzes the information shared within different datasets (Real v.s. Synthetic with 10 Img/Cls The lighter the dot, the more similar the two corresponding layers learned from differ- ent datasets. Higher similar score between two layers' output represents those lay- share more information.
4 Related Work
Dataset Distillation is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Therefore, we propose introducing well-defined metric in information theory, mutual information (MI), to guide the optimization of synthetic datasets.
The formulation of our method for DD, MIMADD absorbs the idea of contrastive learning (i.e., constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of Memory [42] for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset our task. (Further details can be found in the Appendix.)
Conclusion
In this paper, explore that well-defined metrics which measures the amount of shared information between variables in information theory for dataset distillation. Specifically, we introduce MI the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset.
Acknowledgements
This work is supported by NSF IIS-2309073 and NSF SCH-2123521. This article solely reflects the opinions and conclusions of its authors and not the funding agency.

---

References
[1] Pankaj K Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Approximating extent measures of points. Journal of the ACM, 2004.
[2] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. NeurlPS, 2019.
[3] Ondrej Bohdal, Yongxin Yang. and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images, arXiv preprint arXiv:2006.08571 08572, 2020.
[4] Leon Brillouin. Science and information theory. Courier Corporation, 2013.
[5] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
[6] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, 2017.
Liqun Chen, Dong Wang. Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. CVPR, 2021.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[9] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. J,I.R, 2012.
[10] Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- 1k with constant memory. arXiv preprint arXiv:2211.10566, 2022.
[11] Zhiwei Deng and Olga Russakovsky, Remember the past: Distilling datasets into addressable memories for neural networks. NeurIPS, 2022.
[12] Dan Feldman. Introduction core-sets: updated survey. arXiv preprint arXiv:2011.09338 2020.
[13] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, IJCV, 2021.
[14] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf. and Alex Smola. A kernel statistical test of independence. In NewrIPS, 2007.
[15] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
[16] Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurIPS, 2014.
[18] R Devon Hjelm, Alex Fedorov, Samuel Lavoie Marchilnar,,,,,n,, Karan Grewal, Phil Bachman, Adam Trischler. and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018
[19] Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. JCML, 2020.
[20] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
[21] Alexander Kraskov, Harald Stöghauer, and Peter Grassherger. Estimating mutual information. Physical review E, 2004.
[22] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[23] Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 1998.
10

---

[25] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
[26] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML,, 2022.
[27] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In ICCV, 2023.
[28] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In ICMI,, 2015.
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: arXiv:/807.03777 2018.
[30] Ben Poolc, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[32] Maithra Raghu, Thomas Unterthiner, Simon Komblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurlPS, 2021.
[33] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint
arXiv:1609.04747 2016.
[34] Ozan Sener and Silvio Savarese, Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
[35] Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applicd house numbers digit classification. In ICPR, 2012.
[36] Yuzhang Shang, Dan Xu, Ziliang Zong, and Yan Yan. Network binarization via contrastive learning. In ECCV, 2022.
[37] Yuzhang Shang, Bingxin Xu, Gaowen Liu, Ramana Rao Kompella, and Yan Yan. Causal-dfq; Causality guided data-fre network quantization. In ICCV, 2023.
[38] Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In IJCNN, 2021.
[39] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2021.
[40] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
[41] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10953 2018
[42] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via pon-maraettiic instance discrimination. In CVPR, 2018.
[43] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In JCML, 1021.
[44] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
[45] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Datasct condensation with gradient matching. In ICLR, 2021.
[46] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 2021.
[47] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Datasct distillation using neural feature regression. arXiv preprint arXiv:2206.007719 2022.

---

A Appendix
A.1 In-variance of Mutual Information
Theorem (In-variance of Mutual Information): Mutual information is invariant under reparametrization of the marginal variables. If X' F(X) and Y' G(Y) are homeomorphisms (i.e., F-) and G(-) are smooth uniquely invertible maps), then
- (IXXXXX))) Y")
(A.1)
Proof. If =X"=F((X and Y G(Y are homeomorphisms (smooth and uniquely invertible maps), and Jx=
8X 8x
ØY and Jy 8Y?
are the Jacobi determinants, then
('''' (x') JJy (v/ (((,,)))
(A.2)
and similarly for the marginal densities, which gives
[] dy' pl ul .o'... log 1' ,6)
1(x', (XX,Y')
1 ('(a"))' (1)(()))
[] dædy)((x, v) log p(x, v)
(A.3)
Ma (z(x))y(()
(XX Y).
More details can be found in [21].
Discussion on Theorem
Our objective to maximize the Mutual Information (MI) between the synthetic dataset and the real dataset (Eq. 3), task that is numerically unfeasible. To overcome this challenge, we present this theorem. It allows us to transform the target problem at the data level (Eq. 3) into more manageable problem at the feature map level (Eq. 9). Given that each layer's mapping W Rd ب 12 (k 1..., K)) in the network (as per Eq. 4, 5, and 6) can be treated as smooth, uniquely invertible maps, we can achieve the goal of maximizing the mutual information between the two datasets. This is done by maximizing the mutual information between two sets of down-sampled feature maps.
A.2 Datasets and Implementation Details
A.2.1 Datasets
MNIST [24] is dataset for handwritten digits recognition that is widely used for validating image recognition models. contains 60,000 training images and 10,000 testing images with the size of 28 28.
CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of 32 32 from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing.
A.2.2 Implementation Details.
In the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of convolutional layer has 128 channels, and AvgPool represents a 2 x 2 average pooling operation with stride The synthetic images' initial learning rate is 0.1, which is halved at the 1,800th and 2, 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and À in Eq. 17 is 0.8. The effect of À is explored in Sec.3.3.
Each
12

---

Figure (Left) Samples from CIFARIO; (Right) Samples from Synthetic dataset based on CIFARIO. We observe that the heterogeneity in the generated images enhanced, benefited from the contrastive learning loss (Loss LMIMADD in 17.
A.3 Synthetic Samples Visualization.
B
Related Work
Dataset Distillation (DD) is firstly introduced by Wang al. [41], in which they optimize the distilled images using gradient-based hyperparameter optimization [28]. The key problem is to optimize the specific-desinned metrics of networks on real and synthetic datasets to update the optimizable images. Subsequently, several works significantly improve the results by designing different metrics. For example, Bohdal er al.and Sucholutsky er al. [3, 38] use distance between networks' soft labels; Zhao al. [46] define the gradients of networks metric; Zhao et al. [43] further adopts augmentations to enhance alignment ability; Wang el al. [40] utilize distance of network feature maps metrie; and Cazenavette [5] propose long-range trajectory to construct the metric function. Lee et al. [26] propose Dataset Condensation with Contrastive Signals (DCC) by modifying the loss function to enable the DC methods to effectively capture the differences between classes. the other hand, researchers take DD as a bi-level optimization problem. For example, Zhou et al. [47] employ closed-fom approximation for the unrolled inner optimization; Deng et al. [11 revisits the optimization framework [41] and observe that the inclusion of a momentum term in inner optimization significantly enhance performance, leading to state-of-the of-the--rttttttt results in certain settings.
DD is essentially compression problem that emphasizes maximizing the preservation of informa- tion contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement, but
13

---

never considered by previous works. Therefore, we propose to introduce well-defined metrie in information theory, mutual information (MI), guide the optimization of synthetic datasets.
Contrastive Learning and Mutual Information. The fundamental idea of all contrastive learning methods is to draw the representations of positive pairs closer and push those of negative pairs farther apart within a contrastive space. Several self-supervised learning methods are rooted in well-established ideas of MI maximization, such as Deep InfoMax [18], Contrastive Predictive Coding [29]. Memory Bank [42], Augmented Multiscale DIM [2], MoCo [16] and SimSaim [8]. These are based on NCE [15] and InfoNCE [18] which can be seen lower bound on MI [30]. Meanwhile, Tian et al. [39] and Chen et al. [7] extend the contrastive concept into the realm of Knowledge Distillation (KD), pulling and pushing the representations of teacher and student.
The formulation of our method for DD, MIM4DD also absorbs the core idea (i.e., constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of Memory Bank [42] for the exponential number of negative pairs in CRD and WCoRD, thanks the small size of the synthetic dataset in our task. Given that the size of the synthetic dataset M typically ranges from 0.1 - 1 1% of the size of the real dataset N, the product M N is significantly smaller than N N (i.e, M
N NN).
Difference with DCC [26]. Recently, Lee et al. [26] introduced Dataset Condensation with Con- trastive Signals (DCC), modifying the loss function allow Dataset Condensation methods to effectively discern differences between classes. However, several distinctions exist between DCC and method: (i) They are motivated differently. Our approach is predicated information degradation, while DCC hinges on class diversity. (ii) From the perspective of contrastive learning, the view, positive and negative samples differ considerably. Our approach can be implemented the feature map level, thanks to the introduced Theorem 1, while DCC can only be deployed at the gradient level. (iii) The performance of our method significantly surpasses that of DCC.
C Codes
Codes can be found anomalously in Supplement.
14


Reviews:
[{"review_id": "8ko67iEvyL", "reviewer": "Reviewer_BhBC", "summary": "The paper proposes a new method named: MIM4DD that tries to maximize the mutual information shared between synthetic images and real images during the process of dataset distillation.\n\nThe paper derives a lower bound of MI and formulates it as a learning objective for optimization. The proposed new loss can be combined with a wide varieties of DD methods to boost performance.\n\n==========\n\nI have read the author's responses and they have addressed my concerns by running more experiments and showing more proof.\n\n==========", "questions": "- It's hard to interpret the performance boost in Table 1, what are the hyperparameters such as $\\lambda$ and $\\beta$ used in MTT + MIM4DD and BPTT+MIM4DD?\n- Please also see my comments in weakness", "limitations": "- The derivation seems incorrect\n- The method introduces strong scalability limitations such as the number of negative pairs and the per-layer matching loss.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes a new way of boosting the performance of DD methods by measuring the mutual information between synthetic datasets and real datasets\n- This paper studies the mutual information and formulates it mathematically which is largely ignored by a lot of previous methods.\n- Through approximation, the paper derives a lower bound of an accessible MI loss and proposes to tackle it with contrastive learning. \n- The paper is well organized and easy to follow\n- The comparison is thorough such as Table 1.", "weaknesses": "- The foundation of MIM4DD seems incorrect: MI in the targeted data level is equivalent ot MI in the feature maps level. From equation 5, F and G may not be invertible which heavily depend on the activation function. Therefore theorem 1 doesn't apply. \n- The evaluations results are mostly within the variance of the baseline methods such as MTT, there is no strong evidence that the proposed method works.\n- Strong limitations are introduced such as 50K negative pairs which could make the method infeasible to apply to larger datasets. \n- Writing errors such as \"to encounter this obstacle\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Weakness 1 and Limitation 1: Overall invertibility of entire neural networks is well-supported.\nThank you for raising this concern. It's essential to clarify that while individual modules of neural networks might not be inherently invertible, the overall invertibility of entire neural networks is well-supported in the literature. Here's a brief overview:\n\n- Dosovitskiy and Brox [1] demonstrated the possibility of inverting the hidden activations of feedforward CNNs back to the input domain using upsampling deconvolutional architectures.\n- Zhang et al. [2] provided evidence that commonly used CNN architectures like VGGNet and ResNet are almost fully invertible, especially when leveraging pooling switches.\n- Gilbert et al. [3] presented both theoretical and empirical evidence for the invertibility of entire neural networks. Their theoretical explanations were rooted in compressive sensing, and they corroborated these findings with practical analyses on several learned networks.\n\nIn the realm of mutual information in neural networks, this property of overall invertibility is not an oversight but a foundational premise. As a prime example, the information bottleneck theory [4,5,6] emphasizes encoding input data into a compressed representation that maximizes target prediction. The theory is contingent upon minimizing the mutual information between input variables and their latent representations, while simultaneously maximizing the mutual information between the output and these latent representations. Several subsequent studies have employed this principle, implicitly assuming the invertibility of entire neural networks, to shed light on the intricacies of neural network operations.\n\nIn summary, the assumption of invertibility at the level of entire neural networks is not only well-founded but also pervasive in the literature exploring mutual information dynamics within these networks. \n\n[1] Inverting visual representations with convolutional networks. In CVPR, 2016. \n\n[2] Augmenting neural networks with reconstructive decoding pathways for large-scale image classification. In ICML 2016\n\n[3] Towards Understanding the Invertibility of Convolutional Neural Networks. \n\n[4] Opening the black box of deep neural networks via information\n\n[5] MI Neural Estimation, ICML, 2018 \n\n[6] Deep VIB, ICLR, 2017\n## Response to Weakness 2: Results discussion. \nIt's essential to frame our results within the broader context of advancements in dataset distillation (DD):\n\n**Comparative Performance:** Our method consistently surpasses 12 recent DD methods from esteemed conferences like NeurIPS, ICLR, ICML, and CVPR. We emphasize that many of these prior studies have only achieved approximately 1-2% accuracy improvements over their predecessors.\n\n**Relative Improvement:** Absolute improvement might sometimes be deceptive. For instance, if there's just a 3% accuracy gap between real dataset and distilled dataset, expecting a 10% or even 5% absolute improvement is impractical. Notably, our method MIM4DD demonstrates a 13% average relative improvement over the second-best method (BPTT, NeurIPS’2022). Importantly, MTT and BPTT are recently the strongest baselines. \n\n**Peer Validation:** Other reviewers have also acknowledged our method's superiority. For instance:\n- Reviewer sR5S observed, \"Significant improvement over the baseline (see Figure 3)\"\n- Reviewer pJjQ highlighted, \"improves the performance of some methods as an add-on module\".\n- Reviewer 7GDB appreciated that \"Remarkable performance improvements have been achieved when plugging the proposed loss into state-of-the-art distillation methods\".\n\nWe believe it's crucial to view performance advancements not just in isolation, but in relation to the current boundaries of the field. \"\n\n## Response to Weakness 3 and Limitation 2: Scalability Discussion. \nThank you for pointing out concerns related to scalability. I'd like to clarify a few things:\n\n**Contrastive Learning Scalability:** As elaborated in Appendix (A. L71-74), our MIM4DD method leverages fundamental principles from contrastive learning, particularly the ideas central to contrastive KD methods like CRD [A18] and WCoRD [A4]. These established methods offer a framework for managing a vast number of negative samples through a solution called MemoryBank [A21]. This system allows for computationally efficient processing of even millions of negative samples.\n\n**Distinctiveness of Our Approach:** Despite the shared inspiration, MIM4DD holds significant differences from the aforementioned methods: Our mutual information target and the resulting numerical formulation diverge substantially; our method can further decrease the cost of MemoryBank for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. Given that the size of the synthetic dataset $M$ typically ranges from $0.1-1$% of the size of the real dataset $N$, the product $M\\cdot N$ is significantly smaller than $N\\cdot N$ (i.e., $M\\cdot N \\ll N\\cdot N$).\n\n**Efficiency in Practice:** In actual implementations, the overhead introduced by MIM4DD is minimal. Our evaluations show that the additional training time required due to MIM4DD is a mere 3%.\n\nIn essence, while we acknowledge the scalability challenge, our method effectively leverages established techniques and unique task characteristics to remain feasible for large datasets.\n\n## Response to Question 1: $\\lambda$ and $\\beta$ in Table 1. \nWe have detailed the selection process for hyperparameters \n$\\lambda$ and $\\beta$ in Sec.3.3 (L280-295). For clarity, based on our empirical evaluations, we set $\\lambda= 1$ as illustrated in Figure 3 and $\\beta = 2$ , which can be referenced in the corresponding Table 2. Both of these values were consistently used across all experiments presented in Table 1."}, {"review_id": "ZK2HEIXLDH", "reviewer": "Reviewer_sR5S", "summary": "The authors tackle the dataset distillation task — synthesizing a smaller dataset using which one can train models towards comparable test performance to models trained on the full dataset. Unlike the current methods that optimize through heuristic matching between the real and synthetic datasets, the proposed method performs data distribution mutual information maximization.\n\nInterestingly, when it comes down to the implementation, the method looks very much like a regular SimCLR-like contrastive learning formulation. However, the authors provide theoretical groundings on how it is related to the mutual information maximization problem.", "questions": "1. The authors seem to define dataset distillation in slightly different manners in the abstract compared to in the Preliminaries section. In the former case, they said “dataset distillation aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model” whereas in the latter case they said “the goal of dataset distillation is to synthesize a small training set such that models trained on this synthetic dataset can have comparable performance as models (with the same architecture) trained on the large real set”. The nuance makes the two definitions a bit different — the former is helpful for testing acceleration while the latter is helpful for training acceleration. I would recommend the authors to clarify which definition they decide to go with (or maybe both) and ensure consistency.\n\n2. Is there any reason BPTT+MIM4DD is shown but BPTT is not included as a standalone baseline in Table 1?", "limitations": "Nothing to be noted.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is a fairly simple idea and I am surprised that no one has been doing it — verified by a quick literature search. I suppose one reason is that prior researchers who thought of using mutual information encountered the difficulty of attaining an estimate of the data distribution, which the authors worked around (line 135-163).\n\n2. Illustration in Figure 2 is well-made and very informative. \n\n3. The authors provided theoretical grounding for using the “mundane” contrastive learning formation (Figure 2) by showing its optimization target is closely related to the proposed “Accessible Mutual Information”. In some sense this is an interesting explanation of why SimCLR-like contrastive learning (single instance multi-view, positive/negative samples) works so well.\n\n4. Significant improvement over the baseline (see Figure 3).\n", "weaknesses": "1. At the moment, I am unaware of any work showing the theoretical link between mutual information and SimCLR-like contrastive learning. If there exist prior work on this topic, the novelty of this work could be largely mitigated.\n\n2. It might have been better to explore other more sophisticated datasets and architectures beyond the three-layer ConvNet, though not absolutely necessary.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sR5S,\n\nThank you very much for the constructive comments and support.\n\n## Response to Weakness 1: Novelty discussion in the content of contrastive learning.\nActually, we have discussed a more details about dataset distillation and contrastive learning in the Appendix (due to space limit of main paper, we only put the short version related work in the main paper):  \nThe fundamental idea of all contrastive learning methods is to draw the representations of positive pairs closer and push those of negative pairs farther apart within a contrastive space. Several self-supervised learning methods are rooted in well-established ideas of MI maximization, such as Deep InfoMax [A9], Contrastive Predictive Coding [A15], MemoryBank [A21], Augmented Multiscale DIM [A1], MoCo [A8] and SimSaim [A5]. These are based on NCE [A7] and InfoNCE [A9] which can be seen as a lower bound on MI [A16]. Meanwhile, Tian et al. [A18] and Chen et al. [A4] extend the contrastive concept into the realm of Knowledge Distillation (KD), pulling and pushing the representations of teacher and student. \nThe formulation of our method for DD, MIM4DD also absorbs the core idea (i.e., constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD and WCoRD. However, our approach has several differences from those methods: \n- (i) our targeted MI and formulated numerical problem are totally different; \n- (ii) our method can naturally avoid the cost of MemoryBank for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. Given that the size of the synthetic dataset $M$ typically ranges from $0.1-1$% of the size of the real dataset $N$, the product $M\\cdot N$ is significantly smaller than $N\\cdot N$ (i.e., $M\\cdot N \\ll N\\cdot N$).\n\n## Response to Weakness 2: Experiments on ImageNet subset. \nThank you for pointing out the potential benefits of exploring more complex datasets and architectures. We acknowledge the importance of demonstrating the versatility of our method, and in fact, we have taken steps in that direction:\n\n- **Expanded Dataset:** We applied our method to ImageNet subsets, focusing on a limited number of classes, using the MTT codebase as a foundation.\n- **Increased Resolution & Depth:** Given the intricacies of high-resolution images, we tested our method on 128×128 subsets of ImageNet. Accommodating this higher resolution necessitated an architectural adaptation, leading us to employ a depth-5 ConvNet for these experiments.\n- **Specific Subsets:** ImageNette (assorted objects) and ImageWoof (dog breeds) are existing subsets designed to be easy and hard to learn respectively. \n- **Results:** The outcomes of these experiments are showcased in Table 1 of the supplementary one-page PDF. Encouragingly, our method continued to demonstrate its effectiveness and superiority across these tests.\n\n## Response to Question 1: Clarification for Definition of Dataset Distillation. \nQuestion 1: In the former case, they said “dataset distillation aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model” whereas in the latter case they said “the goal of dataset distillation is to synthesize a small training set such that models trained on this synthetic dataset can have comparable performance as models (with the same architecture) trained on the large real set”. The nuance makes the two definitions a bit different — the former is helpful for testing acceleration while the latter is helpful for training acceleration. \n\nThe definition of dataset distillation is the latter. Dataset distillation is helpful for training acceleration. For the former, we intend to say \"dataset distillation aims to synthesize a small dataset where test performance of models trained on the small dataset is comparable to a full dataset using the same model\", which is the same meaning of the latter. \n\nWe recognize the potential for confusion and will ensure that the definitions are consistent and clear throughout our paper in future iterations.\n\n## Response to Question 2: Detailed explanation about Table 1. \nThank you for raising this point about the inclusion of BPTT in Table 1. To clarify, BPTT is indeed present as a standalone baseline. You can find its performance metrics on the third-last line, labeled as BPTT [11]. The subsequent line shows the performance when combined with our method, BPTT+MIM4DD. Finally, the last line depicts the performance improvement attained. We appreciate your attention to detail and will ensure such listings are more prominently highlighted in the future.\n"}, {"review_id": "1CCV3ZdHyO", "reviewer": "Reviewer_pJjQ", "summary": "Dataset distillation aims to synthesize a small dataset with similar test performance to the original full dataset. This paper argues that previous works neglect information theory considerations, and argues that a well-designed information metric between variables is very important. Therefore, it introduces the concept of mutual information maximization and transforms it into a lower bound optimization problem at the feature map level, which improves the performance of some methods as an add-on module.\n", "questions": "1. In some datasets, there are bias and noise samples in the same category. For such datasets, is it necessary to use full original dataset information to maximize mutual information?\n2. How does the module perform on high-resolution datasets such as TinyImageNet and ImageNet Subset? And if ipc is raised to 50 on cifar100, what is the result?\n", "limitations": "see above", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors analyze the previous dataset distillation method from the perspective of information theory for the first time;\n2. The authors provide a rigorous and reasonable mathematical formula derivation for how to convert the maximization of mutual information into the lower bound constraint of the sample feature map representation;\n", "weaknesses": "1. The reason why the method works is not sufficient. The maximum mutual information describes the degree of correlation between two variables from the perspective of information theory, but does this mean that more classification information can be learned from the synthetic images for the model to improve the quality of distillation?\n2. The method is not tested on more popular gradient matching or feature matching based dataset distillation frameworks (e.g. DC, DM, DSA, etc.) as a module.\n3. The NCE loss mentioned in line 307 should correspond to the right of Figure 4.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Weakness 1: Why mutual information should be introduced to dataset distillation (DD).\nThank you for raising this insightful question regarding the connection between mutual information (MI) and the improved quality of distillation.\n\n**Theoretical Background:** At its core, Dataset Distillation (DD) can be conceptualized as a compression problem, with the primary goal being the maximization of preserved information from the original data. To this end, it's imperative to have a robust metric that can measure the degree of shared information between variables – a metric that has been notably absent in previous works. We chose to incorporate MI, a well-established metric in information theory, to steer the optimization of synthetic datasets. The power of MI in the domain of neural networks has been endorsed by the information bottleneck theory [4,5,6]. This theory underscores the principle of encoding input data into a compressed form that optimizes target prediction. Such encoding necessitates minimizing the MI between the input and its latent representation, while concurrently maximizing the MI between the output and this representation. Several works that have built upon this foundational theory have reported improved performance, substantiating the efficacy of MI in the realm of deep learning. More comprehensive discussions on this are available in the related work, Appendix B.\n\n**Empirical Evidence:** We took efforts to corroborate our theoretical assertions with empirical validations, particularly the examination of information flow through CKA analysis (Sec. 3.5). Responding to your astute query, we conducted an additional experiment during the rebuttal phase. We visualized the MI between the real and synthetic datasets, $I(D_{sys}, D_{real})$, utilizing methodologies from references [1-3]. The findings, which are illustrated in **Figure 1 of the supplementary one-page pdf**, reveal that datasets produced by our method exhibit higher MI and yield better accuracy performance. This directly supports the hypothesis that enhanced MI can indeed refine the quality of distillation.\n\n[1] Opening the black box of deep neural networks via information [2] MI Neural Estimation, ICML, 2018 [3] Deep VIB, ICLR, 2017\n\n## Response to Weakness 2: Comparisons to DC, DM, DSA.\nThank you for your observation regarding the inclusion of popular gradient matching or feature matching-based dataset distillation frameworks in our evaluations.\n\nIn our evaluation presented in Table 1, we have indeed compared our approach with a broad selection of recent DD methods from esteemed conferences like NeurIPS, ICLR, ICML, and CVPR. This includes the gradient matching or feature matching-based dataset distillation frameworks you mentioned, such as DC, DM, and DSA.\n\nIt's essential to highlight that while DC, DM, and DSA were indeed state-of-the-art two years ago, the landscape has evolved. Recent works like MTT (CVPR2023) and BPTT (NeurIPS2022) now represent the new state-of-the-art in this domain. As evidenced in Table 1, MTT and BPTT surpass the performance of the earlier methods, including DC, DM, and DSA, by substantial margins - more than 10% absolute accuracy across all benchmarks. Furthermore, our method consistently outperforms all 12 recent DD methods, emphasizing its efficacy and relevance in the current landscape.\n\nDue to the status of DD, we choose to add MIM4DD on MTT and BPTT instead of DC, DM, and DSA.\n\n## Response to Question 1: Dataset usage.\nIndeed, your observation regarding datasets that might contain bias and noisy samples within the same category is valid. In such cases, our approach still utilizes the entire dataset as input. Here's why:\n\nDataset Distillation (DD) can fundamentally be viewed as a data compression problem, where the primary objective is to retain as much pertinent information from the original dataset as possible. In the context of deep learning, compression doesn't typically involve filtering out data. Instead, it focuses on representing data in a more concise form while retaining its essence. This principle applies to our method as well. By using the entire dataset, including its biases and noisy samples, we aim to ensure that the distilled dataset is a genuine representation of the original, capturing all its intricacies and nuances.\n\n## Response to Question 2: Experiments on ImageNet subset. \nThank you for pointing out the potential benefits of exploring more complex datasets and architectures. We acknowledge the importance of demonstrating the versatility of our method, and in fact, we have taken steps in that direction:\n\n- **Expanded Dataset:** We applied our method to ImageNet subsets, focusing on a limited number of classes, using the MTT codebase as a foundation.\n- **Increased Resolution & Depth:** Given the intricacies of high-resolution images, we tested our method on 128×128 subsets of ImageNet. Accommodating this higher resolution necessitated an architectural adaptation, leading us to employ a depth-5 ConvNet for these experiments.\n- **Specific Subsets:** ImageNette (assorted objects) and ImageWoof (dog breeds) are existing subsets designed to be easy and hard to learn respectively. \n- **Results:** The outcomes of these experiments are showcased in Table 1 of the supplementary one-page PDF. Encouragingly, our method continued to demonstrate its effectiveness and superiority across these tests.\n\nFor the IPC=50 experiment on cifar100, conducting it poses significant hardware challenges due to the immense v-ram requirement. Specifically, it would demand around 200G of v-ram, which translates to approximately 10 GPUs of the 3090 or A6000 caliber. This requirement surpasses our current hardware capabilities. Additionally, it's worth noting that most Dataset Distillation (DD) research in academia, to our knowledge, does not typically conduct experiments with IPC=50 on cifar100 due to similar constraints.\n\n\n"}, {"review_id": "qxu2wLcv1n", "reviewer": "Reviewer_7GDB", "summary": "This paper proposes mutual information maximization loss for dataset distillation. Specifically, the authors compute the mutual information between real and synthetic feature distribution in multiple layers by constructing positive and negative pairs. Experiments show that when plugging-in state-of-the-art dataset distillation methods, obvious performance improvements are achieved in multiple datasets.", "questions": "Typo:\n\n1.\tLine 135 D_real*  ->  D_real\n\n2.\tLeft of Figure 2: Synthetic Data  ->  Real Data\n", "limitations": "Not addressed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of introducing mutual information into dataset distillation is natural and under-explored. It will be an interesting work to the researchers in this field.\n2.\tThe paper is well-written and technically solid. Necessary analysis on the learned synthetic data has been provided. \n3.\tRemarkable performance improvements have been achieved when plugging the proposed loss into state-of-the-art distillation methods. \n", "weaknesses": "Incomplete experiment results: \n\na) As a loss function, this paper lacks the experiment results on training with MIM4DD loss independently. \n\nb) The ablation study on beta in Table 2 is incomplete, as larger beta has not been tested. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7GDB\n\nThank you for the very constructive comments and support.\n\n## Response to Weakness 1: Training with MIM4DD loss independently.\nThank you for raising this concern. In response, we conducted supplementary experiments where we trained solely with the MIM4DD loss, without integrating it into any other methods. The results are as follows (for a detailed breakdown of the experimental settings, please refer to Table 1 in the main paper).\n\n| Method | CIFAR10 IPC-1 | CIFAR10 IPC-10 | CIFAR10 IPC-50 | CIFAR100 IPC-1 | CIFAR100 IPC-10 | \n|-------|-------|-------|-------|-------|-------|\n| only MIM4DD | 51.1 ± 0.2 | 63.2 ± 0.4 | 72.0 ± 0.2 | 24.8 ± 0.5 | 36.2 ± 0.8|\n\nOur standalone MIM4DD performance was slightly below its results when utilized as a plug-and-play module, while it still shows its performance superiority. We surmise this marginal performance drop is due to the inherent overfitting tendencies associated with dataset distillation. Furthermore, the regularization property inherent to our method might also play a role in this observation (see Sec. 3.4  Regularization Propriety L300-309 in the main paper).\n\n\n## Response to Weakness 2: Larger $\\beta$ study. \nThank you for highlighting this oversight. We have further expanded our ablation study on the parameter $\\beta$. The additional results can be referenced in the following table (other settings are consistent with Table 2). From our extended analysis, we found that the optimal value for $\\beta$ is indeed 2. \n\n| $\\beta$ | Accuracy |\n|-------|-------|\n| 0.5 | 62.8 ± 0.6 |\n| 1.0 | 63.8 ± 0.6 |\n| 2.0 | **66.0 ± 0.5** |\n| 4.0 | 64.9 ± 0.4 |\n\n## Response to Questions: Typos. \nThanks for pointing out the typos, and we will revise the manuscript. \n\n\n"}]

Author rebuttal:
Here is the one-page pdf for submitting the experiment figure and table.

Meta-review: The manuscript has been reviewed by 4 reviewers. Specifically, most reviewers find the idea, despite being simple, well motivated and effective. The reviewers gave initial comments, based on which the authors gave a rebuttal.

After the rebuttal, all the reviewers were mostly satisfied and all converged on the positive side. There is therefore no basis to overturn the consensus. The AC recommends the acceptance.
Decision: Accept (poster)