Title: MIM4DD: Mutual Information Maximization for Dataset Distillation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dataset Distillation
Abstract: Dataset distillation (DD) aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model. State-of-the-art (SoTA) methods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig.1, Left), such as gradients and training trajectories. DD is essentially a compression problem that emphasizes on maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement, but are never considered by previous works. Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets who share the same labels as positive pairs, and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As a result, the targeted MI can be transformed into a lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that MIM4DD can be implemented as an add-on module to existing SoTA DD methods.

MIM4DD: Mutual Information Maximization for Dataset Distillation

Abstract

Dataset distillation (DD) aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model. State-of-the-art (SoTA) methods optimize synthetic datasets primarily by matching heuristic indicators extracted from two networks: one from real data and one from synthetic data (see Fig. 1, Left), such as gradients and training trajectories. DD is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Thus, we introduce mutual information (MI) as the metric to quantify the shared information between the synthetic and the real datasets, and devise M IM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset. Specifically, we designate the samples in different datasets that share the same labels as positive pairs and vice versa negative pairs. Then we respectively pull and push those samples in positive and negative pairs into contrastive space via minimizing NCE loss. As a result, the targeted MI can be transformed into a lower bound represented by feature maps of samples, which is numerically feasible. Experiment results show that M IM4DD can be implemented as an add-on module to existing SoTA DD methods.

1 Introduction

Deep learning has remarkably successful performance in computer vision tasks [25, 6], but most deep-learning-based methods require enormous amounts of data followed by extensive training resources. For example, in order to train a CLIP model [31] in a self-supervised manner [8], a highresolution dataset with more than 10 million images are collected for training, which consumes tens of hundreds of GPU hours.

Loss

Matching

Loss

Optimize

Mutual

Information

Real

Dataset

Synthetic

Dataset

Loss

Optimize

Synthetic

Dataset

Real

Dataset

Forward Propagation

Network

Distribution

In the era of deep learning, in contrast to

A straightforward solution to eliminate the reliance on training DNNs on data is to construct small training sets. Before the deep learning era, coreset or subset selection is the most prevalent paradigm, in which one can obtain a subset of salient data points to represent the original dataset of interest [1, 12, 34].

previously mentioned selection-based methods, Dataset Distillation (DD), also known as Dataset Condensation [41, 46, 40, 5], offers a revolutionary paradigm to synthesize a small dataset using gradient descent algorithms [33], as shown in Fig.1 (Left). In DD, the key question is how to define metrics to measure the distance between synthetic and real datasets. Only by optimizing a well designed distance metric can gradient descent be properly applied to update synthetic data. To answer this question, researchers design several distance algorithms. For example, Zhao et al. [46] measure distance between the batch gradients of the synthetic samples and original ones; Wang et al. [40] use the similarity between the feature maps extracted by networks on different datasets; Cazenavette et al. [5] resort to the MSE between the training trajectories of networks on real and synthetic datasets.

Dataset distillation task is essentially a compression problem with a strong emphasis on maximizing the preservation of information contained in the data [41, 19, 11]. Admittedly, previous heuristicdesigned distance metrics have achieved promising performance. However, we argue that previous works neglect the well-defined distributional metric in information theory, which is necessary for success measurement. Specifically, if we define a dataset's samples as variables, it is imperative that the high-level distributional properties of these variables from synthetic and real datasets ( e.g. , correlations and dependencies between two datasets) should be captured and utilized to guide the update of synthetic datasets. Motivated by this notion, we introduce Mutual Information (MI), a well-formulated metric in information theory for dataset distillation. In detail, MI quantifies the information amount shared by the real and synthetic datasets. In contrast to the aforementioned works focusing on aligning the indicators extracted by different neural networks, MI can naturally capture non-linear statistical dependencies between variables and be used as a measure of true dependence, which is important information in data compression [4, 39, 36].

Based on MI metric, we propose a novel method, termed Mutual Information Maximization for Dataset Distillation ( M IM4DD ). In particular, we first formulate DD as a problem of MI maximization between two datasets. Then, we derive a numerically feasible lower bound and maximize this lower bound via contrastive learning [15, 18, 8]. Finally, we design a highly effective optimization strategy for the dataset distillation task using contrastive estimation for MI maximization. In this way, contrastive learning theoretically leads to the targeted MI maximization and also contributes to the inter-class decorrelation of synthetic samples. In other words, M IM4DD is built upon a contrastive learning framework to synthesize the small dataset, where samples from the synthetic dataset are pulled closer to the counterparts with the identical label in the real dataset, and pulled further away from the ones with different labels in the contrastive space. To the best of our knowledge, it is the first work aiming at MIM of the datasets over the DD task within a contrastive learning framework.

Overall, the contributions of this paper are three-fold: (i) To distill information from a large real dataset to a small synthetic dataset under a well-defined metric, we formulate the DD task as an MI maximization problem. To the best of our knowledge, this is the first work to introduce MI into the literature of DD. (ii) To maximize the targeted MI, we derive a numerically feasible lower bound and maximize it via contrastive learning. In this way, the heterogeneity in the generated images gets enhanced, which further improves the performance of the DD method. (iii) Experimental results show that our method outperforms existing SoTA methods. Importantly, our method can be implemented as a plug-and-play module for existing methods.

2 Methodology

In this section, we demonstrate the methodology of Mutual Information Maximization for Dataset Distillation ( M IM4DD ). Firstly, we briefly revisit the general framework in previous SoTA DD methods, and we illustrate the MI background. Secondly, we model the DD problem within the literature on mutual information maximization (MIM). Then, we convert the targeted but numerical inaccessible MIM goal into a learnable object optimization problem, and solve the problem via a newly designed M IM4DD loss. Finally, we discuss the potential insights of M IM4DD . Note that we only elaborate on the key derivations in this section due to the space limitation; detailed discussions and technical theorems can be found in the supplemental material.

2.1 Preliminaries

Dataset Distillation. In short, the goal of dataset distillation (also dubbed dataset condensation) is to synthesize a small training set, D syn = { x j syn , y j } j M =1 such that models trained on this synthetic

dataset can have comparable performance as models (with the same architecture) trained on the large real set, D real = { x i real , y i } N i =1 , in which M ≪ N . In this way, after obtaining the targeted synthetic data, the training process of DNNs can be largely accelerated. In this premise, we review some representative DD methods [43, 5, 40], which generally propose to enforce the behaviors of models trained on real and synthetic datasets to be similar. The core idea can be formalized in a form of an optimization problem:

in which θ ⋆ and γ ⋆ are the learned parameters of networks trained on the real dataset D real and the synthetic dataset D syn , respectively; and the distance function Dist ( · , · ) is specifically developed to measure the similarity of two networks across datasets. With aligning the networks, one can optimize the synthetic data. Note that designing the distance functions aligning the networks trained on different datasets to optimize the synthetic dataset is the core of previous DD methods, e.g. , CAFE [40] uses the MSE distance of feature maps, and MTT [5] calculates the difference of training trajectories as distance functions.

Although those heuristically designed distance functions lead to promising results, we argue that well-defined metrics in information theory for measuring the amount of shared information between variables have never been considered, as DD is essentially a compression problem with a different emphasis on the information contained in the data.

Mutual Information and Contrastive Learning. Mutual information quantifies the amount of information obtained about one random variable by observing the other random variable. It is a dimensionless quantity with (generally) units of bits, and can be considered as the reduction in uncertainty about one random variable given knowledge of another. High mutual information indicates a large reduction in uncertainty and vice versa [23]. Strictly, for two discrete variables X and Y , their mutual information (MI) can be defined as [23]:

where P XY ( x, y ) is the joint distribution, P X ( x ) = ∑ y P XY ( x, y ) and P Y ( y ) = ∑ x P XY ( x, y ) are the marginals of X and Y , respectively.

In the content of DD, we would like them to share as much information as possible. Theoretically, considering the samples in real and synthetic datasets as two random variables, the MI between those two variables should be maximized. Recently, contrastive learning has been proven an effective approach to maximize MI. Many methods based on contrastive loss for self-supervised learning are proposed, such as [18], [29], [42]. These methods are theoretically based on NCE [15] and InfoNCE [18]. Essentially, the key concept of contrastive learning is to pull representations in positive pairs close and push representations in negative pairs apart in a contrastive space. Thus the major obstacle for modeling problems in a contrastive way is to define the negative and positive pairs. In this work, we also resort to a contrastive learning framework for maximizing our targeted MI. Meanwhile, we illustrate how we formulate DD as a MI maximization problem, and how we solve this targeted problem within the contrastive learning framework.

2.2 MIM4DD: Mutual Information Maximization for Dataset Distillation

In this section, we first formalize the idea of maximizing the MI between the distributions of real and synthesized data, via constructing a contrastive task based on Noise-Contrastive Estimation (NCE). Specifically, we derive a novel M IM4DD loss to distill task-specific information from real data D real to synthetic data D syn , where NCE is introduced to avoid the direct MI computation by estimating it with its lower bound in Eq.2.14. From the perspective of NCE, straight-forwardly, the real and synthesized samples from the same class can be pulled close, and samples from different classes can be pushed away, which corresponds to the core idea of contrastive learning.

Problem Formulation: Ideally, for variable X real representing the samples in real data and X syn in the synthetic data, we desire to maximize the MI between X real and X syn in terms of X syn , i.e. ,

In this way, synthetic data D ⋆ syn and fixed real data D ⋆ real can share maximal information. However, directly approaching this goal is unattainable since the distributions of datasets themselves are absurd to estimate [15, 18]. To encounter this obstacle, let us include the networks trained on real and synthetic datasets. We define those networks in the form of K -layer Multi-Layer Perceptrons (MLPs). For simplification, we discard the bias term of those MLPs. Then the network f ( x ) can be denoted as:

where x is the input sample and W k : R d k -1 ↦-→ R d k ( k = 1 , ..., K ) stands for the weight matrix connecting the ( k -1) -th and the k -th layer, with d k -1 and d k representing the sizes of the input and output of the k -th network layer, respectively. The σ ( · ) function performs element-wise activation operation on the input feature maps. Based on those predefined notions, the sectional MLP f k ( x ) with the front k layers of the f ( x ) can be represented as:

The k -th layer's feature maps are A k syn and A k real , ( k ∈ { 1 , · · · , K } ) , where A k syn = ( a k, 1 syn , · · · , a k,M syn ) and A k real = ( a k, 1 real , · · · , a k,N real ) can be considered as a series of variables. Specifically, the feature map can be obtained by:

Here, we show the relationship of MI in data X level and in feature maps A level, i.e. , the relationship between I ( X real , X syn ) and I ( A real , A syn ) . We utilize the in-variance of MI to understand it. The property can be illustrated as follows:

Theorem 1 (In-variance of Mutual Information): Mutual information is invariant under the reparametrization of the marginal variables. If X ′ = F ( X ) and Y ′ = G ( Y ) are homeomorphisms ( i.e. , F ( · ) and G ( · ) are smooth uniquely invertible maps), then

Since each layer's mapping W k : R d k -1 ↦-→ R d k ( k = 1 , ..., K ) can be considered as the smooth uniquely invertible maps Theorem 1 . Combining this theorem with the definition of MI in Eq.2.2, we observe that the MI in the targeted data level is equivalent to MI in the feature maps level, i.e. ,

More details and the proof of this theorem are in Supplemental Materials and [21].

Facilitated by this property, we are able to access the calculation of MI between real and synthetic datasets via the feature maps of networks trained on those datasets. In other words, Theorem 1 helps us transfer the targeted MI maximization in Eq.2.3 to a reachable MI in Eq.2.8 in the dataset distillation literature, i.e. ,

in which A k syn = f k ( X syn ) and A k real = f k ( X real ) .

Apart from the theoretical derivation, intuitively, the corresponding variables ( a k syn , a k real ) should share more information, i.e. , MI of the same layer's output feature maps I ( a k syn , a k real ) ( k ∈ { 1 , · · · , K } ) should be maximized to enforce them mutually dependent. This motivation has also been testified from the perspective of KD [17, 13] by CRD [39] and WCoRD [7]. In those methods, the penultimate layer's feature maps of teacher and student are aligned in a contrastive learning manner to enhance the heterogeneity of representations, which can also be explained via MI maximization. However, M IM4DD is different from those methods (details is in Related Work).

To optimize the accessible Mutual Information (MI) as defined in Eq.2.9, we incorporate a contrastive learning framework into our targeted Dataset Distillation (DD) task. The fundamental principle of contrastive learning involves comparing varying perspectives of the data, typically under different data augmentations, to compute similarity scores [29, 18, 2, 16, 8]. This framework is suitable for our case, since the activations from real and synthetic datasets can be seen as two different views.

Network

Network

Representations

Contrastive

Space

Forward Pass

Synthetic Data

Real Data

Update

Pull

Push

=

̸

̸

(Definition: Positive and Negative Samples.) Let's denote each sample from the real and synthetic datasets as { x j cr real } ( j c r ∈ { 1 , · · · , N } ) and { x i cs syn } ( i c s ∈ { 1 , · · · , M } ) , respectively. Here, c r and c s ∈ { 1 , · · · , C } represent the class labels of x j cr real and x i cs syn , respectively. We feed two batches of samples from two datasets to two different neural networks and obtain N · M pairs of k -th layer's activations ( a k,i cs syn , a k,j cr real ) , which can further be utilized for the contrastive learning task. We define a pair containing two activations corresponding to the same labels as the positive pair, i.e. , if c r = c s , ( a k,i cs syn , a k,j cr real ) + and vice versa . Consequently, there is M C · M C · C = 1 C · N · M positive pairs, and thus (1 -1 C ) · N · M negative pairs. The key concept of contrastive learning is to discriminate whether a given pair of activation ( a k,i cs syn , a k,j cr real ) is positive or negative. In other words, it involves estimate the distribution P ( D | a k,i cs syn , a k,j cr real ) , in which D is a scalar variable indicating whether c s = c r or c s = c r . Specifically, D = 1 when c s = c r , and D = 0 when c s = c r . However, we can not directly reach the distribution P ( D | a k,i cs syn , a k,j cr real ) [15], and thus we introduce its corresponding variational form:

Intuitively, q ( D | a k,j cr real , a k,i cs syn ) can be treated as a binary classifier, which can classify a given pair ( a k,i cs syn , a k,j cr real ) into positive or negative. Importantly, q ( D | a k,i cs real , a k,j cr syn ) can be estimated by some mature statistical methods, such as NCE [15] and InfoNCE [18].

Using the Bayes rule, the posterior probability of two activations from the positive pair can be formalized as:

The probability of activations from negative pair is q ( D = 0 | a k,i cs syn , a k,j cr real ) = 1 -q ( D = 1 | a k,i cs syn , a k,j cr real ) . To simplify the NCE derivative, several works [15, 42, 39] build assumption about the dependence of the variables, we also use the assumption that the activations from positive pairs are dependent and the ones from negative pairs are independent, i.e. q ( a k,i cs syn , a k,j cr real | D = 1) = P ( a k,i cs syn , a k,j cr real ) and q ( a k,i cs syn , a k,j cr real | D = 0) = P ( a k,i cs syn ) P ( a k,j cr real ) . Hence, the above equation can be simplified as:

Performing logarithm to Eq.2.12 and arranging the terms, we can achieve

Taking expectation of P ( a k,i cs syn , a k,j cr real ) on both sides, and combining Eq.2.2, we can transfer the MI into:

where I ( a k syn , a k real ) is the MI between the real and synthetic data distributions. Instead of directly maximizing the MI, maximizing the lower bound in the Eq.2.14 is a practical solution.

However, even q ( D = 1 | a k,i cs , a k,j cr ) is still hard to estimate. Thus, as tackled by many contrastive learning works [29, 18, 2, 39, 36, 37], we introduce a discriminator network d ( · , · ) with parameter ϕ ( i.e. , d ( a k,i cs syn , a k,j cr real ; ϕ ) ). Basically, the discriminator d can map a k syn , a k real to [0 , 1] ( i.e. , distinguish given two samples a k syn , a k real belonging to positive or negative pair). Specifically, the discriminator function is designed as follows:

in which g ϕ ( · ) is the embedding function for mapping the activations into the contrastive space and C = exp( <g ( a k,i c s syn ) ,g ( a k,jcr real ) > τ ) + 1 , and τ is a temperature parameter that controls the concentration level of the distribution [17, 39].

Loss Function. We define the contrastive loss function L k NCE between the k -th layer's activations a k syn and a k real as: L k NCE =

64

65

0

65

65

65

65

64

In the view of contrastive learning, the first term in the above loss function about positive pairs is optimized for capturing more intra-class correlations and the second term of negative pairs is for interclass decorrelation. Because we construct the pairs instance-wisely, the number of negative pairs can be the size of the entire real dataset, e.g. , 50K for CIFAR [22]. By incorporating hand-crafted, additional contrastive pairs for the proxy optimization problem in Eq.2.16, the representational quality of generated images can be further enhanced as demonstrated by numerous contrastive learning methods[8, 29, 18, 2].

O

R

丘

<

LL

-

Q

u

o

>

:::i

e

J

n

:::>

<

0.0

0.5

1.0

1.5

2.0

2.5

3.0

入in Overall Loss Function (Eq.17)

2

0

2

0

1

1

0

0

0

0

0

9

9

O

R

士

<

LL

-

O

u

o

>

:

>e

n

3

<

0.0

0.5

1.0

1.5

2.0

2.5

3.0

入in Overall Loss Function (Eq.17)

66.2

66.0

41.0 -

65.6

65.4

402-

65.2-

65.0

40.0

64.8

64.6

39.6

8

6

Finally, by incorporating the set of NCE loss for various layers { L k NCE } , ( k = 1 , · · · , K ) , we can then formulate M IM4DD loss L M IM4DD as:

in which L DD can be any loss functions in previous DD methods [40, 43, 5], λ is used to control the NCE loss, and β is a scalar greater than 1 . In practice, we use MTT [5] or BPTT [11] as default L DD .

2.3 Discussion on MIM4DD

In addition to the theoretical formulation, we are going to provide a more straightforward manifestation of M IM4DD . As illustrated in Fig.2, by embedding the activations to the contrastive space and constructing proper negative-and-positive embedding pairs, we can heterogeneously distill the relevant information from the real dataset to learnable synthetic dataset, and enhance the heterogeneity of the synthetic dataset within the contrastive learning framework as shown in Fig.5. Specifically, networks on different datasets first learn self-supervised information benefited from a large number of the designed pairs [18, 39]. Then, via the backward pass in Fig.2, the synthetic data are updated

4

4

4

4

6

4

8

4

66

66

66

4

6

8

2

4

6

8

3

3

4

4

w.r.t. this contrastive loss. While the actual number of negative samples in practice can be huge, e.g. , 50K for synthesizing 10 pictures per-class to replace CIFAR10 [22] despite only drawing two samples in Fig.2.

Furthermore, we give a direct explanation of why optimizing L MIM4DD in Eq.2.17 can lead to the targeted MIM between synthetic dataset and real dataset in Eq.2.3. Firstly, optimizing the formulated contrastive loss in Eq.2.16 is equivalent to maximizing the derived accessible MI between activations in Eq.2.9, as the inequality in Eq.2.14 (Eq.2.10-2.13 can derive Eq.2.14). Secondly, based on Theorem 1 (Eq.A.1) and the property of the networks itself (Eq.2.4 and Eq.2.5), maximizing the accessible MI in Eq.2.9 equals to maximizing the targeted MI in Eq.2.8, which is our goal, to minimize the MI between synthetic dataset and real dataset. Since real data are fixed and synthetic data are learnable, the optimized synthetic dataset can contain as much information as the real dataset under the metric of MI.

3 Experiments

In this section, we conduct comprehensive experiments to evaluate our proposed method M IM4DD on four different datasets for DD task. We first describe the implementation details of M IM4DD , and then compare our method with several SoTA DD methods to demonstrate superiority of our proposed method. Finally, we validate the effectiveness of MI module (connected with Eq.2.16 and Eq.2.17) by a series of ablation studies.

3.1 Datasets and Implementation Details

Datasets. We use MNIST [24], SVHN [35], and CIFAR10/100 datasets to conduct our experiments. MNIST [24] is a dataset for handwritten digits recognition that is widely used for validating image recognition models. It contains 60,000 training images and 10,000 testing images with the size of 28 × 28 . CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of 32 × 32 from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing. More details of the datasets can be found in Supplemental Materials .

Implementation Details. In the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using a three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of 'Conv-InstNormReLU-AvgPool.' Each convolutional layer has 128 channels, and AvgPool represents a 2 × 2 average pooling operation with stride 2. The synthetic images' initial learning rate is 0.1, which is halved at the 1,800th and 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and λ in Eq.2.17 is 0.8. The effect of λ is explored in Sec.3.3.

3.2 Comparison with SoTA

We compare M IM4DD with a series of state-of-the-art (SoTA) dataset distillation methods, including Dataset Distillation (DD) [41], LD [3], Dataset Condensation (DC) [46], DC with differentiable siamese augmentation (DSA) [43], DC with distribution matching (DM) [45], CAFE [40], FRePo [47], TESLA [10], BPTT [11], and MTT [5]. We report the performances of our method and comparisons on three datasets in Table 1. Taking into account the overall performance of M IM4DD across these mainstream dataset distillation benchmarks, it's apparent that our method consistently outperforms existing SoTAs. For instance, in the setting of generating 10 images per class, our method delivers top-tier results across all datasets. Additionally, when we synthesize 10 images per class using CIFAR100 as a real-world dataset, our method surpasses MTT by a margin of 1.4%. Importantly, our method can serve as an effective plug-and-play module for existing state-of-the-art DD methods.

3.3 Ablation Studies

We conduct a series of ablation studies of M IM4DD in CIFAR-10 and CIFAR100. By adjusting the coefficient λ in the loss function L M IM4DD (Eq.2.16 and Eq.2.17), we investigate the effect of M IM4DD loss in synthesizing distilled datasets. The results are shown in Fig.3. We observe the trend that with λ increasing, the performance improves, which validates the effectiveness of our designed method. However, when the ratio of L NCE in overall loss is greater than a threshold, the performance of student networks drops, which means the main task, dataset distillation is overlooked in the optimization.

M IM4DD as an Add-on Module. Weapply the M IM4DD framework to state-of-the-art Dataset Distillation (DD) techniques, including MTT [5] and BPTT [11]. The results are presented in Table1. It is observed that M IM4DD effectively enhances the performance of all the tested methods, providing substantial evidence that our approach can be utilized as an add-on module to existing techniques.

Hyper-parameters and Relative Module Selection. In addition to ablation studies, we conduct experiments to select the important hyper-parameters and modules. We investigate one hyperparameter, the co-efficient to adjust the weight of layer in overall loss in Eq.2.17, and one module, the architecture of embedding network in Eq.2.15. The results are in Table 2.

3.4 Regularization Propriety

Here, we analyze the trends of dataset distillation loss, L DD in Eq.2.17 during training. As we presented in Sec. 2.2, the term L DD in Eq.2.17 can be any dataset distillation loss. By controling the co-efficient λ in Eq.2.17, we can separately study the DD loss L DD . Their training curves are presented in Fig.4. When turning on NCE loss (assigning λ = 0 . 8 ), we observe that the L DD drops using

(b)

(a)

our designed NCE loss (Left), while the corresponding testing performance improves (see Table 1).

Combining this phenomenon and analysis on the effect of λ in Fig. 3, we conclude that our designed M IM4DD module can act as a regularization term.

3.5 CKA analysis

We formulate DD in a view of variable distribution, and thus we analyze distributional information flow within layers of neural networks. Centered kernel alignment (CKA) [9, 20, 32] is able to compare activations within or across networks quantitatively. Specifically, for a network fed by m samples, CKA algorithm takes X ∈ R m × p 1 and Y ∈ R m × p 2 as inputs which are output activations of two layers (with p 1 and p 2 neurons respectively). Letting K ≜ XX ⊤ and L ≜ YY ⊤ denote the Gram matrices for the two layers CKA computes: CKA( K , L ) = HSIC( K , L ) √ HSIC( K , K )HSIC( L , L ) , where HSIC is the Hilbert-Schmidt independence criterion [14]. Given the centering matrix H = I -1 11 ⊤ and the centered Gram matrices K ′ = HKH and L ′ = HLH , HSIC = vec( K ′ )vec( L ′ ) ( m -1) 2 , MTT MIM4DD

n n the similarity between these centered Gram matrices. Therefore, we employ Centered Kernel Alignment (CKA) to delve into the information interplay between real and synthetic data. Specifically, we input two datasets into two correspondingly trained networks and examine the similarity between the networks' feature maps using CKA. The findings are depicted in Fig.5. The CKA heatmap reveals that the dataset distilled via M IM4DD shares more information with the real dataset compared to the MTT [5]. This is because, in our DD formulation, the output similarity is supposed to be highly related to the data similarity (Theorem 1, Eq.A.1).

4 Related Work

Dataset Distillation is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Therefore, we propose introducing a well-defined metric in information theory, mutual information (MI), to guide the optimization of synthetic datasets.

The formulation of our method for DD, M IM4DD absorbs the core idea of contrastive learning ( i.e. , constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of MemoryBank [42] for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. (Further details can be found in the Appendix.)

5 Conclusion

In this paper, we explore that well-defined metrics which measures the amount of shared information between variables in information theory for dataset distillation. Specifically, we introduce MI as the metric to quantify the shared information between the synthetic and the real datasets, and devise M IM4DD numerically maximizing the MI via a newly designed optimizable objective within a contrastive learning framework to update the synthetic dataset.

References

Pankaj K Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Approximating extent measures of points. Journal of the ACM , 2004.

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS , 2019.

Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006.08572 , 2020.

Leon Brillouin. Science and information theory . Courier Corporation, 2013.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In CVPR , 2022.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 2017.

Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. In CVPR , 2021.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR , 2021.

Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. JMLR , 2012.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. arXiv preprint arXiv:2211.10586 , 2022.

Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In NeurIPS , 2022.

Dan Feldman. Introduction to core-sets: an updated survey. arXiv preprint arXiv:2011.09384 , 2020.

Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. IJCV , 2021.

Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Sch¨ olkopf, and Alex Smola. A kernel statistical test of independence. In NeurIPS , 2007.

Michael Gutmann and Aapo Hyv¨ arinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS , 2010.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS , 2014.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is np-hard. In ICML , 2020.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML , 2019.

Alexander Kraskov, Harald St¨ ogbauer, and Peter Grassberger. Estimating mutual information. Physical review E , 2004.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Solomon Kullback. Information theory and statistics . Courier Corporation, 1997.

Yann LeCun, L´ eon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 2015.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML , 2022.

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In ICCV , 2023.

Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML , 2015.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML , 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurIPS , 2021.

Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR , 2018.

Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In ICPR , 2012.

Yuzhang Shang, Dan Xu, Ziliang Zong, and Yan Yan. Network binarization via contrastive learning. In ECCV , 2022.

Yuzhang Shang, Bingxin Xu, Gaowen Liu, Ramana Rao Kompella, and Yan Yan. Causal-dfq: Causality guided data-free network quantization. In ICCV , 2023.

Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In IJCNN , 2021.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR , 2021.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR , 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR , 2018.

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML , 2021.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV , 2023.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR , 2021.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR , 2021.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719 , 2022.

A Appendix

A.1 In-variance of Mutual Information

Theorem 1 (In-variance of Mutual Information): Mutual information is invariant under reparametrization of the marginal variables. If X ′ = F ( X ) and Y ′ = G ( Y ) are homeomorphisms ( i.e. , F ( · ) and G ( · ) are smooth uniquely invertible maps), then

Proof. If X ′ = F ( X ) and Y ′ = G ( Y ) are homeomorphisms (smooth and uniquely invertible maps), and J X = ∥ ∂X ∂X ′ ∥ and J Y = ∥ ∂Y ∂Y ′ ∥ are the Jacobi determinants, then

and similarly for the marginal densities, which gives

More details can be found in [21].

Discussion on Theorem 1.

Our objective is to maximize the Mutual Information (MI) between the synthetic dataset and the real dataset (Eq. 3), a task that is numerically unfeasible. To overcome this challenge, we present this theorem. It allows us to transform the target problem at the data level (Eq. 3) into a more manageable problem at the feature map level (Eq. 9). Given that each layer's mapping W k : R d k -1 ↦-→ R d k ( k = 1 , ..., K ) in the network (as per Eq. 4, 5, and 6) can be treated as smooth, uniquely invertible maps, we can achieve the goal of maximizing the mutual information between the two datasets. This is done by maximizing the mutual information between two sets of down-sampled feature maps.

A.2 Datasets and Implementation Details

A.2.1 Datasets

MNIST [24] is a dataset for handwritten digits recognition that is widely used for validating image recognition models. It contains 60,000 training images and 10,000 testing images with the size of 28 × 28 .

CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of 32 × 32 from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing.

A.2.2 Implementation Details.

In the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using a three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of 'Conv-InstNorm-ReLU-AvgPool.' Each convolutional layer has 128 channels, and AvgPool represents a 2 × 2 average pooling operation with stride 2. The synthetic images' initial learning rate is 0.1, which is halved at the 1,800th and 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and λ in Eq.17 is 0.8. The effect of λ is explored in Sec.3.3.

A.3 Synthetic Samples Visualization.

B Related Work

Dataset Distillation (DD) is firstly introduced by Wang et al. [41], in which they optimize the distilled images using gradient-based hyperparameter optimization [28]. The key problem is to optimize the specific-designed metrics of networks on real and synthetic datasets to update the optimizable images. Subsequently, several works significantly improve the results by designing different metrics. For example, Bohdal et al. and Sucholutsky et al. [3, 38] use distance between networks' soft labels; Zhao et al. [46] define the gradients of networks as metric; Zhao et al. [43] further adopts augmentations to enhance the alignment ability; Wang et al. [40] utilize distance of network feature maps as metric; and Cazenavette [5] propose long-range trajectory to construct the metric function. Lee et al. [26] propose Dataset Condensation with Contrastive Signals (DCC) by modifying the loss function to enable the DC methods to effectively capture the differences between classes. On the other hand, researchers take DD as a bi-level optimization problem. For example, Zhou et al. [47] employ a closed-form approximation for the unrolled inner optimization; Deng et al. [11] revisits the optimization framework in [41] and observe that the inclusion of a momentum term in inner optimization can significantly enhance performance, leading to state-of-the-art results in certain settings.

DD is essentially a compression problem that emphasizes on maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement, but are

never considered by previous works. Therefore, we propose to introduce a well-defined metric in information theory, mutual information (MI), to guide the optimization of synthetic datasets.

Contrastive Learning and Mutual Information. The fundamental idea of all contrastive learning methods is to draw the representations of positive pairs closer and push those of negative pairs farther apart within a contrastive space. Several self-supervised learning methods are rooted in well-established ideas of MI maximization, such as Deep InfoMax [18], Contrastive Predictive Coding [29], MemoryBank [42], Augmented Multiscale DIM [2], MoCo [16] and SimSaim [8]. These are based on NCE [15] and InfoNCE [18] which can be seen as a lower bound on MI [30]. Meanwhile, Tian et al. [39] and Chen et al. [7] extend the contrastive concept into the realm of Knowledge Distillation (KD), pulling and pushing the representations of teacher and student.

The formulation of our method for DD, M IM4DD also absorbs the core idea ( i.e. , constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of MemoryBank [42] for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. Given that the size of the synthetic dataset M typically ranges from 0 . 1 -1% of the size of the real dataset N , the product M · N is significantly smaller than N · N ( i.e. , M · N ≪ N · N ).

Difference with DCC [26]. Recently, Lee et al. [26] introduced Dataset Condensation with Contrastive Signals (DCC), modifying the loss function to allow Dataset Condensation methods to effectively discern differences between classes. However, several distinctions exist between DCC and our method: (i) They are motivated differently. Our approach is predicated on information degradation, while DCC hinges on class diversity. (ii) From the perspective of contrastive learning, the view, positive and negative samples differ considerably. Our approach can be implemented at the feature map level, thanks to the introduced Theorem 1, while DCC can only be deployed at the gradient level. (iii) The performance of our method significantly surpasses that of DCC.

C Codes

Codes can be found anomalously in Supplement.

Peer reviews:
comments: SUMMARY:
This paper proposes mutual information maximization loss for dataset distillation. Specifically, the authors compute the mutual information between real and synthetic feature distribution in multiple layers by constructing positive and negative pairs. Experiments show that when plugging-in state-of-the-art dataset distillation methods, obvious performance improvements are achieved in multiple datasets.

STRENGTHS:
1.	The idea of introducing mutual information into dataset distillation is natural and under-explored. It will be an interesting work to the researchers in this field.
2.	The paper is well-written and technically solid. Necessary analysis on the learned synthetic data has been provided. 
3.	Remarkable performance improvements have been achieved when plugging the proposed loss into state-of-the-art distillation methods.

WEAKNESSES:
Incomplete experiment results: 

a) As a loss function, this paper lacks the experiment results on training with MIM4DD loss independently. 

b) The ablation study on beta in Table 2 is incomplete, as larger beta has not been tested.

QUESTIONS:
Typo:

1.	Line 135 D_real*  ->  D_real

2.	Left of Figure 2: Synthetic Data  ->  Real Data

LIMITATIONS:
Not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDataset distillation aims to synthesize a small dataset with similar test performance to the original full dataset. This paper argues that previous works neglect information theory considerations, and argues that a well-designed information metric between variables is very important. Therefore, it introduces the concept of mutual information maximization and transforms it into a lower bound optimization problem at the feature map level, which improves the performance of some methods as an add-on module.\n\nSTRENGTHS:\n1. The authors analyze the previous dataset distillation method from the perspective of information theory for the first time;\n2. The authors provide a rigorous and reasonable mathematical formula derivation for how to convert the maximization of mutual information into the lower bound constraint of the sample feature map representation;\n\nWEAKNESSES:\n1. The reason why the method works is not sufficient. The maximum mutual information describes the degree of correlation between two variables from the perspective of information theory, but does this mean that more classification information can be learned from the synthetic images for the model to improve the quality of distillation?\n2. The method is not tested on more popular gradient matching or feature matching based dataset distillation frameworks (e.g. DC, DM, DSA, etc.) as a module.\n3. The NCE loss mentioned in line 307 should correspond to the right of Figure 4.\n\nQUESTIONS:\n1. In some datasets, there are bias and noise samples in the same category. For such datasets, is it necessary to use full original dataset information to maximize mutual information?\n2. How does the module perform on high-resolution datasets such as TinyImageNet and ImageNet Subset? And if ipc is raised to 50 on cifar100, what is the result?\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the dataset distillation task — synthesizing a smaller dataset using which one can train models towards comparable test performance to models trained on the full dataset. Unlike the current methods that optimize through heuristic matching between the real and synthetic datasets, the proposed method performs data distribution mutual information maximization.\n\nInterestingly, when it comes down to the implementation, the method looks very much like a regular SimCLR-like contrastive learning formulation. However, the authors provide theoretical groundings on how it is related to the mutual information maximization problem.\n\nSTRENGTHS:\n1. It is a fairly simple idea and I am surprised that no one has been doing it — verified by a quick literature search. I suppose one reason is that prior researchers who thought of using mutual information encountered the difficulty of attaining an estimate of the data distribution, which the authors worked around (line 135-163).\n\n2. Illustration in Figure 2 is well-made and very informative. \n\n3. The authors provided theoretical grounding for using the “mundane” contrastive learning formation (Figure 2) by showing its optimization target is closely related to the proposed “Accessible Mutual Information”. In some sense this is an interesting explanation of why SimCLR-like contrastive learning (single instance multi-view, positive/negative samples) works so well.\n\n4. Significant improvement over the baseline (see Figure 3).\n\nWEAKNESSES:\n1. At the moment, I am unaware of any work showing the theoretical link between mutual information and SimCLR-like contrastive learning. If there exist prior work on this topic, the novelty of this work could be largely mitigated.\n\n2. It might have been better to explore other more sophisticated datasets and architectures beyond the three-layer ConvNet, though not absolutely necessary.\n\nQUESTIONS:\n1. The authors seem to define dataset distillation in slightly different manners in the abstract compared to in the Preliminaries section. In the former case, they said “dataset distillation aims to synthesize a small dataset whose test performance is comparable to a full dataset using the same model” whereas in the latter case they said “the goal of dataset distillation is to synthesize a small training set such that models trained on this synthetic dataset can have comparable performance as models (with the same architecture) trained on the large real set”. The nuance makes the two definitions a bit different — the former is helpful for testing acceleration while the latter is helpful for training acceleration. I would recommend the authors to clarify which definition they decide to go with (or maybe both) and ensure consistency.\n\n2. Is there any reason BPTT+MIM4DD is shown but BPTT is not included as a standalone baseline in Table 1?\n\nLIMITATIONS:\nNothing to be noted.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method named: MIM4DD that tries to maximize the mutual information shared between synthetic images and real images during the process of dataset distillation.\n\nThe paper derives a lower bound of MI and formulates it as a learning objective for optimization. The proposed new loss can be combined with a wide varieties of DD methods to boost performance.\n\n==========\n\nI have read the author's responses and they have addressed my concerns by running more experiments and showing more proof.\n\n==========\n\nSTRENGTHS:\n- The paper proposes a new way of boosting the performance of DD methods by measuring the mutual information between synthetic datasets and real datasets\n- This paper studies the mutual information and formulates it mathematically which is largely ignored by a lot of previous methods.\n- Through approximation, the paper derives a lower bound of an accessible MI loss and proposes to tackle it with contrastive learning. \n- The paper is well organized and easy to follow\n- The comparison is thorough such as Table 1.\n\nWEAKNESSES:\n- The foundation of MIM4DD seems incorrect: MI in the targeted data level is equivalent ot MI in the feature maps level. From equation 5, F and G may not be invertible which heavily depend on the activation function. Therefore theorem 1 doesn't apply. \n- The evaluations results are mostly within the variance of the baseline methods such as MTT, there is no strong evidence that the proposed method works.\n- Strong limitations are introduced such as 50K negative pairs which could make the method infeasible to apply to larger datasets. \n- Writing errors such as \"to encounter this obstacle\"\n\nQUESTIONS:\n- It's hard to interpret the performance boost in Table 1, what are the hyperparameters such as $\\lambda$ and $\\beta$ used in MTT + MIM4DD and BPTT+MIM4DD?\n- Please also see my comments in weakness\n\nLIMITATIONS:\n- The derivation seems incorrect\n- The method introduces strong scalability limitations such as the number of negative pairs and the per-layer matching loss.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}