SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\overbrace {I \left(\mathbf {a} _ {\text {s y n}} ^ {k} , \mathbf {a} _ {\text {r e a l}} ^ {k}\right)} ^ {\text {A c c e s s i b l e M I i n E q . 2 . 9}} \geq \log (C - 1) + \overbrace {\mathbb {E} _ {P \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c s}} , \mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c r}} \mid D = 1\right)} \left[ \log q \left(D = 1 \mid \mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}}\right) \right]} ^ {\text {o p t i m i z e d l o w e r b o u n d}}, \tag {2.14}
$$
where  $I(\mathbf{a}_{syn}^k, \mathbf{a}_{real}^k)$  is the MI between the real and synthetic data distributions. Instead of directly maximizing the MI, maximizing the lower bound in the Eq.2.14 is a practical solution.
However, even  $q(D = 1 \mid \mathbf{a}^{k,i_{c_s}}, \mathbf{a}^{k,j_{c_r}})$  is still hard to estimate. Thus, as tackled by many contrastive learning works [29, 18, 2, 39, 36, 37], we introduce a discriminator network  $d(\cdot, \cdot)$  with parameter  $\phi$  (i.e.,  $d(\mathbf{a}_{syn}^{k,i_{c_s}}, \mathbf{a}_{real}^{k,j_{c_r}}; \phi)$ ). Basically, the discriminator  $d$  can map  $\mathbf{a}_{syn}^k$ ,  $\mathbf{a}_{real}^k$  to  $[0,1]$  (i.e., distinguish given two samples  $\mathbf{a}_{syn}^k$ ,  $\mathbf{a}_{real}^k$  belonging to positive or negative pair). Specifically, the discriminator function is designed as follows:
$$
d \left(\mathbf {a} _ {\text {s y n}} ^ {k, i _ {c _ {s}}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c _ {r}}}\right) = \exp \left(\frac {<   g _ {\phi} \left(\mathbf {a} _ {\text {s y n}} ^ {k , i _ {c _ {s}}}\right) , g _ {\phi} \left(\mathbf {a} _ {\text {r e a l}} ^ {k , j _ {c _ {r}}}\right) >}{\tau}\right) / C, \tag {2.15}
$$
in which  $g_{\phi}(\cdot)$  is the embedding function for mapping the activations into the contrastive space and  $C = \exp \left( \frac{<g(\mathbf{a}_{syn}^{k,i_{c_s}}), g(\mathbf{a}_{real}^{k,j_{c_r}})>}{\tau} \right) + 1$ , and  $\tau$  is a temperature parameter that controls the concentration level of the distribution [17, 39].
Loss Function. We define the contrastive loss function  $\mathcal{L}_{NCE}^k$  between the  $k$ -th layer's activations  $\mathbf{a}_{syn}^k$  and  $\mathbf{a}_{real}^k$  as:  $\mathcal{L}_{NCE}^k =$
$$
\mathbb {E} _ {q \left(\mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}} \mid D = 1\right)} \left[ \log h \left(\mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}}\right) \right] + (C - 1) \cdot \mathbb {E} _ {q \left(\mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}} \mid D = 0\right)} \left[ \log \left(1 - h \left(\mathbf {a} _ {\text {s y n}} ^ {k, i _ {c s}}, \mathbf {a} _ {\text {r e a l}} ^ {k, j _ {c r}}\right)\right) \right]. \tag {2.16}
$$
In the view of contrastive learning, the first term in the above loss function about positive pairs is optimized for capturing more intra-class correlations and the second term of negative pairs is for interclass decorrelation. Because we construct the pairs instance-wisely, the number of negative pairs can be the size of the entire real dataset, e.g., 50K for CIFAR [22]. By incorporating hand-crafted, additional contrastive pairs for the proxy optimization problem in Eq.2.16, the representational quality of generated images can be further enhanced as demonstrated by numerous contrastive learning methods[8, 29, 18, 2].
Figure 3: Ablation Studies. Effect of  $\lambda$  in  $\mathcal{L}_{\mathrm{MIM4DD}}$  (Eq.2.17) on CIFAR10 (Left) and CIFAR100 (Right).  $\lambda = 0$  is the baseline, MTT [5].
Finally, by incorporating the set of NCE loss for various layers  $\{\mathcal{L}_{NCE}^k\}$ ,  $(k = 1,\dots ,K)$ , we can then formulate MIM4DD loss  $\mathcal{L}_{\mathrm{MIM4DD}}$  as:
$$
\mathcal {L} _ {\mathrm {M I M 4 D D}} = \lambda \sum_ {k = 1} ^ {K} \frac {\mathcal {L} _ {N C E} ^ {k}}{\beta^ {K - 1 - k}} + \mathcal {L} _ {D D}, \tag {2.17}
$$
in which  $\mathcal{L}_{DD}$  can be any loss functions in previous DD methods [40, 43, 5],  $\lambda$  is used to control the NCE loss, and  $\beta$  is a scalar greater than 1. In practice, we use MTT [5] or BPTT [11] as default  $\mathcal{L}_{DD}$ .
2.3 Discussion on MIM4DD
In addition to the theoretical formulation, we are going to provide a more straightforward manifestation of MIM4DD. As illustrated in Fig.2, by embedding the activations to the contrastive space and constructing proper negative-and-positive embedding pairs, we can heterogeneously distill the relevant information from the real dataset to learnable synthetic dataset, and enhance the heterogeneity of the synthetic dataset within the contrastive learning framework as shown in Fig.5. Specifically, networks on different datasets first learn self-supervised information benefited from a large number of the designed pairs [18, 39]. Then, via the backward pass in Fig.2, the synthetic data are updated
w.r.t. this contrastive loss. While the actual number of negative samples in practice can be huge, e.g., 50K for synthesizing 10 pictures per-class to replace CIFAR10 [22] despite only drawing two samples in Fig.2.
Furthermore, we give a direct explanation of why optimizing  $\mathcal{L}_{\text{MIM4DD}}$  in Eq.2.17 can lead to the targeted MIM between synthetic dataset and real dataset in Eq.2.3. Firstly, optimizing the formulated contrastive loss in Eq.2.16 is equivalent to maximizing the derived accessible MI between activations in Eq.2.9, as the inequality in Eq.2.14 (Eq.2.10-2.13 can derive Eq.2.14). Secondly, based on Theorem 1 (Eq.A.1) and the property of the networks itself (Eq.2.4 and Eq.2.5), maximizing the accessible MI in Eq.2.9 equals to maximizing the targeted MI in Eq.2.8, which is our goal, to minimize the MI between synthetic dataset and real dataset. Since real data are fixed and synthetic data are learnable, the optimized synthetic dataset can contain as much information as the real dataset under the metric of MI.
In this section, we conduct comprehensive experiments to evaluate our proposed method MIM4DD on four different datasets for DD task. We first describe the implementation details of MIM4DD, and then compare our method with several SoTA DD methods to demonstrate superiority of our proposed method. Finally, we validate the effectiveness of MI module (connected with Eq.2.16 and Eq.2.17) by a series of ablation studies.
3.1 Datasets and Implementation Details
Datasets. We use MNIST [24], SVHN [35], and CIFAR10/100 datasets to conduct our experiments. MNIST [24] is a dataset for handwritten digits recognition that is widely used for validating image recognition models. It contains 60,000 training images and 10,000 testing images with the size of  $28 \times 28$ . CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of  $32 \times 32$  from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing. More details of the datasets can be found in Supplemental Materials.
Implementation Details. In the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using a three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of 'Conv-InstNorm-ReLU-AvgPool.' Each convolutional layer has 128 channels, and AvgPool represents a  $2 \times 2$  average pooling operation with stride 2. The synthetic images' initial learning rate is 0.1, which is halved at the 1,800th and 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and  $\lambda$  in Eq.2.17 is 0.8. The effect of  $\lambda$  is explored in Sec.3.3.
3.2 Comparison with SoTA
We compare MIM4DD with a series of state-of-the-art (SoTA) dataset distillation methods, including Dataset Distillation (DD) [41], LD [3], Dataset Condensation (DC) [46], DC with differentiable siamese augmentation (DSA) [43], DC with distribution matching (DM) [45], CAFE [40], FRePo [47], TESLA [10], BPTT [11], and MTT [5]. We report the performances of our method and comparisons on three datasets in Table 1. Taking into account the overall performance of MIM4DD across these mainstream dataset distillation benchmarks, it's apparent that our method consistently outperforms existing SoTAs. For instance, in the setting of generating 10 images per class, our method delivers top-tier results across all datasets. Additionally, when we synthesize 10 images per class using CIFAR100 as a real-world dataset, our method surpasses MTT by a margin of  $1.4\%$ . Importantly, our method can serve as an effective plug-and-play module for existing state-of-the-art DD methods.
3.3 Ablation Studies
Table 1: Dataset distillation methods comparisons. The settings are the same as previous SoTAs, BPTT [11], MTT [5], and DREAM [27]. Importantly, MIM4DD can work as an add-on module for SoTA methods.
MNIST	CIFAR10	CIFAR100
IPC-1	IPC-10	IPC-50	IPC-1	IPC-10	IPC-50	IPC-1	IPC-10
Full Set	99.6 ± 0.0	84.8 ± 0.1	56.2 ± 0.3
DD [41]	-	79.5 ± 8.1	-	-	36.8 ± 1.2	-	-	-
LD [3]	60.9 ± 3.2	87.3 ± 0.7	93.3 ± 0.3	25.7 ± 0.7	38.3 ± 0.4	42.5 ± 0.4	11.5 ± 0.4	-
CAFE [40]	93.1 ± 0.3	97.2 ± 0.2	98.6 ± 0.2	30.3 ± 1.1	46.3 ± 0.6	55.5 ± 0.6	14.0 ± 0.3	31.5 ± 0.2
DM [44]	89.7 ± 0.6	97.5 ± 0.1	98.6 ± 0.1	26.0 ± 0.8	48.9 ± 0.6	63.0 ± 0.4	11.4 ± 0.3	29.7 ± 0.3
DSA [43]	88.7 ± 0.6	97.8 ± 0.1	99.2 ± 0.1	28.8 ± 0.7	52.1 ± 0.5	60.6 ± 0.5	16.8 ± 0.2	32.3 ± 0.3
DC [46]	91.7 ± 0.5	97.4 ± 0.2	98.9 ± 0.2	28.3 ± 0.5	44.9 ± 0.5	53.9 ± 0.5	12.8 ± 0.3	25.2 ± 0.3
DCC [26]	-	-	-	32.9 ± 0.8	49.4 ± 0.5	61.6 ± 0.4	13.3 ± 0.3	30.6 ± 0.4
DSAC [26]	-	-	-	34.0 ± 0.7	54.5 ± 0.5	64.2 ± 0.4	14.6 ± 0.3	33.5 ± 0.3
FRePo [47]	92.4 ± 0.5	98.4 ± 0.1	98.8 ± 0.1	41.3 ± 0.5	59.6 ± 0.3	63.6 ± 0.2	24.8 ± 0.2	31.2 ± 0.2
FRePo-w [47]	93.0 ± 0.4	98.6 ± 0.1	99.2 ± 0.0	46.8 ± 0.7	65.5 ± 0.4	71.7 ± 0.2	28.7 ± 0.1	42.5 ± 0.2
MTT [5]	91.4 ± 0.9	97.3 ± 0.1	98.5 ± 0.1	46.3 ± 0.8	65.3 ± 0.7	71.6 ± 0.2	24.3 ± 0.3	40.1 ± 0.4
TESLA [10]	-	-	-	48.5 ± 0.8	66.4 ± 0.8	72.6 ± 0.7	24.8 ± 0.4	41.7 ± 0.3
MTT [5]	91.4 ± 0.9	97.3 ± 0.1	98.5 ± 0.1	46.3 ± 0.8	65.3 ± 0.7	71.6 ± 0.2	24.3 ± 0.3	40.1 ± 0.4
+ MIM4DD	92.0 ± 0.6	98.1 ± 0.2	98.9 ± 0.2	47.6 ± 0.2	66.4 ± 0.2	71.4 ± 0.3	25.1 ± 0.3	41.5 ± 0.2
Δ	(0.6↑)	(0.8↑)	(0.4↑)	(1.3↑)	(1.1↑)	(0.2↓)	(0.8↑)	(1.4↑)
BPTT [11]	94.7 ± 0.2	98.9 ± 0.1	99.2 ± 0.0	49.1 ± 0.6	62.4 ± 0.4	70.5 ± 0.4	21.3 ± 0.6	34.7 ± 0.5
+ MIM4DD	95.8 ± 0.3	98.9 ± 0.1	99.2 ± 0.1	51.8 ± 0.3	66.4 ± 0.8	72.9 ± 0.5	25.0 ± 0.4	38.5 ± 0.6
Δ	(1.1↑)	(0.0-)	(0.0-)	(2.7↑)	(4.0↑)	(2.4↑)	(3.7↑)	(3.8↑)
DREAM [27]	-	-	-	51.1 ± 0.3	69.4 ± 0.4	74.8 ± 0.1	29.5 ± 0.3	46.8 ± 0.7
+ MIM4DD	-	-	-	51.9 ± 0.3	70.8 ± 0.1	74.7 ± 0.2	31.1 ± 0.4	47.4 ± 0.3
Δ	-	-	-	(0.8↑)	(1.4↑)	(0.1↓)	(0.6↑)	(0.6↑)
We conduct a series of ablation studies of MIM4DD in CIFAR-10 and CIFAR100. By adjusting the coefficient  $\lambda$  in the loss function  $\mathcal{L}_{\mathrm{MIM4DD}}$  (Eq.2.16 and Eq.2.17), we investigate the effect of MIM4DD loss in synthesizing distilled datasets. The results are shown in Fig.3. We observe the trend that with  $\lambda$  increasing, the performance improves, which validates the effectiveness of our designed method. However, when the ratio of  $\mathcal{L}_{NCE}$  in overall loss is greater than a threshold, the performance of student networks drops, which means the main task, dataset distillation is overlooked in the optimization.
Table 2: Hyperparameters selection w. 10 Images/Cls on CIFAR10.
Hyper-parameter	Accuracy
critic function w.o network	64.9 ± 0.2
critic function w. 1 fc-layer	65.9 ± 0.4
critic function w. 2 fc-layer	65.3 ± 0.4
β = 1.0 in LIM4DD	63.8 ± 0.6
β = 2.0 in LIM4DD	66.0 ± 0.5
β = 0.5 in LIM4DD	62.8 ± 0.6
MIM4DD as an Add-on Module. We apply the MIM4DD framework to state-of-the-art Dataset Distillation (DD) techniques, including MTT [5] and BPTT [11]. The results are presented in Table1. It is observed that MIM4DD effectively enhances the performance of all the tested methods, providing substantial evidence that our approach can be utilized as an add-on module to existing techniques.
Hyper-parameters and Relative Module Selection. In addition to ablation studies, we conduct experiments to select the important hyper-parameters and modules. We investigate one hyperparameter, the co-efficient to adjust the weight of layer in overall loss in Eq.2.17, and one module, the architecture of embedding network in Eq.2.15. The results are in Table 2.
3.4 Regularization Propriety
Here, we analyze the trends of dataset distillation loss,  $\mathcal{L}_{DD}$  in Eq.2.17 during training. As we presented in Sec. 2.2, the term  $\mathcal{L}_{DD}$  in Eq.2.17 can be any dataset distillation loss. By controlling the co-efficient  $\lambda$  in Eq.2.17, we can separately study the DD loss  $\mathcal{L}_{DD}$ . Their training curves are presented in Fig.4. When turning on NCE loss (assigning  $\lambda = 0.8$ ), we observe that the  $\mathcal{L}_{DD}$  drops using
Figure 4:  $\mathcal{L}_{DD}$  curves while training w.o. (left) and w. (right)  $\mathcal{L}_{NCE}$ .
our designed NCE loss (Left), while the corresponding testing performance improves (see Table 1).
Combining this phenomenon and analysis on the effect of  $\lambda$  in Fig. 3, we conclude that our designed MIM4DD module can act as a regularization term.
3.5 CKA analysis
We formulate DD in a view of variable distribution, and thus we analyze distributional information flow within layers of neural networks. Centered kernel alignment (CKA) [9, 20, 32] is able to compare activations within or across networks quantitatively. Specifically, for a network fed by  $m$  samples, CKA algorithm takes  $\mathbf{X} \in \mathbb{R}^{m \times p_1}$  and  $\mathbf{Y} \in \mathbb{R}^{m \times p_2}$  as inputs which are output activations of two layers (with  $p_1$  and  $p_2$  neurons respectively). Letting  $\mathbf{K} \triangleq \mathbf{X}\mathbf{X}^{\top}$  and  $\mathbf{L} \triangleq \mathbf{Y}\mathbf{Y}^{\top}$  denote the Gram matrices for the two layers CKA computes:  $\mathrm{CKA}(\mathbf{K},\mathbf{L}) = \frac{\mathrm{HSIC}(\mathbf{K},\mathbf{L})}{\sqrt{\mathrm{HSIC}(\mathbf{K},\mathbf{K})\mathrm{HSIC}(\mathbf{L},\mathbf{L})}}$ , where HSIC is the Hilbert-Schmidt independence criterion [14]. Given the centering matrix  $\mathbf{H} = \mathbf{I}_n - \frac{1}{n}\mathbf{1}\mathbf{1}^\top$  and the centered Gram matrices  $\mathbf{K}' = \mathbf{H}\mathbf{K}\mathbf{H}$  and  $\mathbf{L}' = \mathbf{H}\mathbf{L}\mathbf{H}$ , HSIC =  $\frac{\mathrm{vec}(\mathbf{K}')\mathrm{vec}(\mathbf{L}')}{(m - 1)^2}$ ,
the similarity between these centered Gram matrices. Therefore, we employ Centered Kernel Alignment (CKA) to delve into the information interplay between real and synthetic data. Specifically, we input two datasets into two correspondingly trained networks and examine the similarity between the networks' feature maps using CKA. The findings are depicted in Fig.5. The CKA heatmap reveals that the dataset distilled via MIM4DD shares more information with the real dataset compared to the MTT [5]. This is because, in our DD formulation, the output similarity is supposed to be highly related to the data similarity (Theorem 1, Eq.A.1).
Figure 5: CKA analyzes the information shared within different datasets (Real v.s. Synthetic with  $10\mathrm{Img} / \mathrm{Cls}$  on CIFAR100). The lighter the dot, the more similar of the two corresponding layers learned from different datasets. Higher similar score between two layers' output represents those two layers share more information.
Dataset Distillation is essentially a compression problem that emphasizes maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement but are never considered by previous works. Therefore, we propose introducing a well-defined metric in information theory, mutual information (MI), to guide the optimization of synthetic datasets.
The formulation of our method for DD, MIM4DD absorbs the core idea of contrastive learning (i.e., constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of MemoryBank [42] for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. (Further details can be found in the Appendix.)
In this paper, we explore that well-defined metrics which measures the amount of shared information between variables in information theory for dataset distillation. Specifically, we introduce MI as the metric to quantify the shared information between the synthetic and the real datasets, and devise MIM4DD numerically maximizing the MI via a newly designed estimizable objective within a contrastive learning framework to update the synthetic dataset.
This work is supported by NSF IIS-2309073 and NSF SCH-2123521. This article solely reflects the opinions and conclusions of its authors and not the funding agency.
[25] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
[26] Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangwoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML, 2022.
[27] Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. In ICCV, 2023.
[28] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015.
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
[30] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[32] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In NeurIPS, 2021.
[33] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
[34] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
[35] Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In ICPR, 2012.
[36] Yuzhang Shang, Dan Xu, Ziliang Zong, and Yan Yan. Network binarization via contrastive learning. In ECCV, 2022.
[37] Yuzhang Shang, Bingxin Xu, Gaowen Liu, Ramana Rao Kompella, and Yan Yan. Causal-dfq: Causality guided data-free network quantization. In ICCV, 2023.
[38] Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In IJCNN, 2021.
[39] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2021.
[40] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
[41] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
[42] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
[43] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021.
[44] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
[45] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
[46] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 2021.
[47] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719, 2022.
A.1 In-variance of Mutual Information
Theorem 1 (In-variance of Mutual Information): Mutual information is invariant under reparametrization of the marginal variables. If  $X' = F(X)$  and  $Y' = G(Y)$  are homeomorphisms (i.e.,  $F(\cdot)$  and  $G(\cdot)$  are smooth uniquely invertible maps), then
$$
I (X, Y) = I \left(X ^ {\prime}, Y ^ {\prime}\right). \tag {A.1}
$$
Proof. If  $X' = F(X)$  and  $Y' = G(Y)$  are homeomorphisms (smooth and uniquely invertible maps), and  $J_X = \left\| \frac{\partial X}{\partial X'} \right\|$  and  $J_Y = \left\| \frac{\partial Y}{\partial Y'} \right\|$  are the Jacobi determinants, then
$$
\mu^ {\prime} \left(x ^ {\prime}, y ^ {\prime}\right) = J _ {X} \left(x ^ {\prime}\right) J _ {Y} \left(y ^ {\prime}\right) \mu (x, y) \tag {A.2}
$$
and similarly for the marginal densities, which gives
$$
\begin{array}{l} I (X ^ {\prime}, Y ^ {\prime}) = \iint d x ^ {\prime} d y ^ {\prime} \mu^ {\prime} (x ^ {\prime}, y ^ {\prime}) \log \frac {\mu^ {\prime} (x ^ {\prime} , y ^ {\prime})}{\mu_ {x} ^ {\prime} (x ^ {\prime}) \mu_ {y} ^ {\prime} (y ^ {\prime})} \\ = \iint d x d y \mu (x, y) \log \frac {\mu (x , y)}{\mu_ {x} (x) \mu_ {y} (y)} \tag {A.3} \\ = I (X, Y). \\ \end{array}
$$
More details can be found in [21].
Discussion on Theorem 1.
Our objective is to maximize the Mutual Information (MI) between the synthetic dataset and the real dataset (Eq. 3), a task that is numerically unfeasible. To overcome this challenge, we present this theorem. It allows us to transform the target problem at the data level (Eq. 3) into a more manageable problem at the feature map level (Eq. 9). Given that each layer's mapping  $\mathbf{W}^k: \mathbb{R}^{d_{k-1}} \longmapsto \mathbb{R}^{d_k}$  ( $k = 1, \dots, K$ ) in the network (as per Eq. 4, 5, and 6) can be treated as smooth, uniquely invertible maps, we can achieve the goal of maximizing the mutual information between the two datasets. This is done by maximizing the mutual information between two sets of down-sampled feature maps.
A.2 Datasets and Implementation Details
A.2.1 Datasets
MNIST [24] is a dataset for handwritten digits recognition that is widely used for validating image recognition models. It contains 60,000 training images and 10,000 testing images with the size of  $28 \times 28$ .
CIFAR10/100 [22] are two datasets consist of tiny colored natural images with the size of  $32 \times 32$  from 10 and 100 categories, respectively. In each dataset, 50,000 images are used for training and 10,000 images for testing.
A.2.2 Implementation Details.