SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the experiments, we optimize synthetic sets with 1/10/50 Images Per Class (IPC) across all three datasets, using a three-layer Convolutional Network (ConvNet) identical to those used in [46, 40, 5]. The ConvNet comprises three consecutive blocks of 'Conv-InstNorm-ReLU-AvgPool.' Each convolutional layer has 128 channels, and AvgPool represents a  $2 \times 2$  average pooling operation with stride 2. The synthetic images' initial learning rate is 0.1, which is halved at the 1,800th and 2,800th iterations. The training is stopped after 5,000 iterations. To test the ConvNet's performance on the synthetic dataset, we train the network on synthetic sets for 300 epochs and assess the performance using five randomly initialized networks. The network's initial learning rate is 0.01. As per [5], we conduct five experiments and report the mean and standard deviation across the five networks. The default batch size is 256, and  $\lambda$  in Eq.17 is 0.8. The effect of  $\lambda$  is explored in Sec.3.3.
Figure 6: (Left) Samples from CIFAR10; (Right) Samples from Synthetic dataset based on CIFAR10. We observe that the heterogeneity in the generated images enhanced, benefited from the contrastive learning loss (Loss  $\mathcal{L}_{\mathrm{MIM4DD}}$  in Eq.17).
A.3 Synthetic Samples Visualization.
Dataset Distillation (DD) is firstly introduced by Wang et al. [41], in which they optimize the distilled images using gradient-based hyperparameter optimization [28]. The key problem is to optimize the specific-designed metrics of networks on real and synthetic datasets to update the estimizable images. Subsequently, several works significantly improve the results by designing different metrics. For example, Bohdal et al. and Sucholutsky et al. [3, 38] use distance between networks' soft labels; Zhao et al. [46] define the gradients of networks as metric; Zhao et al. [43] further adopts augmentations to enhance the alignment ability; Wang et al. [40] utilize distance of network feature maps as metric; and Cazenavette [5] propose long-range trajectory to construct the metric function. Lee et al. [26] propose Dataset Condensation with Contrastive Signals (DCC) by modifying the loss function to enable the DC methods to effectively capture the differences between classes. On the other hand, researchers take DD as a bi-level optimization problem. For example, Zhou et al. [47] employ a closed-form approximation for the unrolled inner optimization; Deng et al. [11] revisits the optimization framework in [41] and observe that the inclusion of a momentum term in inner optimization can significantly enhance performance, leading to state-of-the-art results in certain settings.
DD is essentially a compression problem that emphasizes on maximizing the preservation of information contained in the data. We argue that well-defined metrics which measure the amount of shared information between variables in information theory are necessary for success measurement, but are
never considered by previous works. Therefore, we propose to introduce a well-defined metric in information theory, mutual information (MI), to guide the optimization of synthetic datasets.
Contrastive Learning and Mutual Information. The fundamental idea of all contrastive learning methods is to draw the representations of positive pairs closer and push those of negative pairs farther apart within a contrastive space. Several self-supervised learning methods are rooted in well-established ideas of MI maximization, such as Deep InfoMax [18], Contrastive Predictive Coding [29], MemoryBank [42], Augmented Multiscale DIM [2], MoCo [16] and SimSaim [8]. These are based on NCE [15] and InfoNCE [18] which can be seen as a lower bound on MI [30]. Meanwhile, Tian et al. [39] and Chen et al. [7] extend the contrastive concept into the realm of Knowledge Distillation (KD), pulling and pushing the representations of teacher and student.
The formulation of our method for DD, MIM4DD also absorbs the core idea (i.e., constructing the informative positive and negative pairs for contrastive loss) of the existing contrastive learning methods, especially the contrastive KD methods, CRD [39] and WCoRD [7]. However, our approach has several differences from those methods: (i) our targeted MI and formulated numerical problem are totally different; (ii) our method can naturally avoid the cost of MemoryBank [42] for the exponential number of negative pairs in CRD and WCoRD, thanks to the small size of the synthetic dataset in our task. Given that the size of the synthetic dataset  $M$  typically ranges from  $0.1 - 1\%$  of the size of the real dataset  $N$ , the product  $M \cdot N$  is significantly smaller than  $N \cdot N$  (i.e.,  $M \cdot N \ll N \cdot N$ ).
Difference with DCC [26]. Recently, Lee et al. [26] introduced Dataset Condensation with Contrastive Signals (DCC), modifying the loss function to allow Dataset Condensation methods to effectively discern differences between classes. However, several distinctions exist between DCC and our method: (i) They are motivated differently. Our approach is predicated on information degradation, while DCC hinges on class diversity. (ii) From the perspective of contrastive learning, the view, positive and negative samples differ considerably. Our approach can be implemented at the feature map level, thanks to the introduced Theorem 1, while DCC can only be deployed at the gradient level. (iii) The performance of our method significantly surpasses that of DCC.
Codes can be found anomalously in Supplement.