SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We conducted experiments to investigate whether  $S^{-}$  solely compensates for the gradient matching error of  $S^{+}$  and is unable to extract knowledge independently. To achieve this, we sorted  $S^{+}$  and  $S^{-}$  by the  $l_{1}$ -norm of the amplification function  $\| R(s_{i},f_{\theta_{m}})\|_{1}$  and trained separate networks with  $S^{+}$  and  $S^{-}$ . As depicted in Figure 2, we observed a significant discrepancy in accuracy, which increased with the difference in magnitude of  $R(s_{i},f_{\theta_{m}})$ . Further details and discussion are provided in subsection 5.3. These experiments verify the coupling issue wherein  $S^{-}$  compensates for the matching error of  $S^{+}$ , thereby reducing its effectiveness in condensing additional knowledge.
4.3 Sequential Subset Matching
We can use a standard deep learning task as an analogy for the dataset distillation problem, then the synthetic dataset  $S$  can be thought of as the weight parameters that need to be optimized. However, simply increasing the size of the synthetic dataset is comparable to multiplying the parameters of a model in an exact layer without architecting the newly added parameters, and the resulting performance improvement is marginal. We thereby propose SeqMatch to reorganize the synthetic dataset  $S$  to utilize the newly added synthetic data.
We incorporate additional variability into the optimization process of synthetic data to encourage the capture of higher-level feature extracted in the latter training progress. To do this, SeqMatch divides the synthetic dataset  $S$  into  $K$  subsets equally, i.e.,  $S = S_{1} \cup S_{2} \cup \dots \cup S_{K}, |S_{k}| = \lfloor \frac{|S|}{K} \rfloor$ . SeqMatch optimizes each  $S_{k}$  by solving
$$
\widehat {\mathcal {S}} _ {k} = \underset { \begin{array}{c} \mathcal {S} _ {k} \subset \mathbb {R} ^ {d} \times \mathcal {Y} \\ | \mathcal {S} _ {k} | = \lfloor | \mathcal {S} | / K \rceil \end{array} } {\arg \min } \underset {\theta_ {0} \sim P _ {\theta_ {0}}} {\mathbb {E}} \left[ \sum_ {m = (k - 1) n} ^ {k n} \mathcal {L} \left(\mathcal {S} _ {k} \cup \mathbb {S} ^ {(k - 1)}, \theta_ {m}\right) \right], \tag {5}
$$
where  $\mathbb{S}^{(k - 1)} = S_1\cup S_2\cup \dots \cup S_{k - 1}$ , which represents the union set of the subsets in the former.  $\mathbb{S}^{(k - 1)}$  are fixed and only  $S_{k}$  will be updated. The subset  $S_{k}$  is encouraged to match the corresponding  $k^{th}$  segment of the teacher trajectory to condense the knowledge in the latter epoch. Let  $n = \lfloor \frac{M}{K}\rfloor$  denote the length of trajectory segment to be matched by each subset  $S_{k}$  in the proposed framework. To strike a balance between providing adequate capacity for distillation and avoiding coupled synthetic data, the size of each subset  $S_{k}$  is well-controlled by  $K$ .
In the distillation phase, each subset is arranged in ascending order to be optimized sequentially. We reveal that the first subset  $S_{1}$  with  $\frac{1}{K}$  size of the original synthetic dataset  $S$  is sufficient to condense adequate knowledge in the former epoch. For the subsequent subset  $S_{k}$ , we encourage the  $k^{th}$  subset  $S_{k}$  condense the knowledge different from those condensed in the previous subsets. This is achieved by minimizing the matching loss  $\mathcal{L}(\mathcal{S}_k\cup \mathbb{S}^{(k - 1)},\theta_m)$  while only  $S_{k}$  will be updated.
During the evaluation phase, the subsets of the synthetic dataset are used sequentially to train the neural network  $f_{\theta}$ , with the weight parameters  $\theta$  being iteratively updated by  $\theta_{k} = \mathrm{alg}(S_{k},\theta_{k - 1})$ . This training process emulates the sequential feature extraction of real dataset  $\mathcal{T}$  during training. Further details regarding SeqMatch and the optimization of  $\theta^{*}$  can be found in Algorithm.1.
In this section, we provide implementation details for our proposed method, along with instructions for reproducibility. We compare the performance of SeqMatch against state-of-the-art dataset distillation methods on a variety of datasets. To ensure a fair and comprehensive comparison, we follow up the experimental setup as stated in [8, 30]. We provide more experiments to verify the effectiveness of SeqMatch including the results on ImageNet subsets and analysis experiments in Appendix A.1 due to page constraints.
Datasets: We evaluate the performance of dataset distillation methods on several widely-used datasets across various resolutions. MNIST [28], which is a fundamental classification dataset, is included with a resolution of  $28 \times 28$ . SVNH [36] is also considered, which is composed of RGB images of house numbers cwith a resolution of  $32 \times 32$ . CIFAR10 and CIFAR100 [25], two datasets frequently used in dataset distillation, are evaluated in this study. These datasets consist of 50,000 training images and 10,000 test images from 10 and 100 different categories, respectively. Additionally, our proposed method is evaluated on the Tiny ImageNet [27] dataset with a resolution of  $64 \times 64$  and on the ImageNet [24] subsets with a resolution of  $128 \times 128$ .
Evaluation Metric: The evaluation metric involves distillation phase and evaluation phase. In the former, the synthetic dataset is optimized with a distillation budget that typically restricts the number of images per class (ipc). We evaluate the performance of our method and baseline methods under the settings  $\mathrm{ipc} = \{10,50\}$ . We do not evaluate the setting with  $\mathrm{ipc} = 1$  since our approach requires  $\mathrm{ipc} \geq 2$ . To facilitate a clear comparison, we mark the factorization-based baselines with an asterisk (*) since they often employ an additional decoder, following the suggestion in [30]. We employ 4-layer ConvNet [14] in Tiny ImageNet dataset whereas for the other datasets we use a 3-layer ConvNet [14].
In the evaluation phase, we utilize the optimized synthetic dataset to train neural networks using a standard training procedure. Specifically, we use each synthetic dataset to train five networks with random initializations for 1,000 iterations and report the mean accuracy and its standard deviation of the results.
Implementation Details. To ensure the reproducibility of SeqMatch, we provide detailed implementation specifications. Our method relies on a single hyperparameter, denoted by  $K$ , which determines the number of subsets. In order to balance the inclusion of sufficient knowledge in each segment with the capture of high-level features in the later stages, we set  $K = \{2,3\}$  for the scenarios where  $\mathrm{ipc} = \{10,50\}$ , respectively. Notably, our evaluation results demonstrate that the choice of  $K$  remains consistent across the various datasets.
As a plug-in strategy, SeqMatch requires a backbone method for dataset synthesis. Each synthetic subset is optimized using a standard training procedure, specific to the chosen backbone method. The only hyperparameters that require adjustment in the backbone method are those that control the segments of the teacher trajectory to be learned by the synthetic dataset, whereas the remaining hyperparameters remain consistent without adjustment. Such adjustment is to ensure each synthetic subset effectively condenses the knowledge into stages. The precise hyperparameters of the backbone methods are presented in Appendix A.3. We conduct our experiments on the server with four Tesla V100 GPUs.
Our proposed SeqMatch is plugged into the methods MTT [5] and IDC [23], which are denoted as SeqMatch-MTT and SeqMatch-IDC, respectively. As shown in Table 1, the classification accuracies of ConvNet [14] trained using each dataset distillation method are summarized. The results indicate that SeqMatch significantly outperforms the backbone method across various datasets, and even surpasses state-of-the-art baseline methods in different settings. Our method is demonstrated to outperform the state-of-the-art baseline methods in different settings among different datasets. Notably, SeqMatch achieves a greater performance improvement in scenarios with a high compression ratio (i.e.,  $\mathrm{ipc} = 50$ ). For instance, we observe a  $3.5\%$  boost in the performance of MTT [5], achieving  $51.2\%$  accuracy on CIFAR-100. Similarly, we observe a  $1.9\%$  performance enhancement in IDC [23],
Table 1: Performance comparison of dataset distillation methods across a variety of datasets. Abbreviations of GM, TM, DM,META stand for gradient matching, trajectory matching, distribution matching, and meta-learning respectively. We reproduce the results of MTT [5] and IDC [23] and cite the results of the other baselines [30]. The best results of non-factorized methods (without decoders) are highlighted in orange font. The best results of factorization-based methods are highlighted in blue font.
Methods	Schemes	MNIST	SVHN	CIFAR-10	CIFAR-100	Tiny ImageNet
10	50	10	50	10	50	10	50	10
DD [47]	META	79.5 ±8.1	-	-	-	36.8 ±1.2	-	-	-	-
DC [53]	GM	94.7 ±0.2	98.8 ±0.2	76.1 ±0.6	82.3 ±0.3	44.9 ±0.5	53.9 ±0.5	32.3 ±0.3	42.8 ±0.4	-
DSA [51]	GM	97.8 ±0.1	99.2 ±0.1	79.2 ±0.5	84.4 ±0.4	52.1 ±0.5	60.6 ±0.5	32.3 ±0.3	42.8 ±0.4	-
DM [52]	DM	97.3 ±0.3	94.8 ±0.2	-	-	48.9 ±0.6	63.0 ±0.4	29.7 ±0.3	43.6 ±0.4	12.9 ±0.4
CAFE [46]	DM	97.5 ±0.1	98.9 ±0.2	77.9 ±0.6	82.3 ±0.4	50.9 ±0.5	62.3 ±0.4	31.5 ±0.2	42.9 ±0.2	-
KIP [37, 38]	KRR	97.5 ±0.0	98.3 ±0.1	75.0 ±0.1	85.0 ±0.1	62.7 ±0.3	68.6 ±0.2	28.3 ±0.1	-	-
FTD [13]	TM	-	-	-	-	66.6 ±0.3	73.8 ±0.2	43.4 ±0.3	50.7 ±0.3	24.5 ±0.2
MTT [5]	TM	97.3 ±0.1	98.5 ±0.1	79.9 ±	87.7 ±0.3	65.3 ±0.7	71.6 ±0.2	40.1 ±0.4	47.7 ±0.2	23.2 ±1.3
SeqMatch-MTT	TM	97.6 ±0.2	99.0 ±0.1	80.2 ±0.6	88.5 ±0.2	66.2 ±0.6	74.4 ±0.5	41.9 ±0.5	51.2 ±0.3	23.8 ±0.3
IDC* [23]3	GM	98.4 ±0.1	99.1 ±0.1	87.3 ±0.2	90.2 ±0.1	67.5 ±0.5	74.5 ±0.1	44.8 ±0.2	51.4 ±0.4	-
SeqMatch-IDC*	GM	98.6 ±0.1	99.2 ±0.0	87.2 ±0.2	92.1 ±0.1	68.3 ±0.2	75.3 ±0.2	45.1 ±0.3	51.9 ±0.3	-
RTP* [11]	META	99.3 ±0.5	99.4 ±0.4	89.1 ±0.2	89.5 ±0.2	71.2 ±0.4	73.6 ±0.4	42.9 ±0.7	-	-
HaBa* [33]	TM	-	-	83.2 ±0.4	88.3 ±0.1	69.9 ±0.4	74.0 ±0.2	40.2 ±0.2	47.0 ±0.2	-
Whole	-	99.6 ±0.0	95.4 ±0.1	84.8 ±0.1	56.2 ±0.3	37.6 ±0.4
achieving  $92.1\%$  accuracy on SVNH, which approaches the  $95.4\%$  accuracy obtained using the real dataset. These results suggest that our method is effective in mitigating the adverse effects of coupling and effectively condenses high-level features in high compression ratio scenarios.
Cross-Architecture Generalization: We also conducted experiments to evaluate cross-architecture generalization, as illustrated in Table 2. The ability to generalize effectively across different architectures is crucial for the practical application of dataset distillation. We evaluated our proposed SeqMatch on the CIFAR-10 dataset with ipc = 50. Following the evaluation metric established in [13, 46], three additional neural network architectures were utilized for evaluation: ResNet [19], VGG [44], and AlexNet [26]. Our SeqMatch approach demonstrated a significant improvement in performance during cross-architecture evaluation, highlighting its superior generalization ability.
Sequential Knowledge Acquisition: We conducted experiments on CIFAR-10 with ipc = 50, presented in Figure 1, to investigate the inability of existing baseline methods to capture the knowledge learned in the latter epoch, as discussed in subsection 4.1. Inspired by [16], we utilized the change in instance-wise loss on real dataset to measure the effectiveness of condensing high-level features. Specifically, we recorded the loss of each instance from the real dataset  $\mathcal{T}$  at every epoch, where the network was trained with synthetic dataset for only 20
Table 2: Cross-Architecture Results trained with ConvNet on CIFAR-10 with ipc = 50. We cite the results reported in Du et al. [13].
Method	Evaluation Model
ConvNet	ResNet18	VGG11	AlexNet
DC [53]	53.9	20.8	38.8	28.7
±0.5	±1.0	±1.1	±0.7
CAFE [46]	55.5	25.3	40.5	34.0
±0.4	±0.9	±0.8	±0.6
MTT [5]	71.6	61.9	55.4	48.2
±0.2	±0.7	±0.8	±1.0
FTD [13]	73.8	65.7	58.4	53.8
±0.2	±0.3	±1.6	±0.9
SeqMatch-MTT	74.4	68.4	64.2	50.7
±0.5	±0.9	±0.7	±1.0
SeqMatch-IDC	75.3	69.7	73.4	72.0
±0.2	±0.6	±0.1	±0.2
<sup>3</sup>Although IDC [23] is not categorized as a factorization-based method, it employs data parameterization to better improve the performance of synthetic dataset. Therefore, we compare IDC to the factorization-based method as factorization can be treated as a kind of special data parameterization.
iterations in each epoch. To distinguish hard instances from easy ones, we employed k-means algorithm [18] to cluster all instances in the real dataset into two clusters based on the recorded instance-wise loss. The distribution of instances in terms of difficulty is as follows:  $77\%$  are considered easy instances, while  $23\%$  are classified as hard instances.
We evaluated MTT[5] and SeqMatch as mentioned above. Our results show that MTT[5] overcondenses the knowledge learned in the former epoch. In contrast, SeqMatch is able to successfully capture the knowledge learned in the latter epoch.
Coupled Synthetic Subsets: In order to validate our hypothesis that the synthetic subset  $S^{-}$  is ineffective at condensing knowledge independently and results in over-condensation on the knowledge learned in the former epoch, we conducted experiments as shown in Figure 2. We sorted the subsets  $S^{+}$  and  $S^{-}$  of the same size by the  $l_{1}$ -norm of the amplification function  $|R(s_i,f_{\theta_m})|1$  as explained in Figure 4.2. We then recorded the accuracy discrepancies between the separate networks trained by  $S^{+}$  and  $S^{-}$  with respect to the mean  $l_{1}$ -norm difference, i.e.,  $\mathbb{E}_{s_i\in S^+}[|R(s_i,f_{\theta_0})|1] - \mathbb{E}_{s_i\in S^{-}}[|R(s_i,f_{\theta_0})|1]$ .
As shown in Figure 2, the accuracy discrepancies increased linearly with the  $l_{1}$ -norm difference, which verifies our hypothesis that  $\mathcal{S}^{-}$  is coupled with  $\mathcal{S}^{+}$  and this coupling leads to the excessive condensation on low-level features. However, our proposed method, SeqMatch, is able to alleviate the coupling issue by encouraging  $\mathcal{S}^{-}$  to condense knowledge more efficiently.
Figure 3: Visualization example of "car" synthetic images distilled by MTT [5] and SeqMatch from  $32 \times 32$  CIFAR-10 (ipc = 50).
Synthetic Image Visualization: In order to demonstrate the distinction between MTT [5] and SeqMatch, we visualized synthetic images within the "car" class from CIFAR-10 [25] and visually compared them. As depicted in Figure 3, the synthetic images produced by MTT exhibit more concrete features and closely resemble actual "car" images. Conversely, the synthetic images generated by SeqMatch in the  $2^{nd}$  and  $3^{rd}$  subsets possess more abstract attributes and contain complex car shapes. We provide more visualizations of the synthetic images in Appendix A.2.
5.4 Limitations and Future Work
We acknowledge the limitations of our work from two perspectives. Firstly, our proposed sequential optimization of synthetic subsets increases the overall training time, potentially doubling or tripling it. To address this, future research could investigate optimization methods that allow for parallel optimization of each synthetic subset. Secondly, as the performance of subsequent synthetic subsets builds upon the performance of previous subsets, a strategy is required to adaptively distribute the distillation budget of each subset. Further research could explore strategies to address this limitation and effectively enhance the performance of dataset distillation, particularly in high compression ratio scenarios.
In this study, we provide empirical evidence of the failure in condensing high-level features in dataset distillation attributed to the sequential acquisition of knowledge in training DNNs. We reveal that the static optimization of synthetic data leads to a bias in over-condensing the low-level features, predominantly extracted from the majority during the initial stages of training. To address this issue in a targeted manner, we introduce an adaptive and plug-in distillation strategy called SeqMatch. Our proposed strategy involves the division of synthetic data into multiple subsets, which are sequentially optimized, thereby promoting the effective condensation of high-level features learned in the later epochs. Through comprehensive experimentation on diverse datasets, we validate the effectiveness of our analysis and proposed strategy, achieving state-of-the-art performance.
This work is supported by Joey Tianyi Zhou's A*STAR SERC Central Research Fund (Use-inspired Basic Research) and the Singapore Government's Research, Innovation and Enterprise 2020 Plan (Advanced Manufacturing and Engineering domain) under Grant A18A1b0045.
A More Experiements
A.1 ImageNet Subsets
To assess the efficacy of our approach, we conducted experiments on subsets of the ImageNet dataset [10]. These subsets were constructed by selecting ten pertinent categories from the ImageNet-1k dataset [10], with a resolution of  $128 \times 128$ . Consequently, the ImageNet subsets pose greater challenges compared to the CIFAR-10/100 [25] and Tiny ImageNet [27] datasets. We adhered to the configuration of the ImageNet subsets as suggested by previous studies [5, 13, 33], encompassing subsets such as ImageNette (diverse objects), ImageWoof (dog breeds), ImageFruits (various fruits), and ImageMeow (cats).
To synthesize the dataset, we employed a 5-layer ConvNet [14] with a parameter setting of  $\mathrm{ipc} = 10$ . The evaluation of the synthetic dataset involved performing five trials with randomly initialized networks. We compared the outcomes of our proposed method, referred to as SeqMatch, with the baseline approach MTT [5], as well as the plug-in strategies FTD [13] and HaBa [33] which build upon MTT.
The comprehensive results are presented in Table 3. Our proposed SeqMatch consistently outperformed the baseline MTT across all subsets. Notably, we achieved a performance improvement of  $4.3\%$  on the ImageFruit subset. Additionally, SeqMatch demonstrated superior performance compared to HaBa and achieved comparable results to FTD.
Table 3: The performance comparison trained with 5-layer ConvNet on the ImageNet subsets with a resolution of  $128 \times 128$ . We cite the results as reported in MTT [5], FTD [13] and HaBa [33]. The latter two methods, FTD and HaBa, are plug-in strategies that build upon the foundation of MTT. Our proposed approach, SeqMatch, exhibits superior performance compared to both MTT and HaBa, demonstrating a significant improvement in results.
ImageNette	ImageWoof	ImageFruit	ImageMeow
Real dataset	87.4 ±1.0	67.0 ±1.3	63.9 ±2.0	66.7 ±1.1
MTT [5]	63.0 ±1.3	35.8 ±1.8	40.3 ±1.3	40.4 ±2.2
FTD [13]	67.7 ±1.3	38.8 ±1.4	44.9 ±1.5	43.3 ±0.6
HaBa* [33]	64.7 ±1.6	38.6 ±1.2	42.5 ±1.6	42.9 ±0.9
SeqMatch-MTT	66.9 ±1.7	38.7 ±1.1	44.6 ±1.7	44.8 ±1.2
SeqMatch-FTD	70.6 ±1.5	41.1 ±1.4	46.5 ±1.2	45.4 ±1.2
A.2 More Visualizations
Instance-wise loss change: We have presented the average loss change of easy and hard instances in Figure 1, revealing that MTT failed to effectively condense the knowledge learned from the hard instances. To avoid the bias introduced by averaging, we have meticulously recorded and visualized the precise loss change of each individual instance. This is accomplished by employing a heatmap representation, as demonstrated in Figure 6. Each instance is depicted as a horizontal line exhibiting varying colors, with deeper shades of blue indicating higher loss values. Following the same clustering approach as depicted in Figure 1, we proceed to visualize the hard instances at the top and the easy instances at the bottom.
The individual loss changes of MTT, depicted in Figure 6, remain static across epochs. The losses of easy instances decrease to a small value during the initial stages, while the losses of hard instances persist at a high value until the end of training. These results confirm that MTT excessively focuses on low-level features. In contrast, the visualization of SeqMatch clearly exhibits the effect of color gradient, indicating a decrease in loss for the majority of instances. Notably, the losses of hard instances experience a significant decrease when a subsequent subset is introduced. These results validate that SeqMatch effectively consolidates knowledge in a sequential manner.
Figure 4: The heatmap illustrates the loss change of each instance in the real dataset across epochs. Each row in the heatmap represents an instance, while the deeper blue color denotes higher instance-wise loss. The network is trained with the synthetic datasets distilled by MTT and SeqMatch. Left: MTT fails to reduce the loss of hard instances while excessively reducing the loss of easy instances. Right: SeqMatch minimizes the loss of both the hard and easy instances.
Figure 5: Visualization example of synthetic images distilled by MTT [5] and SeqMatch from  $32 \times 32$  CIFAR-10 (ipc = {2, 3}). SeqMatch(ipc=2) is seamlessly embedded as the first two rows within the nameSeqMatch(ipc=3) visualization.
Synthetic Dataset Visualization: We compare the synthetic images with ipc = {2, 3} from the CIFAR10 dataset to highlight the differences between subsets of Seqmatch in Figure 5. We provide more visualizations of the synthetic datasets for ipc = 10 from the  $128 \times 128$  resolution ImageNet dataset: ImageWoof subset in Figure 7 and ImageMeow subset in Figure 8. In addition, parts of the visualizations of synthetic images from the  $32 \times 32$  resolution CIFAR-100 dataset are showed in Figure 6. We observe that the synthetic images generated by SeqMatch in the subsequent subset contains more abstract features than the previous subset.
A.3 Hyperparameter Details
The hyperparameters  $K$  of SeqMatch-MTT is set with  $\{2,3\}$  for the settings ipc =  $\{10,50\}$ , respectively. The optimal value of hyperparameter  $K$  is obtained via grid searches within the set  $\{2,3,4,5,6\}$  in a validation set within the CIFAR-10 dataset. We find that the subset with a small size will fail to condense the adequate knowledge from the corresponding segment of teacher trajectories,
Figure 6: Visualization of the first 10 classes of synthetic images distilled by SeqMatch from  $32 \times 32$  CIFAR-100 (ipc = 10). The initial 5 image rows and the final 5 image rows match the first and second subsets, respectively.
resulting in performance degradation in the subsequent subsets. For the rest of the hyperparameters, we report them in Table 4.
ImageFruit has different setting of Max Start Epoch from other ImageNet subsets: {10,10}
Figure 7: Visualization of the synthetic images distilled by SeqMatch from  $32 \times 32$  ImageWoof (ipc = 10). The initial 5 image rows and the final 5 image rows match the first and second subsets, respectively.
Table 4: Hyperparameter values we used for SeqMatch-MTT in the main result table. Most of the hyperparameters "Max Start Epoch" and "Synthetic Step" are various with the subsets, we use a sequential numbers to denote the parameters used in the corresponding subsets. "Img." denotes the abbreviation of ImageNet.
ipc	CIFAR-10	CIFAR-100	Tiny Img.	Img. Subsets
10	50	10	50	10	10
K	2	3	2	3	2	2
Max Start Epoch	{20,10}	{20,20,10}	{20,40}	{40,20,20}	{20,10}	{10,5}4
Synthetic Step	{30,80}	30	30	80	20	20
Expert Epoch	{2,3}	2	2	2	2	2
Synthetic Batch Size	-	-	-	125	100	20
Learning Rate (Pixels)	100	100	1000	1000	10000	100000
Learning Rate (Step Size)	1e-5	1e-5	1e-5	1e-5	1e-4	1e-6
Learning Rate (Teacher)	0.001	0.01	0.01	0.01	0.01	0.01
Figure 8: Visualization of the synthetic images distilled by SeqMatch from  $32 \times 32$  ImageMeow (ipc = 10). The initial 5 image rows and the final 5 image rows match the first and second subsets, respectively.