SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As discussed in the previous section, there are 3 stages in the proposed MGDD, including training, adaptation, and inference stages. In the training stage, we aim at a meta generator  $g_{\omega}$  and adopt Alg. 1 to train  $g_{\omega}$  on a large dataset. In this paper, to ensure that the meta generator can acquire general knowledge for fast adaptation on a target dataset, we use ImageNet1k [6], a large-scale image recognition dataset popular in the computer vision and machine learning communities, as the dataset for meta learning. There are roughly 1,280,000 images in a total of 1,000 classes. We resize all images to the  $32 \times 32$  resolution. In each outer loop of Alg. 1, we randomly select 100 classes at most and a batch of data with 2,000 images in maximal from the selected class as a current target dataset  $T$ , with which we initialize a synthetic dataset  $S$  with 1,000 samples at most in each inner step. The training objective is based on Eq. 1 and the implementation follows the open-source code of FRePo [61]. For computational efficiency, the generator processes each sample independently and the configuration of the architecture can be found in the appendix. The meta generator is trained by the Adam optimizer [21] and the learning rate  $\beta$  is set as  $10^{-5}$ . The learning rate in meta-training is set as  $10^{-4}$  and the number of meta-training steps  $T$  is 5. The meta generator is trained with 200,000 meta-testing iterations. We use a cloud server with a single A40 GPU for meta learning and a workstation with a single 3090 GPU for the subsequent adaptation. The meta learning takes roughly 2 days while the time cost of adaptation is analyzed in Tab. 1.
IPC Steps	Methods	P	A	C	S	PACS	Path	Blood	PathBlood	CUB200
1	Baseline	53.54±0.99	33.53±3.19	51.69±1.25	39.76±1.74	38.03±0.60	54.23±3.12	68.38±0.55	55.11±1.26	4.84±0.19
1,000 w/o Meta	51.74±1.71	40.29±0.68	60.45±0.32	35.87±0.07	42.73±0.25	57.06±1.36	69.14±1.86	58.03±0.51	6.73±0.13
Ours	57.31±1.41	41.18±0.48	60.71±1.02	43.98±0.55	43.46±0.89	57.60±1.19	70.83±0.81	59.86±0.66	7.40±0.20
1	Baseline	55.80±1.16	40.87±1.47	56.40±2.57	44.26±0.52	38.53±0.80	54.70±1.07	68.55±1.39	60.62±1.14	5.65±0.26
2,000 w/o Meta	57.37±1.38	42.15±0.89	62.84±1.19	41.47±1.05	43.89±0.62	58.33±1.76	71.28±1.61	60.72±0.17	7.25±0.09
Ours	60.18±1.99	42.50±0.09	63.96±0.80	46.47±0.96	44.92±0.48	59.06±1.02	72.72±0.90	62.22±0.27	7.68±0.18
Full	Baseline	68.64±0.52	41.59±1.53	59.38±0.93	52.85±0.51	44.87±0.28	64.74±1.10	71.17±0.72	64.57±1.63	12.41±0.20
Ours	66.77±1.81	44.24±0.36	64.48±0.68	53.40±1.58	48.39±0.49	70.25±0.11	72.93±1.70	65.30±0.44	12.51±0.27
10	Baseline	76.33±0.31	47.20±1.61	70.92±1.02	63.26±1.26	59.54±0.34	76.18±0.87	80.09±0.82	74.46±0.91	10.33±0.64
1,000 w/o Meta	77.28±0.44	54.57±0.75	72.81±0.10	69.11±0.17	58.33±0.37	75.00±0.74	79.26±0.17	74.45±0.39	13.69±0.09
Ours	79.40±0.29	56.18±0.34	74.49±0.35	70.63±0.33	61.41±0.13	76.21±0.10	80.20±0.41	75.78±0.69	13.98±0.09
Baseline	77.59±0.41	51.18±2.31	72.36±0.44	67.68±0.86	59.86±0.21	76.96±0.70	81.73±0.37	75.01±0.51	11.25±0.51
2,000 w/o Meta	78.24±0.42	55.52±0.63	74.89±0.23	71.09±0.63	60.69±0.33	77.06±0.18	81.71±0.37	75.26±0.07	13.83±0.24
Ours	80.78±1.05	56.82±0.51	76.12±0.63	72.11±0.48	62.33±0.15	77.44±0.11	82.31±0.45	76.32±0.14	14.69±0.37
Full	Baseline	85.40±0.30	60.64±1.64	78.10±0.54	76.87±1.09	65.38±0.48	77.44±1.53	85.81±0.56	76.58±0.25	16.84±0.12
Ours	82.27±0.89	59.47±1.20	78.88±0.15	76.91±0.25	65.76±0.26	78.21±1.42	84.47±0.23	77.07±0.98	15.05±0.37
Table 2: Evaluations on PACS, PathMNIST, BloodMNIST, and CUB200 datasets. The baseline is FRePo [61]. w/o Meta denotes adapting from scratch instead of a meta generator.
In the adaptation stage, we load the meta generator obtained by the training stage and try to adapt it for a target dataset. The optimization is similar to that in the meta-training step of Alg. 1 and the difference is that we use the Adam optimizer to update the parameters instead of pure gradient descent. Notably, there are two ways to initialize  $X_{s}$  in each adaptation step, namely single-initialization and multi-initialization modes. In the single-initialization mode,  $X_{s}$  is set as the same group of random real samples in the real dataset for all adaptation steps, while in the multi-initialization mode, it can be set differently, with various random seed or/and various sizes of synthetic datasets. We experiment with both ways in this section.
4.2 Comparisons with State of the Arts
In this part, we compare the proposed MGDD with existing state-of-the-art DD methods on standard benchmarks, including CIFAR10 and CIFAR100 [23] datasets. There are 50,000 training images and 10,000 testing images in the  $32 \times 32$  resolution for both datasets, and the numbers of classes are 10 and 100 respectively. For CIFAR10, we compare our method with state of the arts on settings of 1, 10, and 50 images per class (IPC) in synthetic datasets and for CIFAR100, IPCs are 1 and 10. The candidates are three gradient-matching methods DC [60], DSA [58], and IDC [20], one trajectory-matching method MTT [2], one distribution-matching method DM [59], one back-propagation-through-time (BPTT) method RTP [7], and one kernel-ridge-regression-based method FRePo [61]. We obtain synthetic datasets using each algorithm on a 3-layer ConvNet whose architecture is the same as that in [61] and evaluate the accuracy on testing datasets for networks with the same architecture trained on synthetic datasets. For our method, we adapt the generator from the meta model on each comparison setting for 10,000 steps, and for each setting, we report mean and standard deviation over the accuracy of 3 networks initialized from scratch. Moreover, we report the cost of time to derive synthetic datasets.
As shown in Tab. 1, our method achieves comparable performance with DD methods based on heavy iterative loops and yields significant acceleration. For gradient-matching-based based solutions, they typically rely on computing second-order derivatives over a large amount of neural networks to back-propagate the matching loss to synthetic data, which results in high computation overhead. Even worse, for trajectory-matching and BPTT methods, higher-order derivatives are necessary. As a result, their time cost for optimizing synthetic datasets is formidable, and our method achieves up to  $6800 \times$  acceleration compared with BPTT. For DM and FRePo, although they do not rely on high-order gradients, they typically require a massive number of forward-backward iterations to update synthetic datasets, e.g., the FRePo baseline requires 500,000 iterations for optimization, and the efficiency is still unsatisfactory. On the contrary, the results of our method are produced by a generator in a one-stop fashion, which is trained by a meta-learning algorithm and learns to adapt to a target dataset rapidly. It thus enjoys the superior running efficiency of DD. More analysis of the accuracy performance given the same running time can be found in the appendix.
Notably, although the generator is only trained under  $32 \times 32$  RGB images in meta-learning, given that it has a fully-convolutional structure and maintains the same resolution for input and output, it
Dataset	CIFAR10	CIFAR100
IPC	1	10	50	1	10
FRePo	26.8±0.7	49.6±0.1	62.0±0.4	10.1±0.3	29.6±0.2
Ours w FRePo	42.6±0.3	58.9±0.4	66.8±0.2	20.8±0.2	32.2±0.3
DC	24.7±0.4	43.1±0.3	56.0±0.3	6.6±0.2	19.8±0.3
Ours w DC	34.3±0.4	46.2±0.6	60.0±0.4	14.1±0.1	22.3±0.4
DM	25.7±0.5	45.5±0.4	57.4±0.5	9.6±0.2	21.9±0.3
Ours w DM	31.4±0.2	48.9±0.2	60.8±0.4	16.6±0.2	22.9±0.3
Table 3: Performance of using different DD training objectives during adaptation.
Train Arch.	Unseen Arch.
IPC	Method	ConvNet	AlexNet	VGG	ResNet
1	Baseline	49.6±0.1	44.5±0.7	33.0±0.1	31.8±1.6
Ours	58.9±0.4	55.1±0.4	35.9±0.6	32.7±0.8
10	Baseline	26.8±0.7	23.4±0.3	16.9±0.1	15.1±0.8
Ours	42.6±0.3	39.6±0.8	22.9±0.6	19.1±1.3
50	Baseline	62.0±0.4	59.2±0.3	48.7±1.1	48.2±0.4
Ours	66.8±0.2	62.8±0.2	50.9±0.7	52.4±1.2
Table 4: Performance of different network architectures on CIFAR10.
is applicable to datasets with larger resolutions. It can also be adapted for datasets with different numbers of input channels with minor modifications. In the appendix, we provide more experimental results of such cross-resolution and cross-channel-number generalization cases.
4.3 Empirical Studies
In this part, we focus on some interesting properties of the proposed MGDD method, including cross-dataset, cross-objective, cross-architecture, cross-synthetic-initialization, cross-parameterization, and cross-IPC settings. More studies including cross-resolution, cross-channel-number, and cross-class-number that cannot fit into the main content are put in the appendix.
Cross-Dataset Generalization: MGDD proposed in this paper is expected to be generalized to any downstream target datasets, including those unseen and with large domain shifts from datasets used in the meta learning. To evaluate the cross-dataset generalization performance of MGDD, we conduct experiments on more datasets including one domain generalization dataset PACS [29], two medical classification datasets PathMNIST and BloodMNIST [53], and one fine-grain image classification dataset CUB200 [48]. PACS contains 9,991 images from 7 classes and 4 domains: Photo (P), Art Painting (A), Cartoon (C), and Sketch (S). The style variations across the 4 domains are drastic. We perform dataset distillation on each domain both independently and jointly, which formulates a 28-class dataset. PathMNIST and BloodMNIST contain 107,180 images of 9 classes and 17,092 images from 8 classes respectively. We also combine them together to form a 17-class dataset denoted as PathBloodMNIST. CUB200 contains 6,000 images of 200 bird species. We process all the images to the  $32 \times 32$  resolution in the RGB format and compare the performance with the FRePo baseline [61] and a generator from scratch instead of the meta generator, under 1,000 and 2,000 steps as well as full convergence. The quantitative results shown in Tab. 2 validate the robustness of MGDD under various datasets and domain shifts.
Cross-Objective Generalization: By default, both the meta-learning and adaptation objectives used in this paper for MGDD are the KRR objective in Eq. 1 following FRePo [61]. Empirically, we find that it is also feasible to adopt different objectives for adaptation. Here, we switch the adaptation objective to DC [60] and DM [59] respectively. The optimization steps for both baselines and our method are set as 2,000. As shown in Tab. 3, our methods yield consistent improvement over different baselines in a limited number of optimization steps.
Cross-Architecture Generalization: Adapted on an original architecture, a satisfactory generator is expected to produce results that are also valid to train networks with different structures, namely cross-architecture generalization. Tab. 4 shows the performance on CIFAR10, where ConvNet is used in adaptation while AlexNet [24], VGG11 [46], and ResNet18 [14] are used as unseen architectures for evaluation. The results indicate that the accuracy improvement on the original model still holds for unseen structures.
Cross-Initialization Generalization: We find that the single-initialization scheme used in Tab. 1 may lead to over-fitting of the generator to the adopted single initialization. As shown in Tab. 5, if we change the initialization of synthetic data, the performance would drop dramatically. Fortunately, multi-initialization is an alternative to account for this drawback, wherein each adaptation step, a new initialization of the synthetic dataset is sampled from the original dataset. Tab. 5 indicates that multi-initialization typically requires more adaptation steps for convergence and can perform on par with single-initialization. It is useful when samples in the synthetic dataset are concerned with user privacy, given that visualization results produced by the KRR objective are somehow realistic, as illustrated in Fig. 6 and [61]. In such cases, replacing the current dataset with the results of another
Setting	Default	Multi-Init
Original	Different Init	Different Init
Acc. (%)	60.76±0.38	43.27±0.46	60.24±1.28
Time (sec.)	505	1108
Dataset	CIFAR10	CIFAR100
IPC	1	10	1
Ours	46.26±0.27	60.76±0.38	24.66±0.15
Ours-DS	48.17±0.77	63.97±0.30	27.76±0.44
Table 5: Using multiple initialized samples during adaptation can enhance the cross-initialization performance. Performance on CIFAR10 with 10 IPC is shown here.
Table 6: Our method is orthogonal to synthetic dataset parameterization methods and provides improvement gain on small synthetic datasets. DS denotes down-sampling parameterization.
Figure 3: Performance of various IPCs on CIFAR10. Only 10 and 50 are seen in adaptation.
Figure 4: Performance of generators with various adaptation steps on CIFAR10 with 10 IPC.
Figure 5: Performance of continual learning on CIFAR100.
initialization would help solve the problem efficiently, without the necessity to re-run the whole optimization loop of existing methods.
Cross-Parameterization Generalization: Beyond different training objectives, the proposed MGDD is also orthogonal with different synthetic data parameterization tricks. In Tab. 6, we consider storing  $2 \times$  down-sampled synthetic images instead of the raw ones. Thus,  $4 \times$  synthetic samples can be stored given the same storage budget. We find that the simple strategy can lead to additional performance gain for relatively small budgets. The observation is consistent with previous works [20, 30, 7].
Cross-IPC Generalization: One crucial benefit of the MGDD is the cross-IPC generalization. Once the generator is adapted for a target dataset, when the storage budget changes, we do not need to perform the optimization again, unlike previous solutions relying on iteratively updating synthetic datasets. To demonstrate the cross-IPC generalization performance, we conduct experiments on CIFAR10 and adapt the meta generator using multi-initialization, with 10 and 50 IPCs for 2,000 steps. The adapted generator is evaluated on unseen IPCs 20, 30, and 40. The results are shown in the red curve of Fig. 3, where the generator produces satisfactory synthetic datasets with unseen IPCs.
To make the generator aware of the sizes of synthetic datasets, we concatenate size embedding channels to features at the middle layer of the generator. To understand how the embedding works, we remove these channels and conduct the same evaluation. As shown in the yellow curve of Fig. 3, the performance degrades without size embedding. In Fig. 6, we visualize some samples before and after the adapted generator on CIFAR10 with 1 and 50 IPCs. We can observe that the results are quite different: for small sizes, the generated results are vague, while for large sizes the results are more realistic, and their styles are also different. Thus, it is reasonable to take different sizes of synthetic datasets into consideration in the inference stage.
We also try training the generator from scratch instead of the meta model on the target dataset. As shown in Fig. 3, the worse performance in the green curve indicates that meta-learning is crucial for finding a satisfactory initial checkpoint for downstream adaptation.
Moreover, we also evaluate the performance on higher IPCs like 100 and 200, and the results are still encouraging compared with random real samples, which indicates that our method can serve as an alternative when the computational resource cannot support optimization for larger IPCs directly.
Various Adaptation Steps: In Fig. 4, we visualize the accuracy under different adaptation steps on CIFAR10 with 10 IPC as the red curve. Compared with training from scratch and the baseline FRePo [61], as shown in the yellow and green curves respectively, our method results in significantly faster adaptation convergence, which would be attributed to the good initial generator found by the meta-learning algorithm. Thus, our method is more applicable in scenarios requiring high efficiency, like processing streaming data.
Figure 6: Visualizations of samples before and after generator on CIFAR10 with 1 and 50 IPC.
Furthermore, to demonstrate the effectiveness of analytical labels, we replace them with vanilla one-hot labels in synthetic datasets and the performance is shown in the blue curve. The considerable performance drop indicates the importance of minimizing the error in an original space via analytical labels, which is consistent with the theoretical analysis in Theorem 1.
4.4 Application: Continual Learning
Continual learning (CL) aims to learn from a stream of data, where historical and future data are unavailable when learning the current data batch. One important issue is catastrophic forgetting [22]: a model tends to forget knowledge acquired in previous data when learning on newly-coming data. Focusing on this drawback, many works introduce a buffer with limited memory to store core data and knowledge of past experience for future use [40, 1]. Dataset distillation benefits this field by generating informative samples [36, 5, 42, 35, 44] to prevent forgetting as much as possible.
In this paper, we evaluate the CL performance of the proposed MGDD on CIFAR100, following the same protocol of [61, 59], where all 100 classes are divided into 10 tasks randomly with 10 classes for each task. For each task, a buffer with 20 images for each class is allowed for synthetic data. We first try adapting the generator on each new task from the meta model for 2,000 steps and the performance is shown in the yellow curve in Fig. 5. Alternatively, we can choose to adapt the generator from the checkpoint of the previous task, which has already learned some global knowledge of full data and yields better performance, as shown in the red curve. Notably, it is also feasible to only adapt the generator on the first task and the remaining tasks directly adopt this generator to output synthetic data. As shown in the blue curve, with the most significant flexibility, the performance is still comparable with the FRePo baseline [61] shown in the green curve, which suggests great practical value for processing data streams.
In this paper, we propose MGDD, a novel feed-forward paradigm for dataset distillation (DD). Specifically, in our pipeline, synthetic labels are obtained by solving a least-squares problem equipped with an analytical solution, and synthetic samples are transferred from their initial results by a conditional generator instead of taking the whole original training dataset as input. Theoretical derivation indicates an error upper bound of the proposed framework. On the one hand, unlike existing DD approaches requiring time-consuming forward-backward iterations through a massive number of networks, MGDD generates distilled results with a generator adapted rapidly from a meta generator, which improves the efficiency of DD significantly. On the other hand, existing techniques have to repeat the whole iterative algorithms for different sizes of synthetic datasets, while MGDD can perform inference flexibly on various sizes once adapted to the target dataset. Focusing on the efficiency of adaptation on target datasets, we propose a meta-learning algorithm to train a meta generator, such that it can acquire knowledge of target datasets sufficiently in only a few steps. Experiments demonstrate that the proposed MGDD performs on par with existing state-of-the-art DD baselines under  $22 \times$  acceleration. It also exerts strong cross-size generalization ability even on sizes of synthetic datasets unseen during adaptation. Future works may explore advanced feed-forward fashions of DD, focusing on generation pipelines, training algorithms, and network architectures, making improvements on the cross-dataset, cross-size, and cross-architecture generalization.
This work is supported by the Advanced Research and Technology Innovation Centre (ARTIC), the National University of Singapore under Grant (project number: A0005947-21-00, project reference: ECT-RP2), and the Singapore Ministry of Education Academic Research Fund Tier 1 (WBS: A0009440-01-00).
In this part, we include more details about the technical methods, more experimental results of our MGDD method, and more discussion on limitations and future works, which cannot be accommodated in the main paper due to the page limit. Our method first trains a meta generator to generate synthetic samples and then an adaptation stage is executed for a target dataset. We provide algorithmic details of the adaptation stage, a summary of hyper-parameters, and configurations of our generator architecture. Then, we conduct more evaluations on the cross-number-of-channel, cross-resolution, cross-ipc, and cross-number-of-classes performance of our method. More discussions of the adaptation performance and more qualitative examples are also included. Finally, we discuss limitations of the proposed method and potential future works.
A More Details
Adaptation Algorithm: Alg. 1 demonstrates the procedure of meta learning to obtain a meta synthetic sample generator. On a downstream target dataset, the meta network is adapted to a specific network with a limited number of steps. The adaptation algorithm is similar to the meta-training step of the meta learning algorithm. Here, we present the full details in Alg. 2. We can prepare multiple initialization of synthetic samples through randomly sampling from the target dataset. Recall that the main pipeline of our algorithm is to first obtain analytical synthetic labels in a random neural space  $\theta$ :  $Y_{s}^{*} = f_{\theta}(X_{s})W_{t}^{\theta}$ . Here, the optimal kernel-ridge-regression parameters of the target dataset  $W_{t}^{\theta}$  can be computed by  $W_{t}^{\theta} = f_{\theta}(X_{t})^{\top}(f_{\theta}(X_{t})f_{\theta}(X_{t})^{\top})^{-1}Y_{t}$ , if the number of real samples  $n_t$  is smaller than the feature dimension  $p$ . Otherwise,  $W_{t}^{\theta} = (f_{\theta}(X_{t})^{\top}f_{\theta}(X_{t}))^{-1}f_{\theta}(X_{t})^{\top}Y_{t}$ .
Algorithm 2 Adaptation Algorithm of Synthetic Sample Generator for a Target Dataset
Input:  $(X_{t},Y_{t})$  : A Target Dataset;  $T$  : Number of Adaptation Steps;  $\alpha$  : Learning Rate of Generator;  $\theta$  : Parameter of a Random Neural Network;  $\omega$  : Parameter of a Meta Generator;  $\mathcal{I}$  : A Set of Randomly Initialized Synthetic Samples.  
Output:  $\omega^{\prime}$  : Parameter of a Target-Specific Generator.  
1:  $W_{t}^{\theta} = f_{\theta}(X_{t})^{\top}(f_{\theta}(X_{t})f_{\theta}(X_{t})^{\top})^{-1}Y_{t};$   
2: for Each  $X_{s}$  in  $\mathcal{I}$  do  
3:  $Y_{s}^{*} = f_{\theta}(X_{s})W_{t}^{\theta};$   
4: end for  
5: Initialize generator parameters  $\omega^{\prime}$  with  $\omega$ ;  
6: for  $1\leq i\leq T$  do  
7: Sample a batch of real data  $(X_{t}^{i},Y_{t}^{i})$  of from  $(X_{t},Y_{t})$ ;  
8: Sample a initialized synthetic data  $(X_{s},Y_{s}^{*})$  from  $\mathcal{I}$ ;  
9:  $X_{s}^{*} = g_{\omega^{\prime}}(X_{s});$   
10: Sample neural parameters  $\theta^{*}$  from a random distribution;  
11:  $\mathcal{L} = \| f_{\theta^{*}}(X_{t})f_{\theta^{*}}(X_{s}^{*})^{\top}(f_{\theta^{*}}(X_{s}^{*})f_{\theta^{*}}(X_{s}^{*})^{\top})^{-1}Y_{s}^{*} - Y_{t}\|_{2}^{2};$   
12: Update  $\omega^{\prime}$  via  $\omega^{\prime}\gets \omega^{\prime} - \alpha \nabla_{\omega^{\prime}}\mathcal{L}$ ;  
13: end for
After the calculation of analytical labels, we fix them and train the synthetic sample generator initialized by parameters of the meta generator for some steps. The optimization objective is similar to those in Zhou et al. [61] and Loo et al. [32]. The difference is that the optimization target is parameters of the generator instead of synthetic samples.
Summary of Hyper-Parameters: For a clear view, we summarize the hyper-parameters and their values in both meta learning and adaptation stages as shown in Tab. 7. All experiments follow these default settings of hyper-parameters if not specified. Other configurations unmentioned follow the settings of the baseline FRePo [61].