SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Songhua Liu Xinchao Wang*
songhua.liu@u.nus.edu，xinchao@nus.edu.sg
Existing dataset distillation (DD) techniques typically rely on iterative strategies to synthesize condensed datasets, where datasets before and after distillation are forward and backward through neural networks a massive number of times. Despite the promising results achieved, the time efficiency of prior approaches is still far from satisfactory. Moreover, when different sizes of synthetic datasets are required, they have to repeat the iterative training procedures, which is highly cumbersome and lacks flexibility. In this paper, different from the time-consuming forward-backward passes, we introduce a generative fashion for dataset distillation with significantly improved efficiency. Specifically, synthetic samples are produced by a generator network conditioned on the initialization of DD, while synthetic labels are obtained by solving a least-squares problem in a feature space. Our theoretical analysis reveals that the errors of synthetic datasets solved in the original space and then processed by any conditional generators are upper-bounded. To find a satisfactory generator efficiently, we propose a meta-learning algorithm, where a meta generator is trained on a large dataset so that only a few steps are required to adapt to a target dataset. The meta generator is termed as  $MGDD$  in our approach. Once adapted, it can handle arbitrary sizes of synthetic datasets, even for those unseen during adaptation. Experiments demonstrate that the generator adapted with only a limited number of steps performs on par with those state-of-the-art DD methods and yields  $22 \times$  acceleration.
Dataset distillation (DD) introduced by Wang et al. [51] aims to compress an original dataset  $\mathcal{T}$  into a much smaller synthetic set  $S$ , such that the performance of a neural network, trained with the condensed dataset  $S$ , is similar to the network trained with  $\mathcal{T}$ . The derived synthetic datasets not only save the cost of storage and transmission but also significantly reduce the computational resources and time required by training models using original datasets. As such, DD finds its application across a wide spectrum of domains and is receiving increasing attention from the community.
The typical paradigm of DD is to optimize  $S$  in an iterative loop, as shown in Fig. 1(a). In each iteration, a new network is sampled, leveraged by which a matching metric is calculated for  $S$  and  $\mathcal{T}$ , and the matching loss is then back-propagated through the network to update  $S$ . Recently, a large number of approaches have been dedicated to exploring advanced matching objectives to improve the training performance of distilled datasets, including matching training performance via meta learning [51, 7], matching feature regression performance [37, 38, 61, 32, 33], matching training gradients [60, 58, 20], matching training trajectories [2, 8, 4], and matching feature statistics [59, 50].
Although impressive results have been achieved and it has been demonstrated that neural networks trained by a synthetic dataset with even only 1 image per class can yield reasonable performance
(a) Previous DD Paradigm
(b) MGDD Paradigm
Figure 1: Illustration of previous back-propagation-based and our generative fashions after adaptation for dataset distillation.
on real data, the iterative optimization process adopted in existing works results in significant computational overhead. As shown in Fig. 2, for FRePo [61], the method with the best performance requires above 3 hours to obtain a synthetic dataset with 1 image per class for CIFAR100 [23], let alone RTP [7], which is a back-propagation-through-time method and requires over 9 days for optimization. Such a dramatic latency makes existing approaches hard to be applied in scenarios requiring high efficiency, like handling data streams. Moreover, when the memory budget for a synthetic dataset changes, existing methods have to repeat the time-consuming optimization for the different sizes of  $S$ , which lacks flexibility.
To alleviate the drawbacks of the conventional iterative forward-backward process, in this paper, we propose a generative dataset distillation approach, where the optimization loop is replaced by a single feed-forward propagation, as shown in Fig. 1(b). Specifically, given an initialization of synthetic samples in  $S$ , we first obtain the corresponding synthetic labels by analytically solving a least-squares problem in a feature space. Then, a generator is adopted to transfer the initialized samples to the final ones. Our theoretical analysis indicates that  $S$  solved in an original space can be transferred to the final result by any conditional generator with an upper-bounded error, which validates the feasibility of this pipeline.
Then, the key problem of our framework lies in finding a suitable generator as quickly as possible for the feed-forward generation process of  $S$ . To this end, we propose a method called MGDD, where a meta generator is learned with a meta learning algorithm on a large database like ImageNet [6]. Trained in a learning-to-learn fashion, the meta generator is optimized such that only a small number of adaptation steps are required for a target dataset unseen in the meta learning. Experiments demonstrate that our approach yields  $22 \times$  acceleration and comparable performance with existing state-of-the-art methods, as shown in Fig. 2. Beyond that, the generator once adapted can also handle unseen sizes of  $S$  during adaptation, which improves the flexibility of cross-size generalization in existing DD methods significantly. We also validate that MGDD gets competitive performance on target datasets with large domain shifts from those seen in the meta learning.
Our contributions can be summarized from the following three aspects:
- We propose an innovative feed-forward generation fashion for DD without backward propagation after adaptation which significantly improves the efficiency of existing methods;
- We introduce MGDD which uses a meta-learning algorithm to learn a meta generator and helps the generator adapt to a target dataset rapidly;
- The proposed method achieves significant acceleration and improvement in the flexibility of cross-size generalization for existing DD approaches with comparable performance.
Unlike conventional efficient learning schemes that mainly focus on lightening models [9, 34, 55, 54, 10, 18, 19, 17], dataset distillation (DD) looks into data compression: given a real large dataset  $\mathcal{T}$ , DD aims at a smaller synthetic dataset  $S$  which can match the training performance of  $\mathcal{T}$ . The seminal work by Wang et al. [51] proposes a meta learning approach to model this objective: in meta training, a network is trained with the current  $S$  for multiple times, while in meta test, the loss for the updated network is evaluated on  $\mathcal{T}$ , which is then back-propagated through the bi-level optimization
2The acceleration factor estimated here includes adaptation time. If only feed-forward time is considered, the acceleration can be  $1650\times$
to update  $S$ . The following work by Deng et al. [7] improves performances by adopting momentum during meta training.
Considering the concerns on memory and time complexity of unrolling the computational graph in meta learning, a variety of works introduce various surrogate matching objectives to update  $S$ . Zhao et al. [60] propose to match training gradients of  $S$  with those of  $\mathcal{T}$ , and following researches [20, 27, 16, 57, 45] focus on improving the classical gradient-matching objective. Beyond the single-step gradient, Cazenavette et al. [2] and subsequent works [4, 8] consider regulating multi-step training effects and propose matching training trajectories. Without the necessity of calculating higher order gradients, the distribution matching methods [59, 50] minimize the distance between feature statistics of  $S$  and  $\mathcal{T}$ , and result in satisfactory computational efficiency. Another branch of methods [37, 38, 61, 32, 33] turn to kernel ridge regression (KRR) model to improve the efficiency of the seminal meta learning based solution, since KRR enjoys the analytical form of solution, which gets rid of the meta-training process and yields best trade-off between performance and efficiency.
Different from works focusing on objectives of DD, some other researches explore methods of synthetic data parameterization, where synthetic samples can be stored in some other formats except the raw one to improve the data efficiency, and raw samples are recovered via some functions during down-stream training, e.g., data augmentation [58], up-sampling [20], linear transformation [7], and neural networks [30, 26, 3, 49].
For a thorough introduction to dataset distillation, we refer readers to the recent surveys for this area [56, 13, 43, 28]. For existing methods, no matter what objectives and parameterizations are adopted, they all rely on an intensive loop of forward-backward propagation through a massive number of neural networks. Although a concurrent work [31] also adopts a similar pre-training and adaptation pipeline, it still relies on an iterative loop to solve an initialization of synthetic data.
Figure 2: Results of different DD methods on CIFAR100 with 1 image per class. Our MGDD achieves state-of-the-art efficiency and at least comparable performance. DS denotes down-sampling parameterization.
Different from prior works, we innovatively introduce a feed-forward fashion for dataset distillation in this paper. In fact, our method is orthogonal to different training objectives and data parameterizations, and can be built upon any combination of them. In this paper, we consider the KRR-based objective thanks to its favorable performance and computational efficiency and experiment with both raw and down-sampled parameterizations.
In this section, we elaborate on the technical methods of the proposed MGDD pipeline. We first introduce in Sec. 3.1 some preliminary information related to the matching objective. Then, for the main method, according to the overview of the whole pipeline in Fig. 1(b), the final synthetic labels and samples are derived by solving a least-squares problem and a conditional generator given initial synthetic data, which would be illustrated in detail in Sec. 3.2 and 3.3 respectively. Finally in Sec. 3.4, we analyze the feasibility of this two-step pipeline theoretically.
Let us denote datasets before and after distillation as  $\mathcal{T} = (X_t,Y_t)$  and  $\mathcal{S} = (X_s,Y_s)$  respectively, where  $X_{t}\in \mathbb{R}^{n_{t}\times d}$ ,  $Y_{t}\in \mathbb{R}^{n_{t}\times c}$ ,  $X_{s}\in \mathbb{R}^{n_{s}\times d}$ ,  $Y_{s}\in \mathbb{R}^{n_{s}\times c}$ ,  $n_t$  and  $n_s$  are number of real and synthetic data respectively,  $d$  and  $c$  are input and output dimensions respectively. Typically, for the RGB image classification task,  $d$  is equal to  $h\times w\times 3$ ,  $c$  is the number of classes, and  $Y_{t}$  is organized in the one-hot format. For the objective of DD, in this paper, we mainly consider the KRR-based methods in neural feature spaces [61, 32] due to its overall superior performance in terms of accuracy and efficiency. Specifically, assume there is an optimal neural network  $f_{\theta^*}$  to projects  $X_{t}$  and  $X_{s}$  to a feature space with  $p$  dimensions and  $n_s\ll n_t < p$ . The prediction error on  $\mathcal{T}$  for the optimal KRR
parameter solved by  $S$ , denoted as  $W_{s}^{\theta^{*}}$  is adopted as the loss function:
$$
\mathcal {L} (\mathcal {S}; \theta^ {*}) = \| f _ {\theta^ {*}} (X _ {t}) W _ {s} ^ {\theta^ {*}} - Y _ {t} \| _ {2} ^ {2} = \| f _ {\theta^ {*}} (X _ {t}) f _ {\theta^ {*}} (X _ {s}) ^ {\top} \left(f _ {\theta^ {*}} (X _ {s}) f _ {\theta^ {*}} (X _ {s}) ^ {\top}\right) ^ {- 1} Y _ {s} - Y _ {t} \| _ {2} ^ {2}. \tag {1}
$$
In practice, since the optimal parameter  $\theta^{*}$  is unknown, it is approximated by different random initializations [32] or alternately optimization with  $\mathcal{S}$  [61].
3.2 Solving Synthetic Labels
Through Eq. 1, we can find that the loss in a neural space  $\theta$  is upper-bounded by the distance between parameters solved by  $S$  and  $\mathcal{T}$ :
$$
\begin{array}{l} \mathcal {L} (\mathcal {S}; \theta) = \| f _ {\theta} (X _ {t}) W _ {s} ^ {\theta^ {*}} - Y _ {t} \| _ {2} ^ {2} = \| f _ {\theta} (X _ {t}) W _ {s} ^ {\theta} - f _ {\theta^ {*}} (X _ {t}) f _ {\theta} (X _ {t}) ^ {\dagger} Y _ {t} \| _ {2} ^ {2} \\ = \| f _ {\theta} (X _ {t}) W _ {s} ^ {\theta} - f _ {\theta} (X _ {t}) W _ {t} ^ {\theta} \| _ {2} ^ {2} \leq \| f _ {\theta} (X _ {t}) \| _ {2} ^ {2} \| W _ {s} ^ {\theta} - W _ {t} ^ {\theta} \| _ {2} ^ {2} \tag {2} \\ = \| f _ {\theta} \left(X _ {t}\right) \| _ {2} ^ {2} \| f _ {\theta} \left(X _ {s}\right) ^ {\dagger} Y _ {s} - W _ {t} ^ {\theta} \| _ {2} ^ {2}, \\ \end{array}
$$
where  $\dagger$  denotes the pseudo-inverse of a matrix. In our MGDD framework, synthetic samples  $X_{s}$  are initialized as some random real samples in  $X_{t}$ . Given a fixed  $X_{s}$  and a random network  $f_{\theta}$ , the upper bound in Eq. 2 forms a least-squares problem with respect to synthetic labels  $Y_{s}$ , which can be minimized by an analytically optimal solution:
$$
Y _ {s} ^ {*} = f _ {\theta} \left(X _ {s}\right) W _ {t} ^ {\theta} = f _ {\theta} \left(X _ {s}\right) f _ {\theta} \left(X _ {t}\right) ^ {\top} \left(f _ {\theta} \left(X _ {t}\right) f _ {\theta} \left(X _ {t}\right) ^ {\top}\right) ^ {- 1} Y _ {t}. \tag {3}
$$
$Y_{s}^{*}$  obtained with Eq. 3 serves as final synthetic labels. In the next subsection, we will introduce the generation of synthetic samples conditioned on their initialization.
3.3 Learning a Synthetic Sample Generator
Conditioned on initialized synthetic samples  $X_{s}$ , a generator  $g_{\omega}$  is adopted to predict the final synthetic data  $X_{s}^{*}$ , where the parameter  $\omega$  can encode useful information of the target dataset  $\mathcal{T}$  and the optimal neural space parameterized by  $\theta^{*}$ . We expect that the generator can acquire such knowledge through a fast learning process within a limited number of training steps. To this end, we propose a learning-to-learn algorithm based on MAML [12], where a meta generation network is learned to optimize the performance of the network adapted for a few steps from the meta one.
Specifically, to ensure the generality of the meta generator for different target datasets, we perform the training algorithm on ImageNet1k [6], a large-scale dataset for image classification. In each training iteration, a subset of all classes is randomly sampled from it to mimic different target datasets that the generator may encounter in practice. And the meta generator is learned in a bi-level learning framework including a meta-training loop and a meta-testing step, and the meta-testing loss is backpropagated through the computational graph of meta-training steps to update the parameter of the meta generator.
In each meta-training and meta-testing step, from the selected classes, we randomly sample a batch of real images as  $\mathcal{T}$  and initialize synthetic data  $X_{s}$  with part of them. With a random and fixed neural network as  $f_{\theta}$ , the synthetic labels  $Y_{s}^{*}$  are solved via Eq. 3. Then, the final synthetic samples  $X_{s}^{*}$  are predicted by the current generator in a forward pass, and  $\mathcal{S} = (X_s^*, Y_s^*)$  is evaluated by the loss  $\mathcal{L}(\mathcal{S};\theta^{*})$  in Eq. 1. In this paper, following Loo et al. [32], we approximate the optimal neural parameter  $\theta^{*}$  via random sampling in different steps from the distribution for initialization. The loss signal is back-propagated to the current generator and the meta generator to update parameters in meta-training and meta-testing respectively. It is worth noting that in different meta-training and meta-testing times, we use different sizes of synthetic data, which enhances the cross-size generalization ability on target datasets since the meta-testing losses on sizes probably unseen during meta-training are optimized. The main algorithm is summarized in Alg. 1. Given a trained meta generator, a limited number of adaptation steps are performed for a target dataset. The procedure of adaptation is similar to the meta-training loop in Alg. 1.
As for the architecture of the generator, in this paper, we adopt a simple encoder-decoder model, where the encoder consists of three convolutional blocks with two average pooling layers while the decoder has a symmetric structure. Notably, we observe in practice that it is beneficial for different sizes of synthetic datasets to adopt different transfer functions. Taking various sizes into consideration, we concatenate additional size-embedding channels to the bottle-necked layer of the
Algorithm 1 MGDD: Meta-Learning Framework for Training a Synthetic Sample Generator
Input:  $\mathcal{Z}$  : A Large Dataset;  $\theta$  : Parameter of a Random Neural Network;  $T$  : Number of Meta
Training Steps;  $\alpha$ : Learning Rate in Meta-Training;  $\beta$ : Learning Rate in Meta-Testing.
Output:  $\omega$ : Parameter of a Meta Generator.
1: procedure GETTRAININGLOSS(T)  
2: Initialize  $X_{s}$  with some random real images from  $T$ ;  
3: Obtain  $Y_{s}^{*}$  with the analytical solution in Eq. 3;  
4: Forward propagation with  $X_{s}^{*} \gets g_{\omega'}(X_{s})$ ;  
5: Randomly sample  $\theta^{*}$  and compute the loss  $\mathcal{L}((X_s^*,Y_s^*);\theta^*)$  in Eq. 1;  
6: return  $\mathcal{L}((X_s^*,Y_s^*);\theta^*)$   
7: end procedure  
8: Initialize  $\omega$  randomly;  
9: repeat  
10:  $\omega' \gets \mathrm{copy}(\omega)$ ;  
11: Randomly choose a subset of classes  $C$  from  $Z$ ;  
12: Sample a batch of images of  $C$  as  $T$ ;  
13: for  $1 \leq i \leq T$  do  
14:  $\mathcal{L} = \mathrm{GetTrainingLoss}(T)$ ;  
15: Back propagation and update  $\omega': \omega' \gets \omega' - \alpha \nabla_{\omega'} L$ ;  
16: end for  
17:  $\mathcal{L} = \mathrm{GetTrainingLoss}(T)$ ;  
18: Back propagation and update  $\omega: \omega \gets \omega - \beta \nabla_{\omega} L$ ;  
19: until convergence
generator, inspired by the design of the position embedding in Transformer models [47] and the time-step embedding in diffusion models [15, 39, 41]. Please refer to the appendix for detailed architecture configurations.
3.4 Theoretical Analysis
There are two key steps in the proposed MGDD framework: solving the optimal synthetic labels  $Y_{s}^{*}$  as introduced in Sec. 3.2 and generating the corresponding synthetic samples  $X_{s}^{*}$  as introduced in Sec. 3.3. Define the error  $\mathcal{L}(S;\theta)$  in Eq. 1 using the fixed  $X_{s}$  and the optimal  $Y_{s}$  in Eq. 3 with the projection function of  $f_{\theta}$  as  $\epsilon$ . The reason we pursue the optimal  $Y_{s}$  is that the final  $X_{s}^{*}$  is transferred from the initial  $X_{s}$ , whose error in the optimal neural space parameterized by  $\theta^{*}$  is upper-bounded by  $\epsilon$ , as derived in the following theorem.
Theorem 1. Given  $f_{\theta} : \mathbb{R}^d \to \mathbb{R}^p$ ,  $f_{\theta^*} : \mathbb{R}^d \to \mathbb{R}^p$ ,  $X_t \in \mathbb{R}^{n_t \times d}$ ,  $Y_t \in \mathbb{R}^{n_t \times c}$ ,  $X_s \in \mathbb{R}^{n_s \times c}$ ,  $d < p$ ,  $Y_s^*$  obtained by Eq. 3 with the optimal  $\mathcal{L}((X_s, Y_s^*); \theta)$  denoted as  $\epsilon$ , and an arbitrary transfer function  $g_{\omega}$  parameterized by  $\omega$  taking  $X_s$  as input, the transferred  $g_{\omega}(X_s)$  yields an upper-bounded loss  $\mathcal{L}((g_{\omega}(X_s), Y_s^*); \theta^*)$ .
Proof. We first rewrite the given condition:
$$
\mathcal {L} \left(\left(X _ {s}, Y _ {s} ^ {*}\right); \theta\right) = \| f _ {\theta} \left(X _ {t}\right) W _ {s} ^ {\theta} - Y _ {t} \| _ {2} ^ {2} = \| f _ {\theta} \left(X _ {t}\right) f _ {\theta} \left(X _ {s}\right) ^ {\top} \left(f _ {\theta} \left(X _ {s}\right) f _ {\theta} \left(X _ {s}\right) ^ {\top}\right) ^ {- 1} Y _ {s} ^ {*} - Y _ {t} \| _ {2} ^ {2} = \epsilon . \tag {4}
$$
$$
\begin{array}{l} \mathcal {L} \left(\left(g _ {\omega} \left(X _ {s}\right), Y _ {s} ^ {*}\right); \theta^ {*}\right) = \left\| f _ {\theta^ {*}} \left(X _ {t}\right) W _ {s} ^ {\theta^ {*}} - Y _ {t} \right\| _ {2} ^ {2} \\ = \| f _ {\theta^ {*}} \left(X _ {t}\right) f _ {\theta^ {*}} \left(g _ {\omega} \left(X _ {s}\right)\right) ^ {\top} \left(f _ {\theta^ {*}} \left(g _ {\omega} \left(X _ {s}\right)\right) f _ {\theta^ {*}} \left(g _ {\omega} \left(X _ {s}\right)\right) ^ {\top}\right) ^ {- 1} Y _ {s} ^ {*} - Y _ {t} \| _ {2} ^ {2} \tag {5} \\ \leq \left\| f _ {\theta^ {*}} \left(X _ {t}\right) W _ {s} ^ {\theta^ {*}} - f _ {\theta} \left(X _ {t}\right) W _ {s} ^ {\theta} \right\| _ {2} ^ {2} + \left\| f _ {\theta} \left(X _ {t}\right) W _ {s} ^ {\theta} - Y _ {t} \right\| _ {2} ^ {2} \\ = \left\| f _ {\theta^ {*}} \left(X _ {t}\right) W _ {s} ^ {\theta^ {*}} - f _ {\theta} \left(X _ {t}\right) W _ {s} ^ {\theta} \right\| _ {2} ^ {2} + \epsilon , \\ \end{array}
$$
where the inequality is based on the triangle inequality and the last equation is due to Eq. 4.
Theorem 1 indicates that to achieve feed-forward dataset distillation, we do not need to design a neural network taking the whole original dataset  $\mathcal{T}$  as input. We can in fact adopt a conditional
Dataset	CIFAR10	CIFAR100
IPC Ratio (%)	1 0.02	10 0.2	50 1	1 0.2	10 1
Random Full Dataset	Acc. (%)	21.87±0.30	38.86±0.31	57.07±0.54	7.06±0.16	24.50±0.08
Acc. (%)	84.8±0.1	56.2±0.3
DC [60]	Acc. (%)	28.20±0.71	43.74±0.41	53.43±0.28	12.44±0.18	25.08±0.17
Time (sec.)	153	3605	20090	1496	33749
DSA [58]	Acc. (%)	28.10±0.72	52.15±0.48	60.58±0.29	13.81±0.21	32.49±0.30
Time (sec.)	172	3871	21217	1667	35905
IDC [20]	Acc. (%)	35.34±0.87	58.50±0.39	69.32±0.30	17.93±0.15	36.08±0.38
Time (sec.)	39062	40244	45888	189811	198101
MTT [2]	Acc. (%)	45.29±0.86	62.77±0.56	71.09±0.34	24.17±0.57	39.43±0.26
Time (sec.)	2972	8588	8601	8063	10295
DM [59]	Acc. (%)	27.08±0.36	48.80±0.31	62.94±0.28	11.51±0.25	29.33±0.23
Time (sec.)	1123	1182	1423	10942	11355
RTP [7]	Acc. (%)	49.1±0.6*	62.4±0.4*	70.5±0.4*	21.3±0.6*	34.7±0.5*
Time (sec.)	816212	1647559	1660726	842801	1672205
FRePo [61]	Acc. (%)	43.24±0.32	65.76±0.72	71.03±0.34	27.07±0.26	39.97±0.32
Time (sec.)	8507	11133	29185	11231	37607
Ours	Acc. (%)	46.26±0.27	60.76±0.38	69.50±0.17	24.66±0.15	36.47±0.27
Time (sec.)	120×6802	505×22	1395×6	505×22	3004×13
Label (sec.)	7	7	7	7	7
Generator (sec.)	113	498	1388	498	2997
Forward (ms)	2	3	11	3	20
Table 1: Comparisons on test accuracy and running time with state of the arts. The acceleration marked by the red subscript is computed against the method with the best accuracy. We also provide detailed analysis for our method on the time cost for each component in adaptation, including solving synthetic labels, updating the generator from the meta generator, and feedforward generation. IPC: Number of Images Per Class; Ratio: ratio of distilled images to the whole training set. * denotes results from the original paper.
generation function that transfers the initial synthetic samples to the desired ones, which has an error upper bound as shown in Theorem 1. Since the upper bound is related to the error in the original space parameterized by  $\theta$ , it is crucial to solve the optimal synthetic labels with respect to  $\theta$  in the first step. Also, we notice from Eq. 5 that the optimal generator is dependent on  $\mathcal{T}$  and  $\theta^{*}$ . Given that  $\theta^{*}$  is intractable and can only be approximated by iterative sampling, we build a meta learning algorithm in Alg. 1 for the MGDD framework to enforce an efficient process to model this dependency in only a few steps.
4.1 Implementing Details