SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notably, our results achieved with IPC 1 storage space are highly competitive with the previous state-of-the-art results [21, 37] obtained with IPC 10. Specifically, we only need to employ  $10\%$  of their parameters to achieve similar or even better performance. Compared to the previous state-of-the-art [21] using the same IPC 1 storage space, our approach exhibits an average improvement of  $11.2\%$  across all subsets. Moreover, we maintain a substantial lead for IPC 10 storage space. For instance, we achieve  $71.8\%$  accuracy on ImageSquawk, which is a  $15.0\%$  improvement over the previous state-of-the-art [21]. These outstanding outcomes are attributed to our design of sharing SAETs among patches, which enables SPEED to be more effective in reducing spatial redundancy as data resolution increases.
Continual Learning. Owing to its expressive representations and finer-grained image construction, SPEED has the ability to synthesize an informative dataset. Synthetic images of each class are of exceptional quality, thereby contributing to the dataset's overall richness and relevance. Following the DM [17] setup based on GDumb [51], we conduct continual learning experiments on CIFAR100 with IPC 20 storage space and use the default ConvNet and ResNet18 for evaluation. We randomly divide the 100 classes into 5 learning steps, that is, 20 classes per step. As illustrated in Fig. 2, SPEED maintains the highest test accuracy at all steps for both evaluation networks.
3.2 Generalization
Universality to Matching Objectives. SPEED exhibits adaptability to multiple existing matching objectives, including those presented in [13, 16, 17], and can be directly integrated with them. In line with [52], we evaluate a diverse set of architectures, including the default ConvNet, ResNet [2], MLP, and ViT [3]. The results and comparisons to corresponding baselines on CIFAR10 are presented in Tab. 3. For instance, when using MLP for evaluation under the IPC 1 budget, SPEED yields a
(a) ConvNet
(b) ResNet18
Figure 2: Test accuracy of continual learning. SPEED maintains the best performance on all steps.
(a) ConvNet
Figure 3: Robustness of the synthetic dataset on different architectures. Each direction represents each type of corruption.
Evaluation	ConvNet	MLP	ResNet18	ViT
Matching	IPC	1	10	50	1	10	50	1	10	50	1	10	50
Gradient	DC [13]w. SPEEDGain	29.3±0.448.5±0.3+19.2	51.0±0.652.6±0.3+1.6	56.8±0.459.8±0.4+3.0	29.0±0.535.5±0.2+6.5	34.1±0.439.0±0.4+4.9	31.6±0.641.5±0.3+9.9	27.4±0.742.5±0.5+15.1	44.0±1.447.5±0.5+3.5	45.9±1.455.0±0.3+9.1	28.1±1.130.4±0.6+2.3	34.4±0.438.4±0.4+4.0	30.1±0.539.0±0.3+8.9
Distribution	DM [17]w. SPEEDGain	26.5±0.445.0±0.4+18.5	47.6±0.662.0±0.3+14.4	62.0±0.366.4±0.3+4.4	10.0±0.634.8±0.3+24.8	34.4±0.342.2±0.5+7.8	40.5±0.446.3±0.4+5.8	20.6±0.540.7±0.8+20.1	38.2±1.157.6±0.5+19.4	52.8±0.465.2±0.3+12.4	20.5±0.528.4±0.6+7.9	34.4±0.543.2±0.2+8.8	45.2±0.448.9±0.3+3.7
Trajectory	TM [16]w. SPEEDGain	44.2±1.263.2±0.1+19.0	63.7±0.473.5±0.2+9.8	70.3±0.677.7±0.4+7.4	10.4±0.534.1±0.2+23.7	30.8±0.644.4±0.4+13.6	38.5±0.347.7±0.2+9.2	34.2±1.453.9±0.7+19.7	45.2±1.469.5±0.4+24.3	60.0±0.776.4±0.3+16.4	21.5±0.433.6±0.637.5±0.8+16.0	33.6±0.651.5±0.3+17.9	47.7±0.654.5±0.5+6.8
significant improvement of  $24.8\%$  and  $23.7\%$  in accuracy compared to the distribution [17] and trajectory [16] matching baselines, respectively. Our experiments reveal that the improvements in cross-architecture accuracy tend to be more significant than those on the ConvNet. For instance, with an IPC 10 budget for trajectory matching [16], we achieve a  $24.3\%$  gain on ResNet18. This outcome further showcases the strengths of our sparse parameterization approach in terms of generalization.
Cross-Architecture Performance. The main purpose of dataset distillation is to distill a synthetic dataset that is effective on various even unseen architectures. In this study, we evaluate the cross-architecture performance of our method by comparing it with previous synthetic data parameterization approaches [19, 21, 22] on CIFAR10, using IPC 10 storage space. The results presented in Tab. 4 demonstrate that SPEED continues to
outperform other methods significantly in terms of generalization across unseen architectures. As an illustration, our method achieves an accuracy of  $51.5\%$  on ViT, which represents a  $3.6\%$  improvement over the previous state-of-the-art [19]. Although various existing dataset distillation matching objectives tend to overfit the training network, we address this challenge by pruning unnecessary features through sparse parameterization.
Table 3: Universality to different matching objectives. SPEED is compatible with a variety of matching objectives and brings significant accuracy improvements over the corresponding baseline methods. All experiments are conducted with DSA augmentation [14].
Method	ConvNet	MLP	ResNet18	ViT
IDC [19]	67.5±0.5	41.4±0.2	62.9±0.6	47.9±0.8
HaBa [21]	69.9±0.4	35.4±0.4	60.2±0.9	42.2±0.6
RTP [22]	71.2±0.4	27.2±0.2	67.5±0.1	35.7±0.4
SPEED	73.5±0.2	44.4±0.4	69.5±0.4	51.5±0.3
Table 4: Comparison of cross-architecture generalization evaluation. SPEED exhibits outstanding leads on all architectures.
Robustness to Corruption. To explore the out-of-domain generalization of our synthetic dataset, we conduct experiments on CIFAR100-C [53]. In detail, we evaluate on ConvNet and ResNet18, using the synthetic dataset trained under the IPC 1 budget. Fig. 3 shows the average accuracy of 14 types of corruption under 5 levels respectively. Compared with the previous methods, we achieve better performance under all kinds of corruption. Especially on ResNet18, SPEED outperforms previous methods significantly, achieving almost double the test accuracy under every corruption scenario. This demonstrates the generalization and robustness benefits brought by sparse parameterization.
Size of  $k$  for Feature Sparsification. Our total storage parameters are positively correlated with the size of  $k$ , and the representation ability of the SCM is also related to this  $k$ -winner. Therefore, we
(a) Before feature sparsification
(b) After feature sparsification (density  $= 0.3\%$ )
Figure 4: ImageSquawk: synthetic data samples before and after feature sparsification. The process of sparsification does not lead to a significant loss of essential information, resulting in a high degree of similarity between the two images.
k	#Param	Budget	ConvNet	MLP	ResNet18	ViT
full	15M	-	74.0±0.2	44.8±0.3	69.0±0.4	50.8±0.5
96	575K	-	74.6±0.3	44.9±0.1	69.3±0.7	51.8±0.5
48	307K	✓	73.5±0.2	44.4±0.4	69.5±0.4	51.5±0.3
24	173K	✓	68.2±0.3	35.9±0.7	66.4±0.5	45.2±0.4
12	106K	✓	57.8±0.3	24.8±0.4	59.0±0.4	36.7±0.9
R	N/c	ConvNet	MLP	ResNet18	ViT
1	18	20.7±0.4	4.5±0.2	17.3±0.5	9.9±1.1
2	17	26.9±0.3	6.0±0.3	21.3±0.6	14.9±0.3
3	15	25.4±0.1	5.3±0.1	19.5±0.6	13.9±0.2
Table 5: Different  $k$  for feature sparsification.
K	N/c	ConvNet	ResNet18
32	11	38.5±0.2	29.8±0.5
64	11	40.0±0.4	29.9±0.3
96	10	37.3±0.1	29.3±0.4
128	10	38.2±0.3	29.6±0.4
Table 7: Number of SAETs.
D	N/c	ConvNet	ResNet18
48	13	38.0±0.1	29.0±0.4
72	12	38.7±0.3	29.8±0.5
96	11	40.0±0.4	29.9±0.3
144	7	35.5±0.4	26.7±0.3
Table 8: Dimension of SAETs.
Table 6: Depth of FReeNet.
H	N/c	ConvNet	ResNet18
1	33	38.7±0.1	32.2±0.4
2	16	39.3±0.4	30.5±0.3
3	11	40.0±0.4	29.9±0.3
4	8	35.9±0.4	27.4±0.8
Table 9: Head of SAETs.
discuss the preference for the value of  $k$ . Specifically, for cross-architecture evaluation on CIFAR10 with IPC 10 storage space, we set  $k$  to {4096, 96, 48, 24, 12}, as shown in Tab. 5 (Budget denotes whether the storage budget is met). We observe that taking a moderate value of  $k$  can effectively improve the cross-architecture generalization ability of the synthetic dataset while maintaining high test accuracy on the homogeneous network, i.e., the training backbone. When  $k$  is set to 96, the overall accuracy exceeds that of the result obtained without feature sparsification, thereby demonstrating the effective removal of inefficient features and the improvement in generalization. In order to meet the storage budget  $(10 \times 10 \times 48 \times 64 \approx 307\mathrm{K})$ , we finally set  $k$  to 48, which maintains competitive performance.
Feature Sparsification Visualizations. To compare the representation capabilities of SCMs after feature sparsification, we provide some visualization results on ImageSquawk, as shown in Fig. 4. We note that the visualization results using only the sparsified features are highly similar to the corresponding original ones. Differences are noticeable only in a few patches, which are insignificant for classification and mainly consist of meaningless backgrounds. This implies that synthetic images can be constructed effectively using only a few prominent features.
Trade-off between Quality and Quantity. Eq. (7) illustrates that SPEED's parameters consist of three parts. The parameters of SAETs and FReeNet are shared among all synthetic images, while the parameters of SCMs are specific to each synthetic image. Thus, under the same budget, increasing shared parameters will lead to a decrease in the number of synthetic images. We conduct ablation experiments on CIFAR100 with IPC 1 storage space to study the trade-off between quality (more shared parameters) and quantity (more synthetic images). Tab. 7-9 study the size of SAETs. As can be seen, keeping a moderate number of synthetic images is important, i.e.,  $N / c > 10$ . A lack of diversity caused by too few images can impact the performance. However, once a certain number of images is reached, further increasing the quantity can lead to a decrease in the quality of each image and slightly reduce performance. In general, our performance is insensitive to the size of SAETs
and outperforms the previous state-of-the-art [22] in multiple settings, except when the number of synthetic images is too small. Tab. 6 studies the depth of FReeNet, we conduct ablation experiments on TinyImageNet with IPC 1 storage space. Although using only one block can maximize the number of synthetic images per class  $(N / c)$ , the lack of hierarchical features severely compromises the model's representational capacity, resulting in poor image quality. However, having too many blocks can result in increased reconstruction overhead. We find that FReeNet with two blocks already achieves a great compromise between performance and computation cost.
Effects of Increasing Image Resolution. To study the effectiveness of our method in higher resolution scenarios, we performed experimental investigations on both the distribution matching [17] baseline and our method, using the ImageNette dataset with image sizes of  $128 \times 128$  and  $256 \times 256$ . In line with the practices of previous methods that handle higher resolution images with deeper networks, we increased the depth of the ConvNet to 6 for the  $256 \times 256$  image size.
128×128	256×256	Gain
DM [17]	28.6±0.6	29.5±1.1	+0.9
w. SPEED	53.5±1.2	57.7±0.9	+4.2
Gain	+24.9	+28.2	+3.3
Table 10: Effects of increasing image resolution on ImageNette.
As shown in Tab. 10, when the resolution increases from  $128 \times 128$  to  $256 \times 256$ , the gain brought by SPEEDD to the baseline also amplifies from  $24.9\%$  to  $28.2\%$ . The results demonstrate that our method achieves more substantial improvements when applied to higher-resolution images.
Dataset Distillation. Dataset distillation, proposed by Wang et al. [12], aims to learn a smaller synthetic dataset so that the test performance of the model on the synthetic dataset is similar to that of the original dataset. For better matching objectives, Zhao et al. [13] present a single-step matching framework, encouraging the result gradient of the synthetic dataset and the original dataset to be similar, further extended by [15, 19, 54, 55]. Subsequently, Cazenavette et al. introduce TM [16] to alleviate the cumulative error problem of single-step matching, which inspires a series of work [56-58]. To avoid the expensive computational overhead brought by complex second-level optimization, Zhao et al. [17] suggest DM, a distribution matching approach, and Wang et al. [18] explicitly align the synthetic and real distributions in the feature space of a downstream network. There are also some methods based on kernel ridge regression [59, 47, 60, 37], which can bring out a closed-form solution for the linear model, avoiding extensive inner loop training.
In terms of synthetic data parameterization, Kim et al. introduce IDC [19], using downsampling strategies to synthesize more images under the same storage budget. Zhao et al. [20] propose to synthesize informative data via GAN [61, 62]. Deng et al. [22] have explored how to compress datasets into bases and recall them by linear combinations. Liu et al. [21] propose a dataset factorization approach, utilizing image bases and hallucinators for image synthesis. Lee et al. [63] further factorize the dataset into latent codes and decoders. These parameterization methods are designed to find shareable or low-resolution image bases. Still, they either only consider the connections between synthetic images [21, 22, 63], or do not reduce the redundancy inside the synthetic image thoroughly [19]. So the storage space of their image bases is still proportional to the dataset resolution, bringing out unsatisfactory performance on high-resolution datasets. SPEED is a universal synthetic data parameterization framework, in which the distillation and construction are spatial-agnostic. It allows for joint modeling correlations between and within the synthetic images, leading to lower storage redundancy and finer-grained image synthesis, thus performing competently on high-resolution datasets.
Sparse Coding. Sparse coding calls for constructing efficient representations of data as a combination of a few high-level patterns [31-36]. It has been proven to be an effective approach in the field of image reconstruction and image classification [64-70]. Typically, a dictionary of basis functions (e.g. wavelets [71] or curvelets [72]) is used to decompose the image patches into coefficient vectors. By imposing sparsity constraints on the coefficient vectors, efficient sparse representations of the image patches can be obtained. In addition, the dictionary is not necessarily fixed, it can also be learned to adapt to different tasks [25-28], and the penalty function can also be varied [33, 73, 74]. Prior sparse coding research has primarily focused on compressing individual images [75, 76], with an emphasis on achieving high compression ratios while minimizing perceptual distortion. However, there has been limited exploration of how to apply sparse coding to compress entire datasets in a
way that enhances the training of downstream neural networks. Our work demonstrates that theories and techniques in sparse coding can provide valuable inspiration for developing dataset distillation methods.
**Coreset Selection.** Coreset selection identifies a representative subset of the original dataset [77-79, 10]. Its objective is in line with the goals of dataset distillation and can be utilized to tackle challenges such as continual learning [10, 80, 81] and active learning tasks [82]. This technique typically performs better when the storage budget is relatively large, while dataset distillation demonstrates superior performance under extremely limited storage budgets [52].
In this paper, we introduce SPEED, an efficient and generalizable solution for dataset distillation that offers the following merits: First, the spatial-agnostic epitomic tokens distilled by our method are not only shared between the different classes but also shared among patches of every synthetic image, regardless of their spatial locations. Such efficient modeling enables us to perform well on high-resolution datasets with much less spatial redundancy. Second, the proposed feature-recurrent network promotes hierarchical representation in an efficient recurrent manner, resulting in more informative synthetic data. Finally, the proposed feature sparsification mechanism improves both the storage efficiency and the generalization ability. SPEED achieves outstanding performance on various datasets and architectures through extensive experiments.
Limitations. Similar to the previous parameterization methods [19-22], decomposing the original synthetic datasets into different components will slightly increase distilling costs and incur reconstruction overhead. We alleviate this issue by designing the network to be as lightweight as possible. While the dataset synthesized by SPEED offers better privacy protection for users compared to the original dataset, there remains a possibility of privacy leakage.
This work was supported in part by the National Natural Science Foundation of China under Grant 62006183 and Grant 62206271, in part by the National Key Research and Development Project of China under Grant 2020AAA0105600, and in part by the Shenzhen Key Technical Projects under Grant JSGG20220831105801004, CJGJZD2022051714160501, and JCYJ20220818101406014.
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[6] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In ICML, 2020.
[7] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2020.
[77] Max Welling. Herding dynamical weights to learn. In ICML, 2009.
[78] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In UAI, 2010.
[79] Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In NeurIPS, 2011.
[80] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurIPS, 2019.
[81] Felix Wiewel and Bin Yang. Condensed composite memory continual learning. In IJCNN, 2021.
[82] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
[83] Fastai. Fastai/imagenette: A smaller subset of 10 easily classified classes from imagenet, and a little more french.
[84] Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source differentiable computer vision library for pytorch. In WACV, 2020.
[85] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[86] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[87] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017.
[88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Datasets and Preprocessing. We evaluate our methods on the following datasets: i) CIFAR10 [48]: A standard image dataset consists of 60,000  $32 \times 32$  RGB images in 10 different classes, including airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. For each class, 5000 images are used for training and 1000 images are used for testing. ii) CIFAR100 [48]: CIFAR100 contains 100 classes. It has a training set with 50,000 images and a testing set with 10,000 images. iii) TinyImageNet [49]: A  $64 \times 64$  image dataset with 200 classes. Each class has 500 images for training and 50 images for testing. iv) ImageNet [50] subsets: High resolution ( $128 \times 128$ ) datasets from ILSVRC2012 [50]. ImageNette (assorted objects) and ImageWoof (dog breeds) are designed for easy and hard classification tasks [83]. ImageFruit, ImageMeow, ImageSquawk, and ImageYellow [16] consist of fruits, cats, birds, and yellowish things, respectively. All the above subsets contain 10 classes.
For fair comparisons, we follow the previous methods [16, 21], adopting ZCA whitening on CIFAR10 and CIFAR100 with the default Kornia [84] implementation, and no ZCA whitening is used on TinyImageNet and ImageNet subsets.
Evaluation Settings. The objective of dataset distillation is to learn a synthetic dataset that can be utilized across a range of network structures. This is achieved by matching the raw and synthetic datasets with the aid of a specific network, with the expectation that can generalize to unseen network structures. In accordance with the evaluation presented in [52], we consider a diverse set of architectures, as shown below:
- ConvNet [1, 85, 86]: The standard architecture used for both distilling and evaluating synthetic datasets in previous distillation work. The default network contains three  $3 \times 3$  convolution layers, each followed by  $2 \times 2$  average pooling and instance normalization. The hidden embedding size is set to 128. There are around 320K trainable parameters. For TinyImageNet, the number of layers is increased to 4 for improved performance, as suggested in previous work [16, 17]. Similarly, we increase it to 5 for ImageNet subsets, following [16].
- MLP: The simple MLP architecture is applied for evaluation, which includes 3 fully connected layers, and the width is set to 128. For CIFAR10, CIFAR100, and TinyImageNet, the MLP has 411K, 423K, and 2M trainable parameters, respectively.
- ResNet18 [2]: We also evaluate synthetic datasets on the commonly used ResNet architecture with 4 residual blocks. In each block, there are 2 convolution layers followed by ReLU activation and instance normalization (IN) [87]. The number of trainable parameters is around 11M.
- ViT [3]: Vision Transformer applies standard transformer [88] on non-overlapping image patches, demonstrating the variants of transformers can also be a competitive alternative to CNNs. We take it as one of the architectures for evaluating synthetic datasets. There are around 10M trainable parameters in the adopted implementation of ViT.
Following the mainstream evaluation settings, we train 5 randomly initialized evaluation networks on the synthetic dataset, using SGD optimizer, where the momentum and weight decay are set to 0.9 and 0.0005, respectively.
Hyper-Parameters. For CIFAR10  $(32 \times 32)$ , CIFAR100  $(32 \times 32)$ , and TinyImageNet  $(64 \times 64)$ , we fix the default number of patches  $J$  to 64, and increase it to 256 for ImageNet subsets  $(128 \times 128)$ . Then, the default patch dimension  $L$  can be directly derived.