SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Numerous training methods exist to enhance the accuracy of models [28], including extended training cycles/budgets and data augmentation strategies such as Mixup [29] and CutMix [30]. We further examine the performance of synthetic data regenerated from models demonstrating varied levels of accuracy. This investigation is aimed at addressing a compelling question: Does a model with superior squeezing ability yield more robust recovered data? Here, "a model with superior squeezing ability" is defined as a model that exhibits strengthened accuracy on the validation set.
Squeezing Budget. In Table 2, we observe a decrement in the performance of the recovered model as the squeezing budget escalates. This suggests an increasing challenge in data recovery from a model that has been trained over more iterations. Consequently, we adopt a squeezing budget of 50 iterations as the default configuration for our experiments on Tiny-ImageNet.
Data Augmentation. Table 2 also shows that data augmentation methods in the squeeze stage decrease the final accuracy of the recovered data. In summary, results on Tiny-ImageNet indicate that extending the training duration and employing data augmentation during the squeeze phase exacerbates the complexity of data recovery from the squeezed model.
3.2 Recovering Analysis
The condensed data are crafted and subsequently relabeled utilizing a pre-trained ResNet-18 with a temperature of 20. Then, we report the val performance of training a ResNet-18 from scratch.
Image Prior Regularization.  $\mathcal{R}_{TV}$  and  $\mathcal{R}_{\ell_2}$  are extensively applied in image synthesis methods [18, 31]. However, in the pursuit of extracting knowledge from the pre-trained model, our focus is predominantly on the recuperation of semantic information as opposed to visual information. Analyzing from the standpoint of evaluation performance as shown in appendix,  $\mathcal{R}_{\ell_2}$  and  $\mathcal{R}_{TV}$  barely contribute to the recovery of image semantic information, and may even serve as impediments to data recovery. Consequently, these image prior regularizations are omitted during our recovery phase.
Multi-crop Optimization. To offset the RandomResizedCrop operation applied to the training data during the subsequent model training phase, we incorporate a corresponding RandomResizedCrop augmentation on synthetic data during recovery. This implies that only a minor cropped region in the synthetic data undergoes an update in each iteration. Our experimentation reveals that the multi-crop optimization strategy facilitates a notable improvement in validation accuracy, as in Appendix Table 8.
A comparative visualization and comparison with other non-crop settings is shown in Fig. 3. In the last column  $(\mathrm{SRe}^2\mathrm{L})$ , multiple miniature regions enriched with categorical features spread across the entire image. Examples include multiple volcanic heads, shark bodies, bee fuzz, and mountain ridges. These multiple small feature regions populate the entire image, enhancing its expressiveness in terms of visualization. Therefore, the cropped regions on our synthetic images are not only more closely associated with the target categories but also more beneficial for soft label model training.
Dataset	Recovery model	Recovering Budget	Time (ms)
0.5k	1k	2k	4k
Tiny-ImageNet	ResNet-18	34.82	37.88	41.35	41.11	0.75
ResNet-50	35.92	39.19	39.59	39.29	2.75
ImageNet-1K	ResNet-18	38.94	43.69	46.71	46.73	0.83
ResNet-50	21.38	28.69	33.20	35.41	2.62
Table 3: Top-1 validation accuracy of ablation using ResNet-{18, 50} as recovery model with different updating iterations, and ResNet-18 as student model. "Time" represents the consuming time (ms) when training 1 image per iteration on one single NVIDIA 4090 GPU.
Figure 3: Visualization of distilled examples on ImageNet-1K under various regularization terms and crop augmentation settings. Selected classes are {Volcano, Hammerhead Shark, Bee, Valley}.
Recover Budget. We conduct various ablation studies to evaluate the impact of varying recovery budgets on the quality of synthetic data. The recovery budgets are designated as  $[0.5k, 1k, 2k, 4k]$ . As in Table 3, it indicates that employing a longer recovery budget on the same model results in superior classification accuracy on the recovered data. In the case of recovery from diverse models, the results demonstrate that the data recovery under the same iterative setting and budget from ResNet-50 is inferior to that from ResNet-18 on both datasets. This suggests that the process of data recovery from larger pre-trained models is more challenging on large-scale datasets, necessitating more iterations to ensure that the recovered data achieves comparable performance on downstream classification tasks. To strike a balance between performance and time cost, we impose a recovery budget of  $1k$  iterations on Tiny-ImageNet and  $2k$  iterations on ImageNet-1K for the ablations, and  $4k$  for the best in Table 4.
3.3 Relabeling Analysis
We further study the influence of varying relabeling models and distinct softmax temperatures on different architectural models being optimized. In Fig. 4, we present three subfigures that represent
Figure 4: Top-1 val accuracy of models trained on various labels and temperature settings under IPC 50. T and S represent the reference model for relabeling and the target model to be trained, separately. R18, R50, and R101 are the abbreviation of ResNet-18, ResNet-50, and ResNet-101.
the training accuracy of the models on labels generated by three pre-trained models: ResNet-{18, 50, 101}. The training data utilized for these experiments is the same, which is recovered from a pre-trained ResNet-18 model.
Relabeling Model. In each row of Fig. 4 (i.e., on the same dataset), the results of three subfigures indicate that a smaller-scale teacher which is close or identical to the recovery model architecture always achieves better accuracy across ResNet-18, ResNet-50 and ResNet-101. Thus, it can be inferred that the labels of the recovered data are most accurate when the relabeling model aligns with the recovery model. Moreover, Top-1 errors tend to escalate with increasing disparity between the relabeling model and the recovery model. Consequently, in our three-stage methodology, we opt to employ an identical model for both recovery and relabeling processes.
Temperature on Soft Label. We conduct experiments encompassing five different temperature selections [1, 5, 10, 15, 20] specifically for label softmax operations under distillation configuration as [22]. The result indicates that the Top-1 accuracy experiences a rapid surge initially and subsequently plateaus when the temperature setting exceeds 10. The maximum Top-1 accuracy is recorded as  $60.81\%$  when the temperature is fixed at 20, employing ResNet-18 as the teacher model and ResNet-101 as the student model. This observation underscores the beneficial effect of a higher temperature setting for label softmax operations in the training of the student model. Consequently, we elect to utilize a temperature value of 20 in our subsequent evaluation experiments.
Model Training. Contrary to previous data condensation efforts, where different architectures could not be effectively trained on the condensed data due to the mismatch of recovery models or overfitting on the limited synthetic data scale, our condensed data demonstrate the informativeness and training scalability as the inherent properties in real data. In each subfigure within Fig. 4, we observe a progressive increase in accuracy when training ResNet-18, ResNet-50, and ResNet-101 as the student networks. This indicates that our condensed data does not suffer from overfitting recovered from the squeezing model, and during inference, the Top-1 accuracy is consistently increasing using a network with enhanced capabilities.
3.4 Condensed Dataset Evaluation
Tiny-ImageNet. The results derived from the Tiny-ImageNet dataset are presented in the first group of Table 4. Upon evaluation, it can be observed that MTT achieves  $28.0\%$  under the IPC 50. In contrast, our results on ResNet-\{18, 50, 101\} architectures are  $41.1\%$ ,  $42.2\%$ , and  $42.5\%$ , respectively, which significantly surpass the performance of MTT. A noteworthy observation is that our stronger backbones not only accomplish superior accuracy but are also robust for different recovery architectures. In light of the results under IPC 50 and 100 settings on the relatively small Tiny-ImageNet dataset, it is apparent that the larger backbone did not yield a proportional enhancement in performance from ResNet-18 to ResNet-101 by our method in Table 4 and Fig. 4. While, this is different from the observation on full ImageNet-1K that we discuss elaborately below.
Dataset	IPC	MTT [1]	TESLA [11]	TESLA [11] (ViT)	TESLA [11] (R18)	SRe2L (R18)	SRe2L (R50)	SRe2L (R101)
Tiny-IN	50	28.0±0.3	-	-	-	41.1±0.4	42.2±0.5	42.5±0.2
100	-	-	-	-	49.7±0.3	51.2±0.4	51.5±0.3
IN-1K	10	64.0±1.3†	17.8±1.3	11.0±0.2	7.7±0.1	21.3±0.6	28.4±0.1	30.9±0.1
50	-	27.9±1.2	-	-	46.8±0.2	55.6±0.3	60.8±0.5
100	-	-	-	-	52.8±0.3	61.0±0.4	62.8±0.2
200	-	-	-	-	57.0±0.4	64.6±0.3	65.9±0.3
Table 4: Comparison with baseline models.  $\dagger$  indicates the ImageNette dataset, which contains only 10 classes. TESLA [11] uses the downsampled ImageNet-1K dataset. Our results are derived from the full ImageNet-1K, which is more challenging on computation and memory, meanwhile, presenting greater applicability potential in real-world scenarios. The recovery model used in the table is R18.
Squeezed Model	Evaluation Model
DeiT-Tiny	ResNet-18	ResNet-50	ResNet-101
DeiT-Tiny-BN	25.36	24.69	31.15	33.16
ResNet-18	15.41	46.71	55.29	60.81
Table 5: ImageNet-1K Top-1 on Cross-Architecture Generalization. Two recovery/squeezed models are used: DeiT-Tiny-BN and ResNet-18. Four evaluation models: DeiT-Tiny, ResNet-{18, 50, 101}.
ImageNet-1K. As shown in the second group of Table 4, employing the same model architecture of ResNet-18 under IPC 10, our approach improves the performance of TESLA from a baseline of  $7.7\%$  to a significance of  $21.3\%$ . Contrary to TESLA, where the performance deteriorates with larger model architectures, our proposed approach capitalizes on larger architectures, displaying an appreciable proportional performance enhancement. This indicates significant promise in the contemporary era of large-scale models. On IPC 50, 100 and 200, our method obtains consistent boosts on accuracy.
3.5 Cross-Architecture Generalization
It is important to verify the generalization property of our condensed datasets, ensuring its ability to effectively generalize to new architectures that it has not encountered during the synthesis phase. Fig. 4 and Table 5 demonstrate that our condensed dataset exhibits proficient cross-model generalization across ResNet-{18, 50, 101} and ViT-T. The results reveal that our condensed datasets maintain robustness across disparate and larger architectures. However, we observed suboptimal performance of ViT on the condensed dataset, potentially due to the model's inherent need for substantial training data as introduced in [26].
3.6 Synthetic Image Visualization
Fig. 5 provides a visual comparison of selected synthetic images from our condensed dataset and the corresponding images from the MTT condensed dataset. The synthetic images generated by our method manifest a higher degree of clarity on semantics, effectively encapsulating the attributes and contours of the target class. In
contrast, synthetic images from MTT appear considerably blurred, predominantly capturing color information while only encapsulating minimal details about the target class. Consequently,  $\mathrm{SRe}^{2}\mathrm{L}$  produces superior-quality images that not only embed copious semantic information to augment validation accuracy but also demonstrate superior visual performance.
King Penguin
Figure 5: Visualization of MTT [1] and our  $\mathrm{SRe}^2\mathrm{L}$ . The upper two rows are synthetic Tiny-ImageNet and the lower two rows are synthetic ImageNet-1K (the first row is MTT and second is ours).
considerably blurred, predominantly capturing color details about the target class. Consequently,  $\mathrm{SRe}^{2}\mathrm{L}$  is embedded copious semantic information to augment or visual performance.
3.7 Application: Continual Learning
Figure 6: 5-step and 10-step class-incremental learning on Tiny-ImageNet.
Many prior studies [10, 5, 32, 7] have employed condensed datasets for continual learning to assess the quality of the synthetic data. We adhere to the method outlined in DM [10] class-incremental learning implementation, which is based on GDumb [33]. This method sequentially stores prior training data in memory and utilizes both new training data and stored data to learn a model from scratch. To demonstrate the superiority of our method in handling large-scale data, we conduct class incremental learning on Tiny-ImageNet, incorporating an escalating memory budget of 100 images per class, and training with ResNet-18. Fig. 6 shows both 5-step and 10-step class-incremental learning strategies, which partition 200 classes into either 5 or 10 learning steps, accommodating 40 and 20 classes per step respectively. Our results are clearly better than the baselines.
Data condensation or distillation aims to create a compact synthetic dataset that preserves the essential information in the large-scale original dataset, making it easier to work with and reducing training time while achieving the comparable performance to the original dataset. Previous solutions are mainly divided into four categories: Meta-Model Matching optimizes for the transferability of models trained on the condensed data and uses an outer-loop to update the synthetic data when generalized to the original dataset with an inner-loop to train the network, methods include DD [3], KIP [23], RFAD [4], FRePo [5] and LinBa [34]; Gradient Matching performs a one-step distance matching process on the network trained on original dataset and the same network trained on the synthetic data, methods include DC [8], DSA [32], DCC [9] and IDC [7]; Distribution Matching directly matches the distribution of original data and synthetic data with a single-level optimization, methods include DM [10], CAFE [6], HaBa [35], IT-GAN [36], KFS [9]; Trajectory Matching matches the training trajectories of models trained on original and synthetic data in multiple steps, methods include MTT [1] and TESLA [11]. Our proposed decoupling method presents a new perspective for tackling this task, while our BN-matching recovering procedure can also be considered as a special format of Distribution Matching scheme on the synthetic data and global BN statistics distributions.
We have presented a novel three-step process approach for the dataset condensation task, providing a more efficient and effective way to harness the power of large-scale datasets. By employing the sequential steps of squeezing, recovering, and relabeling, this work condenses the large-scale ImageNet-1K while retaining its essential information and performance capabilities. The proposed method outperforms existing state-of-the-art condensation approaches by a significant margin, and has a wide range of applications, from accelerating the generating and training process to enabling the method that can be used in resource-constrained environments. Moreover, the study demonstrates the importance of rethinking conventional methods of data condensation and model training, as new solutions can lead to improvements in both computational efficiency and model performance. As the field of data condensation continues to evolve, the exploration of targeting approaches, such as the one presented in this work, will be crucial for the development of future condensation approaches that are more efficient, robust, and capable of handling vast amounts of data in a sustainable manner.
Limitation and Future Work: At present, a performance disparity persists between the condensed dataset and the original full dataset, indicating that complete substitution of the full data with condensed data is yet to be feasible. Another limitation is the extra storage for soft labels. Moving forward, our research endeavors will concentrate on larger datasets such as the condensation of ImageNet-21K, as well as other data modalities encompassing language and speech.
[39] TorchVision maintainers and contributors. Torchvision image classification reference training scripts. https://github.com/pytorch/vision/tree/main/references/classification, 2016.15
[40] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 17
[41] Angus Galloway, Anna Golubeva, Mahmoud Salem, Mihai Nica, Yani Ioannou, and Graham W Taylor. Bounding generalization error with input compression: An empirical study with infinite-width networks. Transactions on Machine Learning Research, 2022. 18
[42] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019. 18
[43] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (itw), pages 1-5. IEEE, 2015. 18
[44] Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and generalization in deep neural networks, 2019. 18
In the appendix, we provide details omitted in the main text, including:
Section A: Implementation Details.
Section B: Low-Resolution Data.
- Section C: Feature Embedding Distribution.
Section D: Theoretical Analysis.
- Section E: More Visualization of Synthetic Data.
Table 6 enumerates various permutations of ImageNet-1K training set, delineated according to their individual configurations. Tiny-ImageNet [12] incorporates 200 classes derived from ImageNet-1K, with each class comprising 500 images possessing a resolution of  $64 \times 64$ . ImageNette/ImageWoof [37] (alternatively referred to as subsets of ImageNet) include 10 classes from analogous subcategories, with each image having a resolution of  $112 \times 112$ . The MTT [1] framework introduces additional 10-class subsets of ImageNet, encompassing ImageFruit, ImageSquawk, ImageMeow, ImageBlub, and ImageYellow. ImageNet-10/100 [38] samples 10/100 classes from ImageNet while maintaining an image resolution of  $224 \times 224$ . Downsampled ImageNet-1K rescales the entirety of ImageNet data to a resolution of  $64 \times 64$ . In our experiments, we choose two standard datasets of relatively large scale: Tiny-ImageNet and the full ImageNet-1K.
Training Dataset	#Class	#Img per class	Resolution	Method
Tiny-ImageNet [12]	200	500	64×64	MTT [1], FRePo [5], DM [10], SRe2L (Ours)
ImageNette/ImageWoof [37]	10	~1,000	112×112	MTT [1], FRePo [5]
ImageNet-10/100 [38]	10/100	~1,200	224×224	IDC [7]
Downsampled ImageNet-1K [13]	1,000	~1,200	64×64	TESLA [11], DM [10]
Full ImageNet-1K [24]	1,000	~1,200	224×224	SRe2L (Ours)
Table 6: Variants of ImageNet-1K training set with different configurations.
A.2 Squeezing Details
Data Augmentation. Table 2 in the main paper illustrates that the utilization of data augmentation approaches during the squeezing phase contributes to a decrease in the final accuracy of the recovered data. To summarize, the results on Tiny-ImageNet indicate that lengthening the training budget and the application of more data augmentations in the squeezing phase intensify the intricacy involved in data recovery from the compressed model, which is not desired.
Parallel conclusions are inferred from the compressed models for the ImageNet-1K dataset. For our experimental setup, we aimed to extract data from a pre-trained ResNet-50 model with available  $VI$  and  $V2$  weights in the PyTorch model zoo. The results propose that the task of data extraction poses a greater challenge from the ResNet-50 model equipped with  $V2$  weights as compared to the model incorporating  $VI$  weights. This can be attributed to the fact that models utilizing  $VI$  weights are trained employing a rudimentary recipe, whereas models with  $V2$  weights encompass numerous training enhancements, such as more training budget and data augmentations, to achieve cutting-edge performance. It is observed that these additional complexities impede the data recovery process even the pre-trained models are stronger. Therefore, the pre-trained models we employ for the recovery of ImageNet-1K are those integrating  $VI$  weights from the PyTorch model zoo.
Hyper-parameter Setting. We provide hyper-parameter settings for the two datasets in detail.
- Tiny/ImageNet: We train modified ResNet-{18, 50} models on Tiny/ImageNet data with the parameter setting in Table 7a. The well-trained ResNet-{18, 50} models achieve Top-1 accuracy of  $\{59.47\%, 61.17\%\}$  under the 50 epoch training budget.
config	value
optimizer	SGD
base learning rate	0.2
weight decay	1e-4
optimizer momentum	0.9
batch size	256
learning rate schedule	cosine decay
training epoch	100
augmentation	RandomResizedCrop
(a) Tiny-ImageNet squeezing setting.
config	value
optimizer	AdamW
base learning rate	0.001
weight decay	0.01
optimizer momentum	β1, β2 = 0.9, 0.999
batch size	1,024
learning rate schedule	cosine decay
training epoch	300
augmentation	RandomResizedCrop
(b) ImageNet-1K validation setting.
config	value
αBN	1.0
optimizer	Adam
base learning rate	0.1
optimizer momentum	β1, β2=0.5, 0.9
batch size	100
learning rate schedule	cosine decay
recovering iteration	1,000
augmentation	RandomResizedCrop
(c) Tiny-ImageNet recovering setting.
config	value
αBN	0.01
optimizer	Adam
base learning rate	0.25
optimizer momentum	β1, β2=0.5, 0.9
batch size	100
learning rate schedule	cosine decay
recovering iteration	2,000
augmentation	RandomResizedCrop
(d) ImageNet-1K recovering setting.
Table 7: Hyper-parameter settings in three stages.
- ImageNet-1K: We use PyTorch off-the-shelf ResNet-\{18, 50\} with  $V1$  weights and Top-1 accuracy of \{69.76%, 76.13\% \} as squeezed/condensed models. In the original training script [39], ResNet models are trained for 90 epochs with a SGD optimizer, learning rate of 0.1, momentum of 0.9 and weight decay of  $1 \times 10^{-4}$ .
BN-ViT Model Structure. To give a clear illustration of incorporating BN layers into ViT, we provide more details in this subsection. The definitions of vanilla ViT and BN-ViT are presented below to show the structure modification. The vanilla ViT can be formulated as:
$$
\mathbf {z} _ {\ell} ^ {\prime} = \operatorname {M H S A} \left(\operatorname {L N} \left(\mathbf {z} _ {\ell - 1}\right)\right) + \mathbf {z} _ {\ell - 1}
$$
$$
\mathbf {z} _ {\ell} = \operatorname {F F N} \left(\operatorname {L N} \left(\mathbf {z} _ {\ell} ^ {\prime}\right)\right) + \mathbf {z} _ {\ell} ^ {\prime}
$$
where  $\mathbf{z}_{\ell}^{\prime}$  is the intermediate representation before Feed-forward Network (FFN), and  $\mathbf{z}_{\ell}$  is that after FFN and residual connection. FFN contains two linear layers with a GELU non-linearity in between them, i.e.,
$$
\operatorname {F F N} \left(\mathbf {z} _ {\ell} ^ {\prime}\right) = \left(\operatorname {G E L U} \left(\mathbf {z} _ {\ell} ^ {\prime} W _ {\ell} ^ {1} + b _ {\ell} ^ {1}\right)\right) W _ {\ell} ^ {2} + b _ {\ell} ^ {2}
$$
The newly constructed BN-ViT is:
$$
\mathbf {z} _ {\ell} ^ {\prime} = \operatorname {M H S A} \left(\operatorname {B N} \left(\mathbf {z} _ {\ell - 1}\right)\right) + \mathbf {z} _ {\ell - 1}
$$
$$
\mathbf {z} _ {\ell} = \operatorname {F F N} _ {\mathrm {B N}} \left(\operatorname {B N} \left(\mathbf {z} _ {\ell} ^ {\prime}\right)\right) + \mathbf {z} _ {\ell} ^ {\prime}
$$
where we add one additional BN layer in-between two linear layers of FFN, i.e.,
$$
\mathrm {F F N} _ {\mathrm {B N}} \left(\mathbf {z} _ {\ell} ^ {\prime}\right) = \left(\text {G E L U} \left(\mathrm {B N} \left(\mathbf {z} _ {\ell} ^ {\prime} W _ {\ell} ^ {1} + b _ {\ell} ^ {1}\right)\right)\right) W _ {\ell} ^ {2} + b _ {\ell} ^ {2}
$$
We follow the DeiT official training recipe to train a DeiT-Tiny-BN model for 300 epochs with an AdamW optimizer, cosine decayed learning rate of  $5 \times 10^{-4}$ , weight decay of 0.05 and 5 warmup epochs.