SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ablation	Top-1 acc. (%)
RTV	Rl2	Random Crop	Tiny-ImageNet	ImageNet-1K
✓	✓	X	29.87	22.92
✓	X	X	29.92	23.15
X	✓	X	30.11	40.81
X	X	X	30.30	40.37
X	X	✓	37.88	46.71
Table 8: Top-1 validation accuracy under regularization ablation settings. ResNet-18 is used in all three stages with the relabeling temperature  $\tau = 20$ .
A.3 Recovering Details
Regularization Terms. We conduct a large number of ablation experiments under varying regularization term conditions, as illustrated in Table 8. The two image prior regularizers,  $\ell_2$  regularization and total variation (TV), are not anticipated to enhance validation accuracy as our primary focus is on information recovery rather than image smoothness. Consequently, we exclude these two regularization terms from our experiments.
Memory Consumption and Computational Cost. Regarding memory utilization, the memory accommodates a pre-trained model, reconstructed data, and the corresponding computational graph during the data recovery phase. Unlike the MTT approach, which necessitates all model states across all epochs during training to align with the trajectory, our proposed methodology,  $\mathrm{SRe}^2\mathrm{L}$ , merely requires the statistical data from each BN layer, stored within the condensed model, for synthetic image optimization. In terms of total computational overhead, it is directly proportional to the number of recovery iterations. To establish a trade-off between performance and computational time, we enforce a recovery budget of 1k iterations for Tiny-ImageNet and 2k iterations for ImageNet-1K in ablation experiments. Our best accuracy, achieved on condensed data from 4k recovery iterations, is presented in Table 4 in the main paper.
Hyper-parameter Setting. We calculate the total recovery loss  $\ell_{total} = \arg \min_{\mathcal{C}_{\mathrm{syn}},|\mathcal{C}|}\ell (\phi_{\pmb{\theta}_{\mathcal{T}}}(\widetilde{\pmb{x}}_{\mathrm{syn}}),\pmb {y}) +$
$\alpha_{\mathrm{BN}}\mathcal{R}_{\mathrm{BN}}$  and update synthetic data with the parameter setting in Table 7c and Table 7d for TinyImageNet and ImageNet-1K, respectively.
A.4 Relabeling & Validation Details
In this experiment, we utilize an architecture identical to that of a recovery model to provide soft labels as a teacher for synthesized images. We implement a fast knowledge distillation process to isolate the utilization of teacher models in post validation training with a training budget of 300 epochs and a temperature setting of  $\tau = 20$ .
Hyper-parameter Setting. Regarding Tiny-ImageNet, we leverage the condensed data and the retargeted labels to train the validation model over a span of 100 epochs, with all other training parameters adhering to the condensing configurations outlined in Table 7a. In the case of ImageNet-1K, we apply CutMix augmentation with mix probability  $p = 1.0$  and Beta distribution  $\beta = 1.0$  to train the validation model in accordance with the parameter configurations presented in Table 7b.
B Low-Resolution Data
To demonstrate our method's effectiveness on low-resolution datasets, we first conduct additional ablation experiments on ImageNet-1K dataset, including down-sampled resolutions of  $112 \times 112$  and  $64 \times 64$ . In Table 9, the validation accuracy increases as the image resolution size grows, demonstrating that our method is more suitable for handling high-resolution datasets with better effectiveness.
We then conduct experiments on the CIFAR datasets to demonstrate the efficacy of our method when applied to small-scale datasets with fewer classes and a lower image resolution of  $32 \times 32$ . The adapted ResNet-18 is used as a backbone model throughout our  $\mathrm{SRe}^2\mathrm{L}$  's three phases, and the hyper-parameter settings are presented in Table 10. As shown in Table 11, our results on the relatively
Dataset (IPC=50)	ResNet-18	ResNet-50	ResNet-101
IN-1K-64×64	35.27	42.26	44.37
IN-1K-112×112	34.15	42.76	45.25
IN-1K-224×224	46.75	55.62	60.81
large CIFAR-100 dataset are on par with those of leading-edge methods, such as DM, FrePo, MTT, and TESLA, among others. However, on the smaller CIFAR-10 dataset, a clear gap exists. Overall, the additional CIFAR experiments suggest that our approach might also offer significant benefits for the lower-resolution dataset, and we highlight that our approach continues to demonstrate superior computational efficiency and enhanced processing speed when applied to these datasets.
Table 9: Top-1 validation accuracy on distilled and down-sampled ImageNet-1K datasets.
config	value
optimizer	SGD
base learning rate	0.1
momentum	0.9
weight decay	5e-4
batch size	128
learning rate schedule	cosine decay
training epoch	200 (squeeze) / 400 (val)
augmentation	RandomCrop
(a) Squeezing/Validation setting.
config	value
αBN	0.01
optimizer	Adam
base learning rate	0.25
optimizer momentum	β1, β2 = 0.5, 0.9
batch size	100
temperature	30
learning rate schedule	cosine decay
recovery iteration	1,000
(b) Recovery setting.
Table 10: Hyper-parameter settings on CIFAR-100.
IPC	DM [10]	FrePo [5]	MTT [1]	TESLA [11]	SRe2L
10	29.70 ± 0.30	41.30 ± 0.20	40.10 ± 0.40	41.70 ± 0.30	23.48 ± 0.80
50	43.60 ± 0.40	44.30 ± 0.20	47.70 ± 0.20	47.90 ± 0.30	51.35 ± 0.79
100	-	-	-	-	57.06 ± 1.23
200	-	-	-	-	59.64 ± 1.24
Table 11: Top-1 validation accuracy on distilled CIFAR-100 dataset.
C Feature Embedding Distribution
We feed the image data through a pretrained ResNet-18 model, subsequently extracting the feature embedding prior to the classification layer for the purpose of executing t-SNE [40] dimensionality reduction and visualization. Fig. 7a exhibits two distinct feature embedding distributions of synthetic Tiny-ImageNet data, sourced from 3 classes in MTT's and  $\mathsf{SRe}^2\mathsf{L}$  's condensed datasets, respectively. Relative to the distribution present in MTT,  $\mathsf{SRe}^2\mathsf{L}$  's synthetic data from differing classes displays a more dispersed pattern, whilst data from identical classes demonstrates a higher degree of clustering. This suggests that the data synthesized by  $\mathsf{SRe}^2\mathsf{L}$  boasts superior discriminability with respect to feature embedding distribution and can therefore be utilized to train models to attain superior performance. Fig. 7b illustrates feature embedding distributions of  $\mathsf{SRe}^2\mathsf{L}$  's synthetic ImageNet-1K data derived from 8 classes. Our synthetic ImageNet-1K data also exemplifies exceptional clustering and discriminability attributes.
We provide a theoretical analysis of the generalization ability on the condensed dataset. Dataset condensation task generally aims to train on the condensed data meanwhile achieving good performance on the original val data. Given the significance of estimating the generalization error (GE) of deep neural networks as a method for evaluating their ability to generalize, we adopt this approach for
Figure 7: Feature embedding distribution on synthetic data of Tiny-ImageNet and ImageNet-1K. ResNet-18 is used as the feature embedding extractor.
assessing the generalization capability of our condensed dataset for analyzing the generalization/err bounds between models trained on original data and condensed data.
Specifically, we employ the Mutual Information (MI) between the original/condensed input and the final layer representations to carry out this analysis, using the same network architecture limit to bound MI, in line with the methodology outlined in [41]. To elaborate further:
The MI between two variables  $X$  and  $D$  is:
$$
I (X; D) \equiv \sum_ {x, d} p (x, d) \log \frac {p (x , d)}{p (x) p (d)} = \mathbb {E} _ {p (x, d)} \left[ \log \frac {p (d \mid x)}{p (d)} \right]
$$
where  $X$  is the input sample,  $D$  is the input representation, i.e., model's output. The *leave one out* upper bound (UB) [42] can be utilized to conservatively bound MI:
$$
I (X; D) \leq \mathbb {E} \left[ \frac {1}{N} \sum_ {i = 1} ^ {N} \log \frac {p \left(d _ {i} \mid x _ {i}\right)}{\frac {1}{N - 1} \sum_ {j \neq i} p \left(d _ {i} \mid x _ {j}\right)} \right] = I _ {\mathrm {U B}}
$$
Following the information theory fundamentals, by applying the conventional Probably Approximately Correct (PAC) of GE bound, we can obtain the bound on GE as:
$$
\mathrm {G E} <   \sqrt {\frac {\log (| \mathcal {H} |) + \log (1 / \delta)}{2 N _ {\mathrm {t r n}}}}
$$
where  $|\mathcal{H}|$  is the hypothesis-class cardinality and  $N_{\mathrm{trn}}$  is the number of training examples. For the synthetic data,  $N_{\mathrm{trn}} = |\mathcal{C}_{\mathrm{syn}}|$ , while for the full data,  $N_{\mathrm{trn}} = N_{\mathrm{ori}}$ . The confidence parameter, denoted by  $\delta$  and ranging between 0 and 1, specifies the likelihood that the bound remains consistent with respect to the chosen  $N_{\mathrm{trn}}$  training samples.
According to the property of deep neural networks [41], the cardinality of the hypothesis space reduces to  $|\mathcal{H}| \approx 2^{|T|}$  where  $|\mathcal{H}|$  is the number of class-homogeneous clusters that the backbone network distinguishes. An estimate for the number of clusters can then be obtained by  $|T| \approx 2^{H(X)} / 2^{H(X|Z)} = 2^{I(X;Z)}$ .
The ability of Input Compression Bound (ICB) [43, 44] is to predict changes in GE under different dataset interventions, it then can be formulated as:
$$
\mathrm {G E} _ {\mathrm {I C B}} <   \sqrt {\frac {2 ^ {I (X ; D)} + \log (1 / \delta)}{2 N _ {\mathrm {t r n}}}}
$$
Thus, we can have the generalization error bound for the condensed data as:
$$
\mathrm {G E} _ {\mathrm {I C B}} ^ {\mathrm {s y n}} <   \sqrt {\frac {2 ^ {I (X ; D)} + \log (1 / \delta_ {\mathrm {s y n}})}{2 | \mathcal {C} _ {\mathrm {s y n}} |}}
$$
where the generalization error bound for full dataset is  $\mathrm{GE}_{\mathrm{ICB}}^{\mathrm{full}} < \sqrt{\frac{2^{I(X;D)} + \log(1 / \delta_{\mathrm{full}})}{2N_{\mathrm{trn}}}}$ .
E More Visualization of Synthetic Data
We provide more visualization comparisons on synthetic Tiny-ImageNet between MTT and  $\mathrm{SRe}^2\mathrm{L}$  in Fig. 8. Additionally, we visualize synthetic samples pertaining to ImageNet-1K in Fig. 9 and Fig. 10 for a more comprehensive understanding. It can be observed that our synthetic data has the stronger semantic information than MTT with more object textures, shapes and details, which demonstrates the superior quality of our synthesized data.
$\mathsf{SRe}^2\mathsf{L}$
Abacus
Academic Gown
African Elephant
$\mathsf{SRe}^2\mathsf{L}$
American Lobster
Alp
MTT
$\mathsf{SRe}^2\mathsf{L}$
Camel
Apron
Bannister
$\mathsf{SRe}^2\mathsf{L}$
Barbershop
Figure 8: Synthetic data visualization on Tiny-ImageNet from MTT [1] and SRe $^2$ L.
Figure 9: Synthetic data visualization on ImageNet-1K from SRe $^2$ L.
Figure 10: Synthetic data visualization on ImageNet-1K from SRe $^2$ L.