SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In terms of SAETs, which has been studied in Sec. 3.3, we set the number  $K$  according to the resolution of the dataset (e.g. we set  $K$  to 256 for ImageNet subsets, and decrease it to 128 and 64 for TinyImageNet and CIFAR100, respectively). When the budget is enlarged, we expand the dimension of SAETs  $D$  with its head  $H$  increased together, while guaranteeing that their divisor  $D / H \leq K$ , so that each head  $E^h$  is a complete (or overcomplete) dictionary. We always build the FReeNet with shallow blocks, which provides sufficient nonlinear capabilities while making it lightweight enough (e.g. we set 1 block on CIFAR10 with IPC 1 storage space which is excessively stringent, and set 2 blocks for all experiments on TinyImageNet and ImageNet subsets).
(a) Patch size  $4 \times 4$
(b) Patch size  $8 \times 8$
(c) Patch size  $16\times 16$
(d) TM [16] baseline (patch size  $32 \times 32$ )
Figure I: Synthetic data samples under different configurations of patch size. A small patch size can capture intricate details such as the black eye patch of the hamster, truck tires, and rabbit ears.
The number of non-zero elements  $k$  is always set to a value that ensures the matrix density  $k / JK < 5\%$ , then the number of images  $N$  can be computed according to Eq. (7). The setting of the penalty weight  $\lambda$  depends on the number of images, heads, and patches, satisfying  $\lambda \approx 0.064 / NHJ$ .
Compute Resources. Our experiments were run on a mixture of RTX 3090, RTX A6000, and A800 GPUs. The GPU memory consumption is mainly dependent on that of the matching objective and is slightly higher than its baseline, due to the larger data amount. For instance, our experiments based on gradient matching [13] and distribution matching [17] need one 24GB 3090 GPU. In terms of trajectory matching [16], the VRAM usage will be higher, ranging from one 24GB 3090 GPU to six 80GB A800 GPUs.
II Additional Results and Analyses
Patch Size. The patch size plays an important role in determining the granularity of our distillation modeling. Let each patch  $\tilde{x}$  be of  $P\times P$  resolution and contains  $C$  channels, then its dimension  $L$  will be  $P^2 C$ . Tab. I presents the results on CIFAR100 with IPC 1 storage space. We observe that setting a relatively small patch size can lead to better performance, as it reduces redundancy within synthetic images and enables finer-grained synthesis. Therefore, we set a default patch size of  $4\times 4$  for the  $32\times 32$  resolution datasets (CIFAR10 and CIFAR100) and increase it appropriately when the resolution of the dataset increases. For instance, we adopt a default patch size of  $8\times 8$  for TinyImageNet  $(64\times 64)$ .
Patch size	N/c	ConvNet	MLP	ResNet18	ViT
4 × 4	11	40.0±0.4	15.5±0.2	29.9±0.3	20.7±0.5
8 × 8	10	39.3±0.3	14.6±0.3	29.4±0.5	20.2±0.2
16 × 16	7	37.3±0.3	12.9±0.3	27.1±0.8	16.5±0.2
Table I: Patch size. Small patch sizes allow finer-grained modeling.
Furthermore, we provide a visualization of synthetic images with varying patch sizes, as depicted in Fig. I. The utilization of small patches can facilitate the expression of finer-grained features, leading to synthetic images that contain more classifiable information. For instance, when a  $4 \times 4$  patch is employed, it can capture intricate details such as the black eye patch of the hamster, in addition to more noticeable features like truck tires and rabbit ears. Also, sparse parameterization allows for efficient storage of background information with limited significance by allocating fewer non-zero elements in SCMs to represent it.
Figure II: Left: Effects of different penalty weight  $\lambda$  on test accuracy. TM [16] denotes the baseline method of the trajectory matching objective with naive image-independent parameterization. A moderate weight can trade off sparsity and accuracy. Middle: Comparison of different feature sparsification strategies. Global sparsification performs better in various settings, especially when the storage budget is stringent. Right: Test accuracy and distilling time.
full
t
p-48
top-24
top-12
Figure III: Visualization of our SCMs with varying values of  $k$  (patch corresponds to zero column vector in its SCMs is marked grey). SPEED learns to efficiently allocate more storage for foreground objects that contain more classifiable information. Therefore, a small  $k$  still allows for salient features of objects to be synthesized well.
full
top-48
top-24
top-12
Sparsity Penalty Weight  $\lambda$ . In SPEED, the sparsity penalty term constrains the overall scale of elements in SCMs. Ablation studies of the penalty weight  $\lambda$  are conducted on CIFAR100 with IPC 1 storage space, as shown in Fig. II (left). For these experiments, we fix the number of sparsified features to 48. If the penalty weight is too small, our SCM tends to comprise a higher number of elements with larger absolute values, which necessitates more features to be maintained during feature sparsification, thereby guaranteeing the test performance. It consequently implies a larger storage burden. On the other hand, setting the penalty weight too high may limit the representation capability required for synthesizing informative images. Therefore, we prefer the moderate penalty weight, which allows for sufficient combinations of required epitomic tokens while efficiently eliminating inefficient ones without the risk of losing important information.
Local Feature Sparsification. To assess the efficacy of our feature sparsification with global semantic preservation on SCMs, we introduce the contrast, local feature sparsification, that attempts to store an equal number of non-zero elements for each patch, i.e., each column of SCM has the same number of non-zero elements. Consequently, local feature sparsification constructs all patches using the same number of parameters (keep  $k / J$  elements for each column of SCM), while the feature sparsification with global semantic preservation does not (keep  $k$  elements for whole SCM). We evaluate the performance of these two methods using different values of  $k$ , while maintaining the other hyper-parameters constant.
As illustrated in Fig. II (middle), we conduct tests on CIFAR100 with  $k$  set to  $\{64, 128, 192, 256\}$ , where the SCM size is  $64 \times 64$ . Accordingly, the local feature sparsification strategy preserves  $\{1, 2, 3, 4\}$  features for each synthetic patch (e.g. when  $k$  is set at 128,  $J = 64$ , then each patch is represented by the 2 most prominent SAETs). In all cases, the global feature sparsification outperforms the local one. Notably, the advantages of the global approach are more apparent when the storage
(a) no convolutional layers
(b) 1 layer, kernel size  $= 3 \times 3$
(c) 1 layer, kernel size  $= 5 \times 5$
(d) 2 layers, kernel size  $= 3 \times 3$
(e) 2 layers, kernel size  $= 5 \times 5$
Figure IV: Visualizations (border terrier class in ImageNette) regarding the incorporation of convolutional layers after the rearrangement of patches. The presence of the chessboard artifact gradually diminishes with increases in the number of convolutional layers and the size of convolutional kernels.
budget is stringent, i.e., for small  $k$ . This indicates that our global feature sparsification technique produces synthetic images with more classifiable information under the same storage budget, making it more efficient for dataset distillation. Similar to the conclusion drawn in Sec. 3.3, we observe that as  $k$  increases to a certain extent, the accuracy reaches an upper bound and then fluctuates.
Test Accuracy vs. Distilling Time. We observe the test accuracy and distilling time on CIFAR10 with IPC 1 storage space, and compare with the previous methods [16, 21, 22], as illustrated in Fig. II (right). Overall, SPEED consistently outperforms previous methods when compared at similar distilling time points.
Visualization of SCMs. We visualize our SCMs with varying values of  $k$  in Fig. III, where the patch corresponds to zero column vector in its SCMs is marked grey. As  $k$  decreases, we observe a higher preference for column vectors that correspond to background patches to be zero vectors in SCMs. This indicates that SPEED automatically learns to allocate more storage budget to foreground objects that contain more useful clues for classification. Even when  $k$  is set as low as 12, the salient foreground object is still well synthesized. It should be noted that the zeroed-out patches do not occupy any storage budget since the SCMs are saved in the COO compressed format.
Incorporation of Additional Convolutional Layers. We find a chessboard (blocky) artifact on synthetic images of high-resolution datasets. To investigate its influence on test accuracy, we perform multiple experiments on ImageNette with IPC 1 storage space, adding 1 and 2 convolutional layers with kernel sizes 3 and 5, as shown in Tab. II. As evident from the results, the incorporation of additional convolutional layers in our experiments does not yield a significant
improvement in downstream training. However, it does provide slight relief from the chessboard artifact, as depicted in Fig. IV. The impact of the chessboard artifact on downstream training and the exploration of parameter-efficient methods to eliminate these artifacts warrant further investigation.
Kernel size	3×3	5×5
1 layer	66.3±1.8	66.4±1.3
2 layers	65.9±1.3	64.0±0.5
None	66.9±0.7
Table II: Enlargement of the image resolution on ImageNette.
Spar.	Block-shared	Evaluation
SAET	SCM	#Param	ConvNet	MLP	ResNet18	ViT
-	-	-	27M	41.2±0.4	15.6±0.2	29.8±0.5	21.1±0.3
-	✓	-	27M	41.3±0.2	15.6±0.3	29.4±0.4	20.6±0.1
-	✓	✓	14M	41.2±0.4	15.3±0.1	29.6±0.6	20.9±0.4
✓	-	-	549K	40.0±0.6	15.6±0.1	30.3±0.5	20.7±0.4
✓	✓	-	543K	40.3±0.7	15.6±0.1	29.7±0.7	21.0±0.4
✓	✓	✓	305K	40.0±0.4	15.5±0.2	29.9±0.3	20.7±0.5
Table III: Storage efficiency study. The checkmark  $\checkmark$  in the Spar. column indicates the use of sparsified SCMs, while the absence of a mark indicates the use of full-element SCMs. The checkmark  $\checkmark$  in the Block-shared SAET/SCM column indicates sharing SAETs/SCMs across different scales of blocks, while the absence of a mark indicates using block-specific SAETs/SCMs.
IPC
Parameters / Class	1
3,072	10
30,720	50
153,600
IDC [19]	-	40 (44.8±0.2)	-
HaBa [21]	5 (33.4±0.4)	45 (40.2±0.2)	245 (47.0±0.2)
RTP [22]	16 (34.0±0.4)	232 (42.9±0.7)	-
SPEED	11 (40.0±0.4)	62 (45.9±0.3)	100 (49.1±0.2)
Storage Efficiency Study. To investigate the storage efficiency of recurrent blocks that involves utilizing block-shared SAETs and SCMs, we attempt to allocate specific SAETs and SCMs for each block, along with setting a uniform value for  $k$  in feature sparsification, as studied in Tab. III. Our results indicate that applying recurrent blocks leads to the highest efficiency. While the block-specific strategy shows a marginal improvement over the block-shared strategy with or without feature sparsification, it lacks a significant performance advantage. However, recurrent blocks using block-shared SAETs and SCMs come with significantly lower storage, saving about  $44\%$  parameters while achieving competitive results. Furthermore, we observe that feature sparsification can enhance both the overall cross-architecture generalization performance and storage efficiency.
Number of Synthetic Images. The number of our synthetic images on ImageNet subsets is summarized in Tab. IV. Our method synthesizes 15 images under the IPC 1 budget, and remarkably, it achieves performance that is competitive with other methods operating under the IPC 10 budget, i.e., 10 synthetic images, while utilizing only  $10\%$  of the parameters. For instance, on ImageNette, our method achieves an
Table V: Comparisons on quantity  $(N / c)$  and quality (test accuracy) of synthetic images.
IPC
Parameters / Class	1
49,152	10
491,520
# Synthetic images	15	111
Table IV: Number of synthetic images on ImageNet subsets.
impressive accuracy of  $66.9\%$  with IPC 1, surpassing the previous state-of-the-art [37] result of  $66.5\%$  achieved with IPC 10. These findings demonstrate the efficiency of our method and the high quality of the synthetic images it produces.
To further prove the above claim, we conclude the number of synthetic images on CIFAR100, compared with other parameterization work [19, 21, 22], as shown in Tab. V. As evident, while the number of our synthetic images is not the highest among all approaches, our outstanding performance clearly showcases the high quality of the synthetic images. This further emphasizes that our approach enhances performance by improving both the quality and quantity of the synthetic images. Moreover, it demonstrates the highly efficient reduction of spatial redundancy achieved by our method.
III Additional Visualizations
We include additional visualizations here. Fig. V-VII show the synthetic datasets for CIFAR10 and CIFAR100. Higher-resolution datasets are visualized in Fig. VIII-XV. We find that visualizations of lower-resolution datasets are more human-friendly since they contain less spatial redundancy. For high-resolution datasets, storage budgets are primarily allocated to salient foreground features, resulting in a significant reduction of useless patches. Nevertheless, the main texture and rough shape can still be recognized.
Figure V: Synthetic images on CIFAR10.
Figure VI: Synthetic images on CIFAR100. (Classes 1-50)
Figure VII: Synthetic images on CIFAR100. (Classes 51-100)
Figure VIII: Synthetic images on TinyImageNet. (Classes 1-100)
Figure IX: Synthetic images on TinyImageNet. (Classes 101-200)
Figure X: Synthetic images on ImageNette.
Figure XI: Synthetic images on ImageWoof.
Figure XII: Synthetic images on ImageFruit.
Figure XIII: Synthetic images on ImageMeow.
Figure XIV: Synthetic images on ImageSquawk.
Figure XV: Synthetic images on ImageYellow.