Sparse Parameterization for Epitomic Dataset Distillation
Authors: Xing Wei
Anjia Cao
Funing Yang
Zhiheng Ma
Venue: NeurIPS 2023 poster (2023)
Keywords: Dataset Distillation
Dataset Condensation
Sparse Coding
Dictionary Learning

Abstract: The success of deep learning relies heavily on large and diverse datasets, but the storage, preprocessing, and training of such data present significant challenges. To address these challenges, dataset distillation techniques have been proposed to obtain smaller synthetic datasets that capture the essential information of the originals. In this paper, we introduce a Sparse Parameterization for Epitomic datasEt Distillation (SPEED) framework, which leverages the concept of dictionary learning and sparse coding to distill epitomes that represent pivotal information of the dataset. SPEED prioritizes proper parameterization of the synthetic dataset and introduces techniques to capture spatial redundancy within and between synthetic images. We propose Spatial-Agnostic Epitomic Tokens (SAETs) and Sparse Coding Matrices (SCMs) to efficiently represent and select significant features. Additionally, we build a Feature-Recurrent Network (FReeNet) to generate hierarchical features with high compression and storage efficiency. Experimental results demonstrate the superiority of SPEED in handling high-resolution datasets, achieving state-of-the-art performance on multiple benchmarks and downstream applications. Our framework is compatible with a variety of dataset matching approaches, generally enhancing their performance. This work highlights the importance of proper parameterization in epitomic dataset distillation and opens avenues for efficient representation learning. Source code is available at https://github.com/MIV-XJTU/SPEED.

Sparse Parameterization for Epitomic Dataset Distillation
Xing Wei!
Anjia Cao' Funing Yang' Zhiheng Ma²*
'School of Software Engineering, Xi' an Jiaotong University Sheenzhen Institute of Advanced Technology, Chinese Academy of Sciences weixingsmail. w...l.xtueeee edu. en zh. wwwsiat.t........ ac. {caoanjia7, moolink) @stu. estteesttusa. zjtu. edu. cn
Abstract
The success of deep learning relies heavily on large and diverse datasets, but the storage, preprocessing, and training of such data present significant challenges. To address these challenges, dataset distillation techniques have been proposed to ob- tain smaller synthetic datasets that capture the essential information of the originals. In this paper, we introduce a Sparse Parameterization for Epitomic datasEt Distilla- tion (SPEED) framework, which leverages the concept of dictionary learning and sparse coding to distill epitomes that represent pivotal information of the dataset. SPEED prioritizes proper parameterization of the synthetic dataset and introduces techniques to capture spatial redundancy within and between synthetic images. We propose Spatial-Agnostic Epitomic Tokens (SAETs) and Sparse Coding Matrices (SCMs) to efficiently represent and select significant features. Additionally, we build a Recurrent Network (FReeNet) to generate hierarchical features with high compression and storage efficiency. Experimental results demonstrate the su- periority of SPEED in handling high-resolution datasets, achieving state-of-the-an performance on multiple benchmarks and downstream applications. Our framework is compatible with variety of dataset matching approaches, generally enhancing their performance. This work highlights the importance of proper parameteriza- tion in epitomic dataset distillation and opens avenues for efficient representation learning. Source code is available at https: //g/ thub
Introduction
Deep learning has achieved remarkable success across diverse domains, thanks its ability to extract insightful representations from large and diverse datasets [1-5]. Nevertheless, the storage, preprocessing, and training of these massive datasets introduce significant challenges that strain storage and computational resources. In response to these challenges, dataset distillation techniques have arisen means distill information of the original dataset. The central concept of dataset distillation the extraction of an epitomic representation, capturing the core characteristics and patterns of the original dataset while minimizing storage demands. By doing so, deep learning models trained on the distilled dataset can attain performance levels similar those trained on the original dataset but with significantly reduced storage and training costs. This approach opens new horizons for applications requiring cost-effective storage solutions and expedited training times, such as neural architecture search [6-8] and continual learning [9-11].
compact synthetic dataset that encapsulates the pivotal
The success of dataset distillation heavily relies essential factors: proper parameterization of the synthetic dataset and an effective design of the matching objective to align it with the original
*Zhiheng Ma the corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sparse Coding Matrices (SCMS)
Festure-feecrrent Network (FReeNet)
Synthetic Patches
Concatt & Linear
Add Norm
Add & Norm
Dataset Matching
0
133
MLP
Linear
Matching Network
Gradient Feature
0.7
Parameter
Synthetic Data
0-2
xR
Recurrent Input
H heads
Spetial-Agnosiic Epitomic Tokens (SAETs)
Real Data
Figure 1: SPEED Overview. We take -agtial- epitomic tokens as the shared dictionary of the dataset and perform multi-head sparse combinations to synthesize instance- speccefcc features. Subse- quently, we utilize feature-recurrent blocks to generate hierarchical representations for non-linear synthesis of image patches, while reusing the sparse features. this way, we sparsely parameterize the dataset, alleviating the storage burden and producing highly representative synthetic images.
dataset. While many dataset distillation methods have primarily focused on optimizing the matching objective [12-18], the critical aspect of synthetic dataset parameteriiztion has often been overlooked. Typically, these methods employ a naive image-inan cungg--iddeeedeett parameterization approach, where each learnable parameter basis (synthetic image) optimized independently. Although some studies have recognized the inefficiency of mnggeinndependent parameterization and explored the mutual coherence and relationship between different synthetic images improve compression efficiency [19-22], none of the previous methods have fully considered the spatial redundancy that exists within individual images and between different images.
In this paper, we present an efficient parameterization framework, named Sparse Parameterization for Epitomic datasEt Distillation (SPEED), which addresses the aforementioned limitations. SPEED leverages principles from representation learning paradigms, including convolutional neural network (CNN) [23, I], vision transformer (ViT) [3], dictionary learning [24-30], and sparse coding [31-36].
SPEED introduces Spatial-Agnostee Epitomic Tokens (SAETs) that are shared among all synthetic image patches, and employs the Sparse Coding Matrices (SCMs) select the most significant tokens. Subsequently, these selected tokens are assembled sequentially to form higher-level representations that facilitate the reconstruction of synthetic patches via non-linear mapping. To further minimize storage requirements, we propose a Feature-Recd Recturen Network (FReeNet) that utilizes recurrent blocks generate hierarchical features, with SAETs and SCMs shared by all blocks, while leveraging a multi-head mechanism to enhance feature diversity.
In comparison to previous methods, our approach demonstrates significant advantages in handling high-resolution real-world datasets with substantial spatial redundancy. Notably, achieves out- standing performance ImageNet subsets, surpassing most of the previous state-of-the of-the-art meth- ods [21, 37] by achieving an average improvement of 11.2% with image per class storage space. Additionally, our sparse parameterization approach exhibits superior performance architectures, outperforming previous state-of-the-a approaches [19, 1 22], and even surpassing own full-element baseline, highlighting the potential of sparse representation for storage efficiency and improved generalization abilities.
unseen network
Our method demonstrates competitive results across three standard dataset distillation benchmarks, such as surpassing the previous of-te-of-the-art [22] on CIFARIO0 100 by 6.0% and on TinyImageNet by 10.9% when using image per class storage space. It also exhibits strong performance on downstream applications, such as continual learning. Furthermore, our framework is compatible with multiple existing matching objectives [13, 16, 17]. generally enhancing their performance through the use of our sparse parameterization strategy.
In summary, our work highlights the importance of proper parameterization in epitomic dataset distillation and introduces the SPEED framework as a solution. We showcase its superiority in handling high-resolution datasets, achieving exceptional performance benchmarks and downstream

---

applications. Our framework not only enhances storage efficiency but also improves generalization capabilities, opening new avenues for efficient representation learning deep learning upplications.
2 Method
The purpose of dataset distillation [12] is to learn synthetic dataset S = smaller in size than the original dataset T {(X,, y4 1 i.e., .N < M. while minimizing the loss of information. Formally, previous methods optimize the synthetic dataset by minimizing various matching objectives, all of which can be expressed using the following formulation:
that is much
8 arg min Eg-e ((((TT. 0), ((S.
where θ represents the distribution used for initializing the network parameters, 0 parameterize the training network, D(;;) .) is the dataset matching metric, so()) maps the dataset other informative spaces (e.g. gradient [13], feature [17, 18], and parameter spaces [16]).
However, rather than exclusively focusing on matching objectives, this paper introduces universal parameterization method for synthetic datasets that can be seamlessly integrated with most existing matching objectives. The naive parameterization method, which optimizes each synthetic image X independently [12-14, 16-18], fails to leverage shared information between images, resulting in unnecessary redundancy. the following sections, we will present our parameterization framework, which decomposes the synthetic dataset into Spatial-Agnostic Epitomic Tokens, Sparse Coding Matrices, and a Feature- Network. This framework significantly mitigates redundancy within and between images, irrespective of the spatial locations of features.
2.1 Spatial-Agnostic Recurrent Parameterization
Taking inspiration from the Vision Transformer (ViT) [3], we propose a more efficient parame- terization approach applied at the patch level, utilizing cascade non-linear combinations to reduce complex and fine-level redundancy. To further minimize the storage footprint, we share Sparse Coding Matrices (SCMs) and Spatial-Agnostee Epitomic Tokens (SAETs) across all recurrent blocks. The main approach can be formulated as follows:
X,
T (E)
(2)
Here = [= ..... 1 E LRL and a.e E R represents the j-th patch of X,, which is flatten into vector. L equals to the product of the patch height, width, and number of channels, and J is the total patch number. Similar to VIT. we divide the synthetic image into non-overlapping rectangular patches. E- = e2,, e2, eK] E DxX is the Spatial-Agnostic Epitomic Tokens (SAETs) shared by all synthetic patches, where D is the feature dimension, and K is the total number of tokens.
A [a aa ...., ( E Kx is the Sparse Coding Matrix (SCM) for the i-th synthetic image, where ()) E ERK is the specific coding vector for patch ((()) A; will be further sparsified and saved in a storage-efficient format. (.(() non-linear recurrent transformer-style network that maps SAETs and the SCM to a synthetic image. Its learnable parameters are denoted as Φ. This network is referred to as the Recuurent Network (FReeNet), which is described detail below.
(1)
Insture-Recurrent Network. In the Feature- Recuurent Network (FReeNet), each recurrent block shares the same SAETs and SCMs for the sake of parameter efficiency. However, this shared approach can lead to a lack of diversity in the resulting representations, as single pool of SAETs must model multi-scale features. Drawing inspiration from the multi-head mechanism introduced in the transformer architecture, we introduce the concept of "multi-head" SAETs, aiming strike balance between storage efficiency and feature diversity. Initially, the original SAETs are split along the feature dimension to create multiple SAET pools, denoted as { { H
1k=1 where Eh E R x Additionally, each pool is assigned an independent SCM, denoted as {4 [H Jb=1, where A E RRKx The coding matrix A will undergo further sparsification to select the most significant tokens. We refer to the mechanism that combines the multi-head SAETs and SCMs Multi-Hess Sparse Coding (MHSC), which can be formulated as follows:
MHSC, (L (A))))) W,[E' A, AP ..... E A"A + br,
(3)
3

---

where W, E DxD is a linear projection and b, is the bias, both of which are specific to each recurrent block and not shared across blocks. Using MHSC as the central component, we can construct the FReeNet in a recurrent manner, with SAETs and SCMs shared across different scales:
Z (MHSS, ((E")= (A + Zr-1); r= 1,2, ..., R. r= 1.2, R
Z, = LNL + Z,);
X; = W +b,
where R is the total number of recurrent blocks, and Z E DD is the output of the last block. Zo initialized with a zero matrix. W LxD and make up the final linear projection layer, and X; RIX the output synthetic image, which then rearranged into its original shape. MLP stands for the multi-layer perceptron. In our implementation, we set the MLP to have one hidden layer with the same dimension as the imput, and incorporate layer normalization (LN) [38] and residual connection [2] in each block. It is worth noting that the parameters other than the SAETs and SCMs are not shared between different blocks to ensure that each block processes different scale of features. Despite the use of shared SCMs across different blocks, our analysis demonstrates that SCMs still occupy the majority of the stored parameters. Therefore, we introduce a sparsification method to further enhance the storage efficiency of SCMs, motivated by the theories and techniques of sparse coding [31, 32]. In the subsequent section, we will reuse X; ((E" )k= { to refer to the multi-head implementation of the FReeNet, without any ambiguity.
2.2 Training Objective and Feature Sparsification
Since the to norm is not differentiable and difficult to optimize [39-42], we instead adopt the & norm as the sparsity penalty function. By promoting sparsity in solutions, can effectively remove redundant features [40, 43-45]. Our optimization objective can be expressed as follows:
S ( J (a)))))) N
N
H
(5)
arg min (EN)AL ( Temm AV
Ea_e (( 0))]
+
Σ
|14
E=
A=1
where || ||1 is the E, norm of a matrix, À controls the amount of regularization. Using this approach, we decompose synthetic images into a multi-head SAET (Eh) 1 and network parameters ó that
Th=1
are shared by all synthetic images, and a multi-head SCM {AA [ Jhm1 for each synthetic image.
Feature Sparsification with Global Semantic Preservation. Sparse codes allow for the ranking of features [33, 46), with higher coefficients indicating more important features. Therefore, we select the most influential sub-parameters of SCM to achieve the desired sparsity || S k. This process reduces storage inefficiency while preserving the global semantics of synthetic images. Moreover, all existing matching objectives [13, 16-18] optimize synthetic images using a specific training network, but these synthetic images are then used to train various agnostic network architectures. By pruning unnecessary features of synthetic images, we can further enhance their generalization ability on unseen network architectures. We experimentally validate this claim in Sec. 3.3. Specifically, given learedd SCM A E R xxx we assign a binary mask B E IR
xJ to it:
Bii f1. 0.
A[i, E topk(abss(1) otherwise
Á-BOA,
(6)
where topk(-) obtains the largest & clements, and abs(-) takes the absolute value of each element of the input matrix. We operate Hadamard product on the two to preserve the top-k efficient elements of the SCM, i.e., select the most critical epitomic features. By operating on each learned SCM, we Jk- 1 which can be directly applied to synthesize images.
receive { H
For the compressed storage of SCMs, we adopt the widely-used coordinate (COO) format, utilizing the uint8 data type to store the row and column coordinates of non-zero elements as the size of the sparse matrix is always smaller than or equal to 256 x 256 in our implementation. Consequently, the storage needed for each non-zero element is 1.5 times that of single float32 tensor. In this way, the storage complexity SCMs can be greatly reduced from O(NHKJ) to O(N where k < K.J. The algorithm of our approach is summarized in Alg. 1.

---

Algorithm Sparse Parameterization for Epitomic Dataset Distillation (SPEED).
Input: T: original dataset; N: total number of synthetic images; H/: number of heads; A: expected number of elements of SCM: SPARSIFY feature sparsification.
1: Randomly initialize SAETs (E* {E") k== SCMs {(Ar) == ) and parameters o of FReeNet
2: for each distillation step... do
3: Get a random initialized training backbone for the matching strategy
4: Construct the synthetic dataset: S { [[E)))) ) {A, Hk=1), k==1.,9) y.)
5: Inner optimization of the training backbone with respect to the matching strategy (if necessary)
6: Compute the objective functton combined matching loss with sparsity penalty Eq. (5)
7: Optimize { EE" 7
1h=1: {{At) H 1==1)=11 Φ with respect to the objective function
8: end for
9:
for i 1 to N do
10: for h to do Select most significant features: A + SPARSIFY (A,, according Eq. (6)
12:
Convert A to compressed storage format end for
13:
end for
Output and Save: (E* Jh ((A)) H 1 and db.
Storage Analysis. Our storage budget is constrained by the upper bound determined by the storage requirements of the original synthetic images. Specifically, for given budget of images per class (IPC) [12, 13] with total classes, the maximum budget limited en I.J, where I.J the storage requirement for a single synthetic image. Therefore, the following inequality must be satisfied:
SAETs: HE
FReeNet:
DK + 15NHK R(3D² + 7D) L(D S CnLJ
SCMs: (1A)) [h= IN
Budget: IPC
1h=1
Noticing that SAETs and FReeNet are shared by all synthetic images, and 1.5Hk & LJ, we can synthesize much larger amount of synthetic images than the original one. For instance, considering the IPC storage budget on CIFARİ00, i.e., e = 100 and n 1, we set = 96, = 64, k 48, R 2, L 48, and J 64, resulting in more informative synthetic dataset with size of N 1100 (11 final images for each class).
Experiments
In this section, we first evaluate our method and compare with previous work. Then, we conduct generalization experiments and perform ablation studies. More results and detailed values for hyper- parameters will be extensively discussed in the appendix. To quantify the performance and guarantee SestanceNoorm-Rell
the fairness of the comparison, we use the default with 128 channels as our training backbone, consistent with previous methods. We adopt trajectory matching [16] as our default matching objective.
ConvNet
3.1 Comparisons
Standard Benchmarks. We first conduct experiments on three standard dataset distillation bench- mark datasets: CIFARIO [48] and CIFAR 100 [48] at resolution of 32 x 32, and TinylmageNet [49] at a resolution of 64 64. To adhere to the standard protocol, we employ a 3-layer ConvNet and a layer ConyNet for training and evaluating CIFAR and TinylmageNet, respectively. As illustrated Tab. 1, SPEED achieves highly competitive results on all three datasets. Remarkably, we achieve test accuracy of 40.0% and 26.9% CIFAR 100 and TinylmageNet, respectively, using 1 image per class (IPC) storage space, representing improvements of 6.0% and 10.9% over the previous state-of-the-art [22].
ImageNet Subsets. We evaluate the performance of SPEED on the high-resolution ImageNet [50] subsets and achieve new state-of-the-an results, shown in Tab. 2. Consistent with [16, 21], split
5

---

Dataset IPC
CIFARIO
CIFARI00
TinylmageNet
10
50
10
50
10
50
Random
14.4122.
26.0 1.2
43.411.0
4.240.33
14.610.5
30.0-04 0.4
1410.1
5.0:1.22
15.0:10.4
Coreset
Herding
215++1.3
31.6-0.7
40.4+0.6
8.440.3
17.340.3
33.7-0.5
28+022
6.340.2
16.7+0.3
K-Center
23.3+00.9
36.4=0.6
48.720.3
8,640.3
20.7:0.2
33.6=0.4
2.7+0.2
7.440.4
16.7=0.4
Forgetting
13.51:22
23.3=1.0
23.3::1.1
4.5
9.8:0022
1.640.1
5.140.2
15.0:10.3
DC [13]
28.340.5
0.5
53.91-0.5
12.840.3
25.24-.3
32.1-0.3
Matching
DSA [14]
28.8+0.7
52.1-0.5
60.6+t0.55
13.9+0.3
32.3++03
42.8=0.4
KIP [47]
49,910.2
62.7=0.3
68.6+0.2
15.7402
28.340.1
DM [17]
26.010.88
48.9=0.6
63.0104
11.4:0.3
29.7:00.3
43.6=0.4
3.9:0.22
12.9104
24.1:0.3
TM [16]
46.340088
65.3-0.7
716610.2
24.3
40.1
47.7-0.2 0.2
8.8.1..3
23.2:10.2
2..0-1.3
FRePo [37]
46.840.7
65.5-0.4
71.7+-02
28.7+-0.1
42.5+002
44.3-0.2
15.4-0.3
25,4+0.2
Parameters/ / Class
3,072
30,720
153,600
3,072
30,720
153,600
12,288
122,880
614,400
IDC [19]
50.0:0.4
67.5=0.5
74.5+0.1
44,8:0.22
Param.
HaBa [21]
48.310.8
69.9=0.4
74.010.2
33.41044
40,2:1..2
47.0=0.2
RTP [23]
66.4
71.2 0.4
73.6.10.5
34.010.4
42.9.10.7
0.7
SPEED (Ours)
73.5
77.74-0.4
45.9-10.3
28.8
30.1-0.3
Whole Dataset
84.8=0.1
56.2440.3
37.640.
Table I: Comparisons with previous dataset distillation and coreset selection methods on standard benchmarks. "Matching" refers to dataset distillation methods with specific matching objectives. "Param." " refers to dataset distillation methods with synthetic data parameterization. The bold num- bers represent the highest accuracy achieved in each category.
Daraset IPC
ImageNette
ImageWoof
ImagePrait
ImageMoow
ImageSqumkk
Image Yellow
10
10
10
10
10
10
TM (16)
47.7-09
63.0+13
286400.8
38.8
2666408
4.331.3
30.7=16
40.4+22
39.4
12,3+10
45.240.8
60.01.5
[37]
48.1-07
66.51088
29.740.6
Parameters Class]
49,152
491,520
49.152
491,520
49,152
491,520
49.152
491.520
49,152
491,520
49,152
491,520
HaBa [21]
519-17
64.72166
(32.40077
3861133
34.711.1
42.51161
36.9-099
42.91099
(4.9914
$6.8110
50411.
63.0:116
SPEED (Ours)
66.9 0.7
7,
300--099
44.1+1.4
43,4+066
5000-008
43.6
0.7
52,0-133
60.9-19
718+1.3
62.6+133
70.5+115
Whole Datases
87.4+100
67.0-13
63.9+20
66.741.1
87.5-00.3
84.4:0.
Table 2: Comparisons with previous methods on high-resolution ImageNet subsets.
ImageNet into 6 subsets, namely, ImageNette, IoafeWoof, ImageFruit, ImageMeow, ImageSquawk, and Image Yellow, each consisting of 10 classes with resolutions of 128 x 128. And a 5-layer ConvNet employed as the model for both training and evaluation.
Notably, our results achieved with IPC 1 storage space are highly competitive with the previous ofate-of-the-art results [21, 37] obtained with IPC 10. Specifically, we only need to employ 10% of their parameters to achieve similar or even better performance. Compared to the previous state-of- the-art [21] using the same IPC I storage space, our approach exhibits average improvement of 11.2% across all subsets. Moreover, we maintain a substantial lead for IPC 10 storage space. For instance, we achieve 71.8% accuracy on ImageSquawk, which is 15.0% improvement over the previous state-of-te of-the-art [21]. These outstanding outcomes are attributed to our design of sharing SAETs among patches, which enables SPEED to be more effective in reducing spatial redundancy as data resolution increases.
Continual Learning. Owing its expressive representations and finer-grained image construction, SPEED has the ability to synthesize an informative dataset. Synthetic images of each class are of exceptional quality, thereby contributing the dataset's overall richness and relevance. Following the DM [17] setup based on GDumb [51], we conduct continual learning experiments on CIFAR 100 100 with IPC 20 storage space and use the default ConvNet and ResNet18 for evaluation. We randomly divide the 100 classes into 5 learning steps, that is, 20 classes per step. As illustrated in Fig. 2, SPEED maintains the highest test accuracy at all steps for both evaluation networks.
3.2 Generalization
Universality to Matching Objectives. SPEED exhibits adaptability to multiple existing matching objectives, including those presented in [13, 16, 17], and can be directly integrated with them. line with [52], we evaluate a diverse set of architectures, including the default ConvNet, ResNet [2], MLP. and ViT [3]. The results and comparisons to corresponding baselines on CIFARTO are presented in Tab. 3. For instance, when using MLP for evaluation under the IPC 1 budget, SPEED yields

---

W
201
humbar clases
90%
(a) ConvNet
(b) ResNet18
(a) ConvNet
(b) ResNet18
Figure 2: Test accuracy of continual learning. SPEED
Figure 3: Robustness of the synthetic dataset on different architectures. Each di- rection represents each type of corruption.
maintains the best performance on all steps.
Exaluation
ComNNet
MLP
ResNer18
VIT
Masching
10
50
10
50
20.3+04
510+0.6
56.8+04
29.000:5
34.1-044
31,6+066
274-007
44.014
4504-1
34.404
30.+0.5
Gradient
48.510.3
52610.3
35.5102
47.510.
55010..3
30,4006
+19.2
+1.6
43.5
49.1
DM
265504
47610.6
62.0103
100.0.6
36,1103
40.5104
206103
38.21111
52810,6
20.5
34.10.5
45210.
Distribution
SPEED
45.0=004
62020.3
66.4203
543:003
42.2:0.3
46.3404
407208
576=00
08.240.3
28,4306
43
43.220.2
489203
Cain
+18.5
+144
+4.4
+248
+7.8
+588
420.1
+19.4
+124
47.9
+8.8
+37
1M [16
44.211.2
637200
70.3206
104:00
308206
38.5203
34.2+14
45.2414
00030.7
21,504
32.6206
47.720.6
Trejectory
SPEED
63.2+0.1
73.5+0.
77.7+0.
54 1400
44,4004
47,740.
$3.9+07
69.5+04
76.4+0.
37,5408
51.5+033
$4.5+0.5
+19.0
+7.4
+2377
+19.7
+24.3
+16.4
+100
+17.9
Table 3: Universality to different matching objectives. SPEED compatible with a variety of matching objectives and brings significant accuracy improvements over the corresponding baseline methods. All experiments are conducted with DSA augmentation [14].
significant improvement of 24.8% and 23.7% in accuracy compared the distribution [17] and trajectory [16] matching baselines, respectively. Our experiments reveal that the improvements in cross-architecture accuracy tend to be more significant than those the ConvNet. For instance, with an IPC 10 budget for trajectory matching [16], we achieve a 24.3% gain on ResNet18. This outcome further showcases the strengths of our sparse parameterization approach in terms of generalization.
Cross-Architecture Performance. The main purpose of dataset distillation is distill syn- thetic dataset that is effective on various even un- seen architectures. In this study, we evaluate the cross-architecture performance of our method by comparing it with previous synthetic data param- eterization approaches [19, 21, 22] on CIFAR 10, using IPC 10 storage space. The results presented in Tab. demonstrate that SPEED continues to outperform other methods significantly terms of generalization across unseen architectures. As an illustration, our method achieves an accuracy of 51.5% on ViT, which represents a 3.6% improve- ment over the previous state-of-the-art [19]. Although various existing dataset distillation matching objectives tend to overfit the training network, address this challenge by pruning unnecessary features through sparse parameterization.
Method
ConvNet
MLP
ResNet18
VIT
IDC [19]
67.5+0.55
41.420.2
62.9=0.6
47.9=0.8
HaBa [21]
69.922..4
35.420.4
60.2=0.9
42.220.6
RIP [22]
71.2:0.4
27.2:0.2
67.520.11
35.710.4
44410.4
51.5 0.3
Table 4: Comparision of cross-architecture gener- alization evaluation. SPEED exhibits outstanding leads on all architectures.
Robustness to Corruption. To explore the out-of-domain generalization of our synthetic dataset, we conduct experiments on CIFAR 100-C [53]. In detail, we evaluate on ConvNet and ResNet18, 8, using the synthetic dataset trained under the IPC 1 budget. Fig. 3 shows the average accuracy of 14 types of corruption under levels respectively. Compared with the previous methods, we achieve better performance under all kinds of corruption. Especially methods significantly, achieving almost double the test accuracy under every corruption scenario. This demonstrates the generalization and robustness benefits brought by sparse parameterization.
ResNet18, SPEED outperforms previous
3.3 Ablation Study
Size of for Feature Sparsification. Our total storage parameters are positively correlated with the size of k, and the representation ability of the SCM is also related to this w-winner. Therefore,
7

---

(a) Before feature sparsification
(b) After feature sparsification (density = 0.3%)
Figure 4: ImageS synthetic data samples before and after feature sparsification. The process of sparsification does not lead significant loss of essential information, resulting in a high degree of similarity between the two images.
Budget
ConwNer
RoNNelS
R
N/e
ConvNet
MLP
ResNer18
15M
000004
575K
003+0.7
20.7=044
4.520.2
17.3+0.5
99-1.1
73.5003
095:04
2
17
26.9
6.0.1.33
21.3::06
14,9.1.3
173K
350000
664:0.5
452-00
25.420.1
5.320.1
19.520.6
13.9+0.2
106K
578+0.3
248+04
900000.4
367-0.
Table 5: Different k for feature sparsification.
Table 6: Depth of FReeNet.
N/c
ConvNet
ResNet18
D
N/c
ConvNet
ResNet18
H
N/c
ConvNet
ResNet18
38.5:10.2
29.8=0.5
48
38.0.10.1
29.0=0.4
38.7=0.1
32,2:10.4
64
11
40.
29.9 0.3
72
12
38.7400.3
29.8-0.5
2
16
39.3-0.4
30,5++0.3
10
37.340.1
29.3=0.4
96
40.0:10.4
29.9-0.3
40.0=0.4
29.9:10.3
128
10
38.210.3
29.6=0.4
144
35.510.4
2..7=0.3
35.9=0.4
27.4:00.8
Table 7: Number of SAETs.
Table 8: Dimension of SAETs.
Table 9: Head of SAETs.
discuss the preference for the value of k. Specifically, for cross-architecture evaluation on CIFARIO with IPC 10 storage space, we set lo to (4096, 96, 48, 24, 12}. as shown in Tab. 5 (Budget denotes whether the storage budget is met). We observe that taking a moderate value of can effectively improve the cross- archsttcchee generalization ability of the synthetic dataset while maintaining high test accuracy on the homogeneous network, i.e., the training backbone. When À set 96, the overall accuracy exceeds that of the result obtained without feature sparsification, thereby demonstrating the effective removal of inefficient features and the improvement in generalization. order to meet storage budget (10 x 10 x 48 x 64 x 307K), we finally set l to 48, which maintains competitive performance.
Feature Sparsification Visualizations. To compare the representation capabilities SCMs after feature sparsification, provide some visualization results ImageSquawk, shown in Fig. 4. We note that the visualization results using only the sparsified features are highly similar to the corresponding original ones. Differences noticeable only in few patches, which are insignificant for classification and mainly consist of meaningless backgrounds. This implies that synthetic images can be constructed effectively using only few prominent features.
Trade-off' between Quality and Quantity. Eq. (7) illustrates that SPEED's parameters consist of three parts. The parameters of SAETs and FReeNet are shared among all synthetic images, while the parameters of SCMs are specific to each synthetic image. Thus, under the same budget, increasing shared parameters will lead to decrease in the number of synthetic images. We conduct ablation experiments on CIFAR 100 with IPC storage space study the trade-off between quality (more shared parameters) and quantity (more synthetic images). Tab. 7-9 study the size of SAETs. As can be seen, keeping a moderate number synthetic images is important, i.e., N/c > 10. A lack of diversity caused by too few images can impact the performance. However, once certain number of images reached, further increasing the quantity can lead to decrease in the quality of each image and slightly reduce performance. In general, our performance is insensitive to the size of SAETs

---

and outperforms the previous ofte--of-the-rr [22] in multiple settings, except when the number of synthetic images is too small. Tab. 6 studies the depth of FReeNet, we conduct ablation experiments on TinylmageNet with IPC 1 storage space. Although using only one block can maximize the number of synthetic images per class (N/c), the lack of hierarchical features severely compromises the model's 's representational capacity, resulting in poor image quality. However, having too many blocks can result in increased reconstruction overhead. We find that FReeNet with two blocks already achieves a great compromise between performance and computation cost.
Effects of Increasing Image Resolution. To study the ef- fectiveness of our method in higher resolution scenarios, we both the distribu- tion matching [17] baseline and our method, using the Ima- geNette dataset with image sizes of x28x 128 and 256x256. In line with the practices of previous methods that handle higher resolution images with deeper networks, we increased the depth of the ConvNet to 6 for the 256x 256 image size.
128 128
256x256
Gain
performed experimental investigations
DM [17] SPEED
28.6:::0.6
19.5
+0,9
53.5
57.7400.9
+4.2
Gain
+24.9
+28.2
+3.3
Table 10: Effects of increasing image resolution on ImageNette.
As shown in Tab. 10, when the resolution increases from 128x x 128 to 256x256 the gain brought by SPEED to the baseline also amplifies from 24.9% to 28.2%. The results demonstrate that our method achieves more substantial improvements when applied to higher-resolution images.
Related Work
Dataset Distillation. Dataset distillation, proposed by Wang el al. [12], aims to learn a smaller synthetic dataset so that the test performance of the model on the synthetic dataset is similar to that
of the original dataset. For better matching objectives, Zhao et al. [13] present a single-step matching framework, encouraging the result gradient of the synthetic dataset and the original dataset to be similar, further extended by [15, 19, 54, 55]. Subsequently, Cazenavette et al. introduce TM [16] to alleviate the cumulative error problem of single-step matching, which inspires series of work [56- 58]. To avoid the expensive computational overhead brought by complex second-level optimization, Zhao et al. [17] suggest DM, distribution matching approach, and Wang er al. [18] explicitly align the synthetic and real distributions in the feature space of downstream network. There are also some methods based on kernel ridge regression [59, 47, 60, 37], which can bring out closed-form solution for the linear model, avoiding extensive inner loop training.
In terms of synthetic data parameterization, Kim et al. introduce IDC [19], using downsampling strate- gies to synthesize more images under the same storage budget. Zhao et al. [20] propose to synthesize informative data via GAN [61, 62]. Deng et al. [22] have explored how to compress datasets into bases and recall them by linear combinations. Liu et al. [21] propose a dataset factorization approach, utilizing image bases and hallucinators for image synthesis. Lee es al. [63] further factorize the dataset into latent codes and decoders. These parameterization methods are designed to find shareable or low-resolution image bases. Still, they either only consider the connections between synthetic images [21, 22, 63), or do not reduce the redundancy inside the synthetic image thoroughly [19]. So the storage space of their image bases is still proportional to the dataset resolution, bringing out unsatisfactory performance on high-resolution datasets. SPEED is a universal synthetic data parame- terization framework, which the distillation and construction are spatial-agnostic. allows for joint modeling correlations between and within the synthetic images, leading to lower storage redundancy and finer-grained image synthesis, thus performing competently on high-resolution datasets.
Sparse Coding. Sparse coding calls for constructing efficient representations of data as combina- tion of a few high-level patterns [31-36]. It has been proven to be an effective approach in the field of image reconstruction and image classification [64-70]. Typically, a dictionary of basis functions (e.g. wavelets [71] or curvelets [72]) is used to decompose the image patches into coefficient vectors. By imposing sparsity constraints on the coefficient vectors, efficient sparse representations of the image patches can be obtained. In addition, the dictionary is not necessarily fixed, it can also be learned to adapt to different tasks [25-28], and the penalty function can also be varied [33, 73, 74]. Prior sparse coding research has primarily focused on compressing individual images [75, 76], with an emphasis on achieving high compression ratios while minimizing perceptual distortion. However, there has been limited exploration of how to apply sparse coding to compress entire datasets in

---

way that enhances the training of downstream neural networks. Our work demonstrates that theories and techniques in sparse coding can provide valuable inspiration for developing dataset distillation methods.
Coreset Selection. Coreset selection identifies a representative subset of the original dataset [77- 79, 10]. Its objective is in line with the goals of dataset distillation and can be utilized to tackle challenges such as continual learning [10, 80, 81] and active learning tasks [82]. This technique typically performs better when the storage budget is relatively large, while dataset distillation demonstrates superior performance under extremely limited storage budgets [52].
Conclusion and Limitations
In this paper, introduce SPEED, an efficient and generalizable solution for dataset distillation that offers the following merits: First, the spatial-agnostt epitomic tokens distilled by our method not only shared between the different classes but also shared among patches of every synthetic image, regardless of their spatial locations. Such efficient modeling enables us to perform well on high-resolution datasets with much less spatial redundancy. Second, the proposed feature-recurrent network promotes hierarchical representation in an efficient recurrent manner, resulting in more informative synthetic data. Finally, the proposed feature sparsification mechanism improves both the storage efficiency and the generalization ability. SPEED achieves outstanding performance on various datasets and architectures through extensive experiments.
Limitations. Similar to the previous parameterization methods [19-22], decomposing the original synthetic datasets into different components will slightly increase distilling costs and incur recon- struction overhead. We alleviate this issue by designing the network to be as lightweight as possible. While the dataset synthesized by SPEED offers better privacy protection for users compared the original dataset, there remains a possibility of privacy leakage.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 62006188 183 and Grant 62206271, 1, in part by the National Key Research and Development Project of China under Grant 2020AAA0105600, and in part by the Shenzhen Key Technical Projects under Grant JSGG2022083 1058000000 01100/00000000000000000002200110 (GGGG/D22022051774116000000
and JCYJ202208 1 1104406114
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamers. In NeurlPS, 2020.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
[6] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In ICML, 2020.
[7] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2020.
10

---

[8] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20 1):1977-20177 2019. 2019.
[9] lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in meddient-basee neural networks. arXiv preprint drXiv:1312.6211, 2013.
[10] Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
[11] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. In CVPR, 2020.
[12] Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 2018.
[13] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
[14] Bo Zhao and Hakan Bilen. Datasct condensation with diffcrentiable siamese augmentation. In ICML, 2021.
[15] Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In ICML, 2022.
[16] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexci A Efros, and Yun-Yan& Zhu. Dataset distillation by matching training trajectories. In CVPR, 2022.
[17] Bo Zhao and Hakan Bilen. Datasct condensation with distribution matching. In WACV, 2023.
[18] Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang. Guan Huang. Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning condense dataset by aligning features. In CVPR, 2022.
[19] Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML, 2022.
[20] Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. In NeurlPS Workshop. 2022.
[21] Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurlPS, 2022.
[22] Zhiwei Deng and Olga Russakovsky, Remember the past: Distilling datasets into addressable memories for neural networks. In NeurlPS, 2022.
[23] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
[24] Kenneth Kreutz-Delgado, Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te- Lee, and Terrence I Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-396, 2003.
[25] Ke Huang and Selin Aviyente. Sparse representation for signal classification. In NeurlPS, 2006.
[26] John Wright, Allen Y Yang. Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions Patlern Analysis and Machine Intelligence. 31(2):210-227. 2008.
[27] Rajat Raina, Alexis Battle, Honglak Lee, Benjamin Packer, and Andrew Y Ng. Self-taught learning: transfer learning from unlaheled data. In ICMI, 2007.
[28] Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. Discriminative learned dictionaries for local image analysis. In CVPR, 2008.
[29] Bruno A Olshausen, Charles F Cadieu, and David K Warland. Learning real and complex overcomplete representations from the statistics of natural images. In Wavelets XIII, volume 7446, pages 236-246. 2009.
[30] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong, Autoregressive visual tracking. In CVPR, 2023.

---

[31] Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. JEEE Transactions Signal Processing, 41(12):339773311,, 1993.
[32] Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381(6668))60776666) 1996.
[33] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
[34] Scott Shaobing Chen, David I. Donoho, and Michael Saunders. Atomic decomposition by hasis pursuit. SIAM review, 23(1):129-159, 2001.
[35] Joel Tropp. Greed is good: Algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231 222222222,, 2004.
[36] Michal Aharon, Michael Elad, Alfred Bruckstein. K-svd: algorithum for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11):4311 11):44311134222.. 1-4322, 2006.
[37] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. In NeurlPS, 2022.
[38] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint Axiv:1607.06450 2016.
[39] Jason Weston, André Elisseeff, Bernhard Schölkopf, and Mike Tipping. Use of the zero norm with linear models and kernel methods. The Journal of Machine Learning Research, 3: 3:1439-1461, 2003.
[40] Andrew Y Ng. Feature selection, 11 vs. 1 regularization, and rotational invariance. In ICML, 2004.
[41] Christos Louizos, Max Welling, and Diederik Kingma. Learning sparse neural networks through 1_0 regularization. In ICLR, 2018.
[42] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In ACL-IJCNLP, 2021.
[43] Jun Yu, Yong Rui, and Dacheng Tao. Click prediction for web image reranking using multimodal sparse coding, IEEE Transactions Image Processing, 23(S):2019 2032, 2014.
[44] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In AISTATS, 2011.
[45] Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-trim: Convex pruning of deep neural networks with performance guarantee. NeurIPS, 2017.
[46] Jundong Li, Kewei Cheng, Suhang Wang. Fred Morstatter, Robert P Trevino, Jiliang Tang, and Huan Liu. Feature selection: A data perspective. ACM Computing Surveys, 10(6):1-45, 2017.
[47] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurlPS, 2021.
[48] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. Technical report, 2009.
[49] I stanfoorddd. Cs231n: Convolutional neural networks for visual recognition.
[50] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei, Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[51] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual lcarning. In ECCV, 2020.
[52] Justin Cui, Ruochen Wang. Si Si, and Cho-Jui Hsich. De-bench: Dataset condensation benchmark. In NewrIPS, 2022.
[53] Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[54] Zixuan Jiang. Jiaqi Gu, Mingjie Liu, and David Z Pan. Delving into effective gradient matching for dataset condensation. In COINS, 2023.
12

---

[55] Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In CVPR, 2023.
[56] Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Dataset distillation using parameter pruning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2023.
[57] Jiawei Du, Yidi Jiang, Vincent YF Tan, Jocy Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory to improve dataset distillation. In CVPR, 2023.
[58] Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- Ik with constant memory. In ICMI, 2023.
[59] Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Datasct meta-learning from kemel ridge-regression. In ICLR, 2021.
[60] Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. NeurIPS, 2022.
[61] Ian Gaodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11)):39- 144, 2020.
[62] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
[63] Hae Beom Lee, Dong Bok Lee, and Sung Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208. 10494, 2022.
[64] José M Bioucas- Dias and Mário AT Figuciredo. A new twist: Two-step iterative shrinkage/thnesholdinng algorithms for image restoration. IEEE Transactions on Image Processing, 16(12):2992-3) 20044, 2007.
[65] Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12):376-3345 2006.
[66] Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. Image super-resolution as sparse representation of raw image patches. In CVPR, 2008.
[67] Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11)2861-2873 2010.
[68] Julien Mairal, Francis Bach, Jean Ponce, Guillerno Sapiro, and Andrew Zisserman. Non-local sparse models for image restoration. In ICCV, 2009.
[69] Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisserman, and Francis Bach. Supervised dictionary learning. In NeurIPS, 2008
[70] Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In CVPR, 2009.
[71] Antonin Chambolle, Ronald A De Vore, Nam-Yong Lee, and Bradley J Lucier. Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage. IEEE Transactions on Image Processing, 7(3):119-335 335, 1998.
[72] Emmanuel Candes and David Donoho. Recovering edges in ill-posed inverse problems: Optimality of curvelet frames. The Annals of Statistics, 30(3):084-33)3 2002.
[73] Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihond and its oracle properties. Journal of the American Statistical Association, 96(456) 2001.
[74] Yong Liang, Cheng Liu, Xin-Ze Luan, Kwong-Sak Leung, Tak-Ming Chan, Zong- Ben Xu, and Hai Zhang. Sparse logistic regression with 11/2 penalty for gene selection in cancer classification. BMC bioinformatics, 14(1):1-12, 2013.
[75] Emmanuel Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(():489-50) -099, 2006.
[76] David L Donoho. Compressed sensing. IEEE Transactions on Information Theory, 22(4):1289-1300 2006.
13

---

[77] Max Welling. Herding dynamical weights to learn. In ICML, 2009.
[78] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In UAI, 2010.
[79] Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In NeurlPS, 2011.
[80] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NewrIPS, 2019.
[81] Felix Wiewel and Bin Yang. Condensed composite memory continual learning. In IJCN,, 2021.
[82]
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
[83] Fastai, Fastai/imagenette: smaller subset 10 easily classified classes from imagenet, little more french.
[84] Edgar Riba. Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski. Kornia: an open source diffcrentiable computer vision library for pytorch. In WACV, 2020.
[85] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks large-scale image recogni- tion. arXiv preprint 1rXiv:1409.1556, 2014.
[86] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet. Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[87] Dmitry Ulyanov, Andrea Vedaldi. Victor Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017.
[88] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017.
14

---

I Experiment Details
Datasets and Preprocessing. We evaluate our methods on the following datasets: i) CIFARIO [48]: A standard image dataset consists of 60,000 32x RGB images in 10 different classes, including airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. For each class, 5000 images are used for training and 1000 images are used for testing. ii) CIFAR100 [48]: CIFAR100 contains 100 classes. has a training set with 50,000 images and testing set with 10,000 images. iii) TinyImageNet [49]: A 64 x 64 image dataset with 200 classes. Each class has 500 images for training and 50 images for testing. iv) ImageNet [50] subsets: High resolution (128 x 128) datasets from ILSVRC2012 [50]. ImageNette (assorted objects) and ImageWoof (dog breeds) are designed for easy and hard classification tasks [83]. ImageFruit, ImageMeow, ImageSquawk, and imageYellow [16] consist of fruits, cats, birds, and yellowish things, respectively. All the above subsets contain 10 classes.
For fair comparisons, we follow the previous methods [16, 21], adopting ZCA whitening on CIFARIO and CIFARI00 with the default Kornia [84] implementation, and no ZCA whitening is used on TinyImageNet and ImageNet subsets.
Evaluation Settings. The objective of dataset distillation is to learn a synthetic dataset that can be utilized across range of network structures. This is achieved by matching the raw and synthetic datasets with the aid of specific network, with the expectation that can generalize to unseen network structures. In accordance with the evaluation presented in [52], we consider a diverse set of architectures, as shown below:
ConvNet [1, 85, 86]: The standard architecture used for both distilling and evaluating synthetic datasets in previous distillation work. The default network contains three 3 convolution layers, each followed by 2x2 average pooling and instance normalization. The hidden embedding size is set to 128. There are around 320K trainable parameters. For TinylmageNet, the number of layers increased to 4 for improved performance, as suggested in previous work [16, 17]. Similarly, we increase it to 5 for ImageNet subsets, following [16].
MLP: The simple MLP architecture is applied for evaluation, which includes 3 fully connected layers, and the width is set to 128. For CIFARIO 10, CIFAR 100, and TinylmageNet, the MLP has 411K, 423K, and 2M trainable parameters, respectively.
ResNet18 [2]: We also evaluate synthetic datasets on the commonly used ResNet architecture with 4 residual blocks. In each block, there are 2 convolution layers followed by ReLU activation and instance normalization (IN) [87]. The number of trainable parameters is around 11M.
VIT [3]: Vision Transformer applies standard transformer [88] on non-overlapping image patches, demonstrating the variants of transformers can also be competitive alternative to CNNs. We take it as one of the architectures for evaluating synthetic datasets. There are around 10M trainable parameters the adopted implementation of ViT.
Following the mainstream evaluation settings, we train 5 randomly initialized evaluation networks on the synthetic dataset, using SGD optimizer, where the momentum and weight decay are to 0.9 and 0.0005, respectively.
Slyper-Parametery. For CIFARIO (32x32), CIFARI00 (32x32), and TinylmageNet (64x64), we fix the default number of patches J to 64, and increase it to 256 for ImageNet subsets (128x 128). Then, the default patch dimension T. can be directly derived.
In terms of SAETs, which has been studied in Sec. 3.3, we set the number K according to the resolution of the dataset (c.g. we set K to 256 for ImageNet subsets, and decrease it to 128 and 64 for TinyImageNet and CIFAR 10FA respectively). When the budget is enlarged, we expand the dimension of SAETs with its head H/ increased together, while guaranteeing that their divisor D/1 < K, so that each head E a complete (or overcomplete) dictionary. We always build the FReeNet with shallow blocks, which provides sufficient nonlinear capabilities while making it lightweight enough (e.g. we set block on CIFARIO with IPC storage space which excessively stringent, and set blocks for all experiments TinyImageNet and ImageNet subsets).
15

---

Cook
Hamate
Lamp
Vatarcyde
Tobbe
Suntower
(a) Patch size x
(b) Patch size 8
(c) Patch size 16x 16
(d) TM [16] baseline (patch size 32x32)
Figure Synthetic data samples under different configurations of patch size. small patch size can capture intricate details such as the black eye patch of the hamster, truck tires, and rabbit ears.
The number of non-zero elements k is always set to value that ensures the matrix density k/JK 5%, then the number images N can be computed according to Eq. (7). The setting of the penalty weight À depends the number of images, heads, and patches, satisfying À 0.064/NHJ.
Compute Resources. Our experiments were run on mixture RTX 3090, RTX A6000, and A800 GPUs. The GPU memory consumption is mainly dependent on that of the matching objective and is slightly higher than its baseline, due the larger data amount. For instance, our experiments based on gradient matching [13] and distribution matching [17] need one 24GB 3090 GPU. In terms of trajectory matching [16], the VRAM usage will be higher, ranging from one 24GB 3090 GPU to six 80GB A800 GPUs.
II Additional Results and Analyses
Patch Size. The patch size plays important role in determining the granularity of our distillation modeling. Let each patch E be of P x resolution and contains channels, then its dimension L will be C²C. Tab. presents the results CIFAR100 with IPC storage space. We observe that setting a relatively small patch size can lead to better performance, as reduces redundancy within synthetic images and enables finer-graine synthesis. Therefore, set a default patch size of 4x4 for the 32x resolution datasets (CIFAR1Ó and CIFAR 100) and increase it appropriately when the resolution of the dataset increases. For instance, adopt default patch size of 8 x for TinylmageNet 64).
Patch size
N/c
ConvNet
MLP
ResNet18
VIT
40.0
15.5+0.2
29.9-0.3
20.7-10.5
8 x
8
10
39.310.3
14.610.3
29.4=0.5
20.2+10.2
16
16
7
37.310.3
12.910.3
27.1=0.8
16.5+10.2
Table I: Patch size. Small patch sizes allow finer-grained modeling.
Furthermore, we provide visualization of synthetic images with varying patch sizes, as depicted in Fig. I. The utilization of small patches can facilitate the expression of finer- finer-grained features, leading to synthetic images that contain more classifiable information. For instance, when a 4x4 patch is employed, it can capture intricate details such as the black eye patch of the hamster, in addition to more noticeable features like truck tires and rabbit ears. Also, sparse parameterization allows for efficient storage background information with limited significance by allocating fewer non-zero elements in SCMs to represent it.
16

---

TH
42
60
Teat Accuracy (%6
Teet Accuracy
40
30
40
Test
Local
Hada
39
SPEED
16:7
5e.7 16.6
20
40
60
Penallty Weight
Size
Distlling Time (mins)
Figure II: Left: Effects of different penalty weight À on test aceuracy. TM [16] denotes the baseline method of the trajectory matching objective with naive image-independent parametcrization. A moderate weight can trade off sparsity and aceuracy. Middle: Comparison of different feature sparsification strategies. Global sparsification performs better in various settings, especially when the storage budget is stringent. Right: Test accuracy and distilling time.
Aull
Top-48
top-24
top-12
full
1op-48
top-24
top-12
full
top-48
top-24
top-
Figure III: Visualization of our SCMs with varying values of (patch corresponds to zero column vector in its SCMs is marked grey). SPEED learns efficiently allocate more storage for foreground objects that contain more classifiable information. Therefore, a small & still allows for salient features of objects to be synthesized well.
Sparsity Penalty Weight X. SPEED, the sparsity penalty term constrains the overall scale of elements in SCMs. Ablation studies of the penalty weight À are conducted CIFARION with IPC 1 storage space, as shown in Fig. II (left). For these experiments, we fix the number of sparsified features to 48. If the penalty weight is too small, our SCM tends to comprise higher number of elements with larger absolute values, which necessitates more features to be maintained during feature sparsification, thereby guaranteeing the test performance. consequently implies a larger storage burden. On the other hand, setting the penalty weight too high may limit the representation capability required for synthesizing informative images. Therefore, prefer the moderate penalty weight, which allows for sufficient combinations of required epitomic tokens while efficiently eliminating inefficient ones without the risk of losing important information.
Local Feature Sparsification. To assess the efficacy of our feature sparsification with global semantic preservation on SCMs, we introduce the contrast, local feature sparsification, that attempts to store an equal number of non-zero elements for each patch, i.e., each column of SCM has the same number of non-zero elements. Consequently, local feature sparsification constructs all patches using the same number of parameters (keep k/J elements for each column of SCM), while the feature sparsification with global semantic preservation does not (keep h elements for whole SCM). We evaluate the performance of these two methods using different values of k, while maintaining the other hyper-parameters constant.
As illustrated in Fig. II (middle), we conduct tests on CIFAR100 with i: set to (64, 128, 192, 256}. where the SCM size is 64 64. Accordingly, the local feature sparsification strategy preserves 2,,2,3, 4) features for each synthetic patch (e.g. when k: is set at 128, J = 64, then each patch is represented by the most prominent SAETs). all cases, the global feature sparsification outperforms the local one. Notably, the advantages of the global approach more apparent when the storage
17

---

(a) no convolutional layers
(b) layer, kernel size = x
(c) I layer, kernel size 5x5
(d) 2 layers, kernel size 3x3
(e) 2 layers, kernel size 5
Figure IV: Visualizations (border terrier class in ImageNette) regarding the incorporation of convolu- tional layers after the rearrangement of patches. The presence of the chessboard artifact gradually diminishes with increases in the number convolutional layers and the size of convolutional kernels.
budget is stringent, L.e., for small k. This indicates that global feature sparsification technique produces synthetic images with more classifiable information under the same storage budget, making it more efficient for dataset distillation. Similar to the conclusion drawn in Sec. 3.3, observe that as k increases to a certain extent, the accuracy reaches an upper bound and then fluctuates.
Test Accuracy vs. Distilling Time. We observe the test accuracy and distilling time on CIFARIO with IPC I storage space. and compare with the previous methods [16, 21, 22]. as illustrated in Fig. II (right). Overall, SPEED consistently outperforms previous methods when compared at similar distilling time points.
Visualization of SCMs. We visualize our SCMs with varying values of k in Fig. III, where the patch corresponds to zero column vector in its SCMs is marked grey. As & decreases, we observe a higher preference for column vectors that correspond background patches to be zero vectors in SCMs. This indicates that SPEED automatically learns to allocate more storage budget to foreground objects that contain foreground object is still well synthesized. It should be noted that the zeroed-out patches do not occupy any storage budget since the SCMs saved in the COO compressed format.
useful clues for classification. Even when & is set low 12, the salient
Incorporation of Additional Convolutional Layers. We find a chessboard (blocky) artifact on synthetic images of high-resolution datasets. To investigate its influence test accuracy, we perform multiple experiments on ImageNette with IPC storage space, adding and 2 convolutional lay- ers with kernel sizes 3 and 5, as shown in Tab. II. As evident from the results, the incorporation of additional convolu- tional layers in our experiments does not yield significant improvement in downstream training. However, does provide slight relief from the chessboard artifact, as depicted in Fig. IV. The impact of the chessboard artifact on downstream training and the exploration of efrfmeter-efficient methods eliminate these artifacts warrant further investigation.
Kernel size
3x3
5x5
layer
66.3.1 1.8
66.4.11.3
2 layers None
65.9 1.3
64.0.10.5
6..9
Table II: Enlargement of the image res- olution on ImageNette.
18

---

Block-shared
Evaluation
Spar.
SAET
SCM
Param
ConvNet
MLP
ResNet18
ViT
27M
41.2.10.4
15.6_0.2
29.8
21.1100.3
27M
41.3.10.2
15.6-0.3
29.4.10.4
20.6.10.1
/
14M
41.2+t0.4
15.3=0.1
29.6=0.66
20.9+0.4
/
549K
40.0.10.66
15.6=0.1
30.3.10.5
20.7:10.4
543K
40.3:1..77
15.6-0.1
29.7.10.7
21.0:10.4
305K
40.0:10.4
15.5=0.2
29.9
20.7
Table III: Storage efficiency study. The checkmark v/ in the Spar. column indicates the use of sparsified SCMs, while the absence of mark indicates the of full-element SCMs. The checkmark / in the Block-shared SAET/SCM column indicates sharing SAETs/SCMs across different scales of blocks, while the absence of mark indicates using block-specifer SAET</SCMS.
IPC
1
10
50
Parameters/ / Class
3,072
30,720
153,600
IDC [19]
40 (44.8++0.2)
HaBa [21]
5 (33.4+0.4)
45 (40.2++0.2)
245 (47.0++..2
RTP [22]
16 (34.010..4
232 (42.9
SPEED
11 (40.0.10.4)
62 (45.9
100 (49.110.22)
Table V: Comparisons on quantity (N/c) and quality (test accuracy) of synthetic images.
Storage Efficiency Study. To investigate the storage efficiency of recurrent blocks that involves utilizing block-shared SAETs and SCMs, we attempt allocate specific SAETs and SCMs for each block, along with setting a uniform value for in feature sparsification, as studied in Tab. III. Our results indicate that applying recurrent blocks leads the highest efficiency. While the block- specific strategy shows a marginal improvement over the block-shared strategy with or without feature sparsification, it lacks significant performance advantage. However, recurrent blocks using block-shared SAETs and SCMs come with significantly lower storage, saving about 44% parameters while achieving competitive results. Furthermore, we observe that feature sparsification can enhance both the overall cross cross- architerceee generalization performance and storage efficiency.
Number of Synthetic Images. The number of our syn- thetic images on ImageNet subsets is summarized in Tab. IV. Our method synthesizes 15 images under the IPC 1 budget, and remarkably, it achieves performance that is competitive with other methods operating under the IPC 10 budget, i.e., 10 synthetic images, while utilizing only 10% of the param- eters. For instance, on ImageNette, our method achieves an impressive accuracy of 66.9% with IPC 1, surpassing the previous state-of-the-arr [37] result of 66.5% achieved with IPC 10. These findings demonstrate the efficiency of our method and the high quality of the synthetic images it produces.
IPC Parameters/ Class
1
10
49,152
491,520
# Synthetic images
15
111
Table IV: Number of synthetic images on ImageNet subsets.
To further prove the above claim, we conclude the number of synthetic images on CIFARIO0 compared with other parameterization work [19, 21, 22], as shown in Tab. V. As evident, while the number of our synthetic images is not the highest among all approaches, our outstanding performance clearly showcases the high quality of the synthetic images. This further emphasizes that our approach enhances performance by improving both the quality and quantity of the synthetic images. Moreover, it demonstrates the highly efficient reduction of spatial redundancy achieved by our method.
III Additional Visualizations
We include additional visualizations here. Fig. V-VII show the synthetic datasets for CIFARIO and CIFAR100 Higher-resolution datasets are visualized in Fig. VIII-XV. We find that visualizations of lower-resolution datasets are more human-friendly since they contain less spatial redundancy. For high-resolution datasets, storage budgets are primarily allocated to salient foreground features, resulting significant reduction of useless patches. Nevertheless, the main texture and rough shape can still be recognized.
19

---

Figure V: Synthetic images on CIFARIO.
20

---

Figure VI: Synthetic images on CIFARI00. (Classes 1-50)
21

---

Figure VII: Synthetic images on CIFARI00. (Classes 51-100)
22

---

Figure VIII: Synthetic images on TinylmageNet. (Classes 1-100)
23

---

Oo
Figure IX: Synthetic images on TinylmageNet. (Classes 101-200)
24

---

Figure X: Synthetic images on ImageNette.
Figure XI: Synthetic images on Image Woof.
25

---

Figure XII: Synthetic images on ImageFruit.
Figure XIII: Synthetic images on ImageMeow.
26

---

Figure XIV: Synthetic images on ImageSquawk.
Figure XV: Synthetic images on Image Yellow.
27


Reviews:
[{"review_id": "3abfHYOuvB", "reviewer": "Reviewer_kT2g", "summary": "This paper proposes a new parameterization for dataset distillation. The new parameterization considers image patches, use sparse matrix and recurrent feature net to generate synthetic images. The total parameters follows storage constraint. The experimental results show improvement over previous methods.", "questions": "See weaknesses section.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ This paper proposes a new parameterization for synthetic images in Dataset Distillation. \n+ The proposed method works well across various benchmarks, including imagenet, cifar and cross arch generalization\n+ This paper is quite interesting in proceeding the research on Dataset Distillation parameters. Spatial redundancies are quite dominant in standard parameterization, and this method can certainly help with alleviating that.", "weaknesses": "- The paper claims on reducing spatial redundancies. I wonder how the authors compare their method with [19]. Does that also consider reducing spatial redundancies? \n- The paper provides better empirical performance, but the research messages are not quite surprising.\n- Have the authors considered using algorithms similar to [a] for sparsification?\n\n[a] Parameter-Efficient Transfer Learning with Diff Pruning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer kT2g for the valuable comments and feedback. We deeply appreciate the reviewer's acknowledgment of the merits of our proposed method, including its effectiveness in reducing spatial redundancy, its capacity for generalization, and its superior performance compared to previous approaches. We are fully committed to addressing the concerns raised by the reviewer, and we outline our responses below.\n\n1. **How the authors compare their method with [19]. Does that also consider reducing spatial redundancies?**\n\n    As we discussed in _Line 283_, IDC [19] takes a simple downsampling strategy to save more images, which also partially reduces the spatial redundancy. However, naive downsampling suffers from two major drawbacks:\n   * The uniform downsampling operator employed in IDC indiscriminately discards spatial information, including crucial details that are vital for downstream training. In contrast, our method incorporates end-to-end spatial-agnostic learning to preserve informative areas that are crucial for downstream training, while discarding irrelevant and repetitive areas.\n   * Downsampling **can not** effectively reduce spatial redundancy within the same image across different locations, let alone across different images, as it is **not spatial-agnostic**. However, our method is specifically designed to be spatial-agnostic, allowing us to further mitigate spatial redundancy in such scenarios.\n\n\n2. **Have the authors considered using algorithms similar to [a] for sparsification?**\n\n   * We appreciate the introduction to the noteworthy work [a] on transfer learning. This method employs a more sophisticated approach to approximate $l_0$ sparsity by relaxing a binary vector into continuous space. In contrast, our method directly utilizes $l_1$ penalty as a surrogate for $l_0$ penalty. Both techniques are widely recognized and extensively utilized in the machine learning community.  It is important to note that the specific method used to approximate $l_0$ sparsity does not affect our overall parameterization framework.  We will discuss various sparsification techniques in our related work, including the one presented in [a], and plan to investigate and evaluate their respective impacts in future studies. We appreciate your suggestion.\n\n----\n\n[a] Guo, Demi, Alexander M. Rush, and Yoon Kim. \"Parameter-efficient transfer learning with diff pruning.\" arXiv preprint arXiv:2012.07463 (2020)."}, {"review_id": "5fyKdTQNyB", "reviewer": "Reviewer_yNYq", "summary": "This work proposes a new memory-saving method of dataset distillation by distilling the dataset into a set of Spatial-Agnostic Epitomic Tokens which are indexed by Sparse Coding Matrices and decoded into images by a Feature-Recurrent Network. This method is plug-and-play compatible with existing distillation methods, allowing them to achieve more efficient storage. State-of-the-art results are shown for many datasets and problem settings. ", "questions": "I am happy to raise my score if the above weakness is addressed.\n\nI am also just curious about another thing:\n\nThe high resolution images seem extremely blocky due to the nature of the patches.\n\nHave you considered adding 1 or 2 convolutional layers _after_ the patches are stitched back together?", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The overall presentation of the paper is very nice. The figures and equations very clearly explain to the reader the main ideas. The colorfully annotated Eq 7 especially makes the storage budget easy for the reader to digest. \n\nThe SPEED method itself is quite interesting, and algorithm 1 very clearly explains the process.\n\nThe many ablation studies and side-experiments further explain the effectiveness of the method.", "weaknesses": "The biggest issues I have with this paper are the presentation of tables 1 and 2.\n\nThis is not an issue specific to this paper, but methods that do re-paramaterization as a means of memory saving should not be directly compared to methods that only propose a matching algorithm.\n\nIt should be made extremely clear that IDC, HaBa, RTP and SPEED are solving an inherently different problem than the baseline methods.\n\nSynthetic set size should also not be given in \"IPC\" but in the number of learnable parameters, since the given IPC simply isn't true anymore. For example, instead of IPC=10, you could have #Params <= 30,720 (10x3x32x32)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer yNYq for the insightful suggestions and are happy that the reviewer finds our work interesting and effective. We are glad to address the concerns and take the suggestions as follows:\n\n1. **The presentation of tables 1 and 2. Re-parameterization should not be directly compared to methods that only propose a matching algorithm.**\n\n   * Thank you for considering the suggestion. We acknowledge that a direct comparison between pure matching methods and parameterization methods can be a matter of controversy, and we agree on the importance of presenting the tables in a manner that eliminates any ambiguity. As per your request, we have reorganized Table 2 into a _Table A in the submitted PDF of the global response_. In Table A, we have made explicit differentiations between pure matching methods and parameterization methods, ensuring a clear separation between these two categories. Furthermore, we have included an additional row to indicate the corresponding parameter amounts for the parameterization methods. We will also make the same revisions to Table 1.\n\n2. **The high resolution images seem extremely blocky. Have you considered adding 1 or 2 convolutional layers after the patches are stitched back together?**\n\n   * We sincerely appreciate the valuable advice provided. As suggested, we performed multiple experiments on ImageNette under IPC 1 storage budget, adding 1 and 2 convolutional layers with kernel sizes 3 and 5:\n\n     | Kernel Size | 3x3       | 5x5        |\n     | ------      | ---       | ---        |\n     | 1 layer     | 66.3 $\\pm$ 1.8%     |  66.4 $\\pm$ 1.3%     |\n     | 2 layers     | 65.9 $\\pm$ 1.3%     |  64.0 $\\pm$ 0.5%     |\n     | None        | **66.9 $\\pm$ 0.7%** |  **66.9 $\\pm$ 0.7%** | \n\n     As evident from the results, the incorporation of additional convolutional layers in our experiments did not yield a significant improvement in downstream training. However, it did provide slight relief from the chessboard artifact (blocky artifact), as depicted in _Figure A in the submitted PDF of the global response_. Nonetheless, the impact of the chessboard artifact on downstream training and the exploration of parameter-efficient methods to eliminate these artifact warrant further investigation."}, {"review_id": "p27U4hWyPy", "reviewer": "Reviewer_7J1t", "summary": "The paper proposes a new framework(SPEED) to perform dataset distillation.\n\nThe new framework is composed of 3 parts:\n1. Spatial-Agnostic Epitomic Tokens (SAETs)\n2. Sparse Coding Matrices (SCMs)\n3.  A Feature-Recurrent Network (FReeNet)\n\nThe paper also employees multi-head attention to ensure the diversity of distilled images, thus achieving new SOTA. \nThe new framework SPEED is also claimed to work with multiple dataset matching methods and enhance their performances.\n\nThrough various experiments, the method shows strong performances on CIFAR-10/100 and TinyImageNet. Similar results are also observed on ImageNet subsets.\n\n### I have read the author's response and my concerns are addressed by seeing more experiment results.", "questions": "See my comments in weakness section. On general, this paper provides some insights on a totally different way of parameterization in dataset distillation. I am willing to raise the score if the above questions are answered.", "limitations": "The paper proposes a new method to apply parameterization in dataset distillation. However, the generalization ability of the method is unknown such as the recurrent neural network. Especially some evaluation results in table 2 are missing which makes it even harder to tell.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "## originality\nSPEED is claimed to be the first paper studying the spatial redundancy in the field of dataset distillation/condensation\n\nSPEED applies a few methods such as the concepts from ViT, dictionary learning and sparse coding.\n\n## quality\nThe method works very well on dataset with higher resolution.\n\n## clarity\nThe paper is well written and easy to follow\n\nThe framework of the method and learned images are visualized which makes it easy to understand.\n\n## significance\nThe proposed methods achieve SOTA performances under most settings.\n\nThe paper novelly proposes to distill images at image patch level which shows a new way for dataset distillation.\n\nThe proposed method is compatible with multiple matching objectives.", "weaknesses": "- Incomplete evaluation results in table 2. IPC 1/10/50 are used for CIFAR-10/100, but only IPC 1 for TinyImageNet is used. Why that is the case? From [1], even the trajectory matching method this paper adopts has reported the results on TinyImageNet with IPC 1/10/50.\n\n- In table 1, for the parameterization methods including SPEED, can the author include how many synthetic images are generated for evaluations, e.g. 11 for IPC 1 so that we know if the performance gain is due to increased number of images or increased quality of generated images.\n\n\n\n\n\n\n\n[1] Dataset Distillation by Matching Training Trajectories", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer 7J1t for the pertinent and valuable feedback. We are delighted to learn that the reviewer finds our method achieves good performance across multiple datasets demonstrated in Table 1 and Table 2. The concerns are fully addressed as follows.\n\n1. **Incomplete evaluation results in table 2? Only IPC 1 for TinyImageNet is used. Why that is the case?**\n\n   * Thanks for the question. It is important to highlight that our results, achieved under the IPC 1 storage budget (ACC: 26.9%), have already surpassed the performance of the baseline method (trajectory matching [16]) under the IPC 10 storage budget (ACC: 23.2%) on TinyImageNet. Moreover, our results even approached the performance of the baseline method under the IPC 50 (ACC: 28.0%) setting.\n\n   * As highlighted by reviewer yNYq, it is of utmost importance to conduct comprehensive comparisons among different parameterization methods. Previous parameterization studies either lacked reporting results on TinyImageNet [19, 21] or only provided results under the IPC 1 budget [22]. Consequently, experiments conducted under higher IPCs lack meaningful comparisons. Due to the limitations of time and computational resources for submissions, we did not prioritize these experiments accordingly.\n\n   * As per your suggestion, we are willing to report our experimental results on TinyImageNet under IPC 10/50 as an addition, as shown in the following table:\n\n     | IPC (#Param) | 1 (12,288)             | 10 (122,880)             | 50 (614,400)             |\n     | -------- | ----- | ----- | -----         |\n     | TM | 8.8 $\\pm$ 0.3%  | 23.2 $\\pm$ 0.2% | 28.0 $\\pm$ 0.3%        |\n     | IDC | -  | - | -        |\n     | HaBa | -  | - | -        |\n     | RTP | 16.0 $\\pm$ 0.7%  | - | -        |\n     | Ours     | **26.9 $\\pm$ 0.3%** | **28.8 $\\pm$ 0.2%** |    **30.1 $\\pm$ 0.3%**   |\n\n     Our method achieves an accuracy of 28.8% under the IPC 10 budget, showcasing substantial improvements over the baseline (ACC: 23.2%), and even surpassing the baseline under IPC 50 (ACC: 28.0%). Additionally, our method achieves an accuracy of 30.1% under the IPC 50 budget, establishing a state-of-the-art performance. These experimental results have been incorporated into _Table A in the PDF of the global response_, thus making our evaluation more comprehensive. \n\n2. **Can the author include how many synthetic images are generated for evaluations? Is the performance gain due to increased number of images or increased quality of generated images?**\n\n   * Thanks for your insightful attention to the quality and quantity aspects of dataset parameterization. We also recognize the significance of this topic, and therefore, we have dedicated a single section in our paper to thoroughly discuss it. The experimental results can be found in _Tables 6 to 8_, and the discussion can be found in _Lines 254 to 270_. In conclusion, both quality and quantity play crucial roles in parameterization. Excessively sacrificing one in favor of the other without careful consideration can lead to a noticeable decline in performance.\n   \n   * As suggested, the number of synthetic images on ImageNet Subsets in Table 1 is summarized:\n\n     | IPC (#Param)  | 1 (49,152) | 10 (491,520) |\n     | ---- | ----      |----|\n     | #Synthetic Images | 15 |  111  | \n\n     Our method synthesizes 15 images under the IPC 1 budget, and remarkably, it achieves performance that is competitive with other methods operating under the IPC 10 budget, while utilizing only 1/10 of the parameters. For instance, on ImageNette, our method achieves an impressive accuracy of 66.9% under the IPC 1 budget, surpassing the previous state-of-the-art result of 66.5% achieved under the IPC 10 budget. These findings demonstrate the efficiency of our method and the high quality of the synthetic images it produces.\n\n   * To further prove the above claim, we conclude the number of synthetic images on CIFAR100, compared with other parameterization work:\n\n\n     | IPC (#Param) | 1 (3,072)             | 10 (30,720)             | 50 (153,600)             |\n     | ---- | -------------- | --------------- | --------------- |\n     | IDC  | -      | 40 (44.8 $\\pm$ 0.2%)      | -     |\n     | HaBa | 5 (33.4 $\\pm$ 0.4%)      | 45 (42.5 $\\pm$ 0.2%)      | 245 (47.0 $\\pm$ 0.2%)     |\n     | RTP  | 16 (34.0 $\\pm$ 0.4%) | 232 (42.9 $\\pm$ 0.7%)     | -     |\n     | Ours | 11 (**40.0 $\\pm$ 0.4%**)     | 62 (**45.9 $\\pm$ 0.3%**) | 100 (**49.1 $\\pm$ 0.2%**) |\n\n    As evident, while the number of synthetic images generated by our method is **not the highest** among all approaches, our outstanding performance clearly showcases the **high quality** of the synthetic images. This further emphasizes that our approach enhances performance by improving both the quality and quantity of the synthetic images. Moreover, it demonstrates the highly efficient reduction of spatial redundancy achieved by our method.\n\n3. **The generalization ability of the method is unknown such as the recurrent neural network.**\n\n   * We appreciate the reviewer's attention to the generalization abilities of our method. We have discussed and proved the generalization abilities of our method in _Section 3.2 (Lines 203 to 232)_, including cross-architecture performance, universality to matching objectives, and robustness to corruption. To gain a comprehensive understanding of the experimental results and delve into an in-depth analysis, we kindly refer the reviewer to _Section 3.2_ of the paper.\n   \n   * For the ablation study of our recurrent blocks, please kindly refer to _Table II in the supplementary material_ and the second question of reviewer 27jp."}, {"review_id": "Gt2PTA2vvJ", "reviewer": "Reviewer_27jp", "summary": "This paper introduces the insight of sparse coding into dataset distillation and proposes a sound method. This work can efficiently generate syn data by adopting the multi-head SCMs as the shared source of the syn images and using a recurrent model to generate the syn patches. It can cooperate with various previous matching methods. In extensive experiments, the proposed method shows auspicious results and superiority.", "questions": "1. If the image size is larger, what will happen?\n\n2. Some other possible choices of the SCM format and utilization? For future study.", "limitations": "Discussed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ Importing the sparse coding into the syn data is interesting and effective from the experiments. The analysis of the problem of overlooking the syn data sparsity itself is insightful and naturally leads to the proposed method.\n\n+ Good writing, easy to follow.\n\n+ Extensive experiments, ablations, and visualizations.\n\n+ Insightful and valuable discussions and analyses, especially the syn data property, clear formulation, storage analysis, and trade-off between quality and quantity.\n\n+ Fig III in the appendix clearly showcases the effects of the proposed method.", "weaknesses": "- More discussions about the relations between dataset distillation, sparse coding, and corset selection.\n\n- Though the recurrent model is sound according to the cost discussion. I still wonder about the performance of adopting a heavier model following the same idea of the proposed method.\n\n- Fig. 4: should be more self-contained, please explain the comparison and differences/similarities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer 27jp for the insightful and constructive comments and are glad that the reviewer finds our method novel and interesting. In response to the concerns raised, we will address them as follows:\n\n1. **More discussions about the relations between dataset distillation, sparse coding, and coreset selection.**\n\n   * Thanks for the suggestion. As mentioned in _Lines 303 to 305_ of our paper, previous research in sparse coding has primarily focused on compressing individual images [71, 72], with the primary objective of achieving high compression ratios [34] while minimizing perceptual distortion. In contrast, dataset distillation involves condensing informative information from the original dataset into a smaller synthetic dataset to enhance downstream training without explicitly considering perceptual distortion. However, it is worth noting that many techniques and theories in sparse coding can provide valuable inspiration for developing parameterization methods in dataset distillation.\n\n   * Coreset selection [a, b, c, d] aims to identify a representative subset of the original dataset that aligns with the objectives of dataset distillation. As shown in [e], coreset selection typically performs better when the storage budget is relatively large, while dataset distillation demonstrates superior performance under extremely limited storage budgets.\n\n   * We are glad to incorporate more discussions on sparse coding and coreset selection in the paper. Additionally, we have included the results of the coreset methods in the experimental comparison, as presented in _Table A in the submitted PDF of the global response_.\n\n2. **Though the recurrent model is sound according to the cost discussion. I still wonder about the performance of adopting a heavier model following the same idea of the proposed method.**\n\n   * Thanks for your interest in our model design. In _Table II in the supplementary material_, We compare the recurrent model (the last row of the table) and the heavier non-recurrent model (the penultimate row of the table). Further discussion on this comparison can be found in _Lines 125 to 127 of the supplementary material_. In short, a heavier model leads to a notable escalation in the number of parameters (from 305k to 543k) while yielding only a marginal improvement in accuracy (from 40.0% to 40.3%). This experiment provides evidence supporting the efficiency of our recurrent design.\n\n3. **Fig. 4: should be more self-contained, please explain the comparison and differences/similarities.**\n\n   * Thanks for the suggestion. As mentioned in _Lines 248 to 253_, the purpose of Figure 4 is to demonstrate that the process of sparsification does not lead to a significant loss of essential information. Consequently, the two images (before and after sparsification) appear highly similar. To enhance clarity, we will update the caption of Figure 4 to include this conclusion.\n\n4. **If the image size is larger, what will happen?**\n\n   * Thanks for the question. As stated in _Line 68_, our method is designed to efficiently handle high-resolution datasets. However, it is worth noting that previous methods did not specifically perform experiments on larger image sizes, with a maximum size of 128x128. Therefore, conducting a direct comparison with these methods on image sizes of 256x256 is not feasible.\n\n   * Following your suggestion, we performed experimental investigations on both the distribution matching baseline [17] and our method, using the ImageNette dataset with image sizes of 128x128 and 256x256. In line with the practices of previous methods that handle higher resolution images with deeper networks, we increased the depth of the ConvNet to 6 for the 256x256 image size:\n\n     ||128 x 128|256 x 256|Gain|\n     |-|-|-|-|\n     |Baseline|28.6 $\\pm$ 0.6%|29.5 $\\pm$ 1.1%|+0.9%|\n     |Ours|53.5 $\\pm$ 1.2%|57.7 $\\pm$ 0.9%|+4.2%|\n     |Gain|+24.9%|+28.2%|+3.3%|\n     \n        The results demonstrate that our method achieves more substantial improvements when applied to higher-resolution images. These findings will be thoroughly incorporated into the paper, as they contribute to the expanding body of evidence supporting the effectiveness of our method on high-resolution image datasets.\n        \n5. **Some other possible choices of the SCM format and utilization? For future study.**\n\n   * Thanks for the suggestion. Several other efficient sparse matrix storage formats, such as compressed sparse row (CSR), compressed sparse column (CSC), diagonal (DIA), and block compressed sparse row (BSR), have the potential to save more storage compared to the naive coordinate (COO) format. However, DIA and BSR only achieve significant storage savings when the sparse matrix exhibits certain structural patterns. In the context of our work, the sparsity induced by the $l_1$ penalty is unstructured. As a result, we have opted for the simplest COO format to accommodate this unstructured sparsity. For CSR and CSC formats, we can convert directly from COO to them, which can bring further storage savings when the sparsity is not extremely high. We will consider other sparsity penalties and their corresponding efficient storage formats in our future work.\n\n----\n\n[a] Welling, Max. \"Herding dynamical weights to learn.\" Proceedings of the 26th Annual International Conference on Machine Learning. 2009.\n\n[b] Chen, Yutian, Max Welling, and Alex Smola. \"Super-samples from kernel herding.\" arXiv preprint arXiv:1203.3472 (2012).\n\n[c] Feldman, Dan, Matthew Faulkner, and Andreas Krause. \"Scalable training of mixture models via coresets.\" Advances in neural information processing systems 24 (2011).\n\n[d] Sylvestre-Alvise Rebuffi et al. “icarl: Incremental classifier and representation learning”. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.\n\n[e] Cui, Justin, et al. \"DC-BENCH: Dataset condensation benchmark.\" Advances in Neural Information Processing Systems, 2022."}]

Author rebuttal:
We would like to appreciate all the reviewers for their time and effort in the review process. Overall, we are pleased that the reviewers recognize the novelty (reviewer 27jp, yNYq, 7J1t), impressive experimental results (reviewer kT2g, 7J1t), and clear presentation (reviewer 27jp, yNYq, 7J1t) of this work. Please refer to the attached PDF for tables and figures.

Meta-review: The manuscript has been reviewed by four reviewers, all of whom recommended the acceptance of the manuscript.

Essentially, the reviewers found the proposed approach interesting, the paper well organized, and the experiments to be sufficient.

As such, there is no basis to overturn the consensus of the reviewers. The AC recommends acceptance.

Congrats!
Decision: Accept (poster)