Title: Prioritize Alignment in Dataset Distillation
Venue: NeurIPS 2024
Accepted: no
Keywords: dataset distillation

efficient learning
Abstract: Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation (PAD), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, PAD achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance. The code and distilled datasets will be made public.

Abstract

Dataset Distillation aims to compress a large dataset into a significantly more compact, synthetic one without compromising the performance of the trained models. To achieve this, existing methods use the agent model to extract information from the target dataset and embed it into the distilled dataset. Consequently, the quality of extracted and embedded information determines the quality of the distilled dataset. In this work, we find that existing methods introduce misaligned information in both information extraction and embedding stages. To alleviate this, we propose Prioritize Alignment in Dataset Distillation ( PAD ), which aligns information from the following two perspectives. 1) We prune the target dataset according to the compressing ratio to filter the information that can be extracted by the agent model. 2) We use only deep layers of the agent model to perform the distillation to avoid excessively introducing low-level information. This simple strategy effectively filters out misaligned information and brings non-trivial improvement for mainstream matching-based distillation algorithms. Furthermore, built on trajectory matching, PAD achieves remarkable improvements on various benchmarks, achieving state-of-the-art performance. The code and distilled datasets will be made public.

1 Introduction

Dataset Distillation (DD) [43] aims to compress a large dataset into a small synthetic dataset that preserves important features for models to achieve comparable performances. Ever since being introduced, DD has gained a lot of attention because of its wide applications in practical fields such as privacy preservation [5, 44], continual learning [28, 35], and neural architecture search [12, 32].

Recently, matching-based methods [46, 42, 6] have achieved promising performance in distilling high-quality synthetic datasets. Generally, the process of these methods can be summarized into two steps: (1) Information Extraction : an agent model is used to extract important information from the target dataset by recording various metrics such as gradients [49], distributions [48], and training trajectories [1], (2) Information Embedding : the synthetic samples are optimized to incorporate the extracted information, which is achieved by minimizing the differences between the same metric calculated on the synthetic data and the one recorded in the previous step.

In this work, we first reveal both steps will introduce misaligned information, which is redundant and potentially detrimental to the quality of the synthetic data. Then, by analyzing the cause of this misalignment, we propose alleviating this problem through the following two perspectives.

Typically, in the Information Extraction step, most distillation methods allow the agent model to 34

see all samples in the target dataset. This means information extracted by the agent model comes 35

from samples with various difficulties (see Figure 1(a)). However, according to previous study 36

.

37

38

+

100% easy

100% hard

for all IPCs

IPC

100% easy

+

100% easy

50% hard

+

50% easy

100% hard

small

medium

large

…

…

previous methods

ours

(a) Data used for distillation

IPC

small

medium

large

discard

discard

discard

agent model

shallow layer

deep layer

distill

distill

more

semantical

information

more

low-level

information

(b) Parameters used for distillation

[10], information related to easy samples is only needed when the compression ratio is high. This misalignment leads to the sub-optimal of the distillation performance.

To alleviate the above issue, we first use data selection methods to measure the difficulty of each 39 sample in the target dataset. Then, during the distillation, a data scheduler is employed to ensure only 40 data whose difficulty is aligned with the compression ratio is available for the agent model. 41

In the Information Embedding step, most distillation methods except DM [48] choose to use all 42 parameters of the agent model to perform the distillation. Intuitively, this will ensure the information 43 extracted by the agent model is fully utilized. However, we find shallow layer parameters of the 44 model can only provide low-quality, basic signals, which are redundant for dataset distillation in 45 most cases. Conversely, performing the distillation with only parameters from deep layers will yield 46 high-quality synthetic samples. We attribute this contradiction to the fact that deeper layers in DNNs 47 tend to learn higher-level representations of input data [27, 37]. 48

Based on our findings, to avoid embedding misaligned information in the Information Embedding step, 49 we propose to use only parameters from deeper layers of the agent model to perform distillation, as 50 illustrated in Figure 1(b). This simple change brings significant performance improvement, showing 51 its effectiveness in aligning information. 52

Through experiments, we validate that our two-step alignment strategy is effective for distillation 53 methods based on matching gradients [49], distributions [48], and trajectories [1]. Moreover, by 54 applying our alignment strategy on trajectory matching [1, 10], we propose our novel method named 55 Prioritize Alignment in Dataset Distillation (PAD). After conducting comprehensive evaluation 56 experiments, we show PAD achieves state-of-the-art (SOTA) performance. 57

58

2 Misaligned Information in Dataset Distillation

Generally, we can summarize the distillation process of matching-based methods into the following 59 two steps: (1) Information Extraction : use an agent model to extract essential information from the 60 target dataset, realized by recording metrics such as gradients [49], distributions [48], and training 61 trajectories [1], (2) Information Embedding : the synthetic samples are optimized to incorporate the 62 extracted information, realized by minimizing the differences between the same metric calculated on 63 the synthetic data and the one recorded in the first step. 64

65

66

67

10

15

20

25

Ratio (%)

16

17

18

19

Acc (%)

Remove easy

samples

Remove hard

samples

ipc500

ipc1

baseline

(a) Matching gradients

69

70

71

10

15

20

25

Ratio (%)

25

26

27

28

Acc (%)

(b) Matching distributions

82

83

10

15

20

25

Ratio (%)

45

46

47

Acc (%)

(c) Matching trajectories

65

66

0

25

50

75

Ratio (%)

27

28

29

Acc (%)

ipc500

ipc10

baseline

(a) Matching gradients

67

68

0

25

50

75

Ratio (%)

27

28

29

Acc (%)

(b) Matching distributions

83

84

0

25

50

75

Ratio (%)

65

66

67

Acc (%)

(c) Matching trajectories

In this section, through analyses and experimental verification, we show the above two steps both 65 will introduce misaligned information to the synthetic data. 66

67

2.1 Misaligned Information Extracted by Agent Models

In the information extraction step, an agent model is employed to extract information from the target 68

dataset. Generally, most existing methods [1, 6, 49, 46] allow the agent model to see the full dataset. 69 This implies that the information extracted by the agent model originates from samples with diverse 70 levels of difficulty. However, the expected difficulty of distilled information varies with changes in 71 IPC: smaller IPCs prefer easier information while larger IPCs should distill harder one [10]. 72

To verify if this misalignment will influence the quality of synthetic data, we perform the distillation 73 where hard/easy samples of target dataset are removed with various ratios. As the results reported in 74 Figure 2, pruning unaligned data points is beneficial for all matching-based methods. This proves the 75 misalignment indeed will influence the distillation performance and can be alleviated by filtering out 76 misaligned data from the target dataset. 77

2.2 Misaligned Information Embedded by Metric Matching 78

Most existing methods use all parameters of the agent model to compute the metric used for matching. 79

As can be observed in Figure 3, it is evident that across all matching-based methods, the removal 84 of shallow layer parameters consistently enhances performance, regardless of the IPC setting. This 85 proves employing over-shallow layer parameters to perform the distillation will introduce misaligned 86 information to the synthetic data, compromising the quality of distilled data. 87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

3 Method

To alleviate the information misalignment issue, based on trajectory matching (TM) [1, 10], we propose Prioritizing Alignment in Dataset Distillation (PAD). PAD can also be applied to methods based on matching gradients [49] and distributions [48], which are introduced in Appendix A.1.

3.1 Preliminary of Trajectory Matching

Following the two-step procedure, to extract information, TM-based methods [1, 10] first train agent models on the real dataset D R and record the changes of the parameters. Specifically, let { θ ∗ t } N 0 be an expert trajectory, which is a parameter sequence recorded during the training of agent model. At each iteration of trajectory matching, θ ∗ t and θ ∗ t + M are randomly selected from expert trajectories as the start and target parameters.

To embed the information into the synthetic data, TM methods minimize the distance between the expert trajectory and the student trajectory. Let ˆ θ t denote the parameters of the student agent model trained on synthetic dataset D S at timestep t . The student trajectory progresses by doing gradient descent on the cross-entropy loss l for N steps:

Finally, the synthetic data is optimized by minimizing the distance metric, which is formulated as:

3.2 Filtering Information Extraction

In section 2.1, we show using data selection to filter out unmatched samples could alleviate the misalignment caused in Information Extraction step. According to previous work [10], TM-based methods prefer easy information and choose to match only early trajectories when IPC is small. Conversely, hard information is preferred by high IPCs and they match only late trajectories. Hence, we should use easy samples to train early trajectories, while late trajectories should be trained with hard samples. To realize this efficiently, we first use the data selection method to measure the difficulty of samples contained in the target dataset. Then, during training expert trajectories, a scheduler is implemented to gradually incorporate hard samples into the training set while excluding easier ones from it.

Difficulty Scoring Function Identifying the difficulty of data for DNNs to learn has been well studied in data selection area [29, 17, 16, 40]. For simplicity consideration, we use Error L2-Norm (EL2N) score [33] as the metric to evaluate the difficulty of training examples (other metrics can also be chosen, see Section 4.3.2). Specifically, let x and y denote a data point and its label, respectively. Then, the EL2N score can be calculated by:

where p ( w t , x ) = σ ( f ( w t , x )) is the output of a model f at training step t transformed into a probability distribution. In consistent with [40], samples with higher EL2N scores are considered as harder samples in this paper.

Scheduler The scheduler can be divided into the following stages. Firstly, the hardest samples are 121 removed from the training set, ensuring that it exclusively comprises data meeting a predetermined 122 initial ratio (IR). Then, during training expert trajectories, samples are gradually added to the training 123 set in order of increasing difficulty. After incorporating all the data into the training set, the scheduler 124 will begin to remove easy samples from the target dataset. Unlike the gradual progression involved in 125 adding data, the action of reducing data is completed in a single operation, since now the model has 126 been trained on simple samples for a sufficient time. 127

128

3.3 Filtering Information Embedding

To filter out misaligned information introduced by matching shallow-layer parameters, we propose 129 to add a parameter selection module that masks out part of shallow layers for metric computation. 130 Specifically, parameters of an agent network can be represented as a flattened array of length L that 131 stores weights of agent models ordered from shallow to deep layers (parameters within the same 132 layer are sorted in default order). The parameter selection sets a threshold ratio α such that the first 133 k = L · α parameters are not used for distillation. Then the parameters used for matching can now be 134 formulated as: 135

In practice, the ratio α should vary with the change of IPC. For smaller IPCs, it is necessary to 136 incorporate basic information thus α should be lower. Conversely, basic information is redundant in 137 larger IPC cases, so α should be higher accordingly. 138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

4 Experiments

4.1 Settings

We compare PAD with several prominent dataset distillation methods, which can be divided into two categories: matching-based approaches including DC [49], DM [48], DSA [47], CAFE [42], MTT [1], FTD [6], DATM [10], TESLA [4], and kernel-based approaches including KIP [31], FRePo [50], RCIG [26]. The assessment is conducted on widely recognized datasets: CIFAR-10, CIFAR-100[18], and Tiny ImageNet [20]. We implemented our method based on DATM [10]. In both the distillation and evaluation phases, we apply the standard set of differentiable augmentations commonly used in previous studies [1, 6, 10]. By default, networks are constructed with instance normalization unless explicitly labeled with "-BN," indicating batch normalization (e.g., ConvNet-BN). For CIFAR-10 and CIFAR-100, distillation is typically performed using a 3-layer ConvNet, while Tiny ImageNet requires a 4-layer ConvNet. Cross-architecture experiments also utilize LeNet [21], AlexNet [19], VGG11 [39], and ResNet18 [11]. More details can be found in the appendix.

4.2 Main Results

CIFAR and Tiny ImageNet We conduct comprehensive experiments to compare the performance 153 of our method with previous works. As the results presented in Table 1, PAD outperforms previous 154 matching-based methods on three datasets except for the case when IPC=1. When compared with 155 kernel-based methods which use a larger network to perform the distillation, our technique exhibits 156 superior performance in most cases, particularly when the compression ratio exceeds 1% . As can be 157 observed, PAD performs relatively better when IPC is high, suggesting our filtering out misaligned 158 information strategy becomes increasingly effective as IPC increases. 159

Table 3: (a) Cross-Architecture evaluation on CIFAR-100 IPC50. (b) Ablation studies on the modules of our method on CIFAR-10 IPC10. (c) Results of different sets of data selection hyper-parameters on CIFAR-10 IPC10.

Cross Architecture Generalization We evaluate the generalizability of our distilled data in both 160 low and high IPC cases. As results reported in Table 3(a), when IPC is small, our distilled data 161 outperforms the previous SOTA method DATM on ResNet and AlexNet while maintaining comparable 162 accuracy on VGG. This suggests that our distilled data on high compressing ratios generalizes well 163 across various unseen networks. Moreover, as reflected in Table 2, our distilled datasets on large IPCs 164 also have the best performance on most evaluated architectures, showing good generalizability in the 165 low compressing ratio case. 166

167

168

169

170

171

172

173

174

175

176

4.3 Ablation Study

To validate the effectiveness of each component of our method, we conducted ablation experiments on modules (section 4.3.1) and their hyper-parameter settings (section 4.3.2 and section 4.3.2).

4.3.1 Modules

Our method incorporates two separate modules to filter information extraction (FIEX) and information embedding (FIEM), respectively. To verify their isolated effectiveness, we conduct an ablation study by applying two modules individually. As depicted in Table 3(b), both FIEX and FIEM bring improvements, implying their efficacy. By applying these two modules, we are able to effectively remove unaligned information, improving the distillation performance.

4.3.2 Hyper-parameters of Filtering Information Extraction

Initial Ratio and Data Addition Epoch To filter the information learned by agent models, we 177 initialize the training set with only easy samples, and the size is determined by a certain ratio of 178 the total size. Then, we gradually add hard samples into the training set. In practice, we use two 179 hyper-parameters to control the addition process: the initial ratio (IR) of training data for training 180 set initialization and the end epoch of hard sample addition (AEE). These two parameters together 181 control the amount of data agent models can see at each epoch and the speed of adding hard samples. 182

(a) Using EL2N to measure the difficulty of samples has the best performance.

(b) As IPC increases, removing more shallow-layer parameters becomes more effective.

(c) Using layer depth to select parameters outperforms using matching loss.

(a) with 100% parameters

(b) with 75% parameters

(c) with 50% parameters

In Table 3(c), we show the distillation results where different hyper-parameters are utilized. In 183 general, a larger initial ratio and faster speed of addition bring better performances. Although the 184 distillation benefited more from learning simpler information when IPC is small [10], our findings 185 indicate that excessively removing difficult samples (e.g., more than a quarter) early in the training 186 phase can adversely affect the distilled data. This negative impact is likely due to the excessive 187 removal leading to distorted feature distributions within each category. On the other hand, reasonably 188 improving the speed of adding hard samples allows the agent model to achieve a more balanced 189 learning of information of varying difficulty across different stages. 190

191

192

193

194

195

196

Other Difficulty Scoring Functions Identifying the difficulty of data points is the key to filtering out misaligned information in the extraction step. Here, we compare the effect of using other difficulty-scoring functions to evaluate the difficulty of data. (1) prediction loss of a pre-trained ResNet. (2) uncertainty score [3]. (3) EL2N [33]. As can be observed in Table 4(a), EL2N performs the best across various IPCs; thus, we use it to measure how hard each data point is as default in our method. Note that this can also be replaced with a more advanced data selection algorithm.

197

198

199

200

201

202

203

204

205

206

207

208

4.3.3 Ratios of Parameter Selection

It is important to find a good balance between the percentage of shallow-layer parameters removed from matching and the loss of information. In Table 4(b), we show results obtained on different IPCs by discarding various ratios of shallow-layer parameters. The impact of removing varying proportions of shallow parameters on the distilled data and its relationship with changes in IPC is consistent with prior conclusions. For small IPCs, distilled data requires more low-level basic information. Thus, removing too many shallow-layer parameters causes a negative effect on the classification performance. By contrast, high-level semantic information is more important when it comes to large IPCs. With increasing ratios of shallow-layer parameters being discarded, we can ensure that low-level information is effectively filtered out from the distilled data.

5 Discussion

5.1 Distilled Images with Filtering Information Embedding

To see the concrete patterns brought by removing shallow-layer parameters to perform the trajectory 209 matching, we present distilled images obtained by discarding various ratios of shallow-layer parame210 ters in Figure 4. As can be observed in Figure 4(a), without removing any shallow-layer parameters 211

1

2

3

4

5

6

Layer

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

iter 0

iter 1000

iter 5000

Loss

(a) CIFAR-10 IPC1

1

2

3

4

5

6

Layer

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

iter 0

iter 1000

iter 5000

Loss

(b) CIFAR-10 IPC10

1

2

3

4

5

6

Layer

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Loss

iter 0

iter 1000

iter 5000

(c) CIFAR-10 IPC500

(a) Match shallow layers only

(b) Original

(c) Match deep layers only

中

to filter misaligned information, synthetic images are interspersed with substantial noises. These 212 noises often take the form of coarse and generic information, such as the overall color distribution 213 and edges in the image, which provides minimal utility for precise classification. 214

By contrast, images distilled by our enhanced methodology (see Figure 4(b) and Figure 4(c)), which 215 includes meticulous masking out shallow-layer parameters during trajectory matching according to the 216 compressing ratio, contain more fine-grained and smoother features. These images also encapsulate 217 a broader range of semantic information, which is crucial for helping the model make accurate 218 classifications. Moreover, we observe a clear trend: as the amount of the removed shallow-layer 219 parameters increases, the distilled images exhibit clearer and smoother features. 220

221

5.2 Rationale for Parameter Selection

In this section, we analyze from the perspective of trajectory matching why shallow-layer parameters 222 should be masked out. In Figure 5, we present the changes in trajectory matching loss across different 223 layers as the distillation progresses. Compared to the deep-layer parameters of the agent model, 224 a substantial number of shallow-layer parameters exhibit low loss values that fluctuate during the 225 matching process (see Figure 5). By contrast, the loss values of the deep layers are much higher but 226 consistently decrease as distillation continues. This suggests that matching shallow layers primarily 227 conveys low-level information that is readily captured by the synthetic data and quickly saturated. 228 Consequently, the excessive addition of such low-level information produces noise, reducing the 229 quality of distilled datasets. 230

For a concrete visualization, we provide distilled images resulting from using only shallow-layer 231 parameters or only deep-layer parameters to match trajectories in Figure 6. The coarse image features 232 depicted in Figure 6(a) further substantiate our analysis. 233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

5.3 Parameter Selection Strategy

In the previous section, we observed a positive correlation between the depth of the model layers and the magnitude of their trajectory-matching losses. Notably, the loss in the first layer of the ConvNet was higher compared to other shallow layers. Consequently, we further compared different parameter alignment strategies, specifically by sorting the parameters based on their matching losses and excluding a certain proportion of parameters with lower losses. Higher loss values indicate greater discrepancies in parameter weights; thus, continuing to match these parameters can inject more information into the synthetic data. As shown in Table 4(c), sorting by loss results in an improvement compared with no parameter alignment, but filtering based on parameter depth proves to be more effective.

6 Related Work

Introduced by [43], dataset distillation aims to synthesize a compact set of data that allows models to achieve similar test performances compared with the original dataset. Since then, a number of studies have explored various approaches. These methods can be divided into three types: kernel-based, matching-based, and using generative models [45].

Kernel-based methods are able to achieve closed-form solutions for the inner optimization [31] via kernel ridge regression with NTK [22]. FRePo [50] distills a compact dataset through neural feature regression and reduces the training cost.

Matching-based methods first use agent models to extract information from the target dataset by recording a specific metric [7, 23, 38, 24]. Representative works that design different metrics include DC [49] that matches gradients, DM [48] that matches distributions, and MTT [1] that matches training trajectories. Then, the distilled dataset is optimized by minimizing the matched distance between the metric computed on synthetic data and the record one from the previous step. Following this workflow, many works have been proposed to improve the efficacy of the distilled dataset. For example, CAFE [42] preserves the real feature distribution and the discriminative power of the synthetic data and achieves prominent generalization ability across various architectures. DREAM [25] employs K-Means to select representative samples for distillation and improves the distillation efficiency. DATM [10] proposes to match early trajectories for small IPCs and late trajectories for large IPCs, achieving SOTA performances on several benchmarks. Moreover, new metrics such as spatial attention maps [36, 15] have also been introduced and achieved promising performance in distilling large-scale datasets.

Generative models such as GANs [8, 13, 14, 41] and diffusion models [34, 30, 9] can also be used to distill high quality datasets. DiM [41] uses deep generative models to store information of the target dataset. GLaD [2] transfers synthetic data optimization from the pixel space to the latent space by employing deep generative priors. It enhances the generalizability of previous distillation methods.

7 Conclusion

In this work, we find a limitation of existing Dataset Distillation methods in that they will introduce misaligned information to the distilled datasets. To alleviate this, we propose PAD, which incorporates two modules to filter out misaligned information. For information extraction, PAD prunes the target dataset based on sample difficulty for different IPCs so that only information with aligned difficulty is extracted by the agent model. For information embedding, PAD discards part of shallow-layer parameters to avoid injecting low-level basic information into the synthetic data. PAD achieves SOTA performance on various benchmarks. Moreover, we show PAD can also be applied to methods based on matching gradients and distribution, bringing remarkable improvements across various IPC settings.

Limitations Our alignment strategy could also be applied to methods based on matching gradients and distributions (see Appendix A.1). However, due to the limitation of computing resources, for methods based on matching distributions and gradients, we have only validated our method's effectiveness on DM [48] and DC [49] (see Table 5 and Table 6).

283

References

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. 284 Dataset distillation by matching training trajectories. 2022 IEEE/CVF Conference on Computer 285 Vision and Pattern Recognition (CVPR) , pages 10708-10717, 2022. 286

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. 287 Generalizing dataset distillation via deep generative prior. 2023 IEEE/CVF Conference on 288 Computer Vision and Pattern Recognition (CVPR) , pages 3739-3748, 2023. 289

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, 290 Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for 291 deep learning. arXiv preprint arXiv:1906.11829 , 2019. 292

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet293 1k with constant memory. In International Conference on Machine Learning , 2022. 294

295

296

Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? ArXiv , abs/2206.00240, 2022.

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

Jiawei Du, Yiding Jiang, Vincent Y. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3749-3758, 2022.

Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. ArXiv , abs/2311.01570, 2023.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63:139 - 144, 2014.

Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. ArXiv , abs/2311.15529, 2023.

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. ArXiv , abs/2310.05773, 2023.

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2015.

Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2018.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4396-4405, 2018.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8107-8116, 2019.

Samir Khaki, Ahmad Sajedi, Kai Wang, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. 323 Plataniotis. Atom: Attention mixer for efficient dataset distillation, 2024. 324

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, and 325 Rishabh K. Iyer. Grad-match: Gradient matching based data subset selection for efficient 326 deep model training. In International Conference on Machine Learning , 2021. 327

Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Rishabh Iyer Univer328 sity of Texas at Dallas, Indian Institute of Technology Bombay Institution One, and IN Two. 329 Glister: Generalization based data subset selection for efficient and robust learning. In AAAI 330 Conference on Artificial Intelligence , 2020. 331

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 332

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep 333 convolutional neural networks. Communications of the ACM , 60:84 - 90, 2012. 334

Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. 335

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning 336 applied to document recognition. Proc. IEEE , 86:2278-2324, 1998. 337

338

339

340

341

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Narain Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Journal of Statistical Mechanics: Theory and Experiment , 2020, 2019.

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sung-Hoon Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , 2022.

Haoyang Liu, Tiancheng Xing, Luwei Li, Vibhu Dalal, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. ArXiv , abs/2311.18531, 2023.

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Hua Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 17268-17278, 2023.

Noel Loo, Ramin M. Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. ArXiv , abs/2302.06755, 2023.

Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision , 120:233 - 255, 2016.

Wojciech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning on synthetic data. In CVPR Workshop , 2020.

Baharan Mirzasoleiman, Jeff A. Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning , 2019.

Brian B. Moser, Federico Raue, Sebastián M. Palacio, Stanislav Frolov, and Andreas Dengel. Latent dataset distillation with diffusion models. ArXiv , abs/2403.03881, 2024.

Timothy Nguyen, Zhourung Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. ArXiv , abs/2011.00050, 2020.

Ramakanth Pasunuru and Mohit Bansal. Continual and multi-task architecture search. ArXiv , abs/1906.05226, 2019.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. In Neural Information Processing Systems , 2021.

Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10674-10685, 2021.

Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. In International Workshop on Continual Semi-Supervised Learning , 2021.

Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 17097-17107, October 2023.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi 375 Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based 376 localization. International Journal of Computer Vision , 128:336 - 359, 2016. 377

378

379

380

Seung-Jae Shin, Heesun Bae, DongHyeok Shin, Weonyoung Joo, and Il-Chul Moon. Losscurvature matching for dataset selection and condensation. In International Conference on Artificial Intelligence and Statistics , 2023.

381

382

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR , abs/1409.1556, 2014.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond 383 neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information 384 Processing Systems , 35:19523-19536, 2022. 385

Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Hua Zhu, Wei Jiang, and Yang You. Dim: 386 Distilling dataset into generative model. ArXiv , abs/2303.04707, 2023. 387

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Hua Zhu, Shuo Yang, Shuo Wang, Guan Huang, 388 Hakan Bilen, Xinchao Wang, and Yang You. Cafe learning to condense dataset by aligning 389 features. 2022. 390

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 391 2020. 392

Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via 393 contrastive representation ensemble. ArXiv , abs/2302.08888, 2023. 394

Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. 395 IEEE Transactions on Pattern Analysis and Machine Intelligence , 46:150-170, 2023. 396

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In 397 International Conference on Machine Learning , 2021. 398

399

400

Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning , 2021.

401

402

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 6503-6512, 2021.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. 403 ArXiv , abs/2006.05929, 2020. 404

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature 405 regression. ArXiv , abs/2206.00719, 2022. 406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

(a) Removing various ratios of hard/easy samples improves DC on small/large IPCs.

(b) Removing various ratios of hard/easy samples improves DM on small/large IPCs.

(a) Matching gradients from deep-layer parameters leads to improvements.

(b) Matching distributions from deep-layer parameters leads to improvements.

Table 6: Results of filtering information embedding by masking out shallow-layer parameters for metric computation in DC(a) and DM(b) on CIFAR-10.

A Appendix

A.1 Filtering Misaligned Information in DC and DM

Although PAD is implemented based on trajectory matching methods, we also test our proposed data alignment and parameter alignment on gradient matching and distribution matching. The performances of enhanced DC and DM with each of the two modules are reported in Table 5 and Tabl 6, respectively. We provide details of how we integrate these two modules into gradient matching and distribution matching in the following sections.

Gradient Matching We use the official implementation 1 of DC [49]. In the Information Extraction step, DC uses an agent model to calculate the gradients after being trained on the target dataset. We employ filter misaligned information in this step as follows: When IPC is small, a certain ratio of hard samples is removed from the target dataset so that the recorded gradients only contain simple information. Conversely, when IPC becomes large, we remove easy samples instead.

In the Information Embedding step, DC optimizes the synthetic data by back-propagating on the gradient matching loss. The loss is computed by summing the differences in gradients between each pair of model parameters. Thus, we apply parameter selection by discarding a certain ratio of parameters in the shallow layers.

Distribution Matching We use the official implementation of DM [48], which can be accessed via the same link as DC. In the Information Extraction step, DM uses an agent model to generate embeddings of input images from the target dataset. Similarly, filtering information extraction is applied by removing hard samples for small IPCs and easy samples for large IPCs.

In the Information Embedding step, since DM only uses the output of the last layer to match distributions, we modify the implementation of the network such that outputs of each layer in the model are returned by the forward function. Then, we perform parameter selection following the same practice as before.

A.2 Experiment Settings

We use DATM [10] as the backbone TM algorithm and our proposed PAD is built upon. Thus, our 432 configurations for distillation, evaluation, and network are consistent with DATM. 433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

Distillation. We conduct the distillation process for 10,000 iterations to ensure full convergence of the optimization. By default, ZCA whitening is applied in all the experiments.

Evaluation. We train a randomly initialized network on the distilled dataset and evaluate its performance on the entire validation set of the original dataset. Following DATM [10], the evaluation networks are trained for 1000 epochs to ensure full optimization convergence. For fairness, the experimental results of previous distillation methods in both low and high IPC settings are sourced from [10].

Network. We employ a range of networks to assess the generalizability of our distilled datasets. For scaling ResNet, LeNet, and AlexNet to Tiny-ImageNet, we modify the stride of their initial convolutional layer from 1 to 2. In the case of VGG, we adjust the stride of its final max pooling layer from 1 to 2. The MLP used in our evaluations features a single hidden layer with 128 units.

Hyper-parameters. Hyper-parameters of our experiments on CIFAR-10, CIFAR-100, and TinyImageNet are reported in Table 7. Hyper-parameters can be divided into three parts including data alignment (DA), parameter alignment (PA) and trajectory matching (TM). Soft labels are applied in all experiments , we set its momentum to 0.9.

Compute resources. Our experiments are run on 4 NVIDIA A100 GPUs, each with 80 GB of memory. The amount of GPU memory needed is mainly determined by the batch size of synthetic data and the number of steps that the agment model is trained on synthetic data. To reduce the GPU usage when IPC is large, one can apply TESLA [4] or simply reducing the synthetic steps N or the synthetic batch size. However, the decrement of hyper-parameters shown in Table 7 could result in performance degradation.

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claim does accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations at the end of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We didn't present any theoretical results in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All hyper-parameters and computing resources needed for experiments are listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

Answer: [Yes]

Justification: Our code will be made public.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our experiment results are reflected by classification accuracy.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All details are listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work doesn't have societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is not applicable to our work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all code, data, and previous works in a proper manner.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

Answer: [NA]

Justification: This is not applicable to our work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is not applicable to our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is not applicable to our work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper makes the key observation that existing dataset distillation methods often introduce misaligned information during both the extraction and embedding stages, which leads to suboptimal performances. In response to this observation, the authors propose a method called Prioritize Alignment in Dataset Distillation (PAD), which aims to filter out misaligned information through two steps: 1). Pruning the target dataset based on sample difficulty according to the compression ratio, and 2) using only deep layers of the agent model during distillation to avoid encoding low-level, redundant information.\n\nSTRENGTHS:\n1. This paper synthesize a universal framework for existing data distillation methods by abstracting those methods into two steps: 1). Information extraction and 2) information embedding. Furthermore, it identifies a common theme of information misalignment in both steps. This observation enhances the understanding of current limitations as well as provides a clear direction for future research. \n2. The method presented in this paper effectively combines known conclusion from two distinct areas of research: data selection and representation learning. By leveraging established principles from both domains, the provide improvements to existing data distillation methods. More importantly, their analysis builds on developing an understanding of data distillation and the underlying mechanism: 1) small datasets require simple data 2) large datasets do not benefit from low-level information/features.\n\nWEAKNESSES:\n1. The proposed method’s filtering of information extraction is supported experiments shown in Figure 2. However, in practice, the method introduces two sets of hyper parameters - initial ratio and data addition epoch. The sensitivity to these hyperparameters (especially AEE shown in Table c) relative to the incremental performance gain presents a challenge, as running AEE can be complex and time-consuming (involves retraining the agent). This sensitivity and the associated tuning complexity could hinder its practical adoption in larger-scale datasets. \n2. The proposed method is adapted from the DATM framework with modifications at enhancing information alignment. However, the performance improves over DATM (shown in Table 1 and in the cross-architecture generalization results in Table 3) are not significant. This marginal improvement raises concerns about the practical value of the proposed changes, as they may not justify the added complexity.\n\nQUESTIONS:\n1. How is data selection applied in Figure 2 (b)? In the original work, DM[1] uses untrained networks (random initialized network to embed features). In your experiment. are you using untrained agent models but applying data selection score (EL2N) from a differently pre-trained model? Overall, I am confused about how the EL2N is generated? Are you replicating the score from [2] and apply it in all your methods or are you recomputing it based on different distillation architectures?  \n2. Can the same analysis not applied to more recent SOTA methods such as SRe2L[3], which also involved a component similar to DM [1]? Would the same information alignments trick works as well? \n\n[1] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching [arXiv:2110.04181](https://arxiv.org/abs/2110.04181)\n\n[2] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: 364 Finding important examples early in training. [arXiv:2107.07075](https://arxiv.org/abs/2107.07075)\n\n[3] Zeyuan Yin, Erix Xing, and Zhiqiang Shen. Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective [arXiv:2306.13092](https://arxiv.org/abs/2306.13092)\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper proposes to study the information misalignment problem in dataset distillation.
It proposes two basic pruning strategies: (1) learn the synthetic data with easy real samples first, and gradually change to harder samples, and (2) only match deep layers of the network during trajectory matching. The proposed method could enhance the current method in a wide range of dataset settings.

STRENGTHS:
- The paper is well-written.
- The motivation is reasonable.
- The idea of using a scheduler to dynamically adjust the real sample difficulty is smart.

WEAKNESSES:
1. The experimental observations to support the two strategies (Information Extraction and Information Embedding) involving Figure 2 and 3, is not sufficient. Experiments on a wider range of datasets and more pruning ratios could rationalize the method. And a comparison of discarding deeper-layer parameters is missing.
2. The wide existence of difficulty-aware dataset distillation could **potentially** weaken the contribution. Some discussion is appreciable:
```
[1] Prune Then Distill: Dataset Distillation with Importance Sampling
[2] Distill Gold from Massive Ores: Efficient Dataset Distillation via Critical Samples Selection
[3] (DATM) Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching
[4] On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm
```

QUESTIONS:
1. Line 125-127: is there any experimental comparison to support the conclusion?
2. Is it possible to combine PAD with DATM, since they do not conflict?
3. The ablation study in Table 3(b) is weak. Conducting the experiments on other datasets may help.

LIMITATIONS:
The authors have discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors claim that existing data distillation methods introduce misaligned information, so they propose Prioritize Alignment in Dataset Distillation (PAD). PAD prunes the target dataset and uses only deep layers of the agent model to perform the distillation, achieving state-of-the-art performance.\n\nSTRENGTHS:\n1. The paper is somewhat well-written and mostly easy to follow. And the tables/figures are well-demonstrated.\n\n2. The authors analyze the misaligned information from two perspectives and propose method. \n\n3. PAD achieves improvements on various benchmarks, achieving state-of-the-art performance.\n\nWEAKNESSES:\nThe performance gains brought by the method proposed by the authors are subtle and limited, potentially attributable to other explanations. For instance, as mentioned in [1], discarding original data in certain ways, or even randomly, can yield minor performance improvements under different IPC conditions. Or tricks mentioned in [2]. \n\nThe trend changes in Figure 2 are not pronounced, and there are even instances where the trends contradict the explanations. Could additional test ratios or test IPCs be included to validate the findings?\n\n[1] Distill Gold from Massive Ores: Efficient Dataset Distillation via Critical Samples Selection.\n[2] Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality\n\nQUESTIONS:\nIn Figure 2, Figure 3, and Table 5, the baselines do not all align with those in Table 1 or the results presented in the original paper. Is there an explanation I have missed? Please clarify this discrepancy, as I am unable to evaluate the correctness of \"Misaligned Information Extracted\" without understanding the reason for this variation.\n\nLIMITATIONS:\nDue to the limitation of computing resources, the authors only validated their method’s effectiveness on DATM, DM, and DC.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}