Title: DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation
Venue: NeurIPS 2024
Accepted: no
Keywords: Dataset Distillation

Diversity-driven EarlyLate Training
Abstract: Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new EarlyLate training scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conducted extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our empirical results demonstrate that the proposed approach significantly improves over previous state-of-the-art methods under various IPCs.

1

Abstract

Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-toglobal matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe 2 L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new EarlyLate training scheme to enhance the diversity of images in batch-toglobal matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conducted extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our empirical results demonstrate that the proposed approach significantly improves over previous state-of-the-art methods under various IPCs 1 .

1 Introduction

In the era of large models and large datasets, dataset 23 distillation has emerged as a crucial strategy to en24 hance training efficiency and make AI technologies 25 more accessible and affordable for the general public. 26 Previous approaches [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] 27 primarily employ a batch-to-batch matching tech28 nique, where information like features, gradients, 29 and trajectories from a local original data batch are 30 used to supervise and train a corresponding batch 31 of generated data. This method's strength lies in its 32 ability to capture fine-grained information from the 33 original data, as each batch's supervision signals vary. 34 However, the downside is the necessity to repeatedly 35

1 represents 𝑛 I mages P er C lass for the distilled dataset.

Our DELT Distillation

Prior

Ours

Computation

!×#

# +

# - ./ …+./

IPC

IPC

IPC

1:

N

IPC

1:

N

batch-to-global

matching

batch-to-global

matching

…

…

…

…

Full

Full

IPC

1

IPC

2

IPC

N

IPC

1

IPC

2

IPC

N

#

!

#

"

#

#

#

#

#

bird

bird

1

Early-optimized image

N

Late-optimized image

Our DELT generated images

Plot curves of lr

IPC=1

适⽤，只能是初始化

起作⽤

(1) SRe

2

L

(3) Ours [1

×

1 initialized]

(2) CDA

Class Index

input both original and generated data for each training iteration, which significantly increases memory 36 usage and computational costs. Recently, a new decoupled method [11, 12, 13] has been proposed 37 to separate the model training and data synthesis, also it leverages the batch-to-global matching to 38 avoid inputting original data during distilled data generation. This solution has demonstrated great 39 advantage on large-scale datasets like ImageNet-1K [11, 14] and ImageNet-21K [12]. However, as 40 shown in Fig. 2 right subfigure, a significant limitation of this method is its strategy of synthesizing 41 each data point individually, where supervision is repetitively applied across various synthetic images. 42 For instance, SRe 2 L[11] utilizes globally-counted layer-wise running means and variances from the 43 pre-trained model for supervising different intra-class image synthesis. This methodology results in a 44 pronounced lack of diversity within the same category of generated images. 45

To address this issue, previous studies such as G-VBSM [14] and RDED [15] have been conducted. 46 Specifically, G-VBSM [14] introduces a framework that utilizes a diverse set of local-match-global 47 matching signals derived from multiple backbones and statistical metrics, offering more precise and 48 effective matching than the singular model. However, as the diversity of matching models grows, the 49 overall complexity of the framework also increases, thus diminishing its conciseness. RDED [15] 50 crops each original image into multiple patches and ranks these using realism scores generated by an 51 observer model. Then it amalgamates every four chosen patches from previous stage into a single new 52 image, maintaining the resolution of the original images, and produce IPC-numbered distilled images 53 for each class. While RDED is effective for selecting and combining data, it does not enhance or 54 optimize the visual content within the distilled dataset. Thus, the diversity and richness of information 55 it encapsulates largely dependent on the distribution of the original dataset. 56

Our solution, termed the EarlyLate training scheme, is straightforward and also orthogonal to 57 these prior methods: by initializing each image in the same category at a different starting point for 58 optimization, we ensure that the final optimized results vary across images. We also use teacher-ranked 59 real image patches to initialize the synthetic images. This prevents some images from being short60 optimized and ensures they provide sufficient information. As shown in Fig. 1 of the computation 61 comparison, our approach not only enhances intra-class diversity but also significantly reduces the 62 computational load of the training process. Specifically, while conventional training requires 𝑇 63 optimization iterations per image or batch, in our EarlyLate scheme, the first image undergoes 𝑇 1 64 iterations (where 𝑇 1 = 𝑇 ). Subsequent batches are processed with progressively fewer iterations, such 65 as 𝑇 2 ( 𝑇 2 = 𝑇 1 -RI 2 ) for the next set, and so forth. The iterations for the final batch are reduced 66 to RI which is 1 / 𝑗 of the standard count (where typically 𝑗 = 4 or 8), meaning the total number of 67 optimization iterations required is just about 2 / 3 of prior batch-to-global matching methods, such as 68 SRe 2 L and CDA. We further visualize the average cosine similarity between each sample of 50 IPCs 69 with the associated cluster centroid within the same class on ImageNet-1K, as shown in Fig. 2 left 70 subfigure, DELT shows significantly better diversity than other counterpart methods across all classes. 71

We perform extensive experiments on datasets of CIFAR-10, Tiny-ImageNet, ImageNet-1K and its 72 subsets. On ImageNet-1K, our proposed approach achieves 66.1% under IPC 50 with ResNet-101, 73 outperforming previous state-of-the-art RDED by 4.9%. On small-scale datasets of CIFAR-10, our 74 approach also obtains 2.5% and 19.2% improvement over RDED and SRe 2 L using ResNet-101. 75

76

77

78

Our main contributions in this work are as follows:

We propose a simple yet effective EarlyLate training scheme for dataset distillation to enhance the intra-class diversity of synthetic images from batch-to-global matching.

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

We demonstrate empirically that the proposed method can generate optimized images at different distances from their initializations, to enlarge informativeness among generations.

We conducted extensive experiments and ablations on various datasets across different scales to prove the effectiveness of the proposed approach 3 .

2 Related Work

Dataset Distillation. Dataset distillation or condensation [1] focuses on creating a compact yet representative subset from a large original dataset. This enables more efficient model training while maintaining the ability to evaluate on the original test data distribution and achieve satisfactory performance. Previous works [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] mainly designed how to better match the distribution between original data and generated data in a batch-to-batch manner, such as the distribution of features [6], gradients [2], or the model weight trajectories [4, 8]. The primary optimization method used is bi-level optimization [16, 17], which involves optimizing model parameters and updating images simultaneously. For instance, using gradient matching, the process can be formulated as to minimize the gradient distance:

where the function 𝐷 (· , ·) is defined as a distance metric such as MSE [18], 𝜃 denotes the model parameters, and ∇ 𝜃 ℓ (· ; 𝜃 ) represents the gradient, utilizing either the original dataset T or its synthetic version S . 𝑁 is the number of 𝑑 -dimensional synthetic data. During distillation, the synthetic dataset S and model 𝜃 are updated alternatively,

where 𝜆 and 𝜂 are learning rates designated for S and 𝜃 , respectively.

Batch-to-global matching used in [11, 14, 12, 13] tracks the distribution of BN statistics derived from the original dataset for the local batch synthetic data, the formulation can be:

where 𝑙 is the index of BN layer, 𝜇 𝑙 (S) and 𝜎 2 𝑙 (S) are mean and variance. BN RM 𝑙 and BN RV 𝑙 are running mean and running variance in the pre-trained model at 𝑙 -th layer, which are globally counted. Fig. 3 illustrates the difference of batch-to-batch and batch-to-global matching mechanisms, where 𝑏 represents a local batch in data T and S .

(1)

Batch-to-Batch

Matching

(2)

Batch-to-Global

Matching

Global mean,

var statistics

Batch mean, var

statistics

weight pretraining

weight sharing

1

3

Gradient/Trajectory/Feature, etc.

Gradient/Trajectory/Feature, etc.

ℒ

2

ℒ

ℒ

+,

0

3

0

3

Original

dataset

.

.

.

1

!

represents a local

batch in

#

2 Other loss like CE and R is not plotted in the figures and $ Moreover, for the recent advances of multi-stage dataset distillation methods, MDC [10] proposes to compress multiple condensation processes into a single one by including an adaptive subset loss on top of the basic condensation loss, so that to obtain datasets with multiple sizes. PDD [9] generates multiple small batches of synthetic images, each batch is conditioned on the accumulated data from previous batches. Unlike PDD, our current synthetic batch is independent with different operation iterations and not relevant to any previous batches. D3 [19] partitions large datasets into smaller subtasks and employs locally trained experts to distill each subset into distributions. These distilled distributions from the subtasks demonstrate effective generalization when applied to the entire task.

Initialization. Weight initialization [20, 21, 22, 23] is pivotal in training neural networks, significantly influencing their optimization process. Proper initialization is essential for ensuring model convergence and mitigating issues such as gradient vanishing. Recently, weight selection [24] introduces a strategy for initializing smaller models by selecting a subset of weights from a pretrained larger model. This

Teacher

Ranker

Diversity-driven IPC Concatenation

Scheduler

+

+

+

Concatenation node

I

mages

P

er

C

lass whose rank

CDA Gradient updates for

iterations

op Medium Selection Criteria

1

2

3

4

5

6

7

4 method facilitates the transfer of learned attributes from the pretrained weights, enhancing the smaller 124 model's performance. Weight subcloning [25] involves manipulating the pretrained model to derive a 125 correspondingly scaled-down version with equivalent initialization. This involves two main steps: 126 initially, it applies a neuron importance ranking to reduce the embedding dimension per layer within 127 the pretrained model. Subsequently, it eliminates blocks from the transformer model to align with the 128 layer count of the scaled-down network. 129

5

130

3

131

6

132

133

2

134

Unrolled version This work focuses on data initialization for generation processes. Few studies have examined this angle. While, PCA-K [26] appears to be the most relevant. It employs an initialization method that involves drawing samples from a distribution that accurately mirrors and is easily sampled from the training distribution. During training, it is possible to retrieve some details from the original image using the initial noisy sample, which at best provides a blurred representation of the original image.

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

3 Our Approach

Preliminaries. The objective of a regular dataset distillation task is to generate a compact synthetic dataset S = {( ˆ 𝒙 1 , ˆ 𝒚 1 ) , . . . , GLYPH<0> ˆ 𝒙 | S | , ˆ 𝒚 | S | GLYPH<1> } as a student dataset that captures a substantial amount of the information from a larger labeled dataset T = {( 𝒙 1 , 𝒚 1 ) , . . . , GLYPH<0> 𝒙 | T | , 𝒚 | T | GLYPH<1> } , which serves as the teacher dataset. Here, ˆ 𝒚 represents the soft label for the synthetic sample ˆ 𝒙 , and the size of S is much smaller than T , yet it retains the essential information of the original dataset T . The learning goal using this distilled dataset is to train a post-validation model with parameters 𝜽 :

where ℓ is a standard loss function such as soft cross-entropy and 𝜙 𝜽 S represents the model.

The primary aim of dataset distillation is to produce synthetic data that ensures minimal performance difference between models trained on the synthetic dataset S and those trained on the original dataset T using validation data 𝑉 . The optimization procedure for generating S is given by:

where ( 𝒙 𝑣𝑎𝑙 , 𝒚 𝑣𝑎𝑙 ) are the sample and label pairs in the validation set of the real dataset T . The learning task then focuses on the <data, label> pairs within S , maintaining a balanced representation of distilled data across each class.

Initialization. Previous dataset distillation methods [11, 14, 150 12] on large-scale datasets like ImageNet-1K and 21K employ 151 Gaussian noise by default for data initialization in the synthesis 152 phase. However, Gaussian noise is random and lacks any 153 semantic information. Intuitively, using real images provide a 154 more meaningful and structured starting point, and this struc155 tured start can lead to quicker convergence during optimization 156 because the initial data already contains useful features and 157 patterns that are closer to the target distribution, which further 158

2

3

5

1

4

Teacher

Ranker

1

2

3

4

5

3

2

4

IPC initialization

Random crop pool

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

enhances realism, quality, and generalization of the synthesized images. As shown in Fig. 2 right subfigure, our generated images exhibit both diversity and a high degree of realism in some cases.

Selection Criteria. Here, we introduce how to select real image patches to initialize the synthetic images. In our final syntheses, a significant fraction of our data has been subject to limited optimization iterations, making effective initialization crucial. A proper initialization also dramatically minimizes the overall computational load required for the updating on data. Prior approach [15] has demonstrated that choosing representative data patches from the original dataset without training can yield favorable performance without any additional training. Our observation, however, underscores that applying iterative refinement to original patches can lead to markedly improved results. As illustrated in Fig. 1, our selection criterion is based on a pretrained teacher model as a ranker, we calculate all patches' probabilities and sort them as the initialization pool. Then, we choose lowest, medium, or highest probability patches as the initialization for our optimization.

Diversity-driven IPC Concatenation Training. As shown in Fig. 4, to further emphasize diversity and avoid potential distribution bias from initialization, we optimize the initialized images starting from different points. The motivation behind this design is that different data samples require varying numbers of iterations to converge which is similar to the early stopping idea [27]. Importantly, as images become easier to predict with more updates by class labels, training primarily on easy data points can hinder model generalization. Therefore, our method enhances generalization by generating data samples with varying difficulty levels, acting as a regularizer by limiting the optimization process to a smaller volume of image pixel space. Previous work [28] studies how to perform early stopping training on different layers' weights of the model with progressive retraining to mitigate noisy labels. We are pioneering to study how to leverage early-late training when optimizing data. Moreover, we improve the efficiency of our approach by performing gradient updates in a single scan. Initially, we conduct a single gradient loop, continually introducing new data for distillation by concatenating them at different time stamps. Consequently, the 𝑀 batch receives the synthetic images of all preceding batches, IPC 0: 𝑀𝑘 -1, as final generations. This process can be simplified as follows:

where [ ˆ 𝒙 0 , ˆ 𝒙 1 , . . . , ˆ 𝒙 𝑀𝑘 -1 ] refers to the concatenation of the generated image. 𝑀 is the number of batches, 𝑘 is the number of generated images in each batch. We train these different batches at different starting points, each batch goes through a completed learning phase, but the total number of iterations varies. Then, the multiple IPCs of ˆ 𝒙 are concatenated into a simple batch. Because of its early-late training property, we refer to this simple training scheme as EarlyLate training.

Training Procedure. As illustrated in Fig. 4, our learning procedure is extremely simple using an incremental learning process: We split the total IPCs to be learned into multiple batches. The training begins with the first batch. Following a predefined number of iterations, the second batch commences its iterative training, and this process continues sequentially with subsequent batches. Batch-to-global matching algorithm [12] of Eq. 3 has been utilized between each round.

4 Experiments

4.1 Datasets and Results Details

We first run DELT on five standard benchmark tests including CIFAR-10 (10 classes) [29], Tiny197 ImageNet (200 classes) [30], ImageNet-1K (1,000 classes) [31] and it variants of ImageNette (10 198 classes) [32], and ImageNet-100 (100 classes) [33] with performances reported in Table 1 and Table 2. 199 The evaluation protocol is following prior works [15, 11]. We compare DELT to six baseline dataset 200 distillation algorithms including Matching Training Trajectories (MTT) [4], Improved Distribution 201 Matching (IDM) [34], TrajEctory Matching with Constant Memory (TESLA) [8], Squeeze-Recover202 Relabel (SRe 2 L) [11], Difficulty-Aligned Trajectory-Matching (DATM) [35], Realistic-Diverse203 Efficient Dataset Distillation (RDED) [15]. Following previous dataset distillation methods [2, 15, 204 11], we use ConvNet [36], ResNet-18/ResNet-101 [37], EfficientNet-B0 [38], MobileNet-V2 [39], 205 MnasNet1_3 [40], and RegNet-Y-8GF [41], as our backbone for training or post-validation. All our 206 experiments are conducted on 4 NVIDIA RTX 4090 GPUs. 207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

As shown in Table 1, our approach establishes the new state-of-the-art accuracy in 13 out of 15 of the configurations on five datasets from small-scale CIFAR-10 to large-scale ImageNet-1K using relatively large backbone architecture of ResNet-101, in many cases with significant margins of improvement. The results using small-scale architecture ConvNet are shown in Table 2, our approach also achieves the state-of-the-art accuracy in 8 out of 12 of the configurations on four datasets.

4.2 Cross-architecture generalization

An important characteristic of distilled datasets is their effectiveness in generalizing to novel training architectures. In this context, we assess the transferability of DELT's distilled datasets tailored for ImageNet-1K with 10 images per class. Following previous studies [11, 15], we test our models using five distinct architectures: ResNet-18 [37], MobileNet-V2 [39], MnasNet1_3 [40], EfficientNetB0 [38], and RegNet-Y-8GF [41]. As shown in Table 4, our proposed approach demonstrates significant better performance than other competitive methods on all these architectures.

4.3 Ablation Study

Mosaic splicing pattern. Mosaic stitching method [43] in RDED selects four crops from the train set as the optimal hyper-parameter, and puts the contents of the four crops into a synthetic image that is directly used for post-validation. In this work, considering that we use different difficulty levels of

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

3×3

4×4

5×5

6×6

selection for initialization, we examine different strategies of the Mosaic splicing patterns, including 1 × 1, 2 × 2, 3 × 3, 4 × 4, and 5 × 5 patches, as illustrated in Fig. 11. The ablation results are shown in Table 3, it can be observed that 1 × 1 achieves the best accuracy.

Initialization. We examine how different initialization strategies affect final performance, including: choosing lowest probability crops, medium probability crops and highest probability crops. Our results are shown in Table 3. Overall, the performance gap between different strategies is not significant, and selecting the medium probability crops as the initialization achieves the best accuracy.

Optimization iterations. We examine two types of optimization iterations: maximum iteration (MI) for the earliest batch training and round iteration (RI). MI presents the number of optimization iterations that the earliest batch goes through. RI represents the number of iterations used for each round in Fig. 4. It essentially indicates the iteration gap between the optimization of two adjacent batches. As shown in Table 3, we test MI values of 1K, 2K, and 4K, using 500 and 1K iterations for each RI. Note that when MI is set to 1K, it is not feasible to use 1K as RI. The results show that 4K (same as [11, 12]) MI and 500 RI achieves the best accuracy.

Early-only vs. EarlyLate. Early-only is equivalent to using constant MI to optimize each image. The method will transform to baseline batch-to-global matching of CDA [12] + real image initialization. Our results in Table 3 clearly show that the EarlyLate training bring a significant improvement on final performance. More importantly, this strategy is the key factor in enhancing generation diversity.

Real image stitching vs. Minimax diffusion vs. Ours. We further compare the performance of our approach with real image stitching [15] and diffusion generation [44]. The results are presented in Table 3d. While the first two methods produce more realistic images, each image contains limited information. In contrast, our method achieves the best final performance.

4.4 Computational Analysis

For image optimization-based methods like SRe 2 L and CDA, the total computational cost is calculated as 𝑁 × 𝑇 , where 𝑁 is the MI. In our EarlyLate scheme, the first batch images undergo 𝑇 1 iterations (where 𝑇 1 = 𝑇 ). Subsequent batches are processed with progressively fewer iterations, such as 𝑇 2 ( 𝑇 2 = 𝑇 1 -RI) for the next set, and so forth. The iterations for the final batch are reduced to RI which is 1 / 𝑗 of the standard count (where 𝑗 = 4 or 8 in our ablation), the total number of our optimization iterations required is 𝑁 × 𝑇 -𝑗 ( 𝑗 -1 ) 2 RI, which is roughly 2 / 3 of prior batch-to-global matching methods. Our real time consumptions for data generation are shown in Table 5, note that the smaller the dataset like CIFAR, the more time is spent on loading and processing the data, rather than training.

4.5 Visualization of DELT

Fig. 7 illustrates a comprehensive visual comparison between randomly selected synthetic images from our distilled dataset and those from the real image patches [15], MinimaxDiffusion [44], MTT [4], IDC [45], SRe 2 L [11], SCDD [46], CDA [12] and G-VBSM [14] distilled data. It can be observed that

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

# Patches

1

×

1

2

3

4

5

×

×

×

×

2

3

4

5

Top 1 acc

57.57

56.92

56.62

56.71

56.51

Iterations

Round Iterations

500

1K

44.87

45.61

46.42

1K

2K

4K

n/a

44.40

44.66

Selection criteria

Lowest probability

Medium probability

Top 1 acc

57.55

57.67

Highest probability

57.03

(b) Selection criteria . Initializing 1 × 1 images selected according to teacher model's probability

Dataset

CDA [12] + Our init.

ImageNet-1K

Tiny-ImageNet

CIFAR-10

(d) Ablation on init. and

43.5

42.2

39.4

Ours

45.8

43.0

43.0

EarlyLate under IPC 10.

IPC

RDED [15]

MinimaxDiffusion [44]

10

50

42.0

56.5

Ours

44.3

58.6

45.8

59.2

(c) Round Iterations . Top-1 acc. of our method for IPC 10 using different round iterations with ResNet-18. (e) Comparison with real and diffusion generated data.

the images generated by each method have their own characteristics. MinimaxDiffusion leverages the diffusion model to synthesize images which is close to the real ones. However, as in our above ablation, both real and diffusion-generated data are inferior to ours. MTT results show noticeable artifacts and distortions, the objects in all images are located in the middle of the generations, the diversity is limited. IDC results also show distorted and less recognizable dog images, but diversity is increased. SRe 2 L exhibits some dog features but with significant distortions and similar simple background. SCDD shows more recognizable dog features but still the color is simple and monochromatic, the same situation happens in CDA. G-VBSM shows more colorful patterns, possibly due to recovery from multiple different networks, but all generations are in the same pattern and the diversity is not large. Our approach's synthetic images exhibit a higher degree of diversity, including both compressed distorted images from long-optimized initializations and clear, recognizable dog images from short-optimized initializations, a unique capability not present in other methods.

4.6 Application I: Data-free Network Pruning

Our distilled dataset acts as a multifunctional training tool and boosts the adaptability for diverse downstream applications. We validate its utility in the scenario of data-free network pruning [47]. Table 6 shows the applicability of our dataset in this task when pruning 50% weights, where it significantly surpasses previous methods such as SRe 2 L and RDED under IPC 10 and 50.

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

Ours

G-VBSM

CDA

SCDD

2

L

MTT

IDC

Real

M-Diffusion

4.7 Application II: Continual Learning

We examine the effectiveness of DELT generated images in the continual learning scenario. Following the setup in prior studies [11, 6], we perform 100-step class-incremental experiments on ImageNet-1K, comparing our results with the baselines G-VBSM and SRe 2 L. As shown in Fig. 8, our DELT distilled dataset significantly outperforms GVBSM, with an average improvement of about 10% in 100-step class-incremental learning task. This highlights the significant benefits of deploying DELT, particularly in mitigating the challenges of continual learning.

5 Conclusion

Wehave introduced a new training strategy, EarlyLate , to improve image diversity in batch-to-global matching scenarios for dataset distillation. The proposed approach organizes predefined IPC samples into smaller, manageable subtasks and utilizes local optimizations. This strategy helps in refining each subset into distributions characteristic of different phases, thereby mitigating the homogeneity typically caused by a singular optimization process. The images refined through this method exhibit robust generalization across the entire task. We have extensively evaluated this approach on CIFAR-10 and 100, Tiny-ImageNet, ImageNet-1K, and its variants. Our empirical findings indicate that our approach significantly outperforms prior state-of-the-art methods across various IPC configurations.

Limitations. Our method effectively avoids the issue of insufficient data diversity generated by batch-to-global methods and reduces the computational cost of the generation process. However, there is still a performance gap when training the model on our generated data compared to training on the original dataset. Additionally, our short-optimized data exhibits similar semantic information to the original images, which may potentially leak the privacy of the original dataset.

100

200

300

400

500

600

700

800

900

Class Number

10

15

20

25

30

35

40

Top-1 Accuracy (%)

G-VBSM

Ours

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

References

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959 , 2018.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929 , 2020.

Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems , 35:9813-9827, 2022.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4750-4759, 2022.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning , pages 12352-12364. PMLR, 2022.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023 , 2023.

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems , 35:1100-1113, 2022.

Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning , pages 6565-6590. PMLR, 2023.

Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirzasoleiman. Data distillation can be like 321 vodka: Distilling more times for better quality. In The Twelfth International Conference on Learning 322 Representations , 2024. 323

Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. ICLR , 2024. 324

Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet 325 scale from a new perspective. In NeurIPS , 2023. 326

Zeyuan Yin and Zhiqiang Shen. Dataset distillation in large data era. arXiv preprint arXiv:2311.18838 , 327 2023. 328

Haoyang Liu, Tiancheng Xing, Luwei Li, Vibhu Dalal, Jingrui He, and Haohan Wang. Dataset distillation 329 via the wasserstein metric. arXiv preprint arXiv:2311.18531 , 2023. 330

Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data 331 condensation via various backbone and statistical matching. In CVPR , 2024. 332

Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient 333 dataset distillation paradigm. In CVPR , 2024. 334

335

336

337

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):10045-10067, 2021.

338

339

340

341

342

343

344

Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sĳia Liu. An introduction to bi-level optimization: Foundations and applications in signal processing and machine learning. arXiv preprint arXiv:2308.00788 , 2023.

Zhou Wang and Alan C Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine , 26(1):98-117, 2009.

Tian Qin, Zhiwei Deng, and David Alvarez-Melis. Distributional dataset distillation with subtask decomposition. arXiv preprint arXiv:2403.00999 , 2024.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural 345 networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , 346 pages 249-256. JMLR Workshop and Conference Proceedings, 2010. 347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422 , 2015.

Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940 , 2019.

Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov, Yida Yin, Zhiqiang Shen, Trevor Darrell, Lingjie Liu, and Zhuang Liu. Initializing models with larger ones. In The Twelfth International Conference on Learning Representations , 2024.

Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299 , 2023.

Jeffrey Zhang, Shao-Yu Chang, Kedan Li, and David Forsyth. Preserving image properties through initializations in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5242-5250, 2024.

Lutz Prechelt. Early stopping-but when? In Neural Networks: Tricks of the trade , pages 55-69. Springer, 2002.

Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems , 34:24392-24403, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Fastai. Fastai/imagenette: A smaller subset of 10 easily classified classes from imagenet, and a little more french.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16 , pages 776-794. Springer, 2020.

Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7856-7865, 2023.

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth International Conference on Learning Representations , 2024.

Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4367-4375, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Mingxing Tan, Bo Chen, Ruoming Pang, Vĳay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2820-2828, 2019.

Ilĳa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network 396 design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 397 pages 10428-10436, 2020. 398

399

400

401

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 11953-11962, 2022.

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 , 2020.

Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. In CVPR , 2024.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: A good compression is all you need. arXiv preprint arXiv:2404.07976 , 2024.

Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv:1507.06149 , 2015.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision , pages 2736-2744, 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data 421 augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision 422 and pattern recognition workshops , pages 702-703, 2020. 423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

Appendix

A Broader Impacts

Our dataset distillation framework can significantly reduce the computational resources required for training machine learning models. This leads to lower energy consumption and cost, making AI more accessible and sustainable. By generating smaller, more manageable datasets, researchers and developers can iterate and experiment more quickly, accelerating the pace of innovation in various AI applications. However, condensed datasets might inadvertently amplify biases present in the original data. If the distillation process does not adequately address bias, it could lead to unfair or discriminatory AI systems. Also, simplifying datasets may lead to a loss of important nuances and context, potentially degrading the performance of models in real-world applications where such details are crucial. Moreover, the models may overfit to condensed data, indicating that models trained on distilled datasets might perform well on the condensed data but poorly on more diverse real-world data, limiting their generalizability and robustness.

B Training Details

(a) Validation settings

(b) Recovery settings

(c) Dataset-specific settings in recovery

For reproducibility, we provide all our hyper-parameter settings used in our experiments in Table 7, we outline such details below.

Squeezing and Pre-trained models. Following the previous works [11, 12, 15], we use the official PyTorch [50] pre-trained ResNet-18 model for ImageNet-1K, and we use the same official Torchvision [50] code to produce our pre-trained models, ResNet-18 and ConvNet, for the other datasets.

Ranking. A crucial part of our method is initialization, we simply use ResNet-18 pre-trained models to rank and select the top-medium images as initialization for all our datasets, except for ImageNet-100 where we simply extracted the top-medium images based on the rankings of the original ImageNet-1K.

Recovery. For our synthesis, we provide the details of the general hyper-parameters used for different datasets, including ImageNet-1K, ImageNet-100, ImageNette, Tiny-ImageNet, and CIFAR10, in Table 7b. Because synthesizing a single image per class, i.e., IPC 1, is quite special as we cannot use

agenet

rounds, we apply different numbers of iterations based on both the dataset scale and the validation 449 teacher model as outlined in Table 7c. 450

Validation. This includes both the soft-label generation, Relabel in SRe 2 L, and evaluation, or 451 post-training. We outline such details in Table 7a. We use timm 's version of RandAugment [51] with 452 different settings depending on the synthesized dataset being validated as outlined in Table 7c. 453

454

C More Visualizations

We provide more visualizations on synthetic Tiny-ImageNet, ImageNette and CIFAR-10 datasets. In 455 each figure, each column represents a different class, with images progressing from long optimization 456 at the top to short optimization at the bottom. 457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated the contributions and scope of this paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations have been discussed in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

Justification: The paper does not include theoretical assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided all the experimental details to reproduce the results. Code is also available in the supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

Answer: [Yes]

Justification: We have included the code in the supplemental materials and shared the data link anonymously in the main paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specified all the training and test details to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have performed our experiments three times for each to provide the mean and variance accuracy suitably and correctly in our tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the details of computer resources in the experimental section. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed both potential positive societal impacts and negative societal impacts in Sec. A.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited all papers and credited all code we utilized in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should state which version of the asset is used and, if possible, include a URL.

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code has been included in the supplemental materials and is well documented, we have also shared the synthetic data in the main paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a simple but novel approach to enhance image diversity in dataset distillation. previous methods face challenges in balancing computational efficiency and diversity in synthetic images. The proposed EarlyLate training scheme addresses these issues by partitioning predefined IPC samples into smaller subtasks and using local optimizations for each subset.\n\nSTRENGTHS:\n1. The EarlyLate training scheme effectively enhances the diversity of synthetic images. This is a very simple but novel approach to increase the diversity of distilled datasets. I believe it can provide some inspiration for future work.\n2. The method, or the training scheme reduces the computational load compared to batch-to-global matching methods.\n3. The experiments are comprehensive, including performance, cross-architecture generalization, ablation, and application. These experiments verify the method's superiority.\n\nWEAKNESSES:\n1. Compared to previous methods, the work in this paper is incremental.\n2. The motivation and the advantages of the \"Selection Criteria\" in the initialization approach are not clear. And I am confused about how to rank, which is presented in Fig 5, could the authors explain it here?\n3. There are a lot of hyperparameters involved. How should these hyperparameters be tuned? Are there any principled approaches?\n4. I want to know the impact of the initialization method. In the ablation study, only CDA+init is shown. More advanced methods with init and whether EarlyLate uses init are not presented.\n5. The performance of other sota methods on MobileNet-v2 is not presented in Table 1. Is the proposed method still better than other sota methods on MobileNet-v2?\n6. Training tricks like random crop play a significant role in methods such as SRe2L. I would like to know to what extent the method proposed in this paper relies on such tricks.\n\nQUESTIONS:\nPlease refer to weaknesses.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an EarlyLate curriculum learner, which distills the easiest samples first and gradually add harder samples. Based on batch-to-global distillation algorithms, the proposed method consistently enhances the distillation performance.\n\nSTRENGTHS:\n- The writing is clear.\n- The proposed curriculum learning scheduler seems effective, which is also an interesting point to analyze.\n- Good distillation performance.\n\nWEAKNESSES:\nLimited contribution and potential overclaiming: \n\n1. In section 3, the initialization with real samples is common in DD, the data selection is proposed by RDED, and only the training scheduler is proposed in this paper. I suggest that, at least, add some diversity analysis and comparison of the distilled data.\n2. Though the paper is titled with \"diversity-driven\", the method part lacks justification of the relation between \"diversity\" and the proposed scheduler. It seems that only the real initialization contributes to the diversity.\n\nQUESTIONS:\n1. Equation 7: I assume that the initialization samples for $\\{x_0, x_1, ...\\}$ are ordered by the same criteria in Line 168-170, which is not clarified. The sample order here is critical since $x_0$ will be trained for more iterations than $x_{Mk−1}$. A comparison between random order and ascending/descending order by patch probability could rationalize the model design.\n2. Table 3(c) is the most important experiment to support the motivation of the whole paper. Maybe the authors could rewrite the story in the future version by: finding these observations -> thorough analysis -> proposing the method. The current table is not comprehensive enough, and more analysis is appreciated:\n    1. Comparison of 1K/1K is non-trivial (baseline).\n    2. It is interesting that 4K/2K is worse than 2K/1K, which means training longer leads to a performance drop (they have the same two phases, but 4K/2K trains longer in each phase). The same happens to 4K/1K and 2K/500.\n3. In Table 5: 4K-iteration is not fair for comparison since original SRe2L only trains for 1K iterations. A similar concern exists in the model design: e.g. if the base algorithm could converge within 500 iterations, DELT becomes trivial since the algorithms consume all training samples in the last 500 iterations (under default setting MI=4K and RI=500).\n\n\nMinors:\n- Line 66, footnote links to Fig.2\n- A figure with training time as the x-axis and training sample number as the y-axis could make the curriculum learning clearer and easier to demonstrate the computation time reduction.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
Recent advancements in dataset distillation have led to two main approaches: batch-to-batch and batch-to-global matching. While the former excels in small datasets, the latter, though popular for large datasets, faces a diversity challenge due to independent optimization. Authers propose an EarlyLate training scheme that enhances diversity in batch-to-global matching by partitioning IPC samples into subsets and optimizing them locally. Experiments show significant improvements over previous methods.

STRENGTHS:
1) The technical approach is solid and robust, demonstrating a high level of technical competence.
2) The performance metrics presented are highly competitive, showcasing the method's effectiveness in comparison to existing benchmarks.

WEAKNESSES:
1) The motivation for the research is unclear, lacking an explicit articulation of the unifying challenges faced by current state-of-the-art works.
2) The resolution of the figures is inadequate, impeding clear interpretation of the results.
3) There is inconsistency in the styling of table borders and captions, with captions for Table 1 and 2 placed in different positions compared to subsequent tables, some above and some below the table.
4) The experimental settings are not uniformly aligned, and efforts should be made to cover all datasets and settings consistently across all experiments to ensure comparability and rigor.

QUESTIONS:
see weaknesses

LIMITATIONS:
see weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies batch-to-global dataset distillation, optimizing the synthetic dataset by matching the statistical information of the synthetic batches to that of the full real dataset. Previous batch-to-global methods lacked diversity because each batch had the same optimization objective, leading to redundant information being learned across different batches. Based on this, the paper proposes an early-late training method. First, the real data is divided into lowest, medium, or highest probability patches based on a pretrained model, and these patches are sampled to initialize the synthetic dataset. During training, within-class samples are divided into smaller sub-batches, which are gradually concatenated for batch-to-global training.\n\nSTRENGTHS:\n1. Previous batch-to-global methods indeed faced the problem of synthetic datasets receiving the same supervision signal, leading to redundant information being learned. This paper attempts to propose a new solution to this issue.\n\n2. Extensive experiments demonstrate the good efficacy of the proposed method.\n\nWEAKNESSES:\n##  The main problem of this work is the writing.\n\n>*It dedicates too much space to introducing previous work.*\n\nThe introduction describes previous methods in too much detail, leading to redundancy with the content in the related work section. The related work section also spends too much space summarizing and describing previous methods.\n\n>*The technical part is confused*\n\nThe proposed method appears straightforward, but the authors describe the entire process almost entirely in text, lacking mathematical descriptions and definitions, which makes it somewhat difficult to understand. I suggest the authors dedicate more space to explaining the Concatenation Training and Training Procedure, incorporating some formulas to clearly demonstrate how the training is conducted.\n\n>*It is doubtful whether the proposed method can effectively solve the issues present in previous approaches.*\n\nAlthough the synthetic dataset is further divided within classes and different initializations are used, the supervision signal for each sub-batch seems to still be the same global signal as in other batch-to-global methods. This means that each sub-batch is still optimized in the same direction, potentially resulting in redundant information being learned. I suggest that the authors try to provide a more sound explanation.\n\nQUESTIONS:\nThe Training Procedure in Section 3 is somewhat difficult to understand. When training the later sub-batches in DELT, are the previous parts frozen, or are they trained together?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}