DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 16:29:55

Abstract: Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new EarlyLate training scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conducted extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our empirical results demonstrate that the proposed approach significantly improves over previous state-of-the-art methods under various IPCs.

Main paper: 
 related work: 
 Dataset Distillation. Dataset distillation or condensation [1] focuses on creating a compact yet representative subset from a large original dataset. This enables more efficient model training while maintaining the ability to evaluate on the original test data distribution and achieve satisfactory performance. Previous works [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] mainly designed how to better match the distribution between original data and generated data in a batch-to-batch manner, such as the distribution of features [6] , gradients [2] , or the model weight trajectories [4, 8] . The primary optimization method used is bi-level optimization [16, 17] , which involves optimizing model parameters and updating images simultaneously. For instance, using gradient matching, the process can be formulated as to minimize the gradient distance:EQUATIONwhere the function 𝐷 (•, •) is defined as a distance metric such as MSE [18] , 𝜃 denotes the model parameters, and ∇ 𝜃 ℓ(•; 𝜃) represents the gradient, utilizing either the original dataset T or its synthetic version S. 𝑁 is the number of 𝑑-dimensional synthetic data. During distillation, the synthetic dataset S and model 𝜃 are updated alternatively,EQUATIONwhere 𝜆 and 𝜂 are learning rates designated for S and 𝜃, respectively.Batch-to-global matching used in [11, 14, 12, 13] tracks the distribution of BN statistics derived from the original dataset for the local batch synthetic data, the formulation can be:EQUATION)where 𝑙 is the index of BN layer, 𝜇 𝑙 (S) and 𝜎 Moreover, for the recent advances of multi-stage dataset distillation methods, MDC [10] proposes to compress multiple condensation processes into a single one by including an adaptive subset loss on top of the basic condensation loss, so that to obtain datasets with multiple sizes. PDD [9] generates multiple small batches of synthetic images, each batch is conditioned on the accumulated data from previous batches. Unlike PDD, our current synthetic batch is independent with different operation iterations and not relevant to any previous batches. D3 [19] partitions large datasets into smaller subtasks and employs locally trained experts to distill each subset into distributions. These distilled distributions from the subtasks demonstrate effective generalization when applied to the entire task.Weight initialization [20, 21, 22, 23] EQUATIONwhere ℓ is a standard loss function such as soft cross-entropy and 𝜙 𝜽 S represents the model. The primary aim of dataset distillation is to produce synthetic data that ensures minimal performance difference between models trained on the synthetic dataset S and those trained on the original dataset T using validation data 𝑉. The optimization procedure for generating S is given by: arg minEQUATIONwhere (𝒙 𝑣𝑎𝑙 , 𝒚 𝑣𝑎𝑙 ) are the sample and label pairs in the validation set of the real dataset T . The learning task then focuses on the <data, label> pairs within S, maintaining a balanced representation of distilled data across each class. Selection Criteria. Here, we introduce how to select real image patches to initialize the synthetic images. In our final syntheses, a significant fraction of our data has been subject to limited optimization iterations, making effective initialization crucial. A proper initialization also dramatically minimizes the overall computational load required for the updating on data. Prior approach [15] has demonstrated that choosing representative data patches from the original dataset without training can yield favorable performance without any additional training. Our observation, however, underscores that applying iterative refinement to original patches can lead to markedly improved results. As illustrated in Fig. 1 , our selection criterion is based on a pretrained teacher model as a ranker, we calculate all patches' probabilities and sort them as the initialization pool. Then, we choose lowest, medium, or highest probability patches as the initialization for our optimization.As shown in Fig. 4 , to further emphasize diversity and avoid potential distribution bias from initialization, we optimize the initialized images starting from different points. The motivation behind this design is that different data samples require varying numbers of iterations to converge which is similar to the early stopping idea [27] . Importantly, as images become easier to predict with more updates by class labels, training primarily on easy data points can hinder model generalization. Therefore, our method enhances generalization by generating data samples with varying difficulty levels, acting as a regularizer by limiting the optimization process to a smaller volume of image pixel space. Previous work [28] studies how to perform early stopping training on different layers' weights of the model with progressive retraining to mitigate noisy labels.We are pioneering to study how to leverage early-late training when optimizing data. Moreover, we improve the efficiency of our approach by performing gradient updates in a single scan. Initially, we conduct a single gradient loop, continually introducing new data for distillation by concatenating them at different time stamps. Consequently, the 𝑀 batch receives the synthetic images of all preceding batches, IPC 0:𝑀 𝑘-1 , as final generations. This process can be simplified as follows:IPC 0:𝑀 𝑘-1 = [ x0 , x1 , . . . , x𝑘-1 IPC 0:𝑘-1 , . . . ..., x𝑀𝑘-1EQUATIONwhere Training Procedure. As illustrated in Fig. 4 , our learning procedure is extremely simple using an incremental learning process: We split the total IPCs to be learned into multiple batches. The training begins with the first batch. Following a predefined number of iterations, the second batch commences its iterative training, and this process continues sequentially with subsequent batches. Batch-to-global matching algorithm [12] of Eq. 3 has been utilized between each round.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations have been discussed in the conclusion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We have included the code in the supplemental materials and shared the data link anonymously in the main paper.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
datasets and results details: 
 We first run DELT on five standard benchmark tests including CIFAR-10 (10 classes) [29] , Tiny-ImageNet (200 classes) [30] , ImageNet-1K (1,000 classes) [31] and it variants of ImageNette (10 classes) [32] , and ImageNet-100 (100 classes) [33] with performances reported in Table 1 and Table 2 .The evaluation protocol is following prior works [15, 11] . We compare DELT to six baseline dataset distillation algorithms including Matching Training Trajectories (MTT) [4] , Improved Distribution Matching (IDM) [34] , TrajEctory Matching with Constant Memory (TESLA) [8] , Squeeze-Recover-Relabel (SRe 2 L) [11] , Difficulty-Aligned Trajectory-Matching (DATM) [35] , Realistic-Diverse-Efficient Dataset Distillation (RDED) [15] . Following previous dataset distillation methods [2, 15, 11] , we use ConvNet [36] , ResNet-18/ResNet-101 [37] , EfficientNet-B0 [38] , MobileNet-V2 [39] ,MnasNet1_3 [40] , and RegNet-Y-8GF [41] As shown in Table 1 , our approach establishes the new state-of-the-art accuracy in 13 out of 15 of the configurations on five datasets from small-scale CIFAR-10 to large-scale ImageNet-1K using relatively large backbone architecture of ResNet-101, in many cases with significant margins of improvement. The results using small-scale architecture ConvNet are shown in Table 2 , our approach also achieves the state-of-the-art accuracy in 8 out of 12 of the configurations on four datasets.An important characteristic of distilled datasets is their effectiveness in generalizing to novel training architectures. In this context, we assess the transferability of DELT's distilled datasets tailored for ImageNet-1K with 10 images per class. Following previous studies [11, 15] , we test our models using five distinct architectures: ResNet-18 [37] , MobileNet-V2 [39] , MnasNet1_3 [40] , EfficientNet-B0 [38] , and RegNet-Y-8GF [41] . As shown in Table 4 , our proposed approach demonstrates significant better performance than other competitive methods on all these architectures.Mosaic splicing pattern. Mosaic stitching method [43] in RDED selects four crops from the train set as the optimal hyper-parameter, and puts the contents of the four crops into a synthetic image that is directly used for post-validation. In this work, considering that we use different difficulty levels of 3 , it can be observed that 1 × 1 achieves the best accuracy.Initialization. We examine how different initialization strategies affect final performance, including:choosing lowest probability crops, medium probability crops and highest probability crops. Our results are shown in Table 3 . Overall, the performance gap between different strategies is not significant, and selecting the medium probability crops as the initialization achieves the best accuracy.Optimization iterations. We examine two types of optimization iterations: maximum iteration (MI) for the earliest batch training and round iteration (RI). MI presents the number of optimization iterations that the earliest batch goes through. RI represents the number of iterations used for each round in Fig. 4 . It essentially indicates the iteration gap between the optimization of two adjacent batches. As shown in Table 3 , we test MI values of 1K, 2K, and 4K, using 500 and 1K iterations for each RI. Note that when MI is set to 1K, it is not feasible to use 1K as RI. The results show that 4K(same as [11, 12] ) MI and 500 RI achieves the best accuracy.Early-only vs. EarlyLate. Early-only is equivalent to using constant MI to optimize each image. The method will transform to baseline batch-to-global matching of CDA [12] + real image initialization.Our results in Table 3 clearly show that the EarlyLate training bring a significant improvement on final performance. More importantly, this strategy is the key factor in enhancing generation diversity.Real image stitching vs. Minimax diffusion vs. Ours. We further compare the performance of our approach with real image stitching [15] and diffusion generation [44] . The results are presented in Table 3d . While the first two methods produce more realistic images, each image contains limited information. In contrast, our method achieves the best final performance.For image optimization-based methods like SRe 2 L and CDA, the total computational cost is calculated as 𝑁 × 𝑇, where 𝑁 is the MI. In our EarlyLate scheme, the first batch images undergo 𝑇 1 iterations (where 𝑇 1 = 𝑇). Subsequent batches are processed with progressively fewer iterations, such as 𝑇 2 (𝑇 2 = 𝑇 1 -RI) for the next set, and so forth. The iterations for the final batch are reduced to RI which is 1/ 𝑗 of the standard count (where 𝑗 = 4 or 8 in our ablation), the total number of our optimization iterations required is 𝑁 × 𝑇 -𝑗 ( 𝑗 -1) 2 RI, which is roughly 2/3 of prior batch-to-global matching methods. Our real time consumptions for data generation are shown in Table 5 , note that the smaller the dataset like CIFAR, the more time is spent on loading and processing the data, rather than training.Fig. 7 illustrates a comprehensive visual comparison between randomly selected synthetic images from our distilled dataset and those from the real image patches [15] , MinimaxDiffusion [44] , MTT [4] ,IDC [45] , SRe 2 L [11] , SCDD [46] , CDA [12] and G-VBSM [14] distilled data. It can be observed that the images generated by each method have their own characteristics. MinimaxDiffusion leverages the diffusion model to synthesize images which is close to the real ones. However, as in our above ablation, both real and diffusion-generated data are inferior to ours. MTT results show noticeable artifacts and distortions, the objects in all images are located in the middle of the generations, the diversity is limited. IDC results also show distorted and less recognizable dog images, but diversity is increased.SRe 2 L exhibits some dog features but with significant distortions and similar simple background.SCDD shows more recognizable dog features but still the color is simple and monochromatic, the same situation happens in CDA. G-VBSM shows more colorful patterns, possibly due to recovery from multiple different networks, but all generations are in the same pattern and the diversity is not large. Our approach's synthetic images exhibit a higher degree of diversity, including both compressed distorted images from long-optimized initializations and clear, recognizable dog images from short-optimized initializations, a unique capability not present in other methods.Our distilled dataset acts as a multifunctional training tool and boosts the adaptability for diverse downstream applications. We validate its utility in the scenario of data-free network pruning [47] .Table 6 shows the applicability of our dataset in this task when pruning 50% weights, where it significantly surpasses previous methods such as SRe 2 L and RDED under IPC 10 and 50.  
conclusion: 
 We have introduced a new training strategy, EarlyLate, to improve image diversity in batch-to-global matching scenarios for dataset distillation. The proposed approach organizes predefined IPC samples into smaller, manageable subtasks and utilizes local optimizations. This strategy helps in refining each subset into distributions characteristic of different phases, thereby mitigating the homogeneity typically caused by a singular optimization process. The images refined through this method exhibit robust generalization across the entire task. We have extensively evaluated this approach on CIFAR-10 and 100, Tiny-ImageNet, ImageNet-1K, and its variants. Our empirical findings indicate that our approach significantly outperforms prior state-of-the-art methods across various IPC configurations.Limitations. Our method effectively avoids the issue of insufficient data diversity generated by batch-to-global methods and reduces the computational cost of the generation process. However, there is still a performance gap when training the model on our generated data compared to training on the original dataset. Additionally, our short-optimized data exhibits similar semantic information to the original images, which may potentially leak the privacy of the original dataset. 7c .Validation. This includes both the soft-label generation, Relabel in SRe 2 L, and evaluation, or post-training. We outline such details in Table 7a . We use timm's version of RandAugment [51] with different settings depending on the synthesized dataset being validated as outlined in Table 7c . 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have specified all the training and test details to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We have performed our experiments three times for each to provide the mean and variance accuracy suitably and correctly in our tables.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided the details of computer resources in the experimental section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have discussed both potential positive societal impacts and negative societal impacts in Sec. A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We believe this paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 
licenses for existing assets: 
 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all papers and credited all code we utilized in this work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 In the era of large models and large datasets, dataset distillation has emerged as a crucial strategy to enhance training efficiency and make AI technologies more accessible and affordable for the general public.Previous approaches [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] primarily employ a batch-to-batch matching technique, where information like features, gradients, and trajectories from a local original data batch are used to supervise and train a corresponding batch of generated data. This method's strength lies in its ability to capture fine-grained information from the original data, as each batch's supervision signals vary.However, the downside is the necessity to repeatedly input both original and generated data for each training iteration, which significantly increases memory usage and computational costs. Recently, a new decoupled method [11, 12, 13] has been proposedto separate the model training and data synthesis, also it leverages the batch-to-global matching to avoid inputting original data during distilled data generation. This solution has demonstrated great advantage on large-scale datasets like ImageNet-1K [11, 14] and ImageNet-21K [12] . However, as shown in Fig. 2 right subfigure, a significant limitation of this method is its strategy of synthesizing each data point individually, where supervision is repetitively applied across various synthetic images.For instance, SRe 2 L [11] utilizes globally-counted layer-wise running means and variances from the pre-trained model for supervising different intra-class image synthesis. This methodology results in a pronounced lack of diversity within the same category of generated images.To address this issue, previous studies such as G-VBSM [14] and RDED [15] have been conducted.Specifically, G-VBSM [14] introduces a framework that utilizes a diverse set of local-match-global matching signals derived from multiple backbones and statistical metrics, offering more precise and effective matching than the singular model. However, as the diversity of matching models grows, the overall complexity of the framework also increases, thus diminishing its conciseness. RDED [15] crops each original image into multiple patches and ranks these using realism scores generated by an observer model. Then it amalgamates every four chosen patches from previous stage into a single new image, maintaining the resolution of the original images, and produce IPC-numbered distilled images for each class. While RDED is effective for selecting and combining data, it does not enhance or optimize the visual content within the distilled dataset. Thus, the diversity and richness of information it encapsulates largely dependent on the distribution of the original dataset.Our solution, termed the EarlyLate training scheme, is straightforward and also orthogonal to these prior methods: by initializing each image in the same category at a different starting point for optimization, we ensure that the final optimized results vary across images. We also use teacher-ranked real image patches to initialize the synthetic images. This prevents some images from being shortoptimized and ensures they provide sufficient information. As shown in Fig. 1 We perform extensive experiments on datasets of CIFAR-10, Tiny-ImageNet, ImageNet-1K and its subsets. On ImageNet-1K, our proposed approach achieves 66.1% under IPC 50 with ResNet-101, outperforming previous state-of-the-art RDED by 4.9%. On small-scale datasets of CIFAR-10, our approach also obtains 2.5% and 19.2% improvement over RDED and SRe 2 L using ResNet-101.Our main contributions in this work are as follows:• We propose a simple yet effective EarlyLate training scheme for dataset distillation to enhance the intra-class diversity of synthetic images from batch-to-global matching.2RI is the number of round iterations and will be introduced in Sec. 4.3.• We demonstrate empirically that the proposed method can generate optimized images at different distances from their initializations, to enlarge informativeness among generations.• We conducted extensive experiments and ablations on various datasets across different scales to prove the effectiveness of the proposed approach3. 
 
 Appendix: We provide more visualizations on synthetic Tiny-ImageNet, ImageNette and CIFAR-10 datasets. In each figure, each column represents a different class, with images progressing from long optimization at the top to short optimization at the bottom. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This submission presents a new training approach for dataset distillation from the perspective of enhancing the diversity of synthetic images per class. The proposed approach first selects patches from real images to initialize synthetic images, and then optimizes the generation process for the current batch by reusing synthetic images from all preceding batches and introducing new batched images, which is called the early-late training scheme. The submission was finally scored (5 yet less positive,5,4,5) by four knowledgeable reviewers, who mostly acknowledged the simplicity as well as the good performance of the proposed method, but raised several major concerns about **1)** incremental technical novelty, especially to previous methods; **2)** poor presentation of the method section; **3)** unclear motivation/insight of the real data selection criteria in the initialization step; **4)** questionable core contribution, that is, performance improvement is mostly from the initialization step rather than the early-late training scheme; **5)** inconsistent experimental settings in performance comparison.  

The authors provided detailed responses to these concerns, and all reviewers acknowledged that some of their concerns are addressed, but three of them did not increase their scores after the rebuttal. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback. I also think the proposed method is simple and shows good performance, but lacks strong technical novelty (using real data for synthetic data initialization and the concept of early-late training are both not new in dataset distillation research [15,27,28,12]) or any theoretical principles in the method formulation. Besides, the experiments provided in the original manuscript and in the rebuttal show that the initialization step clearly outweighs the early-late training (claimed core contribution) in terms of final performance improvement, though the method/criteria used in the initialization step is common. The early-late training brings small gain (sometimes less than 1%, at most 1.5%) to the initialization step. Therefore, this submission at its current form does not meet the bar for acceptance. The authors are encouraged to consider the reviewers' comments and suggestions to improve their work for a future conference.

Reviews: review: paper_summary: paper_summary: This work studies batch-to-global dataset distillation, optimizing the synthetic dataset by matching the statistical information of the synthetic batches to that of the full real dataset. Previous batch-to-global methods lacked diversity because each batch had the same optimization objective, leading to redundant information being learned across different batches. Based on this, the paper proposes an early-late training method. First, the real data is divided into lowest, medium, or highest probability patches based on a pretrained model, and these patches are sampled to initialize the synthetic dataset. During training, within-class samples are divided into smaller sub-batches, which are gradually concatenated for batch-to-global training.
strength_weakness: strength_weakness: ##  The main problem of this work is the writing.

>*It dedicates too much space to introducing previous work.*

The introduction describes previous methods in too much detail, leading to redundancy with the content in the related work section. The related work section also spends too much space summarizing and describing previous methods.

>*The technical part is confused*

The proposed method appears straightforward, but the authors describe the entire process almost entirely in text, lacking mathematical descriptions and definitions, which makes it somewhat difficult to understand. I suggest the authors dedicate more space to explaining the Concatenation Training and Training Procedure, incorporating some formulas to clearly demonstrate how the training is conducted.

>*It is doubtful whether the proposed method can effectively solve the issues present in previous approaches.*

Although the synthetic dataset is further divided within classes and different initializations are used, the supervision signal for each sub-batch seems to still be the same global signal as in other batch-to-global methods. This means that each sub-batch is still optimized in the same direction, potentially resulting in redundant information being learned. I suggest that the authors try to provide a more sound explanation.
questions: questions: The Training Procedure in Section 3 is somewhat difficult to understand. When training the later sub-batches in DELT, are the previous parts frozen, or are they trained together?
limitations: limitations: N/A
paper_summary: paper_summary: Recent advancements in dataset distillation have led to two main approaches: batch-to-batch and batch-to-global matching. While the former excels in small datasets, the latter, though popular for large datasets, faces a diversity challenge due to independent optimization. Authers propose an EarlyLate training scheme that enhances diversity in batch-to-global matching by partitioning IPC samples into subsets and optimizing them locally. Experiments show significant improvements over previous methods.
strength_weakness: strength_weakness: 1) The motivation for the research is unclear, lacking an explicit articulation of the unifying challenges faced by current state-of-the-art works.
2) The resolution of the figures is inadequate, impeding clear interpretation of the results.
3) There is inconsistency in the styling of table borders and captions, with captions for Table 1 and 2 placed in different positions compared to subsequent tables, some above and some below the table.
4) The experimental settings are not uniformly aligned, and efforts should be made to cover all datasets and settings consistently across all experiments to ensure comparability and rigor.
questions: questions: see weaknesses
limitations: limitations: see weaknesses
paper_summary: paper_summary: This paper proposes an EarlyLate curriculum learner, which distills the easiest samples first and gradually add harder samples. Based on batch-to-global distillation algorithms, the proposed method consistently enhances the distillation performance.
strength_weakness: strength_weakness: Limited contribution and potential overclaiming: 

1. In section 3, the initialization with real samples is common in DD, the data selection is proposed by RDED, and only the training scheduler is proposed in this paper. I suggest that, at least, add some diversity analysis and comparison of the distilled data.
2. Though the paper is titled with "diversity-driven", the method part lacks justification of the relation between "diversity" and the proposed scheduler. It seems that only the real initialization contributes to the diversity.
questions: questions: 1. Equation 7: I assume that the initialization samples for $\{x_0, x_1, ...\}$ are ordered by the same criteria in Line 168-170, which is not clarified. The sample order here is critical since $x_0$ will be trained for more iterations than $x_{Mk−1}$. A comparison between random order and ascending/descending order by patch probability could rationalize the model design.
2. Table 3(c) is the most important experiment to support the motivation of the whole paper. Maybe the authors could rewrite the story in the future version by: finding these observations -> thorough analysis -> proposing the method. The current table is not comprehensive enough, and more analysis is appreciated:
    1. Comparison of 1K/1K is non-trivial (baseline).
    2. It is interesting that 4K/2K is worse than 2K/1K, which means training longer leads to a performance drop (they have the same two phases, but 4K/2K trains longer in each phase). The same happens to 4K/1K and 2K/500.
3. In Table 5: 4K-iteration is not fair for comparison since original SRe2L only trains for 1K iterations. A similar concern exists in the model design: e.g. if the base algorithm could converge within 500 iterations, DELT becomes trivial since the algorithms consume all training samples in the last 500 iterations (under default setting MI=4K and RI=500).


Minors:
- Line 66, footnote links to Fig.2
- A figure with training time as the x-axis and training sample number as the y-axis could make the curriculum learning clearer and easier to demonstrate the computation time reduction.
limitations: limitations: The authors have adequately discussed the limitations.
paper_summary: paper_summary: The paper presents a simple but novel approach to enhance image diversity in dataset distillation. previous methods face challenges in balancing computational efficiency and diversity in synthetic images. The proposed EarlyLate training scheme addresses these issues by partitioning predefined IPC samples into smaller subtasks and using local optimizations for each subset.
strength_weakness: strength_weakness: 1. Compared to previous methods, the work in this paper is incremental.
2. The motivation and the advantages of the "Selection Criteria" in the initialization approach are not clear. And I am confused about how to rank, which is presented in Fig 5, could the authors explain it here?
3. There are a lot of hyperparameters involved. How should these hyperparameters be tuned? Are there any principled approaches?
4. I want to know the impact of the initialization method. In the ablation study, only CDA+init is shown. More advanced methods with init and whether EarlyLate uses init are not presented.
5. The performance of other sota methods on MobileNet-v2 is not presented in Table 1. Is the proposed method still better than other sota methods on MobileNet-v2?
6. Training tricks like random crop play a significant role in methods such as SRe2L. I would like to know to what extent the method proposed in this paper relies on such tricks.
questions: questions: Please refer to weaknesses.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.

Comments: title: Thanks for raising the score
Thanks for pointing this out
Thank you for your post-response
Response to Reviewer d2h5
comment: Thank you for your kind words and for raising the score. We have no intention of complaining but are thankful and sincerely appreciate the time and effort you invested in reviewing our work, and further helping us improve it. We assure that we will polish our paper carefully according to the suggested comments. Wishing you a great day!
Thank you for pointing this out! The mistake in the caption was unintentional, likely due to following other works. We will remove it in the revision to ensure it aligns with our table.
For Q4, plz check table 2, why CIFAR-10 is not in the list while it is mentioned in the caption?
I want to clarify that I have the background of batch-to-global matching, e.g., Sre$^2$L, but the initial draft gave me the impression that too much space was devoted to unnecessary parts, while the original content lacked detail. As a result, the paper feels unclear when reading.

However,  I wouldn't be too harsh on this. I am not strongly against accepting the paper, although, a major revision would be necessary. I increase my score to 5.
Thank you for your post-response, we are glad that some of your concerns are addressed by our rebuttal, and we respect your comments and decision. 

However, we would like to further clarify two points (primarily for the ACs):

1. The perception of whether our writing is understandable largely depends on the reader's background. For example, another reviewer mentioned that our writing is clear. The difficulty in understanding may stem from a lack of familiarity with recent developments in *batch-to-global* matching in large-scale dataset distillation methods and related works like SRe$^2$L, CDA, RDED, etc.

2. The three stages mentioned in our rebuttal are well-established in recent literature. In our paper, we focused on highlighting the differences and our contributions relative to previous works, instead of reiterating well-known frameworks.

It seems that the subjective perception of writing clarity was a key concern for this reviewer, while we have proposed revisions to address this, we are unsure if this alone should be a determining factor in the rejection.

We hope the ACs will consider these points in the final decision.
I thank for the detailed rebuttal. Some concerns are addressed. Nevertheless, I think a major revision would be important so I maintain my initial score.
Thanks for the reply. Some of my concerns are addressed and I will keep my score.
We sincerely appreciate the reviewer's kind reply and further questions. 

>Some concerns on W2 still remain. The authors give empirical results on enhancing diversity, but should clarify why the EarlyLate training enhances diversity. I understand that the method could learn samples at diverse difficulty levels, but there is a gap to pixel diversity. Perhaps the diversity all comes from the initialization.

Initialization forms the foundation, while the proposed EarlyLate training enhances diversity by varying the optimization length for different samples post-initialization. Without this variation, using the same optimization budget/iteration with the initialization strategy for all samples would result in similar style patterns (e.g., similar mosaics or abstract drawings), as shown in the MTT and SRe$^2$L columns of Figure 7. In our method, fully-optimized images will look like SRe$^2$L/CDA synthesis, under-optimized images will look closer to the original images like RDED/MinimaxDiffusion, and this change is gradual for each class in the entire dataset. If the reviewer examines Figures 6 and the last column of Figure 7 (from top to bottom), they will be seen how our method intuitively increases the diversity of the generated images.

>Q1: Surprisingly this is not detailed in the paper.

Given the paper's length limitation and our original belief that Figure 5 was sufficiently informative, we will include the additional details mentioned in our response in the revised version.

>I suggest the authors polish the writing of section 3, also according to reviewers 1XsL and cugo.

Thanks for the suggestion. We will polish section 3 carefully according to reviewers 1XsL and cugo's comments.
Thanks for the authors' response and some of my concerns are addressed.

**W1-W2**: Thanks for the clarification but some concerns on W2 still remain. The authors give empirical results on enhancing diversity, but should clarify why the EarlyLate training enhances diversity. I understand that the method could learn samples at diverse difficulty levels, but there is a gap to pixel diversity. Perhaps the diversity all comes from the initialization.

**Q1**: Surprisingly this is not detailed in the paper.

**Q2-Q3 and Minors**: Thanks for the details and I appreciate the efficiency analysis.

I suggest the authors polish the writing of section 3, also according to reviewers *1XsL* and *cugo*.