Dataset Distillation via Curriculum Data Synthesis in Large Data Era
Authors: Zeyuan Yin
Zhiqiang Shen
Venue: Accepted by TMLR ()

Abstract: Dataset distillation or condensation aims to generate a smaller but representative subset from a large dataset, which allows a model to be trained more efficiently, meanwhile evaluating on the original testing data distribution to achieve decent performance. Previous decoupled methods like SRe$^2$L simply use a unified gradient update scheme for synthesizing data from Gaussian noise, while, we notice that the initial several update iterations will determine the final outline of synthesis, thus an improper gradient update strategy may dramatically affect the final generation quality. To address this, we introduce a simple yet effective global-to-local gradient refinement approach enabled by curriculum data augmentation ($\texttt{CDA}$) during data synthesis. The proposed framework achieves the current published highest accuracy on both large-scale ImageNet-1K and 21K with 63.2% under IPC (Images Per Class) 50 and 36.1% under IPC 20, using a regular input resolution of 224$\times$224  with faster convergence speed and less synthetic time. The proposed model outperforms the current state-of-the-art methods like SRe$^2$L, TESLA, and MTT by more than 4% Top-1 accuracy on ImageNet-1K/21K and for the first time, reduces the gap to its full-data training counterparts to less than absolute 15%. Moreover, this work represents the inaugural success in dataset distillation on the larger-scale ImageNet-21K dataset under the standard 224$\times$224 resolution. Our code and distilled ImageNet-21K dataset of 20 IPC, 2K recovery budget is available at https://github.com/VILA-Lab/SRe2L/tree/main/CDA.

Published in Transactions on Machine Learning Research (11/2024)
BOOM
Dataset Distillation via Curriculum Data Synthesis in
Large Data Era
Zeyuan Yin
zeyuan, way.uan.yiinnnbzaai.coo
VILA Lab
Mohamed bin Zayed University of Artificial Intellignnce
Zhiqiang Shen*
zhiqiong. www.s@mzenmaaa...mmm ac.ee
VILA Lab
Mohamed bin Zuyed Universily of Artificial Intelligence
Reviewed on OpenReview: https: // openreview. net/ formf id= PlaZDanGC1
Abstract
Dataset distillation or condensation aims to generate a smaller but representative subset from a large dataset, which allows a model to be trained more efficiently, meanwhile eval- uating the original testing data distribution to achieve decent performance. Previous decoupled methods like SRe²L simply use unified gradient update scheme for syuthesizing data from Gaussian noise, while, we notice that the initial several update iterations will determine the final outline of synthesis, thus an improper gradient update strategy may dramatically affect the final generation quality. To address this, we introduce a simple yet effective global-to-local gradient refinement approach enabled by curriculum data augmen- tation (CDA) during data synthesis. The proposed framework achieves the current published highest accuracy both large-scale ImageNet-1K and 21K with 63.2% under TPC (Images Per Class) 50 and 36.1% under IPC 20, using regular input resolution of 224x224 with faster convergence speed and less synthelic time. The proposed model outperforms the current state-of-the-art methods like SRe?L, TESLA, and MTT by more than 4% Top-1 accuracy on ImageNet-1K/2111 and for the first time, reduces the gap to its full-data train- ing counterparts to less than absolute 15%. Moreover, this work represents the inaugural success dataset distillation on the larger-scale ImageNet-21KK dataset under the standard 224x224 resolution. Our code and distilled ImageNet-21IK dataset of 20 IPC, 2K recovery budget are available at lhttss://githuu wtt..///A----aa//RRR//////AAAA/ httpaa/////eeeeeee//aaaaa////// main/CDA
1
Introduction
65
SRe'L (4K)
Our CDA (1K)
Dataset distillation condensation (Wang et al.)
Our CDA (4K)
2018] has attracted considerable attention across
Top-1 Accuarty (9))
60
various fields of computer vision (Cazenavette et al.)
2022b Cui et al. 2023 Yin et al. 2023) and natural
language processing (Sucholutsky Schonlan 2021
55
Maekawa al. 2023). This task aims optimize
the process of condensing massive datasct into
smaller,, yet representative subset, preserving the es- sential features and characteristics that would allow a model to learn from scratch as effectively from the distilled dataset as would from the original large
Resnet-18
Resnet-50
Resnet-101
DenseNet-122
RegNet-V.8GF
Figure 1: ImageNet-1K comparison with S.
* *Corresponding author.

---

Published in Transactions on Machine Learning Research (11/2024)
dataset. As the scale of data and models continue to grow, this dataset distillation concept becomes even more critical in the large data era, where datasets are often voluminous that they pose storage, computational, and processing challenges. Generally, dataset distillation can level the playing field, allowing researchers with limited computation and storage resources to participate in statc-of-the-art foundational model training and
application development, such as affordable ChatGPT (Brown et. al. 2020 OpenAT 2023 and Stable Dif-
fusion (Rombach et al., [002], the current large data and large model regime. Morcover, by working with distilled datasets, which are synthesized to retain the most representative information from Gaussian noise initialization through gradient optimization at high-level abstraction instead of closely resembling the original dataset, there is potential to alleviate data privacy concerns, as raw, personally identifiable data points might be excluded from the distilled version.
Recently, there has been a significant trend in adopting large models and large data across various research and application areas. Yet, many prior dataset distillation methods (Wang et al. 2018 Zhao et al., 2020; Zhou et al. 2022; Cazenavette et al. 2022a Kim et al. 2022a Cui ct al. |023) prodominantly target datasets finding challenging to scale their frameworks
like CIFAR, Tmmy-lmageNet, and downsampled ImageNet-1K.
2009). This suggests that these approaches have not fully evolved in line with contemporary ad advancements and dominant methodologies.
for larger datasets, such as full ImageNet-IK 1K (Deng et al.
In this study, we extend our focus even beyond the ImageNet-IK dataset, venturing into the un- charted territories of the full ImageNet-211 (Deng et al., 2009 Ridnik et al.) 2021) at a conven- tional resolution of 224x224. This marks pio- neering effort in handling such a vast dataset for dataset distillation task. Our approach harnesses a straightforward yet effective global-to-local learn- ing framcwork. We moticulously address each as- pect and craft a robust stralegy to effectively train on the complete ImageNet-2IK,, ensuring compre- hensive knowledge is captured. Specifically, follow- ing prior study (Yin et al., 2023), our approach initially trains model to encapsulate knowledge from the original datasets within its dense parame- ters. However, we introduce a refined training recipe that surpasses the results of Ridnik et al. (202T) h on ImageNet-21K. During the data recovery/ synthesis phase, we employ a strategic learning scheme where partial image crops sequentially updated based on the difficulty of regions: transitioning either from simple to difficult, or vice versa. This progression is modulated by adjusting the lower and upper bounds of the Random ReidzomdeiszzeCrrop data augmentation throughout varying training iterations. Remarkably, we ob- serve that this straightforward Icarning approach substantially improves the quality of synthesized data. In this paper, we delve into three learning paradiguns for data synthesis linked to the curriculun learning frame- work. The first is the standard curriculum learning, followed by its alternative approach, reverse curriculum learning. Lastly, we also consider the basic and previously employed method of constant learning.
Initial
intermediate
final
Figure 2: Molivation of our work. The left column is the syuthesized images after a few gradient up- date itcrations from Gaussian noise. Middle and right columns are intermediate and final synthesized images.
Motivation and Intuition. We aim to maximize the global informativeness of the synthetic data. Both SRe"L (Yin et al., 2023) and our proposed approach utilize local mini-batch data's mean and variance statistics to match the global statistics of the entire original dataset, synthesizing data by applying gradient updates directly to the image. The impact of such strategy is that the initial few iterations set the stage for the global structure of the ultimately generated image, as shown in Figure 2 Building upon the insights derived from the analysis, we can leverage the global-to-local gradient refinement scheme for more expressive synthesized data, in contrast, SRe2L does not capitalize on this characteristic. Specifically, our proposed approach exploits this by initially employing large crops to capture a more accurate and complete outline of
2

---

Published in Transactions on Machine Learning Research (11/2024)
Hardest
Easiest
[0.08, 0.20]
[0.08. 0.40]
[0.08, 0.60]
[0.08, 0.80]
[0.08. 1.00]
[0.20. 1.00]
[0.40. 1.00]
[0.60, 1.00]
[0.80. 1.00]
Figure 3: Illustration of crop distribution from different lower and upper bounds in CandomResizedlo The first row is the central points of bounding boxes from different sampling scale hyperparameters. The second and last rows correspond to 30 and 10 boxes of the crop distributions. In each row, from left to right, the difficulty of crop distribution is decreasing.
objects, for building a better foundation. As the process progresses, it incrementally reduces the crop size to enhance the finer, local details of the object, significantly elevating the quality of the synthesized data. Global-to-local via Curriculum Sampling. RandomResizedChop randomly crops the image to certain area and then resizes it back to the pre-defined size, ensuring that the model is exposed to different regions and scales of the original image during training. As illustrated in Figure 3, the difficulty level of the cropped region can be controlled by specifying the lower and upper bounds for the area ratio of the crop. This can used to ensure that certain portions of the image (small details or larger context) are present in the cropped region. If we aim make the learning process more challenging, reduce the minimum crop ratio. This way, the model will often see only small portions of the image and will have to learn from those limited contexts. If we want the model to larger context more frequently, increase the minimum crop ratio. Tn this paper, we perform comprehensive study how the gradual difficulty changes by sampling strategy influence the optimization of data generation and the quality of synthetic data for dataset distillation. Our proposed curriulum data augmentation (CDA) is a heuristic and intuitive approach to simulate global-to- local lcarning procodure. Morcover, it is highly effective on largo-scale datasets like ImageNet-1K and 21K, achieving state-of-the-arl performance on dataset distillation.
The Significance of Large-scale Dataset Condensation. Large models trained on large-scale datasets consistently outperform smaller models and those trained on limited data (Dosovitskiy et al. 2020 Dehghani et al. 2023 OpenAl 2023). Their ability to capture intricate patterns and understand muanced contextual information makes them exceptionally elfective across a wide range of tasks and domains. These models play a crucial role in solving complex industrial challenges and accelerating the development of AI-driven products and services, thereby contributing to economic growth and innovation. Therefore, adapting dataset condensation or distillation methods for large-scale data scenarios is vital to unlocking their full potential in both academic and industrial applications.
We conduct extensive experiments on the CIFAR, Imng-ImageNet, ImageNet-IK, and ImageNet-21IK datasets. Employing resolution of 224x224 and IPC 50 on ImageNet-IK, the proposed approach al- tains impressive accuracy of 63.2%, surpassing all prior state-of-the-art methods by substantial margins. As illustrated in F our proposed CDA outperforms SRc²L by 4~6% across different architectures under 50 IPC, on both 1K and 4K recovery budgets. When lested on ImageNel-211K with IPC 20, our hod achieves a top-1 accuracy of 35.3%, which is closely competitive, exhibiting only minimal gap compared to the model pre-trained with full data, at 44.5%, while using 50x fewer training samples.
Our contributions of this work:
We propose new curriculum data augmentation (CDA) framework enabled by global-to- gradient update in data synthesis for large-scale dataset distillation.
3

---

Published in Transactions on Machine Learning Research (11/2024)
We are the first to distill the ImageNett22KK dataset, which reduces the gap to its full-data training counterparts to less than an absolute 15% accuracy.
We conduct extensive experiments on CIFAR-100, Tiny-ImageNet,, ImageNet- and ImageNet-21ns
datasets to demonstrate the effectiveness of the proposed approach.
2 Related Work
Dataset condensation or distillation strives to form compact, synthetic dataset, retaining crucial informa- tion from the original large-scale dataset. This approach facilitates casier handling, reduces training time, and aims for performance comparable to using the full dataset. Prior solutions typically fall under four cat- egories: Moda- Matching optimizes for model transferability on distilled data, with an outer-loop for
synthetic data updates, and an inner-loop for network training, such as DD [Wang et al., 2020). KIP Ngguyen
jet al. 2021D, RFAD (Loo et al., 2022). FRePo (Zhou et al. 2022 LinBa (Dng & Russakovsky 2022). and
MDC (He et al.] (2024;; Grudient Matching performs a one-step distance matching between models, such
as DC (Zhao et al., (2020) DSA (Zhao & Bilen 2021), DCC (Lee et al.) 2022). IDC (Kim et al. 2022b);
and MP (Zhou et al. 2024aa): Distribution Matching directly matches the distribution of original and syn-
thetic data with single-level optimization, such DM (Zhao & Bilen 2023), CAFE (Wang et al.
2022
HaBa (Liu et al. 2022a). KFS (Lee et al., 2022). DataDAM (Sajedi et al. 2023) FreD Shin et al.
(0244
and GUARD (uee el al. 2024); Trajectory Malching matches the weight trajectories of models trained on
original and synthetic data in multiple steps, methods include MTT (Cazenavette et al. 2022b): TESLA CCi
ct al. 2023). APM (Chen et al. 2023 and DATM (Guo et al. 2024).
Moreover, there are some recent. methods out of these categories that have further improved the existing dataset distillation. SeqMatch (Du et al. 2023 reorganizes the synthesized dataset during the distillation and evaluation phases to extract both low-level and high-level features from the real dataset, which can be integrated into existing dataset distillation methods. Deep Generative Prior (Cazenavette et al. 2023) utilizes the learned prior from the pre-trained deep generative models to synthesize the distilled images. RDED (Sun et al. 2024) proposes a non-optimization method to concatenate multiple cropped realistic patches from the original data to compose the distilled dataset. D3M (Abbasi et al.) 2024] condenses an entire category of images into a single textual prompt of latent diffusson models. SC-DD (Zhou et al,, 2024b) proposes a self-supervised paradigm by applying the self-supervised pre-trained backbones for dataset distillation. EDC (Shao et al.) 2024b) explores a comprehensive design space that includes multiple specific, effective strategies like soft category-awarr matching and learning rate schedule to establishe benchmark for both small and large-scale dataset distillation. Ameliorate Bias (Cui et al. 2024) studies the impact of bias within the original dataset on the performance of dataset condensation. It introduces a simple yet effective approach based on a sample reweighting scheme that utilizes kernel density estimation.
SRe2L (Yin et al., 2023) is the first and mainstream framework to distill large-scale datasets, such as ImageNet-1K, and achieve significant performance. Thus, we consider it our closest baseline. More specif- ically, SRe?T. proposes decoupling framework to avoid the bilevel optimization of model and synthesis during distillation, which consists of three stages of squeezing, recovering, and relabeling. In the first squeez- ing stage, a model is trained on the original dataset and serves as a frozen pre-train model in the following two stages. During the recovering stage, the distilled images are synthesized with the knowledge recovered from the pre-train model. At the last relabeling stage, the soft labels corresponding to synthetic images are generated and saved by leveraging the pre-train model. Recently, distilling on large-scale datasets has received significant attention in the community, and many works have been proposed, including (Sun et al,
2023
Liu et al.) 2023 Chen et al. 2023 Shao et al.] 2024a; Zhou et al. 2024a Wu et al.) 2024 Abbasi
ct al.
2024
Zhou ct al.) 2024b
Shao ct al.] 2024b: Xuc et al. 2024 Qin ct al. 2024 Gu et al.] 2023 Ma
el al.)
2024
Shang et al.
2024
Theses recent methods represent a comprehensive sludy and literature
on framework design space (Sao et al., 2024b) and adversarial robustness benchmarks (Wu et al., 2024
in dataset distillation. They are substantially different from our input-optimizzaioo-beers approach. Addi- tionally, GUARD (Xue et al.) 2024) incorporates curvature regularization to embed adversarial robustness, focusing on different objective than our CDA.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Compared to earlier traditional dataset distillation baselines, CDA is fundamentally different in its approach: (1) CDA exhibits better scalability. The previous works like DM (Zhao & Bilen 2023), DSA (Zhao & Bilen 2021), and FRePo (Zhou et al. 2022), work well on small-scale dataset distillation, but they are limited by the huge computational cost and cannot be scaled to large datasets and models. (2) Different generation
paradigms. MTT (Cazenavelle et al. 2022a) matches the model Irajectories (weights) of training on distilled and raw datasets; RDED (Sun et al. 2023) selects and combines the raw image patches with diversity; D3M (Abbasi et al., 2024) leverages text-to-image diffusion models to generate distilled images. Thus, MTT proposes matching trajectories, RDED proposes non-optimizing, and D3M proposes iiffusion-modde-bbseee generation paradigms which do not align with to our know/edge-cissillatiinn-
generation approach. (3) Unique evaluation recipes. For instance, RDED utilizes a unique smoothed LR schedule for the learning rate reduction throughout the evaluation, which improves evaluation performance effectively -
3 Approach
3.1
Preliminary: Dataset Distillation
The goal of dataset distillation is to derive concise synthetic dataset that maintains significant proportion of the information contained in the original, much larger dataset. Suppose there is large labeled dataset D. [(11,11,,- ....., (.I }, our target is to formulate compact distilled dataset, represented as
Da (zt,,11) ..... (D/ 3 (al) ) J. where y' is the soft label corresponding to synthetic data x', and [Dal << [Dol, preserving the essential information from the original dalaset D,,. The learning objective based on this distilled synthetic dataset is:
PD.- = arg min
0
LD. (0) (o' w/ DD) ( ((00,,
(2)
where f is the regular loss function such as the soft cross-entropy, and Φ0p, is model. The primary objective of the dataset distillation task is to generate synthetic data aimed at attaining a specific or minimal performance disparity on the original validation data when the same models are trained on the synthetic data and the original dataset, respectively. Thus, we aim to optimize the synthetic data Da by:
Suu {|e (der., (Irat), Yeat) - l (dee,, (2lo,( (Ivvt): Yeol Da.|Dal
arg min
where (Ivel, )eat) are sample and label pairs in the validation set of the real dataset Do. Then, we learn < <data, label>EDa Da with the corresponding number of distilled data in each class.
3.2 Dataset Distillation on Large-scale Datasets
Currently, the prevailing majority of research studies within dataset distillation mainly employ datasets of a scale up to ImageNet-1K (Cazenavette et al. [202b Cui et al., 2023: Yin et al. (2023) as their benchmarking standards. In this work, we are the pioneer in showing how to construct a strong baseline on ImageNet- 21K (the approach is equivalently applicable to ImageNet- by incorporating insights presented in recent studies, complemented by conventional optimization techniques. Our proposed baseline is demonstrated to achieve state-of-the-art performance over prior counterparts. We believe this provides substantial significance towards underslanding the true impact of proposed methodologies on dataset distillation task and towards assessing the true gap with full original data training. We further propose a curriculum training paradigm to achieve a more informative representation of synthetic data. Following prior work dataset distillation (Yin) Recover-Relabbeee to save computation
et al. 2023), we focus on the decoupled training framework, and memory consumption on large-scale TetgeNe the procedures are listed below:
Squeeze: Building A Strong Pre-trained Model on ImagNNet-21.. To obtain a squeezing model, we use a relatively large label smooth of 0.2 her with Cutout (DeVries & Taylor, 2017) and RandAug:
ment (Cubuk et al., (2020), as shown in Appendix B.4 This recipe helps achieve 22% improvement over the
default training (Ridnik et al. 2021 on ImageNet-222 21K,, as provided in Table 21
Therefore, ensure a fair and straightforward comparison, these baseline results in our experimental section have been taken directly from the best evaluation performance reported by their original papers.
5

---

Published in Transactions on Machine Learning Research (11/2024)
Recover: Curriculum Training for Better Representation of Synthetic Data. A well-crafted curriculum data augmentation is employed during the synthesis stage to realize the global-to-local learning scheme and enhance the representational capability of the synthetic data. This step is crucial, serving to enrich the gencrated images by embedding more knowlodge accumulated from the original datasct, thereby making them more informative. Detailed procedures will be further described in the following Section 3.3
Relabel: Post-training on Larger Models with Stronger Training Recipes. Prior studies, such TESLA (Cui et al., 2023). have encountered difficulties, particularly, decline in accuracy when utilizing models of larger scale. The reason may be that the trajectory-based matching approaches, c.g., MTT and TESLA, generate images by excessively optimizing to aligu the dense training trajectories of model weights at each epoch between real and distilled datasets on specific backbone models. As a result, the distilled dataset becomes overly dependent on these models, potentially leading to overfitting and reduced effectiveness when training other models, particularly larger ones. This further suggests that the synthetic data used is potentially inadequate for training larger models. Conversely, the data we relabel show improvement with the use of larger models combined with enhanced post-training methodologies, displaying promise when applied to larger datascts in distillation processes.
We have also observed that maintaining a smaller batch size is crucial for post-training on synthetic data to achieve commendable accuracy. This is attributed to the Generalization Gap (Keskar et al. 2016 Hoffer ct al. 2017), which suggests that when there is a deficiency in the total training samples, the model's 's capacity to generalize to new, unseen data is not robust. In the context of synthetic data, the generalization gap can be exacerbated due to the inherent differences between synthetic and real data, distributions. Smaller batch sizes tend to introduce more petaalssnoisse into the gradient updates during training, which, counterintuitively, can help in better generalizing to unseen data by avoiding overfitting to the synthetic dataset's ' general patterns. The noise can also act as a regularizer, preventing the model from becoming too confident in its predictions on the synthetic data, which may not fully capture the complexitics of large batch-size data. Employing smaller batch sizes while training on the small-ssall synthetic data allows models to explore the loss landscape more meticulously before converging to an optimal minimum. In Table we empirically notice that utilizing a small batch size improve model evaluation performance. This observed phenomenon aligns with the Generalization Gap theory, which arises when there is lack of training samples.
3.3 Global-to-local Gradient Update via Curriculum
In SReeL @Yin et al. 2023 approach, the key of data synthesis revolves around utilizing the gradient information emanating from both the semantic class and the predictions of the pro-trained squeezing model, paired with BN distribution matching. Let (æt y) be an example æ for optimization and its corresponding one-hot label y for the pre-trained squeezing model. Throughout the synthesis process, the squeezing model is frozen to recover the encoded information and ensure consistency and reliability in the generated data. Let T(x) be the target training distribution from which the data synthesis process should ultimately learn function of desired trajectories, where T is data transformation function to augment. input samples to various levels of difficulties. Following Bengio et al. (2009), weight 0 < W, (x) < 1 is defined and applied to example æ at stage in the curriculum sequence. The training distribution (z(z) is:
(x(x) & (x)x)((z)) Vr
In our scenario, since the varying difficulties are governed by the data transformation function T, we can straightforwardly employ (x) across all stages. Consequently, the training distribution solely depends on Txx and can be simplified as follows:
(xx) x T(x) Vr
(5)
By integrating curriculum learning within the data synthesis phase, this procedure can be defined as: Definition 1 (Curriculum Data Synthesis). In the data synthesis optimization, the corresponding sequence of distributions D(x) will be a curriculum if there is an increment in the entropy of these distributions, i.e., the difficully of the transformed input samples escalates and becomes increasingly challenging for the pre-trained model to predict as the training progresses.
6

---

Published in Transactions on Machine Learning Research (11/2024)
Thus, the key for our curriculum data synthesis becomes how to design T(x) across different training iterations. The follow- ing section discusses several strategies to construct this in the curriculum scheme.
initial
Baseline: Constant Learning (CTL). This is the regular training method where all training examples typically treated equally. Each sample from the training dataset has an equal chance of being transformed in given batch, assuming no diffi- culty imbalance or biases across different training iterations.
Curriculum
Crop
Cropai
Crople
CTL is straightforward to implement since we do not have to rank or organize examples based on difficulty. In practice, we use Rondom to crop a small region via current crop ratio randomly sampled from a given interval [min_ crop, max_ _aa_mmappppppap crop] and then resize the cropped image to its original size, formulated as follows:
Open
Optil
Opti::
Opti:
Optimizer at step S
synthetic
xy RandommesszzdCCopplaa min_crop crop ai, max_ crop 0a)
Figure 4: Illustration of global-to-local data synthesis. This figure shows our spe- cific curriculum procedure in data synthe- sis to provide a comprehensive overview of our dataset distillation framework. It starts with large area (single bounding- box in each step) to optimize the image, building a better initialization, and then gradually narrows down the image area of learning process so that it can focus on more detailed areas.
(6)
where aj and au are the constant lower and upper bounds of crop scale.
Curriculum Learning (CL). As shown in Algorithm ! in our CL, data samples organized based on their difficulty. The difficulty level of the cropped region can be managed by defining the lower and upper scopes for the area ratio of the crop. This en- ables the assurance that specific crops of the image (small details or broader context) are included in the cropped region. For the difficulty adjustment, the rate at which more difficult examples are introduced and the criteria used to define difficulty are adjusted dynamically as predetermined using the following schedulers.
Constart
Step
Coene
Step. Step scheduler reduces the minimal scale by factor for every fixed or specified number of iterations, as shown in Figure 5 right.
Crop ratlo
Linear. Linear scheduler starts with a high initial value and decreases linearly by a factor y to a min- imum value over the whole training.
0
Cosine. Cosine scheduler modulates the distribution according to the cosine function of the current itera- tion number, yielding a smoother and more gradual adjustment, compared to step-based methods.
0.08
100
milestone 100
Distillaionn progress (%)
Distillation progress (%)
Figure 5: Crop ratio schedulers of prior CTL solu- tion (left) and our Global-to-local (right) enabled by curriculum. The colored regions depict the random sampling intervals for the crop ratio value in each itcration under different schedulers.
As shown in Figure 5. the factor distribution manages the difficulty level of crops with adjustable an and ai for CTL and milestone for CL.
Data Synthesis by Recovering and Relabeling.
After receiving the transformed input æ7, update it by aligning between the final classification label and intermediate Batch Normalization (BN) statistics, i.e., mean and variance from the original data. This stage forces the synthesized images to capture a shape of the original image distribution. The learning goal for this stage can be formulated as follows:
s' arg min l (de (xT) .y) Rieg
(7)
where Φe the pre-trained squeezing model and will be frozen this stage. During synthesis, only the input crop area will be updated by the gradient from the objective. The entire training procedure is illustrated in Figure 4 After synthesizing the data, follow the relabeling process in SRe"L. to generate soft labels using
7

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1: Our CDA via. RandomResizzdCrop
Input: squeezed model Φg, recovery iteration S, curriculum milestone T, target label y, default lower and upper bounds of crop scale 31 and Bu in RandomlResizedCrrpp decay of lower scale bound 7 Output: synthetic image Initialize: zo from standard normal distribution for step s from 0 to S-1 do if & I T then
Bu
if step
a +
B1 + 7 * (Bu - 8/T)
if linear
B1 + y (Bu + cos (Λ s//T)) /2 if cosine
else
B
end
TT f Random.ResizeCChoppc min_ crop = a. max_crop = 3n))
4/ + vт is optimized w.r.t og and y in Eq. 7
Es+1 t Reverse ReoorssRaadom@ReesseCCCp.pee x'y)
end
return x <- g
FKD Shen & Xing 2022) with the integration of the small batch size setting for post-training. Ryeg is the regularization term used in Yin et al. 2023 its detailed formulation using channel-wise mean and variance matching is:
Rieg (xx)
Σ 1ax (a') B (Nx Do))ll + Σ 1|of (a') - TE ( 1
(8)
Pk (x') BNRMIL Σ -
where & is the index of BN layer, pa (x') and of (x') are the chamnel-wise mean and variance in current batch data. BNRM and BN are mean and variance in the pre-trained model at k-th BN layer, which are globally counted.
Advantages of Global-to-local Synthesis. The proposed CDA enjoys several advantages: (1) Stabilized training: Curriculum synthesis can provide a more stable training process as it reduces drastic loss fluctu- ations that can occur when the learning procodure encounters a challenging sample carly on. (2) Better generalization: By gradually increasing the difficulty, the synthetic data can potentially achieve better gener- alization diverse model architectures in post-training. It reduces the chance of the synthesis getting stuck in poor local minima early in the training process. (3) Avoid overfitting: By ensuring that the synthetic data is well-tuned on simpler examples before encountering outliers or more challenging data, there is potential to reduce overfitting. Specifically, better generulization here refers to the ability of models trained on the distilled datasots to perform well across a wider range of evaluation scenarios. However, avoiding overfitting particularly refers our curriculum strategy during the distillation process, where we use flexible region update in each iteration to prevent overfitting that could occur with a fixed region update.
4 Experiments
4.1
Datasets and Implementation Details
We verify the effectivencss of our approach on small-scale CIFAR-100 and various ImageNet scale datascts, including Tmay-ImageNet (Le & Yang, 2015), ImageNet-1KK (Deng el al. 2009), and ImageNet-211 [Ridnik] et al. 2021). For evaluation, we train models from scratch on synthetic distilled datasets and report the Top-1 accuracy on real validation datascts. Default lower and upper bounds of crop scales 13 and By are 0.08 and 1.0, respectively. The decay y 0.92. Curriculum Learning (CL) settings, the actual lower bound is dynamically adjusted to control difficulty, whereas the upper bound is fixed to the default value of 1.0 to
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Comparison with state-of-thh-art methods on various datasets.
Dataset
CIFAR-100
Ting-enageNet
ImageNet- IK
Net-gKN
100
100
200
20
Ratio (%)
10
10
20
0.8
16
0.8
1.6
DM
29.710.3
43.6.10.4
12.9.0.4
24.110.3
5.7.10.1
11.410.9
DSA
32.3:0.3
12.840.1
FRoPo
42.5+002
44.3+0.2
25.4+0.2
MTT
39.7=0.1
17.7=0.2
23 2=0.2
28 02000
DataDAM
34.8-005
49.4+0.3
18.7+033
7-.03
6,3+0.00
15.5+0022
TESLA
41.710.3
47.9.10.3
17.8.1..
27.91122
DATM
17.2=0.4
55.0220.2
31.110.3
39.720.3
Dataset
79.1
61.2
69.8
38.3
SRe"T.
23.5-0.8
51.4+0.8
17.740.7"
41.1+00.4
49.7+0.3
21.3++0.6
46.8+0.2
6288+0.4
57.0+0.3
18.5+0.2" 22.6:0.2
21.8+0.11
CDA (Ours)
19.8+0.6
64.4+0.5
21.3=0.3
48.7=0.1
53.2 53.2+0.1
33.5+0.3
53.5+0.3
68.0+0.2
63.3=0.2
26.4+0.1
Replicated experiment results are marked with ", while the other baseline results are referenced from original papers.
The full dataset resuits refer to Top-1 val accuracy achieved by a ResNet-18 18 model trained on the full dataset, the architecture is the SRc"r. and CDA.
ensure there is a probability of cropping and optimizing the entire image any progress. More details are provided in the Appendix B
4.2
CIFAR-100
Result
comparisons with baseline
meth-
Table Comparison on CIFAR-100.
ods,
including DM Zhao & Bilen
2023
CIFAR-100 (IPC)
DC
DSA
DM
MTT
SRe
Ours
DSA
Zhao & Bilen 2021), FRePo
(Zhou
et al.
2022). MTT (Cazenavette et al.)
2022b
1
12.8
13.9
11.4
24.3
13.4
DataD
DAM Sajedi et al.] 2023), TESLA LAA
[Cui
10
25.2
32.3
29.7
40.1
49.8
50
42.8
43.6
47.7
49.4
64.4
ct al.
2023). DATM (Guo et al., 2024), and
Yiin et al., 2023) on CIFAR- 100 are presented in Table 2 and Table 1 Our model is trained with an 800ep budget. It can be observed that our CDA validation accuracy outperforms all baselines under 10 and IPC. And our reported results have the potential to be further improved as training budgets increase. Overall, our CDA method is also applicable to small-scale dataset distillation.
SReaL
4.3 ImageNet
Results on the Tiny-ImageNe dataset are detailed in the second group of Table | and the first group of Table 4 Our CDA outperforms all baselines except DATM under 10 IPC. Compared to SRe"L, our CDA achieves average improvements 7.7% and 3.4% under IPC 50 and IPC 100 settings across ResNet-(11, 50, 101} validation models, respectively. Importanttly CDA stands as the inaugural approach to diminish the Top-1 accuracy performance disparity to less than 10% between the distilled dataset employing IPC 100 and the full Imag-lm, signifying breakthrough on this dataset.
4.4 ImageNet-1K
Table 3: Constant learning result. ay and a, stand for the ResizedCrop. represents the results from SRe'L implementation but following the setting in the table.
and max_crop parameters in Random-
Constant learning type \
0.08
0.2
0.4
0.6
0.8
1.0
Easy (oq = a, an = 3B (1.0))
44.90
47.88
46.34
45.35
43.48
41.30
Hard (a: Be (0.08), aa = a)
22.99
34.75
42.76
44.61
45.76
44.90
Constant Learning (CTL). We leverage a ResNet-18 and employ synthesized data with 1K recovery iterations. As observed Table 3. the results for exceedingly straightforward or challenging scenarios fall below the reproduced SR bascline accuracy of 44.90%, especially when a 2 0.8 in easy and a S 0.4 hard type. Thus, the results presented in Table 3 suggest that adopting larger cropped range assists in circumventing extreme scenarios, whether easy or hard, culminating in enhanced performance. A noteworthy
9

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: Comparison with baseline on various datasets.
ResNet-11
ResNet-50
ResNet-101
Dataset
IPC
SRe²L
Ours
S
Ours
SRe²L
Ours
50
41.1
48.717.
42.2
49.717.5
42.5
50.678.1
Tiny-IN
100
49.7
53.213.5
51.2
54.413.2
51.5
55.013.5
50
46.8
53.5
55.6
61.316.7
57.6
61.614.0
IN-1K
100
52.8
58.015.2
61.0
65.114.1
62.8
65.9
200
57.0
63.316.3
64.6
67.613.0
65.9
68.4
18.5
22.614.1
27.4
32,415.0
27.3
34.216.9
IN-21K
20
21.8
26.414.6
31.3
35.374.0
33.2
36.1 36.1 12.9
observation is the crucial role of appropriate lower and upper bounds for constant learning in boosting validation accuracy. This highlights the importance of employing curriculum data augmentation strategies in data synthesis.
Curriculum Learning (CL). We follow the recovery recipe of SRe2T.'s best result for 4K recovery iterations. As illustrated in Table | and the second group of Table 4 when compared to the strong baseline SRe²L CDA enhances the validation accuracy, exhibiting average margins of 6.1%, 4.3%, and 3.2% on ResNet- (18, 50, } across varying TPC settings. Furthermore, as shown in Figure [ the results achieve with our CDA utilizing merely 1K recovery iterations surpass those of SRe"L encompassing the entire 4K iterations. These results substantiate the efficacy and effectiveness of applying CDA in large-scale dataset distillation.
4.5 ImageNet-211 21K
Pre-training Results. Table 21 of the Appendix presents the accuracy for ResNet-18 and ResNet-50 ImageNet-21K-P, considering varying initial weight configurations. Models pre-trained by us and initial- ized with ImageNet-1K weight exhibit commendable accuracy, showing 2.0% improvement, while models initialized randomly achieve marginally superior accuracy. We utilize these pre-trained models to recover ImageNet-22X data and to assign labels to the synthetic images generated. An intriguing observation the heightened difficulty in data recovering from pro-trained models that are initialized randomly compared those initialized with ImageNeel-IK weight. Thus, our experiments employ CDA specifically pre-trained models that are initialized with ImageNet-Ile IK weight.
Validation Results. As illustrated in Table D and the final group of Table 4 we perform validation experiments on the distilled ImageNet-225 employing IPC 10 and 20. This yields an extreme compression ratio of 100x and 50x. When applying IPC 10, i.c., the models are trained utilizing a distilled datasct that is a mere 1% of the full dataset. Remarkably, validation accuracy surpasses 20% and 30% on ResNet-18 and approach attains
ResNet-(50, 101), respectively. Compared to reproduced TR TmageNet-21KK, elevation of 5.3% on average under IPC 10/20. This not only highlights the efficacy of our approach in maintaining dataset essence despite high compression but also showcases the potential advancements in accuracy over existing methods.
4.6 Ablations
Table 5: Applicability of our proposed mothod on SC-DD
Zhou et al. 2024a
Method
Tiny-ImageNe
ImageNet-IKK
SC-DD
45.5
53.1
SC-DD+Curiculun (Ours)
4..5
54.0
Applicability. Our method designed be applicable to general decoupled dataset distillation approaches. To demonstrate this, we apply our approach to the self-supervised SC-DD (Zhou et al. 2024b). As shown Table 5 our method achieves 1.0 and 0.9 improvements on Tiny-TmageNNt and ImageNet-1k respectively.
10

---

Published in Transactions on Machine Learning Research (11/2024)
Curriculum Scheduler. To schedule the global-to-local learning, present three distinct types of cur- riculum schedulers, step, linear, and cosine to manipulate the lower bounds data cropped augmentation. As illustrated in Figure 5 the dataset distillation progress is divided into two phases by a milestone. It is observed that both linear and cosine with continuous decay manifest robustness across diverse milestone configurations and reveal trend of enhancing accuracy performance when the milestone is met at later phase, as shown in Figure Morcover, cosine marginally outperforms lineor terms of accuracy towards the end. Consequently, we choose to implement the cosine scheduler, assigning milestone percentage of 1.0, modulate the minimum crop ratio adhering to the principles of curriculum learning throughout the progression of synthesis.
Step
Linear
Cosine
48
Top-1 Accuarcy (%)
47
46
45
44
43
42
41
40
50 60
70
80
Milestone Percentage (%)
Figure 6: Ablation study on three different schedulers with varied milestone settings. Each number is obtained by averaging repeated experiments with three different seeds. The figure shows that the difference between linear and cosine schedulers marginal and the best result for linear at the milestone of 90% while the cosine scheduler performs similarly better in the end. To avoid manually setting the milestone percentage for the linear scheduler, we adopt the cosine scheduler with a milestone percentage of 100% in our experiments.
Batch Size in Post-training. We perform an ablation study to assess the influence of utilizing smaller batch sizes the generalization performance of models when the synthetic data is limited. We report results the distilled ImageNet-21K from ResNet-18. In Table rise in validation accuracy is observed as batch size reduces, peaking at 16. This suggests that smaller batch sizes enhance performance on small-scale synthetic datasets. However, this leads to more frequent data loading and lower GPU utilization in our case, extending training times. To balance training time with performance, we chose a batch size of 32 for our experiments.
4.7 Analysis
Cross-Model Generalization. The challenge of ensuring distilled datasets gen- eralize effectively across models unseen during the recovery phase remains signifi- cant, as in prior approaches (Zhao et al., 2020; Cazenavelle al. 2022a),, synthetic images were optimized to overfit the recovery model. In the first group of Table we deploy our ImageNet-1KK 1K distilled datasots to train various validation models, and we attain over 60% Top-1 accuracy with most of these models. Additionally, our performance in Top-1 accuracy surpasses that of SRe?L, across all validation models spanning various architectures. Although DefT-Tiny not validated with comparable Top-1 accuracy other CNN models due to the ViT's inherent characteristic requiring more training data, CDA achieves double cross-model gen- cration performance the DeiT-Tiny validation model, compared with SRc*L. More validation models distilled ImageNet-1K are included in Table 19 of the Appendix. The second group of Table 7 supports further empirical substantiation of the CDA's efficacy in the distillation of large-scale ImageNet-21K datasets. The results demonstrate that the CDA's distilled datasets exhibit. reduced dependency on specific recovery models, thereby further alleviating the overfitting optimization issues.
Table 6: Ablation
batch size in validation.
Batch Size
Acc. (%)
128
20.79
21.85
22.54
16
22.75
22.41
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table Cross-model generation on distilled ImageNet- 1K with 50 IPC and ImageNet-21K with 20 IPC.
Validation Model
Dataset
Method
R18
R50
R101
DenseNet-121
RegNeetYY-SG ConvNeXt-Ting
DeiT-Tiny
SRe2L
46.80
55.60
57.60
49.74
60.34
53.53
15.41
IN-IK
CDA (ours)
53.45
61.26
61.57
57.35
63.22
62.58
31.95
S 21.83
31.26
33.24
24.66
34.22
34.95
15.76
IN-2IK
CDA (ours)
26.42
35.32
36.12
28.66
36.13
36.31
18.56
Impact of Curriculum. To study the curriculum's advantage on syuthetic image characteristics, we evaluate the Top-1 accuracy on CDA, SReeT. and real ImageNet-1K training set, using mean of ran- dom 10-crop and global images. We employ PyTorch's pre- traitra MobileNet-VV V2 to classify these images. As shown in Table g CDA im- ages closely resemble real ImageNet images in prediction accuracies, better than S. Consequently, curriculum data augmentation improves global image prediction and reduces bias and overfitting post-training on simpler, cropped images of SRe2L.
Table 8: Classification accuracy using MobileNet- V2.
Dataset
Top-1 (%)
SRe2L
CDA (ours)
Real
global cropped
79.34
81.25
82.16
87.48
82.44
72.73
Visualization and Discussion. Figure 7 provides a comparative visualization of the gradient synthetic images at recovery steps of {100, 500, 1K, 2K} illustrate the differences between SRe²L and CDA within the dataset distillation process. S im- ages in the upper line exhibit a significant amount of noise, indicating slow recovery progression in the carly recovery stage. On the contrary, due to the mostly entire image optimization stage, CDA images the lower line can establish the layout of the entire image and reduce noise rapidly, And the final synthetic images contain more visual information directly related to the target class Plant. Therefore, the comparison highlights CDA's ability to synthesize images with enhanced visual coherence to the target class, offering a more efficient recovery process. More visualizations are provided in Appendix D
the early
Figure 7: Syuthetic ImageNet-211K images (Pland).
Synthesis Cost. We highlight that there is no additional synthesis cost incurred our CDA to SRePL. (Yin ct al. 2023) under the same recovery iteration sctting. Specifically, for ImageNet-IK,, it takes about 29 hours to generate the distilled IetgeNet-1IK with 50 IPC on single A100 (40G) GPU and the peak GPU memory utilization is 6.7GB. For ImageNet-2IK,, it takes 11 hours to generate ImageNet- NeK images per IPC a single RTX 4090 GPU and the peak GPU memory utilization is 15GB. In our experiment, it takes about 55 hours to generate the entire distilled ImageNet-211 with 20 IPC on 4x RTX 4090 GPUs in total. Seleeting representative ive real images for input data initialization instead of the Gaussian noise initialization will be an effective way to further accelerate image distillation and reduce synthesis costs. can potentially reduce the number of recovery itcrations required to achieve high-quality distilled data and this approach could lead laster convergence and lower synthesis costs. Nevertheless, present our detailed training time several tested models, shown in Table g
Table 9: Detailed training time on different models.
Model
ResNet-18 2.3
ResNet-50
ReaNet-101
DeuseNet-121
RegNet-Y-8GE 13.6
ConvNeXt-Tiny 18.3
DerT-Tiny 4.6
Training time (A100 GPU hours)
7.1
12.5
9.1
4.8 Application: Continual Learning
The distilled datascts, comprising high-semantic images, possoss a boosted representation capacity compared the original datasets. This attribute can be strategically harnessed to combat catastrophic forgetting in continual learning. We have further validated the effectiveness of our introduced CDA synthesis within
12

---

Published in Transactions on Machine Learning Research (11/2024)
various continual learning scenarios. Following the setting introduced in SRe*L (Yin et al. 2023 we conducted 5-step and 10-step class-incremental experiments on Tiny-ImageNet, aligning our results against the baseline SRe²L and randomly selected subset on Liny-mmaeNee for comparative analysis. As illustrated in Figure our CDA distilled dataset notably surpasses SR exhibiting an average advantage of 3.8% and 4.5% on 5-step and 10-step class-incremental learning assignments respectively. This demonstrates the substantial benefits inherent in the generation of CDA, particularly in mitigating the complexities associated with continual learning.
Top-1 Accuracy (%)
55
Top-1 Accuracy (%)
55
50
50
45
45
CDA
CDA SRe²L
40
40
SRe²L
35
35
40
80
120
160
200
20
40
100 120 140 160 180 200 Number of classes
80
Number of classes
Figure 5/10-step dlass-incremental learning on Tiny-IN.
5
Conclusion
We presented a new framework focused on global-to-locul gradient refinement through curriculum data syn- thesis for large-scale dataset distillation. Our approach involves a practical paradigm with detailed pertaining for compressing knowledge, data synthesis for recovery, and post-training recipes. The proposed approach enables the distillation of ImageNet-21K to 50x smaller while maintaining competitive accuracy levels. In regular benchmarks, such ImageNet-1K and CIFAR-100, our approach also demonstrated superior per- formance, surpassing prior stale-of-the-ar methods by substantial margins. We further show the capability of our synthetic data on downstream tasks of cross-model generalization and continual learning. With the recent substantial growth in the size of both models and datasets, the critical need for dataset distillation on large-scale datasets and models has become inereasingly prominent and urgent. Our future work will focus on distilling more modalities like language and speech.
Limitations. Our proposed approach is robust generate informative images, while we also clarify that the quality of our generated data is not comparable to the image quality achieved by state-of-the-art generative models large-scale datasets. This difference is expected, given the distinct goals of dataset distillation versus generative models. Generative models aim to synthesize highly realistic images with detailed features, whereas dataset distillation methods focus on producing images that capt the most representative infor- mation possible for efficient learning in downstream tasks. Realism is not the primary objective in dataset distillation.
Acknowledgments
This research is supported by the MBZUAI-WIS Joint Program for AI Research and the Google Research award grant.
References
Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash, and Soheil Kolouri. One category prompt: Dataset distillation using diffusion models. arXie preprint orXiv:240300714 2024.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2024)
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun- Yan Zhu. Dataset distilla- tion by matching training trajectories. In Proceedings of the TEEE/CA CVF Conference on Computer Vision and Pattern Recognition, pp. 4750 4759, 2022a.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Yun Zhu. Wearable ima- genet: Synthesizing tileable textures via dataset distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, 2022b.
George Cazenavette, Tongzhou Wang, Autonio Torralba, Alexei A Efros, and Jun-Yar Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVE Conference Computer 2023.
Vision and Pattern Recognation, pp.
Mingyang Chen, Bo Huang, Junda Lu, Bing Li, Yi Wang, Minhao Cheng, and Wei Wang. Dataset distillation via adversarial prediction matching. arXiv preprint or XrXim:2312 2. 2.08912, 2023.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reducod search space. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognilion workshops, pp. 702 703, 2020.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenct- 1k with constant memory. In International Conference on Machine Leurning, pp. 6565 6590. PMLR, 2023.
Justin Cui, Ruochen Wang, Yuanhao Xiong, and Cho-Jui Hsich. Ameliorate spurious correlations in dataset condensation. In Forty-first International Conference on Machine Learning, 2024.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Tbrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480 7512. PMLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference compater vision and pullern recognilion, pp. 248 255. Teee, 2009.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. Xix preprint arXiv:2206.02: 02916, 2022.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint Xrviv:1708. 04555,, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXio:2010. 11929, 2020.
Jiawei Du, Qin Shi, and Joey Tianyi Zhou. Sequential subset matching for dataset distillation. Advances in Newral Information Processing Systems, 36, 2023.
Jianyang Gu, Saced Vahidian, Vyacheslav Kungurtsev, Haonan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. arXiv preprint arXiv:2311.15529 2023.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via aifficult--lignedd trajectory matching. In The Twelfth International Conference Learning Representations, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervisod visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 97389,977,110 2020.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. ICLR, 2024.
Elad Hoffer, Itay Hubara, and Danicl Soudry. Train longer, generalize better: closing the gencralization gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.0/4 04836, 2016.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102-1118 PMLR, 2022a.
Jang-Hy Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation vía efficient symthetic-data parameterization. In Proceedings of the 39th International Conference on Machine Learning, 2022b.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pp. 12352-12362 PMLR, 2022.
Haoyang Liu, Tiancheng Xing, Luwei Li, Vibhu Dalal, Jingrui He, and Haohan Wang. Dataset distillation via the wasserstein metric. arXiv preprint 1rXhw:2311.1855 2023.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in Newral Information Processing Systems, 35:1100 1113, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern 2022b.
recognition, pp.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXim:2200 12067, 2022.
Zhiheng Ma, Anjia Cao, Funing Yang, and Xing Wei. Curriculum dataset distillation. ai preprinl arXiv:2405.088 09150, 2024.
Aru Maekawa, Naoki Kobayashi, Kotaro Funakoshi, and Manabu Okumura. Dataset distillation with at- tention labels for fine-tuning bert. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 119-127, 2023.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jachoon Loc. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186 5198, 2021.
OpenAL. Gpt-4 technical report, 2023.
Tian Qin, Zhiwei Deng, and David Alvarez-Melis. Distributional dataset distillation with subtask decompo- sition. arXiv preprint arXiv:2403, 00999, 2024.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 10428-100000 10436, 2020.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Manor. Imagenet-211 21k pretraining for the masses. arXiv preprint ar Xiv:21004 10972, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the TEEE/CVF conference on compuler vision and pattern recognition, pp. 10684 10695, 2022.
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Konstantinos N. Pla- taniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 17097- 17107, October 2023.
Xinyi Shang, Peng Sun, and Tao Lin. Gift: Unlocking full potential of labels in distilled dataset al near-zero 2024.
cost. arXiv preprint or Nin:22405..14
Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. CVPR, 2024a.
Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. arXiv preprint 0rXiv22404117335 2024b.
Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In European Conference on Computer Vision, pp. 673 690. Springer, 2022.
Donghyeok Shin, Seungiae Shin, and II-Chul Moon. Frequency domain-based dataset distillation. Advances in Neural Informalion Processing Systems, 36, 2024.
Ili Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. Xiv preprint arXiv:2312.09522 2023.
Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the DEE//CVF Conference on Computer Vision and Pallern Recognilion (CVPR), 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.1095,, 2018.
Tongzhou Wang, Jun-Yan Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2020.
Yifan Wu, Jiawei Du, Ping Liu, Yuewei Lin, Wenqing Cheng, and Wei Xu. Dd-robustbench: An adversarial robustness benchmark for dataset distillation. arXiv preprint Niv:244003 13322, 2024.
Fric Xue, Yijiang Lí, Haoyang Liu, Yifan Shen, and Haohan Wang. Towards adversarially robust dataset distillation by curvature regularization. arXie preprint arXiv:2403. 10045, 2024.
Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspoctive. In NewrIPS, 2023.
Bo Zhao and Hakan Bilen. Datasct condensation with differentiable siameso augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching, In IEEE/CVE Winter Con- ference on Applications of Computer Vision, WACV 2023, Waikolon, HI, USA, January 2-7, 2029, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint ar Xrx:20006. 05929, 2020.
Binglin Zhou, Linhao Zhong, and Wentao Chen. Improve cross-architecture generalization on dataset distil- lation. Xiv preprint arXiv:2402. 13007, 2024a.
Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: good compression is all you need. arXie preprint arXiv:2404.07970 2024b.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Newral Information Processing Systems, 35:9813 9827, 2022.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A Datasets Details
We conduct experiments three ImageNet scale datasets, Tiny-ImageNet (Le & Yang 2015 ImageNel-
1K (Deng et al. 2009). and ImageNet-211 21K (Ridnik et al. 2021 The dataset details are as follows:
CIFAR-101 dataset composes 500 training images per class, each with a resolution of 32x pixels, across 100 classes.
Timy-ImageNet dataset is derived from ImageNet-IIe IK and consists of 200 classes. Within each category, there are 500 images with a uniform 64x64 resolution.
ImageNet-1K dataset comprises 1,000 classes and 1,281,167 images in total. We resize all images into standard 224x224 resolution during the data loading stage.
The original ImageNet-21K dataset an extensive visual recognition dataset containing 21,841 classes and 14,197,122 images. We use ImageNet- 21K-P (Ridnik et al., 2021 which utilizes data processing to remove infrequent classes and resize all images to 224x224 resolution. After data processing, ImageNet- 21K-P dataset consists of 10,450 classes and 11, 11,060,223 images.
B
Implementation Details
B.1 CIFAR-100
param-parreeter Setting. We train modified ResNet-18 18 model (He et al.
2020) on CIFAR-100 training
data with a Top-1 accuracy of 79.1% using the parameter setting in Table 10a
The well-trained model serves
as the recovery model under the recovery setting in Table [0b
Table 10: Hyper-parameter settings on CIFAR-100.
(a) Squeezing/
cannezzin//alidaatioon validation setting.
(b) Recovery setting.
config
value
config
value
optimizer
SGD
OBN
0.01
base learning rate
0.1
optimizer
Adam
momentum
0.9
base learning rate
0.25
weight decay
5e-4
momentum
31, 33 0.5, 0.9 100
batch size learning rate schedule
(squeeze) / (val) cosine decay
128
batch size
learning rate schedule
cosine decay
training epoch augmentation
200 (squeeze) / 800 (val)
recovery iteration
1,000
RandomResizedCCop
augmentation
RandomResizzodCrn
Due to the low resolution CIFAR images, the default lower bound B noods to be raisod from 0.08 (ImageNet setting) to higher reasonable value in order to avoid the training inefficiency caused by extremely small cropped areas with little information. Thus, conducted the ablation to select the optimal value for the default lower bound 3r in GandomFReszzdCCrog operations in Table I1 We choose 0.4 as the default lower bound 3, in Algorithm to exhibit the best distillation performance on CIFAR-100. We adopt a small batch size value of 8 and extend the training budgets in the following validation stage, which aligns with the strong training recipe on inadequate datasets.
Table 11: Ablation on the lower bound B sctting in distilling CIFAR-100.
default lower bound &
0.08
0.2
0.4
0.6
0.8
1.0
validation accuracy (800ep) (%)
58.5
62.14
64.0
63.36
61.65
54.43
18

---

Published in Transactions on Machine Learning Research (11/2024)
B.2 ImageNtt
Hyper-parameter Setting. We train a modified ResNet-18 model (He et al. 2020 Tiny-ImageNel training data with the parameter setting in Table [2a] and use the well-trained ResNet- 18 model with a Top-1 accuracy of 61.2% as a recovery model for CDA. The recovery sctting is provided in Table 12b
Table 12: Hyper-parameter settings Tiny-ImageNet.
(a) Squeezing/ boneeziing/valddateee validation setting.
(b) Recovery setting.
config
value
config
value
optimizer
SGD
OWN
1.0
base learning rate
0.2
optimizer
Adam
momentum
0.9
base learning rate
0.1
weight decay
le-4
momentum
B1, 32 0.5, 0.9 100
batch size
256 (squeeze) / 64 (val)
batch size
learning rate schedule
cosine decay
learning rate schedule
cosine decay
training epoch
50 (squeeze) 100 (val)
recovery iteration
4,000
augmentation
RandommessizedCrop
augmentation
RandomResizzedCrpp
Small IPC Setting Comparison. Table 13 presents the result comparison among our CDA, DM (Zhao & Bilen 2023 and MTT (Cazenavette et al. 2022b). Consider that our approach a decoupled process of
dataset compression followed by recovery through gradient updating. It is well-suited to large-scale datasets but less so for small IPC values. As anticipated, there no advantage when IPC value extremely low, such as IPC 1. However, when the IPC increased slightly, our method demonstrates considerable benefits accuracy over other counterparts. Furthermore, emphasize that our approach yields substantial improvements when alforded a larger training budgel, i.e., more training epochs.
Table 13: Comparison with baseline methods on Tiny-ImageNet.
Tiny-ImageNe IPC
DM
MTT
CDA (200ep)
CDA (400ep)
CDA (800ep)
3.9
8.8
2.38
0.08
2.82
0.06
3.29 + 0.26
10
12.9
23.2
30.41
1.53
37.41 0.02
43.04 + 0.26
20
43.93
0.20
47.76 0.19
50.46 + 0.14
50
24.1
28.0
50.26 + 0.09
51.52
0.17
55.50 + 0.18
Continual Learning. We adhere to the continual learning codebase outlined Zhao et al. 2020 and val- idate provided SRe²L and our CDA distilled Tiny-ImageNe dataset under IPC 100 as illustrated in Figure Detailed values are presented in the Table 14 and Table 15
Table 14: 5-step class-incremental learning on of Figure
This complements details the left subfigure
# class
40
80
120
160
200
L
45.60
48.71
49.27
50.25
50.27
CDA (ours)
51.93
53.63
53.02
52.60
52.15
Table 15: 10-step class- incremental learning on Tiny-Image NmageNet..
This complements details in the right
subfigure of Figure
# class
20
40
60
80
100
120
140
160
180
200
L
38.17
44.97
47.12
48.48
47.67
49.33
49.74
50.01
49.56
50.13
CDA (ours)
44.57
52.92
54.19
53.67
51.98
53.21
52.96
52.58
52.40
52.18
19

---

Published in Transactions on Machine Learning Research (11/2024)
B.3 ImageNet- 1K
Hyper-parameter Settings. We employ PyTorch off-the-shelf ResNet-18 and DenseNett-111 with the Top-1 1 accuracy of (69.8%, 74.4%) which are trained with the official recipe in Table 16a And the recovery settings are provided in Table T6c. and it is noteworthy that we tume and set distinct parameters OBN and learning rate for different recovery models in Table 16d Then, we employ ResNet-(18, 50, 101, 152] (He et al. 2016), DenseNel-121 (Huang et al. 2017], RegNet. (Radosavovic et al. 2020). ConvNeXi (Liu et al.,
2022b), and DeiT--Tiny (Touvron et al., (021) as validation models evaluate the cross-model generalization distilled ImageNet-1Ih dataset under the validation sctting in Table 16b
Table 16: Hyper-parameter settings on ImageNet-IK.
(a) Squeezing setting.
(b) Validation setting.
config
value
config
value
optimizer
SGD
optimizer
AdamW
base learning rate
0.1
base learning rate
1e-3
momentum
0.9
weight decay
1c-2
weight decay
le-4
batch size
128
batch size
256
learning rate schedule
cosine decay
Ir step size
30
training epoch
300
Ir gamma
0.1
augmentation
RandomResizeedrrpp
training epoch
90
augmentation
RandommeesizedCrrp
(c) Shared
recovery setting.
(d) Model-specific recovery setting.
config
value
config
ResNet-18
DenseNet-122
optimizer
Adam
QBN
0.01
0.01
momentum
81. B2 0.5, 0.9 100
base learning rate
0.25
0.5
batch size
recovery iteration
1,000
4,000
1,000
learning rate schedule
cosine decay
augmentation
RandomResizeddrrp
Histogram Values. The histogram data of ImageNet-1K comparison with SRe2L in Figure D conveniently found in the following Table 17 for reference.
can be
Table 17: ImageNet-1K comparison with SR This table complements details in Figure 1
Method Validation Model
ResNet-18
ResNet-50
ResNet-101
DenseNet-121
RegNet- Y-8GF
SRe2L (4K)
46.80
55.60
57.59
49.74
60.34
Our CDA (IK)
52.88
60.70
61.10
57.26
62.94
Our CDA (4K)
53.45
61.26
61.57
57.35
63.22
To conduct the ablation studics efficiently Table 3 Table 22 and Figure 0 we recover the data for 1,000
iterations and validate the distilled dataset with a batch size of 1,024, keeping other settings the same Table 16 Detailed values of the ablation study schedulers provided in Table 18
20

---

Published in Transactions on Machine Learning Research (11/2024)
Table 18: Ablation study on three different schedulers with varied milestone settings. complements details in Figure
Scheduler \
Milestone
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Step
45.46
46.08
46.65
46.75
46.87
46.13
45.27
44.97
42.49
41.18
Linear
45.39
46.30
46.59
46.51
46.60
47.18
47.13
47.37
48.06
47.78
Cosine
45.41
45.42
46.15
46.90
46.93
47.42
46.86
47.33
47.80
48.05
Cross-Moder Generalization. To supplement the validation models distilled ImageNet- 1K in Table 7 including more different architecture models to evaluate the pross-architecturg performance. We have con- ducted validation experiments on a broad range of models, including SqueezeNet, MobileNet, Efficient.Ne,, MNASNet, ShuffleNet, ResMLP, AlexNet, DeiT-Base, and VGG family models. These validation models are selected from a wide variety of architectures, encompassing a vast range of parameters, shown Table I9] In the upper group of the table, the selected models are relatively small and efficient. There is a trend that its validation performance improves validated earlier models AlexNet and VGG. These models also show trend of performance improvement with increasing size, but due to the simplicity of carly model architectures, such as the absence of residual connections, their performance is inferior compared to more recent models. Additionally, we evaluated our distilled dataset on ResMLP, which is based on MLPs, and the DeíT-Bass model, which is based on trans- formers. In summary, the distilled dataset created using our CDA method demonstrates strong validation performance across wide range of models, considering both architectural diversity and parameter size.
the number of model parameters increases. In the lower group,
Table 19: ImageNet-Ike 1K Top-1 on cross-model generation. Our CDA dataset consists of 50 IPC.
Model
SqueezeNet
MobileNet
EfficientNet
MNASNet
ShuffleNet
ResMLP
#Params (M)
1.2
3.5
5.3
6.3
7.4
30.0
accuracy (%)
19.70
49.76
55.10
55.66
54.69
54.18
Model
AlexNet
DeiT-Bass
VGG-11
VGG-13
VGG-16
VGG-19
# Params (M)
61.1
86.6
132.9
133.0
138.4
143.7
accuracy (%)
14.60
30.27
36.99
38.60
42.28
43.30
B.4 ImageNet-21K
Hyper-parameter Setting. ImageNet-2IK-P (Ridnik et al. 2021 proposes two training recipes train ResNet-[18,, 50} models. One way is to initialize the models from well-trained ImageNet-1K weight and train Ima for 80 epochs, another is to train models with random initialization for 140 epochs,
shown in Table 20a] The accuracy metrics on both training recipes are reported in Table 21] In our experiments, we utilize the pre-trained ResNet-(18, 50} models initialized by ImageNet- 1K weight with the Top-1 accuracy of (38.1%, 44.2%} as recovery model. And the recovery setting is provided in Table 20c Then, we evaluate the quality of the distilled ImageNet-211K dataset on ResNet-(18,, 50, 1013 validation models under the validation setting in Table 20b To accelerate the ablation study on the batch size setting in Table 6 we train the validation model ResNet-18 for 140 epochs.
C Reverse Curriculum Learning
Reverse Curriculum Learning (RCL). We use reverse step scheduler in the RCL experiments, starting with the default cropped range from 81 to 3n and transitioning at the milestone point to optimize the whole image, shifting from challenging to simpler optimizations. Other settings follow the recovery recipe on ResNet-18 for 1K recovery iterations. Table 22 shows the RCL results, smaller step milestone indicates an earlier difficulty transition. The findings reveal that CRL does not improve the generated dataset's quality compared to the baseline SRe2L,, which has 44.90% accuracy.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Table 20: Hyper-parametee settings on ImageNet-2IK.
(a) Squeezing setting.
(b) Validation setting.
config
value
config
value
optimizer
Adam
optimizer
AdamW
base learning rate
30-4
base learning rate
2c-3
weight decay
lc-4
weight decay
lc-2
batch size
1,024
batch size learning rate schedule
32
learning rate schedule
cosine decay
cosine decay
label smooth
0.2
label smooth training epoch
0.2
training epoch
80/140
300
augmentation
CutoutPIL,
augmentation
CutoutPII,
Rugmett
RandomnesizeedCrgg
(c) Recovery setting.
config
value
OBN
0.25 Adam
optimizer
base learning rate
0.05 (ResNet-18), 0.1 (ResNet-50) B1, B2 0.5, 0.9
momentum
batch size
100
learning rate schedule
cosine decay
recovery iteration
2,000
augmentation
RandommesizedCCopp
Table 21: Accuracy of ResNet-(18,, 50} on Imag
Model
Initial Weight
Top-1 Acc. (%)
Top-5 Acc. (%)
ImageNet-1K
38.1
67.2
ResNet-18 (Ours)
Random
67.8
Ridnik al.
2021
ImageNet-1K
42.2
72.0
ImageNet-1K
44.212.0
74.612.6
ResNet-50 (Ours)
Random
44.5
75.1
Table 22:
Ablation of reverse curriculum learning.
Step Milestone
Accuracy (%)
0.2
41.38
0.4
41.59
0.6
42.60
0.8
44.39
D
Visulization
We provide additional comparisons of four groups of visualizations on synthetic ImageNet-21K, images recovery steps of {100, 500, 1,000, 1,500, 2,000} between SRe*L (upper) and CDA (lower) in Figure g The chosen target classes are Benthos, Squash Rackets, Marine Animal, and Scavenger.
In addition, present our CDA's synthetic ImageNet-1K NK images in Figure 10 and Net-211 images in Figure I1 and Figure I2
22

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 9: Synthetic ImageNet-21KK data visualization comparison.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 10: Synthetic ImageNet-IK 1K data visualization from CDA.
24

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 11: Synthetic ImageNet-21K data distilled from ResNet-11 by CDA.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 12: Synthetic ImageNet-211K data distilled from ResNet-50 by CDA.
26


Reviews:
[{"review_id": "Z1DqboC6LZ", "reviewer": "Reviewer_Rerr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Although CDA is the only major technical modification, it is effective and works well across datasets and architectures. It's interesting to know that a simple change in curriculum suffices to provide good performance.\n- The paper provides thorough experiments and ablation studies.\n  - This includes comparison at various Image-per-class (IPC), curriculum scheduling (or ranges for constant cropping ratios), and batch sizes.\n- It discusses the computation cost for synthesis.\n\nWeakness: My main complaint is that the method is highly specific to the baseline SRe$^2$L, which is the only baseline in this work.\n- The paper mentions several related works at the end of the related work section. Could you provide a brief description on these works, and the reason why these methods are not compared against?\n- Are there other baselines that can achieve comparable performance, and if yes, can the curriculum idea be applied to these baselines?\n\nSome other comments:\n- The title should be made more specific. Currently the title is overly broad and vague.\n- Eq (3): why taking the sup rather than the average?\n- In the paragraph of \"Relabel: Post-training on Larger Models with Stronger Training Recipes.\", the paper mentions that another prior method TESLA has a decline in accuracy when scaled up. Could you comment on why?\n- Fig 4: Each step has 2 boxes, one with solid borders, one with dashed borders. Are the two boxes of the same type (i.e. simply 2 crops), or does the border type signal anything?\n- At the end of section 3, the paragraph of \"Advantages of Global-to-local Synthesis\": this paragraph mentions 3 advantages of the proposed CDA: (1) stabilized training, (2) better generalization, and (3) avoiding overfitting. Aren't \"better generalization\" and \"avoiding overfitting\" the same?\n- Are $\\beta_l, \\beta_u$ used other than in Table 2? If not, please remove these notations and use 0.08, 1 directly.\n- The ablation on the curriculum schedule (i.e. Fig 6): I wonder how much difference linear vs cosine schedules have -- how many seeds are used in producing the bars in Fig 6? Also, it seems that the best performance is from Linear at 90%.\n- There's a minor typo next to Table 6.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to improve large-scale dataset distillation using a proposed curriculum data augmentation (CDA), which gradually increases the training difficulty by lowering the minimal crop ratio.\n\nThe rest of the method is the same as a prior method SRe$^2$L. There are 3 steps:\n- **Squeeze**, where a model is trained on the real dataset and frozen in the following stages. This step follows from the practices in SRe$^2$L.\n- **Recover**, which is the synthesizing step where images are generated using gradient information. The motivation for CDA is that we want to refine the gradients from global to local throughout training.\n  - For CDA, the paper considers 2 scheduling for the curriculum (i.e. linear and cosine), and searches over the length of the curriculum (called the \"milestone\").\n- **Relabel**, where a model is . The paper finds that small-batch training helps improve the performance, especially when the data is limited.\n\nThe paper shows that CDA improves over the baseline SRe$^2$L across datasets (CIFAR-100, Tiny-ImageNet, ImageNet-1K and 21K) and models (ResNet-18, 50, 101, DenseNet, RegNet, ConvNeXt, DeiT). The paper provides ablation study on the effects of curriculum scheduling and batch size.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n- In \"Synthesis Cost\", could you add a comment comparing the synthesis time to the training time of the models being tested?\n- We may want to trade off quality versus efficiency. Could you comment on how the quality of the pretrained model and the number of recovery iterations affect the quality of the generated data? For example, why Table 7(b) uses 1000 recover iterations?\n- Could you comment on potential ways to reduce the synthesis cost? For example, would selecting (rather than randomly sampling) the images per class provide improvements?\n- Table 1: What would the results be for higher IPC (e.g. 32%, 64%)? I’m wondering whether there’s a smooth growth in performance w.r.t. IPC. Please treat this question as optional since it's expensive to generate more images.", "broader_impact_concerns": "There are no direct societal or ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wedp05LX1Z", "reviewer": "Reviewer_G33W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is very well-written and easy to understand.\n\n* The proposed method significantly outperforms previous dataset distillation methods.\n\n* It is the first work to validate on a large dataset (i.e., ImageNet-21K).\n\n* The authors perform a good analysis of their method, including an application to Continual Learning.\n\n**Weaknesses** \n\nThe authors have done a wonderful job explaining and conveying their methods, and the method has been validated with rigorous experiments. Some weaknesses related to the formatting is as follows: \n\n* Although it has been mentioned in related works of the different categories of previous methods, it would be great to reiterate why SRe2L and CDA are tied together in Table 1 or somewhere in the Experiments Section (Section 4). \n\n* It would be good to highlight that Figure 4 is the central figure for explaining the proposed method, with detailed captions provided to enhance understanding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of dataset distillation, which involves generating a smaller yet representative subset from a large dataset to enable efficient model training while maintaining strong performance on the original testing data distribution. Previous methods have struggled with improper gradient update strategies, leading to a decline in the quality of the distilled datasets. To overcome this, the authors propose a novel global-to-local gradient refinement approach that is enhanced by curriculum data augmentation (CDA) during the data synthesis process. This method significantly improves the quality of the distilled datasets, achieving state-of-the-art accuracy on both ImageNet-1K and ImageNet-21K benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the Weaknesses above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "axUKyhSsGb", "reviewer": "Reviewer_2jfx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The proposed curriculum based Random cropping is simple and works nicely at scale.\n1. The quantitative evaluation is extensive which helps clarify a lot of questions.\n\n**Weaknesses**:\n\n**Limited methodological novelty**: Apart from the proposed data augmentation setup, most of the framework for learning distilled samples has already been proposed in prior work. Though for TMLR, I don't think this is a major problem per se.\n\nSee Requested Changes for other questions/queries.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Claims and Contributions**: The paper introduces a method for dataset distillation with a special focus at scale using the ImageNet 21k dataset. More specifically, the authors propose curriculum data augmentation (CDA) which utilizes modulated random cropping during the synthesis stage of dataset distillation to better model the global and local structures in dataset samples. Empirically, the proposed method achieves highest accuracy on large scale ImageNet 1k and 21k datasets and outperforms competing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Rephrase**: The authors mention: “ Moreover, by working with distilled datasets, there is potential to alleviate some data privacy concerns, as raw, personally identifiable data points might be excluded from the distilled version”. This seems inaccurate as the representative subset might still violate privacy concerns. It might be worth mentioning approaches which utilize recent large scale generative models to synthesize novel samples (not a dataset subset) which have the potential to mitigate these concerns to a large extent (although these methods have their own set of challenges).\n\n1. **Clarification on Quantitative Results**: For the quantitative results in Table 1, are the baselines also evaluated using the improved Squeeze network training as discussed in Section 3.2? If not, can the authors show an ablation of CDA applied to the ResNet 18 model for a small dataset like CIFAR-100 or Tiny ImageNet and compare it with other baselines for fair comparison?\n\n1. The authors mention: “ We have also observed that maintaining a smaller batch size is crucial for post-training on synthetic data to achieve commendable accuracy. This is attributed to the Generalization Gap”. This is not obvious to me. Can the authors elaborate more on this in the main text?\n\n1. While the quantitative results are good, the visualizations in Fig. 7 indicate that the quality of samples even with 2k recovery steps is not comparable to the sample quality achieved by state-of-the-art generative models on large scale datasets. I would highly recommend the authors to include a note in the conclusion highlighting some of the limitations/strengths as compared to generating datasets using powerful image generation models.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the authors' revision, most of the reviewers' concerns have been addressed. The reviewers unanimously recommend acceptance of the paper.

I find the work valuable and recommend its acceptance and publication.
Decision: Accept (poster)