Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality
Authors: Xuxi Chen
Yu Yang
Zhangyang Wang
Baharan Mirzasoleiman
Venue: ICLR 2024 poster (2024)
Keywords: dataset distillation
dataset condensation
Primary area: optimization
TL;DR: We propose a multi-stage dataset distillation framework to improve the quality of synthetic samples.

Abstract: Dataset distillation aims to minimize the time and memory needed for training deep networks on large datasets, by creating a small set of synthetic images that has a similar generalization performance to that of the full dataset. However, current dataset distillation techniques fall short, showing a notable performance gap compared to training on the original data. In this work, we are the first to argue that the use of only one synthetic subset for distillation may not yield optimal generalization performance. This is because the training dynamics of deep networks drastically changes during training. Therefore, multiple synthetic subsets are required to capture the dynamics of training in different stages. To address this issue, we propose Progressive Dataset Distillation (PDD). PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets without requiring additional training time. Our extensive experiments show that PDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%. In addition, our method for the first time enables generating considerably larger synthetic datasets. Our codes are available at https://github.com/VITA-Group/ProgressiveDD.

Published as a conference paper at ICLR 2024
DATA DISTILLATION CAN BE LIKE VODKA: DISTILL- ING MORE TIMES FOR BETTER QUALITY
Xuxi Chen"), Yu Yang*? Zhangyang Wang', Baharan Mirzasoleiman' University of Texas at Austin Uniivrriiy of California, Los Angeles fxxchen, at laswang] Butexas. . edu, (yuyang, vaharan)ecss ucla. eduu
ABSTRACT
Dataset distillation aims to minimize the time and memory needed for training deep networks on large datasets, by creating small set of synthetic images that has similar generalization performance to that of the full dataset. However, current dataset distillation techniques fall short, showing a notable performance gap compared to training on the original data. In this work, we are the first argue that the use of only one synthetic subset for distillation may not yield optimal generalization performance. This is because the training dynamics of deep networks drastically changes during training. Therefore, multiple synthetic subsets are required capture the dynamics of training in different stages. To address this issue, we propose Progressive Dataset Distillation (PDD). PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets without requiring additional training time. Our extensive experiments show that FDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%. In addition, our method for the first time enables generating considerably larger synthetic datasets. Our codes are available at hhtpp:: /ttphbggtttt..t//. com//III TA hrap:///rroooop/rrrrrssiiiaa
INTRODUCTION
Dataset distillation aims to generate very small num- ber of synthetic examples from a large dataset, which can provide a similar generalization performance to
Our (PDD-MTT)
74
Binglowlaga (MIT)
TestAcouray
72
that of training on the full dataset (Wang et al. 2018 70
Loo et al. 22022 Nguyen et al.) 2202a by Zhou et al. 2022b. If this can be achieved, it can significantly re-
duce the costs and memory requirements of training a deep network on large datasets. Therefore, dataset distillation has gained a lot of recent interest and has found various applications, ranging from contin- uous learning, neural architecture search, privacy- preserving MI. (Zhao et al. 2021 Dong et al., 2022).
30
50
Number of imagaa Per Class (ICC)
Figure 1: Our multi-stage dataset distillation framework, PDD, improves the state-of- the- art algorithms by iteratively distilling smaller synthetic subsets that together capture longer training dynamics on full data. In the setting shown in the figure, PDD uses MTT as base distillation method to incrementally generate synthetic subsets of size 10, where each subset captures 15 epochs of training full data. This yiclds much better performance compared to generating the same number of images (IPC) in a single stage, i.e., based on the first 15 epochs of training on full data.
Existing dataset distillation methods generate set of synthetic examples that match the gradient (Zhao al.] 2021 Zhao & Bilen 2021b), Neural Tan- gent Kernel (NTK) (Loo et al., [2022] Nguyen et al.
2021a b Zhou et al., 2022). or weights (Kim et 2022) of number of randomly initialized models being trained on the original (Zhao et al.) (02T) or synthetic data (Zhao& & Bilen. 2021b). However, as matching the entire training dynamics intractable, existing methods only match the dynamics of early training iterations, as short the first four epochs (Zhao et al. 2021). As the training dynamics of deep networks drastically changes during training. synthetic subset generated based the early training dynamics cannot represent the dynamics
"The first authors contributed equally.

---

Published as a conference paper at ICLR 2024
of the later training phases. Hence, existing dataset distillation methods suffer from substantial performance gap to that of training on the original data (Zhao et al.
2021 Kim et al.) 2022).
Recent results on optimization and generalization of neural networks revealed that gradient methods have an inductive bias towards learning simple functions, especially early in training (Kalimeris et al.
2019. Hu et al. 2020 Herrmnn & Lampinen 2020 Neyshabur al. 2014 Shah et al. (200)). That is, models trained with (stochastic) gradient methods learn nearly linear functions in the initial training iterations (Kalimeris al., 20199] Hu al., 20020) As iterations progress, SGD learns functions of increasing complexity (Kalimeris et al. 2019). This implies that synthetic examples generated based on early training dynamiis can only train low-compyyx neural networks that perform well on easy examples that are separable by lomplexyyy models. This limitation is further supported by recent studies (Pooladzandi et al. 2022] Yang et al. 2023). which observed that deep models benefit the most from learning examples of increasing difficulty levels at various training stages and one subset of the training data is not enough to support the entire training.
Building on this observation, to bridge the gap training on the full data, is crucial to synthesize examples that can capture the dynamics of later training phases. However, this is very challenging. First, synthesizing examples that match the training dynamics of many randomly initialized networks over longer training intervals has a very high computational cost. Moreover, capturing complex train- ing dynamics over longer intervals requires synthesizing more images, which makes it prohibitively expensive. Finally, even if a larger subset can be generated to match the dynamics of a longer training interval, it is not enough to bridge the gap training on the full data.
In this work, we address the above challenges by proposing a Progressive Dataset Distillation (PDD) pipeline. We are the first to employ multi-stage idea: specifically, to generate multiple synthetic subsets that can capture the training dynamics in different phases. To do so, we synthesize examples that can capture the training dynamics of the full data in given training interval. Then, we train the model on the synthesized examples and generate another set of synthetic examples that, together with the previous synthetic sets, capture training dynamics of the full data in the consecutive training interval. Importantly, our progressive distillation approach captures the training dynamics of neural networks more effectively, resulting in superior generalization performance. Figure confirms that by distilling the dynamics of the later training stages on CIFAR-10 10, PDD effectively improves the performance when training on the distilled data.
Our extensive experiments confirm that our multi-stage distillation approach outperforms existing methods by up to 5% on ConvNet and 5% for cross-aree aros--achitecture generalization to ResNet- 10 and ResNet-18. Remarkably, PDD is the first method to enable generating larger synthetic datasets. In doing so, it considerably bridges the gap to training on the full data by achieving 90% of the full accuracy with only 5% of the full data size CIFAR-10 10 and 8% of full data size CIFAR- 100 accuracy on Imag-ema (Le & Yang 2015).
(Krizhevsky et al. 2009) and provides
We also conduct studies showing that: 1) our multi-stage synthesis framework achieves consistent improvement if new intervals are introduced, which confirms its ability to serve as an effective base method for further condensation (Liu et al.) 2023a) where the budgets (i.e., number of images) are subjected to change; 2) our framework generates synthetic samples with strong generalization ability across various architectures; 3) the distillation process can be performed on progressively challenging subsets of the full data at each stage, resulting in minimal performance degradation.
2 RELATED WORKS
Dataset Distillation (DD) (Wang et al. 2018) aims to generate a synthetic subset from a large training data that can achieve a similar generalization performance to that of training on the full dataset, when trained on. To achieve this, DD adopted an optimization process comprising two nested loops. The inner loop involves training a model using the synthesized data until it reaches convergence, while the outer loop aims to optimize the synthetic data such that the trained model generalizes well on the original dataset. More recent studies (Loo et al. 2022 Nguyen et al. 202 102lap Zhou et al.) 2022) leverage the same framework but use kernel methods, such as Neural Tangent Kernel (NTK), to approximate the inner optimization in a closed form. While kernel-based algorithms achieved higher accuracy than DD (Wang et al. (0188) on networks that satisfy the infinite- aafinite-width width assumption, they do not work well in practice as the constant NTK assumption does not generally hold.
Another set of methods relies on gradient matching. In particular, DC (Zhao et al., (021) minimizes the distance between the gradients of the synthetic and original data on the network being trained
2

---

Published as a conference paper at ICLR 2024
on the synthetic data. DSA (Zhao & Bilen 2021b) improves upon DC by applying differentiable siamese augmentations to both the original and synthetic data while matching their training gradients. Incorporating differentiable data augmentation has been adopted by almost all subsequent studies, Later on, IDC (Kim et al. 2022) proposed a multi-formulation framework to generate more augmented examples from the same set of synthetic data, to boost the performance with the same storage budget. The synthetic data is generated by minimizing the distance between the gradients of the synthetic and original data on the network being trained on the full data. Concurrent to our work, [enng al. (2024) proposes to unroll gradients within a randomly picked small window along the training trajectory and generate nested synthetic datasets of different IPCs.
Besides matching the gradients, other methods involve matching the training trajectories of the network parameters (Cazenavette et al., (2022) or the data distribution (Wang et al. 2022 Zhao & Bilen 2023). MTT (Cazenavette et al. (022)) pre-computes and stores training trajectories of expert networks trained on the original data, and then minimizes the distance between the parameters of the
network trained on the synthetic data and the expert networks. CAFE (Wang et al. 2022) matches the features between the synthetic and real data in all intermediate layers. To avoid the expensive bi-level optimization, DM (Zhao & Bilen 2021a) minimizes the distance between feature embeddings of the synthetic and real data based on randomly initialized networks. More recently, HuBa (Liu al.] (2022) proposed to distill dataset into two components, Bases and Hallucination increase the representation capability of distilled datasets. IT-GAN (Zhao & Bilen (022) inverted the training samples into latent spaces and further fine-tuned towards distillation objective, GLaD (Cazenavette iet al. 2023) used generative adversarial networks as prior to help the cross- cross-architecture generaliza- tion. More recently, DREAM (Liu et al. 2023b) selects representative samples during matching to improve the performance and effciency of dataset distillation, and DATM (Guo et al. 2024) aligns the difficulty of the generated patterns with the size of the synthetic dataset.
Most existing works generate a set of synthetic examples that match the dynamics of neural networks during early-training stage or at multiple random initializations. In contrast, we show that progres- sively generating multiple synthetic subsets to match the training dynamics in different stages of training yields superior performance.
3 PROBLEM FORMULATION AND PRELIMINARY
Given large dataset T {( vi)) consisting of [T] samples from C' classes, data distillation aims to learn synthetic set = {(,,, }:)} with synthetic samples |S] so that deep neural networks can be trained in S and achieve comparable generalization performance to those trained on T. Formally,
(E)) (z). v)] = EE- (
[[(ps (z), v)].
(1)
where P(D) is the real data distribution, G07 (.) and (0s (.) are models trained on T and S respectively. _(...) is the loss function, e.g., cross-entropy loss.
State-of-tue the-art dataset distillation methods condense the real dataset into small synthetic set by matching the gradient of full data along the synthetic or real trajectory, expressed as follows:
arg min Ee Peo S
Σ D(VoL=(())) (())). 2=
(2)
where 0, denotes the model parameters, and D computes distance between the gradients. DC (Zaa et al. 2021 and DSA (Zhao & Bilen 2021b) update 0: by minimizing the loss L (0.) on the synthetic data. TDC (im et al. | (022)) showed that updating O, by minimizing the loss LT (0.) on the full data yiclds superior performance. Matching the gradient of the augmented version of the training and
synthetic examples further improves the performance (Zhao al. 202 Kim et al., 2022).
Alternatively, MTT (Cazenavette et al.) 2022) trains two models on synthetic and real data and matches weight trajectories OitN of length N when training the model synthetic data S with weight trajectories 0H of length M > N when training the model on real data T:
0 0 Maalll
arg
min argmin
0,
0 M+Mll 12
Existing dataset distillation methods synthesize examples based on the gradients or weights of the
models during the initial training epochs (Cazenavette et al. 2022 Kim et al. 2022), match outputs
3

---

Published as a conference paper at ICLR 2024
Stage
Transtion
Stage
Transition
Stage
PDD Output
Sp
S2
S₂
Optimizabre
S3
S truin
S
S
S
Fixed
(9..
distiilie,
train
(9s T)
S₂
S1
0
S2
2
S3
S
An illustration of the proposed Progressive Dataset Distillation (PDD) framework. consists of multiple distillation stages and transitions in between. In each distillation stage, we distill set of images conditioned images synthesized in the previous stages. In transitions, train models on all synthesized images so far, as the starting weights for the next distillation stage to capture longer training dynamics. Our framework can be applied any dataset distillation algorithm.
Figure 2:
of multiple randomly initialized models (Zhao & Bilen, 2021a). The most successful methods, synthesize examples that capture the training dynamics of models trained on full data (Kim et al.) 2022, Zhao et al. 202I). However, they only capture the early training dynamics. For example, IDC (Zhao et al. 202T) and MIT (Kim et al.) 2022) synthesize examples by matching gradients and weights of the first and 15 epochs of a 200 training pipeline respectively, when distilling CIFAR-10 10 and CIFAR-100 This is because matching weights gradients over longer intervals becomes computationally difficult and does not yield high-quality synthetic data. This introduces a performance gap that of training the original data.
4 PROGRESSIVE DATASET DISTILLATION (PDD)
Next, we introduce our Progressive Dataset Distillation (PDD) framework to generate synthetic images that match the training dynamics of different stages of training.
4.1 DISTILLING MULTIPLE TRAINING STAGES
To capture the learning dynamics of different training stages, our key idea, shown in Fig-
Algorithm Progressive Dataset Distillation (PDD)
2 is to generate a sequence of small syn- thetic datasets Si, S2. ... .Sp, so that each synthetic dataset S captures the training dy- namics in the full data a different stage training. Then at test time, when the model is trained on the synthetic images, we can train the model different subsets mimic differ- ent stages of training on full data.
Input: A dataset distillation algorithm A, full training set T
Output: Model trained series of synthetic
datasets: S₁, S2,...... Sy Generating synthetic subsets: PDD
So < 0 Initialize O randomly for 2, ..., P do
S, A(0,, T 7 8,) O, arg ming C(B (,,,,,-),,,0--)1 end for Evaluation: Training on the PDD subsets Initialize On randomly for = 1, .P do
However, naively dividing the full training pipeline into P intervals and generating a sub- based the training dynamies of each interval does not yield a satisfactory perfor- mance, due the following reasons. First, generating different synthetic subsets indepen- dently results capturing redundant informa- tion in the subsets and does not improve the performance, Second, since the synthetic sub- sets are small, at test time when the model is trained on the synthetic images, minimizing the loss on subset S results in forgetting the previously learned information from subsets -.. S,. Finally, if forgetting changes the training loss and interrupts the optimization pipeline. This does not allow the model to learn well from multiple synthetic subsets.
0, arg min, C(B, ,,0,,,=
end for
be prevented, transitioning from training on S, to training on S test time
4

---

Published as a conference paper at ICLR 2024
To address the above issues, synthesize each subset St based on the dynamics of training on the full data at stage i, conditioned on the previous subsets S1, S2, ........ S.. That is, we generate S, such that S U S2 U... U S; captures the training dynamics at stage i. Note that only synthesize S; at interval T3 while keeping S,,S2. .... S fixed. This prevents capturing redundant information in subset St that are already captured by previous subsets S1, S2, ,S Next, to address the discontinuity in training on multiple subsets, we synthesize every subset S; based on the training dynamics of full data starting from parameters 0,, where training on S US2 U... ....USS, S is finished. This allows smooth transitioning between different subsets when training on the synthetic data. Finally, at test time when the model trained on the synthetic subsets, to prevent forgetting the information learned from the previous subsets, we first train the model on Si, then Si U S2, and keep training on the union of the previous subsets addition to the new one S₁ U S2 U.,
We summarize our pipeline in Algorithm Formally, for i 1, P. we generate a synthetic subset S, as follows:
S A(0,. T| T/ S2)
s.t.
0: arg min C(,, 1 S 0,,),-)),
where L( S, 1) is the loss of the model trained on data S starting from 0,-1. A can be any dataset distillation method, such as DC (Zhao et al. (02I), DSA (Zhao & Bilen 20216). IDC (Zhao)
ctt al. 202I), and MTT (Kim et al.] 2022). described in Equation and
Distillation and training costs Note that conditioning the distillation on previous subsets does not increase the cost of synthesizing a new subset, as we generate the same number of synthetic images at every interval. On the other hand, at test time, we train on fewer images in total. This is because instead of training on k = S] synthetic examples during the entire training, PDD with m intervals first trains the model on k/m synthetic images. Then, it trains the model on / synthetic images and keeps increasing the number of training examples until it trains on k examples at the final interval.
4.2 DISCARDING EASIER-TO- EXAMPLES AT LATER STAGES
As training progress, PDD generates synthetic examples that enable the network to learn higher complexity functions. This implies that at later stages, we can safely discard the examples that are learned early in training with complexxyy functions from the distillation pipeline. To calculate the learning difficulty of examples, we use the forgetting score (Toneva et al. 2019) defined as the number of times the prediction of every example changes from correct to wrong during the training. Examples with higher forgetting scores are learned later during the training with higher complexity functions. On the other hand, examples that have very low forgetting score are those that can be classified by lower complexity functions, early in training. At every distillation stage, we drop examples with low forgetting scores and focus the distillation on examples with increasing levels of difficulty, measure by forgetting score. This improves the efficiency of PDD without harming the performance, as we will confirm experimentally.
Next, we will show experimentally that PCD effectively trains higher-quality neural networks with superior generalization performance without increasing the training time on the synthetic examples.
EXPERIMENTS
In this section, we assess the classification performance of neural networks trained on synthetic images generated by our framework. In addition to evaluating on the architecture used for distillation, we also investigate the transferability of the distilled images to larger models with different architectures. We further show with ablation studies that PDD trains models with increasing classification accuracy when we increase the number of intervals, and confirm the importance of conditioning and transitions.
5.1 EXPERIMENTAL SETTINGS
Datasets. We conduct our experiments three datasets: CIFAR-10. 10, CIFAR-100 (Krizhevsky et al., 2009) and Tiny- ImageNet (Le & Yang 2015). CIFAR-IO and CIFAR- 100 consist of 50, 000 training images, with 10 and 100 classes, respectively. The image size for CIFAR is 32 x 32. Tiny-ImageNet contains 100, 000 training images from 200 categories, with a size of 64 x 64.
Baselines. We consider both data selection and distillation algorithms as baselines, including random selection, Herding (Welling 2009). K-center (Farahani & Hekmatfar) 2009). and Forgetting (Toneva et al. 2019) for selection and DC (Zhao et al. 2021). DSA (Zhao & Bilen 2021b), DM (Zhao & Bilen
5

---

has Published as a conference paper at ICLR 2024
Table I: Test accuracy of ConvNets on CIFAR-10/100 and Tiny-ImageNet, trained on synthetic samples generated by various models with different numbers of images per class (IPC). Our algorithm (PDD) improves upon baseline methods through its multi-stage distillation pipeline, narrowing the performance gap relative training on the full dataset. PDD results are reported for 5 stages.
Dataset
CIFAR-10
CIFAR-100
Tiny-ImageNe
IPC
10
50
10
50
10
50
Random
26.0
43.4 1.0
14.6 + 0.5
30.0 0.4
0.2
15.0 0.4
Herding
0.7
40.4 0.6
17.3 + 0.3
33.7 0.5
6.2
16.7 0.3
Selection
K-Center
14.7
27.0 1.4
7.1
Forgetting
23.3
23.3 1.1
+.2
30.5 + 0.3
5.1
15.0 0.3
DC
1..
0.5
25.2
DSA
5..
0.5
32.3+ 0.3
42.8 0.4
CAFE
46.3 0.5
55.5+
27.8 0.3
37.9 0.3
CAFE
0.4
31.5
42.9
DM
0.6
6..0
10.7.0..3
43.6
12.9
24.1 - 0.3
Distillation
MTT
0.7
1.2
0.3
47.7 0.2
23.2
28.0 0.3
PDD+MTT
66.9
0.4
74.2 0.5
43.1
0.7
52.0
0.5
27.3 0.5
29.2 +2++0.6 0.6
IDC
0.5
74.5 0.1
45.1 + 0.3
52.5 0.4
PDD l IDC
67.9
0.2
76.5
45.8 = 0.5
54.4 0.4
Full Data
88.1
56.2
37.6
2021a), CAFE (Wang al. 2022). IDC (Kim et al. 2022 and MTT (Cazenavette al.. 2022) for
distillation. Herding (Welling 2009) greedily selects samples to approximate the mean of the entire 2019) keeps track of how many times training sample is learned and forgotten during the training and keeps examples with the highest forgetting score;
dataset; Forgetting score (Toneva
K-Center (Farahani Hekmatfar 2009) selects the samples to minimize the maximum distance between a data point and its center. Distillation baselines are introduced in Section 2
Architectures. Our experimental settings follow that of Cazenavette et al. (0022): we employ ConvNet for distillation, with three convolutional blocks for CIFAR- 10 and CIFAR- 100 and four convolutional blocks for Tiny-Imma ImageNet, each containing 128-kernel convolutional layer, an instance normalization layer (Ulyanov et al.) 2016), ReLU activation function (Nair & Hinton 2010) and an average pooling layer. We include ResNet- 18 and ResNet-10 (He al. 2016) to assess the transferabilityy of the synthetic images to other architectures.
Distillation Settings. We adopt two representative baseline methods on which we apply our frame- work: IDC and MIT, which are widely used state-of-the-art dataset distillation methods. During the matching process, we adopt the optimal hyper-paramete reported in the original paper of each dataset distillation method in each stage of PDD without further tuning. We report the number of images PDD distills at each stage and also report the number of synthetic sets in our results to enable comparison between PDD and the baselines. Note that the number of synthetic monotonic effect on the models' testing accuracies.
Evaluation. Once the synthetic subsets have been constructed for each dataset, they are used train randomly initialized networks from scratch, followed by evaluation on their corresponding testing sets. For PDD, sequentially train models after each interval all synthetic samples that have already been generated up to the current interval. For each experiment, we report the mean and the standard deviation of the testing accuracy of 5 trained networks. To train networks from scratch at evaluation time, we use the SGD optimizer with momentum of 0.9 and weight decay of 5 10 For IDC, the learning rate is set to 0.01. For MTT, the learning rate is simultaneously optimized with the synthetic images. During the evaluation time, we follow the augmentation strategies of each method to train networks from scratch.
5.2 EVALUATING DISTILLED DATASETS
Setup. We demonstrate the effectiveness of the proposed multi-stage distillation by applying PDD to MIT and IDC to distill CIFAR- 10/100 and Tiny-ImageNet. Table compares PDD with -tate-of-the- art baselines for different values of Images Per Class (IPC) distilled in stages. We specify baselines' IPC and PDD's 's IPC to be 10 and 50 for all the benchmarks. For Tiny-ImageNer we only conduct experiments with MTT IDC's distillation time is prohibitively expensive in this higher resolution. Based on the default settings, single-stage IDC distills 4 epochs training on the real images; MIT distills 15 epochs for CIFAR-10, 20 for CIFAR-100. and 40 for ImageNet

---

Published as a conference paper at ICLR 2024
per stage IPC
55
10
Aramay 30-
50
Arsing
50-
75-
45-
per-ttge IPC
per-stage IPC
70-
10
65-
40-
Numbor of stoges
Number stages
Number of stages
(a) CIFAR-10
(b) CIFAR-100
(c) Tiny-Imagen
Figure 3: ConvNets' test accuracy on CIFAR-10, CIFAR-100 and
after training on
samples distilled by PDD + MTT with multiple stages with larger per-stage IPCs. Left: performance on CIFAR- 10; Middle: performance on CIFAR-100; Right: performance on Tiny-ImageNet. The red lines indicate the performance of training on respective full data.
Comparing to Single Stage Distillation. We see that PDD consistently improves the performance across all data selection and distillation baselines with the same IPCs, especially when we distill longer training dynamics (i.e., 15 epochs with MTT) on the real images in each stage. Specifically, PDD + MIT outperforms MTT by significant margins 1.6%/2.0 2.33% on CIFAR- 10 IPC- IPC-10/50, 3.5% 4.3% on CIFAR-100 and 4.1% 1.2% on Tmay-ImageNet. After applying PDD to IDC, we witness an substantial improvement on performance across different datasets: 0.4% on CIFAR- 10 IPC- 10/50, respectively, and 0.7%/1.9% 11.% on CIFAR-100 100 1PC-10/50, respectively.
Scaling up synthetic datasets: towards bridging the gap training on the full data. In Figure Balana 3b) we extend our experiments with MTT by maintaining a constant per-stage IPC while progressively increasing the number of stages. This setting enables scaling synthesis process to generate larger total IPC, because the images generated in earlier stages are employed in subsequent stages. We conduct these experiments on CIFAR- 10 and CIFAR- 100, respectively, and set the per- stage IPC to 10/50 for CIFAR-I0, 10/20 20 for CIFAR-IO0 100, and 2/10 for Tiny- Iiny-mmaeeNet. Remarkably, PDD considerably bridges the gap to training on the full dataset by achieving 90% the full accuracy with only 5% of the full data size CIFAR- 10 (which means that IPC 250) and 10% of full data size on CIFAR-100 (which means that IPC 50). Notably, for CIFAR-100, we utilize 20% of the complete dataset, resulting On Tiny-ImageNer, applying PDD with MTT could also reach 80% of the performance obtained by training on the full data after distilling 50 images per class.
IPC value of 100, yet achieve comparable performance.
5.3
GENERALIZATION
Next, evaluate the generalization perfor- mance of PDD on architectures that are differ- from the one used distill CIFAR-10. Following the settings of cross-architecture ex- periments the original papers, we use batch normalization layers when evaluating on IDC, and use instance normalization layers for MTT. We follow the same evaluation pipeline for each baseline method to acquire and present the test accuracy in Table
Table 2: Performance on other architectures of networks with synthetic datasets generated Con- vNets by baselines vs. PDD + baselines.
(Total) IPC
Method
ResNet-10
ResNet-18
IDC
63.0 0.6
63.6 =
IDC
63.2
63.9 0.6
10
46.4 0.4
45.2
PDD+NIT
47.1 1 0.3
46.0
IDC
70.7 0.6
69.8 = 0.3
PDD+IDC
7.4
71.6
50
Images distilled by PDD improves other architec- tures' performance (1.7% (1.7% 1155% on ResNet-10 and 1..%% on ResNet-18) when using IPC
6..1
62.6 - 0.4
PDD+MIT
64.6 0.9
63.5 0.5
50, and show considerable improvement (0.2%/0.7 ResNet-10 10 and 0.3% on ResNet- 18) compared to using the single-stage MTT and IDC when the total IPC is 10. These results indicate that distilled images from multiple stages robust changes in network architectures.
5.4 ABLATION STUDIES
Effect of Progressive Training. When training a model on the P synthetic subsets, PDD pro- gressively trains the union of the first synthetic sets, for i = 1,- P. To demonstrate the effectiveness this progressive way of training, we explore multiple choices training pipelines with the PDD generated synthetic sets: (1) Union: train the union of the synthetic sets generated
7

---

Published as a conference paper at ICLR 2024
in all P stages, Le., Sf (2) Sequential: train different S; in the order they generated; (3) Progressive: we progressively train on union of the first i synthetic sets, i.c. U3=1š}.
Table 3 compares the above training methods when evaluating the synthetic sets PDD distilled for CIFAR-10 with a fixed per-stage IPC = 10 and dif- ferent numbers of stages P. For the base distilla- tion algorithms, namely MTT and TDC, progressive training is consistently better than union and out- performs sequential training with a large margin in particular for larger P. This confirms the necessity of progressive training prevent forgetting the pre- viously learned information. Note that PDD + MTT performs poorly with the union pipeline because MTT learns the learning rate for each set of synthetic images, so a single learning rate suitable for training on the union.
Table 3: Effect of training on PDD distilled subsets. Testing accuracy on CIFAR-10 10 af- ter being trained on 10 IPC per stage distilled by PDD + different base methods. In 'Train- ing' column, U, correspond training on Uf J or St, or S at stage respec- tively.
P Training
Test Accuracy
MTT PDD
IDC + PDD
65.3 0.7
67.5 + 0.5
60.4 0.6
71.1 0.2
2
64.1 1 1.0
68.5 0.1
Importance of transitions and conditioning. There are two key designs in PDD that are essen- tial for the success of multi-stage dataset distillation: (1) transition between stages by generating new synthetic subset based on the training trajectory start- ing from the point where training the previous synthetic subsets is finished; and (2) conditioning synthetic images distilled in carlier stages when generating a new synthetic set for the current training stage.
P
68.7 + 0.8
71.4 0.2
65.4
74.2 0.4
3
67.4 1.1
68.2 0.7
P
71.5+
74.0 0.3
63.2
75.4 + 0.3
the union of
4
66.0 0.9
69.9 0.5
P
73.1
75.4
65.9
76.2 0.6
5
67.410.
69.9 0.5
74.2 + 0.5
76.5 0.2
In Table we show both components are crucial by comparing the test accuracy of ConvNet after being trained the PDD distilled datasets with
both or without one of the two designs. For PDD + MTT and both variants, we fix the number of images per class distill in each stage be 10. We observe decreased performance of PDD when it distills images for each training stage independent of the previous stages, and the difference is more significant when we distill longer training intervals with more stages.
Table 4: ConvNet's performance on CIFAR-10 10 with differ- synthesis modes using MTT with PDD,
Table 5: Models' testing accuracy on CIFAR-10, PDD with different numbers of stages (P) and per-stage IPC.
P
w/o transition
w/o conditioning
PDD
P
per-stage IPC
Accuracy
65.3 0.7
65.3 0.7
65.3 0.7
2
66.0 + 0.6
67.9 0.5
68.7 13.7=0.8
1
10
65.3 + 0.7 65.5 + 0.9
66.3 + 0.4
69.8 0.9
71.3 0.4
2
65.6 - 0.5
71.4 0.5
5
66.9 + 0.4
63.6 + 0.7
71.9 + 0.7
74.2 - 0.7
10
64.4 + 0.6
Distilling more training stages vs more images per stage. Given fixed total number of images per class, we can distill longer training dynamics by having more stages, or choose to distill more images each stage to capture the dynamics better. To understand which of the above two strategies leads to better performance, we study four different combinations of the number of stages and per-stage IPC, and record the models' test accuracy Table 5 We observe that establishing more stages can generally improve the results, as long per-stage IPC is not too small (IPC 1 per stage leads to degraded performance). In particular, with 10 as a fixed number of images in total, best result corresponds to P 5 and per-stage IPC=22
Discarding easy-to-learn examples at later stages. Next, we confirm that samples that are easier to learn can be dropped from the distillation pipeline in later intervals. To do so, use the forgetting score (Toeva et al.) 2019) defined as the number of times the prediction of every example changes from being correctly classified to incorrectly classified during the training. Examples with higher
8

---

Published as a conference paper at ICLR 2024
forgetting scores are more difficult to learn for the network and are learned later during the training (Toneva et al. 2019).
Table 6: ConvNet's performance on CIFAR-10 trained on synthetic set with 10 images per class using MIT with PDD by distilling from easy to difficult samples. i-th stage we select samples with forgetting score within [3(i 11) We report the portion of training samples used each setting.
We separate training examples into multiple par- titions based on their forgetting scores, with an increment of 3. More specifically, at the i-th stage only the examples with a number of for- getting events between 3 (i 1) and 3 i. Subsequently, we apply PDD to distill the cor- responding partition of data examples at each stage. starting from the partition that contains examples with the lowest forgetting scores and progressing to those with the highest scores. Ta- ble [ shows that when PDD explicitly distills examples with increasing learning difficulty at different stages, models trained on the distilled images have the same test performance when the distillation is based on the full training set at all stages. This observation not only confirms that PDD naturally creates curriculum with its synthetic sets but also confirms the possibility of reducing the distillation cost of PDD as the training examples used in each stage can be significantly reduced.
Data Used
Testing Accuracy
36.5%
65.9%
100%
65.3%
55.8%
71.6%
3
100%
71.5%
66.4%
73.7%
5
100%
74.2%
5.5 CONTINUAL LEARNING
Table 7: Continual learning performance using distilled sam- ples generated by different methods on CIFAR- 1000
In this section, we adopt class in- cremental setting (Zhao et al. 2021 Zhao & Bilen 2021b) to show that PDD can improve the performance in the application of continual learn- ing. We apply PDD on MTT to dis- till CIFAR-IOM 100 across phases, in each of which we can only access 20 classes with 20 images distilled in total per class. During the evaluation, a model will be trained
Methods
Stagel
Stage 2
Stage
Stage 4
Stage 5
DSA
52.5
45.7
40.4
35.0
31.1
Herding
48.4
43.3
39.6
36.4
33.1
MTT
55.7
52.1
18.3
43.0
11.2
PDD+MTT 61.2
56.6
51.5
48.3
45.1
sequentially on samples available at each stage. Table 7 shows the performance using different methods, which demonstrates that PDD + MTT consistently outperforms MTT at each stage and showcases PDD's ability to improve baselines' performance in the application of continual learning.
5.6 SYNTHESIZED SAMPLES VISUALIZATION
In Figure 4 we provide examples of synthetic samples on CIFAR-10 10 using PDD MTT at different stages. We distill CIFAR-IO in 5 stages with a per-stage IPC of 10. From the images we can observe that the synthetic samples at later stages show diversified patters,, demonstrating lower saturation in color and more abstract textures. This evolution of visual patterns indicates shift in the focus of the distillation process and thus provides an empirical support to our multi-stage design. Figure A5 shows all the samples from Stage 1 to 5 where the transition of distilled patterns on all classes are clearly presented.
Stage
Smage 3
Stage 5
Figure 4: Synthesized images of CIFAR- 10 using PDD + MTT from Stage 1, 3 and 5. The images from classes "automobile" and "birds" at each stage are selected for demonstration.
6 CONCLUSION
In this work, we proposed progressive dataset distillation framework. PDD, that generates multiple sets of synthetic samples sequentially, conditioned on the previous ones, to capture dynamics of different training intervals. The multi-stage nature of PDD enables it to capture the training dynamics of neural networks more effectively, and simultaneously reduce the complexity associated with training the entire synthetic dataset. Extensive experiments confirm the effectiveness of PDD in improving the performance of existing dataset distillation methods on various benchmark datasets. For future works, are interested in exploring PDD's potential to serve effective base method for further condensation (Liu et al. 2023a) where the budgets may vary, aligning well with the inherent design of our approach.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The work of Y. Yang supported in part by the Amazon PhD fellowship. The work of B. Mirzasoleiman was supported in part by the U.S. National Science Foundation under the grant IIS-2146492 (CAREER Award). X. Chen and Z. Wang are in part supported by the NSF Al Institute for Foundations of Machine Learning (IFML).
REFERENCES
George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexei A Efros, and Jun-Yan Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750-4759, 2022.
George Cazenavette, Tongzhou Wang. Antonio Torralba, Alexci A Efros, and Jun- Yan Zhu. General- izing dataset distillation via deep generative prior. arXiv preprini 2305::2330000444 2023.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In International Conference Machine Learning, pp. 5378-5396. PMLR, 2022.
Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Yunzhen Feng, Shanmukha Ramakrishna Vedantam, and Julia Kempe. Embarrassingly simple dataset distillation. In The Twelfth International Conference on Learning Representations, 2024. URL
atttps://openzeviews net wtw.://wwweee/eooomeeeeeeeee forum?id-PLowee 7Mjc
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligner trajectory matching. ICI.R, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995-10006, 2020.
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics of neural networks. Advances Neural Information Processing Systems, 33: 17116-17128, 2020.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in neural information processing systems, 32, 2019.
Jang- Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, pp. 11102-11118 PMLR, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang, Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Songhua Liu, Kai Wang, Xingyi Yang. Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. arXiv preprint arXiv:2210 10. 16774, 2022.
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3759-376666 June 2023a.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. arXiv preprint arXiv:2302. 240.:22332..411111 14416, 2023b.
10

---

Published as a conference paper at ICLR 2024
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210.12067, 2022.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814,
2010.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization deep learning. arXiv preprint 141166442.66/44 2014.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression. In International Conference on Learning Representations, 2021a. URL https
/oopenreview net saett/oorrmidd--Prroop
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186-5196
2021b.
Omead Pooladzandi, David Davini, and Baharan Mirzasoleiman. Adaptive second order coresets for data-efficient machine learning. In International Conference on Machine Learning. pp. 17848- 17869. PMLR, 2022.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls simplicity bias in neural networks. Advances in Neural Information Processing Systems, 13:9573-9588 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL
met/forue 3dd33 xx333x xm30cKm
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint 100.::666770002,, 2016
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1196-12205, 2022.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 11I1::8811.10999 10959, 2018.
Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128, 2009.
Yu Yang, Hao Kang, and Baharan Mirzasoleiman. Towards sustainable learning: Coresets for data-efficient deep learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 Proceedings of Machine Learning Research, pp. 39314-39330. PMLR, 23-29 Jul 2023. BtLhttts: htttps::/proceeeinnngn wwtpeeeedinnn.m.......eeeeeeeeee wwt.::.mw..ppp..eeeeeemereeeseee wtt.:ppesss//2//eeeeeeee
httl
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. arXiv preprint 2 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 2674-126855 PMLR, 2021b.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint arXiv:2204.07513 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. ICLR, 1(2)3, 20.1.
11

---

Published as a conference paper at ICLR 2024
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In Alice Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. BtLL:tts::/// httptp::///penneveww...... forum?
(d=2clwrA2t.fik
A1 EXPERIMENT DETAILS
A1.1 EXPERIMENT SETTINGS
On CIFAR-1O, the networks are trained for 2000
P+1 epochs at each stage. Consequently, the total 2 P41 B 1000Pm,
iterations taken P(P+1) 2000
B where B is the batch size and n the number of images newly distilled each stage. This quantity proves to be adequate in achieving favorable outcomes without inflating the computational burden of network training. Notably, it aligns with utilizing all available images for a training duration of 1000 epochs. Additionally, note that augmenting the number of epochs could lead to further enhancements in the test accuracy of the trained networks. For CIFAR-100 100, the networks undergo training for 500 epochs during each stage to facilitate improved convergence.
important to
A2 MORE VISUALIZATION
we visualize the synthetic samples of CIFAR-10 distilled at stages to 5 using PDD + MTT. We observe significant shift of visual features in these distill images. The images distilled the first stage are the most colorful among all the distilled samples, while the images distilled at later stages contain more abstract features and less focus on colours. These figures show that PDD helps distill diverse features according to different stages.
In Figure A5
Stage
Stage
Stage
Stage 4
Stage
Figure A5: Visualization of synthesized samples from Stage to Stage 5.
A12

---

Published as a conference paper at ICLR 2024
A3 MORE EXPERIMENT RESULTS
A3.1 RESULTS ON MORE METHODS
Experiments on DC, DSA and DM. We further apply PDD to DC (Zhao al., 2021 and DSA (Zao & Bilen 2021b) to distill images from CIFAR-10. Table A8 shows the ConvNet's accuracy after trained on the distilled images. On DC and DSA, compared using the single stage synthesis, PDD + DC and PDD DSA generates samples that lead to higher performance, improving the baselines' performance by 2.4% and 0.7%, respectively.
Additionally, we have conducted experiments with DM on CIFAR-10 with IPC=50. Note that DM does not distill the training dynamics but rather enforces the similarity between features extracted from synthetic and real samples by enormous models with random weights. Our experiments show that for IPC=50, DM hits an accuracy of 63.0% while PDD + DM can reach 63.4%, showing a slight 0.4% improvement in performance. This mild improvement is expected, DM does not capture any training dynamics of training on full data, and only captures similarity between features based on random weights.
Table A8: ConvNets' test accuracy CIFAR- 10 after trained on synthetic samples generated by DC, DSA and DM with different numbers of images per class.
Table A9: ConvNets' test accuracy on CIFAR- 10 after trained with samples distilled by DREAM and DREAM+PDD.
Dataset IPC
CIFAR-I0 50
Dataset IPC
CIFAR-
10
10
50
DC
53.9
0.5
Accuracy of DREAM
68.7
74.8
DSA
60.6 + 0.5
Accuracy of PDD + DREAM
69.3
76.7
DM
63.0 0.4
Improvement of PDD
0.6
1.9
PDD + DC
56.3 + 0.5
Standard Deviation
0.3
0.2
PDD + DSA
61.3 0.4
PDD + DM
63.4 + 0.5
Full
88.1
Full
88.1
Experiments on DREAM, We have conducted an additional set of experiments applying PDD to DREAM on CIFAR-10. The results using the IPC values of 10 and 50 are presented Table A9 where we observe that PDD can further improve the performance of DREAM beyond the current state-of-the-art method.
A3.22 EXPERIMENTS ON UPDATING PREVIOUS FROZEN SUBSETS
We have conducted an additional set of experiments that allows images synthesized early stages to be updated during late stages with smaller learning rate. We choose MIT as the base method and set up stages with a per-stage IPC of 10, resulting we ensure the faimess of comparison by only using the last set of synthesized images to train models. This is to keep the number of synthesized samples to the same when compared with other results. The trained model hits a test accuracy of 69.1%, which is inferior to our proposed method.
total IPC of 50. During the evaluation phase,
The decline in performance can likely be attributed to the fact that expert trajectories, i.e, the subsequent model checkpoints obtained by training on the full real dataset and synthetic samples (if applicable, from previous stages), are not derived from a consistent set of synthetic samples. Note that we first train on synthetic examples produced in previous stages to generate expert trajectories and synthesize data for the next stage. If the synthesized images the previous stages are updated, the distilled samples from stage 1, which serve the foundation for generating trajectories in stages 2 to 5, vary with each subsequent stage. When the samples synthesized in stage 1 are updated in stage 2, they gradually deviate from the initial synthetic samples (i.e., those employed to generate the expert trajectories for stage 2). Hence, training on the synthesized images that are being updated results different training dynamics each time. This eventually leads discrepancy and ultimately results in a decrease in performance.
A13


Reviews:
[{"review_id": "Dkcj1MvGm1", "reviewer": "Reviewer_S7ej", "summary": "This paper proposes a dataset distillation algorithm that generates synthetic data in a progressive manner: the next batch of synthetic data would be dependent on previous batches. The training using the distilled datasets also contains several stages. The training data for each stage come from the corresponding batch of the distilled dataset and its previous batches in a cumulative way. Experiments demonstrate that the proposed strategy improves the baseline method.", "questions": "Please refer to the Weaknesses part. I would also like the authors to discuss more benefits of PDD as mentioned in the 1st point of Strengths in the camera-ready version or in the next revision cycle.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea of progressive dataset distillation is interesting. This can 1) capture the training dynamic of neural networks better as demonstrated by authors, 2) reduce the complexity of training the whole synthetic dataset together, and 3) serve as a strong method for slimmable dataset condensation [a].\n\n[a] Slimmable Dataset Condensation, CVPR 2023.", "weaknesses": "1. Many places are unclear.\n    * Fig. 1 needs some explanations. How do the results come in detail, like what's the IPC of each stage, and how to conduct multi-stage training? Although some of these questions are answered in the following parts, the writing is not coherent.\n    * The networks for the next stage come from the training results with previous batches of synthetic data. In IDC, the networks are periodically re-initialized randomly. In MTT, the networks come from checkpoints of training with original datasets. We have to conduct some modifications to these baselines before using PDD. These operations are unclear.\n2. A comment: this method makes an assumption on downstream training using synthetic datasets: models must be trained in a multi-stage way using the provided multi-stage synthetic data, which would introduce a lot of hyperparameters, especially in the cross-architecture setting and make the dataset distillation less elegant. Given that the performance gain is not significant in most cases, the practical value of the proposed method is somewhat limited.\n3. Through the results in Tab. 6, the effect of discarding easy-to-learn examples at later stages is not significant. More evidence is necessary to demonstrate the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdkBvKw41k", "reviewer": "Reviewer_rp5C", "summary": "The paper proposes a dataset distillation method. The core idea is to freeze the previously distilled subset and then only optimize the new subset. This method can be applied to several dataset distillation methods and show marginal improvement over the original method.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall writing is clear.\n2. The comparison with a few previous is clear to validate the effectiveness of the method.\n3. The explanation looks sufficient.", "weaknesses": "1. Some important reference is missing such as DEARM [a]. Since this is the SOTA method in dataset distillation, the authors should cite and compare the paper.\n2. After Eq.4, the authors claim PDD can be used to **any** dataset distillation method. Therefore, it is necessary to do experiment if the PDD can augment DREAM.\n3. The accuracy is not as good as DREAM. For example, on CIFAR-10 IPC-10, DREAM can achieve 69.4% accuracy. But the proposed PDD can just achieve 67.9%. On CIFAR-100 IPC-10, DREAM can achieve 46.8% accuracy. But the proposed PDD can just achieve 45.8%. \n4. Based on question 3, why does the proposed method have bad performance on IPC-10?\n5. The idea is too straightforward. Is it possible to update the previous frozen subset with small learning rate?\n\n\n I will increase the score if the DREAM-related experiments are added.\n\n[a] DREAM: Efficient Dataset Distillation by Representative Matching, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oYgYheJsv", "reviewer": "Reviewer_xT2K", "summary": "The significant contribution of this work lies in the introduction of Progressive Dataset Distillation (PDD). It is a novel methodology in the field of dataset distillation. PDD effectively addresses the limitations inherent in existing DD works by recognizing that relying solely on a single synthetic subset for distillation does not lead to optimal generalization. \n\nSpecifically, PDD innovatively synthesizes multiple small sets of synthetic images, with each set being conditioned on the knowledge acquired from the preceding sets. The alteranated updated model is then trained on the cumulative union of these subsets. This sophisticated approach results in a noteworthy performance improvement.", "questions": "Is it possible to provide additional experimental results about \"DM+PDD\"?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea presented in this work is both intriguing and groundbreaking. As far as my knowledge extends, this is the first instance where the synthesis of multiple small sets of distilled images has been proposed. The authors have adeptly addressed the challenge of minimizing bias and ensuring training stability by introducing a conditioning mechanism for each distilled set, based on the knowledge accumulated from the preceding sets.\n\n2. The clarity and coherence of the writing are commendable. The motivation behind the research is robust, and the overall structure of the paper is meticulously organized.\n\n3. The experimental results provided in the paper are highly promising and effectively illustrate the efficacy of the proposed solution.", "weaknesses": "N.A.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)