Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching
Authors: Ziyao Guo
Kai Wang
George Cazenavette
HUI LI
Kaipeng Zhang
Yang You
Venue: ICLR 2024 poster (2024)
Keywords: Dataset Distillation
Primary area: general machine learning (i.e., none of the above)

Abstract: The ultimate goal of Dataset Distillation is to synthesize a small synthetic dataset such that a model trained on this synthetic set will perform equally well as a model trained on the full, real dataset. Until now, no method of Dataset Distillation has reached this completely lossless goal, in part due to the fact that previous methods only remain effective when the total number of synthetic samples is extremely small. Since only so much information can be contained in such a small number of samples, it seems that to achieve truly loss dataset distillation, we must develop a distillation method that remains effective as the size of the synthetic dataset grows. In this work, we present such an algorithm and elucidate why existing methods fail to generate larger, high-quality synthetic sets. Current state-of-the-art methods rely on trajectory-matching, or optimizing the synthetic data to induce similar long-term training dynamics as the real data. We empirically find that the training stage of the trajectories we choose to match (i.e., early or late) greatly affects the effectiveness of the distilled dataset. Specifically, early trajectories (where the teacher network learns easy patterns) work well for a low-cardinality synthetic set since there are fewer examples wherein to distribute the necessary information. Conversely, late trajectories (where the teacher network learns hard patterns) provide better signals for larger synthetic sets since there are now enough samples to represent the necessary complex patterns. Based on our findings, we propose to align the difficulty of the generated patterns with the size of the synthetic dataset. In doing so, we successfully scale trajectory matching-based methods to larger synthetic datasets, achieving lossless dataset distillation for the very first time. Code and distilled datasets will be released.

Published as a conference paper at ICLR 2024
TOWARDS LOSSLESS DATASET DISTILLATION VIA EEFICULTY-ALIGNER TRAJECTORY MATCHING
Ziyao Guo Kai Wang't George Cazenavette" Hui Li Kaipeng Zhang Yang You'
1 National University of Singapore "Massachusetts Institute of Technology
3 Shanghai Artificial Intelligence Laboratory 4 AXidian University
gryaffeerattheatt.ook com, (kai wang, youy) @comp. nus. edu. sg, gcazêmit.edu
ABSTRACT
The ultimate goal of Dataset Distillation is to synthesize small synthetic dataset such that a model trained on this synthetic set will perform equally well as model trained the full, real dataset. Until now, method of Dataset Distillation has reached this completely lossless goal. in part because they only remain effective when the total number of synthetic samples is extremely small. Since only much information can be contained in such a small number of samples, it seems that to must develop distillation method that
achieve truly lossless dataset distillation, remains effective as the size of the synthetic dataset grows. In this work, we present such an algorithm and elucidate why existing methods fail to generate larger, high- quality synthetic sets. Current state-of-the-a methods rely on trajectory-matchun or optimizing the synthetic data to induce similar long-term training dynamics as the real data. We empirically find that the training stage of the trajectories we choose to match (i.e., early or late) greatly affects the effectiveness of the distilled dataset. Specifically, early trajectories (where the teacher network learns easy patterns) work well for a low-cardinality synthetic set since there are fewer examples wherein distribute the necessary information. Conversely, late trajectories (where the teacher network learns hard patterns) provide better signals for larger synthetic sets since there are Based on our findings, we propose to align the difficulty of the generated patterns with the size of the synthetic dataset. In doing so, successfully scale trajectory matching- based methods to larger synthetic datasets, achieving lossless dataset distillation for the very first time. Code and distilled datasets are available https: /tle://gitttu cow COW.uuu..cOONUUUUS-UUEUU NAI//NUSSHHERE -I- -AA-://AUSSS----AA////AAAAA-AA Lab/DATM.
enough samples to represent the necessary complex patterns.
INTRODUCTION
Dataset distillation (DD) aims distilling large dataset into small synthetic one, such that models trained on the distilled dataset will have similar performance as those trained on the original dataset. In recent years, several algorithms have been proposed for this important topic, such gradient matching (Zhao et al., 2020; Kim et al., 2022: Zhang et al., 2023; Liu et al., 2023b), kernel inducing points (Nguyen et al., 2020; 2021), distribution matching (Wang al., 2022: Zhao Bilen, 2023; Zhao et al., 2023), and trajectory matching (Cazenavette et al., 2022; Cui et al., 2023; Du et al., 2023). So far, dataset distillation has achieved great success in the regime of extremely small synthetic sets. For example, MTT (Cazenavette al., 2022) achieves 71.6% test accuracy on CIFAR- 10 using only 1% the original data size. This impressive performance led to its application a variety of downstream tasks such as continual learning (Masarczyk & Tautkute, 2020; Rosasco et al., 2021), privacy protection (Zhou et al., 2020; Sucholutsky & Schonlau, 202 la: Dong et al., 2022; Chen et al., 2022: Xiong et al., 2023), and neural architecture search (Such et al., 2020; Wang al., 2021).
However, although previous DD methods have achieved great success with very few IPC (images-per per- class), there still remains significant gap between the performance of their distilled datasets and the full, real counterparts. To minimize this gap, one would intuitively think to increase the size of the
Project lead.
Cooreeponning author.

---

Published as a conference paper at ICLR 2024
Acc. (%)
Comparable
Distilation Algarthim
generato
add
90
Pattems
Synthelic Danaset
train
(b) Normal
70
ganorate
add
Random
50-
DM
Large Original Dateset
Small Syrthetic Dataset
Distilation Aigorthi
Easy Patterns
Small Synthelic Dataset
DSA
optimize
generate
add
30
10
50
1000
2000
Distilation Aigorithm
IPC
(c)
Hard Patters
Large Synthetic Dataset
(d) Performance Comparison
Figure 1: (a) Illustration of the objective of dataset distillation. (b) The optimization in dataset distillation can be viewed the process of generating informative patterns on the synthetic dataset. (c) We align the difficulty of the synthetic patterns with size the distilled dataset, to enable our method to perform well in both small and large IPC regimes. (d) Comparison of the performance of multiple dataset distillation methods on CIFAR- 10 with different IPC. As IPC increases, the performance of previous methods becomes worse than random selection.
synthetic dataset. Unfortunately, as IPC increases, previous distillation methods mysteriously become less effective, even performing worse than random selection (Cui et al., 2022; Zhou et al., 2023). In this paper, we offer an answer as to why previous dataset distillation methods become ineffective as IPC increases and, in doing so, become the first to circumvent this issue, allowing us to achieve lossless dataset distillation.
We start our work by observing the patterns learned by the synthetic data, taking trajectories matching (TM) based distillation methods (Cazenavette al., 2022; Du et al., 2023) example. Generally, the process of dataset distillation can be viewed as the embedding informative patterns into a set of synthetic samples. For TM-based distillation methods, the synthetic data learns patterns by matching the training trajectories of surrogate models optimized over the synthetic dataset and the real one. According to (Arpit et al., 2017), deep neural networks (DNNs) typically learn to recognize easy patterns early in training and hard patterns later on. As a result, we note that the properties of the data generated by TM-based methods vary widely depending on from which teacher training stage we sample our trajectories from (carly or late). Specifically, matching early or late trajectories causes the synthetic data to learn easy or hard patterns respectively.
We then empirically show that the effect of learning easy and hard patterns varies with the size of
the synthetic dataset (i.e, IPC). In low-IPC settings, easy patterns prove the most beneficial since they explain a larger portion of the real data distribution than an equivalent number of hard samples. However, with sufficiently large synthetic set, learning hard samples becomes optimal since their union covers both the easy and "long-tail" hard samples of the real data. In fact, learning easy patterns in the high-IPC setting performs worse than random selection since the synthetic images collapse towards the mean patterns of the distribution and no longer capture the long-tät cong-tail parts. Previous distillation methods default toward distilling easy patterns, leading to their ineffcctiveness in high-IPC cases.
The above findings motivate us to manage to align the difficulty of the learned patterns with the size of the distilled dataset, in order to keep our method effective in both low and high IPC cases. Our experiments show that, for TM-based methods, we can control the difficulty of the generated patterns by only matching the trajectories of specified training phase. By doing so, method is able to work well in both low and high IPC settings. Furthermore, we propose to learn easy and hard patterns sequentially, making the optimization stable enough for learning soft labels during the distillation, bringing further significant improvement. Our method achieves -ttte-of-the-art performance in both low and high IPC cases. Notably, we distill CIFAR- 10 and CIFAR- 100 to 1/5 and Tiny ImageNet to 1/10 of their original sizes without any performance loss on ConvNet, offering the first lossless method dataset distillation.
PRELIMINARY
For a given large, real dataset Dreat dataset distillation aims to synthesize a smaller dataset D such that models trained on Deyn will have similar test performance as models trained Dreat-
2

---

Published as a conference paper at ICLR 2024
|PC=100
IPC=100
IPC=1000
IPC=2000
70
Acc. 1%0
Acc. 1%6
Acc. 1%6
30
60-
75-
78-
March Thm March Early Tm
Match All Tral
Match All Tra:
Match All Trai
March Earty Trn
March Early Tra).
Match Early Tra)
Mazch Losc
Match Lato TTg.
Match Loce Tral.
Match Late Tra).
10
50
70
74
inraatio 1301)
Itoration (10%)
Itcration (10%)
Iteration (109)
(d)
Figure We train expert models on CIFAR-IO for 40 epochs. Then the distillation is performed under different IPC settings by matching either early trajectories {0] (0 s t s. 20), late trajectories [07]20 S t S 40), or all trajectories {0r |0 & I 40). As IPC increases, matching late trajectories becomes beneficial while matching early trajectories tends be harmful.
For trajectory matching (TM) based methods, the distillation is performed by matching the training trajectories of the surrogate models optimized over Dreal and Day Specifically, let 7" denote the expert training trajectories, which is the time sequence of parameters {0r y obtained during the training of a network on the real dataset real Similarly, 9 denotes the parameters of the network trained on the synthetic dataset Dayn at training step t.
In each iteration of the distillation, 02 and iM+ are randomly sampled from a set of expert trajectories {r"} as the start parameters and target parameters used for the matching, where M is a preset hyper- parameter. Then TM-based distillation methods optimize the synthetic dataset Dayn by minimizing the following loss:
10++N
07 M
C
(1)
10;
04 As
where N is preset Syyper-paramete and d++ obtained in the inner optimization with cross-entropy (CE) loss & and the trainable learning rate a:
0ct141 = 014s VV Dayu); where 0, = 07.
(2)
3
METHOD
In this section, we first analyze the influence of matching trajectories from different training stages. Then, we introduce our method and its carefully designed modules.
3.1 EXPLORATION
TM-based methods generate patterns the synthetic data by matching training trajectories. Ac- cording to Arpit et al. (2017), DNNs tend to learn easy patterns early in training, then the harder ones later on. Motivated by this, we start our work by exploring the effect of matching trajectories from different training phases. Specifically, we train expert models for 40 epochs and roughly divide their training trajectories into two parts: the early trajectories (0; (0 S S 20} and the latter ones [2, < t < 40). Then we perform the distillation by matching these two sets of trajectories under various IPC settings. Experimental results are reported in Figure 2. Our observations and relevant analyses are presented as follows.
Observation shown in Figure 2, matching early trajectories works better with small synthetic datasets, but matching late trajectories performs better the size of the synthetic set grows larger.
Analysis 1. Since DNNs learn to recognize easy patterns early in training and hard patterns later we infer that matching early trajectories yields distilled data with easy patterns while matching late trajectories produces hard ones. Combined with the empirical results from Figure 2, we can conclude that distilled data with easy patterns perform well for small synthetic sets while data with hard features work better with larger sets. Perhaps unsurprisingly, this highly coincides with a common observation in the area of dataset pruning: preserving easy samples works better when very
3

---

Published as a conference paper at ICLR 2024
few samples are kept, while keeping hard samples works better when the pruned dataset is larger (Sorscher et al., 2022).
Observation 2. As can be observed in Figure 2 (a), matching late trajectories Icads to poor perfor- mance in the low IPC setting. When IPC is high, matching early trajectories will consistently under- mine the performance of the synthetic dataset as the distillation goes on, as can be observed in Figure 2 (d). Also, as reflected in Figure 2, simply choosing to match all trajectories is not a good strategy.
Analysis 2. In low IPC settings, due distilled data's 's limited capacity, it is challenging to learn data that models the outliers (hard samples) without neglecting the more plentiful easy samples; since easy samples make up most of the real data distribution, modeling these samples is more efficient performance-wise when IPC is low. Therefore, matching early trajectories (which will generate easy patterns) performs better than matching later ones (for low IPC). Conversely, in high IPC settings, distilling data that models only the easy samples is no longer necessary, and will even perform worse than a random subset of real samples. Thus, we must now consider the less-common hard samples by matching late trajectories (Figure 4). Since previous distillation methods focus on extremely small IPC cases, they tend to be biased towards generating easy patterns, leading to their ineffective in large IPC cases.
Based on the above analyses, to keep dataset distillation effective in both low and high IPC cases, we must calibrate the difficulty of the generated patterns (i.e., avoid generating patterns that are too easy or too difficult). To this end, we propose our method: Difficulty-Alignee Trajectory Matching, or DATM.
3.2 DIFFICULTY-LIGNNE TRAJECTORY MATCHING
Since patterns learned by matching earlier trajectories are easier than the later ones, we can control the difficulty of the generated patterns by restricting the majectory-matchingg range. Specifically, let =- {8) [0 < t I n denote expert trajectory. To control the matching range flexibly, we set lower bound T and an upper bound T on the sample range of I, such that only parameters within [0 < < T+} } can be sampled for the matching. Then the trajectory segment used for the matching can be formulated as:
100 07, EDO easy
07
07
0%
(3)
matching range too hard
To further enrich the information contained in the synthetic dataset, an intuitive choice is using soft labels (Hinton et al., 2015). Recently, Cui et al. (2023) show that using soft labels to guide the distillation can bring non-trive non-trivial improvement for the performance. However, their soft labels are not optimized during the distillation, leading to poor consistency between synthetic data and soft labels. To enable learning labels, we find the following challenges need to be solved:
Mislabeling, We use logits L, fe- (I;) to initialize soft labels, which are generated by the pre- trained model fa- sampled from expert trajectories. However, labels initialized in this way might be incorrect (i.e,, target class doesn't have the highest logit score). To avoid mislabeling, we sift through Dreat to find samples that can be correctly classified by model fa- and use them to construct the subset Deub. Then we randomly select samples from Daub to initialize Dayn = {1,, D) = softmax (L:)), }. such that we can avoid the distillation being misguided by the wrong label.
Instability. During the experiments, of the distillation when the IPC is low. In low IPC settings, the distillation loss tends to be higher and less stable overall since the smaller synthetic set struggles to induce a proper training trajectory. This issue becomes fatal when labels are optimized during the distillation, as the labels are too fragile to take the wrong guidance brought by the mismatch, leading to increased instability. To alleviate this, propose to generate only easy patterns in the early distillation phase. After enough easy patterns are embedded into the synthetic data for surrogate models to learn them well, we then gradually generate harder ones. By applying this sequential generation (SG) strategy, the surrogate model can match the expert trajectories better. Accordingly, the distillation becomes more stable.
found that optimizing soft labels will increase the instability
In practice, to generate only easy patterns at the early distillation stage, we set floating upper bound T on the sample range of t, which is set to be relatively small in the beginning and will be gradually increased as the distillation progresses until it reaches its upper bound T+,, Overall, the
4

---

Published as a conference paper at ICLR 2024
Dutase
CIEAR-10 10
CIFAR: 100
Tiny ImageNe:
IPC
10
50
500
1000
10
50
100
10
50
Ratio
20
Random
15.420.3
31.060.5
50.640.3
73.220.3
78.490
4,211.3
14.640.5
23.420.4
42.860.3
1,480.1
50002
15,020.4
DC
28.310.5
46,9.0.5
53,9005
72.1
36.640.3
12.8:0.3
25.240.3
75.120.3
78.549.1
29.720.3
43.640.4
3.96002
129804
24.120.3
DSA
28 7
52.1.0.5
60660.5
73.640.3
78.740.3
13.900.
$2.3+n.3
42. 42.8+01.4
CAFE
10.3s1.1
6
6
12.9.0.3
27.8.0.3
37.9.0.3
KIP
19.940.2
62.720
68.660.2
15.720.2
28.320.1
TRaPo
16.810.7
65.54004
71.760.2
28.7.0.1
42.5g0.2
44.310.2
15/420.3
25.4.0.2
ROIG
33.9+10
69.1s0.4
73.5603
39.3e0.4
44.1204
46,740.3
25.6290.3
29.420.2
65,4+077
59.7=04
47.7+0.2
29.
8.8+0.3
23.2=0.2
0300
TESLA" 48.5
86.4
72 72.6
24.8:0.4
41.7G0.3
47,910.3
49.2 2:044
PID 46.010.4
65.3
73.260.2
24.4104
42.520.2
48.540.3
49.740.4
10.540.2
23,400.3
28.240.4
DATM (Ouss
66.8e0..
76.1=0.3
83.5el..2
W5.5e00.4
27.9+00.2
47.2+0.4
55.0+00.2
57.5e00.
17.1+00.
31.1=0.3
39.760.3
Full Datnset
84
56.250.3
57.6e0.4
Table I: Comparison with previous dataset distillation methods on CIFAR- 10, CIFAR- 100 and Tiny ImageNet. ConvNet is used for the distillation and evaluation. Hilighted results indicate we achieve lossless distillation.
Our method consistently out-penn out-performs prior works and the only to achieve lossless distillation.
"Kemel-base methods use much larger neural network; we underline their results when they perform best. "Previous TM-based methods perform worse than random initialization in higher IPC cases, indicated by ls.. For a fair comparison, reproduce FTD without using EMA (exponential moving average).
process of sampling the start parameters 07 can be formulated as:
0% u((0 07)), where T + 7
In each iteration, after deciding the value of f, we then sample 0 and 87+M from expert trajectories the start parameters and the target parameters for the matching. Then d.+ can be obtained by Eq. 2. Subsequently, after calculating the matching loss using Eq. 1, we perform backpropa- gation to calculate the gradients and then use them to update the synthetic data 2; and Li, where (z,, Di softmax(L.)) (L,)) E Dayu- See Algorithm for the pseudocode of our method.
4 EXPERIMENTS
4.1 SETUP
We compare our method with several representative distillation methods including DC (Zhao et al.. 2020), DM (Zhao & Bilen, 2023), DSA (Zhao & Bilen, 2021), CAFE (Wang et al., 2022), KIP (Nguyen al., 2020), FRePo (Zhou et al., 2022), RCIG (Loo et al., 2023), MTT (Cazenavette et al., 2022), TESLA (Cui et al., 2023), and FTD (Du et al., 2023). The evaluations are performed on several popular datasets including CIFAR- 10, CIFAR- 100 (Krizhevsky et al., 2009), and Tiny ImageNet (Le & Yang. 2015). We generate expert trajectories in the same way FTD without modifying the involved hyperparameters. We also use the same suite of differentiable augmentations (Zhao & Bilen, 2021) in the distillation and evaluation stage, which is generally utilized in previous works (Zhao & Bilen, 2021; Wang et al., 2022; Cazenavette et al., 2022: Du et al., 2023).
Consistent with previous works, networks with batch normalization are indicated with "-BN" (e.g., ConyNet-BN), Without particular specification, we perform distillation using 3-layer ConvNet for CIFAR-10 and CIFAR-100, while we move up to a depth-4 ConvNet for Tiny ImageNet. We also use LeNet (LeCun et al., 1998), AlexNet (Krizhevsky al., 2012), VGG11 I (Simonyan & Zisserman, 2015), and ResNet18 (He et al., 2016) for cross- architectuh experiments, More details can be found in Section A.8.
networks with instance normalization by default, while
4.2 MAIN RESULTS
CIFAR-10/100 and Tiny ImageNet. As the results reported in Table 1, our method outperforms other methods with the same network architecture in all settings but 11 with IPC=1. As can be observed, the improvements brought by previous distillation methods are quickly saturated as the distillation ratio approaches 20%. Especially in CIFAR- 10, almost all previous methods have similar or even worse performance than random selection when the ratio is greater than 10%. Benefiting from our difficulty alignment strategy, method remains effective in high IPC cases. Notably,
5

---

Published as a conference paper at ICLR 2024
Iritalizttion
80.0
Without learning labels
With leamnnn labels
50
Distilacin cost
9.0 tm, 20 (0hitenn
75.6
76.3
Act. (8)
Acc (N)
(1.2
ACC. 20
67.0-
67.0-
75
Labeinng Export
Labeing Expert + 20th spach
+ 20th spoch
60th apech
PCC-50
Random selection
80th spach
spoch
IPC- 1000
15
54.0
$4.0
5
10
60
5
10
Daration (10%)
Iteration (10')
Iter Iteration (109)
Faration CL0%)
(b)
Figure 3: (a): (CIFAR- 100, IPC=10) Synthetic datasets are initialized by randomly sampling data from the original dataset (random selection) or a subset of data that can be correctly classified (ours). Our strategy makes the optimization converge faster. (b): (CIFAR-10, IPC=50) Ablation on learning soft labels, where soft labels are initialized with expert models trained after different epochs. Learning labels relieves us from carefully selecting the labeling expert. (c): (CIFAR-10) The optimization with higher IPC converges fewer iterations.
Method ComeNet
ResNec18
VGG
AlexNet
Soll Label
Difliculty Alignment
Acc
Label Learning
Sequential Gen. Ace
Randons
33.46
31.95
52.18
26.65
72.8
MTT
45.68
42.56
41.22
40.20
50.79
75.0
75.6
DATM
55.03
51.71
45.38
45.74
55.03
76.1
(a) 100,R-100, IPC=50
(b) CIFAR-100, IPC=50
(c) CIFAR-10, IPC=50
Table 2: (a): Arosstrrchitecture evaluation. Our distilled dataset performs well across various unseen networks. (b): Ablation studies the components of our method; all bring non-trivial improvement. (c): Ablation on learning soft labels and our sequential generation (SG) strategy.
successfully distill CIFAR- 10 and CIFAR-100 to 1/5, and Tiny ImageNet to 1/10 their original size without causing any performance drop.
Here we evaluate the generalizability of our distilled datasets in various IPC settings. As the results reported in Table 2 (Left), our distilled dataset performs best on unseen networks when IPC is small, reflecting the good generalizability of the data and labels distilled by our method. Furthermore, performance with two representative coreset selection methods including Glister (Killamsetty et al., 2021) and Forgetting (Toneva et al., 2018). As shown in Table 4, although coreset selection methods are applied case by case, they are not universally beneficial for all networks. Notably, although our synthetic dataset is distilled with ConvNet, it generalizes well on all networks, bringing non-trivial improvement. Furthermore, on CIFAR-100, the improvement of AlexNet is even higher than that of ConvNet. This reflects the overfitting problem of synthetic datasets to distillation networks is somewhat alleviated in higher IPC situations.
Cross-architecture generalization.
evaluate the generalizability in high IPC settings and compare the
4.3 ABLATION
Ablation on components of our method. We perform ablation studies by adding the components of our methods one by one to measure their effect. As the results reported in Table 2 (b,c), all the components of our method bring non-trivial improvement. Especially, when soft labels are utilized, the distillation becomes unstable if our proposed sequential generation strategy is not applied, leading to poor performance and sometimes program crashes.
Soft label. In our method, we use logits generated by the pre-trained model to initialize soft labels, since having an appropriate distribution before the softmax is critical for the optimization of the soft label (Section A.2). However, using logits will introduce additional information to the distilled dataset (Hinton et al.,
Dataset IPC
CIFAR-10
CIFAR-I00
Tiny ImageNet
50
10
FID
46.0
65.3
73.2
24.4
42.5
48.5
10.5
23.4
28.2
FID+ASI.
45.5
66.1
72.8
24.2
44.5
51.2
12,4
26.7
30.9
DATM
46.9
66.8
76.1
27.9
47.2
53.0
17.1
29.0
33.7
Table 3: We assign soft labels (ASL) for datasets distilled by FTD. For fairness, our difficult alignment strategy is not utilized here. Results in red indicate the case when ASI. is harmful.

---

Published as a conference paper at ICLR 2024
Dataset
Ratio
Method
ConvNet
Conv-eet-B BN
ResNet18
ResNert8-BN
VGG11
AlexNet
LeNet
MLP
Avg.
Random
78.38
80,25
84. 58
87.21
80.81
80.75
61.85
50.98
75.60
Glister
70.55
56.56
CIFAR-10
20%
Forgetting
76.27
80.06
85 66
87.18
82.04
81.35
64.59
52.21
76.17
DATM
85,50
85,23
87.22
88,13
84.65
85,14
66.70
52.40
79.37
12.64
14.85
+3.77
Random
42,80
46.18
47.48
55.62
42,69
38.05
25.91
20.66
39.95
Glister
35.45
37-13
28.
23.33
34.16
CIFAR-100
20%
Forgeting
45.52
49.99
51.44
54.65
43.28
43.47
27.22
22.90
42.30
DATM
57.50
57.75
$7.98
63.34
$5,10
55.69
33,57
26,39
50.92
+10.50
+17.64
+7.66
Random
15.00
24.21
17.73
28.07
22,51
14,03
9:25
5.85
17.08
Glister
17.32
19.77
18.84
23.12
19,10
11.68
8.84
3.86
15.32
Forgeting
12.13
5.54
18,20
DATM
39.68
40.32
36.12
43.14
38,35
35.10
12.41
9.02
31.76
+24.68
+16.11
+18.39
+15.07
+15.84
+21.07
+3.16
+3.17
+14.68
Table 4: We evaluate our distilled lossless datasets on unseen networks and compare them with two coreset selection methods. Results worse than random selection are indicated with red color. 1 denotes the performance improvement brought by our method compared with random selection. TI denotes Tiny ImageNet.
2015). To see if this information can be directly integrated into the distilled datasets, we assign soft labels for datasets distilled by FTD (Du et al., 2023).
As shown in Table 3. directly assigning soft labels for the distilled datasets will even hurt its performance when the number of categorics in the classification problem is small. For CIFAR-100 and Tiny ImageNet, although assigning soft labels slightly improves the performance of FTD, there still a huge gap between its performance and ours. This is because the soft labels synthesized by method are optimized constantly during the distillation, leading to better consistency between the synthetic data and their labels.
Furthermore, the information contained in logits varies with the capacity of the teacher model (Zong et al., 2023; Cui et al., 2023). In the experiments reported in Figure 3 (b), we use models trained after different epochs initialize the soft labels. In the beginning. this difference has non-trivial influences on the performance of the synthetic datasets. However, the performance gaps soon disappear as the distillation goes on if labels are optimized during the distillation. This indicates learning labels relieves us from carefully selecting models to initialize soft labels. Moreover, as can be observed in Figure 3 (b), when soft labels are not optimized, the distillation becomes less stable, leading to the poor performance of the distilled dataset. Because using unoptimized soft labels will enlarge the discrepancy between the training trajectories over the synthetic dataset and the original one, considering the experts are trained with one-hot labels. More analyses are attached in Section A.2.2.
Synthetic data initialization. To avoid mislabeling, we initialize the synthetic dataset by randomly sampling data from subset of the original dataset, which only contains samples that can be correctly classified by the model used for initializing soft labels. The process of constructing the subset can be viewed simple coreset selection. Here perform ablation study to see its effect. Specifically, synthetic datasets are either initialized by randomly sampling data from the original dataset (random selection) a subset of data that can be correctly classified by pre-trained ConvNet (ours).
As shown in Fig (a), our initialization strategy can significantly speed up the convergence of the optimization. This is because data selected by our strategy are relatively easier for DNNs learn. Thus, models trained on these easier samples will perform better when only limited training data are provided (Sorscher al., 2022). Although this gap is gradually bridged as the distillation goes on, our initialization strategy can be utilized as distillation speed-up technique.
EXTENSION
5.1
VISUALIZATION
For a better understanding of easy patterns and hard patterns, we visualize the distilled images and discuss their properties. In Figure 4, visualize the images synthesized by matching early trajectories and late trajectories under the same IPC setting, where easy patterns and hard ones are learned respectively. In Figure 5, visualize the images distilled under different IPC settings.
7

---

Published as a conference paper at ICLR 2024
Match Early Trajectories
Original Images
Match Late Trajectories
Figure 4: We perform the distillation on CIFAR-10 with IPC=50 by matching either early trajectories {0. < t s 10} late trajectories {0, < t < 40). All synthetic images are optimized 1000 times. Matching earlier trajectories will blur the details the target object and change the color more drastically.
IAD
Random
ConvNet
ResNet18
END
Random
ComvNet
ResNet18
PDD
Randocn
ConvNet
ResNeti8
ResNar18
26.70
48.66
45
ResNer18
54.96
66.27
61.18
ResNet18
84.58
87.22
87.70
VGGI
31,65
45.93
VGGI1
50.22
VGGU1
84.05
84.85
29.17
MLP
34.41
IPC=10
IPC=50
IPC=1000
Table 5: We use ConvNet and R8 to perform the distillation (D) on CIFAR- 10 with various IPC settings. Then evaluations (E) are performed using networks with various architectures. As IPC increases, datasets distilled using ResNet18 perform relatively better.
As can be observed Figure 4, compared with hard patterns, the learned easy patterns drive the synthetic images to move farther from their initialization and tend to blend the target object into the background. Although this process seems to make the image more informative, blurs the texture and fine geometric details of the target object, making it harder for networks to learn to identify non-typical samples. This helps explain why generating easy patterns turned out to harmful in high IPC cases. However, generating easy patterns performs well in the regime of low IPC, where the optimal solution limited data budget. For example, shown in Figure 5, the synthetic images collapse almost only contain color and vague shape information when IPC extremely low, which helps networks learn to identify easy samples from this basic property.
model the most dense areas of the target category's 's distribution given the
Furthermore, we find matching late trajectories yields distilled images that contain more fine details. be observed in Figure 4, matching late trajectories transforms the simple background in the dog images into a more informative one and gives more texture details to the dog and horse. This transformation helps networks learn identify outlier (hard rd) samples; hence, matching late trajectories is better choice high IPC cases.
For example,
5.2 DISTILLATION COST
In this work, we scale dataset distillation to high IPC cases. Surprisingly, find the distillation cost does not increase linearly with IPC, since the optimization converges faster in large IPC cases. This is because only make few changes on the images, as we have analyzed Section 5.1 and can be observed in Figure 5. In practice, reflected in Figure 3 (c), although the distillation with IPC=1000 needs optimize 20x more data than the case with IPC=50, the former one's cost is only 1.05 times higher.
match only late trajectories in high IPC cases, where the learned hard patterns
5.3 DISTILLATION BACKBONE, NETWORKS
So far, almost all representative distillation methods choose to ConvNet to perform the distillation (Zhao Bilen, 2021; Cazenavette et al., 2022; Loo et al., 2023). Using other networks as the distillation backbone will result in non-trivial performance degradation (Liu et al., 2023b). What makes ConvNet so effective for distillation remains an open question.
Here offer answer from our perspective: part of the specialness of ConvNet comes from its low capacity. general, networks with more capacity can learn more complex patterns. Accordingly, when used distilling networks, their generated patterns relatively harder for DNNs leam.
8

---

Published as a conference paper at ICLR 2024
Original Image
IPC=1
IPC=10
IPC=100
Original Image
IPC=1
IPC=10
IPC=50
CIFAR-100
Tiny ImageNet
Figure Visualization synthetic datasets distilled with different IPC settings. As IPC increases, synthetic images move less far from their initialization.
As have analyzed in section 3.1, in small IPC cases (where previous distillation methods focus their attention), most improvement comes from the easy patterns generated on the synthetic data. Thus networks with more capacity such ResNet will perform worse than ConvNet when IPC is low because their generated patterns are harder for DNNs to learn. However, in high IPC cases, where hard patterns play an important role, using stronger networks should perform relatively better. To verify this, we use ResNet18 8 and ConvNet to perform distillation on CIFAR-10 with different IPC settings. As shown in Table 5, when IPC low, ConvNet performs much better than ResNet18 the distillation network. However, when IPC reaches 1000, ResNetl 8 has a comparable or even better performance compared with ConvNet.
RELATED WORK
Dataset distillation introduced by Wang et (2018) is naturally a bi-level optimization problem, which aims at distilling large dataset into a small one without causing performance drops. The following works can be divided into two types according to their mechanism:
Kernel-based distillation methods use kernel ridge-regression with NTK (Lee al., 2019) to obtain a closed-form solution for the inner optimization (Nguyen et al., 2020). By doing so, dataset distillation can be formulated significantly reduced the training cost (Zhou et al., 2022) and improved the performance (Loo et al., 2022; 2023). However, since the heavy resource consumption of inversing matrix operation, it is hard to scale kernel-based methods to larger IPC.
single-level optimization problem. The following works have
Matching-based methods minimize defined metrics of surrogate models learned from the synthetic dataset and the original one. According to the definition of the metric, they can be divided into four categories: based on matching gradients (Zhao et al., 2020; Kim et al., 2022: Zhang al., 2023), features (Wang et al., 2022), distribution (Zhao & Bilen, 2023; Zhao et al., 2023), and training trajectories (Cazenavette al., 2022; Cui et al., 2023; Du et al., 2023). So far, trajectory matching- based methods have shown impressive performance on every benchmark with low IPC (Cui et al., 2022; Yu et al., 2023). In this work, we further explore and show great power higher IPC cases.
CONCLUSION AND DISCUSSION
In this work, find the difficulty of patterns generated by dataset distillation algorithms should be aligned with the size of the synthetic dataset, which is the key to keeping them effective both low- and high-IPC cases. Building upon this insight, our method excels only in low IPC cases but also maintains its efficacy in high IPC scenarios, achieving lossless dataset distillation for the first time.
However, our distilled data are only losslesy for the distillation backbone network: when evaluating them with other networks, the performance drops still exist. We think this is because models with different capacities need varying amounts of training data. How overcome this issue is still challenging problem. Moreover, it is hard to scale TM-based methods to large datasets due to its high distillation cost. How to improve its efficiency would be the goal of our future work.

---

Published as a conference paper at ICLR 2024
Acknowledgements. This work is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-PhD-2021-1 008). This work is also supported in part by the National Key R&D Program of China (NO.2022Z0016010) Part of Hui Li's work supported by the National Natural Sci-
and N0.20227AD110101), Foundation of China (61932015), Shaanxi Innovation Team project (2018TI--00),, Higher Education Discipline Innovation 111 project (B (B16037). Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant (WBS number: A-8001 104-00-00), Alibaba grant, and Google grant for TPU usage. We thank Bo Zhao for valuable discussions and feed- back.
REFERENCES
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et closer look at memorization in deep networks. In ICML, 2017. 2,
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. In NewrlPS Workshop. 2020. 16
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun Yan Zhu. Dataset distillation by matching training trajectories. CVPR, 2022. 1, 2, 5, 8, 9, 16, 17
Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private set generation with discriminative information. NeurIPS, 2022.
Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirzasoleiman. Data distillation be like vodka: Distilling more times for better quality. In ICLR, 2024. 16
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation benchmark. In NeurlPS, 2022. 9,
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- lk with constant memory. In ICML, 2023. 5,
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In ICML, 2022.
Jiawei Du, Yidi Jiang, Vincent YF Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the ac- cumulated trajectory error to improve dataset distillation. In CVPR, 2023. 1, 2, 5, 9, 14, 17
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NeurlPS Workshop, 2015,
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In ICLR, 2022. 16
Krishnateja Killamsetty, Durga Sivasubramanian. Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In AAAI, 2021.
Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In ICML, 2022. 1,9
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurIPS, 2012. 5
10

---

Published as a conference paper at ICLR 2024
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. 5
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NewrIPS, 2019. 9
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In NeurIPS, 2022. 16
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In CVPR, 2023a. 16
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. ICCV, 2023b. 1, 8, 16
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In NeurlPS, 2022. 9
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. In ICML, 2023. 5, 8, 9
Wojciech Masarczyk and Ivona Tautkute. Reducing catastrophic forgetting with learning on synthetic data. In CVPR Workshop, 2020. 1
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-leaming from kernel ridge- regression. In ICLR, 2020. 1, 5,
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In NeurlPS, 2021. 1
Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. In International Workshop on Continual Semi-Supervisedd Learning, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurIPS, 2022. 4, 7, 15
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In /CML, 2020.
Ilia Sucholutsky and Matthias Schonlau. Secdd: Efficient and secure method for remotely training neural networks (student abstract). AAAI, 2021a.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In International Joint Conference on Neural Networks, 2021b. 16
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR,, 2018. 6
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022. 5,
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. arXiv, 2023. 16
11

---

Published as a conference paper at ICLR 2024
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable nas. In ICLR, 2021.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:/811.10959, 2018. 16
Xindi Wu, Byron Zhang, Zhiwei Deng, and Olga Russakovsky. Vasion-language dataset distillation. arXiv preprint arXiv:2.38..075544 075445 2023. 16
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsich. Feddm: Iterative distribution matching for communication-eficcee federated learning. CVPR, 2023.
Beining Yang, Kai Wang, Qingyun Sun, Cheng Ji, Xingcheng Fu, Hao Tang, Yang You, and Jianxin Li. Does graph distillation see like vision dataset counterpart? In NeurIPS, 2023. 16
Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: comprehensive review. arXiv preprint arXiv:2301.07014 2023. 9
Lei Zhang, Jie Zhang, Bowen Lei, Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. In CVPR, 2023. 1, 16
Tianle Zhang, Yuchen Zhang, Kun Wang, Kai Wang, Beining Yang, Kaipeng Zhang, Wenqi Shao, Ping Liu, Joey Tianyi Zhou, and Yang You. Two trades is not baffled: Condensing graph via crafting rational gradient matching, 2024a. 16
Yuchen Zhang, Tianle Zhang, Kai Wang, Ziyao Guo, Yuxuan Liang, Xavier Bresson, Wei Jin, and Yang You. Navigating complexity: Toward lossless graph condensation via expanding window matching. arXiv preprint arXiv:2402.050111 2024b.
Bo Zhao and Hakan Bilen. Dataset condensation with diffcrentiable siamese augmentation. In ICML, 2021. 8
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023. 1, 9
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In ICLR, 2020. 5,
Ganlong Zhao, Guanbin Li. Yipeng Qin, and Yizhou Yu. Improved distribution matching for dataset condensation. In CVPR, 2023. 1,9
Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng. Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. arXiv preprint arXiv:2308. /0524, 2023. 2
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.09999, 2020.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regres- sion. In NeurIPS, 2022. 5,
Martin Zong. Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In ICLR, 2023. 7
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 SOFT LABEL DISTRIBUTION
To observe the changes in soft labels' distribution during the distillation, we record the standard deviation (std) of soft labels (after softmax) for each synthetic image, and report the average value of their std in Figure 6. As can be observed, for all datasets, their labels' std tends to increase as the distillation goes on. However, this increase does not arise due to the diversity between the values of soft labels becoming larger, but the values of non-target categories' labels are suppressed. Since the value the target category much higher than others, to facilitate observation, we only report the values of non-target categories in Figure 7. As can be observed, after the distillation, the values of non-target categories' labels are suppressed more drastically when IPC is smaller. This is because, in low IPC cases, basic patterns of target category are embedded into the synthetic data since only early expert trajectories are used for the matching. Accordingly, the model becomes more confident that the generated sample belongs the target category.
A.2 SOFT LABEL INITIALIZATION
We have tried to initialize soft labels with the original one-hot labels and directly optimize their values during the distillation, but the distillation soon crashed. We also have tried to add softmax layer on it. However, the distillation is still not stable. After initializing labels with class probabilities calculated using softmax and logits output by a pre-trained model, finally, soft labels can be optimized stably during the distillation.
For labels with a given distribution, their values before softmax can be different. this case, due to the utilization of softmax, when performing backpropagation, the gradients of pre-softmax logits will also be different even if their values after softmax are the same. Through experiments, we find using appropriate distribution before softmax initialize soft labels is crucial to maintaining the stability of the distillation. We have also tried to modify the distribution of logits without changing their values after softmax. However, we see this operation will greatly increase the instability of distillation. Moreover, we have tried to scale the values of logits during the initialization, which also leads to the degradation of performance.
A.2.1 STABABILITY
In the manuscript, propose generate easy and hard patterns sequentially make the distillation stable enough for learning soft labels. The insight here is enabling the surrogate model to learn more easy patterns through the finite training steps and limited samples in inner optimization, such that the model can match the expert trajectories better. We find that simply increasing the update times in inner optimization can also stabilize the distillation. Because the surrogate model can learn easy patterns better through longer learning process. However, increasing the update times in inner optimization will increase the memory requirement and the training cost. Thus we use the method introduced in the manuscript by default.
A.2.2 SOFT LABEL OPTIMIZATION
We can choose to only replace original one-hot labels with soft labels but don't 't optimize them during the distillation. However, this will make the surrogate model harder to match the expert training trajectories, because the expert trajectories are trained with one-hot labels. As reflected in Figure (Left), when soft labels are not optimized, the matching loss is higher than using one-hot labels. Although using unoptimized soft labels still performs better than one-hot labels because of the additional information contained in the soft labels, its performance is undermined by the under-matching.
The muder-matching issue can alleviated by optimizing soft labels during the distillation. As can be observed in Figure (Left), when soft labels are optimized during the distillation, the matching loss becomes lower than using one-hot labels. Accordingly, the performance is improved, as can be observed in Figure 8 (Right).
13

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-100
Tiny ImageNet
Avarage Standard Destalion
Average Standard Devtation
Average Standrd Devistion
0.294
0.075
0.05
0.070
0.200
0.065
0.03
Figure 6: Visualization of the changing of soft labels in various datasets where IPC=50. The average standard
deviation of soft labels tends to increase as the distillation goes on.
tarations,
3.005
0.006
5,005
Lobel Valu
0.004-
LabelVabee Valoe
0.004
Value
0.004
0.002
0.002
3.002
8
Categgry
Category
Category
Figure 7: Visualization of the distribution of non-target soft labels of synthetic image initialized with the same soft labels and image but distilled with different IPC settings. The values of non-target categories are suppressed more drastically when IPC is smaller.
A.3 ABLATION ON SYNTHETIC STEPS
We find increasing the synthetic steps N (Algorithm 1) will bring more performance gain when soft labels are utilized, reflected in Figure 9 (Left). This is because the optimization of the surrogate model in the inner optimization affects how well it can match the expert trajectories. When soft labels are not utilized, the information contained in the synthetic dataset is relatively limited, thus the surrogate barely benefits from a longer learning process. Moreover, we find increasing the synthetic steps N can also bring improvement in high IPC cases. As shown in Figure 9 (middle), setting N=80 performs best in the case with IPC=1000, where the batch size is set to 1000. In this case, in every iteration, the parameters of the surrogate model are optimized over 80K images (10K unduplicated images) contained in the synthetic dataset, while the target parameters are obtained by the optimization over 100k images (50k unduplicated images) contained in the original datasets. In this case, although the length of the training trajectory over the synthetic dataset Dayn and the original dataset Dreal are similar, matching trajectories still can improve the training performance of the synthetic datasets. Based this observation, we conjecture that the key to keeping TM-baded methods effective is to ensure the number of unduplicated images contained in Dayn smaller than that of Dreat, rather than use the short trajectory trained Dayn match the longer optimized over Dreal-
A.4 PREVIOUS TM-BASED METHODS IN LARGE IPC SETTINGS
We have tried to use previous TM-based methods to perform the distillation in larger IPC settings. The distillation logs of FTD (Du et al., 2023) are reported in Figure 9. As can be observed, FTD will undermine the training performance of the datasets in larger IPC cases. We have tried to tune its syper-parametery including the learning rate, batch size, synthetic steps, and the upper bound of the sample range, but the effort can only slow down the rate of performance degradation it arose. Without aligning the difficulty of the generated patterns with the size of the synthetic datasets, previous T'-based methods can not keep being effective in high IPC settings.
14

---

Published as a conference paper at ICLR 2024
CIFAR-1ON AROR, IPC=10
CIFAR-100, IPC=10
0.9
50
Matching Loss
0.7-
43
Acc (99)
0.5
36-
bookk
burring astt uses
wdo warriing baces
0.3
29
lteration (3)
Iteraiio (k)
Figure 8: Left: Logs of the matching loss (smoothed with EMA), where the labels of the synthetic dataset are either one-hot labels, unoptimized soft labels, soft labels that are optimized during the distillation. Right: Logs performance of the distilled datasets. Learning soft labels during the distillation enables surrogate models match expert trajectories better. Accordingly, its synthetic datasets have a better performance.
N=
N=80
(%)
Acc. 196)
46
78
IPC-1000
IPC-500
IPC-50D
IPC-1000
72
w/o soft label
W/ soft label
40
80
100
Synthetic Steps (N)
Iteration (2.0
Figure 9: Left: (CIFAR-100, IPC=10) Ablation on synthetic step N, distillation with soft labels benefits more
from larger N. Middle: (CIFAR-10) Distillation in larger IPC settings still benefit from higher larger N. Distillation log FTD on CIFAR-10 with larger IPC. The performance the synthetic datasets keeps being degraded as the distillation goes on.
Right:
A.5 MORE DETAILED ANALYSIS ON MATCHING Loss
Here we provide more results and additional analysis about the matching loss over expert trajectories. As can be observed in Figure 10, initially, the matching loss over the former part of the trajectories is always lower than the one over the later part. This indicates earlier trajectories are relatively easier to match compared with the later ones. In other words, the patterns that surrogate models need to learn to match the early trajectories are relatively easier.
Moreover, it is interesting observe that matching later trajectories also reduce the matching loss over early trajectories in high IPC cases. We hypothesize that this because, in the late training phases, DNNs just prefer learn hard patterns help identify the outlers, while few easy patterns are also learned in late training phases. From this perspective. TM-based methods might not be the most efficient way distill datasets with large IPC, since matching later trajectories still will generate a few easy patterns.
We can also find that when IPC small, matching late trajectories will raise the matching loss over the early trajectories. This indicates generating ha rd patterns is harmful for the model to learn basic (easy) patterns obtain the basic capacity to perform the classification when data is limited. This coincide with the observation in the dataset pruning area: preserving hard samples perform worse when only limited samples are kept (Sorscher et al., 2022).
A.6 GUIDANCE FOR ALIGNING DIFFICULTY
Our difficulty alignment aims at letting the models trained on the synthetic dataset learn as many hard patterns as possible, without compromising their capacity to classify easy patterns. For TM-based methods, this can be quantified by the matching loss over iestillationuuniivlllee trajectory, as we have analyzed section A.5. Specifically, we want to add patterns that can help to reduce the matching loss over the latter part of the expert trajectory without increasing the matching
expert
15

---

Published as a conference paper at ICLR 2024
Algorithm 1: Pipeline of our method
Input: {-"}: set of expert parameter trajectories. N: update times of the surrogate network each inner optimization. M: update times between the start and target expert parameters.
TT,T. : lower, current upper, final upper bound of the sample range of Dreat: original dataset. I: interval for expanding the sampling range.
Sample model fø- from {T"}.
Construct Daub {(1,, softmax softmax(L))) (L:)) v:) Dreat and argmax(L.) = y:3. where
L, = fo (z;).
Randomly sample data from Suub to initialize synthetic dataset D Dayu-
for iteration +- 0 to max_ mix_ttraaiion do
Randomly sample an expert training trajectory T E {T*} with -0, Select random start timestamp i, where T < St ST
Sample 0j 07- M from r", initialize O, 0j
for i 0 to N-1 do
bi++ ~ Dayn
sample mini-batch of distilled dataset b update surrogate model with CE loss
0eti+1 = 8iti V bete)
Compute matching loss between Oc+ and 07 with Eq. 1
Update (I,,, softmax
(D)(+N-1 and a with respect the matching loss
(iteration % I == 0) and (T T+) then
L T ++1
Output: distilled dataset Dayu and learning rate a
loss over the former part. In practice, we realize it by only matching a certain part of the expert trajectories during the distillation.
Here we introduce how to tune the values of T-, T. and T (Algorithm 1). For initialized synthetic dataset Dayn. we first set 7- = and T = T + 10 to perform the distillation for 50 iterations, where the matching loss Then we simultaneously increase the values of T and T until the distillation will not increase the matching loss over the latter part of the expert trajectory. Subsequently,
distillation-uminvor
expert trajectory is recorded.
increase the value of T until the distillation will increase the matching loss over the former part of the expert trajectory. After deciding the values of T and 7* we let them respectively be the lower- and upper-bound of the sample range and perform the distillation. During the distillation, we record the value of t if the matching loss larger than 1, which denotes the surrogate model can match the expert trajectory. Then T' is set as the minimum recorded value, to avoid matching too hard trajectories in the beginning.
A.7 MORE RELATED WORK
Two early works (Bohdal et al., 2020; Sucholutsky & Schonlau, 2021b) in the dataset distillation area also focus on optimizing the labels of the datasets. Specifically, based on the dataset distillation algorithm proposed by Wang et al. (2018), Sucholutsky & Schonlau (2021b) propose to optimize labels during the distillation, while Bohdal et al. (2020) choose to only distill soft labels without optimizing the training data. Different from them, we use the pre-trained model to initialize soft labels, which contain more information. Furthermore, our method is based on matching training trajectories (Cazenavette et al., 2022) rather than the method proposed by Wang et al. (2018).
Recently, several methods are proposed to improve the performance, efficiency and suitability of dataset distillation. For example, Liu al. (2022) proposed to use hallucinations to enlarge the synthetic datasets in the deployment stage. Subsequently, Wang et al. (2023) achieved this goal by distilling the target dataset into generative model. Moreover, (Zhang et al., 2023; Liu et al., 2023b) were proposed to accelerate the distillation and Liu et al. (2023a) proposed method that allows adjusting the size of the distilled dataset during the deployment stage. Recently, Chen et al. (2024) proposed to improve the quality of the synthetic dataset with carefully designed distillation schedule. Moreover, dataset distillation has also been successfully applied in condensing gragh data (Jin et al., 2022; Yang et al., 2023; Zhang et al., 2024a;b) and multi-modality data (Wu al., 2023).
16

---

Published as a conference paper at ICLR 2024
Loa
L20
Log
3.
1.
0.4
Laation
fansion
lansion
tanaton
20
1.1
1.3
0.7
Figure 10: More detailed results of experiments reported in Figure 2. We train the expert models CIFAR-1O for 40 epochs. Then we distill datasets with two strategies: (1) matching the early part of expert training trajectories. where 0g E {0g...0a.. (2) matching the latter part of expert training trajectories, where 02 E (0.0...0..) The first row shows the matching loss over a distillation performed with strategy I, and the second distillation-uninooeeer uninvolved expert trajectory, where the distillation is darker color indicates the matching start parameters used matching (Algorithm 1). The last row shows the performance of the datasets distilled by different strategies with various IPC settings.
shows the loss of strategy 2. In the first rows, lines with over more later part the trajectories. denotes timestamp of the
A.8 SETTINGS
Distillation. Consistent with previous works (Cazenavette et al., 2022; Du et al., 2023), perform the distillation for 10000 iterations to make sure the optimization is fully converged. We use ZCA whitening in all the involved experiments by default.
Evaluation. We keep our evaluation process consistent with previous works (Cazenavette et al., 2022; Du al., 2023). Specifically, we train randomly initialized network on the distilled dataset and then evaluate its performance on the entire validation set of the original dataset. Following previous works (Cazenavette et al., 2022; Du al., 2023), the evaluation networks are trained for 1000 epochs to make sure the optimization is fully converged. All the results are the average over five trials. For fairness, experimental results of previous distillation methods low IPC settings are obtained from (Du et al., 2023), while their results in high IPC cases come from (Cui et al., 2022).
Since the exponential moving average (EMA) used in FTD (Du et al., 2023) technique that hasn't it been utilized by previous matching- based methods, for a fair comparison, reproduce FTD with the official released code without using EMA. Accordingly, we do not use EMA in our method.
plug-and-play
Network. We use various networks to evaluate the generalizability of our distilled datasets. Specifi- cally, scale ResNet, LeNet, and AlexNet ImageNe, increase the stride of their first convolution layer from to 2. For VGG, we increase the stride of its last max pooling layer from to 2. The MIB _ utilized in our evaluation has one hidden layer with 128 units.
17

---

Published as a conference paper at ICLR 2024
Plyper-paramette We report the hyper-paramett of our method in Table 6. Additionally, for all the experiments with optimizing soft labels, we set its momentum to 0.9. We find learning labels with low momentum will somewhat increase the instability of the distillation. We conjecture this is because the optimized soft labels are easy to overfit the expert trajectories considering we only match trajectory in each iteration.
Compute resources. Our experiments are run on 4 NVIDIA A100 GPUs, each with 80 GB of memory. The heavy reliance on GPU memory can be alleviated by TESLA (Cui et al., 2023) or simply reducing the synthetic steps N, which will not cause too much performance degradation. For example, reducing the synthetic steps N from 80 40 saves about half the GPU memory. while it only makes the performance drop by around 0.8% for CIFAR-10 with IPC=1000, 0.7% for CIFAR- 100 with IPC=100, and 0.4% for Imageee with IPC=50.
Synthetic Batch
Learning Rute (Label)
Leurning Rate
Dataset
IPC
N
M
T
T
T+
Interval
(Pixels)
1
80
2
0
4
10
5
100
10
80
2
0
10
20
100
100
2
100
CIFAR-10
50
80
2
40
100
500
1000
2
1000
80
2
1000
50
40
3
0
10
20
100
100
10
1000
10
80
2
50
100
1000
1000
CIFAR-10 100
1000
100
80
2
50
1
60
2
0
15
20
400
200
10
10000
TI
10
60
2
10
50
50
250
100
50
80
2
100
Table 6: Hyper-parameters for different datasets.
18

---

Published as a conference paper at ICLR 2024
Figure 11: (Tiny ImageNet, IPC=1) Visualization of distilled images (1/2).
19

---

Published as a conference paper at ICLR 2024
Figure 12: (Tiny ImageNet, IPC=1) Visualization of distilled images (2/2).
20

---

Published as a conference paper at ICLR 2024
ATM
Figure 13: (Tiny ImageNet, IPC=50) Visualization of distilled images (1/2).
21

---

Published as a conference paper at ICLR 2024
Figure 14: (Tiny ImageNet, IPC=50) Visualization of distilled images (2/2).
22


Reviews:
[{"review_id": "o24HDaHErG", "reviewer": "Reviewer_5Mev", "summary": "As a compression algorithm, it remains unclear whether dataset distillation can be used to fully recover the full training accuracy of original dataset. The authors in this paper propose to address this issue through specifically aligning the difficult trajectories using part of the data. The experiment result show promising results.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed framework is quite well motivated, and the experiment results aligns well with the observation\n+ In the experiments, the authors are the first to demonstrate that distilled synthetic data can outperform real training dataset\n+ The authors perform extensive studies and analysis on the algorithm", "weaknesses": "- The algorithm is extending MTT, although important, but there is not much algorithmic novelty\n- In table 4, authors only compared with some simple baselines for cross-architecture generalization. It would be great if the authors can have a more thorough comparison with other dataset distillation algorithms as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D1m64lmZH8", "reviewer": "Reviewer_8eCM", "summary": "The paper introduces a novel method of dataset distillation. It introduces the concept of difficulty-aligned trajectory matching, which controls the difficulty of the patterns generated on the synthetic data by matching different training phases of the expert model. \n\nOverall, the paper makes a substantial contribution to the field of dataset distillation, introducing a novel method and providing valuable insights. Despite the need for clarification on certain points and a deeper exploration into the method’s performance across different scenarios, the paper stands out for its clarity, quality, and originality. With sufficient details provided for reproducibility, this work paves the way for further research and development in trajectory matching-based dataset distillation algorithms.", "questions": "I have a few questions and concerns about the method:\n\nQ1. Manually setting lower bound and upper bound for the matching range is somehow incremental.\n\nQ2. From Table 2a, the performance of “complex” networks is worse than 3-layer convolutional networks. While in table 2, the performance of “complex” networks is better than the 3-layer convolutional networks. It would be interesting to see the performance across architectures with the increasing number of IPC.\n\nQ3. How is the synthetic dataset performance on the downstream tasks, such as object detection?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper provides a novel and intuitive insight into the effect of matching different training trajectories on the quality of the distilled dataset. The observation in the paper is insightful for the trajectory matching based dataset distillation. The paper is well-written and organized, with clear problem formulation, method description, and experimental setup. \n\nClarity, Quality, Novelty And Reproducibility:\nThe paper is clearly written, and the idea seems novel. The implementation details are provided for reproducing", "weaknesses": "Please refer to the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3enWbgzmM", "reviewer": "Reviewer_G6x3", "summary": "This work aims to achieve lossless dataset distillation by synthesizing a small synthetic dataset such that a model trained on this synthetic set will perform equally well as a model trained on the full, real dataset. Concretely, the author identified the problem of performance not being maintained as the size of the synthetic dataset grows. The authors propose a way to process early trajectories (easy patterns) and late ones (hard patterns) separately.", "questions": "1) Please explain why the performance across model architecture is so low.\n\n2) Please see my statements in weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The author identified an important problem from an interesting perspective. The current dataset distillation algorithms cannot handle it well.\n\n2) The model performance on the synthesized dataset is better than that on the dataset synthesized by other dataset distillation methods.", "weaknesses": "1. The goal of this work is to achieve lossless dataset distillation. This requires a formal definition, what is lossless dataset distillation exactly?  Do models achieve the same performance on the original dataset as on the synthesized one?\n\n2. Why the model performance on the synthesized one is better than the one on the full dataset? Does that mean that the proposed whole dataset distillation works like a regularization method somehow?\n\n3. If a synthesized dataset is indeed lossless, a model with a totally different architecture can also perform well on the synthesized dataset. There lack of such experiments.\n\n4. This work improves previous work when the size of the synthesized dataset becomes larger. The lossless dataset distillation is an overclaim to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pY3QPRGjze", "reviewer": "Reviewer_VsZC", "summary": "The papers proposes a new method to perform dataset distillation under large IPCs. The authors made server observations regarding the trajectory matching process and found that easy patterns are learnt at early states and hard patterns are learnt at later stages. The authors also found that learning labels will help boost the distillation performances. Through gradually increasing the limit of matched expert trajectory epochs, the method achieves SOTA performances on large IPCs. Especially, when IPC increases to 500 or 1000 where all previous methods fail to perform better than random baseline, the proposed method is the first to be able to perform equivalently well as training on the whole dataset or better. The proposed method also achieves SOTA results on cross-architecture evaluations.", "questions": "See my questions above inside weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the method solves the problem that previous DD methods failed to solve: when IPC becomes large such as 500, their performances start to degrade to similar levels as random selection.\n2. The paper is well written and easy to follow.\n3. The paper introduced the observations gradually and solved the problems through carefully designed strategies. \n4. The comparison to baseline methods are extensive. \n5. The method is the first to scale to large IPCs such as 500, 1000 and achieves lossless performance on the 3 datasets.\n6. The paper also showed lots of insights for the reason community to gain a better understanding of Matching Training Trajectories(MTT) which is one of the recently proposed SOTA methods.", "weaknesses": "1. In table 1, can the authors indicate which DATM's results are achieved with label learning and which are not? Performance-wise, DATM's performance is very similar to other MTT based methods such as MTT itself with small IPCs, if soft label is used, does that mean DATM will cause MTT's performance to drop under smaller IPCs which is then compensated by soft label learning?\n\n2. Although mentioned briefly in 5.2, can the authors also specify the training resources such as number of GPUs and their type and the training cost in either 4.1 or appendix. This will provide a guidance for other researchers, especially on large IPCs and datasets. This leads to my next question\n\n3. Does this 1.05 times higher cost apply to all settings or it's just for CIFAR-10 (Sec 5.2 mentioned it's for CIFAR-10)? How about the extra training cost on other datasets?\n\n4. Some claims conflicts with the settings adapted by the method, e.g. in section 4.3, the authors first mention that \"We use logits generated by the pre-trained model to initialize soft labels\", then later in the same section, the authors claim again \"This is because using soft labels without optimizing them will enlarge the discrepancy between the training trajectories over the synthetic dataset and the original one, **considering the experts are trained with one-hot labels**\". If the later claim (considering the experts are trained with one-hot labels) is correct, shouldn't initialize the soft labels with a distribution close to one-hot have a smaller gap with expert training trajectories during the matching phase? The decision to use pre-trained model to initialize soft labels can be understood, however I think the explanation given later is confusing\n\n5. Have the authors tried to increase $T^{-}$ as well instead of just increasing the current upper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)