Embarrassingly Simple Dataset Distillation
Authors: Yunzhen Feng
Shanmukha Ramakrishna Vedantam
Julia Kempe
Venue: ICLR 2024 poster (2024)
Keywords: Dataset Distillation
Data Condensation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a new simple method for dataset distillation to achieve state-of-art results on a vast majority of benchmarks, providing further improvements through a boosted variant.

Abstract: Dataset distillation extracts a small set of synthetic training samples from a large dataset with the goal of achieving competitive performance on test data when trained on this sample. In this work, we tackle dataset distillation at its core by treating it directly as a bilevel optimization problem. Re-examining the foundational back-propagation through time method, we study the pronounced variance in the gradients, computational burden, and long-term dependencies. We introduce an improved method: Random Truncated Backpropagation Through Time (RaT-BPTT) to address them. RaT-BPTT incorporates a truncation coupled with a random window, effectively stabilizing the gradients and speeding up the optimization while covering long dependencies. This allows us to establish new state-of-the-art for a variety of standard dataset benchmarks. A deeper dive into the nature of distilled data unveils pronounced intercorrelation. In particular, subsets of distilled datasets tend to exhibit much worse performance than directly distilled smaller datasets of the same size. Leveraging RaT-BPTT, we devise a boosting mechanism that generates distilled datasets that contain subsets with near optimal performance across different data budgets.

Published as a conference paper at ICLR 2024
EMBARRASSINGLY SIMPLE DATASET DISTILLATION
Yunzhen Feng" |
Ramakrishna Vedantam" Julia Kempe*
Center for Data Science, New York University
f Courant Institue of Mathematical Sciences, New York University Eyf2231 @nyu.edu
ABSTRACT
Dataset distillation extracts a small set of synthetic training samples from a large dataset with the goal of achieving competitive performance test data when trained on this sample. In this work, we tackle dataset distillation at its core by treating it directly as a bilevel optimization problem. Re-examining the founda- tional pack-propagation through time method, we study the pronounced variance in the gradients, computational burden, and long-term dependencies. We intro- duce an improved method: Random Truncated Backpropagation Through Time (RaT-BPTT) address them. RaT-BPTT incorporates truncation coupled with a random window, effectively stabilizing the gradients and speeding up the op- timization while covering long dependencies. This allows us to establish new state-of-the-art for variety of standard dataset benchmarks. A deeper dive into the nature of distilled data unveils pronounced intercorrelation. In particular, subsets of distilled datasets tend to exhibit much worse performance than directly distilled smaller datasets of the same size. Leveraging RaT-BPTT, we devise a boosting mechanism that generates distilled datasets that contain subsets with near optimal performance across different data budgets.
INTRODUCTION
Learning deep, overparameterized neural networks with stochastic gradient descent, backpropagation and large scale datasets has led tremendous advances in deep learning. In practice, often observed that for deep learning algorithm to be effective, vast amount of training samples and numerous training iterations are needed.
In this work, we aim to explore the genuine necessity of vast training data and numerous training steps for achieving high test accuracy. To investigate, we limit the number of samples in the training set to be small (e.g., 1. 5, or 10 images per class) and the number of training steps to be small (e.g., on the order of 300 steps). This leads us to the concept of optimizing small synthetic dataset, such that neural networks trained on this dataset perform well the desired target distribution, problem known as Dataset Distillation (Wang et al. 2018).
This is instance of a bilevel optimization problem (Dempe) 2020) where the output of one optimization problem (in this instance, the learning algorithm trained on the small dataset) is fed into another optimization problem (the generalization minimize. In general, this problem is intractable, as the inner loop involves a multi-step computation with large number of steps. Early works (Wang al.. 2018 Sucholutsky & Schonlau, 2021 1 Deng
the target set) which we intend to
& Russakovsky] 2022) directly approached this problem via back-propagation through time (BPTT), unrolling the inner loop for a limited number of steps, before hitting an optimization bottleneck that called for alternative techniques. Later works have made steady progress by replacing the inner loop with closed-form differentiable surrogates, like the Neural Tangent Kernel (Nguyen et al. 2021a b). Neural Features (Zhou et al.) (022) and Gaussian Process (Loo et al. 2022). This approach often requires looping thought a diverse pool of randomly initialized models during the optimization to alleviate the mismatch between the surrogate model and the actual one. Moreover, these approaches are limited to MSE loss; they tend to work better on wider models, where the surrogate approximation the set of frequently used narrower models. Another
holds better, but give worse performance line of works has modified the outer loop objective using proxy mraiiing-mttric like matching the trajectories of the network (Cazenavettee al. 2022 or the gradients during training (Zhao & Bilen

---

Published as a conference paper at ICLR 2024
BPTT
Truncated
Ra-Truncated
Jcea
Jeap
Suage
outeri/f (2))
outer (B))
Forward Pars
11
IPC
Backward Pass
Figure 2: Boosting Dataset Distillation (Boost-DD). We start with 5 randomly initial- ized images per class, distill the dataset into them (Stage yielding five images per class (IPC5), then add five more random images and distill while reducing the learning rate on the first 5 (Stage 2) to yield IPC10, and so on, resulting in a nested dataset of different IPC. Boosting reduces higher order dependencies in the distilled datasets.
Figure 1: Illustration of bilevel optimization of the outer r loss when training for steps. We show Full Backpropagation Through Time (BPTT) (left), Truncated BPTT (middle) and our proposed Ran- domized Truncated BPTT (right) (RaT-BPTT). RaT- BPTT picks a window in the learning trajectory (randomly) and tracks the gradients dataset UI for the chosen window, as opposed to T- BPTT that uses fixed window, and BPTT that uses the entire trajectory.
the training
2021a). However, these methods either necessitate the storage of different trajectories or are impeded by subpar performance. These observations lead to the question: Does there exist a simple and direct method for dataset distillation?
In this paper, refine BPTT to address distinct challenges in dataset distillation, and achieve state-of-theearr performance across a vast majority of the CIFARIO. 10, CIFAR CUB, TinylmageNet, and ImageNet-1K benchmarks. We start by re-examining BPTT, the go-to method for bi-level optimization problems (Finn et al., (017b)] Lorraine et al. 2020). Notably, the inner problem of dataset distillation presents unique challenges- the pronounced non-convex nature when training a neural network from scratch on the distilled data. One has to use long unrolling of BPTT to encapsulate the long dependencies inherent in the inner optimization. However, this results in BPTT suffering from slow optimization and huge memory demands, consequence of backpropagating through all intermediate steps. This further complicated by considerable instability in meta- gradients, emerging from the multiplication of Hessian matrices during long unrolling. Therefore, the performance limited.
To address these challenges, we integrate the concepts of randomization and truncation with BPTT, leading to the Random Truncated Backpropagation Through Time (RaT-BPTT) method. The refined approach unrolls within a randomly anchored smaller fixed-size window along the training trajectory and aggregates gradients within that window (see Figure for a cartoon illustration). The random window design ensures that the RaT-BPTT gradient serves as random subsample of the full BPTT gradient, covering the entire trajectory, while the truncated window design enhances gradient stability and alleviates memory burden. Consequently, RaT-BPTT provides expedited training and superior performance compared to BPTT.
Overall, our method is embarrassingly simple we show that a careful analysis and modification of backpropagation lead to results exceeding the current state-of-the-art, without resorting to various approximations, a pool of models in the optimization, or additional heuristics. Since our approach does not depend on large-width approximations, it works for any architecture, in particular commonly used narrower models, for which methods that use inner-loop approximations perform less well. Moreover, our method can be seamlessly combined with prior methods on dataset re- parameterriztioonn (Deng & Russakovsky] 2022), leading to further improvements. To our knowledge, we are the first to introduce truncated backpropagation through time (Shaban al. (209)) to the dataset distillation setting, and to combine it with random positioning of the unrolling window.
Having established the strength of our method, we proceed to dissect the structure of the learned datasets to catalyze further progress. In particular we address the observation, already made in prior work (c.g. Nguuyn et al. (202Ib)) that distilled data seems to show a large degree of intercorrelation. When training on a subset of distilled data, for instance 10 images per class extracted from a 50-image per class distilled dataset we observe a large degradation in test accuracy: the resulting dataset
2

---

Published as a conference paper at ICLR 2024
performs much worse than if it were distilled from scratch; even worse than training dataset of random train images of the same size! This property makes distilled data less versatile since for each desired dataset size need to distill from scratch. To produce datasets that contain high performing subsamples, we propose Boost-DD, a boosting algorithm that produces a nested dataset without these higher-order correlations and only marginal performance loss. It works essentially every existing gradient-basee based distillation algorithm (see Figure for an illustration). To further our understanding of the learned dataset, we discuss the role of intercorrelation, as well as what information is captured in the distilled data through the lens of hardness metrics.
plug-and-play for
Overall, our contributions are follows:
RaT-BPTT algorithm: We propose RaT--PPT by integrating truncation and randomization with BPTT, and achieve state of the art performance various dataset distillation benchmarks. RaT-BPTT can be combined with data parametrization methods, leading to further improvements.
Boosting: We propse apppting-approach to dataset distillation (Boost-DD) that generates a modular synthetic dataset that contains nested figh-performance subsets for various budgets.
This paper is structured follows:
Section 2 surveys prior work, Section dediiineeeeee and motivates our algorithm. RaT-BPTT, Section presents experimental results and compares to prior art, and Section 5 details and evaluates our boosting algorithm. In Section 6 we summarize and discuss bottlenecks to further improvements.
BACKGROUND AND RELATED WORK
Dataset Distillation, introduced by Wang et al. (018)), aims to condense a given dataset into a small synthetic version. When neural networks trained on this distilled version, they achieve good performance on the original distribution. Dataset distillation shares many characteristics with coresel selection Jubran et al. (019), which finds representative samples from the training set to still accurately represent the full dataset on downstream tasks. However, since dataset distillation generates synthetic samples, is not limited the set of images and labels given by the dataset and has the benefit of using continuous gradient-baad based optimization techniques rather than combinatorial methods, providing added flexibility and performance. Both coresets and distilled datasets have found numerous applications including speeding up catastrophic forgetting SSnggeraaoo al.) (2022); Zhou al. (2022). federated learning Hu al.)
Mirzasoleiman et al. (2020). reducing
(2022): Song et al. (2022) and neural architecture search Such et al. (2020).
Numerous follow up works have proposed clever strategies to improve upon the original direct bilevel
optimization(see] Lei & Tao (2023); (Sachdeva & McAuley (2023): ; Yu et al. (2023); Geng al.) (2023) for recent surveys and Cui the core bilevel optimization problem, most works have focused on I) approximating the function in the inner-loop with more tractable expressions or 2) changing the outer-loop objective.
Cui et al. (2022) for benchmarking). Yet, given the apparent intractability of
Inner-loop surrogates: The first innovative works Nguyen et al. (202 b) tackle inner-loop intractabil- ity by approximating the inner network with the Neural Tangent Kernel (NTK) which describes the neural net in the infinite-widdi width limit with suitable initialization (Jacot al. (2018): [Le al. (2019); Arora et al.] (2019) and allows for convex optimization, but scales unfavorably. To alleviate the scaling, random feature approximations have been proposed: [oo al. (2022) leverage a Neural Network Gaussian process (NNGP) to replace the NTK, using MC sampling to approximate the averaged GP. Zhou al. (2022) propose use the Gram matrix of the feature extractor the kernel, equivalent to only training the last layer with MSE loss. A very recent work Loo et al. (2023) assumes that the inner optimization is convex by considering linearized training in the lazy regime and replaces the meta-grade meta-gradient with implicit gradients, thus achieving most recent state-of-the-art. Yet all of these approaches inevitably face the discrepancies between learning in the lazy regime and feature learning in adta-adeptive neural nets (c.g. Ghorbani et al. (2019) and numerous follow ups) and often need to maintain a large model pool. Moreover, inner-loop surrogates, be it NTK, NNGP or random features, tend to show higher performance on wide networks, where the approximation holds better, and be less effective for the narrower models used in practice.
Modified objective: A great number of interesting works try to replace the elusive test accuracy objective with metrics that match the networks trained on full data and on synthetic data. Zhao & Bilen 2021a) propose to match the gradient between the two networks with cosine similarity,
3

---

Published as a conference paper at ICLR 2024
69
T-8PTT RaTBPTT
100
Goddent Nrom
67
1
65-
BPTT
TEPTTT
63-
R-EPTT
RaTaPTT(ours)
100
200
Steps
300
400
500
10000
200D0
30000
40000
50000
60000
Steps
Figure 4: Test Accuracy during distillation with BPTT, T-BPTT, R-BPTT, and RaT- BPTT. Using random unrolling (R-BPTT) and truncated window (T-BPTT) both worse than BPTT. Combining them into RaT- BPTT gives the best performance. CIFAR1O, IPC10.
Figure 3: Meta-gradient norm in the first 500 steps. BPTT (unroll 120 steps) have unstable gra- dients. T-BPTT (unroll 120 steps and backprop- agate 40 steps) stabilizes the gradient. For RaT- BPTT, for each epoch (25 batch-update steps) we randomly place the 40-step backpropagation win- dow along the 120 unrolling. CIFARIO, IPC10.
with various variations (like differentiable data augmentation (Zhao & Bilen 2021b) (DSA)) and improvements (Jiang et al. 2022 Lee et al. 2022b). Other works pioneer feature alignment (Wang al., (202)), matching the training trajectories (MTT, introduced in Cazenavette et al. (2022) and refined in Du et al. (2023); Cui et al. (2023); Zhang et al. (2022b)), and loss-curvature matching (Shin et al. 2023). However, is unclear how well modified outer-loop metrics align with the test-loss objective and most of these methods ends up with subpar performance.
METHODS
In this section, start by defining the dataset distillation problem motivate our RaT-BPTT method. Denote the original training set as 'D and the distilled set as U. With an initialization 0g for the inner-loop learner A, we perform the optimization for steps obtain (2()) with loss C(0r (24), D). We add (4) to denote its dependence on U. The dataset distillation problem can be formulated as
min (1)) (outer loop) such that 0T(4)) = A(0o, U, (inner loop) "
The principal method for tackling bilevel optimization problems is backpropagation through time (BPTT) in reverse mode. When the inner-loop learner A is gradient descent with learning rate a, we obtain the meta-gradient with respect the distilled dataset by leveraging the chain rule:
GBPTT (L() D)
82 C ((0)())) 2 ((0,(1),44
1
(2)
00
002
a00u
The aforementioned computation reveals that the meta-gradient can be decomposed into parts. Each part essentially represents matrix product of the form I[ HH where every H matrix corresponds to Hessian matrix. Nonetheless, computing the meta-gradient demands the storage of all intermediate states backpropagate through every unrolling step. This imposes significant strain on GPU memory resources and diminishes computational efficiency.
To circumvent these challenges, the prevalent strategy truncated BPTT (T-BPTT) method (Williams & Peng [990; Puskorius & Feldkamp) [194),, which unrolls the inner loop for the same T' steps but only propagates backwards through a smaller window of M steps. In T-BPTT, the gradient is
GP-BPTT @C(B) D)
T-1
S L(0,((0),2))
O ((B( a00u
Σ İ-T-
1 1-a
00
002
The distinguishing feature of T-BPTT is its omission of the first T M terms in the summation; each omitted term is a product of more than M Hessian matrices. Under the assumption that the inner loss function is locally a e-stroggly convex, [Shaban al. (2019) shows that T-BPTT
4

---

Published as a conference paper at ICLR 2024
inherits convergence guarantees. The theoretical result comes from the diminishing contributions of the Hessian products. Strong convexity assumptions endow the Hessian matrices with positive eigenvalues. Consequently, 1 aH will have all cigenvalues smaller than 1, and the product term I1 aH] vanishes as the number of factors increases. Therefore, T-BPTT could enjoy a similar performance compared with BPTT but with less memory requirement and faster optimization time.
However, the inner task in our context diverges significantly from the realm of strong convexity. contains training a neural network from scratch on the current distilled data with random initialization. This problem is intrinsically non-convex with multiple local minima. This beckons the question: how do BPTT and T-BPTT fare empirically?
We visualize the training curve and the norm of meta-gradients through outer- loop optimization steps in Figure The experiment performed on CIFAR10 with IPC 10. A comparison between BPTT120 and T-BPTT reveals that: 1) The veta-gradieets of BPTT manifest significantly greater instability than their T-BPTT counterparts. This observed volatility and norm discrepancy can be attributed the omitted T - M gra- dient terms. It underscores the highly non-convex nature of the inner problem, characterized by Hessian matrices with negative eigenvalues. The compounded effects of these negative eigenvalues am- plifies the variance from different initial-
Algorithm 1 Dataset Distillation with RaT-BPTT. Differ- ences from BPTT are highlighted in purple.
Input: Target dataset DD T: total number of unrolling steps. M: truncated window size.
1: Initialize distilled data 24 from Gaussian
2: while Not converged do
3: Uniformly sample N in [M,T] as the current un-
rolling length
4: Sample batch of data d ~ 'D
5: Randomly initialize fg from p(0)
6:
for 0 N do
7:
If n- N M,,,,, start accumulating gradients
8:
Sample mini-batch of distilled data u, ~ U Update network pn+1 = 0n a V 0n)
9:
end for
11: Compute classification loss L = E(d, 0x)
Update UI with respect to L.
izations, creating the unstable gradient behavior. With the gradient stabilized, T- BPTT achieves faster improvement dur- ing the initial phase. 2) BPTT ends up with higher accuracy than T-BPTT. This indicates that important information from the initial phase is disregarded in T-BPTT notable concern given that the rapid optimization of neural networks usually happens during the early stage of the inner loop. The challenge thus is how harmonize the good generalization performance of BPTT with the computational speedup of T-BPTT.
13:
end while
To this end, we propose the Random Truncated BPTT (RaT-BPTT) in Algorithm 1 which randomly places the truncated window along the inner unrolling chain. The gradient of RaT-BPTT is
OL(0N(U),
N-1 Σ N=N-M
82 (C((,()) 8 (L)() 202
GRAT GRAT-BPTT BPTT -a
N-1
00
000u
RaT-BPTT varies from BPTT by randomly selecting M consecutive segments from GAPTT and excluding shared Hessian matrix products. Essentially, RaT-BPTT is a subsample of BPTT, covering the entire learning trajectory, but limits the number of Hessians in the product to less than M. This approach combines T-BPTT's accelerated performance and gradient stabilization. As shown in Figure 4 RaT-BPTT consistently surpasses other methods in the optimization process. Contrastingly, R-BPTT, involving full unrolling over randomly sampled trajectory lengths, yields unstable gradients and inferior performance compared to full BPTT unrolling. Section 4.3 includes an ablation study on the importance of a moving truncated window and the rationale behind random uniform sampling in Algorithm|/ [ We investigate the stability of meta mneaa-grddentts using gradient norms as a metric, predicated on the notion that stable and efficient learning should manifest consistent and decreasing gradient norms throughout training. We further provide instability analysis in Appendix [D3] with another metric, the normalized variance. In light of the significant change of the network within the inner loop, concurrent work Chen et al. 2024) introduces a progressive concept that bears resemblance our random design approach.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTAL RESULTS
In this section, we present an evaluation of our method, RaT-BPTT, comparing it to range of SOTA methods across multiple benchmark datasets.
Datasets We run experiments on four standard datasets, CIFAR-10 (10 classes, 32 x 32). CIFAR-100 (100 classes, 32 x 32. Krizhevsky et al. (2009)), Caltech Birds 2011 1 (200 classes, CUB200, 32 x 32, (200 classes, 64 x 64, & Yang (0155) ), ImageNet- IK (1,000 classes, 64 x 64, Deng et al. ).09).. o. We distill datasets with 1, 10, and 50 images per class for the first two datasets, with and 10 images per class for the last two datasets, and and images per
Wah et al. (01TH
class for the big ImageNet-1K K dataset.
Baselines We compare our methods the first two lines of works we discussed in related work (Section 2) including 1) inner-loop surrogates: standard BPTT (the non-factorized version of LinBa in (Deng Russakovsky 2022 ). Neural Tangent Kernel (KIP) (Nguyen et al. (022I). Random Gaussian Process (RFAD) (Loo et al. (022). and empirical feature kernel (FRePO) (Zhou et al.
2022). and reparameterized convex implici gradient (RCIG) (Loo et al. 2023). 2) Modified objectives:
gradient matching with augmentation (DSA) (Zhao & [ distribution matching (DM)
(Zhao Bilen 2023), trajectory matching (MTT) (Cazenavette et al.] (202), a memory- friendll
Version of MTT (TESLA) (Cui et al. (023) and flat trajectory distillation (FTD) (Cui et al. 2023). ).
The works on parametrization (Deng & Russakovsky. 2022] Liu et al. 2022 Kim et al. 2022) are
complementary to optimization framework and can be combined with RaT-BPP for improved
performance, as we illustrate for the SOTA case of linear basis (Deng & Russakovsky 2022) in
Section 4.2
Setup Building upon existing literature, we employ standard ConvNet architectures (Zhao & Bilen
2021b: Deng & Russakovsky 2022 Cazenavette al., 2022) -three layers for 32 x 32 images and
four layers for 64 x 64 images. Our distilled data is trained utilizing Algorithm 1, with the Higher
package (Grefenstette al. 2019 aiding in the efficient calculation of meta-gradients. We opt for simple setup: using Adam for inner optimization with a learning rate of 0.001, and applying standard augmentations (flip and rotation) on the target set. Parameters such as unrolling length and window size are determined via a validation set.
Evaluation During the evaluation phase, we adhere to the standard augmentation protocol as per Deng & Russakovsky (2022): Zhao & Bilen (202Tb). We conduct evaluations of each distilled data set using ten randomly selected neural networks, reporting both the mean and standard deviation of the results. For all other baseline methodologies, we record the best value reported in the original paper. Note that Zhhou et al. (2022); [oo et al.] (2023) employs a 4 or times wider ConvNet to reduce discrepancies between surrogate approximations and actual training. To ensure aligmment with this protocol, we provide a transfer evaluation of our method, that is we distill with a narrow network and evaluate with wide network. We also re-evaluate their checkpoints for narrow networks. Complete details can be found in the Appendix. We release our code at https: htttps://gttub.coom thw.. www.o////egggpuu///cccooco wttppppw////mple..ee./e Dataset Dist illation
Table 1: Performance of different dataset distillation techniques on standard datasets. The AVG column denotes the average performance across the other columns. denotes works where performance evaluated with wider ConvNets. FRePO and RCIG the re-evaluated results with narrow networks. Results denotes the best results for narrow networks while results denotes hest for wide networks.
Dutset Ins
CIFAR-10
CIFAR-100 10
CUB220
T- T-lmgeNi
ImageNes IK
10
50
50
10
10
BPTT
Dane & Rasilmnl
2022
40.3ons
62.4
70.5ema
21.3aea
347405
KBP"
62.7 203
157rra
2831.1
Inner
RFAI
66 Lon
36 1am
Loop
FRePO
23100 c
46.8
63.5
71.
28.7
12.4.02
16.81c1
15.4
7.5k3
9.712
FRePO
45.6181
63.5.00
16.9
13.3
12
RC3
66.800
DSA
Phao Riles 10
38.8e9
$2.1am
60.6
13.0
32.3
42.8am
1.3
4.5
14.4
Modided
DM
Parior Vilin AUp4
26.013
43.9ad
53.0184
11.41C3
29.7113
43.024
16111
4,4.02
3.9.02
12.114
15:M1
17:M
Objectives
MT
Sove 1 2021
46.31s
60.3m0
T16118
243363
40.11-
4T.3n3
22:31
8,5003
TE3
11 m1063
48.5aax
66.4ma
72.6ee:
245
41.7as
47.0um
4.7
10.5
M11>
888535
25.2ar2
434115
50020
10/4
Ouex
33.2
36.3
47.5
30,5aa
13,8
12.7
22.
10.7aam
12
Uur Oranster to wide)
71,0003
41
36.9003
47M983
51,000
1422.03
179
22.5
13.04.8
Dunsse
K3,5
55.3183
20.
37.6002
33,8

---

Published as a conference paper at ICLR 2024
4.1 BENCHMARK PERFORMANCE
Our simple approach to dataset distillation demonstrates competitive performance across a number of datasets (Table T). With 10 and 50 images per class, our approach gets the state-of-the-art (SOTA) results on the CIFAR- 100, CIFAR-I0, CUB200, and ImageNet- K datasets (Table [ Tlast last two rows). Moreover, we achieve these results without any approximations the inner loop. When considering all IPC values in {1, 10, 50), across all datasets, our approach performs as well as the RCIG method up to statistical significance. Encouragingly, even though our bilevel optimization towards wider networks, we obtain state-of-the- ahrt performance even for wide networks on CIFAR 10, CIFAR 100, CUB200, and ImageNet-Ile IK across all IPC values. Moreover, when the datasets from the wider-network approaches are evaluated on practical, narrower settings we find that they show significant drop in performance, going from 39.3% to 35.5% (RCIG, CIFAR 100, IPC1) or from 25. 25.4% to 22.4% (FrePO, Tiny ImageNet IPC10). Thus, work generalizes gracefully to wider networks that are used by previous work (improving in performance) as well as narrower networks (for which can tune directly). This is significant advantage of our work over prior state-of-the-art.
not biased
4.2 COMBINATION WITH PARAMETRIZATION METHODS
A separate and complimentary line of work aims to improve the optimization via parameterization of the distilled dataset. [iu et (2022): Wang al. (2023) leverage encoder-decoders, Lee et al.)
(2022a): Cazenavette et al. (2023) use generative priors, Kim et al. (2022); Liu et al. (2023b) propose
multi-scale augmentation, and Deng & Russakovsky (2022) designs linear basis for the dataset.
Note that our performance improve- ments come from careful study of the bilevel optimization problem. In principle, RaT-BPTT is complimen- tary to most of these parameteriza- tion idcas and can scamlessly inter- adopt the lin-
Table 2: Combination of RaT-BPTT with linear parameteri- zation leads to further improvement. We only present those settings where parameterization outperforms the standard RaT-BPPT.
Dataset Img/class(IPC)
CIFAR-10
grated. For instance, ear basis from Deng & Russakovsky (2022) within study the case of CIFAR 10, as for all the other benchmarks RaT-BPP gives better performance even without data parametrization. Without any hyper- parameter tuning, we can improve their performance by around 1.6%, leading to astonishing numbers of 68.2% for IPC1 and 72.8% for IPC10 (the numbers w/o parameterization are 53.2% and 69.4% respectively). The results are shown in Table 2
Para-
IDC (Kim et al. 2022
50.0.u.
67.5.u
framework. We only
meteri- LinBa (Deng & Kussakovsky
2022
66 41.4
71.2:0.
zation
HaBa (Liu et al. 2022
48.3
69.910.4
Linear RaT-BPTT
68.2+a4
72.8+0.
4.3 ABLATIONS ON THE RANDOM TRUNCATED WINDOW
In Section we justify the necessity of performing truncation speed up and stabilize the gradient, and the necessity of changing the truncated window to cover the entire trajectory. We present an ablation study on selecting the truncated window, comparing random uniform truncation with backward and for- ward moving methods. The latter two involve initial window placement at the trajectory's start end and shifting it when the loss stagnates.
Accursa
Randem
Backward
Forwand
10000
20000
30000
40000
56000
60000
Steps
Figure reveals that random uniform truncation out- performs other methods throughout training. A closer examination of the forward and backward moving curves suggests that altering the window's position- ing can spur noticeable enhancements in accuracy. This suggests that different truncation windows cap- ture unique aspects of knowledge, emphasizing the need for comprehensive trajectory coverage. Actually, uniform sampling isn't the optimal design. By
Figure 5: Comparison between random uni- form truncation, backward moving, and for- ward moving. Random uniform truncation gives the best performance across the whole training process. N=120, T-40 for IPC10 with CIFARIO.
7

---

Published as a conference paper at ICLR 2024
focusing more the initial phase, accuracy on CIFARIO with IPC10 improves by 0.4%, though it requires line-tuning an extra parameerr For simplicity, we opt for the uniform approach.
INTERCORRELATIONS AND BOOSTING
Having established the strength of our method, we proceed to dissect the structure of the learned datasets to catalyze further progress. Nearly all the current distillation method optimize the data jointly. Such joint optimization often lead to unexpected behaviors which are largely absent in the original dataset. For instance. Nguyen al. (2021b) observes that the intrinsic dimension of distilled data increases, compared to the training data. The data is jointly learned, and therefore can be correlated. A particularly noticeable consequence of the correlation a significant drop in performance when using a subset of the distilled data for downstream tasks, as shown in Figure We see that subsampling 5 images per class from an IPC50 distilled dataset not only gives performance way below an IPC5 dataset learned from scratch, but performs even worse than training on 5 random training images per class.
To address these challenges, we propose boosted dataset distillation (Boost-DD) (in Algorithm 2 Boost-DD controls inter-correlation within distilled data and can be integrated with any gradient-based dataset distillation algorithm. The central idea is construct the distilled data iteratively with smaller data groups called "blocks" illustrated Figure For IPC50, we divide all images into blocks of IPC5. We start from a distilled IPC5. Each time, we add an- other fresh block of IPC5, and optimize the new block with reduced learning rate on the existing blocks by a factor of B < 1. The extreme case where the learning rate zero for previous blocks is termed strongly- boost (3 = 0). The advantage of this approach is that initial blocks remain unchanged when the new block is added, ensuring consistent performance. This re- sults a "nested" dataset where subsampling earlier blocks yields high-performin subsets! We call the algorithm weakly-booss
Algorithm Boosted Dataset Distillation (Boost-DD)
Input: Target dataset D. Distillation- Algorithm A with initiatlization procedure I (ssize and meta-learning rate a, outputring distilled data U = A(D, I(size) (M) size). Block size b. Number of blocks J. noosting-strengtue 8 E 1].
Output: Distilled data 24 with [u| b J. 1: Distill first block of size b: Uo =
2: for j= 1... do
Distill increased data Ug A(D, U I(b)) using "stale" meta-leaming rate a, B a the first
(j (--1) data points and a on the last b. end for 5: u==
is non-zero and perform experiments with 3 0.1. When performing distilla- tion for each fix IPC setting, we follow the same configurations as our RaT-BPTT.
Figure 7 shows how weakly-boost and strongly-boost compare distilling each of the sub-datasets from scratch. It is important to highlight that the curve for "strongly-boos" represents the accuracy curve obtainable when subsampling across different budgets. We observe that even strongly- broost
74
60
Accuracy
72
70
IPC50 Subsample
40
IPC25 Subsample
Il Strongly Boest
IPC10 Subsampla
Random
66
Weakly Boost
20
20
30
40
50
IPC number
IPC number
Figure 6: Test performance of random subsam- ples of larger distilled datasets, compared to per- formance of real data of the same size. We aver- age over 10 random samples. Setting: CIFAR10, RaT-BPTT.
Figure 7: Performance of fully distilled, strongly and weakly boosted distilled data. Boosting es- sentially retains performance compared to jointly distilled data. CIFARIO, RaT-BPTT.
8

---

Published as a conference paper at ICLR 2024
results in exceedingly minor sacrifice in performance compared to joint distillation from scratch, especially with larger distilled datasets. In CoppendixD.9]wwe present a visual comparison of images distilled jointly with images distilled with boosting, Boosted images seem to show larger diversity and seem closer to real images. In Appendix D.8. we further combine our framework with other distillation methods like the MTT.
DISCUSSION
In this work, we proposed simple yet effective method for dataset distillation, based on random trun- cated backpropagation through time. Through a careful analysis of BPTT, we show that randomizing the window allows to cover long dependencies in the inner problem while truncation addressed the unstable gradient and the computational burden. Our method achieves state of the art performance across multiple standard benchmarks, across both narrow as well as wide networks. We note that the utility of the RaT-BPTT approach has the potential to extend to a wider variety of bilevel optimization problems, for instance meta-learning of hyperparameters (Andrychowicz al., 2016), MAML--lik problems (Finn et al. (0177)) for good few-shot adaptation to new tasks, and Poisoning Attacks (Muñoz González et al. 2017) for crafting training-data to corrupt a system's test behavior. All these problems have bilevel optimization its core which, in general, requires optimization through unrolling and is thus amenable to the RaT-BPP approach.
Nonetheless, several design choices are guided by intuitions and observations, leaving room for improvement. For instance, we could envision adopting methods from Maclaurin et al. (2013) to improve efficiency of meta-gradient computation or from (Sachdeva al. (2023) for algorithmic design for Adam with constant memory consumption. Integrating such methods into our framework presents an exciting avenue for developing scalable algorithm. This integration could potentially enhance the computational efficiency and memory management of our method, an aspect we are keen explore in future work. We defer detailed limitation discussion to Appendix
Further, we address the catastrophic degradation in performance of subsets of distilled data with our boosting method. It allows us create single versatile dataset for various distillation budgets with minimal performance degradation. However, the boosted dataset still has inner correlation between blocks. This is evident Figure X when comparing the performance of the first IPC5 block with the second one obtained via strongly-boost (though both of them are much higher than sampling random IPC5 from the jointly distilled IPC50). Moreover, as shown in Figure 7 weakly-booss for larger IPC eventually outperforms joint training. Since weakly-boost generates less inter-correlated datasets, this hints at the possibility that strong intercorrelations are one reason for diminishing returns observed when increasing the size of distilled datasets. While higher-order correlations may potentially encode more information, they also compromise data interpretability. diverging from the standard IID paradigm. Is it possible further minimize these correlations, especially in larger IPC datasets? We leave these questions to future research.
We also attempt understand other factors bottlenecking the gains when scaling up distilled data. Specifically, we try to understand what information is learnt in the distilled data by dissecting the accuracy on evaluation samples. We leverage a hardness score that characterizes whether data is easy or hard, stratify the accuracy by hardness and compare it for the original network and the network trained on distilled data for range of IPC in Figure (details in Appendix EE. One would have hoped that adding more distilled data would help to distill more of the hardness tail, but this is not the case. This suggests that future work might benefit from focusing on how one can distill data that better adapted to larger hardness scores, for instance by infusing validation batches with harder data, placing emphasis on the middle of the distribution. A preliminary promising study is presented in Appendix B

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This work was supported by the National Science Foundation under NSF Award 1922658. This work was also supported in part through the NYU High Performance Computing resources, services, and staff expertise. The authors extend their gratitude to all reviewers for their insightful suggestions. YF would like thank Di He, Weicheng Zhu, Kangning Liu, and Boyang Yu for discussions and suggestions.
REFERENCES
Marein Andrychowiez, Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford. and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In D. Lee, M. Sugiyama, Luxburg. I. Guyon, and Garnett (eds.). Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buee Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8139-8148, 2019.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 4750-4759 2022.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. General- izing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Xuxi Chen, Yu Yang. Zhangyang Wang, and Baharan Mirzasoleiman, Data distillation can be like vodka: Distilling more times for better quality. In The Twelfth International Conference Learning Representations, 2024. URL http:///opeeeiieeeeeeeeeeee net/ forum did-:mmmmiiiiiim NHgnkg0z2
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. De-bench: Dataset condensation benchmark. Advances in Neural Information Processing Systems, 35:810-822, 2022.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pp. 6565-6590 PMLR, 2023.
Stephan Dempe. Bilevel optimization: theory, algorithms, applications and bibliography. Bilevel Optimization: Advances and Next Challenges, pp. 581 581-672, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Proceedings of the Advances Neural Information Processing Systems (NeurIPS), 2022.
Jiawei Du, Yidi Jiang, Vincent T. F. Tan, Joey Tianyi Zhou, and Haizhou Li. Minimizing the accumulated trajectory error to improve dataset distillation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Fartash Faghri, David Duvenaud, David Fleet, and Jimmy Ba. A study of gradient variance in deep learning. arXiv preprint arXiv:2007.044. 043343 2020.
Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong. Transferred discrepancy: Quantify- ing the difference between representations. arXiv preprint arXiv:2007. /2446, 2020.
10

---

Published as a conference paper at ICLR 2024
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learing for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 1126-1135. JMLR.org, 2017a.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 06-11 Aug 2017b.
Zongxion Geng, Jiahui andg Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAT), 2023.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32.
Curran Associates, Inc., 2019.
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala. Generalized inner loop meta-learning. arXiv preprint arXiv: 2019.
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. Fedsynth:
Gradient compression via synthetic data in federated learning. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurlPS 2022), 2022. |
//openreview.
forummrrmmmm id= 2--rmmm2c--LL3Vh044eE 1k8VhkQ4eE3
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018
Zixuan Jiang, Jiaqi Gu, Mingjie Liu, and David Z. Pan. Delving into effective gradient matching for dataset condensation. arXiv preprint 2rXhv:2208.00311, 2022.
Ibrahim Jubran, Alaa Maalouf, and Dan Feldman. Introduction coresets: Accurate coresets. ArXiv, abs/1910.08707, 2019.
Jang-Hyun Kim, Jinuk Kim, Scong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song, Dataset condensation via efficient synthetic-data parameterization. In Proceedings of the International Conference on Machine Learning (ICML), pp. 11108-1118,, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.0011 00719, 2022a.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve Linear Models Under Gradient Descent. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensa- tion with contrastive signals. In Proceedings of the International Conference Machine Learning (ICML), pp. 12352-12364, 2022b.
Shiye Lei and Dacheng Tao. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301. 0056003 2023.
11

---

Published as a conference paper at ICLR 2024
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3759-3768, 2023a.
Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. DREAM: Efficient dataset distillation by representative matching, arXiv preprint (tt.:3302.14166, 2023b.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, and Daniela Rus. Dataset distillation with convexified implicit gradients. arXiv preprint arXiv:2302.06755, 2023.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1540-1552. PMLR, 26-28 Aug 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- -based hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 2113-2122. PMLR, 2015.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6950 6960. PMLR, 13-18 Jul 2020.
Luis Muñoz González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee. Emil C. Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back teek-gradeent optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec 17, pp. 27-38, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450352024.
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression. In Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). pp. 5186-5198, 2021b.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL net. Fortt//ormmmme (oormmmid=0:pp----22
https:://ppenrev//
GV Puskorius and LA Feldkamp. Truncated backpropagation through time and kalman filter training for neurocontrol. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume pp. 2488-2493. IEBE, 1994.
Noveen Sachdeva and Julian McAuley. Data distillation: A survey. arXiv preprint a4X1422301.04272, 2023.
Noveen Sachdeva. Zexue Hc, Wang-Cheng Kang, Jianmo Ni, Derek Zhiyuan Cheng, and Julian McAuley. Farzi data: Autoregressive data distillation, 2023.
Mattia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. 2022 International Joint Conference on Neural Networks (IJCNN), pp. 01-08, 2022.
12

---

Published as a conference paper at ICLR 2024
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagatton for bilevel optimization. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Sen Swenty-Second International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1723-1732. PMLR, 16-18 Apr 2019.
Seungjae Shin, Heesun Bae, Donghyeok Shin, Weonyoung Joo, and Il-Chul Moon. matching for dataset selection and condensation. In Proceedings of the International Conference on Artificial Intelligence and Statisties (AISTATS), 2023.
Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll. Federated learning via decentralized dataset distillation in resource-constrained edge environments, 2022.
Felipe Petroski
Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pp. 9206-9216. PMLR, 13-18 Jul 2020.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In Proceedings of the International Joint Conference Neural Networks (IJCNN), pp. 1-8, 2021.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. https:t:::/ /ttttps://openreviee. net/ feet/forue 3oo
Nikolaos Tsilivis and Julia Kempe. What the neural tangent kernel tell about adversarial robustness? In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2022.
Nikolaos Tsilivis, Jingtong Su, and Julia Kempe. Can we achieve robustness from data alone? arXiv e-prints, pp. arXiv-2207 2022.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. CAFE: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12196-12205, 2022.
Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang. and Yang You. Dim: Distilling dataset into generative model. arXiv preprint arXiv:2303.04777, 2023.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10995 2018
Ronald Williams and Jing Peng. An efficient gradient-based algorithm for on-line training of recurrent network trajectories. Neural computation, 2(4):490-501, 1990.
Ruonan Yu, Songhua Liu, and Xinchao Wang. A comprehensive survey to dataset distillation. arXiv preprint arXiv:2301.0000 2023.
Chia-Hung Yuan and Shan-Hung Wu. Neural tangent generalization attacks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 12230-12240. PMLR, 18-24 Jul 2021.
Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu. Dense: Pata-free one-shot federated learning. Advances in Neural Information Processing Systems, 25:21414-211428, 2022a.
13

---

Published as a conference paper at ICLR 2024
Lei Zhang, Jie Zhang, Bowen Lei. Subhabrata Mukherjee, Xiang Pan, Bo Zhao, Caiwen Ding, Yao Li, and Dongkuan Xu. Accelerating dataset distillation via model augmentation. arXiv preprint arXiv:2212.06152, 2022b.
Bo Zhao and Hakan Bilen. Dataset condensation with gradient matching. In Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 12674-12685, 2021b.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature re- gression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
14

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
Algorithm Design The design of our method is primarily guided by intuitions and observations from empirical studies. Throughout the algorithm's development, aim to strike balance between scalability and effectiveness. Our approach currently involves tuning the ratio between the unrolling length and window size, scaling the unrolling length in accordance with the IPC number and GPU size. While this approach has demonstrated promise, we acknowledge that the current algorithmic choice might not represent the absolute optimal solution. Further research could investigate alternative algorithm designs.
Application to larger models and datasets A notable strength of our methodology is its versatility: it is compatible with all differentiable loss functions and network architectures, emphasizing its broad applicability. However, we only focus on illustrating the method's capabilities with standard benchmarks in the literature. This decision leaves promising avenue for future work to apply and validate our method across various domains and tasks beyond image classification. It's also worth highlighting that while surrogate-based techniques are constrained to using the MSE loss to convexify the inner problem, our approach agnostic to the specific loss function employed. This flexibility paves the way for our method's application in other realms, such as audio and text data distillation.
GPU memory usage Despite the significant improvements introduced by RaT-BPTT, it still ne- cessitates unrolling and backpropagating over several steps, which require storing all intermediate parameters in the GPU. Consequently, this method incurs substantial memory consumption, often exceeding that of directly training the model. For larger models, one might need to implement checkpointing techniques to manage memory usage effectively.
B FIGURES FOR DISCUSSION
IPCS analysis
70
Correitiio
Boosting
Insers
PC25
corre ardne
60
0.5
40
0.4
20
Hertiness
Real IPCS
Bandom from b-Pecaa
Dostiled (iC)
Randon from DJPC2S
Rendon from D-PC10
Brst from Street
Secresd from Stond
Heat from Wook
Seecod from Wook
Rasdimm from Slong
fundom from Wook
Random from D-incio
Histogram the Hardness Score
100
Figure 8: Performance of subsamples compared to fully distilled and random real images for IPC5: y-axis shows test accuracy a) Random IPC5 from IPC50 and IPC25 performs worse than random real IPC5, indicating the strong inter-correlation learned in the dataset. b) IPC5 building blocks of boosting perform quite well. c) Random IPC5 from boosted IPC10 performs better than random IPC5 from standard IPC10. Boosting somewhat lowers the inter-correlation.
Figure 9: Top: Hardness score accuracy of dataset distillation (for various images per class (IPC)). "Optimal" indicates accuracy when train- ing on the entire real data. Bottom: Histogram of the hardness scores. Notice how score 0 (un- forgettable) examples are distilled well (top), but harder examples are progressively harder to dis- till. Details in Appendix L
C OTHER RELATED WORK
In this section, we discuss further works related to dataset distillation hardness metrics.
15

---

Published as a conference paper at ICLR 2024
Boosting: It is noteworthy that Liu et al. (2023a) has also identified challenges associated with retraining distilled datasets for varying budgets. Their proposed solution adopts a top-down approach, aiming to slim large distilled dataset. In contrast, our method follows a bottom-up strategy, producing a modular dataset designed to accommodate various budgets. Moreover, one of our motivations is address and study the intercorrelation problem.
Hardness metrics: One way to study generalization performance of neural nets is to understand which data points are "easy" "hard" to learn for neural network. There is an intimate relationship to data pruning which tries to understand and quantify which subsets of the data can be pruned with impunity, while maintaining the performance of the neural net when trained on the remainder") Inspired by the phenomenon of catastrophic forgetting, [Toneva et al. (2019) are the first to study how the learning process of different examples in the dataset varies. whether some examples are harder to learn than others (examples that are forgotten and relearned multiple times through learning.) and define a forgetting score to quantify this process. To our knowledge, our work is the first to use this tool to understand how learning on distilled data differs from learning on full data, to identify bottlenecks. The idea to "enrich" the validation data during the data distillation process appears in Liu et al. (2023b), who chose more "representative" data to learn from, as determined by k-means clustering. To our knowledge, we are the first to propose learning from "harder-to-leam"" data towards more efficient data distillation.
particular, the authors analyze
Extensions of Dataset Distillation: Beyond the conventional dataset distillation formula that aims to minimize generalization error, there have been advances in optimizing metrics for various objectives. These include dataset generation tailored for generalization attacks Yuan & Wu (2021), adversarial perturbations Tsilivis & Kempe (2022), and generating distilled data with an emphasis on robustness
Tsilivis et al. (2022).
D EXPERIMENTS
D.1 EXPERIMENTAL DETAILS
Data Preprocessing: Leveraging a regularized ZCA transformation with regularization strength of X = 0.1 across all datasets except only the ImageNet-1K: our approach adheres the methods established by prior studies Nguyen et al. ():la(b):(har et al. (2022); Loo et al. (2022); Deng & Russakovsky (0022). We apply the inverse ZCA transformation matrix for distillation visualization, using the mean and standard deviation to reverse-normalize optimized data.
Models Following previous works, we use Instance Normalization for all networks for both training and evaluation.
Initialization In contrast to conventional real initialization widely used nearly all previous works, we employ random initialization for distilled data, hypothesizing that there is a reduction in bias from such uninformative initialization. Data are initialized via a Gaussian distribution and normalized to norm 1. For RaT-BPTT, note comparable performance and convergence between random and real initialization.
Label Learning Following previous works that leverage learnable labels, we optimize both the data and label for CIFARIO IPC10 and IPC50, all IPCs for CIFAR100 100, CUB-200, ImageNet-IK. We forego normalization for label probability, hence the labels retain their positive real value representation.
and
Training In addition to the RaT-BPTT algorithm, we incorporate meta-gradient clipping with an exponential moving average to counter gradient explosion. We find that the proper combination of normalizing initialization and learning rate (0.001 for Adam) is crucial for successful distillation image training. While using instance normalization, an image scaled by a leads to meta-gradient scaling by As result, one should use an times larger learning rate for Adam or a² times larger for SGD to achieve the same optimization trajectory. We thus adopt a similar initialization scale to that of neural network training (normalized norm 1), combined with standard learning
'The boundary between coresets and data praning is fluid: the former term is used for small subsets of the training set, while the latter usually refers to removing only fraction of the training data, like 25% Paul et al.
202T).
16

---

Published as a conference paper at ICLR 2024
rate of 0.001 when using Adam. To maintain meta-gradien stability, employ batch sizes of 5,000 for CIFAR- 10 and CIFAR-100, 3,000 for CUB-200, 1,000 for Tiny-ImageNet, and 1,500 for ImageNet- IK. Note that one should aim to further increase the batch size for Imy-ImageNet until all the GPU memory is used. For sampling from the distilled data, a batch size of 100 or 200 samples is adopted. This translates to 10 samples per class in CIFARIO (10 classes), 1 sample per class in CIFAR 100 (100 classes, except for IPC 50 where it's 2 samples per class), and sample per class in CUB200 (200 classes). For TinyImageNet and ImageNet-IK, due to GPU limitations, we use sample per class from 50 randomly selected classes in each batch. Ideally, one should further increase the number of classes for these large scale dataset.
Hyperparameters In an effort to minimize tuning requirements, we adhere to standard baseline across all configurations. Specifically, we utilize the Adam optimizer for both the inner loop (network unrolling) and the outer loop (distilled dataset optimization) with learning rates uniformly set to 0.001 for CIFAR-10, 10, CIFAR- and CUB-200 and to 0.0003 for Tiny-ImageN, and ImageNet- I.. We refrain from applying weight decay or learning rate schedules that are used in prior works Zhou et al. (2022) Loo et al. (2023).
Evaluation We evaluate our optimized data using seperate held-out test dataset (the test set in the corresponding dataset). We adopt the same data augmentation as in previous work Deng & Russakovsky (2022). We train standard convolutional networks using the Adam optimizer with learning rate of 0.001. For wide convolutional networks, we employ the same optimizer but adjust the learning rate 0.0003. No learning rate schedule is used. During the evaluation phase, maintain the same batch size for the distilled data as used in the training phase.
D.2 ABLATIONS ON CURRICULUM
unroll 80
window 40
unroll
window 60
68.0
unrall 200
window 100
67.5
Accuracy
67.0
67.0
66.5
66.5
66.0
66.0
65.5
65.5
65.0
10000
15000
20000
65.0
5000
10000
15000
20000 25000 30000 35000 Steps
Steps
Figure 10: Left Test accuracy during distillation for different unrolling length of 80, 120, 200, 300 with fixed window size 40, CIFAR- 10, IPC10. Right Test accuracy during distillation for different window size in 40, 60, 100 with fixed unrolling length 200.
Our RaT-BPTT implementation hinges on tuning two hyperparameters: unrolling length and back- propagation window size. This section presents an ablation study exploring these parameters for 10, IPC10.
Unrolling length
We initially fix the window size at 40 while varying the unrolling length. Notably, unrolling length governs the long-term dependencies we can capture within the inner loop. Figure [ reveelss that a moderate unrolling length, between twice and five times the window size, yields similar performance. However, disproportionate unrolling, as seen with window size of 40 and unrolling length of 300, detrimentally affects performance.
Window size
Next, we fix the unrolling length at 200 and experiment with window sizes of 40, 60, and 100. Figure 10 shows the latter two sizes yield comparable performance. In RaT-BPTT, GPU memory consumption is directly proportional to the window size, thus a window size of 60, with unrolling length of 200 steps, emerges as an optimal balance. As such, we typically maintain a window size to unrolling length ratio of around 1:3.
17

---

Published as a conference paper at ICLR 2024
In our implementation, employ window size and unrolling length of (60, 200) for CIFAR-10 IPC1 and CUB-200, (80, 250) for CIFAR-10 IPC10, and (100, 300) for all other datasets.
D.3 OTHER METRICS ON GRADIENT STABILITY
In Figure 3J we investigated the stability of meta geett-ggaseents using gradient norms as metric, predi- cated on the notion that stable and efficient learning should manifest consistent and decreasing gradient norms throughout training. Expanding on this analysis, we now introduce another metric evaluating gradient stability:
normalized gradient variance, in line with the methodology proposed by Faghri et al. (2020). Each variance value reflects the instability across the batch samples, and the values across time steps reflects the instability across training steps.
To calculate this metric, compute the average variance of all gradient entries using a set of 100 samples from the evaluation batch. Given the different scales gradient norms across different methods, normalize this variance against the square of the norm. This normalization yields more consistent metric, termed the normalized variance. Employing the same experimental setup as in Figure 3 present the results in Figure 11 It shows that RaT-BPTT not only maintains lower variance at each training step but also demonstrates more consistent variance trajectories over the course of training. These findings, in conjunction with the earlier results from Figure collectively offer a comprehensive view of the argued training instability.
D.4 ABLATION STABILIZING BPTT
In Figure 3 we have demonstrated the notable instability of meta-gradients via the gradient norm. This section extends our analysis with ablation studies, indicating that both controlling the gradient norm and incrementally increasing the unrolling parameter T of BPTT result in only marginal improvements, which cannot compare to the gains garnered through RaT-BPP. We follow the setting in Figure
Our foundational approach has already incorporated gradient clipping to manage extreme gradient norm values, employing standard exponential moving average (EMA) with 0.9 decay rate and capping the gradient norm at twice the adaptive norm.
To further stabilize the gradient norm, explored two additional methods: 1) BPTT-Gradient Clipping. limiting the gradient norm to no more than 1.05 times the adaptive norm, and 2) BPTT- Normalized Gradient, ensuring a consistent gradient norm of throughout training. However, as Figure 12 illustrates, these methods achieve only marginal enhancements over the basic BPTT approach. Their performance trails behind RaT-BPTT, with threefold increase in optimization time due to extended backpropagation.
These findings highlight challenges such as deviation from the kernel regime, variance from ex- tensive unrolling, and pronounced non-co non-convexity, contributing to gradient instability, as evidenced by fluctuating gradient norms. Addressing these issues solely by adjusting gradient norms proves insufficient.
T-PPTT R-BPTT
Norralzzd Verfance
102
67-
101
65
RaTBPIT(ours)
Clipping
100
BPTT-Nomme Nomalizzd Gradient
increanngg Window
100
200
300
400
Steps
10000
Staps
Figure 11: Normalized variance across batch samples the meta-gradient. RaT-BPTT has stable and small variances. Same setting as in Figure
Figure 12: Ablations stabilizing BPTT: only controlling the gradient norm or gradually in- creasing the window is not enough.
18

---

Published as a conference paper at ICLR 2024
Alternatively, examine limiting the maximum Hessian matrices in Equation by gradually extending the unrolling length T in BPTT. In Figure [2 the BPTT-Increasing Windows variant, which linearly scales T from 10 to 180, underperforms both R-BPTT and standard BPTT. This underlines the complexity within the inner loop, deviating significantly from the kernel regime and emphasizing the importance of managing the unrolling window size.
D.5 OTHER ARCHITECTURES
Table 3:
Generalization to other architectures. We con-
Boosting
duct transfer evaluation of a distilled dataset trained with a 3-layer ConvNet and directly training the dataset with the architecture the inner loop. CIPARIO, IPC10.
Accuracy
68
66
Distill All Strongly Boost
Architecture
VGG-11
AlexNet
ResNet-18
10
20
30
40
50
Transfer
46.640.
60. 60.1
49.2
IFC number
Direct Training
47.7108
63.71006
53.0
Figure 13: Boosting with MTT
We further assessed our method across various architectures to demonstrate its universality. noteworthy that our approach is already effective across different widths of the convolutional networks (narrow and wide) we used. Additionally, we conducted tests using the standard VGG-11, 11, AlexNet, and ResNet-18,, both training from scratch and transferring from the distilled dataset. To our knowledge, we are the pioneers in applying direct distillation to a standard-sizer network like ResNet- 18 and VGG-11. Prior works never train directly on VGG11 and they only use small or modified ResNets like ResNet-10 10 (Zhang et al. 2022a Kim et al. (2022) ResNet- 12 (Deng & Russakovsky 2022) and ResNet-APPO (Kim et al. 2022 Liu et al. 2023b) in these settings.
The results are presented in Table 3 Our results yield better comparable transfer results compared
with previous methods. Direct training further increases the numbers.
D.6 ABLATION ON THE INNER OPTIMIZER
We have opted for Adam instead of SGD simplify the tuming process for the inner loop. This decision was based on the ability to use a common learning rate without requiring decay in the inner loop. In this section, perform ablation studies on how the inner loop optimizer affects the performance.
We implement RaT-BPTT (SGD) using SGD with learning rate 0.01 and learning rate decays at [120, 200) by 0.2. For IPC10 on CIFAR10, RaT-BPTT (SGD) achieves a 69.0% accuracy (std 0.3%), while RaT-BPIT (Adam) results in slightly higher accuracy of 69.4% (std 0.4%). Thus, RaT-BPTT (SGD) also outperforms previous methods in this setting by a large margin. It is crucial to note that our improvement attributed factors beyond merely employing Adam in the inner loop.
It is also noteworthy to point out that we are not the first to use Adam for the inner loop during training. Loo al. (2022) also uses Adam for their linearized inner loop. Some other papers (Loo et al. 2023] ZZou et al. 2022) have also adopted Adam for the linear loop during evaluations. We suspect that whenever Adam was an option, the benchmarking papers probably tried without significant improvements.
D.7 DISCUSSIONS ON EFFICIENCY
We have conducted a comparative analysis of the total training time for several methods, utilizing a consistent computational environment on an RTX8000 with 48GB. It should be noted that we have excluded RCIG from this comparison, as our reproduced accuracy falls short the reported number. The following the recorded training times (in hours) for CIFAR10 with IPC10: KIP (over 150), LinBa (100), FrePO (8), RaT-BPTT (22), MTT (14), and IDC (35), Among these methods, our cost ranks the third best.
19

---

Published as a conference paper at ICLR 2024
There ways to further improve the efficiency. 1) The current package utilize for meta meta-gradien calculation, the higher package, as noted in Maclaurin et al. (2015), 9 lacks efficiency compared to other methods. We could lower the time cost by altering our implementation to more efficient methodologies. 2) The references (Maclaurin et al. 2015: Sachdeva et al. 2023) contain efficient designs for the meta-gradient calculation. As reported in Sachdeva al. (2023 , it could lead to
up to 2x speedups compared with the higher package. This improvement would not only enhance the performance of our method but also bring it line with the efficiency benchmarks set by methodologies like FrePO. 3) Similar to FrePO, we may keep pool of parameter checkpoints to further optimize our method. This strategy would reduce the need for inner training from new random initializations.
D.8 BOOSTING FOR OTHER METHOODS
Our boosting framework is a method that can be combined with any gradient- based dataset distillation method. In this section, combine boosting with MTT Cazenavette al. (2022) proof of principle, choosing a representative method. We implement strong boosting (with 3 = 0) for MTT from IPC 10 to IPC 50 in steps of IPC 10. In Figure 13| the final generalization accuracy is 71.4%.
The accuracy improves rapidly throughout the boosting and the final performance is marginally lower (exact percentage pending) than the direct distillation of MTT, which stands at 71.6%.
D.9 VISUALIZATION
We incorporate visualizations for IPC10 CIFAR-10, 10, representing standardly trained (Figure I4). weakly boosted (Figure and strongly boosted images (Figure 15). Upon inspection, the images from both boosted categories appear more diverse compared to their standard counterparts.
20

---

Published as a conference paper at ICLR 2024
autsmosie
sutomable
automatte
sutomab
s.tomoble
sutsmoste
sctomable
b-1
brdd
trd
283
b.dd
Hrd
brd
brd
25g
dog
drg
day
cog
frg
frg
frog
honce
hore
horee
horos
:Hfp
8115
1112
1:b
#1c
INp
Figure 14: Visualization for RaT-BPTT standardly trained on CIFAR-10 with IPC10.
21

---

Published as a conference paper at ICLR 2024
austare
Piplanc
airsbree
brdd
brd
brd
hind
cut
cot
cit
Ext
cot
sct
frog
hog
frgg
frog
hrgg
htg
frog
htg
10335
hits
4112
mudk
1sck
trusk
Tret
truck
Figure 15: Visualization for RaT-BPTT with strong boosting (Boost-DD). CIFAR- 10 with IPC10.
22

---

Published as a conference paper at ICLR 2024
anstare
eratans
brdd
brd
brd
cutt
cel
5dt
cac
frog
hog
frgg
frog
hog
htg
fro2
frog
htg
hog
ha2s
15:35
4112
trdk
tscc
Sruce
trus
trac
Trxtt
Truck
Figure 16: Visualization for RaT-BPTT with weak boosting (Boost-DD), CIFAR-I0 with IPC10.
23

---

Published as a conference paper at ICLR 2024
E HARDNESS ANALYSIS
E DISSECTING THE DATA
Now, we attempt understand what is bottlenecking the gains when scaling up the distilled data from say IPCI to IPC50 (see Table 1) by analyzing the performance of dataset distillation on examples that are easy or hard to learn, using a hardness score to stratify the data. We first leverage the forgetting score Toneva et al. (2019) an empirical implementation of the hardness score it characterizes the difficulty of learning a datapoint along the training trajectory.
Forgetting score, For each data point and network trained for T epochs, we say has forgetting event at time t æ is correctly classified at time and misclassified at time t + 1. The forgetting score of data point is the sum of its forgetting events before time T, averaged over 10 random initializations. Toneva et al. (2019) show that based on the forgetting score a significant amount of training data can be omitted, without sacrificing test performance.
We first compute the forgetting score of the original network for each training data point. We then stratify the accuracy by forgetting score and compare it for the original network and the network trained on distilled data (Figure 9p. Notice that the highest boost in performance, especially for the easy datapoints (score 0) comes from simply having one image per class (IPC1). Further, one would have hoped that increasing the number of images per class would help distill more of the tail, enabling models to generalize better data points with larger forgetting scores. We notice that this happens to some extent till a score of 4, but after that despite there being lot of datapoints with larger hardness score (Figure 9 bottom) IPC50 seems yield minimal marginal improvements. This suggests that future works might benefit from focusing on how one can distill datapoints with larger forgetting scores.
E.2 HARDNESS SAMPLER
We present initial approach to enhance learning on challenging examples through a "hardness sampler" that modifies the data batch distribution. Specifically, our objective is to enrich validation batches with more challenging examples, concentrating primarily those lying mid-way between the extremes of very easy and very hard examples. This approach is inspired by the parabolic shape depicted in Figure
However, the calculation of the forgetting score is often computationally demanding and thus may not be prac- tical for all applications, especially as part of the outer loop dataset distillation. Moreover, Forgetting score has only been defined for and analysed on networks that are trained on the data that being scored. To address propose adaptive hardness score
Ouaprat 4
74.75-
Ousprsb
74.50-
14.00
these challenges, that is both efficient and versatile. This score computed based on the disagreement in predictions Feng et al. (2020) across 8 randomly trained networks using the current dis- tilled dataset. To stay relevant to the evolving challenges, this score is updated adaptively every 50 epochs.
3235-
73:00
30000
40020
50000
stéco 10033 Se000 5093E 100000 Stps
Figure 17: Test performance of hardness sampler versus standard for thresholds I and 4. We start to sample at 25K steps since we believe it becomes more rel- evant in the later stages of distillation. Setting: CIFARIO, IPC50, RaT-BPTT.
Based on this adaptive hardness score, we down-weight the easiest and hardest examples by giving the following weight us to sample with hardness score HS(()) e 8): w(z)= thr + (bs(H -),
where thr is a threshold which we set to either 1 or 4. For each update of the meta-gradion we sample from a training data point x proportional to w(x), which upweights medium cardumm-harddd examples the most. Figure 17 demonstrates notable performance improvement for the IPC50 distillation on CIFARIA 10, and hints that this direction might be fruitful for future work to pursue.
24


Reviews:
[{"review_id": "DyMAEMsx6I", "reviewer": "Reviewer_Qtcx", "summary": "This paper proposes Random Truncated BPTT (RaT-BPTT) for solving the bi-level optimization problem in dataset distillation. RaT-BPTT incorporates a truncation coupled with a random window, effectively stabilizing the gradients and speeding up the optimization while covering long dependencies. Empirical results show that such simple method outperforms existing methods for dataset distillation application.", "questions": "Is the proposed algorithm applicable for general bi-level optimization problems, not only for the dataset distillation application? If so, it would be great to add some discussions on what other applications using bi-level optimization (which currently use BPTT) one can use RaT-BPTT instead.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple and easily applicable in practice\n- The empirical results are positive", "weaknesses": "- No theoretical results supporting the empirical results. Analyzing the bi-level optimization problem might be difficult in general, but it is better to find some simple setting where theory can explain when/why the proposed method outperforms existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yoOG5jWitO", "reviewer": "Reviewer_dNLL", "summary": "The paper proposes three key changes to the meta-matching framework of data distillation: (1) randomly truncated backpropagation through time (BPTT); (2) boosting in data distillation; and (3) using Adam optimizer in the inner-loop (although the authors don’t state this as a primary difference). These changes lead to SoTA results on numerous image classification datasets, especially compared to the naive meta-matching algorithm [1].\n\n[1] Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks.", "questions": "- (Figure 3) The change in gradient norm with more steps is not a good indicator of training stability. I would suggest looking at either the variance of these gradients [1], or the eigenvalues of the hessian matrix (a.k.a. sharpness) [2] as better aligned indicators.\n- Do you expect the boosting idea to work with techniques other than RaT-BPTT, e.g., DSA [3], MTT [4]?\n- One subtle change in RaT-BPTT is the usage of Adam optimizer in the inner-loop. Notably, all existing distillation techniques use SGD. Is this the main reason for improvement compared to other techniques? How does RaT-BPTT (SGD) compare with RaT-BPTT (Adam)? I would suggest referring to [5] for a better understanding of using Adam in data distillation.\n\nI'd be happy to consider increasing my overall rating if some of these questions are addressed in the rebuttal.\n\nOther comments and suggestions (not used in deciding my overall rating):\n- Please include full-data performance in Table 1.\n- The meta-gradient of SGD in the inner-loop can be efficiently computed [6]. Please mention it in the full-text.\n- It would be great to include an analysis of the effect of varying $\\beta$ in Boost-DD on downstream generalization.\n\n[1] Faghri, Fartash, et al. \"A study of gradient variance in deep learning.\" arXiv preprint arXiv:2007.04532 (2020).\n\n[2] Cohen, Jeremy M., et al. \"Gradient descent on neural networks typically occurs at the edge of stability.\" arXiv preprint arXiv:2103.00065 (2021).\n\n[3] Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 12674–12685, 2021b.\n\n[4] George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4750–4759, 2022.\n\n[5] Sachdeva, Noveen, et al. \"Farzi Data: Autoregressive Data Distillation.\" arXiv preprint arXiv:2310.09983 (2023).\n\n[6] Maclaurin, Dougal, David Duvenaud, and Ryan Adams. \"Gradient-based hyperparameter optimization through reversible learning.\" International conference on machine learning. PMLR, 2015.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Thorough empirical evaluation, and SoTA performance on multiple datasets and IPC settings.\n- Improved efficiency for meta-gradient computation compared to full BPTT [1] in terms of both memory & time.\n- I really like the brief hardness-stratified analysis in Figure 10.", "weaknesses": "- Using meta-gradient norm as an indicator for stability (more in questions).\n- Boosting is optimization-agnostic procedure however it’s only tested with RaT-BPTT (more in questions).\n- Section 5 (boosting) is not well-discussed with key details shifted to the appendix. I would suggest keeping a subset of results from Figures 7-10 (and moving others to the appendix), but with all details for those experiments complete in the main-text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T7s7DvlSeW", "reviewer": "Reviewer_wwkm", "summary": "This paper proposes simple and effective method for dataset distillation. Since Back Propagating Through Time (BPTT) is computationally expensive, we truncate a trajectory of inner  optimization steps, called truncated BPTT. However, the performance of truncated BPTT is worse than BPTT due to the biased gradient. In order to get better trade-off between computational cost and performance, the paper proposes to sample random window of inner trajectory, which uses the same amount of computational cost as the truncated BPTT but with better performance. Moreover, it proposes \"Boosted Dataset Distillation\" to mitigate intercorrelation between distilled images, where it iteratively learns to distill a dataset into a small set while setting the learning rate of previously distilled instances to small value.", "questions": "- What happens if we scale the meta-gradient of BPTT with the norm of the gradient during training? The authors argue training instability of BPTT based on the larger gradient norm. Then the most straightforward way to mitigate the issue would be gradient clipping or scaling the gradient.\n\n- What happens if we gradually increase the total number of unrolling steps for BPTT instead of using fixed $T$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is simple and effective. It outperforms most of the baselines on CIFAR, CUB and Tiny-ImageNet datasets.\n\n- Ablation studies show the effective of proposed method.\n\n- The proposed method can be easily integrated with other bilevel optimization based dataset distillation methods.", "weaknesses": "- Although the proposed method uses the same amount of computational cost as the truncated BPTT method, it still requires to store a trajectory of length $M$. If the dataset becomes large, we may need large $M$ for convergence at inner loop, which hinders scalability of the proposed algorithm.\n\n- It is hard to analyze why the proposed method helps improving performance compared to BPTT or truncated BPTT.\n\n- The paper requires more comprehensive empirical experiments to verify the effectiveness of the proposed method. First and foremost, the authors of the paper did not conduct experiments for architecture generalization. While Table 1 simply demonstrates the method's ability to generalize from a shallow convnet to a wide convnet, I believe this is not adequate. I am curious about how the proposed dataset distillation method can be applied to other architectures, such as VGG, EfficientNet, and ResNet. Secondly, I wonder the proposed method scales to ImageNet dataset which has 1000 classes. \n\n- There is no comparison of computational cost between the proposed method and the other baselines. I think the proposed method is way more expensive than other baselines (such as FrePo) which simplify the  inner loop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was originally rejected because its post-rebuttal version was slightly over 9 pages long. The authors correctly pointed out that the CFP didn't specify length limits for intermediate versions of papers and that there were accepted papers whose post-rebuttal versions also exceeded the page limit. The PC chairs deliberated over the merits of the paper, concluded that the paper would have been accepted if the slight length overflow hadn't been a consideration, and decided to accept the paper.
Decision: Accept (poster)