Data Distillation: A Survey
Authors: Noveen Sachdeva
Julian McAuley
Venue: Accepted by TMLR ()

Abstract: The popularity of deep learning has led to the curation of a vast number of massive and multifarious datasets. Despite having close-to-human performance on individual tasks, training parameter-hungry models on large datasets poses multi-faceted problems such as (a) high model-training time; (b) slow research iteration; and (c) poor eco-sustainability. As an alternative, data distillation approaches aim to synthesize terse data summaries, which can serve as effective drop-in replacements of the original dataset for scenarios like model training, inference, architecture search, etc. In this survey, we present a formal framework for data distillation, along with providing a detailed taxonomy of existing approaches. Additionally, we cover data distillation approaches for different data modalities, namely images, graphs, and user-item interactions (recommender systems), while also identifying current challenges and future research directions.

Published in Transactions on Machine Learning Research (07 /023)
Data Distillation: A Survey
Noveen Sachdeva
nosachde Cucsd. edu
Computer Science & Engineering University of Colifornia, San Diego
Julian McAuley
jmcouley@ucsd. edu
Computer Science & Engineering University of Colifornia, San Diego
Reviewed on OpenReview: https: // openreview. net/ forum? id- LmXMXP74TO
Abstract
The popularity of deep learning has led to the curation of a vast number of massive and multifarious datasets. Despite having close-to- to-human performance on individual tasks, training parameter-hungry models on large datasets poses multi-faccted problems such as (a) high model-training time; (b) slow research iteration; and (c) poor eco-sustainaablity. As an alternative, data distillation approaches aim to synthesize terse data summaries, which can serve effective drop-in replacements of the original dataset for scenarios like model training, inference, architecture search, etc. In this survey, we present a formal framework for data distillation, along with providing detailed laxonomy of existing approaches. Additionally, we cover data distillation approaches for different data modalities, namely images, graphs, and user-item interactions (recommender systems), while also identifying current challenges and future research directions.
1 Introduction
(Loose) Definition (Data distillation) Approaches that aim to synthesize tiny and high-fidelity data summaries which distill the most imporlant knowledge from a given larget dataset. Such distilled summaries are oplimized to serve effective drop-in replacements of the original dalasel for efficient and accurate data-usage applications like model training, inference, architecture search, etc.
The recent "scalleib-everyyhinng" viewpoint. (Ghorbani et al., 2021; Hoffmann et al., 2022; Kaplan et al., 2020), argues that training bigger models (i.e., consisting of a higher number of parameters) on bigger datasets, and using larger computational resources the sole key for advancing the frontier of artificial intelligence. Such studies observe and hypothesize the generalizability of neural networks as power-law w.r.t. the aforementioned factors, albeit with small exponents. On the other hand, reasonable argument is that principled and well-reasoned solution will be more amenable to various scaling-laws, thereby leading to faster progress. Data distillation (Definition 1) is clearly task rooted in the latter school of thought by introducing the fidelity of data as in important covariate in such neural scaling- Sorscher et al. (2022) demonstrate this viewpoint analytically by using simple houristics to prune away data with low measures of signal for model training. Clearly, the scale viewpoint still holds, that if we keep increasing the amount of data (albeit now compressed and of higher quality), we will observe an improvement in both upstream and downstream generalization, but a faster rate.
Motivation. A terse, high-qualily data summary has use cases from a variely of standpoints. First and foremost, leads to a faster model-training procedure, In turn, faster model training equates to (1) compute- cost saving and expedited research iterations, e.e the investigative procedure of manually experimenting different ideas; and (2) improved aco-sustainablitty, é.e., lowering the amount of compute time directly leads lower carbon footprint from running power-hungry accelerated hardware (Gupta et al., 2022). Additionally,

---

Published in Transactions on Machine Learning Research (07 2022)
Data Distillation
Train
<< 50K distilled images
Similar
Performance
Train
50K images
Learning algorithm
[HQ Image Link] Figure 1:
The premise of data distillation demonstrated using an image dataset.
a small data summary democratizes the entire pipeline, as more people can train state-of-Che-arl algorithms on reasonably accessible hardware using the data summary. Finally, a high-quality data summary indirectly also accelerates orthogonal procedures like neural architecture search (Liu et al., 2019), approximate nearest neighbour search (Arya et al., 1998), knowledge distillation (Hinton et al., 2015), elc., where the procedure needs to iterate over the entire dataset multiple times.
Comparison with data pruning. Another reasonable avenue for summarizing large datasets is pruning away low-quality data which presumably does not carry large amount of signal to be captured during model- training. The primary challenge for such data pruning approaches (a.k.a. coreset construction) lies in tagging the hardness of each data-point which can be used for subsequent pruning (typically in a greedy fashion). Prominent data-pruning approaches propose heuristics for the same, relying concepts such as shapley values (Ghorbani & Zou, 2019), confidence scores (Coleman et al., 2020), coror-conttibution (Toneva et al., 2019), feature-space geometry (Abbas et al., 2023; Sorscher et al., 2022; Welling, 2009), etc. Another line of work builds on the advances in submodular optimization (see Bilmes (2022) for a review) to approxinately solve the NP-Hard combinatorial optimization of selecting the subset that maximizes a set-level goodness function, when such goodness functions are provably submodular (Killamsetty et al., 2021; Mirzasoleiman et al., 2020; S et al., 2021). Notably, such data pruning methodologies inherently share the same goal data distillation but are severely restricted due to only retaining data already in the target dataset, thereby leading to finite expressivity and hence, generally, lower sample-fidelity (sce Ayed & Hayou (2023) for deeper theoretical outlook on the fundamental limitations of data pruning). Further, recent empirical studies of data pruning methodologies (Guo et al., 2022) demonstrate that the efficacy of such data pruning heuristics rarely and irregularly transfers to practical scenarios, with random downsampling being a hard bascline.
Comparison with knowledge distillation & transfer learning. Despite inherently distilling some notion of knowledge, we would like to highlight both knowledge distillation and transfer learnóng are orthogonal procedures to data distillation, and can potentially work in conjunction to perform both tasks more efficiently. More specifically, knowledge distillation (Hinton et al., 2015) entails distilling the knowledge from a trained teacher network into a smaller student network, On the other hand, transfer learning (Pratt, 1992) focuses on transferring knowledge across similar tasks, e.g., from image classification image sogmentation. Orthogonally, data distillation aims to distill the knowledge from given dataset into a terse data summary. Such data summaries can be used in conjunction with knowledge distillation or transfer learning procedures for both (1) faster learning of the teacher models; and (2) faster knowledge transfer to the student models. The same holds true for model compression techniques (LeCun et al., 1989), where similar to knowledge distillation, the goal is to reduce model storage size rather than reducing the training time or increasing the sample-iidelity.
In this survey, we intend to provide a succinct overview of various data distillation frameworks across different. data modalities. We start by presenting a formal data distillation framework in Section 2, and present technicalities of various existing techniques. We classify all data distillation techniques into four categories
2

---

Published in Transactions on Machine Learning Research (07/2023)
(see Figure 2 for taxonomy) and provide detailed empirical comparison of image distillation techniques in Table 1. Subsequently, in Section 3, we discuss existing data distillation frameworks for synthesizing data of different modalities, as well as outlining the associated challenges. In Section 4, we discuss alternative applications of synthesizing a high-fidelity data summary rather than simply accelcrating model training along with pointers to existing work. Finally, in Section 5, we conclude by presenting common pit/aalls existing data distillation techniques, along with proposing interesting directions for future work.
2 The Data Distillation Framework
Before going into the specifics of data distillation, we start by outlining useful notation. Let D - {a1 ) be a given dataset which needs to be distilled, where z; E X are the set of imput features, and y E y is the desired label for zy. For classification tasks, let C. be the set of unique classes in V, and DDE 2 {(T.. v1) y = c) be the subset of 'D with class c. We also define the matrices _ [xa][1] and - [y]] for convenience. Zt, data distillation techniques aim to synthesize a high-fidelity data summary Dayn A { D=)) such that < |D|- We define Di sy:: Xsyn; and Y, syn similarly as defined for D. Let Φg : x represent learning algorithm parameterized by 0. We also assune access to twice-differentiabll cost function / )' x y ~ I. and define (D( - (((n)) summarized in Appendix A. Notably, since D and Dayn share the same data domain (2), under reasonable systems' assumptions, training & using gradient descent (GD) on Dsyn will have compared to training t on D.
Given a data budget
for convenience. Notation is also
D n
training-lime speedup
For the sake of uniformity, we refer to the data synthesized by data distillation techniques as data summary henceforth. Inspired by the definition of coresets (Bachem et al., 2017), we formally define an a-pproximmaee data summary, and the data distillation task as follows:
Definition 2. (E (e-approxiaate data summary) Given a learning algorithm &, let 0o, 8Pan represent the optimal set of parameters for & estimated on D and Dsyn= and E IR; we define an c-approzimate data summary as one which solisfies:
- ( sup (1 (x), Jang S
(1)
Definition 3. (Data distillation) Given a learning algorithem 4 let 02, @pm represent the optimal set of parameters for & estimated on 'D and Dayn: we define data distillation as optimizing the following:
arg min sup (1((4g0()))) (2) Den.
(2)
From Definition 3, we highlight three cornerstones of evaluating data distillation methods: (1) Performance: downstream evaluation of models trained on the syuthesized data summary us. the full dataset (e.g., accuracy, FID, nDCG, etc.): (2) Efficiency: how quickly can models reach full-data performance (or even excoed it), i.e., the scaling of vs. downstream task-performance; and (3) Transferability: how well can data summaries generalize to diverse pool of learning algorithms, in terms of downstream evaluation.
No free lunch. The universal "No Free Lunch" theorem (Wolpert & Macready, 1997) applics to data distillation as well. For example, looking at the transferability of a data summary, it is strongly dependent on the set of encoded inductive biascs, i.e., through the choice of the lcarning algorithm & used while distilling, as well as the objective function I(.,). Such biases are unavoidable for any data distillation technique, in a sense that learning algorithms closely following the set of encoded inductive biases, will be able to generalize better on the data summary than others.
Keeping these preliminaries in mind, we now present a formal framework for data distillation, encapsulating existing data distillation approaches. Notably, the majority of existing techniques intrinsically solve a bilevel optimization problem, which are tractable surrogates of Equation (2). The inner-loop typically optimizes a representative learning algorithm on the data summary, and using the optimized learning algorithm, the outer-loop optimizes tractable proxy of Equation (2).
3

---

Published in Transactions on Machine Learning Research (07/2023)
Data Distillation
Image Distillation
RecSys Distilltion
Graph Distillation
Meta-Model Matching
Gradient Matching
Trajectory Matching
Distribution Matching
DD. KIP, READ, FRaPO. LINBA. D1aTILI-C
DC, DSA, DOC, IDG, GOOND, Do8ComD
MTT. HABA,
DM, CAFE, GAN. KFS.
TESLA
GCDM
Section 2.1
Section 2.2
Section 2.3
Section 2.4
[QQ Image Link] Figure 2: A taxonomy of cxisting data distillation approaches.
Some common assumptions that existing data distillation techniques follow are: (1) static-length data summary, i.e,, n is fixed and is treated as a tunable hyper-parameter; and (2) we have on-demand access to the target dataset D which is also assumed to be iid. Notably, the outer-loop optimization of Day happens simply through GD on the analogously defined Xsn E Raxdim(A)) which instantiated as free parameters. Note that the labels, syn E B=xdim())) can be similarly optimized through GD as well (Bohdal al., 2020). For the sake of notational clarity, we will interchangeably optimization of Dayn (Xsyn: syn) henceforth.
2.1 Data Distillation by Meta-modes Matching
Meta-model matching-based data distillation approaches fundamentally optimize for the transferability of models trained on the data summary when generalized to the original dataset:
arg min LD (0Pm)
s.t. Pay
arg min L (0),
(3)
Den
8
where intuitively, the inner-loop trains a representative learning algorithm on the data summary until convergence, and the outer-loop subsequently optimizes the data summary for the transferability of the optimized learning algorithm to the original dataset. Besides common assumptions mentioned earlier, the key simplifying assumption for this family of methods is that a perfect classifier exists and can be estimated on D, i.e., 3 0 s.t. 1( (x). v) = 0, Vz ~ X,y ~ y Plugging the second assumption along with the iid assumption of 'D in Equation (2) directly translates to Equation (3). Despite the assumption, Equation (3) is highly expensive both in terms of computation time and memory, due to which, methods from this family typically resort to making further assumptions.
Wang et al. (2018) (DD) originally proposed the task of data distillation, and used the meta-model matching framework for optimization. DD makes the optimization in Equation (3) tractable by performing (1) local optimization à la stochastic gradient descent (SGD) in the inner-loop, and (2) outer-loop optimization using Truncated Back Paok-Prrooggttoo Through Time (TBPTT), f.e., unrolling only limited number of inner-loop optimization steps. Formally, the modified optimization objective for DD is as follows:
arg min Dey
[LD (0r) s.t. Oc41 + - -1 VoLr, (0:), Pr
E
(4)
where Pp is a parameter initialization distribution of choice, T accounts for the truncation in TBPTT, and n is tunable learning rate. We also elucidate DD's control-flow in Algorithm 1 for reference.
Notably, TBPTT has been associated with drawbacks such as (1) computationally expensive inner-loop unrolling; (2) bias involved with truncaled unrolling (Wu et al., 2018); and (3) poorly conditioned loss landscapes, particularly with long unrolls (Metz et al., 2019). Consequently, the TEPTT framework was empirically shown to be incffective for data distillation (Zhao et al., 2021). However, recent work (Deng & Russakovsky, 2022) claims that using momentum- based optimizers and longer inner-loop unrolling can greatly improve performance. We delay deeper discussion of this work to Section 2.5 for clarity.
4

---

Published in Transactions on Machine Learning Research (07/2023)
Algorithm 1: Control-flow of data distillation using naive meta-matching (Equation (4)) Target dataset D, outer-loop iterations K, parameter initialization distribution Po, inner-loop iterations T, inner-lopp learning rate 7), outer-loop learning rate Tayn
Input:
1 Initialize: (X syn
Y syn) ~
2 for == K do
// Outer-loop: optimize Dayn
3 Initialize OU ~ Po
for t= I....,I I do
1/ Inner-loop: optimize & on D 1
O2 + 0 - 11 V V D (0t-1)
X
Xk-1
lyyn VX (0T)
// Update Xsy by computing unrolled meaa-egr
syn
syn
Y syn
+ Yk-1
Tyyn Vy (0T)
// Update Yom by computing unrolled meta meta-gradient
syn
Output: DK =
X
syn
YK
syn
Analogously, a separate line of work focuses on using Neural Tangent Kernel (NTK) (Jacot et al., 2018) based algorithms to solve the inner-loop in closod form. As brief side note, the infinito-width correspondence states that performing Kernelized Ridge Regression (KRR) using the NTK of a given neural network, is equivalent to training the same co-width neural network with T.2 reconstruction loss for do SGD-steps. These "oo-width" neural networks have been shown to perform reasonably compared to their finite-width counterparts, while also being solved in closed-form (see Lee et al. (2020) for a detailed analysis on finite us. infinite neural networks for image classification). KTP uses the NTK of a fully-comnected neural network (Nguyen et al., 2021a), or a convolutional network (Nguyen et al., 2021b) in the inner-loop of Equation (3) for efficient data distillation. More formally, given the NTK K: x X' + R of a neural network architecture, KIP optimizes the following objective:
arg min Xam. Y
Y - -xxx, (Kx,nXen + A1)- Y
55)
where KAB E [||4 represents the gramian matrix of two sets A and B, and whose (i,j)th element is defined by XCAA,, B,). Although KIP docsn't impose any additional simplifications to the meta-model matching framework, it has an O(D) - dim(X)) time and memory complexity, limiting its scalability. Subsequently, RFAD (Loo et al., 2022) proposes using (1) the light-weight Empirical Neural Network Gaussian Process (NNGP) kernel (Neal, 2012) instead of the NTK; and (2) classification loss (e.g., NLL) instead of the L2-reconstruction loss for the outer-loop to get Onn) time complexity while also having better performance. On a similar note, FRePO (Zhou et al., 2022b) decouples the feature extractor and a linear classifier in $ and alternatively optimizes (1) the data summary along with the classifier, and (2) the feature extractor. To be precise, let fa xx X" be the feature extractor, Go : Xx" + y be the linear classifier, s.t. ((x) ge (x) Vz E X: the optimization objective for FRePO can be written as:
(K& ( + AT)-1 Ysy |
arg min Xays: Yom
E ParPP
Σ
Y
Kdc X
(6)
s.t. 0++1 + O₂ -
V (ollow (a.,))/DDe
E
K Xm fo. (Xeyn fo, (Xayn)".
where 7' represents the number of inner-loop update steps for the feature extractor fe. Notably, (1) wide architecture for fe is crucial for distillation quality in FRePO; and (2) despite the bilevel optimization, FRePO shown to be more scalable compared to KTP (Equation (5)), while also being more generalizable.
2.2 Data Distillation by Gradient Matching
Gradient matching based data distillation, at a high level, performs one-step distance matching on (1) the network trained on the target dataset (D) vss. (2) the same network trained on the data summary (Dayn) In contrast to the mota-model matching framework, such an approach circunvents the unrolling of the inner-loop, thereby making the overall optimization much more efficient. First proposed by Zhao et. al. (2021) (DC),
5

---

Published in Transactions on Machine Learning Research (07 (07/2023)
data summarics optimized by mundient-matching significantly outperformed data pruming methodologies, well as TBPTT-based data dist.illation proposed by Wang et al. (2018). Formally, given learning algorithm
2, DC solves the following optimization objective:
Σ D (VLLD= (o). VoLDa, (h))) s.t. act1 t 01 10-nn VoL (0:),
T
arg min Dey
E Po C
LI=U
where T' accounts for model similarity T-steps in the future, and D R R R is a distance metric of choice (typically cosine distance). In addition to assumptions imposed by the meta-model matching framework (Section 2.1), pradient-matching assumes (1) inner-loop optimization of only T steps; (2) local smoothness:
two sets of model parameters close to each other (given distance metric) imply model similarity; and (3) first-order approximation of 0P: instead of exactly computing the training trajectory of optimizing 0g on D (say OP); perform first-order approximation on the optimization trajectory of 8g on the much smaller Dayn (say ",""), i..,, approximate 6P single gradient-descent update on 02r using DD rather than 00_ (Figure 3).
Subsequently numerous other approaches have been built atop this framework with subtle variations. DSA (Zhao & Bilen, 2021) improves over DC by performing the same image-augmentationn (e.g., crop, rotate, jitter, etc.) on both D and Dayn while optimizing Equation (7). Since these augmentations are universal and are applicable across data distillation frameworks, augmentations performed by DSA have become a common part of all methods proposed henceforth, but we omit them for notational clarity. DCC (Tee el. al., 2022b) further modifies the gradient-matchine objective incorporate class contrastive signals inside each gradient-matchine step and is shown to improve stability as well performance. With 6, evolving similarly as in Equation (7), the modified optimization objective for DCC can be written as:
T D
(0.),, cEe [Vologn (0)))]
arg min Dn
VoCx))))) CEC
(8)
Oa-Pa
Most recently, Kim et al. (2022) (TDC) extend the gradient matching framework by: (1) multi-formation: to synthesize a higher amount of data within the same memory budget, store the data summary (e.g., images) in lower resolution to remove spatial redundancies, and upsample (using e.g., bilinear, FSRCNN (Dong et al., 2016)) to the original scale while usage; and (2) matching gradients of the network's training trajectory over the full dataset D rather than the data summary Dayn- To be specifie, given kx upscaling function f Rdxdd + kkd the modified optimization objective for IDC can be formalized as:
T
Σ D (V)Co=00)).aCCrro....)) s.t. + -1-1 VoLD(O.)
arg min Dan
E 0g
Kim et al. (2022) further hypothesize that training models Day instead of D in the inner-loop has two major drawbacks: (1) strong coupling of the inner- and outer-loop resulting in chicken-egg problem (McLachlan & Krishnan, 2007); and (2) vanishing network gradients due to the small size of D Dayn, leading to an improper outer-loop optimization for gracheng based techniques.
2.3 Data Distillation by Trajectory Matching
Cazenavette et al. (2022) proposed MTT which aims to match the training trajectories of models trained on 'D es. Dayn- More specifically, let (8P)T_ represent the training trajectory of training 4o on D; trajectory matching algorithus aim to solve the following optimization:
T-M M
D (OP PPPN))
arg min D Dan
E B
Σ
D ( OP)
(10)
L=U
S.t.
VoLDm
(DPan)
Pptn + OP- VoLomn (0P)
where D:R!1 x R is distance metric of choice (typically L2 distance). Such an optimization can intuitively be seen as optimizing for similar quality models trained with N SGD steps on D D compared to

---

Published in Transactions on Machine Learning Research (07/202) 2023)
0 Pa
8
D
6
Minimize: D
Update Payn by minimizing:
! (0,,0000 (0,))
$ Eu
Update on
(0 6P)
Updale 20yn
Meta-Model Matching
Minimize:
Gradient Matching
al D D
Trajectory Matching
8
G2D
[Q Image Link] Figure 3: The underlying optimization in various data distillation frameworks.
>> N steps on D, thereby invoking long-horizon trajectory matching. Notably, calculating the gradient of Equation (10) w.r.t. Dayn encompasses gradient unrolling through N-timesteps, thereby limiting the scalability of MTT. On the other hand, since the trajectory of training De on D, i.e., (6P))- is independent of the optimization of Dayn it can be pre-computed for various 0o Po initializations and directly substituted. Similar to gradient matching methods (Section 2.2), the trajectory matching framework also optimizes the first-order distance between parameters, thereby inheriting the local smoothness assumption. As a scalable alternative, Cui et al. (2022b) proposed TESLA, which re-parameterizes the parameter matthiigg loss of MTT in Equation (10) (specifically when D is set as the L2 distance), using linear algebraic manipulations to make the bilevel optimization's memory complexity independent of N. Furthermore, TESLA uses learnable soft-labels (m)) during the optimization for an increased compression efficiency.
2.4 Data Distillation by Distribution Matching
Even though the aforementioned madient-mmaching or trajectory-matching based data distillation tochniques have been empirically shown to synthesize high-quality data summaries, the underlying bilevel optimization, however, is oftentimes computationally expensive procedure. To this end, distribution-matehie techniques solve correlated proxy task via a single-level optimization, Icading to a vastly improved scalability. More specifically, instead of matching the quality of models on 'D us. Dayn: distributtion-malcinng techniques directly match the distribution of 'D us. Dayn- The key assumption for this family of methods is that two datasets that are similar according to a particular distribution divergence metric, lead to similarly trained models.
First proposed by Zhao & Bilen (2023), DM uses (1) numerous parametric encoders cast high-dimensional data into respective low-dimensional latent spaces; and (2) an approximation of the Maximum Mean Discrepancy to compute the distribution mismatch between D and Day in each of the latent spaces. More precisely, given set of l encoders & (wh X X} the optimization objective be written
arg min Den
[+(x)] - [+())]
E De
(11)
DNDO
DM uses set of randomly initialized neural networks (with the same architecture) to instantiate E. They observe similar performance when instantiated with more mcaningful, task-optimized neural networks, despite it being much less efficient. CAFE (Wang et al., 2022) further refines the distributtion-mallchnng idea by: (1) solving a bilevel optimization problem for jointly optimizing a single encoder (Ø) and the data summary, rather than using a pre-determined set of encoders (E); and (2) assuming a neural network encoder (Ø), match the latent representations obtained at all intermediate layers of the encoder instead of only the last layer. Formally, given (I. 1a-layer neural wetwork to X + where o6 represents 4's output at the tth
7

---

Published in Transactions on Machine Learning Research (07 2023)
layer, the optimization problem for CAFE can be specified as:
arg min
E enc c
Σ
E Do
(x)]
E
1 (2)] -
E (awg)DD
Dar
am D
(12)
s.t. 0141 t 02 - 17. VoLD (0r) fayla, 0) softmax
((
(ob(((0)) D'
y
where ((1|:,0 0) intuitively represents the nearest centroid classifier on Dsyn using the latent representations obtained by last layer of Dg. Analogously, IT-GAN (Zhao & Bilen, 2022) also uses the distribution-matchnn framework in Equation (11) to generate data that is informative for model training, in contrast to the traditional GAN (Goodfellow et al., 2014) which focuses on generating realistic data.
2.5 Data Distillation by Factorization
All of the aforementioned data distillation frameworks intrinsically maintain the syuthesized data summary as a large set of free parameters, which are in turn optimized. Arguably, such a setup prohibits knowledge sharing between synthesized data points (parameters), which might introduce data redundancy. On the other hand, factorization- based data distillation techniques parameterize the data summary using two separate components: (1) bases: set of mutually independent base vectors; and (2) hallucinators: mapping from the bases' vector space to the joint data- and label-space. In Lurn, both the bases and hallucinators are optimized for the task of data distillation.
Formally, let B a {b, e B) be the set of bases, and H. A {h+ Br X y be the set of hallucinators, then the data summary is parameterized as Dayn {h(b)) h-H- Even though such a two-pronged approach seems similar to generative modeling of data, note that unlike classic generative models, (1) the input space consists only of a fixed and optimized set of latent codes and isn't meant to take any other inputs; and (2) given a specific B and H. we can generate at most [B] |7| sized data sunmaries. Notably, such a hallucinator-baser data paramctcrization can be optimized using any of the aforementionod data optimization frameworks (Sections 2.1 to 2.4)
This framework was concurrently proposed by Deng & Russakovsky (2022) (we take the liberty to term their unnamed model as "Lin-car Ba-ses") and Liu et al. (2022c) (HaBa). LinBa modifies the general hallucinator-bases framework by assuming (1) the bases' vector space (IB) to be the same as the task input space (A'); and (2) the hallucinator to be linear and additionally conditioned on a given predictand. More specifically, the data parameterization can be formalized as follows:
Dayn & (y H 3) } J
H~H
(13)
H. H. R BIIi |H|
s.t. B B ((xdmmx) a lb. E x) B
where for the sake of notational simplicity, assume y E R represents the one-hot vector of the label for which we want to generate data, and the maximum amount of data that can be synthesized S |C|- [H]. Since the data generation (Equation (13)) is end-to-end differentiable, both B and H are jointly optimized using the TBPTT framework discussed in Section 2.1, albeit with some crucial modifications for vastly improved performance: (1) using momentum- -base optimizers instead of vanilla SGD in the inner-loop; and (2) longer unrolling (z 100 steps) of the inner-loop during TBPTT. Liu al. (2022c) (HaBa) relax the linear and predictand-connittooaall hallucinator assumption of LinBa, equating to the following data parameterization:
Dayn A (h(b), w) boyB ho7
s.t. B 2 (b. E E 1! ; H (ho, X + x3/31
(14)
where B and H are optimized using the trajectory matching framework (Section 2.3) with an additional contrastive constraint to promote diversity in Dayn (cf. Liu et al. (2022c), Equation (6)) Following this setup, HaBa can generate at most [B] [7]] sized data summarics. Furthermore, one striking difference between HaBa (Equation (14)) and LinBa (Equation (13)) is that to generate each data point, LinBa uses a linear combination of all the bases, whereas HaBa generates data point using single base vector.
8

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Table Comparison of data distillation methods. Each method (1) synthesizes the data summary on the train-set; (2) unless mentioned, trains a 128-width ConvNet (Gidaris & Komodakis, 2018) on the data summary; and (3) evaluates it on the test-sct. Confidence intervals obtained by training at least 5 networks the data summary. LinBa (No Fact.) represents LinBa with the factorization. Methods evaluated using KRR are marked as (oo-Conv) or (oc-FC). The equivalent storage-in-bytes is used for factoriaation-base techniques instead of IPC. The best method in their category is emboldened, the best-overall non-factorized method evaluated on ConvNet colored orange, and the best-overall factorized method colored blue.
Dataset
MNIST
CIFAR-10
CIFAR-100
Tiny ImageNet
Imgs/Class Class (IPC)
10
50
10
50
50
50
Random
64.9
95.1
97.9
14.4
26.0
43.4
4.2
14.6
30.0
1.5
6.0
16.8
Baselines
13.5
20.0
10.2
22.0
11.2
21.
10.3
20.5
20.4
10.1
10.8
21.8
Herding!
89.2
93.7
94.9
21.5
31.6
40.4
8.4
17.3
33.7
=1.6
+0.3
+0.2
+1.2
+0.7
+0.66
+0.3
#0.5
+0.5
Forgetting
35.5
68.1
88.2
13.5
23.3
23.3
4.5
15.1
30.5
13.8
1:3.3
21.22
2:1.0
1:1.1
1..2
30.3
20.3
DD
79.5
36.8
+8.1
+1.2
LinBa (No Fact.) 16
95.2
98.8
99.2
49.1
62.4
70.5
21.3
34.7
20.3
20.1
2:01
20.6
2.0.4
20.4
20.8
20.5
Meta-model Matching
KIP (ConvNet)"
90.1
97-5
98.3
49.9
62.7
68.6
15.7
28.3
10.1
10.0
10-1
20.2
10.3
1.
10.3
10.1
RFAD (ConvNet)
94.4
08.5
53.6
66.3
71.1
33.0
10.1
FRePO (CouvNet)"
93.0
98.6
99.2
46.8
65.5
42.5
44.3
15.4
25.4
KIP (oo-FCy7
85.5
97.2
98.4
40.5
53.1
58.6
10.1
10.2
20-1
2:04
2:0.5
1:0.1
KTP (oc-Conv)*
99.1
80.6
READ 5
97.2
09.1
99.1
61.4
73.7
76.6
44.1
46.8
#0.2
+0.3
FRePO (x--Conv)
92.6
98.6
99.2
47.9
68.0
74.4
32.3
44.9
43.0
19
26.5
#0.4
+0.1
2001
1:0.6
1.0.2
+0.11
+0.1
+h0.2
20.3
+0.3
+0.1
DC
91.7
97.4
98.2
28.3
44.9
53.9
12.8
25.2
30.5
4.6
11,2
10.9
Gradient Matching
+0.2
+0.6
=0.3
#0.7
DSA
88.7
97.8
99.2
52.1
60.6
13.9
32.3
42.5
6.6
14.4
22.6
DCC10
34.0
54.5
64.2
14.6
33.5
39.3
1007
20.3
2.0.
20.4
DM
89.7
97.5
98.6
26.0
48.9
63.0 0
11.4
29.7
43 6
3.9
12.9
24.1
Distr.
20.1
20.1
2:0.6
10.3
20.3
1:0.4
10.2
20.3
Matching
CAFE12
97.5
31.5
42.9
MTT13
46.3
65.3
71.6
24.3
40.1
47.7
8.8
23.3 2
28.0
Traj.
20.8
1:0.7
20.2
10.3
20.4
20.2
1.0.
30022
200.3
Matching
TESLA14 14
48.5
66.4
72.6
24.8
41.7
47.9
10.8
250.0
20077
20.4
20.3
10.3
1D
50.0
67.5
74.5
44.8
Factorization
2:0.4
1:0.8
2.0.1
10.2
LinBa 16
98.7 7
99.3
99.4
66.4
71.2
73.6
34.0
42.9
16.0
20.7
20.5
10.4
10.4
1004
10.5
10.4
10.7
20.7
HaBa 17
48.3
60.9
74.0
33.4
40.2
47.0
2..5
2:0 4
20022
10.4
20.3
2.02
8FS
59.8
72.0
75.0
40.0
50.6
22.7
27.8
-0.2
Full Dataset
09.6
84.8 2-0.1
56.2
37.6
20.1
#0.3
20.4
(Welling, 2009), (TTneva et al., 2019), (Wang et al., 2018), (Nguyen al., 2021b), (Loo al., 2022)
(Zhou et al., 2022b), (Nguyen et al., 2021a), (Zhao et al., 2021). (Zhao & Bilen, 2021), 10 (Lee al., 2022b)
11 (Zhao & Bilen, 2023), (Wang et al., 2022), (Cazenavette et al., 2022), (Cui al., 2022b)
15 (Kim et al., 2022), (Deng & Russakovsky, 2022), (Liu al., 2022c). (Lee et al., 2022a)

---

Published in Transactions on Machine Learning Research (07/2023)
Lee et al. (2022a) (KFS) further build atop this framework by maintaining different bases' vector space BB from the data domain X, such that dim(B) < dim(2). This parameterization allows KFS to store an even larger number of images, with comparable storage budget to other methods. Formally, the data parameterization for KFS can be specifiod as:
Dayn - U (M(b), c) JbmB
OEC
(15)
s.t. B B UB.
B. a {b E B) : H a {ho, B + x1
DEC
where KFS stores B bases per class, equivalent to total of |C| B- B- sized data summaries. Following this data parameterization, B and H are optimized using the distribution matching framework for data distillation (Equation (11)) to ensure fast, single-level optimizatiin.
Data Distillation vs. Data Compression. We highlight that is non-trivial to ensure a fair comparison between data distillation techniques that (1) are "non-lactorized"" i.e., maintain each synthesized data point as set of tree-parameters (Sections 2.1 to 2.4); and (2) use factorized approaches discussed in this section efficiently organize the data summary. If we use the size of the data summary (n) as the efficiency metric, factorized approaches are adversely affected as they need a much smaller storage budget to synthesize the same-sized data summaries. On the other hand, if we use "end-to-end bytes of storage" as the efficiency metric, non-factorized approaches are adversely affected as they perform no kind of data compression, but focus solcly on better understanding the model-to-data relationship through the lens of optimization. For a belter intuition, one can apply posthoc lossless compression (e.g., Huffman coding) on dala synthesized by non-factorized data distillation approaches to fit more images the same storage budget (Schirrmeister et al., 2022). Such tcchniques unintentionally deviate from the original intent of data distillation, and progress more toward better data compression techniques. As a potential solution, encourage the community to consider reporting results for both scenarios: a fixed data summary size n, well as fixed bytes-of-storage. Nonetheless, for the ease of empirical comparison amongst the discussed data distillation techniques, we provide a collated set of results over four image-llasiffiattoon datasets in Table 1.
3 Data Modalities
Having learned about different kinds of optimization frameworks for data distillation, orthogonal (and important) aspect of data distillation - what kinds of data can data distillation techniques summarize? From continuous-valued images to heterogeneous and semi-structured graphs, the underlying data for each unique application of machine learning has its own modality, structure, and set of assumptions. While the earliest data distillation techniques were designed to summarize images, recent steps have been taken to expand the horizon of data distillation into numerous other scenarios. In what follows, we categorize existing data distillation methods by their intended data modality, while also discussing their unique challenges.
discuss
Images. A large-portion of existing data distillation techniques are designed for image classification data (Cazenavette et al., 2022; Deng & Russakovsky, 2022; Kim et al., 2022; Lee et al,, 2022a;b; Liu et al., 2022c; Loo et al., 2022; Nguyen et al., 2021a;b; Wang et al., 2022; 2018; Zhao & Bilen, 2021; 2022; 2023; Zhao et al., 2021; Zhou et al., 2022b) simply because images have a real-valued, continuous data-domain (X' = Rdxd). This allows SGD-based imizationn directly on the data, which is treated as a set of free parameters. Intuitively, incrementally changing each pixel value can be treated as slight perturbations in the color space, and hence given a suitable data distillation loss, can be naïvely optimized using SGD.
Text, Textual data available in large amounts from sources like websites, news articles, academic manuscripls, elc., and is also readily accessible with datasets like the common crawl which sizes up to almost. 541TB. Furthermore, with the advent of large language models (LLM) (Brown et al., 2020; Devlin et al., 2019; Thoppilan et al., 2022), training such models from scratch on large datasets has become an increasingly
hhtpps//commmoncrawllor////// theer/tttaeaa////
10

---

Published in Transactions on Machine Learning Research (07 (02)))
(A,
Users
Items (Movies/Ad//Songa)
Data Distillation
BI I941 TOAT
EF
Fake Users
Items (Movies/Ads/Songs)
[Q Image Link] Figure Overview of distilling data for a few commonly observed data modalities.
expensive procedure. Despite recent efforts in democratizing LLM training (Geiping & Goldstein, 2022; Scao et al., 2022; Wolf et al., 2020), effectively distilling large-scale textual data solution is yet to be explored. The key bottlenecks for distilling textual data are: (1) the inherently discrete nature of data, where token should belong in a limited vocabulary of words; (2) the presence of a rich underlying structure, f.e., sentences of words (text) obey fixed patterns according to a grammar; and (3) richness of context, i.e., a given piece of text could have wildly different semanttcc interpretations under different contexts.
Sucholutsky & Schonlau (2021) take a latent-embeddiin approach textual data distillation. On a high level, to circumvent the discreteness of the optimization, the authors perform distillation in continuous embedding space. More specifically, assuming access to latent space specified by fixed text-encoder, the authors learn continuous representations of each word in the distilled text and optimize it using the TBPTT data-distillation framework proposed by Wang et al. (2018) (Equation (4)). Finally, the distilled text representations are decoded by following a simple nearest-neighhor protocol.
Graphs. A wide variety of data and applications can inherently be modeled as graphs, e.g., user-item interactions (Mittal et al., 2021; Sachdeva & McAuley, 2020; Wu et al., 2020), social networks (Fan et al., 2019), aulonomous driving (Casas et al., 2020; Sachdeva al., 2022b), elc. Taking the example of social nelworks, underlying user-user graphs typically size up to the billion-scale (Chen al., 2021), calling for principled scaling solutions. Graph distillation trivially solves a majority of the scale challenges, but synthesizing tiny, high-fidelity graphs has the following hurdles: (1) nodes in a graph products, elc. and could be discrete, heterogeneous, or even numerical IDs; (2) graphs follow variety of intrinsic patterns (e.g., spatial (Kipf & Welling, 2017)) which need to be retained in the distilled graphs; and (3) quadratic size of the adjacency matrix could be computationally prohibitive for data optimization.
be highly abstract, e.g., users,
Jin al. (2022b) propose GCOND which distills graphs in the inductive node-classification setting, specified by its nodc-feature matrix X, adjacency matrix A, and node-target matrix Y. GCOND distills the given graph by learning a synthetic node-feature matrix Xsyn, and using Xsyn to generate Asyn - fx(X)))) which can be realized, e.g., through parametric similarity function (im between the features of two nodes, i.e., A assimo syn: X)m)),, where a(-) is the sigmoid function. Finally, both X and 0 are optimized using the peadient-matchinn et al., 2022a) (GCDM) shares the same framework as GCOND but instead uses the distribution matching framework proposed by Zhao & Bilen (2023) (Equation (11)) to optimize Xsm and 0. Extending to a graph-clasiffcation setting, Jin et al. (2022a) further propose DOSCOND with two major changes compared to GCOND: (1) instead of parameterizing the adjacency matrix using similarity function X they maintain a free-parameter matrix Ω with the same size as the adjacency matrix, and sample each A entry through an independent Bernoulli draw on Di as the prior using the reparameterization trick (Maddison et al., 2017). Such procedure ensures differentiability well discrete matrix synthesis; and (2) X Xsyn and
framework proposed by Zhao et al. (2021) (Equation (7)). Another work (Liu
11

---

Published in Transactions on Machine Learning Research (07/2023)
Ω are still optimized using the gradient-matching framework (Equation (7)), albeit with only a single-step, L.e., T= for improved scalability and without empirically observing loss in performance.
Recommender Systems. The amount of online user-feedback data available for training recommender systems is rapidly increasing (Wu et al., 2022). Furthermore, typical user-facing recommender systems need to be periodically re-trained (Naumov et al., 2019), which adds to requirements for smarter data summarization solutions (see Sachdeva et. al. (2022c) for background on sampling recommender systems data). However, distilling recommender systems data has the following challenges: (1) the data is available in the form of abstract and discrcte (userID, itemID, relevance) tuples, which departs from the typical (features, label) setup; (2) the distribution of both user- and item-popularity follows a strong power-law which leads to data scarcity and unstable optimization; and (3) the data inherits a variety of inherent structures, e.g., sequential patterns (Kang& & McAuley, 2018; Sachdeva et al., 2019), user-item graph patterns (Wu et al., 2019), item-item co-occurrence patterns (Steck, 2019), misting-nottat--andoomm Schnabel et al., 2016), elc.
(Sachdeva et al., 2020;
Sachdeva et al. (2022a) propose DISTILL-CF which distills implicit-feeddback recommender systems data, i.e., when the observed user-item relevance is binary (e.g., click or no-click). Such data can be visualized as a binary user-item matrix R where each row represents single user, and each column represents an item. On a high-level, DISTILL-CF synthesizes fake users along with their item-consumption histories, visualized as synthetic user-ilem malrix Reyn. Notably, to preserve semantic meaning, the item-space in Rayn is the same as in R. To alleviate the data discreteness problem, DISTILL-CF maintains a sampling prrio matrix Ω which has the same size as Rayn, and can in-turn be used to generate Rayn using multi-stop Gumbel sampling with replacement (Jang et al., 2017) for each user's prior in Z (equivalent to each row). Such a formulation automatically also circumvents the dynamic user- and nem-pooulyrityy artifact in recommender systems data, which can analogously be controlled by the row- and column-wise entropy of Ω. Finally, 1 is optimized using the meta-model matching framework proposed by Nguyen et al. (2021a). Notably, Sachdeva et al. (2022a) also propose infinite-width autoencoders which suit the task of item recommendation while also leading to closed-form computation of the inner-loop in the meta- model matching framework (Equation (5)).
4 Applications
While the data distillation task was originally designed to accelerate model training, there are numerous other applications of high-fidelity data summary. Below we briefly discuss few such promising applications, along with providing pointers to existing works.
Differential Privacy. Data, distillation was recently shown to be a promising solution for differential privacy defined by Dwork (2008). Dong et al. (2022) show that data distillation techniques can perform better than existing slate-of-the-arl differeettallly-privaa data generators (Cao et al., 2021; Harder el al., 2021) on both performance and privacy grounds. Notably, the privacy benefits of data distillation techniques are virtually free, as none of these methods were optimized for genorating diffcrentially-privaee data. Chen et al. (2022) further modify the gradient matching framework (Equation (7)) by clipping and adding white noise to the gradients obtained on the original dataset while optimization. Such a routine was shown to have better sample utility, while also satisfying strict differential privacy guarantees. From a completely application perspective, data distillation has been used to effectively distill sensitive medical data well (Li et al., 2020a; 2022).
Neural Architecture Search (NAS). Automatic searching of neural-network architectures can alleviate majority of manual effort, well as lead to more accurate models (see Elsken et al. (2019) for a detailed review). Analogous to using model extrapolation, i.e., extrapolating the performance of an under-trained model architecture on the full datasct; data extrapolation, on the other hand, aims to train models on small, high-fidelity data sample till convergence. Zhao el al. (2021) show promise of their technique (DC) on a small custom NAS test-bed consisting of only 720 variations of the ConvNet architecture (Gidaris & Komodakis, 2018) by employing the data extrapolation framework. However, Cui et al. (2022a) show that data distillation does not perform well when evaluating diverse architectures on the bigger test-bed, NAS-Bench- (Dong & Yang, 2020), calling for better rank-preserving data distillation techniques.
12

---

Published in Transactions on Machine Learning Research (07/2023)
Continual Learning. Never-ending learning (sce Parisi et al. (2019) for detailed review) has been frequently associated with catastrophic forgetting (French, 1999), Lee, patterns extracted from old fata/tasks are easily forgotten when patterns from new data/tasks are learned. Data distillation has been shown as an effective solution to alleviate catastrophic forgetting, by simply using the distilled data summary in a replay busfer that is continually updated and used in subsequent data/ training (Rosasco et al., 2021; Sangermano et al., 2022; Wiewel & Yang, 2021). Deng & Russakovsky (2022) show further evidence of a simple cesppres--thnn-eeaee strategy outperforming existing state-of-the-art continual learning approaches. Notably, only the data summary is stored for each task, and new model is trained (from scratch) using all previous data summaries, for each new incoming task.
Federated Learning. Federated or collaborative learning (see Li et al. (2020b) for a detailed survey) involves training a learning algorithm in a decentralized fashion. standard approach to federated learning is to synchronize local parameter updates to a central server, instead of synchronizing the raw data itself (Konečný et al., 2016). Data distillation, on the other hand, alleviates the need synchronize large paramet ric models across clients and servers, by synchronizing tiny synthesized data summaries to the central server instead. Subsequently, the entire training happens only on the central server. Such data distillation-base federated learning methods (Goetz & Tewari, 2020; Hu et al., 2022; Liu et al., 2022b; Song et al., 2022; Xiong et al., 2022; Zhou et al., 2020) are shown to perform better than modelssynchronization based federated learning approaches, while also requiring multiple orders lesser client-server communication.
5 Challenges & Future Directions
Despite achicving remarkable progress in data-efficient learning, there are numerous framework-based, thcoret- ical, and application-basel directions yet to be explored in data distillation. In what follows, we highlight and discuss such directions for the community to further explore, based either on early evidence or our intuition.
New data modalities. Extending the discussion in Section 3, existing data distillation techniques have largely been restricted to cater to image datasets, primarily due to the amenable data-optimization in the continuous pixel-domain of images. Despite recent efforts in increasing the horizon of data distillation to other data modalitics such as graphs (Jin et al., 2022a;b) and recommender systems (Sachdeva et al., 2022a); each data modality poses ils unique challenges and calls for future work, e.g., handling long sequences of time-series data in audio-clasiification (Hershey et al., 2017), video classification (Karpathy et al., 2014), self-driving (Sun et al., 2020); millions of categorical features in tabular data (Wang et al., 2021); sparse and noisy financial data (Xu & Cohen, 2018); etc.
New predictive tasks. Another limitation of existing data distillation techniques is that their underlying optimization is primarily designed for classification scenarios. However, a large number of predictive tasks fail to naïvely fit. into the existing supervised data distillation framework, e.g-, image-generation (Ramesh et al., 2022; Rombach et al., 2022), language modcling (Brown et al., 2020; Devlin et al., 2019; Touvron et al., 2023), representation learning (Chen et al., 2020; Grill et al., 2020), elc. Further, the aforementioned tasks have gained immense popularity and have seen widespread practical use in the recent years, calling for future work in developing data distillation techniques for more predictive tasks.
Better scaling. Existing data distillation techniques validate their prowess only in the super low-data regime (typically 1 50 data points per class) due to (i) computational difficulties in synthesizing large data summaries with existing techniques; and (ii) collapse to the random-sane samplinn bascline when syuthesizing large data summaries, as noted by Cui al. (2022a). This calls for future work from both directions developing efficient data distillation techniques that are scalable to web-scale datasets, and deeper investigations of the cause and potential fixes of the observed scaling artifacts of existing techniques.
Improved optimization. A unifying thread across data distillation techniques is an underlying bilevel optimization, which provably NP-hard even in the lincar inner-optimization case (Vicente et al., 1994). Notably, bilevel optimization has been successfully applied in a variety of other applications like meta-learning (Finn et al., 2017; Li et al., 2017), hyper-parameter optimization (Lorraine et al., 2020; Maclaurin et al.,
13

---

Published in Transactions on Machine Learning Research (07/2023)
2015), neural architecture search (Liu et al. 2019), coreset construction (Borsos et al., 2020; Zhou et al., 2022a), elc. Despite its success, many ihelretical underpinnings are yet. to be explored, e-g,, the effect of commonly-used singleton solution assumption (Franceschi et al., 2018), the effect of over-parammttriizaoon on bilovel optimization (Vicol et al., 2022), connections to statistical influence functions (Bac et al., 2022), the bias-variance tradeoff (Vicol et al., 2021), elc. Clearly, an overall better understanding of bilevel optimization will directly enable the development of better data distillation techniques.
Improved data-quality evaluation. As briefly discussed in Section 2, data synthesized using data distillation is evaluated from performance, efficiency, and transfcrability standpoints. However, numerous high-stakes use-cases call for being able to train robust models from variety of angles such fairness (Mehrabi et al., 2021), adversarial robustness (Madry et al., 2018), etc. Hence, synthesizing data summaries able to support such robust model training is practical and an important direction for future work. Notably, while popular metrics exist for evaluating the robustness of learning algorithms from the aforementioned standpoints, developing such notions at the dataset-level is non-trivial, and with little existing literalure (Ben-Eliezes & Yogev, 2020; Celis al., 2018).
Acknowledgments
We sincercly thank Zhiwci Dong, Bo Zhao, and George Cazenavette for their feodback on carly drafts of this survey.
References
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semdedup: Data-efficient learning at web-scale through semantic deduplication. orXiv preprint arXiv:2303.095: 2023.
Sunil Arya, David M Mount, Nathan S Netanyahu, Ruth Silverman, and Angela Y Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. Journal of the ACM (JACM), 45(6):891-923, 1998.
Fadhel Ayed and Soufiane Hayou. Data pruning and neural scaling laws: fundamental limitations of score-based algorithms. arXi preprint or Xiv:22302. 06960, 2023.
Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. er preprint Xiv:::1703. 06476, 2017.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger Grosse. If influence functions are the answer, then what the question? Xiv preprint orXiv:2209.0553 05364, 2022.
Omri Ben-Eliczer and Eylon Yogev. The adversarial robustness of sampling. In Proceedings of the 39th ACM STCMOD-STCACTT-STCA Symposium on Principles of Database Systems, pp. 49 62, 2020.
Jeff Bilmes.
Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.0011 2022.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint ar wwXi:200660885 2020.
Zalán Borsos, Mojuir Mutny, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming, Advances Neurul Information Processing Systems, 33:14879 14890, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 2021.
Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urlasun. Spagnu: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9491-99999 9497. TEEE, 2020.
14

---

Published in Transactions on Machine Learning Research (07/2023)
George Cazenavetle, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 4750-4759, 2022.
Elisa Celis, Vijay Keswani, Damian Straszak, Amit Deshpande, Tarun Kathuria, and Nisheeth Vishnoi. Fair and diverse dpp-based data summarization. In International Conference on Machine Learning, pp. 716-725. PMLR, 2018.
Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private set generation with discriminative information. In Proceedings of the Advonces in Neural Information Processing Systems (NewrIPS), 2022.
Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. Adeances in Newral Information Processing Systems, 34:5199 5212, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597 1607. PMLR, 2020.
Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In Infernational Conference on Learning Representations, 2020.
Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsich. DC-BENCH: Dataset condensation benchmark. In shirty-sizth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022a.
Justin Cui, Ruochen Wang, Si, and Cho-Jui Hsich. Scaling up dataset distillation to imagenet- Ik with constant memory. arXiv preprint Xrxi::2211 10586, 2022b.
Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. In Advances in Newrul Information Processing Systems, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapler of the Association for Compulational Linguisties: Human Language Technologies, Volume 1 (Leng und Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European conference on compuler vision, pp. 391 407. Springer, 2016.
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2020.
Cynthia Dwork. Differential privacy: A survey of results. In International conference theory and applications of models of computation, pp. 1-19. Springer, 2008
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-200 2019.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendatton. In The world wide web conference, pp. 417-426, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic leaannea for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and leaanng In International Conference on Machine Learning, pp. 1568-1577. PMLR, 2018.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128 1(4):128-135, 1999.
Jonas Geiping and Tom Goldstein. Cramming: Training language model a single gpu in one day, 2022.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Amirala Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Leurning, pp. 2242 2251. PMLR, 2019.
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.00 109.0740,, 2021.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference computer vision and pottern recognation, pp. 1367 4375, 2018.
Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint ar 2020.
lan Goodfellow, Jean Ponget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Jean- Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 -22224, 2020.
Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I, pp. 181-195. Springer, 2022.
Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, and Carole-Jean Wu. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro, 42(4):37-47, 2022.
Frederik Harder, Kamil Adamczewski, and Mijung Park. Dp-merf: Differentially private mean embeddings with randomfeatures for practical privacy-preserving data generation. In International conference on artificial intelligence and statistics, pp. 1819 1827. PMLR, 2021.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cun architectures for large-scale audio classification. 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 131 135. IEEE, 2017.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint TrXii:1503.02533 2(7), 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compate-optimal large language models. Xrxie preprint arXiv:2209. 15556, 2022.
Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. Fedsynth: Gradient compression via synthetic data federated learning. arXio preprint ari::22204.013 012273, 2022.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural langent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proccedings. OpenReview.ng net, 2017.
Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, and Bing Yin. Condensing graphs via one-step gradient matching. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 720 730, 2022a.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In International Conference on Learning Representations, 2022b.
Wang- Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE international conference on data mining (ICDM), pp. 197-206. IEFE, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXio preprint arXio:2001.08300 2020.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Andrej Karpathy, George Toderici, Sanketh Shetly, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Compater Vision and Pollern Recograition, pp.
2014.
Krishnateja Killamsetly, Durga S, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 5464-5474. PMLR, 18-24 Jul 2021.
Jang-Hyun Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Thomas N. Kipf and Max Welling. Semi-Superviaed Classification with Graph Convolutional Networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR '17, 2017.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint Xri:: 110.0220, 2016.
Yann LeCum, John Denker, and Sara Solla. Optimal brain damage. Advonces in neural information processing systems, 2, 1989.
Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing, Xrv preprint (tto:2288.1449, 2022a.
Jachoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl- Dickstein. Finite versus infinite neural networks: an empirical study, Advances in Neural Information Processing Systems, 33:15156 15172, 2020.
Sachyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In Proceedings of the 39th International Conference on Machine Learning, pp- 12364, 2022b.
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP), pp- 305-309. IEEE, 2020a.
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Compressed gastric image generation based on soft-labe dataset distillation for medical data sharing. Computer Methods and Programs in Biomedicine, pp. 107189, 2022.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3)500-60, 2020b.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning, arXiv preprint Xiv: 09835, 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Mengyang Liu, Shanchuan Li, Xinshi Chen, and Le Song. Graph condensation via receptive field distribution matching. arXiv preprint ar arXiv:2206.19699 2022a.
Ping Liu, Xin Yu, and Joey Tianyi Zhou. Meta knowledge condensation for federated learning. arXiu preprint arxiv:2209. 14851, 2022b.
Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. NewrIPS, 2022c.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. In Advances in Newral Information Processing Systems, 2022.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp- 1540 1552. PMLR, 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- baseed hyperparameter optimization through reversible learning. In International conference on machine learning, pp. 2113-2122 PMLR, 2015.
17

---

Published in Transactions on Machine Learning Research (07/2023)
Chris J. Maddison, Andriy Muih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Leurning Representations, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Geoffrey McLachlan and Thriyambakam Krishnan. The EM elgorithm and extensions. John Wiley & Sons, 2007.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey bias and fairness in machine learning. ACM Computing Survegs (CSUR), 51(6):1 35, 2021.
Luke Metz, Niru Mabeswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sick-Dic Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556 4565. PMLR, 2019.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960. PMLR, 2020.
Anshul Mittal, Noveen Sachdeva, Sheshansh Agrawal, Sumeet Agarwal, Purushottam Kar, and Manik Varma. Eclare: Extreme classification with label graph correlations. In Proceedings of the Web Conference 2021, WWW 221 2021.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs/1906.0009// 00099,, 2019.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021a.
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Tnformation Processing Systems, 34, 2021b.
German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: review. Newral Networks, 11:55471,, 2019.
Lorien Y Pratt. Discriminablitty-aaer transfer between neural networks. Advances in neural information processing systems, 5, 1992.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Lext-conditional image generation with clip latents. arXiv preprind arXiv:220,.0611 061125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Andrea Rosasco, Antonio Carla, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through syuthetic samples. arXie preprint arXie:2103. 15851, 2021.
Durga S, Rishabh Iyer, Ganesh Ramakrishnan, and Abir De. Training data subset selection for regression with controlled generalization error. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Muclene Learning, volume 139 of Proceedings of Muchine Learning Research, pp. 9202 92122 PMLR, 18-24 24 Jul 2021.
Noveen Sachdeva and Julian McAuley. How Useful Are Reviews for Recommendation? A Critical Review and Potential Improvements, pp. 1845-1848, SIGIR. '20. Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781450380164. doi: 10.1145 11445//3397771.34412
Noveen Sachdeva, Giuseppe Manco, Ettore Ritacco, and Vikram Pudi. Sequential variational autoencoders for collaborative filtering, In Proceedings of the twelfth ACM international conference on web search and data mining, pp. 2000608. 2019.
18

---

Published in Transactions on Machine Learning Research (07/2023)
Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledgee Discovery & Dala Mining, KDD '20, pp. 965 965-975, New York, NY, USA, 2020. Association for Computing Machinery. doi:
Noveen Sachdeva, Mehak Preet Dhaliwal, Carole-Jean Wu, and Julian McAuley. Infinite recommendation networks: A data-centric approach. In Advances in Newral Information Processing Systems, 2022a.
Noveen Sachdeva, Ziran Wang, Kyungtae Han, Rohit Gupta, and Julian McAuley. Gapformer: Fast autoregressive transformers meet rmns for personalized adaptive cruise control. In 2022 IEEE 25th International Conference on Intelligent Trunsportation Systems (ITSC), pp. 2528 2535, 2022b. doi: 10. 1411109/TTC5511/0 1D..C/5511.22222999999 2022.992227775
Noveen Sachdeva, Carole-Jean Wu, and Julian McAuley. On sampling collaborative filtering datasets. In Proczedings of the Fiffeenth ACM International Conference on Web Search and Data Mining, WSDM '22 2022c.
Mattia Sangermano, Antonio Carla, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. In 2022 International Joinl Conference on Newrul Networks (IJCNN), pp. 01- 01-08. IEEE, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexan- dra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXio preprinl Xr:::22 1.05100. 2022.
Robin Tibor Schirmmeister, Rosanne Liu, Sara Hooker, and Tonio Ball. When less is more: Simplifying imputs aids neural network understanding. arviv preprint or 2rv:: 0056110, 2022.
Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as trealments: Debiasing learning and evaluation. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Resenrch, pp. 1670-1679. PMT.R, 2016.
Rui Song, Dai Liu, Dave Zhenyu Chen, Andreas Festag, Carsten Trinitis, Martin Schulz, and Alois Knoll. Fed- erated learning via decentralized dataset distillation in resource-constrained edge environments. arXiv preprint 2208 1 18111, 2022.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advonces in Neural Information Processing Systems, 2022.
Harald Steck. Embarrassingl shallow autoencoders for sparse data. In The World Wide Web Conference, 2019.
Ilia Sucholutsky and Matthias Schonlan. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEFE, 2021.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognaliin, pp. 2446 2454, 2020.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- The Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXie preprint arXix:2201.088 008239, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations: 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bapliste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint ar Xiv:23302 18971, 2023.
Luis Vicente, Gilles Savard, and Joaquim Júdice. Descent approaches for quadratic bilevel programming. Journal of Optimization theory and applications, 81 1994.
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein. Unbiased gradient estimation in unrolled computation graphs with persistent evolution strategies. In International Conference on Machine Learning, pp. 10553 10563. PMLR, 2021.
Paul Vicol, Jomathan P Lorraine, Fabian Pedregosa, David Duvenaud, and Roger B Grosse. On implicit bias in oxerparameterized bilevel optimization. In International Conference Machine Learning. PMLR, 2022.
19

---

Published in Transactions on Machine Learning Research (07 2273))
Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shno Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cale: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CV Conference Computer Vision and Pallern Recognition, pp. 12196 12196-12205 2022.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, pp. 1785-1797, 2021.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXi::111.10955, 2018.
Max Welling, Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, 2009.
Felix Wiewel and Bin Yang. Condensed composite memorry continual learning. 2021 International Joinl Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernile, Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empérical Methods in Natural Longuage Processing: System Demonstrations, pp. 38 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v 10////11/22000 22000.emmlp-dmmos.f.. 6.
D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization. IEEE Transactions Evolutionary Compulation, 1(1):67 82, 1997. doi: 1LL109944335558890
Carole-Jean Wu, Ramya Raghavendra, Ldit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, al. Sustainable ai: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795 813, 2022.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey. ACM Computing Surveys (CSUR), 2020.
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-based recommendation with graph neural networks. In Proceedings of the AAAT conference on artificial àntelligence, 2019.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse, Understanding short-horizon bias in stochastic meta- optimization. In International Conference on Learning Representations, 2018
Yuanhao Xiong, Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsich. Feddm: Iterative distribution matching for communictiion-eficien federated learning, arXiv preprint or Xiv:2200 09653,, 2022.
Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1970-1979, 2018
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning, pp- 12674-12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. arXiv preprint NXiu:2204.07510 2022.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proccedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and Tong Zhang. Probabilistic bilevel coreset selection. In International Conference on Machinee Learning, pp. 27287 27302. PMLR, 2022a.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arxiv preprint 2al::000007999 2020.
Yongchao Zhou, Ehsan Nezhadarya, and Jimmyy Ba. Dataset distillation using neural feature regression. In Advances in Neural Information Processing Systems, 2022h.
20

---

Published in Transactions on Machine Learning Research (07 (023))
A
Notation
Dataset related
D _ (1x
X,
))
The target datasct to be distilled
X
Data domain
y
Predictand domain
C
Set of unique classes in y
D - {(x;))) yi) 8/ c)
Portion of DD with class c
X [0,1/21
Matrix of all features in D
Y A [
|P]
Matrix of all predictands in DD
7
Size of data summary
Day A {(ã, 81)
Data summary
Diyn - {60, 9i) 0i c)
Portion of Dsyn with class c
Ssyn |
Matrix of all features in Dayn
Y syn A [ū
Matrix of all predictands in Dayn
Learning related
To 10: X
Learning algorithm parameterized by Twiie-diiffereniabll cost function
1: x y + R
(D) 2 (( d ((
(20(()).))
Expected loss of Φ on D Expected loss of on Dayn
(D) A (2 ((0p(z)))))
General
dim(A)
Size of basis of A
[A]
Number of elements in
sup
Supremum
arg min f())
Optimum value of which minimizes f(0)
[[(())] p(z)) f(x
Expected value of (()) when domain of is discrete
21


Reviews:
[{"review_id": "V6CUiUPBuS", "reviewer": "Reviewer_pzQW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The author proposed a unified framework for data distillation and surveyed recent algorithms. In addition, the data distillation methods are compared in Table 1.\n- The author not only discussed the algorithms, but also listed the applications that can benefit from data distillation.\n\nWeaknesses\n- The author has not cited other surveys about data distillation, including \"A Survey on Dataset Distillation: Approaches, Applications and Future Directions\" which has been accepted by IJCAI 2023, and \"A Comprehensive Survey of Dataset Distillation\" (https://arxiv.org/pdf/2301.05603.pdf). In fact, the paper is quite similar to the IJCAI2023 survey paper For example, the IJCAI2023 survey paper also discussed model matching, gradient matching, trajectory matching, and distribution matching algorithms. One exception is that the previous survey paper has not discussed factorization-based data distillation methods. In addition, the IJCAI 2023 paper surveyed different data modalities and has included the audio modality, which was not included in the paper under review. The IJCAI 2023 paper also discussed different applications.\n- The author compared data distillation with data compression in Section 2.5. I think the author may also compare data distillation with data pruning, especially discuss the relationship with recent paper like \"Beyond neural scaling laws: beating power law scaling via data pruning\".\n- If we check the data modality section, we may notice that most problems are still classification problems. Applying data distillation to other tasks like language modeling or representation learning should also be a reasonable direction and the author may like to include it the survey.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveyed methods, data modalities and applications of a newly emerging machine learning area called data distillation. The goal of data distillation is to synthesize small-scale data summaries from a large-scale dataset, which can act as drop-in replacement of the original dataset in training or inference scenarios. The author first proposed the concept of \\epsilon-approximate data summary and defined data distillation as an optimization problem. Existing data distillation algorithms can be encapsulated in the framework as different ways of solving the optimization problem. The author grouped data distillation algorithms into five categories in the survey: 1) model matching, 2) gradient matching, 3) trajectory matching, 4) distribution matching, and 5) factorization. Afterwards, the author discussed the modalities that existing algorithms have been applied on, including image, text, graph and recommender systems. The author also introduced the applications that can benefit from the high-fidelity data summary, including differential privacy, neural architecture search, federated learning. In the end, the author briefly discussed the challenges in this area.\n\nThe paper is well-written, the unified framework is neat and the survey has included most of the recent publications.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The author needs to clarify the relationship / difference of the paper with \"A Survey on Dataset Distillation: Approaches, Applications and Future Directions\".\n2. The author should discuss the relationship between data distillation and data pruning\n3. Most data distillation algorithms are still compared on classification datasets. The author should clarify such limitation and point out potential directions to expand data distillation to more tasks.\n", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BWY7Rqc83N", "reviewer": "Reviewer_hCkx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The topic discussed in the paper is highly significant, especially in the context of the prevalence of large-scale models today. With the escalating costs associated with model training, utilizing data distillation to expedite both training and inference processes is very important. This review has the potential to advance this objective.\n\n2. The paper provides a concise and accurate summary of the existing five data distillation methods. It includes a comprehensive summary of primary references and offers method descriptions that are not only mathematically rigorous but also provide sufficient details.\n\n3. The future directions outlined in Section 5 are valuable references for subsequent research in the field.\n\n4. The paper is well-organized and written-well.\n\n\nWeaknesses:\n1. It seems that the paper only focused on labeled datasets. However, regarding tasks without annotations, such as unconditioned image synthesis, are there any works related to data distillation?\n\n2. Currently, large-scale models and datasets are highly popular. Could the author summarize if there are any data distillation methods specifically designed for large datasets?\n\n3. On Page 3, below Definition 3, the paper highlights three key criteria for evaluating data distillation methods: performance, efficiency, and transferability. I'm curious if robustness should be considered as a fourth criterion. For instance, when training with a distilled dataset, is the model more vonerable to adversarial attack methods?\n\n4. An important role of data distillation is to boost training. Could the author provide a summary of the existing methods and their impact on training speed improvement?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a review on Data Distillation, which aims to reduce the size of datasets of various learning tasks and synthesize terse data summaries that can be used as alternatives for modeling training/inference. The paper begins by presenting the concept and rationale behind data distillation. Subsequently, it delves into a thorough examination of five primary categories of data distillation methods. Furthermore, the paper explores the modality and application of data distillation, providing valuable insights. Lastly, it outlines several prospective directions for future research in this field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please attempt to address the concerns within the \"Weakness\" section.", "broader_impact_concerns": "I have no concerns on the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f3l11NgoIi", "reviewer": "Reviewer_MxvN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper's classification of different methods based on definitions, algorithms, and data modalities is clear and systematic, making it a valuable survey.\n- The paper's categorization of applications, challenges, and future directions is also clear and insightful.\n\n**Weaknesses:**\n\n- The paper could benefit from a taxonomy figure summarizing existing methods, which would make it easier for readers to locate and compare different studies.\n- It would be helpful if the paper indicated whether there is related work on surveys of data distillation.\n- The results presented are limited to image modalities, and it would be beneficial if the paper discussed other modalities.\n- The paper would benefit from a more detailed algorithm explaining the overall flow of the most basic model.\n- In the applications section, it would be helpful to define and quantify the results in each field. For example, for NAS, it would be useful to mention the scale of small and large NAS test-beds, whether they refer to the number of models in the search space, model size, or exploration time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contribution of this survey paper is to present a formal framework for data distillation and provide a detailed taxonomy of existing approaches. It also covers data distillation approaches for various data modalities, identifies current challenges, and proposes future research directions. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A taxonomy figure summarizing existing methods would be useful for readers to locate and compare different studies.\n- The paper could indicate whether there is related work on data distillation.\n- The results presented are limited to image modalities, and it would be beneficial if the paper discussed other modalities.\n- The paper could benefit from a more detailed algorithm explaining the overall flow of the most basic model.\nIn the applications section, it would be helpful to define and quantify the results in each field. For example, for NAS, it would be useful to mention the scale of small and large NAS test-beds, whether they refer to the number of models in the search space, model size, or exploration time.", "broader_impact_concerns": "This paper does not discuss any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper surveys methods, data modalities and applications of a newly emerging machine learning area called data distillation. The author first proposed the concept of epsilon-approximate data summary and defined data distillation as an optimization problem. The paper is well-written, the unified framework is neat and the survey has included most of the recent publications.

Based on three qualified reviews, this paper can be accepted and the authors are encouraged to merge the comments into their update versions.
Decision: Accept (poster)