Dataset Condensation with Gradient Matching
Authors: Bo Zhao
Konda Reddy Mopuri
Hakan Bilen
Venue: ICLR 2021 Oral (2021)
Keywords: dataset condensation
data-efficient learning
image generation

Abstract: As the state-of-the-art machine learning methods in many fields rely on larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes a training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into a small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained on the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods. Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.

Published as a conference paper at ICLR 2021
DATASET CONDENSATION WITH GRADIENT MATCHING
Bo Zhao, Konda Reddy Mopuri, Hakan Bilen School of Informatics, The University of Edinburgh {bo. zhao, kmopuri, hbilen @ed. @eddea...... ac. uk
ABSTRACT
As the state-of-the-art machine learning methods in many fields rely larger datasets, storing datasets and training models on them become significantly more expensive. This paper proposes training set synthesis technique for data-efficient learning, called Dataset Condensation, that learns to condense large dataset into small set of informative synthetic samples for training deep neural networks from scratch. We formulate this goal as a gradient matching problem between the gradients of deep neural network weights that are trained the original and our synthetic data. We rigorously evaluate its performance in several computer vision benchmarks and demonstrate that it significantly outperforms the state-of-the-art methods" Finally we explore the use of our method in continual learning and neural architecture search and report promising gains when limited memory and computations are available.
INTRODUCTION
Large-scale datasets, comprising millions of samples, are becoming the norm to obtain state-of- the-art machine learning models in multiple fields including computer vision, natural language pro- cessing and speech recognition. At such scales, even storing and preprocessing the data becomes burdensome, and training machine learning models on them demands for specialized equipment and infrastructure. An effective way to deal with large data is data selection identifying the most repre- sentative training samples - that aims at improving data efficiency machine learning techniques. While classical data selection methods, also known coreset construction (Agarwal et al.) 2004 Har-Peled & Mazumdar) 2004 Feldman et al.] 2013). focus on clustering problems, recent work can be found in continual learning (Rebufh et al. [017]] Toneva al. 2019 Castro et al.) 2018 Allundi et al. 2019) and active learning (Sener & Savarese 2018) where there is typically a fixed budget in storing and labeling training samples respectively. These methods commonly first define a criterion for representativeness (e.g. in terms of compactness (Rebuffi et al.] 2017: Castro et al.,
2018). diversity (Sener& & Savarese 2018) Aljundi et al., 2019)). forgetfulness (Toneva al. (219)). ) then select the representative samples based on the criterion, finally use the selected small set to train their model for a downstream task.
Unfortunately, these methods have two shortcomings: they typically rely on i) heuristics (e.g. pick- ing cluster centers) that does not guarantee any optimal solution for the downstream task (e.g. image classification), ii) presence of representative samples, which is neither guaranteed. A recent method, Dataset Distillation (DD) (Wang al., 2018)) goes beyond these limitations by learning small set of informative images from large training data. In particular, the authors model the network param- eters as function of the synthetic training data and learn them by minimizing the training loss over the original training data w.r.t. synthetic data. Unlike in the coreset methods, the synthesized data are directly optimized for the downstream task and thus the success of the method does not rely on the presence of representative samples.
Inspired from DD WWang et al., 2018), we focus on learning to synthesize informative samples that optimized to train neural networks for downstream tasks and not limited to individual samples in original dataset. Like DD, our goal is to obtain the highest generalization performance with model trained on small set of synthetic images, ideally comparable performance to that of model trained on the original images (see Figure (a)). In particular, we investigate the following
The implementation available at http://///ttttiaa..o./oooo/aaaaa

---

Published as a conference paper at ICLR 2021
(a] (b)
train
Test
CE
Large training set
Large training set
Comparable
Matching Loss
pass
Update synthetic
+
Backpropagation
CE
Small synthetic set
Small syntheti sat
Figure 1: Dataset Condensation (left) aims to generate small set of synthetic images that can match the performance of network trained on large image dataset. Our method (right) realizes this goal by learning a synthetic set such that a deep network trained on it and the large set produces similar gradients w.r.t its weights. The synthetic data can later be used to train a network from scratch in a small fraction of the original computational load. CE denotes Cross-Entropy.
questions. Is it possible i) compress a large image classification dataset into small synthetic set, ii) train an image classification model on the synthetic set that can be further used to classify real images, iii) learn a single set of synthetic images that can be used train different neural network architectures? To this end, we propose a Dataset Condensation method to learn small set of "coodensedd" synthetic samples such that a deep neural network trained on them obtains not only similar performance but also close solution to network trained on the large training data in the network parameter space. We formulate this goal as a minimization problem between two sets of gradients of the network parameters that are computed for a training loss over large fixed training set and a learnable condensed set (see Figure [b). We show that our method enables effective learning of synthetic images and neural networks trained them, outperforms (Wang et al. 2018) and coreset methods with a wide margin in multiple computer vision benchmarks. addition, learning a compact set of synthetic samples also benefits other learning problems when there is a fixed budget on training images. We show that our method outperforms popular data selection methods by providing more informative training samples in continual learning. Finally, we explore a promising use case of our method in neural architecture search, and show that once our condensed images are learned they can be used to train numerous network architectures extremely efficiently.
Our method is related to knowledge distillation (KD) techniques (Hinton et al. 2015 Buciluà et al., 2006 Ba & Caruana 2014 Romero et al., 2014) that transfer the knowledge in an ensemble of models to single one. Unlike KD, we distill knowledge of large training set into small synthetic set. Our method is also related to Generative Adversarial Networks (Goodfellow et al. [2014a Mirza
& Osindero 2014 Radford et al.) 2015) and Variational AutoEncoders (Kingma & Welling 2013)
that synthesize high-fidelity samples by capturing the data distribution. In contrast, our goal is to generate informative samples for training deep neural networks rather than to produce "real-looking" samples. Finally our method is related to the methods that produce image patches by projecting the feature activations back to the input pixel space (Zeiler & Fergus 2014). reconstruct the input image by matching the feature activations (Mahendran & Vedaldi 2015). recover private training images for given training gradients (Zhu et al. 20199) Zhao al.] 2020), synthesize features from semantic embeddings for zero-shot learning (Sariyildiz & Cinbis 2019). Our goal is however to synthesize a set of condensed training images
recover the original or missing training images.
In the remainder of this paper, we first review the problem of dataset condensation and introduce our method in section present and analyze our results in several image recognition benchmarks in sec- tion 3.1 showcase applications in continual learning and network architecture search in section 3.2 and conclude the paper with remarks for future directions section 4
METHOD
2.1 DATASET CONDENSATION
Suppose are given large dataset consisting of IT pairs of a training image and its class label T= [xx, y:)) [=1 where x E C Rd, y E (o...., C- - 1}. X is a d-dimensional input space and C is the number of classes. We wish learn differentiable function & (i.e. deep neural network)
2

---

Published as a conference paper at ICLR 2021
with parameters 0 that correctly predicts labels of previously unseen images, i.e. y (0(x). One can learn the parameters of this function by minimizing an empirical loss term over the training set:
o arg min L (0)
(1)
0
where (T = 17| Etayer (2) v) e(ss) is task specific loss (i.e. cross-entropy) and oT is the minimizer of L The generalization performance of the obtained model ΦØT can be written as Pe>Pp|<(@0> (z)).) where PD the data distribution. Our goal is to generate a small set of condensed symthetic samples with their labels. S = {(s,, W where 8 E R and E V, |S] < [T]. Similar to eq. ① once the condensed set learned, one can train ø on them as follows
@ = arg min L (0)
(2)
where (0) E Eeayes ((00(s) v) and 0 is the minimizer of LS. As the synthetic set S is significantly smaller (2-3 orders of magnitude), we expect the optimization in eq. (2) to be significantly faster than that in eq. ① We also wish the generalization performance of Φgs to be close to Φ0T i.e. P [c(por (z)) v) Ex- ((oggs (az). y)] over the real data distribution Po.
Discussion. The goal of obtaining comparable generalization performance by training the con- densed data can be formulated in different ways. One approach, which is proposed in (Wang et al.] 2018) and extended in (Sucholutsky & Schonlau [2099] [Bohdal et al. 20200 Such et al. 2020 is to pose the parameters 0 as a function of the synthetic data S:
S = arg min L (0$ (S))
subject to
0 (S) min (²(())
(3)
s
0
The method aims to find the optimum set of synthetic images S such that the model óag trained on them minimizes the training loss over the original data. Optimizing eq. involves nested loop optimization and solving the inner loop for 05( (S) at each iteration to recover the gradients for S which requires a computationally expensive procedure unrolling the recursive computation graph for S over multiple optimization steps for 0 (see (Samuel & Tappen| [2009] Domke (0112). Hence, it does not scale to large models and/or accurate inner-lopp optimizers with many steps. Next we propose an alternative formulation for dataset condensation.
2.2 DATASET CONDENSATION WITH PARAMETER MATCHINN
Here we aim to learn S such that the model &gs trained on them achieves not only comparable generalization performance to ΦΦT but also converges to a similar solution in the parameter space (i.e. gs Rs 07)) ). Let Φe be a locally smooth functior similar weights (0S Rs 0²) imply similar mappings in local neighborhood and thus generalization performance, iee Po (C(oor (m), v)) =
Ex~Po [((dos (x), v)]. Now we can formulate this goal as
min D OT) subject to
(*(S) arg min CS
(4)
0
where 0J arg ming (T()) and D(..-) distance function. In deep neural network, @T typically depends on its initial values Og- However, the optimization in eq. 4 aims to obtain an optimum set of synthetic images only for one model deg with the initialization 6g. while our actual goal is to generate samples that can work with distribution of random initializations Pe.- Thus we modify eq. as follows:
min Fag (D(O((0)) OT (0o))) subject to 0$( = arg arggin min L3 (0(0o) S
(5)
where 0 arg ming LT (O)O)) For brevity,
only 0 and @ to indicate (0 and
@ @ (0n) respectively in the next sections. The standard approach to solving eq. (5) employs implicit differentiation (see (Domke 2012) for details), which involves solving an inner loop optimization for gs As the inner loop optimization (S) = arg ming L can be computationally expensive in
Local smoothness is frequently used to obtain explicit first-order local approximations in deep networks (e.g. see (Rifai al. 2012 Goodfellow et al. 2014b Koh & Liang 2017)..
3

---

Published as a conference paper at ICLR 2021
case of large-scale models, adopt the back-optimization approach in (Dokee 2012) which
re-defines 0 as the output of an incomplete optimization:
0 (S) = opt- apt--alg(C (0),s)
(6)
where opt 1 g specific optimization procedure with fixed number of steps (s).
In practice, H for different initializations can be trained first in an offline stage and then used as the target parameter vector in eq. (S. However, there two potential issues by learning to regress OJ as the target vector. First the distance between 0 and intermediate values of 0 can be too big in the parameter space with multiple local minima traps along the path and thus it can be too challenging to reach. Second opt apt-alg involves a limited number of optimization steps as a trade- off between speed and accuracy which may not be sufficient take enough steps for reaching the optimal solution. These problems are similar to those of (Wang et al. 2018), as they both involve parameterizing A with S and to.
2.3 DATASET CONDENSATION WITH CURRICULUM GRADIENT MATCHING
Here we propose a curriculum based approach address the above mentioned challenges. The key idea is that we wish 0 to be close to not only the final 0 but also to follow a similar path to 0² throughout the optimization. While this can restrict the optimization dynamics for 0, we argue that it also enables more guided optimization and effective use of the incomplete optimizer. We can now decompose eq. (5) into multiple subproblems:
Σ D 0g) subject to t=
min Eg, Poo
(5) opt-aalpppappp ( (,,), and T1 =
(OTT)(O)) (0T),s?)
where T is the number of iterations, 5 and sT are the numbers of optimization steps for 0 and 0 respectively. In words, we wish to generate a set of condensed samples S such that the network parameters trained on them (0$) are similar to the ones trained on the original training set (0%) ) at each iteration t. In our preliminary experiments, we observe that 05 which is parameterized with S, can successfully track 0 by updating and minimizing D(0 07) close to zero.
In the of one step gradient descent optimization for opt-alg, the update rule is:
0 o. V and 0f - 07 - - VoL (07).
(8)
where 1/ is the learning rate. Based on observation (D(0S 07 R 0), simplify the formula- tion in eq. (7 by replacing 0% with 0 and use O to denote 0 in the rest of the paper:
T-1
min E Pou
D(VoL (o)
(9)
t=0
We now have single deep network with parameters 0 trained on the synthetic set S which is optimized such that the distance between the gradients for the loss the training samples LT w.r.t. 0 and the gradients for the loss over the condensed samples LS w.r.t. is minimized. In words, goal reduces matching the gradients for the real and synthetic training loss w.r.t. via updating the condensed samples. This approximation has the key advantage over (Wang et al.) 2018) and eq. (5) that it does not require the expensive unrolling of the recursive computation graph over the previous parameters (Ou....,0 0 J. The important consequence is that the optimization is significantly faster, memory efficient and thus scales up to the state-of-the-art deep neural networks (e.g. ResNet (He et al. ).16)).
The synthetic data contains not only samples but also their labels (s,y) that can be jointly learned by optimizing eq. (9) in theory. However, their joint optimization is challenging, as the content of the samples depend on their label and vice-versa. Thus in our experiments we learn to synthesize images for fixed labels, e.g. synthetic image per class.
Discussion.
4

---

Published as a conference paper at ICLR 2021
Algorithm. We depict the optimization details in Alg. 1 At the outer level, it contains loop over random weight initializations, as we want to obtain condensed images that can later be used train previously unseen models. Once 0 is randomly initialized, both the training samples (L"), ), synthetic samples (C$) and their gradients w.r.t. 0, then optimize the synthetic samples S match these gradients VoL$ to VoL by applying <s gradient descent steps with learning rate nJ. We use the stochastic gradient descent optimization for both opt-alo and
Og to first compute the loss
Next train 0 on the updated synthetic images by minimizing the loss L with learning rate ne for çø steps. Note that we sample each real and synthetic batch pair from T and containing samples from a single class and the synthetic data for each class are separately (or parallelly) updated at each iteration (1) for the following reasons: i) this reduces memory use at train time, ii) imitating the mean gradients w.r.t the data from single class is easier compared to those of multiple classes. This does not bring any extra computational cost.
Algorithm 1: Dataset condensation with gradient matching
Input: Training T
Required: Randomly initialized set synthetic samples for C' classes, probability distribution over randomly initialized weights Pa,, deep neural network Φe, number of outer-loop steps K, number of inner-loop steps T, number of steps for updating weights and synthetic samples <s in each inner-loop step respectively, learning rates for updating weights no and synthetic samples 11s. for 0, ................................ Initialize 0, ~ Paa
do
for t = 0, 0............................... T do
for 0, C do
Sample minibatch pair D. ~ T and _ ~ P Bb and B are of the same class c.
Compute C!
T DinyERn E(oo, (a). y) and C TaTT Einate ag E(óo, (s).
Update S. + opt opt-alls
VoL? (0.)),ss,n)
Update ac+1 + opt-alleee ( ( (0.), so, no)
b Use the whole S
Output:
Gradient matching loss. The matching loss D(.,)) in eq. measures the distance between the gradients for CS and L w.r.t O. When de a multi-layered neural network, the gradients corre- spond to set of learnable 2D (out xn) and 4D (out x xh xw) weights for each fully connected (FC) and convolutional layer resp where out, in, h, w are number output and input channels, kernel height and width resp. The matching loss can be decomposed into sum of layerwise losses D(Vo VoLT)= ΣL_ ((Vo LS Vam LT) where / is the layer index, is the number of layers with weights and
A₁. B A -
d(A B)
(10)
B
where A; and B, are flattened vectors of gradients corresponding each output node i, which is in dimensional for FC weights and in x X dimensional for convolutional weights. In contrast to Lopez-Paz et al. 2017/Aljundi Alluudii al. [019] Zhu et al.) (209) that ignore the layer-wise structure by flattening tensors over all layers to one vector and then computing the distance between two vectors, group them for each output node. We found that this is better distance for gradient matching
(see the supplementary) and enables using single learning rate across all layers.
EXPERIMENTS
3.1
DATASET CONDENSATION
First we evaluate classification performance with the condensed images on four standard benchmark
datasets: digit recognition on MNIST (LeCun et al. 1998), SVHN (Netzer et al. 2011) and object classification on FaShionMNIST (Xiao et al. 2017). CIFARIO (Krizhevsky et al. 2009). We test our method using six standard deep network architectures: MLP, ConvNet (Gidaris & Komodakis, 2018), LeNet (LeCun al. 1998), AlexNet (Krizhevsky al. 2012), VGG-11 (Simonyan & Zis- serman 2014) and ResNet- 18 (He et al. 2016). MLP is a multilayer perceptron with two nonlinear hidden layers, each has 128 units. ConvNet is commonly used modular architecture in few-shot
5

---

Published as a conference paper at ICLR 2021
Coresel
Selection K-Conter
Img/Cls
Ratio S
Random
Herding
Forgetring
Ours
Whole Dataset
0.017
64.9
1.33.1..5 19
35,51.5.6
9..7
MNIST
0.17
95.1=09
93.7+033
84.41.77
68. 1433
97.4=0.2
99.6=0000
50
0.83
97.9=0.2
94,92222
97.4+0.3
88224:2
98.8:00.2
0.017
514438
67.0+119
669-18
42.04-5.5
70.5-00.6
FashionMNIST
10
0.17
73.8-077
71.1::7
54.7415
53.94200
82.3-2044
93.5
50
0.83
82.5=0.7
719408
68.3+03
55.0+:111
83.62:0.4
0.014
14.6=1.6
20.9+11.3
21.011.5
12.1+117
312+114
SVHN
0.14
35.1=4.1
50.54:3.
14.01:33
16.8412
76.1::0.6
95.40.11
50
0.7
70.9-0.9
72.6.0088
2..1
27.2.1.5
0.3
0.02
1..4=2.
21.5::12
21.54:13
13.54:12
2.5
CIFARIO
10
0.2
1.2
31.64007
14.740.99
23,34:10
4..9-0.55
84.8.0.1
1..4=1.0
40.4
27.0+1.4
23.3+111
53.9-0.5
Table I: The performance comparison coreset methods, This table shows the testing accuracies (%) of different methods on four datasets. ConvNet is used for training and testing. Img/Cls: image(s) per class, Ratio (%): the ratio of condensed images to whole training set.
learning (Snell et al. 2017, Vinyals et al. 2016; Gidaris & Komodakis, 2018) with D duplicate blocks, and each block has a convolutional layer with W (3 x 3) filters, normalization layer N, an activation layer A and a pooling layer P. denoted as [W, NN P] x D. The default ConvNet (unless specified otherwise) includes 3 blocks, each with 128 filters, followed by InstanceNorm (Ulyanov jet al.,, 2016), ReLU and AvgPooling modules. The final block is followed by a linear classifier. We use Kaiming initialization (He et al.) (2155) for network weights. The synthetic images can be initialized from Gaussian noise randomly selected real training images. More details about the datasets, networks and syyper-parametes can be found in the supplementary.
The pipeline for dataset condensation has two stages: learning the condensed images (denoted as C) and training classifiers from scratch on them (denoted as T). Note that the model architectures used in two stages might be different. For the coreset baselines, the coreset selected in the first stage. We investigate three settings: 1, 10 and 50 image/class learning, which means that the condensed set or coreset contains 1, 10 and 50 images per class respectively. Each method is run for 5 times, and 5 synthetic sets are generated in the first stage; each generated synthetic set is used to train 20 randomly initialized models in the second stage and evaluated on the test set, which amounts to evaluating 100 models in the second stage. In all experiments, we report the mean and standard deviation of these 100 testing results.
Baselines. We compare our method to four coreset baselines (Random, Herding, K-Center and Forgetting) and also to DD (Wang al.) 2018). In Random, the training samples are randomly selected as the coreset. Herding baseline, which selects closest samples to the cluster center, is based on (Weling 2009) and used in (Rebufti et al., [017]] Castro et al. 2018 Wu et al. 2019 Belouadah & Popescul 2020). K-Center (Wolf) 2011 Sener & Savarese 2018) picks multiple center points such that the largest distance between data point and its nearest center is minimized. For Herding and K-Center, we use models trained on the whole dataset to extract features, compute l2 distance to centers. Forgetting method (Toneva et al., 2019) selects the training samples which are easy to forget during training. We do not compare to GSS-Greed) (Aljundi et al. 2019)) because it is also a similarity based greedy algorithm like K-Center, but GSS- GSS-Greedy trains an online learning model to measure the similarity of samples, which is different from general image classification problem. More detailed comparisons can be found in the supplementary.
Comparison to coreset methods. We first compare our method to the coreset baselines on MNIST, FashionMNIST, SVHN and CIFARIO in Table | using the default ConvNet in classifi- cation accuracy. Whole dataset indicates training on the whole original set which serves as an approximate upper-bound performance. First we observe that our method outperforms all the base- lines significantly and achieves a comparable result (98.8% in case of 50 images per class to the upper bound (99.6%) in MNIST which uses two orders of magnitude more training images per class (6000). We also obtain promising results in FaShionMNIST. however, the gap between our method and upper bound is bigger in SVHN and CIFAR 10 which contain more diverse images with varying foregrounds and backgrounds. We also observe that, (i) the random selection baseline is competitive to other coreset methods in 10 and 50 images per class and (ii) herding method is average the best coreset technique. We visualize the condensed images produced by our method under 1 image/class setting in Figure Interestingly they are interpretable and look like "prototypes" of each class.
6

---

Published as a conference paper at ICLR 2021
0
1
2
3
4
5
6 7
8
9
AiT
MLP
ConvNet
LeNet
AlexNet
VGG
ResNet
MLP
7..5111.2
63,916.5
77.31..8
70.911166
53.2.1..0
80.9.3.6
ConsNet
69.61116
91.7:0555 85.3118
85.1=30
8344
90.0-00.8
10
14p
Partaru oveltress Coat 50 noa Shirbsneakee Sweaker Bag book
11
3
4
5
7
9
AlexNet
LeNet
71,01:16 90.31122
72.1-117
17.5+:1.
85.0.1..7
84.0228
84.7.22
82.7429
81:243.0
80.3.2.7
89.0.0.8.
1.1911111
VGG
70.3+1.6
90.1+007
83.9+2.7
83,43377
81.7226
89.1
ResNet
73.6:112 91.600.5 86,4+15
8544=19
834424 89.4099
Plane
Bar
Git
Deer
Dog
Frog
Honse
Ship Trukk
Figure 2: Visualization of condensed im-
Table 2: Cross-architecture performance in testing accu- racy (%) for condensed 1 image/class in MNIST.
age/class with ConvNet for MNIST, Fashion- MNIST, SVHN and CIFARIO.
Dataset
Img/Cis
DD
Ours
Whole Dalasel
Random Herding Ours Farly-stopping Whole Dataset
MNIST
85.0::1.6
99.5+
Performance (%)
76.2
76.2
84.5
85.9
10
7..
9..9
Comelaaion
-0.21
-0.20
0.79
0.42
1.00
CIFARIO
3..8.1..2
24.2++00.9
1.2
83.1+002
Time cost (min)
Storage (imgs)
10a
18.8
10a
18.8
10a
18.8
10*
18.8
8604.3
10
Table 3: Comparison to DD [Wang et al. 2018) in terms of testing accuracy (%)
Table 4: Neural Architecture Search. Methods are compared in performance, ranking correlation, time and memory cost.
Comparison to DD (Wang et al., 2018). Unlike the setting in Table I DD (Wang et al., 2018)
reports results only for 10 images per class on MNIST and CIFAR 10 over LeNet and Alex Cet (a customized AlexNet). We strietly follow the experimental setting in (Wang et al., 2018). use the same architectures and report our and their original results in Table for a fair comparison. Our method achieves significantly better performance than DD on both benchmarks; obtains 5% higher accuracy with only 1 synthetic sample per class than DD with 10 samples per class. In addition, our method obtains consistent results over multiple runs with a standard deviation of only 0.6% on MNIST, while DD's performance significantly vary over different runs (8.1%). Finally our method trains 2 times faster than DD and requires 50% less memory on CIFAR 10 experiments. More detailed runtime and qualitative comparison can be found in the supplementary.
Cross-architecture generalization. Another key advantage of our method is that the condensed images learned using one architecture can be used to train another unseen one. Here we learn 1 con- densed image per class for MNIST over a diverse set of networks including MLP, ConvNet (Gidaris) & Komodakis 2018), LeNet (LeCun et al. 1998), AlexNet (Krizhevsky al. 2012), VGG-11 (Si-
monyan & Zisserman 2014) and ResNet- 88 (He et al., 2016) (see Table 2). Once the condensed sets are synthesized, we train every network on all the sets separately from scratch and evaluate their cross architecture performance in terms of classification accuracy on the MNIST test set. Table 2 shows that the condensed images, especially the ones that are trained with convolutional networks, perform well and are thus architecture generic. MLP generated images do not work well for training convolutional architectures which is possibly due to the mismatch between translation invariance properties of MLP and convolutional networks. Interestingly, MLP achieves better performance with convolutional network generated images than the MLP generated ones. The best results are obtained in most cases with ResNet generated images and ConvNet or ResNet classifiers which is inline with their performances when trained on the original dataset.
Number of condensed images. We also study the test performance of ConvNet trained on them for MNIST, FashionMNIST, SVHN and CIFAR1O 10 for various number of condensed images per class in Figure 3 in absolute and relative terms- normalized by its upper-bound. Increasing the number of condensed images improves the accuracies in all benchmarks and further closes the gap with the upper-bound performance especially in MNIST and MNShionMNIST, while the gap remains larger in SVHN and CIFAR1O. In addition, our method outperforms the coreset method Herding by a large margin in all cases.
Activation, normalization & pooling. We also study the effect of various activation (sigmoid, ReLU (Nair & Hinton 2010 Zeiler et al. 22013), leaky ReLU (Maas et al. 2013)). pooling (max, average) and normalization functions (batch (Toffe & Szegedy 2015). group (w & He 2018), layer (Ba et al. 2016), instance norm (Ulyanov et al. (0166) and have the following observations: i) leaky ReLU over ReLU and average pooling over max pooling enable learning better condensed images, they allow for denser gradient flow; ii) instance normalization obtains better classifica-
7

---

Published as a conference paper at ICLR 2021
5T
FashionMNiST
95.2
-99
90
95
95-
94.5
Abxolute sccuracy (95)
80
60
Hardig
Our
-90 70
hard
Relsive sccurscy
93
93.0
SVHN
CIFAR10
92.4
80-
92
Herding w. KD
60-
60 40
91
Herding w.o. KD
Durs w. KD
90.7
40
Hendrrg rg
30
Durs
20
Duy'
20
90
3.0
20
10
60
50
30
20
30
40
50
Images/cioss
Images/cass
Training stage
Figure 3: Absolute and relative testing accuracies for varying the number of condensed images/class for MNIST, FasHionMNIST, SVHN and CIFARIO. The relative accuracy means the ratio compared upper- bound, Le. training with the whole dataset.
Figure 4: Continual learning performance in
accuracy (%). Herding denotes the original E2E (Castro et al. 2013). T1, T2, T3 are three learning stages. The performance at each stage is the mean testing accuracy on all learned tasks.
tion performance than its alternatives when used in the networks that are trained on small set of condensed images. We refer to the supplementary for detailed results and discussion.
3.2
APPLICATIONS
Continual Learning First we apply our method to a continull-leaming scenario (Rebuffi et al., Castro et al. 2018) where new tasks are learned incrementally and the goal to preserve the performance on the old tasks while learning the new ones. We build our model on E2E method in (Castro et al., (018)) that uses a limited budget rehearsal memory (we consider 10 images/class here) to keep representative samples from the old tasks and knowledge distillation (KD) to regularize the network's 's output w.r.t. to previous predictions. We replace its sample selection mechanism (herding) with ours such that set of condensed images are generated and stored in the memory, keep the rest of the model same and evaluate this model on the maskiincrementl learning problem on the digit recognition datasets, SVHN (Netzer et al. 2011), MNIST (LeCun al. 1998) and USPS (Hull 1994] in the same order. MNIST and USPS images are reshaped to 32 x 32 RGB images.
2017
We compare our method to E2E (Castro et al. 2018). depicted as herding in Figure with and without KD regularization. The experiment contains 3 incremental training stages WVHNN-MNISTT*USIIE
and testing accuracies are computed by averaging over the test sets of the previous and current tasks after each stage. The desired outcome is to obtain high mean clas- sification accuracy at T3. The results indicate that the condensed images are more data-efficient than the ones sampled by herding and thus our method outperforms E2E in both settings, while by larger margin (2.3% T3) when KD is not employed.
Neural Architecture Search. Here we explore the use of our method in simple neural architec- ture search (NAS) experiment on CIFAR 10 which typically requires expensive training of numerous architectures multiple times the whole training set and picking the best performing ones a val- idation set. Our goal is to verify that our condensed images can be used to efficiently train multiple networks identify the best network. To this end, we construct the search space of 720 ConvNets as described Section 3.I by varying hyper-parameter W, N, A, P, over an uniform grid (see supplementary for more details), train them for 100 epochs on three small proxy datasets (10 im- ages/class) that are obtained with Random sampling, Herding and our method. Note that we train the condensed images for once only with the default ConvNet architecture and use them to train all kinds of architectures. We also compare to early-stopping (Li & Talwalkar 2020) in which the model is trained on whole training set but with the same number of training iterations as the one required for the small proxy datasets, in other words, for the same amount of computations.
Table 4 depicts i) the average test performance of the best selected model over runs when trained on the whole dataset, ii) Spearman's 's rank correlation coefficient between the validation accuracies obtained by training the selected top 10 models on the proxy dataset and whole dataset, iii) time for training 720 architectures images. Our method achieves the highest testing performance (84.5%) and performance correlation (0.79), meanwhile significantly decreases the the searching time (from 8604.3 to 18.8 minutes) and storage space (from x 10 to x 102 images) compared to whole-dataset training. The competitive early-stopping baseline achieves on par performance for the best performing model with
NVIDIA 10X1080-Ti GPU, and iv) memory print of the training
8

---

Published as a conference paper at ICLR 2021
ours, however, the rank correlation (0.42) of top 10 models is significantly lower than ours (0.79) which indicates unreliable correlation of performances between carly-stopping and whole-dataset training. Furthermore, stopy-stopping needs 100 times many training images that the training time for synthetic images is around 50 minutes (for K = 500) which is one time off and negligible cost when training thousands even millions of candidate architectures in NAS.
needs. Note
CONCLUSION
In this paper, we propose dataset condensation method that learns synthesize a small set of informative images. We show that these images are significantly more data-efficient than the same number of original images and the ones produced by the previous method, and they are not archi- tecture dependent, can be used to train different deep networks. Once trained, they can be used to lower the memory print of datasets and efficiently train numerous networks which are crucial in continual learning and neural architecture search respectively. For future work, we plan to explore the use of condensed images in more diverse and thus challenging datasets like ImageNet (Deng al. 2009) that contain higher resolution images with larger variations in appearance and pose of objects, background.
Acknowledgment.
This work is funded by China Scholarship Council 2018 806010331 and the EPSRC programme grant Visual AI EP/T028572/1. We thank lain Murray and Oisin Mac Aodha for their valuable feedback.
REFERENCES
Pankaj K. Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Approximating extent measures of points. Journal of the ACM (JACM), 51(4):606-635, 2004.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816- 11825, 2019.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural informa- fion processing systems, pp. 2654-2662, 2014.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learn- ing. In The IEEE Winter Conference on Applications of Computer Vision, 2020.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. Neural Information Processing Systems Workshop, 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 535-541, 2006.
Francisco M Castro, Manuel J Marín JiméénJimménz, Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 233-248. 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. Ieee, 2009.
Justin Domke. Generic methods for optimization- based modeling. In Artificial Intelligence and Statistics, pp. 318-326, 2012.
Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-sizz coresets for k-means, pea and projective clustering. In Proceedings of the twenty-fourth annual ACM-SIAM symposium Discrete algorithms, pp. 1453. SIAM, 2013.

---

Published as a conference paper at ICLR 2021
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018.
Jack Goetz and Ambuj Tewari. arXiv:2008.04489 2020.
Federated learning via synthetic data. arXiv preprint
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium Theory of computing, 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
Jonathan J. Hull. A database for handwritten recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550-554, 1994.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, 1bs/1500.03316,, 2015.
Diederik P Kingma and Max Welling, arXiv:1312.6114 2013.
Auto-encoding variational bayes. arXiv preprint
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1885- 1894. JMLR. org, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11):2278-2322 1998.
Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference Image Processing (ICIP), pp. 305- 309. IEEE, 2020.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. LLD Workshop at Neural Information Processing Systems (NIPS) 2017.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
10

---

Published as a conference paper at ICLR 2021
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In International conference on machine learning (ICML), 2013.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5188-5196, 2015.
Mehdi Mirza and Simon Osindero. arXiv: 1784, 2014.
Conditional generative adversarial nets. arXiv preprint
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation deep networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 2001-2010, 2017.
Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. generative process for sampling contractive auto-encoders. arXiv preprint 1rXiv:1206.666 1206.6434, 2012.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Kegan GG Samuel and Marshall F Tappen. Learning optimized map estimates in continuously- valued mrf models. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp- 477-484. IEEE, 2009.
Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gradient matching generative networks for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168-2178, 2019.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. ICLR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp- 4077-4087, 2017.
Felipe Petroski Such, Aditya Rawal, Jocl Lehman, Kenneth O Stanley, and Jeff Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. International Conference on Machine Learning, 2020.
Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. arXiv preprint (eXXi:1910.025511 2019.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 1607.0802, 2016.

---

Published as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, 2016.
Tongzhou Wang, Jun- Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint 1rX9v 2018.
Max Welling. Herding dynamical weights learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121-1128. ACM, 2009.
G W Wolf. Facility location: concepts, models, algorithms and case studies. 2011.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint
2017.
Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
Matthew D. Zeiler, Mare' Aurelio Ranzato, Rajat Monga. Mark Z. Mao, Kyeongcheol Yang. Quoc V. Le, Patrick Nguyen, Andrew W. Senior, Vincent Vanhoucke, Jeffrey Dean, and Geoffrey E. Hin- ton. On rectified linear units for speech processing. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3517-3521, 2013.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.0266 2020.
Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999, 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, pp. 14747-14756, 2019.
12

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
In this part, we explain the implementation details for the dataset condensation, continual learning and neural architecture search experiments.
Dataset condensation. The presented experiments involve tuning of six hyperparameters the number of outer-loop K and inner-loop steps T, learning rates Tis and number of optimization steps <s for the condensed samples, learning rates ne and number of optimization steps so for the model weights. In all experiments, we set K = 1000, 7s = 0.1, Stochastic Gradient Descent (SGD) as the optimizer. The only exception is that we set is to 0.01 for synthesizing data with MLP in cross- crosttrchitecture experiments (Table as MLP requires a slightly different treatment. Note that while K is the maximum number of outer-loop steps, the optimization can early-stop automatically if it converges before K steps. For the remaining hyperparameters, we use different sets for 1, 10 and 50 imagess)/class learning. We set T = 1, <o = 1 for 1 image/class, T = 10, <g 50 for 10 images/class. T 50, se 10 for 50 images/class learning. Note that when T = I, it is not required to update the model parameters (Step 9 in Algorithm DT as this model is not further used. For those experiments where more than 10 images/class are synthesized, we set T to be the same number as the synthetic images per class and so = 500/T, e-g T = 20, so 25 for 20 images/class learning. The ablation study on hyper-parameters are given in Appendix B which shows that our method is not sensitive to varying hyper-parameters.
0.01, <s = 1 and employ
We do separate-class mini-batch sampling for Step 6 in Algorithm I Specifically, we sample mini-batch pair BT and B that contain real and synthetic images from the same class at each inner iteration. Then, the matching loss for each class is computed with the sampled mini-bator pair and used to update corresponding synthetic images S. by back-propogation (Step 7 and 8). This is repeated separately (or parallelly given enough computational resources) for every class. Training as such is not slower than using mixed-class batches. Although our method still works well when we randomly sample the real and synthetic mini-batches with mixed labels, we found that separate- class strategy is faster to train as matching gradients w.r.t. data from single class is easier compared to those of multiple classes. In experiments, we randomly sample 256 real images of class as Mini-bacch& to calculate the mean gradient and match it with the mean gradient that is averaged over all synthetic samples with the same class label. The performance is not sensitive to the size of real-image mini-batch if is greater than 64.
In all experiments, we use the standard train/test splits of the datasets the train/test statistics are shown in Table [5] We apply data augmentation (crop, scale and rotate) only for experiments (coreset methods and ours) on MNIST. The only exception is that we also use data augmentation when compared to DD (Wang et al. 2018) on CIFARIO with AlexCifarNet, and data augmentation is also used in (Wang et al.) 2018). For initialization of condensed images, we tried both Gaussian noise and randomly selected real training images, and obtained overall comparable performances in different settings and datasets. Then, we used Gaussian noise for initialization in experiments.
USPS MNIST FashionMNIST SVHN CIFARIO CIFARI00
Train 7,291 60,000
60,000
73,257
50,000
50,000
Test 2,007
10,000
10,000
26,032
10,000
10,000
Table T5: Train/test statistics for USPS, MNIST, FashionMNIST, SVHN, CIFARIO and CIFAR 100 datascts.
In the first stage while training the condensed images we use Batch Normalization in the VGG and ResNet networks. For reliable estimation of the running mean and variance, we sample many real training data to estimate the running mean and variance and then freeze them ahead of Step 7. In the second stage - while training deep network on the condensed set -, we replace Batch Normalization layers with Instance Normalization in VGG and ResNet, due to the fact that the batch statistics are not reliable when training networks with few condensed images. Another minor modification that we apply to the standard network ResNet architecture in the first stage is replacing the strided convolutions where stride = 2 with convolutional layers where stride = 1 coupled with an average pooling layer. We observe that this change enables more detailed (per pixel) gradients w.r.t. the condensed images and leads to better condensed images.
Continual learning. In this experiment, we focus on a insk-ennremental learning on SVHN MNIST and USPS with the given order. The three tasks share the same label space, however have
13

---

Published as a conference paper at ICLR 2021
Random: Correlation - 0.21
Herding: Correlation -0.20
Our:: Correltion - 0.79
Eufl-siopohng Correlation - 0.42
Peformance
0.85
0.85
0.86
0.86
0.83
0.83
0.84
0.80
0.82
0.80
0.78
6.80
0.78
0.75
0.78
0.73
0.33
0.32
0.33
0.34
0:35
0.36
0.37
0.38
0.42
0.44
0.46
0.52
0.54
0.56
Procy Dataset Pet formancc
Prozy Dataset Performance
Proxy Dutaset Performance
Prony Dataset Performance
Figure F5: The performance correlation between the training on proxy dataset and whole-dataset. For each proxy dataset, the best 10 models are selected based on validation set performance. In the figure, each point represents an architecture.
Sigmoid
ReLu
LeakyRoLu
C\T
None
MaxPooling
AvgPooling
Sigmoid
86.740.7
91 01.66
91.24:.6
None
78.
80.8
88.3+1
ReLu
86. 16:99
91.7=0.5
91.7+0,55
MaxPooling
81.2+28
1..5
91.10066
LeakyRoLu
86,3:099
91.7:0055
91.79004
Avgpocing
81.422999
90.2
91.7-00055
Table T6: Cross-activation experiments in accu-
Table T7:
Cross-pooling experiments in accu-
racy (%) for condensed image/class in MNIST.
racy (%) for condensed image/class in MNIST.
significantly different image statistics. The images of the three datasets are reshaped to 32 x 32 RGB size for standardization. We use the standard splits for training sets and randomly sample 2,000 test images for each datasets to obtain a balanced evaluation over three datasets. Thus each model is tested growing test set with 2,000, 4,000 and 6,000 images the three stages respectively. We use the default ConvNet in this experiment and set the weight of distillation loss to 1.0 and the temperature to 2. We 5,000 and 500 iterations for training and balanced finetuning in (Castro eet al. 2018) with the learning rates 0.01 and 0.001 respectively. We run 5 experiments and report the mean and standard variance in Figure
Neural Architecture Search. To construct the searching space of 720 ConvNets, we vary hyper- parameters W E (32. 64, 128, 256). D E f1, 3, 4}. N E (None, BatchNorm, LayerNorm, InstanceNorm, GroupNorm), A E (Sigmoid, ReLu, LeakyReLu), P E (None, MaxPooling, AvgPooling). We randomly sample 5,000 images from the 50,000 training images in CIFARIO 1 the validation set. Every candidate ConvNet is trained with the proxy dataset, and then evalu- ated on the validation set. These candidate ConvNets are ranked by the validation performance. 10 architectures with top validation accuracies are selected calculate Spearman's rank correlation coefficient, because the best model that we want will come from the top 10 architectures. We train each ConvNet for 5 times to get averaged validation and testing accuracies.
We visualize the performance correlation for different proxy datasets in Figure F5 Obviously, the condensed proxy dataset produced by our method achieves the highest performance correlation (0.79) which significantly higher than early-stopping (0.42). It means our method can produce more reliable results for NAS.
B FURTHER ANALYSIS
Next we provide additional results on ablative studies over various deep network layers including activation, pooling and normalization functions and also over depth and width of deep network architecture. We also study the selection of hyper-parameters and the gradient distance metric. An additional qualitative analysis on the learned condensed images is also given.
Ablation study on activation functions. Here we study the use of three activation functions Sigmoid, ReLU,, LeakyReLu (negative slope is set 0.01) - in two stages, when training condensed images (denoted
C) and when training a ConvNet from scratch on the learned condensed im- ages (denoted as T). The experiments are conducted in MNIST dataset for condensed image/class setting. Table T6 shows that all three activation functions are good for the first stage while gener- ating good condensed images, however, Sigmoid performs poor in the second stage while learning classifier the condensed images its testing accuracies are lower than ReLu and LeakyReLu by around 5%. This suggests that ReLU can provide sufficiently informative gradients for learning condensed images, though the gradient of ReL.U w.r.t. to its input is typically sparse.
14

---

Published as a conference paper at ICLR 2021
CXT
None
BatchNorm
LayerNorm
InstanceNorm
GroupNorm
None
79.0:12..2
80.8-2.0
1..8:11.7
90.7:40.7
8..9
BatchNorm
78.6:12.11
80.7:1188
15.7:116
90.9:10.6
15
LayerNorm
81.2.41.88
78.6433
17.41113
90.7440.77
8..341..4
InstanceNorm
72.9+7.1
56,746.5
82.745.3
91.7+-0.5
84,3+44,2
GroupNorn
79.5+2211
81.8+2.3
87.3+1122
91.6+005
87.2+1.2
Table T8: Cross-normalization experiments in accuracy (%) for 1 condensed image/class in MNIST.
C/2
c\r
128
256
1 613435 78.2=30
77.1+400 76.4+355
32 90.6=208 91,4:005 91.5+0055 91 3:60.66
78.3+2233 89.0=088
91001066
89.4:0.8
9100=008 91.6::066 91,8:0.5 91.46066
3
1. 89
91.7.10.5 90.4.10.6
128
908..0.7 91.5.10.6 91.7.10.55 91.2.10.7
1.3541.3 899-008 91.9.0.5 90.6.04
256
91.0410.7 91.6:10.6 91.7100. 91.410.5
Table T9: Cross-depth performance in accuracy
Table T10: Cross-width performance in accuracy (9) for 1 condensed image/class in MNIST.
(%) for 1 condensed image/class in MNIST.
Ablation study on pooling functions. Next we investigate the performance of two pooling func- tions average pooling and max pooling- also no pooling for 1 image/class dataset condensation with ConvNet in MNIST in terms of classification accuracy. Table ( shows that max and average pooling both perform significantly better than no pooling (None) when they are used in the second stage. When the condensed samples are trained and tested on models with average pooling, the best testing accuracy (91.7 + 0.5%) is obtained, possibly, because average pooling provides more informative and smooth gradients for the whole image rather than only for its discriminative parts.
Ablation study on normalization functions. Next we study the performance of four normal- ization options No normalization, Batch (loffe & Szegedy 2015), Layer (Ba et al.) 2016), In- stance (Ulyanov et al. 2016) and Group Normalization (Wu & 2e (number of groups is set to be four)- for 1 image/class dataset condensation with ConvNet architecture in MNIST classi- fication accuracy. Table T8 shows that the normalization layer has little influence for learning the condensed set, while the choice of normalization layer important for training networks on the con- densed set. LayerNorm and GroupNorm have similar performance, and InstanceNorm is the best choice for training a model on condensed images. BatchNorm obtains lower performance which is similar to None (no normalization), as it is known to perform poorly when training models on few condensed samples as also observed in (Wu & He, 2018). Note that Batch Normalization does not allow for a stable training in the first stage (C); thus we replace its running mean and variance for each batch with those of randomly sampled real training images.
Ablation study on network depth and width. Here we study the effect of network depth and width for I image/class dataset condensation with ConvNet architecture in MNIST in terms of clas- sification accuracy. To this end we conduct multiple experiments by varying the depth and width of the networks that are used to train condensed synthetic images and that are trained to classify testing data in ConvNet architecture and report the results in Table T9 and Table TTO In Table T9) observe that deeper ConvNets with more blocks generate better condensed images that results in better classification performance when a network is trained on them, while ConvNet with 3 blocks performs best as classifier. Interestingly, Table T10 shows that the best results are obtained with the classifier that has 128 filters at each block, while network width (number of filters at each block) in generation has little overall impact on the final classification performance.
Our performance not sensitive to hyper- parameterr se- lection. The testing accuracy for various K and T, when learning 10 images/class condensed sets, is depicted in Figure F6 The results show that the optimum K and T are around similar values across all datasets. Thus we simply set K to 1000 and T to 10 for all datasets. Similarly, for the remaining ones including learning rate, weight decay, we use single set of hyperparameters that are observed to work well for all datasets and architectures in our preliminary experiments.
Ablation study on hyper-parameters.
Ablation study on gradient distance metric. To prove the effectiveness and robustness of the proposed distance metric for gradients (or weights), we compare to the traditional ones (Lopez- Paz et al. 2017: Aljundi et al. 22119; Zhu et al. 2019) which vectorize and concatenate the whole gradient, GT, G E "R and compute the squared Euclidean distance ||GT - G and the
Cosine distance 1 - cos (GT,GS), where D is the number of all network parameters. We do
15

---

Published as a conference paper at ICLR 2021
100
100
Testing accuracy (%)
90
80
80
60
MNIST
50
PashionMNIST
FashiorMMSS
40
OIARRO
SVHN
40
SVHN
CIFAR10
50
100
200
500
1000
1500
2000
2
5
10
20
50
pypprr-arameter:
Hyper-parameter:
Figure F6: Ablation study on the papammtaraeeters K and T' when learning 10 images/class condensed sets.
image/class learning experiment on MNIST with different architectures. For simplicity, the synthetic images are learned and tested on the same architecture in this experiment. [TT shows that the proposed gradient distance metric remarkably outperforms others on complex architectures (e.g- LeNet, AlexNett VGG and ResNet) and achieves the best performances in most settings, which means it is more effective and robust than the traditional ones. Note that we set vis 0.1 for MLP-Euclidean and MLP-Cosine because it works better than T/s = 0.01.
MLP
ConvNet
LoNet
AlexNet
VGG
ResNet
Euclidean
69.3+009 92.7-0.3 65.045.1 66.245.6 57.147.0 68.045,2
Cosine
45.2=#36 69.2+2.7
61.1#82
58.3+44.1 55.0=5500 68.8+7.8
Ours
70.5 91.7.00.5 85.0::1.7 82.7129 81.71:266 8.94
Table T11: 1: Ablation study on different gradient distance metrics. Ohviously, the proposed distance metric is more effective and robust. Euclidean: squared Euclidean distance, Cosine: Cosine distance.
Further qualitative analysis We first depict the condensed images that are learned on MNIST, FashionMNIST, SVHN and CIFARIO datasets in one experiment using the default ConvNet in 10 images/class setting in Figure F7 It is interesting that the 10 images/class results in Figure F7 are diverse which cover the main variations, while the condensed images for image/class setting (see Figure 2 look like the "prototype" of each class. For example, in Figure F7 (a), the ten images of "four" indicate ten different styles. The ten "bag" images in Figure F7 (b) are significantly different from each other, similarly "wallet" (1st row), "shopping bag" (3rd row), "handbag" (8th row) and
"schoolbag" (10th row). Figure F7 (c) also shows the diverse house numbers with different shapes, colors and shadows. Besides, different poses of a "horse" have been learned in Figure F7 (d).
C COMPARISON TO MORE BASELINES
Optimal random selection. One interesting and strong baseline is Optimal Random Selection (ORS) that we implement random selection experiments for 1,000 times and pick the best ones. Table T12 presents the performance comparison to the selected Top 1000 (all), Top 100 and Top 10 coresets. These optimal coresets are selected by ranking their performance. Obviously, the condensed set generated by our method surpasses the selected Top 10 of 1000 coresets with large margin on all four datasets.
Generative model. We also compare to the popular generative model, namely, Conditional Gener- ative Adversarial Networks (cGAN) (Mirza & Osindero 2014). The generator has two blocks which consists of the Up-sampling (scale factorr.. Convolution (stride=1), BatchNorm and LeakyReLu layers. The discriminator has three blocks which consists of Convolution (stride=2), BatchNorm and LeakyReLu layers. In additional the random noise, we also input the class label the condition. We generate and 10 images per class for each dataset with random noise. Table T12 shows that the images produced by cGAN have similar performances those randomly selected coresets (i.e. Top 1000). It is reasonable, because the aim of cGAN is to generate real-look images. In contrast, our method aims to generate images that can train deep neural networks efficiently.
Analysis coreset performances We find that K-Center (Wolf 2011 Sener & Savarese 2018) and Forgetting Toneva et al. 2019) don't work well other general coreset methods, namely
16

---

Published as a conference paper at ICLR 2021
2
5
Top
PantsRutoveDDess Sandai Sneaker Bag
Boot
3
2
3
4
5
6
7
0
&
2
3
7
6
7
8
9
6
2-
3
4
5
6
7
8
9
0
/
5
3
4
f
G
7
8 4
9
4
/
1
3
4
5
6
7
9
P
3
4
5
6
7
8
9
f
2
3
4
5
6
7
8
4
4
L
3
4
5
7
9
7
2
7
5
6
7
8
9
0
2
3
5
7
8
9
(a) MNIST
(b) FaSH
1
3
6
Plane
Car
Bird
cat
Deer
Dog
Prog
Horse
Ship
Truck
5
5
6
8
9
0
1
7
2
3
4
5
5
6
7
8
9
0
1
2
3
5
6
7
9
0
13
2
3
4
6
8
0
2
5
5
8
9
3
4
5
6
8
9
11
2
4
5
0
7
8
9
3
5
8
3
9
(c) SVHN
(d) CIFARIO
Figure F7: The synthetic images for MNIST, FashionMNIST, SVIN and CIFARIO produced by our method with ConvNet under 10 images/class setting.
Optimal Random Selection
Img/Cls
Ratio 1%:
Top 1000
Top 10D
Top
cGAN
Ours
Whole Dataset
0.017
64.3=6.1
74.41:188
78.24:1.7
64.0:1322
91.7-1-0.5
MNIST
99.6::0.0
10
0.17
94.8. 0.7
96.0.10.2
96.410.1
94.910.6
97.4
FashionMNIST
I
0.017
51.3=5.4
59.6::1.3
62.4:099
51.110088
70.5:1-0.6
93.5:20.1
10
0.17
73.8 1.6
76.4
77.610.2
73.9
82.3.10.4
SVHN
1
0.014
14.3=2.11
18.11:099
19.9:0022
16.11:09
31.2-1:14
95.4=001
0.14
34.62.3.2
40.3.1..3
42.1109
33.911.1
76.1
CIARROO
0.02
15.0=200
18.5:10.8
20.1+:0.5
16.34114
28.3:10.55
8..
10
0.2
1.6
29.8:10.7
314102
27.91111
44..10.5
The performance comparison to optimal random selection (ORS) and conditional generative versarial networks (cGAN) baselines. This table shows the testing accuracies (%) of different methods on four datasets. ConvNet is used for training and testing. Img/Cls: image(s) per class, Ratio (%): the ratio of con- densed images to whole training set. Top 1000, Top 100 and Top 10 means the selected 1000, 100 and 10 optimal coresets by ranking their performances.
Table T12:
17

---

Published as a conference paper at ICLR 2021
Corc-set Selection
Ing/Cls
Ratio
LD"
Ours
Whole Datasct
Random
Herding
K-Center
Forgetting
CIFAR100
0.2
4.2:10.3
8.4.0.3
8.3:10.3
3.500.3
115100
12.8-0.3
$6.2.10.3
14.61..55
17.1.0.3
7.140.3
9.8
Table TI3: The performance comparison on CIFARI00. This table shows the testing accuracies (%) of different methods. ConvNet is used for training and testing except that LD' uses AlexNet. Img/Cls: image(s) per class, Ratio (%): the ratio of condensed images whole training set.
Method
MLP
ConvNet
LeNet
AlexNet
VGG
ResNer
DD
72.7122 776.129
79.518.1 51.31 19,9 1144266 63.7112.7
Ours
83.0225 92.900.5 939.00.6
90.6119 92.90..5 94,5:10.4
Table T14:
Generalization ability comparison DD. The 10 condensed images per class trained with LeNet, and tested on various architectures. It shows that condensed images generated by method have better generalization ability.
Random and Herding (Rebuffi et al. 2017 in this experimental setting. After analyzing the algo- rithms and coresets, we find two main reasons. 1) K-Center and Forgetting are not designed for training deep networks from scratch, instead they are for active learning and continual learning spectively. 2) The two algorithms both tend to select "hard" samples which are often outliers when only a small number of images selected. These outliers confuse the training, which results in worse performance. Specifically, the first sample per class K-Center coreset is initialized by se- lecting the closest each class center. The later ones selected by the greedy criterion that pursues maximum coverage are often outliers which confuse the training.
Performance on CIFAR100. We supplement the performance comparison on CIFAR100 dataset which includes 10 times many classes other benchmarks. More classes while fewer images per class makes CIFAR 100 significantly more challenging than other datasets. We use the same set of hyper-parameters for CIFAR100 as other datasets. Table T13 depicts the performances of coreset selection methods, Label Distillation (LD) Bohdal al. (2020) and ours. Our method achieves 12.8% and 25.2% testing accuracies on CIFAR 100 when learning and images per class, which the best compared with others.
D FURTHER COMPARISON TO DD (WANG ET AL., 2018)
Next we compare our method to DD WWang et al. 2018) first quantitatively in terms of cross- architecture generalization, then qualitatively in terms of synthetic image quality, and finally in terms of computational load for training synthetic images. Note that we use the original source code obtain the results for DD that is provided by the authors of DD the experiments.
Generalization ability comparison. Here we compare the generalization ability across different deep network architectures to DD. To this end, we use the synthesized 10 images/class data learned with LeNet on MNIST to train MIP, ConvNet, LeNet, AlexNet, VGG11 and ResNet18 and report the results in Table T14 We see that that the condensed set produced by method achieves good classification performances with all architectures, while the synthetic set produced by DD perform poorly when used to trained some architectures, e.g. AlexNet, VGG and ResNet. Note that DD generates learning rates to be used in every training step addition to the synthetic data. This is in contrast to method which does not learn learning rates for specific training steps. Although the tied learning rates improve the performance of DD while training and testing on the same architecture, they will hinder the generalization unseen architectures.
Method
Dutaset
Architecture
Memory (MB)
Time (min)
Test Acc.
DD
MNIST
LeNet
785
160
79.5::8.1
Ours
MNIST
LeNet
653
46
93.9
0..6
DD
CIFARIO AlexCifarNet
3211
214
36.8:::1.2
AlexCifarNet
39.111.2
Table T15: Time and memory use for training DD and our method in 10 images/class setting.
18

---

Published as a conference paper at ICLR 2021
Hane
Bird
Deer
Dog
lorre
Ship
Truck
(a) MNIST of DD
(b) CIFARIO of DD
Frog
/
2
3
4
5
6
7
7
0
1
3
4
6
3
9
0
2
3
4
5
4
7
9
0
1
3
S
6
7
<
9 9
0
1
2
3
5
6
7
8 2 8
/
2
3
4
5
6
7
9
0
/
>
3
4
5
6
7
0
/
3
4
6
7
8
9 9
0
3
4
6
7
3
4
5
7
9
(c) MNIST of ours
(d) CIFARIO of ours
Figure F8: Qualitative comparison between the condensed images produced by DD and ours under 10 im- ages/class setting. LeNet and AlexCifarNet are utilized for MNIST and CIFAR 10 respectively.
Qualitative comparison. We also provide a qualitative comparison to DD in terms of image quality in Figure F8 Note that both of the synthetic sets are trained with LeNet on MNIST and AlexCifarNet on CIFAR1O. Our method produces more interpretable and realistic images than DD, although it is not our goal. The MNIST images produced by DD are noisy, and the CIFAR 10 images produced by DD do show any clear structure of the corresponding class. In contrast, the MNIST and CIFARIO images produced by our method are both visually meaningful and diverse.
Training memory and time. One advantage of our method is that we decouple the model weights from its previous states in training, while DD requires to maintain the recursive computation graph which is not scalable to large models and inner-loop optimizers with many steps. Hence, our method requires less training time and memory cost. We compare the training time and memory cost re- quired by DD and our method with one NVIDIA 1080-T GPU, Table TI5) shows that our method requires significantly less memory and training time than DD and provides an approxima- tion reduction of 17% and 55% memory and 71% and 51% in train time to learn MNIST and CIFAR10 10 datasets respectively. Furthermore, our training time and memory cost be signifi- cantly decreased by using smaller hyper-parameters, e.g. K, T and the batch size sampled real images, with slight performance decline (refer to Figure F6
19

---

Published as a conference paper at ICLR 2021
E EXTENDED RELATED WORK
Variations of Dataset Distillation. There exists recent work that extends Dataset Distillation (Wang al. 2018). For example, (Sucholutsky & Schonlau 2019 Bohdal
2020 aim to improve DD by learning soft labels with/without synthetic images. (Such et al. 2020 utilizes generator to synthesize images instead of directly updating image pixels. However, the reported quantitative and qualitative improvements over DD are minor compared to our improvements. In addition, none of these methods have thoroughly verified the arositerciee generalization abil- ity of the synthetic images.
Zero-shot Knowledge Distillation. Recent zero-shot KD methods (Lopes et al., 2017: Nayak al.) (201)) aim to perform KD from trained model in the absence of training data by generating synthetic data as the intermediate production to further use. Unlike them, our method does not require pretrained teacher models provide the knowledge, i.e. obtain the features and labels.
Data Privacy & Federated Learning. Synthetic dataset is also promising solution protecting data privacy and enabling safe federated learning. There exists some work that uses synthetic dataset protect the privacy of medical dataset (Li et al.) (020) and reduce the communication rounds in federated learning (Zhou et al. 2020). Although transmitting model weights or gradients (Zhu et al., Zhao et al. 2020) may iecraase the transmission security, the huge parameters of modern deep neura networks are prohibitive to transmit frequently. In contrast, transmitting small-scal synthetic dataset between clients and server is low-cost (Goetz Tewari 2020).
2019
20


Reviews:
[{"review_id": "zPngfLS4lwn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for generating synthetic datasets from the large realworld datasets. The CNN trained on such synthetic dataset supposed to have similar accuracy on the realworld data, as trained on the original one. The benefit of a such procedure is reduced model training time and storage space (for data).\n\n\nThe method is built on the idea that the gradients of the network being trained on the real images should be similar to gradients, which were obtained by the training on the synthetic images.\n   \nThe method is validated on MNIST, SVHN, FashionMNIST and CIFAR-10 on several different architectures: MLP, AlexNet, VGG-like and ResNet architectures. \n\nMoreover, the paper compares the proposed method vs  many other baselines, e.g. methods, which select \"representative\" image from the dataset (coreset methods), as well as Dataset Distillation and cGAN. \n\n*****\nOverall I like the paper a lot. The method is well-motivated, shows good (sota) results and also often (for MNIST, SVHN, FashionMNIST)  produces human-recognizable examples, although there is no term/regularization directly encouraging this. \n\nThe paper also studies how architectural choices like normalization, pooling, etc. influence the generated samples and how samples generated for the architecture A are suitable for training architecture B.\n\nI don't see any major weakness in the paper.\n\n\nQuestions and comments.\n\n- In Figure 5, IMO it is bad practice to fit linear regression into blob-like data. (minor)\n- Have you tried to add some term, encouranding the diversity for the different synthetic samples belonging to the same class? \n\n********\n\nPost-rebuttal. \n\nThe rebuttal didn't raised any concerns and made the paper even stronger, thus I am keeping my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "uh5Ay9WH5Pm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper proposes a novel dataset condensation technique that generates synthetic samples by matching model gradients with those obtained on the original input dataset. This technique is investigated empirically on several smaller datasets like MNIST, SVHN and CIFAR10. Two applications to continual learning and neural architecture search (NAS) are also explored and show some promising results.\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for accepting this paper. The technique is intuitive and well-justified. Experimental results seem to suggest that it produces a synthetic set that compares favorably to those obtained using alternative methods. Also, additional applications of this technique to continual learning and NAS appear to be quite promising.\n\n##########################################################################\n\nPros: \n\n1. The paper is well written. The core idea is arrived at systematically and is carefully explained.\n\n2. The paper does a good job referencing prior work (with most papers that I knew of being included) and empirical results obtained by the authors appear to compare very favorably to this existing prior work. I do not think that presented empirical results are exhaustive, but they are definitely very promising.\n \n##########################################################################\n\nCons.\n\nI did not see any major problems with the paper. But wanted to make a few comments that could potentially be addressed:\n\n1. I found a sentence \"Note that each real and synthetic batch sampled from T and S contains samples from a single class and the synthetic data for each class are separately updated at each iteration\" a bit confusing. Specifically, does this mean that all samples in T and S have the same label? If so, does this mean that in the case when we have only one sample per class, S contains essentially the same input sample (with or without image augmentations depending on the experiment)?\n\n2. While S is built to match the gradients on the original architecture, it is a little counterintuitive that such a small training set would not cause dramatic overfitting on other (possibly \"heavier\") architectures. Do we need to use multiple synthetic sets in practice, or rely on heavy data augmentation to avoid overfitting? Or just limiting the number of training steps would be enough? It would be interesting to see a more detailed discussion of this aspect. While I am encouraged by NAS results, this remains one of my concerns.\n\n3. It would be interesting to see results on larger datasets like ImageNet or at least CIFAR-100. I understand that this exploration would be quite computationally intensive, but this would make the results much more convincing.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above. (I will update the score depending on the authors reply.)\n\n#########################################################################\n\nPost-rebuttal.\n\nThanks for a detailed response that clarified some questions and concerns that I previously had. I think the updated paper is stronger and I am inclined to raise the score to 8.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qpiz_1WvBbe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper tackles the challenging dataset condensation problem. The goal is to learn to synthesize a small dataset, so that a neural network trained on the small synthetic dataset can have similar performance as a network trained on the full dataset. The proposed method tackles the problem by gradient matching. The proposed method achieves state-of-the-art performance, and shows promising results on two other downstream tasks, continual learning and neural architecture search.\n\nStrength:\n+ Comparing to existing approaches, the proposed method is efficient and effective, achieving the state-of-the-art performance\n+ The authors use the synthetic dataset for two other downstream tasks and achieve promising results\n+ The authors conduct extensive experiments to study and analyze the proposed method\n+ The synthetic dataset trained on one architecture can be also used to train any other networks with different architectures, which makes to method more applicable \n\nWeakness:\n- In Section 2.3 “Gradient matching loss”, the authors claim that the proposed distance metric is “a better distance”. It is probably better to use experiments or results to support this claim.\n\nOther comments:\n* Currently, the loss for the target task is cross-entropy loss only (Figure 1(b)). I wonder if this method can be used for other loss functions as well. Also, I wonder if the proposed method can be used for self-supervised tasks as well. Can the authors comment on these?\n* It seems that the authors do not have any special designs to make the condensed synthetic dataset cross-architecture generalizable. I wonder why the proposed method has such a good cross-architecture generalization.\n* In Figure 3, it seems that the performance on CIFAR10 saturates quickly as the number of images per class grows. Just curious, will the relative performance eventually reach 80~90% as in other datasets? If so, then what is the data ratio?\n\n---- Post-rebuttal comments----\n\nThanks for the response. After reading other reviews' comments and the rebuttal, I think this paper is in a good shape now. Thus, I am willing to increase my score to 8 and recommend acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel dataset condensation technique that generates synthetic samples (images) by matching model gradients with those obtained on the original input samples (images). The authors also show that these synthetic images  are not architecture dependent and can be used to train different deep neural networks. The approach is validated on several smaller datasets like MNIST, SVHN and CIFAR10. This work is well-motivated and the methodological contributions convincing. All reviewers were enthusiastic and indicated that there were no flaws in this work. The rebuttal clarified outstanding questions and made the paper stronger.
Decision: Accept (Oral)