Dataset Distillation with Infinitely Wide Convolutional Networks
Authors: Timothy Nguyen
Roman Novak
Lechao Xiao
Jaehoon Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: dataset distillation
infinite-width limit
convolutional neural networks
kernel ridge-regression
meta-learning
feature learning

Abstract: The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel-based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10  datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.

Dataset Distillation with Infinitely Wide Convolutional Networks
Timothy Nguyen'
Roman Novak
Lechao Xiao
Jaehoon Lee"
DeepMind
Google Research, Brain Team
cimotaayagggyyea@eeppnn....mom com
{romann, xlc, jaehles) com
Abstract
The effectiveness machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply novel distributed cernel-based meta-learning framework to achieve state-of-the-ane results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR- image classification task, dramatic improvement over the previous best test accuracy of 40%. Our ofte-offthhe-att results extend across many other settings for MNIST, Fashion-MNIST. CIFAR-10, CIFAR-I00 and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets shed light on how they differ from naturally occurring data.
Introduction
Deep learning has become extraordinarily successful a wide variety of settings through the availability of large datasets [Krizhevsky al., 2012, Devlin et al., 2018, Brown et al., 2020, Dosovitskiy et al., 2020]. Such large datasets enable a neural network to learn useful representations of the data that are adapted solving tasks of interest. Unfortunately, it can be prohibitively costly to acquire such large datasets and train neural network for the requisite amount of time.
One way to mitigate this problem is by constructing smaller datasets that are nevertheless informative. Some direct approaches to this include choosing a representative subset of the dataset (i.e. a coreset) or else performing low-dimensional projection that reduces the number of features. However, such methods typically introduce a tradeoff between performance and dataset size, since what they produce is coarse approximation of the full dataset. By contrast, the approach of dataset distillation is to synthesize datasets that are more informative than their natural counterparts when equalizing for dataset size [Wang et al., 2018, Bohdal et al., 2020, Nguyen al., 202 Zhao and Bilen, 2021]. Such resulting datasets will not arise from the distribution of natural images but will nevertheless capture features useful to a neural network, capability which remains mysterious and is far from being well-understood [Ilyas et al., 2019, Huh et al., 2016, Hermann and Lampinen, 2020].
The applications of such smaller, distilled datasets are diverse. For nonparametric methods that scale poorly with the training dataset (e.g. nearest-neighhors or kernel-ridge regression), having a reduced dataset decreases the associated memory and inference costs. For the training of neural networks, such distilled datasets have found several applications in the literature, including increasing the
Work done while at Google Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

effectiveness of replay methods in continual learning [Borsos et al., 2020] and helping to accelerate neural architecture search [Zhao et al., 2021, Zhao and Bilen, 2021].
In this paper, we perform a large-scale extension of the methods of Nguyen et al. [2021] to obtain new state-of-the-art (SOTA) dataset distillation results. Specifically, apply the algorithms KIP (Kernel Inducing Points) and LS (Label Solve), first developed in Nguyen et al. [2021], to infinitely wide convolutional networks by implementing novel, distributed meta-lcarning framework that draws upon hundreds of accelerators per training. The need for such resources is necessitated by the computational costs of using infinitely wide neural networks built out of components occurring in modern image classification models: convolutional and pooling layers (see $B for details). The consequence that we obtain distilled datasets that are effective for both kernel ridge-regression and neural network training.
Additionally, we initiate a preliminary study of the images and labels which KIP learns. We provide a visual and quantitative analysis of the data learned and find some surprising results concerning their interpretability and their dimensional and spectral properties. Given the efficacy of KIP and LS learned data, we believe a better understanding of them would aid in the understanding of feature learning in neural networks.
To summarize, our contributions are as follows:
1. We achieve SOTA dataset distillation results on a wide variety of datasets (MNIST, Fashion- MNIST, SVHN, CIFAR-10, 10, CIFAR-100) for both kernel regre-oen and neural network training. In several instances, our results achieve an impressively wide margin over prior art, including over 25% and 37% absolute gain in accuracy on CIFAR-10 and SVHN image classification, respectively, when using only 10 images (Tables 2, A11).
2. We develop novel, distributed meta-learning framework specifically tailored the compu- tational burdens of sophisticated neural kernels ($2.1).
3. We highlight and analyze some of the peculiar features of the distilled datasets we obtain, illustrating how they differ from natural data ($4).
4. We open source the distilled datasets, which used thousands of GPU hours, for the re- search community to further investigate at https:/ thu.. com/ wtt. -eam/goggl--eeerchh google-research/ research/tree httt///eeeeem//eettt/tteeee master/kip.
2 Setup
Background on infinitely wide convolutional networks. Recent literature has established that Bayesian and gradient-descent trained neural networks converge to Gaussian Processes (GP) as the number of hidden units in intermediary layers approaches infinity (see $5). These results hold for many different architectures, including convolutional networks, which converge to a particular GP in the limit of infinite channels [Novak al., 2019, Garriga-Alonso et al., 2019, Arora et al., 2019]. Bayesian networks are described by the Neural Network Gaussian Process (NNGP) kernel, while gradient descent networks are described by the Neural Tangent Kernel (NTK). Since we interested in synthesizing datasets that can be used with both kernel methods and common gradient-descen trained neural networks, focus on NTK in this work.
Infinitely wide networks have been shown to achieve SOTA (among non-parametric kernels) results on image classification tasks [Novak et al., 2019, Arora et al., 2019, Li et al., 2019, Shankar et al., 2020, Bietti, 2021] and even rival finite-width networks in certain settings [Arora et al., 2020, Lee et al., 2020]. This makes such kernels especially suitable for our task. As convolutional models, they encode useful inductive biases of locality and translation invariance [Novak et al., 2019], which enable good generalization. Moreover, flexible and efficient computation of these kernels are possible due to the Neural Tangent library [Novak et al., 2020].
Specific models considered. The central neural network (and corresponding infinite-width model) we consider a simple 4-layer convolutional network with average pooling layers that we refer to as ConvNet throughout the text. This architecture is a slightly modified version of the default model used by Zhao and Bilen [2021], Zhao et al. [2021] and was chosen for ease of baselining (see SA for details). In several other settings we also consider convolutional networks without pooling

---

layers ConvVec.
and networks with no convolutions and only fully cully-connected layers FC. Depth of architecture (as measured by number of hidden layers) indicated by an integer suffix.
Background on algorithms. We review the Kernel Inducing Points (KIP) and Label Solve (L.S)
algorithms introduced by Nguyen et al. [2021]. Given kernel K, the kernel ridge-regression (KRR) loss function trained on a support dataset (X,,. Us) and evaluated on a target dataset (X,,8t) is
L L( v.) 2 llv. Kx,, (Kx,x, + +XII"'y.1 9.112-
where U and V are sets, Kuv is the matrix of kernel elements (K(u) (u, u) wet..ev. Here À > 0 is fixed regularization parameter. The KIP algorithm consists of minimizing (1) with respect to the support set (either just the X, or along with the labels y.). Here, we sample (X,, y1 from target dataset at every (meta)step, and update the support set using gradient-basse methods. Additional variations include augmenting the X, or sampling different kernel K (from a fixed family of kernels) at each step.
The Label Solve algorithm consists of solving for the least-norm minimizer of (1) with respect to ys- This yields the labels
y² = (Kxxx + yr.
where A denotes the pseudo-inverse of the matrix A. Note that here (X,, yt) = D. i.c. the labels solved using the whole target set.
In our applications of KIP and Label Solve, the target dataset always significantly larger than the support set (X,, V.). Hence, the learned support set or solved labels can be regarded as distilled versions of their respective targets. We also initialize our support images to be a subset of natural images, though they could also be initialized randomly.
Based the infinite-widhh correspondence outlined above and in $5, dataset distillation using KIP or LS that is optimized for KRR should extend to the corresponding finite-widhh width neural network training. Our experimental results in $3 validate this expectation across many settings.
2.1 Client-Server Distributed Workflow
We invoke a client-server model of distributed computation" in which server distributes independent workloads to large pool of client workers that share a queue for receiving and sending work. Our distributed implementation of the KIP algorithm has two distinct stages:
Forward pass: In this step, we compute the support-support and target-support matrices K(X,, X²) and K (X,, X,). To do so, we partition X, x X, and X, X X, into pairs of images (x, x'', ,2),, each with batch size B. We send such pairs to workers compute the respective matrix block K ((x,z) z¹). The server aggregates all these blocks to obtain the K K(X,, X,) and K(X, X.) matrices.
Backward pass: In this step, we need to compute the gradient of the loss I. (1) with respect to the support set Xg. We need only consider 0 since ÖT, is cheap to compute. By the chain rule, we can write
AL OK(X,, 0x, O(K(X,,X, X.)) ax, O(K(,,, 0x,
ØL
OL OK(X,,
The derivatives of L with respect to the kernel matrices are inexpensive, since L depends in simple way on them (matrix multiplication and inversion). What is expensive to compute is the derivative of the kernel matrices with respect to the inputs. Each kernel element is an independent function of the inputs and a naive computation of the derivative of a block would require forward- mode differentiation, infeasible due to the size of the input images and the cost compute the individual kernel elements. Thus our main novelty is to divide up the gradient computation into backward differentiation sub-computations, specifically by using the built-in function j jax. vjp in JAX [Bradbury et al., 2018]. Denoting K K(X,, X.) or K K(X,, X.) for short-hand, we divide the
"The abbreviation "Vec" stands for vectorization, to indicate that activations of the network are vectorized (flattened) before the top fully-connected layer instead of being pooled. Implemented using Courier available https:/ / //gi git..b.... com http///eppmmnnddeede www.h//dddaaaahaaa..a
3

---

Table 1: Comparison with other methods. The left group consists of neural network based methods. The right group consists of kernel ridge-regression All settings KIP involve use label-learning Grayscale datasets use standard channel-wise preprocessing while RGB datascts use regularized ZCA preprocessing.
Imgs/ Class
DC'
DSA
KIP FC' aug
LS ConvNet 2.3
KIP ConvNet"
no aug
aug
91.7+0.5
88.7+0.6
85.5+0.1
73.4
97.3+0.1
96.5+0.1
MNIST
10
97.4
97.8+0.1
97.2+0.2
96.4
99.1=0.1
99.1=0.11
50
98.8+0.1
99.2#0.1
98.4+0.1
98.3
99.4=0.1
99.5
70.5
70.6+0.6
65.3
82.9+0.2
76.7+0.2
Fashion-
10
82.340.4
84.6+0.3
80.8
91.0=0.1
88.8+0.1
MNIST
50
83.640.4
88.7+0.2
86.9
92.4+0.1
91.0+0.1
1
1..2+114
1..5+14
23.9
62.4+0.2
64.3+0.4
SVHN
76.1
79.2+0.5
52.8
79.3+0.1
81.1
50
82.3+0.3
84.410.4
76.8
82.0+0.1
84.3+0.1
1
2..
28.8+0.7
40.
26.1
64.7+0.2
63.4+0.1
CIFAR-10
10
44.940.5
52.1
$3.1
53.6
75.610.2
75.5+0.1
50
53.9
60.6+0.5
58.6+0.4
65.9
78.2+0.2
80.6+0.1
12.8+0.3
13.9+0.3 32.3+0.3
23.8
34.9
33.3+0.3
CIFAR-100
10
25.240.3
39.2
47.9+0.2
49.5
DC [Zhao ul., 2021]. DSA [Zhao and Bilen, 2021]. KIP FC [Nguyen el al., 2021].
Ours.
LD [Bohdal et al., 2020] is another baseline which distills only labels using the AlexNet architecture. Our LS achieves higher test accuracy than theirs in every dataset category.
matrix 0T/OKK, computed on the server, into B x blocks corresponding to aL/aK s'), where r and a' each have batch size B. We send each such block, along with the corresponding block of image data (z, z') to a worker. The worker then treats the åL/aK (z/kkKz,a , 2') receives as the cotangent vector argument of jax. jax.vjp that, via contraction, converts the derivative of K(x, .²) with respect into scalar. The server aggregates all these partial gradient computations performed by the workers, over all possible x B blocks, to compute the total gradient /L/ 0x,, used to update X..
Experimental Results
3.1 Kernel Distillation Results
We apply the KIP and LS algorithms using the ConvNet architecture on the datasets MNIST [LeCun et al., 2010], Fashion MNIST [Xiao et al., 2017], SVHN [Netzer et al., 201 1]. CIFAR-10 (Krizhevsky, 2009], and CIFAR-100. Here, the goal is condense the train dataset down to a learned dataset of size 1, 10, or 50 images per class. We consider a variety of hyperparameter settings (image preprocessing method, whether augment target data, and whether train the support labels for KIP), the full details of which are described in SA For space reasons, we show a subset of our results in Table 1, with results corresponding the remaining set of hyperparameters left to Tables 13-A10. We highlight here that a crucial ingredient for our strong results in the RGB dataset setting the use of regularized ZCA preprocessing. Note the variable effect that our augmentations have on performance (see the last two columns of Table I): they typically only provides a benefit for sufficiently large support set. This result is consistent with Zhao al. [2021]. Zhao and Bilen [2021]] in which gains from augmentations are also generally obtained from larger support sets. We tried varying the fraction (0.25 and 0.5) of each target batch that augmented at each training step and found that while 10 images still did best without augmentations, 100 and 500 images typically did slightly better with some partial augmentations (versus none or full). For instance, for 500 images on CIFAR-10, 10, we obtained 81.1% %% test accuracy using augmentation rate 0.5. Thus, our observation is that given that larger support sets can distill larger target datasets, as the former increases in size, the latter can be augmented more aggressively for obtaining optimal generalization performance.

---

Korne Samp NN Trander (NSE)
Kome Sampling NN Transter (XENT)
Teat Acturacy
0.4
0.5
Conoliet
Cony-Voc
KIP Traning Stepe
KIP Traning Steps
0.3
Condiet Traiing NN Transfer (MSE)
Conyler Traiing NN Transkur (XENT)
0.2
Train kemnls
Toul Amuuvey
0.4
Conwitet
Con--Vec3
0.0
ConviNet
Voca
Venv
Test kernel
Traning
Training Stops
Figure 1: KIP with kernel sampling individual kernels. Left: Evaluation of three kernels, ConvNet, Conv-Vec3. Conv-Vect for KRR with respect to four train settings: sampling KIP ("all") which uses all the kernels else KIP trained with the individual kernels. For all three kernels, "all" is a close second place, outperformed only if the kernel used for training is exactly the same as the one used for testing. Right: We take the learned images of the four train settings described above and transfer them to finite-width neural networks corresponding to ConvNet, Conv-Vec3. Conv- Vec8. Each point is a neural network trained on specified KIP learned checkpoint. Top row sampling KIP images and bottom row is the baseline using just ConvNet for KIP. These plots indicate that sampling KIP improves performance across the architectures that are sampled, for both MSE and cross entropy loss. Settings: CIFAR-10, 100 images, no augmentations, no 7CA, no label learning.
Remarkably, our results in Table outperform all prior baselines across all dataset settings. Our results are especially strong in the small support size regime, with our 1 image per class results for KRR outperforming over 100 times as many natural images (see Table A1). We also obtain significant margin over prior art across all datasets, with our largest margin being a 37% absolute gain in test accuracy for the SVHN dataset.
3.2 Kernel Transfer
In $3.1, we focused obtaining state-of-the-art dataset distillation results for image classification using a specific kernel (ConvNet). Here, we consider the variation of KIP in which we sample from a family of kernels (which we call sampling KIP) We validate that sampling KIP adds robustness to the learned images in that they perform well for the family of kernels sampled during training.
In Figure 1, we plot test performance of sampling KIP when using the kernels ConvNet, Conv-Veck and Conv-Vec8 (denoted by "all") alongside KIP trained with just the individual kernels. Sampling KIP performs well at test time when using any of the three kernels, whereas datasets trained using single kernel have significant performance drop when using a different kernel.
3.3 Neural Network Transfer
In this section, we study how our distilled datasets optimized using KIP and L. transfer to the setting of finite-width neural networks. The main results are shown in Table 2. The third column shows the best neural network performance obtained by training with respect to some choice of KIP and neural network training hyperparameters (see ŠA for details). Since the datasets optimized for kernel ridge-regression and not for neural network training itself, we expect some performance loss when transferring to finite-wdtth networks, which we record the fourth column. Remarkably, the drop due to this transfer is quite moderate or small and sometimes the transfer can even lead to gain in performance (see LS for SVHN dataset with 10 images per class).
KIP dataset of the corresponding size
Overall, our transfer to finite-width networks outperforms prior art based on DC/DSA [Zhao et al., 2021, Zhao and Bilen, 2021] in the image per class setting for all the RGB datasets (SVHN, CIFAR-10. 10, CIFAR- 100). Moreover, for CIFAR-10, outperform DC/DSA in all settings.
5

---

Table 2: Transfer of KIP and LS to neural network training. Datasets obtained from KIP and LS using the ConvNet kernel are optimized for kernel ridge-regression and thus have reduced performance when used for training the corresponding Unite-width ConvNet neural network. Remarkably, the loss performance is mostly moderate and even small in many instances. Grayscale datasets use standard channel -wise preprocessing while RGB datasets use regularized ZCA preprocessing. The KIP datasets used here can have augmentations or no augmentations and, unlike those in Table I, can have either fixed or learned labels. * denotes best chosen transfer is obtained with learned labels.
Perf. change
Perf. change
Imgs/Class
DC/DSA
KTP to NN
LS to NN
91.7+
90.1+0.1
-5.5
71.0+0.2
-2.4
MNIST
10
97.8+0.1
97.5+0.0
-1.1
95.2+0.1
-1.2
50
99.240.1
98.3+0.1
-0.8
97.9+0.0
-0.4
70.640.6
73.5
-9.8
61.2+0.1
-4.1
Pashion-MNIST
10
84.610.3
86.810.1
-1.3
79.7+0.1
-1.2
50
88.7+0.2
88.0+0.1
-4.5
85.0+0.1
-1.8
3..2+1.4
57.3+0.1"
-8.3
23.8+0.2
-0.2
SVHN
10
79.2
75.0+0.1
-1.6
53.240.3
0.4
50
84.410.4
80.5+0.1
-1.0
76.5+0.3
-0.4
28.8+0.7
49.910.2
.9.2
24.7#0.1
-1.4
CIFAR-10
10
52.1
62.7+0.3
-4.6
49.3+0.1
-4.3
50
60.6+0.5
68.640.2
-4.5
62.0+0.2
-3.9
13.940.3
15.7+0.2
-18.1
11.8+0.2
-12.0
CIFAR-100
32.3+0.3
28.3+0.1
-17.4
25.0+0.1
-14.2
Figures 2 and 3 provide closer look at KIP transfer changes under various settings. The first of these tracks how transfer performance changes when adding additional layers as function the number of KIP training steps used. The normalization layers appear to harm performance for MSE loss, which can be anticipated from their absence in the KIP and LS optimization procedures. However they appear to provide some benefit for cross entropy loss. For Figure training progresses, the downstream finite-width network's performance also improves in general. A notable exception is observed when learning the labels in KTP, where longer training steps lead to deterioration of information useful to training finite-width neural networks. We also observe that as predicted by infinite-widde theory [Jacot al., 2018, Lee et al., 2019], the overall gap between KIP or LS performance and finite-width neural network decreases as the width increases. While our best performing transfer is obtained with width 1024, Figure (middle) suggest that even with modest width of 64, our transfer can outperform prior art of 60.6% by Zhao and Bilen [2021].
observe that KIP
Finally, in Figure 4, investigate the performance of KIP images a single run, as compared to natural images, over range of hyperparameters. We find the outper- formance of KIP images above natural images consistent across hy perparammesess and checkpoints. This suggests that our KIP images may also be effective for accelerated hyperparameter search, an application of dataset distillation explored in Zhao al. [2021], Zhao and Bilen [2021].
the course of training of
Altogether, we find our neural network training results encouraging. First, it validates the applicability of infinite-width methods to the setting of finite width [Huang and Yau, 2020, Dyer and Gur-Ari, 2020, Andreassen and Dyer, 2020, Yaida, 2020, Lee et al., 2020]. Second, we find some of the transfer results quite surprising, including efficacy of label solve and the use of cross-entropy for certain settings (see SA for the full details).
Understanding KIP Images and Labels
natural question to consider is what causes KIP to improve generalization performance. Does simplify support images, removing noise and minor sources of variation while keeping only the core features shared between many target images of a given class? does it make them more complex by producing outputs that combine characteristies of many samples in single resulting collage? While

---

Compare ONN Archiletures Thansfer (MSE)
Compare Archileck.es: NN (XENT)
0.75
Text Antuney
0.05
0.60
0.50
Base
Batch-Nam
LayerNomm Norm
Myrtie
Instance Norm
KIP Width
10
10
10'
10
10P
10"
10"
KIP Training Staps
KIP Training Steps
Figure 2: Robustness to neural network variations. KIP ConvNet images (trained with fixed labels) are tested on variations of the ConvNet neural network. including those which have vari- ous normalization layers (layer, instance, batch). A similar architecture to ConvNet, the Myrtle5 architecture (without normalization layers) [Shankar et al., 2020], which differs from the ConvNet architecture by having an additional convolutional layer at the bottom and global average pooling that replaces the final local average pooling at the top, is also tested. Finally, mean-square error is compared with cross-entropy loss (left versus right), Settings: CIFAR-10, 500 images, ZCA, no label learning.
Comnect CIFAR-10
Conviet CIFAR110 10 Translervs Stadaadd
Conshet CIRAR-10
Eiaiiing Transfer
Labo Soleo Transfor v Standard
0.3
1024
KP Steps
Plumbr Channel))
(Number Charnel))
Figure 3: Variations for neural network transfer. Left: Plot of transfer performance a function of KIP training steps across various train settings. Here (a) denotes augmentations used during KIP training and (a+l) denotes that additionally the labels were learned. MSE and XENT denote mean-square-error and cross entropy loss for the neural network, where for the case of XENT and (a+l), the labels for the neural network are the argmax of the learned labels. Middle: Exploring the effect of width on transferability of vanilla KIP data. Right: The effect of width on the transferability of label solved data. Settings: CIFAR-10. 10, 500 images, ZCA.
0.7
CIFAR 10 suppont S.Ne 50rthags:
CIRAR-O ogeer 002
OA 15 support 9ondrrig 57
CIFAP- SOPAP-O-AA wwp.rrpeerooooo..oo.oooooooo caN (2eibogg 112
0.6
0.5-
0.4
x NNNeRc66 KDNT
xom
0.3
0.4
0.6
07
0s
04
04
07 63
0.
05 64
Nezural TeetAooray
hakore Acuuaay
Natusl Arcuacyy
Naaur AestAcuuyy
C1744-10 suport 20
CIPAB-12 support tbe Soodheipi 1000
c CFA Stodholgt 5000
CI suppor Scolltiii 50000
0.4
0.5-
MEP No ALD. 5002 yot MiriM DUB
Morppo Shtera RLB
UPP por 3Nimral 253)
03
04
0.5 Noosa AocAAooett
0.5 06
04 05 Natusl Arcuray
07
04
Natura Tecraayy
Natural Tor.Acouscs
Figure 4: Hyperparameter robustness. In the above, KIP images across eight different checkpoints are used to train the ConvNet neural network. Each point in each plot is a neural network training with different hyperparameter, and its location records the final test accuracy when training natural images versus the KIP images obtained from initializing from such images. For both MSE and cross entropy loss, KIP images consistently exceed natural images across many hyperparameters. Settings: CIFAR-10 10, 500 images, ZCA, no augmentations, no label learning.
7

---

CIFAR-100
apple
aquarlum_ fish
baby
bear
beaver
bed
bee
beetle
blcycle
bottle
Trained Init
Figure 5: Examples of learned images. Images are initialized from natural images in the top row (Init) and converge to images in the bottom row (Trained). Settings: 100 images distilled, no ZCA, no label training, augmentations.
properly answering this question is subject to precise definitions of simplicity and complexity, we find that KIP tends to increase the complexity of pictures based on the following experiments:
Visual analysis: qualitatively, KIP images tend to be richer in textures and contours.
Dimensional analysís: KIP produces images of higher intrinsic dimensionality (i.e. an estimate of the dimensionality of the manifold on which the images lie) than natural images.
Spectral analysis:
unlike natural images, for which the bulk generalization performance is explained by a small number of top few cigendirections, KIP images leverage the whole spectrum much more evenly.
Combined, these results let us conclude that KIP usually increases complexity, integrating features from many target images into much fewer support images.
Visual Analysis. A visual inspection of our learned data leads to intriguing observations in terms of interpretability. Figure 5 shows examples of KTP learned images from CIFAR-100. The resulting images are heterogeneous in terms of how they can be interpreted as distilling the data. For instance, the distilled apple image seems consist of many apples nested within possibly larger apple, whereas the distilled bottle image starts off as two bottles and before transforming into one, while other classes (like the beaver) are altogether visually indistinct. Investigating and quantifying aspects that make these images generalize so well is a promising avenue for future work. We show examples from other datasets in Figure A2.
In Figure 6, we compare MNIST KIP data learned with and without label learning. For the latter case with images and labels optimized jointly, while labels become more informative, encoding richer inter-class information, the images become less interpretable. This behavior consistently leads to superior KRR results, but appears not be leveraged as efficiently in the neural network transfer setting (Table 2). Experimental details can be found in SA.
Dimensional Analysis. We study the intrinsic dimension of KIP images and find that they tend to grow. Intrinsic dimension (ID) was first defined by Bennett [1969] as "the number of free parameters required in hypothetical signal generator capable of producing a close approximation to each signal in the collection". In our context, it is the dimensionality of the manifold embedded into the image space which contains all the support images. Intuitively, simple datasets have low ID as they be described by small number of coordinates on the low-dimensional data manifold.
ID can be defined and estimated differently based on assumptions on the manifold structure and the probability density function of the images on this manifold (see [Camastra and Staiano, 2016] for review). We the "Two-NN" method developed by [Facco et al., 2017], which makes relatively few assumptions and allows to estimate ID only from two nearest-neighbor distances for each datapoint.
Figure shows that the ID is increasing for the learned KIP images function of the training step across a variety of configurations (training with or without augmentations/llael learning) and datasets. One might expect that a distillation procedure should decrease dimensionality. On the other hand, Ansuini et al. [2019] showed that ID increases the earlier layers of a trained neural network. remains to be understood if this latter observation has any relationship with our increased ID. Note that ZCA preprocessing, which played an important role for getting the best performance for our RGB datasets, increases dimensionality of the underlying data (see Figure A4).

---

MNIST
Label Imaages
3
Init
Trained Trained images only
labels and images
0123436789
0123456789
0123456759
0123156789
0123456789
Figure 6: Dataset distillation with trainable and non-trainable labels. Top row: initialization of support images and labels. Middle row: trained images if labels remain fixed. Botton row: trained images and labels, jointly optimized. Settings: 100 images distilled, no augmentations.
Dataset
Images intrinaic Dimension
GIFARTO
FASHION MNIST
MNIST
SVHN GROPPED
60-
Data Augmentation
20-
305
5,000
335 6,000
335 6,000
335 6,000
Training step
Training step
Training stop
Training
Figure 7: Intrinsic dimension of a learned datasets grows during training. As training progresses, intrinsic dimension of the learned dataset grows, indicating that training non-trivially transforms the data manifold. See Figures A2 and 6 for visual examples of learned images, and Figures A3 and A4 for similar observations using other metrics and settings. Settings: 500 images distilled, no ZCA.
Spectral Analysis. Another distinguishing property of KIP images is how their spectral components contribute to performance. In Figure 8, we spot how different spectral bands of KIP learned images affect test performance compared to their initial natural images. Here, and ConvNet architectures. We note that for natural images (light bars), most of their performance captured by the top 20% of eigenvalues. For KIP images, the performance is either more evenly distributed across the bands (FC and Conv-Vec8) or else is skewed towards the tail (ConyNet).
the FC2, Conv-Vecte
Related Work
Dataset distillation was first studied in Wang al. [2018]. The work of Sucholutsky and Schonlau [2019], Bohdal et al. [2020] build upon it by distilling labels. Zhao et al. [2021] proposes condensing training set by harnessing a gradient matching condition. Zhao and Bilen [2021] takes this idea further by applying a suitable augmentation strategy. Note that while Zhao and Bilen [2021] is limited augmentation expressiveness (they have to apply single augmentation per training iteration), we can sample augmentations independently per image in our target set per train step. Our work together with Nguyen et al. [2021] are, to the best of our knowledge, the only works using kernel-based methods for dataset distillation image classification datasets.

---

Relative Performance using Banded Eigenvectors
1.0
FCNaturrl
FCXIP
Relative Performance
Con- Nactauralllural
0.8
Con-VeecKPP
Natural
0.6
KiP
0.4
0.2
0.0
ToP 0-20%
Top 20400%
Top 40-60%
Top 80-80%
80- 80-100%
Figure 8: Spectral contribution to test accuracy shifts to the tail. By setting the param--r to zero kernel-ridge regression and composing Kx,x, with the spectral projection onto various eigenspaces, we can explore how different spectral bands affect test accuracy of kernel ridge- regression. We plot the relative change in test accuracy using contiguous bands of 20% of the eigenvalues. Settings: CIFAR-10, 500 images. Further details in $A
Our use kernels stems from the correspondence between infinitely-wide neural networks and kernel methods [Neal, 1994, Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018, Novak et al., 2019, Garriga-Aonso et al., 2019, Arora et al., 2019, Yang. 2019a,b, Hron et al., 2020] and the extended cor- respondence with the finite width corrections [Dyer and Gur-Ari, 2020, Huang and Yau, 2020, Yaida, 2020]. These correspondences underlic the transferability of our KRR results to neural networks, and have been utilized in understanding trainability [Xiao et al., 2020], generalizations [Adlam and Pennington, 2020], training dynamics [Lewkowycz et al., 2020, Lewkowycz and Gur-Ari, 2020], uncertainty [Adlam et al., 2021], and demonstrated their effectiveness for smaller datasets [Arora et al., 2020] and neural architecture search [Park et al., 2020, Chen et al., 2021].
6 Conclusion
We performed an extensive study of dataset distillation using the KIP and LS algorithms applied to convolutional architectures, obtaining SOTA results variety of image classification datasets. In some cases, our learned datasets were more effective than a natural dataset two orders of magnitude larger in size. There are many interesting followup directions and questions from work:
First, integrating efficient kernel- aamnolllppooiiiioonn methods into our algorithms, such as those of Zandich et al. [2021], will reduce computational burden and enable scaling up to larger datasets. In this direction, the understanding when optimizing for neural network performance has received significant attention machine learning models continue to stretch computational limits [Hestness et al., 2017, Rosenfeld et al., 2020, Kaplan al., 2020, Bahri et al., 2021]. Developing understanding of how to harness smaller, yet more useful representations data would aid in such endeavors. In particular, it would be especially interesting to explore how well datasets be compressed they scale up in size.
how various resources (e.g. data, parameter count, compute) scale
Second, LS and KIP with label learning shows that optimizing labels a very powerful tool for dataset distillation. The labels we obtain are quite far away from standard, interpretable labels and we feel their effectiveness suggests that understanding of how to optimally label data warrants further study.
Finally, the novel features obtained by our learned datasets, and those of dataset distillation methods in general, may reveal insights into interpretability and the nature of sample-effe -efilelfffcientttt representations. For instance, observe the bee and bicycle images Figure 5: the bee class distills into what appears to be spurious visual features (e.g. pollen), while the bicycle class distills to the essential contours of a typical bicycle. Additional analyses and explorations of this type could offer insights into the perennial question of how neural networks learn and generalize.
10

---

Acknowledgments We would like to acknowledge special thanks to Samuel S. Schoenholz, who proposed and helped develop the overall strategy for our distributed KIP learning methodology. We are also grateful to Ekin Dogus Cubuk and Manuel Kroiss for helpful discussions.
References
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training deep bidirectional transformers for language understanding, arXiv preprint arXiv:18/0.04905, 2018.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers image recognition scale. arXiv preprint arXiv:2010.1192) 2020.
Tongzhou Wang, Van Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.1095, 2018.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006. 08572, 2020.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, 2021.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019.
Minyoung Huh, Pulkit Agrawal, and Alexei Efros. What makes imagenet good for transfer learning? arXiv preprint 1608 08614, 2016.
Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006.124: 12433, 2020.
Zalán Borsos, Mojmir Mutný, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming, arXiv preprint 2r06v:2006. 03875, 2020.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang. Jiri Hron, Daniel A. Abolafia. Jeffrey Pennington, and Jascha Dickksien.. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
Adrià Marriga-Alonso Laurence Aitchison, and Carl Edward Rasmussen. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. In Advances in Neural Information Processing Systems, pages 8141-8150. Curran Associates, Inc., 2019.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kemnlss. arXiv preprint arXiv: 1000009, 2019.
Vaishaal Shankar, Alex Chengyu Fang, Wenshuo Guo, Sara Fridovich-Kae Ludwig Schmidt, Jonathan Ragan- Kelley, and Benjamin Recht. Neural kernels without tangents. In International Conference on Machine Learning, 2020.
Alberto Bietti. Approximation and learning with deep convolutional models: a kemel perspective. arXiv preprint axXiv:2102.10032. 2021.

---

Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets small-data lasks. In International Conference on Learning Representations, 2020. URL
Jachoon Lee, Samuel S Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: empirical study. In Advances Neural Information Processing Systems, 2020.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks python. International Conference on Learning Representations, 2020. URL https: //
//tttp://github /ttphbb.ttttuuttg// cww/cooo/ooo/.ooo grogle/neural. neural- -taggetts.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milnee JAX: composable transformations Python+NumPy programs, 2018. URL http://githuue com/ google/jaz.
Yann LeCun, Corinna Cortes, and Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: 2, 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemell: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenbolz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advonces in Neural Information Processing Systems, 2019.
Jiaoyang Huang and Horn--Tzar Yau. Dynamies of deep neural networks and neural tangent hierarchy. In International Conference on Machine Learning, 2020.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In International Confer- ence on Learning Representations, 2020. URL https://openrevieg
Anders Andreassen and Ethan Dyer. Asymptotics of wide convolutional neural networks. arxiv preprint arXiv:2008. 08675, 2020.
Sho Yaida. Non-Gaussian processes and neural networks at finite widths. Mathematical and Scientific
Machine Learning Conference, 2020.
R. Bennett The intrinsic dimensionality signal collections. IEEE Transactions on Information Theory, 15(5): 517-525, 1969. doi: 10.1109/TTT.1961 1054365..
Francesco Camastra and Antonino Staiano. Intrinsic dimension estimation: Advances and open problems. Information Sciences, 328:26-41, 2016. ISSN 0020-0255. doi: https///dbi.rr//1..000jjjji in..::m................... URL https: hwwp://wwwww./
Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):18 8, 2017.
A. Ansuini, Laio, Macke, and Zoccolan. Intrinsic dimension of data representations in deep neural networks. In NeurlPS, 2019.
Ilia Sucholutsky and Matthis Schonlau. Soft-label dataset distillation and text dataset distillation. arXiv preprint arXiv:1910.02551. 2019.
Radford M. Neal. Priors for infinite networks (tech. rep. no. crg 94-11), University of Toronto, 1994.
Jachoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl-dickstein. Deep neural networks gaussian processes. In Internutional Conference Learning Representations. 2018.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations. 2018.
12

---

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural langent kernel derivation. arXiv preprint arXiv:1902.04760, 2019a.
Greg Yang. Wide feedforward recurrent neural networks of any architecture gaussian processes. In Advances in Neural Information Processing Systems, 2019b.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: NNGP and NTK for deep attention networks. In International Conference Machine Learning, 2020.
Lechao Xiao, Jeffrey Pennington, and Samuel S Schoenholz. Disentangling trainability and generalization in deep learning. In International Conference Machine Learning, 2020.
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference on Machine Learning, pages 74-84. PMLR, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase deep learning: catapult mechanism. arXiv preprint arXiv:2003.022 0.20000,.222,,... 2020.
Aitor Lewkowycz and Guy Gur-Ari. On the training dynamics of deep networks with 1_2 regularization. In Advances Neural Information Processing Systems, volume 33, pages 4790-4799, 2020.
Ben Adlam, Jachoon Lee, Lechao Xiao, Jeffrey Pennington, and Jasper Snoek. Exploring the uncertainty properties neural networks' implicit priors infinite-width limit. International Conference on Learning Representations, 2021.
Daniel Park, Jaehoon Lee, Daiyi Peng. Yuan Cao, and Jascha Sohl-Dickstein. Towards ungp-guided neural architecture search. arXiv preprint arXiv:201 2020.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet four (gpu) hours: A theoretically inspired perspective. In International Conference on Learning Representations, 2021.
Amir Zandieh, Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, Jinwoo Shin. Scaling neural tangent kernels via sketching and random features. In Advances in Neural Information Processing Syxtems, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00409, 2017. URL http: /ttp///axvvvv.v. org/abs/ / 111.1.11.000 00409.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference Learning Representations, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXtv:2 102006701, 2021.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jascha
Roman Novak, Samuel Schoenholz, and Jachoon Lec. On the infinite width limit of neural networks with standard parameterization. arXiv preprint arXiv:2001.07300 2020.
13


Reviews:
[{"review_id": "XQdEY320lYw", "reviewer": "Reviewer_Vxyf", "summary": "The authors proposed a dataset distillation method using Kernel Inducing Points (KIP) and Label Solve (LS). The proposed method was designed for infinitely wide convolutional networks, however, when training a finite-width neural network model with the compressed dataset, the performance drop is small or moderate.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed method is very neat, as it is basically optimising the support set (training data for the training kernel matrix) for a kernel ridge regression problem.\n\n2. The performance is reasonably good, even it is applied to the finite-width neural networks.\n\n3. The discussion for the choice of lambda appears to be missing.\n\n4. Line 235 -- 236, \", distributed metalearning framework that leverages hundreds of GPUs per training run\" is confusing, can author elaborate which part of the model needs hundreds of GPUs and why?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "VUvgYyqKCwl", "reviewer": "Reviewer_ET9m", "summary": "In this work the authors extend the Kernel Inducing Points (KIP) method for dataset distillation to a new set of kernel functions and achieve new state of the art results on dataset compression. Since the KIP algorithm is a key algorithmic mechanism in this work, I will summarize it: The KIP method optimizes what can be seen as a test residual of a target dataset fitted by a kernel regression model that is created using a support dataset. This means that given a target dataset distribution, one can differentiate through the kernel computations to evolve the support data points in the support dataset freely in the input space and optimize them. After optimization the support dataset (Which is usually taken to be much smaller than the target distribution) can be used as a distilled version of the dataset. This compressed dataset enables classification and training without training with extensively large datasets. In this work the authors use the KIP algorithm using a family of kernels corresponding to infinitely wide neural networks that were discovered in the realm of NTK. Such infinite NTK convolution kernels are known to provide state of the art results (when compared with other kernels) on classic classification tasks such as MNIST, CIFAR, etc. The biggest issue of such infinite-network kernels is that they require immense amounts of computation to construct the kernel. In this work, the authors construct such kernels at each training step (!) and then back-propagate through the kernels (!!) to optimize the datapoint. This impressive feat is made possible by considering 3 layer convolutional neural networks and working with a relatively small number of data-points (5K as opposed to the whole dataset). This is still highly computationally challenging and the authors create a careful implementation including an orchestration system working with 100s of GPUs for optimizing the support dataset. With this new kernel family the authors achieve state of the art results on dataset compression and analyze the compressed dataset using high dimensional statistics tools.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance:\n\nThis work presents new state of the art results for dataset distillation in the cases of simple classification datasets. Dataset distillation is an important topic for neural networks as it can enable various knowledge transfer methods via a shorter training loop. In addition the authors demonstrate some results on actual neural network transfer, since this is an important contribution, can the authors clarify and add in the main text what neural network architectures were used as the networks trained with the distilled datasets? Dataset distillation is most useful when the distilled dataset can be used among many different architectures. Can the authors share about experiments of using the distilled dataset to train different neural network architectures?\n\nSince the authors utilize infinite-width networks kernels, the required computation for the method becomes very large and makes it naively infeasible. One of the key contributions of this work is the engineering component of this work which is highly non-trivial and is interesting and significant by itself.\n\nQuality and clarity:\n\nOverall this submission is very high quality with broad literature review, thorough experimental setup and in-depth analysis of the properties of the distilled datasets. Further the attached supplement provides ample details on the experimental setup in the appendix in addition to the new client-server set-up used for the large experiments.\nOverall this is an interesting and well delivered work on dataset distillation but at the same time I am a bit torn due to some weaknesses. Some of the weaknesses of the work are the apparent lack of novelty of the idea, and irreproducibility (stemmed from the large amount of computation required). Further given the very large computational requirements of this method on simple datasets such as MNIST, and CIFAR I am concerned about the applicability of this method to larger datasets of interest. At this time I am going to assign this a score of (6) but if the authors can clarify my questions with regards to the transferability of the distilled datasets to different neural architectures I am willing to improve my score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Large computational costs and focus on smaller datasets.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Rlj8rAgUaJK", "reviewer": "Reviewer_xee8", "summary": "The paper proposes to use infinitely wide CNNs to perform dataset distillation – obtaining a tiny dataset that enables high accuracy. The approach is an extension of a prior work (Kernel Inducing Points (KIP), etc.). Results are shown on MNIST, Fashion-MNIST, SVHN, CIFAR-10/100 datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has following strengths:\n1.\tResults for ConvNet (3-layer ReLU CNNs) outperform prior models.\n2.\tResults for transferring to NN show good results for 1-sample distilled datasets.\n3.\tPreliminary but interesting study of generated images is presented.\n\nThe paper has several major weaknesses:\n1.\tIt is unclear if this proposed method will lead to any improvement for hyper-parameter search or NAS kind of works for large scale datasets since even going from CIFAR-10 to CIFAR-100, the model's performance reduced below prior art (if #samples are beyond 1). Hence, it is unlikely that this will help tasks like NAS with ImageNet dataset.\n2.\tThere is no actual new algorithmic or research contribution in this paper. The paper uses the methods of [Nguyen et al., 2021] directly. The only contribution seems to be running large-scale experiments of the same methods. However, compared to [Nguyen et al., 2021], it seems that there are some qualitative differences in the obtained images as well (lines 173-175). The authors do not clearly explain what these differences are, or why there are any differences at all (since the approach is identical). The only thing reviewer could understand is that this is due to ZCA preprocessing which does not sound like a major contribution.\n3.\tThe approach section is missing in the main paper. The reviewer did go through the “parallelization descriptions” in the supplementary material but the supplementary should be used more like additional information and not as an extension to the paper as it is.\n\nTimothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021.\n\n\n\n--------------------\nUpdate: Please see my comment below. I have increased the score from 3 to 5.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KLjksMD_oON", "reviewer": "Reviewer_HQy7", "summary": "Paper mainly extends the dataset distillation algorithms KIP and LS (Nguyen et al. 2021) to infinitely wide neural networks. To that end, authors present a distributed framework that draws upon huge hardware resources and show improvement on the distillation performance. Paper also analyzes the synthesized data samples (images, labels) via multiple studies.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Main drawback of the paper is its limited technical contributions. Authors extend the existing distillation methods to (primarily) infinite width neural networks.\n- Also, while the proposed framework obtains SOTA results on multiple datasets, the transfer results (to finite width NNs) suffer performance drop and result in non SOTA results in 50% of the cases. However, the results are still very strong.\n- The distributed meta-learning framework (one of the claimed contributions) developed for scaling the distillation algorithms to complex neural networks has only been discussed in appendix.\n- Despite the above issues, the paper presents an important result of adapting the successful distillation algorithms to crucial neural network configurations. Further, the experimental analysis and studies performed on the distilled samples add strong value to the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a distributed kernel-based meta-learning framework for dataset distillation. The paper extends previous work Nguyen et al. [2021] to a large-scale training setting. 

The paper has received mixed reviews. The reviewers appreciate the significant improvement of dataset distillation performance on MNIST and CIFAR. However, several reviewers are concerned about the technical novelty compared to [Nguyen et al., 2021],  the scalability to large-scale datasets and large batch sizes, and the performance of downstream tasks. The authors have provided a rebuttal and addressed some of the concerns. The AC considers the improvement of dataset distillation on MNIST and CIFAR is significant compared to prior works. Before this paper, it’s hard to imagine that one can get 64.3% performance on CIFAR with 1 image per class for any machine learning algorithm. The AC also appreciates the engineering efforts and recommends that the authors move the distributed meta-learning algorithm from the appendix to the main text. The AC understands that the current method cannot handle ImageNet-level datasets. But given the low performance (29%) of previous dataset distillation methods even on CIFAR-10, this paper has presented a meaningful intermediate step towards the goal of dataset distillation. How to make it more efficient could be left as future work. Therefore,  the AC recommended accepting the paper. 



Decision: Accept (Poster)