Dataset Distillation with Infinitely Wide Convolutional Networks
Authors: Timothy Nguyen
Roman Novak
Lechao Xiao
Jaehoon Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: dataset distillation
infinite-width limit
convolutional neural networks
kernel ridge-regression
meta-learning
feature learning

Abstract: The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel-based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10  datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.

Dataset Distillation with Infinitely Wide Convolutional Networks
Timothy Nguyen'
Roman Novak
Lechao Xiao
Jaehoon Lee"
DeepMind
Google Research, Brain Team
cimotaayagggyyea@eeppnn....mom com
{romann, xlc, jaehles) com
Abstract
The effectiveness machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply novel distributed cernel-based meta-learning framework to achieve state-of-the-ane results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10 datapoints (0.02% of original dataset), we obtain over 65% test accuracy on CIFAR- image classification task, dramatic improvement over the previous best test accuracy of 40%. Our ofte-offthhe-att results extend across many other settings for MNIST, Fashion-MNIST. CIFAR-10, CIFAR-I00 and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets shed light on how they differ from naturally occurring data.
Introduction
Deep learning has become extraordinarily successful a wide variety of settings through the availability of large datasets [Krizhevsky al., 2012, Devlin et al., 2018, Brown et al., 2020, Dosovitskiy et al., 2020]. Such large datasets enable a neural network to learn useful representations of the data that are adapted solving tasks of interest. Unfortunately, it can be prohibitively costly to acquire such large datasets and train neural network for the requisite amount of time.
One way to mitigate this problem is by constructing smaller datasets that are nevertheless informative. Some direct approaches to this include choosing a representative subset of the dataset (i.e. a coreset) or else performing low-dimensional projection that reduces the number of features. However, such methods typically introduce a tradeoff between performance and dataset size, since what they produce is coarse approximation of the full dataset. By contrast, the approach of dataset distillation is to synthesize datasets that are more informative than their natural counterparts when equalizing for dataset size [Wang et al., 2018, Bohdal et al., 2020, Nguyen al., 202 Zhao and Bilen, 2021]. Such resulting datasets will not arise from the distribution of natural images but will nevertheless capture features useful to a neural network, capability which remains mysterious and is far from being well-understood [Ilyas et al., 2019, Huh et al., 2016, Hermann and Lampinen, 2020].
The applications of such smaller, distilled datasets are diverse. For nonparametric methods that scale poorly with the training dataset (e.g. nearest-neighhors or kernel-ridge regression), having a reduced dataset decreases the associated memory and inference costs. For the training of neural networks, such distilled datasets have found several applications in the literature, including increasing the
Work done while at Google Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

effectiveness of replay methods in continual learning [Borsos et al., 2020] and helping to accelerate neural architecture search [Zhao et al., 2021, Zhao and Bilen, 2021].
In this paper, we perform a large-scale extension of the methods of Nguyen et al. [2021] to obtain new state-of-the-art (SOTA) dataset distillation results. Specifically, apply the algorithms KIP (Kernel Inducing Points) and LS (Label Solve), first developed in Nguyen et al. [2021], to infinitely wide convolutional networks by implementing novel, distributed meta-lcarning framework that draws upon hundreds of accelerators per training. The need for such resources is necessitated by the computational costs of using infinitely wide neural networks built out of components occurring in modern image classification models: convolutional and pooling layers (see $B for details). The consequence that we obtain distilled datasets that are effective for both kernel ridge-regression and neural network training.
Additionally, we initiate a preliminary study of the images and labels which KIP learns. We provide a visual and quantitative analysis of the data learned and find some surprising results concerning their interpretability and their dimensional and spectral properties. Given the efficacy of KIP and LS learned data, we believe a better understanding of them would aid in the understanding of feature learning in neural networks.
To summarize, our contributions are as follows:
1. We achieve SOTA dataset distillation results on a wide variety of datasets (MNIST, Fashion- MNIST, SVHN, CIFAR-10, 10, CIFAR-100) for both kernel regre-oen and neural network training. In several instances, our results achieve an impressively wide margin over prior art, including over 25% and 37% absolute gain in accuracy on CIFAR-10 and SVHN image classification, respectively, when using only 10 images (Tables 2, A11).
2. We develop novel, distributed meta-learning framework specifically tailored the compu- tational burdens of sophisticated neural kernels ($2.1).
3. We highlight and analyze some of the peculiar features of the distilled datasets we obtain, illustrating how they differ from natural data ($4).
4. We open source the distilled datasets, which used thousands of GPU hours, for the re- search community to further investigate at https:/ thu.. com/ wtt. -eam/goggl--eeerchh google-research/ research/tree httt///eeeeem//eettt/tteeee master/kip.
2 Setup
Background on infinitely wide convolutional networks. Recent literature has established that Bayesian and gradient-descent trained neural networks converge to Gaussian Processes (GP) as the number of hidden units in intermediary layers approaches infinity (see $5). These results hold for many different architectures, including convolutional networks, which converge to a particular GP in the limit of infinite channels [Novak al., 2019, Garriga-Alonso et al., 2019, Arora et al., 2019]. Bayesian networks are described by the Neural Network Gaussian Process (NNGP) kernel, while gradient descent networks are described by the Neural Tangent Kernel (NTK). Since we interested in synthesizing datasets that can be used with both kernel methods and common gradient-descen trained neural networks, focus on NTK in this work.
Infinitely wide networks have been shown to achieve SOTA (among non-parametric kernels) results on image classification tasks [Novak et al., 2019, Arora et al., 2019, Li et al., 2019, Shankar et al., 2020, Bietti, 2021] and even rival finite-width networks in certain settings [Arora et al., 2020, Lee et al., 2020]. This makes such kernels especially suitable for our task. As convolutional models, they encode useful inductive biases of locality and translation invariance [Novak et al., 2019], which enable good generalization. Moreover, flexible and efficient computation of these kernels are possible due to the Neural Tangent library [Novak et al., 2020].
Specific models considered. The central neural network (and corresponding infinite-width model) we consider a simple 4-layer convolutional network with average pooling layers that we refer to as ConvNet throughout the text. This architecture is a slightly modified version of the default model used by Zhao and Bilen [2021], Zhao et al. [2021] and was chosen for ease of baselining (see SA for details). In several other settings we also consider convolutional networks without pooling

---

layers ConvVec.
and networks with no convolutions and only fully cully-connected layers FC. Depth of architecture (as measured by number of hidden layers) indicated by an integer suffix.
Background on algorithms. We review the Kernel Inducing Points (KIP) and Label Solve (L.S)
algorithms introduced by Nguyen et al. [2021]. Given kernel K, the kernel ridge-regression (KRR) loss function trained on a support dataset (X,,. Us) and evaluated on a target dataset (X,,8t) is
L L( v.) 2 llv. Kx,, (Kx,x, + +XII"'y.1 9.112-
where U and V are sets, Kuv is the matrix of kernel elements (K(u) (u, u) wet..ev. Here À > 0 is fixed regularization parameter. The KIP algorithm consists of minimizing (1) with respect to the support set (either just the X, or along with the labels y.). Here, we sample (X,, y1 from target dataset at every (meta)step, and update the support set using gradient-basse methods. Additional variations include augmenting the X, or sampling different kernel K (from a fixed family of kernels) at each step.
The Label Solve algorithm consists of solving for the least-norm minimizer of (1) with respect to ys- This yields the labels
y² = (Kxxx + yr.
where A denotes the pseudo-inverse of the matrix A. Note that here (X,, yt) = D. i.c. the labels solved using the whole target set.
In our applications of KIP and Label Solve, the target dataset always significantly larger than the support set (X,, V.). Hence, the learned support set or solved labels can be regarded as distilled versions of their respective targets. We also initialize our support images to be a subset of natural images, though they could also be initialized randomly.
Based the infinite-widhh correspondence outlined above and in $5, dataset distillation using KIP or LS that is optimized for KRR should extend to the corresponding finite-widhh width neural network training. Our experimental results in $3 validate this expectation across many settings.
2.1 Client-Server Distributed Workflow
We invoke a client-server model of distributed computation" in which server distributes independent workloads to large pool of client workers that share a queue for receiving and sending work. Our distributed implementation of the KIP algorithm has two distinct stages:
Forward pass: In this step, we compute the support-support and target-support matrices K(X,, X²) and K (X,, X,). To do so, we partition X, x X, and X, X X, into pairs of images (x, x'', ,2),, each with batch size B. We send such pairs to workers compute the respective matrix block K ((x,z) z¹). The server aggregates all these blocks to obtain the K K(X,, X,) and K(X, X.) matrices.
Backward pass: In this step, we need to compute the gradient of the loss I. (1) with respect to the support set Xg. We need only consider 0 since ÖT, is cheap to compute. By the chain rule, we can write
AL OK(X,, 0x, O(K(X,,X, X.)) ax, O(K(,,, 0x,
ØL
OL OK(X,,
The derivatives of L with respect to the kernel matrices are inexpensive, since L depends in simple way on them (matrix multiplication and inversion). What is expensive to compute is the derivative of the kernel matrices with respect to the inputs. Each kernel element is an independent function of the inputs and a naive computation of the derivative of a block would require forward- mode differentiation, infeasible due to the size of the input images and the cost compute the individual kernel elements. Thus our main novelty is to divide up the gradient computation into backward differentiation sub-computations, specifically by using the built-in function j jax. vjp in JAX [Bradbury et al., 2018]. Denoting K K(X,, X.) or K K(X,, X.) for short-hand, we divide the
"The abbreviation "Vec" stands for vectorization, to indicate that activations of the network are vectorized (flattened) before the top fully-connected layer instead of being pooled. Implemented using Courier available https:/ / //g git..b.... com http///eppmmnnddeede www.h//dddaaaahaaa..a
3

---

Table 1: Comparison with other methods. The left group consists of neural network based methods. The right group consists of kernel ridge-regression All settings KIP involve use label-learning Grayscale datasets use standard channel-wise preprocessing while RGB datascts use regularized ZCA preprocessing.
Imgs/ Class
DC'
DSA
KIP FC' aug
LS ConvNet 2.3
KIP ConvNet"
no aug
aug
91.7+0.5
88.7+0.6
85.5+0.1
73.4
97.3+0.1
96.5+0.1
MNIST
10
97.4
97.8+0.1
97.2+0.2
96.4
99.1=0.1
99.1=0.11
50
98.8+0.1
99.2#0.1
98.4+0.1
98.3
99.4=0.1
99.5
70.5
70.6+0.6
65.3
82.9+0.2
76.7+0.2
Fashion-
10
82.340.4
84.6+0.3
80.8
91.0=0.1
88.8+0.1
MNIST
50
83.640.4
88.7+0.2
86.9
92.4+0.1
91.0+0.1
1
1..2+114
1..5+14
23.9
62.4+0.2
64.3+0.4
SVHN
76.1
79.2+0.5
52.8
79.3+0.1
81.1
50
82.3+0.3
84.410.4
76.8
82.0+0.1
84.3+0.1
1
2..
28.8+0.7
40.
26.1
64.7+0.2
63.4+0.1
CIFAR-10
10
44.940.5
52.1
$3.1
53.6
75.610.2
75.5+0.1
50
53.9
60.6+0.5
58.6+0.4
65.9
78.2+0.2
80.6+0.1
12.8+0.3
13.9+0.3 32.3+0.3
23.8
34.9
33.3+0.3
CIFAR-100
10
25.240.3
39.2
47.9+0.2
49.5
DC [Zhao ul., 2021]. DSA [Zhao and Bilen, 2021]. KIP FC [Nguyen el al., 2021].
Ours.
LD [Bohdal et al., 2020] is another baseline which distills only labels using the AlexNet architecture. Our LS achieves higher test accuracy than theirs in every dataset category.
matrix 0T/OKK, computed on the server, into B x blocks corresponding to aL/aK s'), where r and a' each have batch size B. We send each such block, along with the corresponding block of image data (z, z') to a worker. The worker then treats the åL/aK (z/kkKz,a , 2') receives as the cotangent vector argument of jax. jax.vjp that, via contraction, converts the derivative of K(x, .²) with respect into scalar. The server aggregates all these partial gradient computations performed by the workers, over all possible x B blocks, to compute the total gradient /L/ 0x,, used to update X..
Experimental Results
3.1 Kernel Distillation Results
We apply the KIP and LS algorithms using the ConvNet architecture on the datasets MNIST [LeCun et al., 2010], Fashion MNIST [Xiao et al., 201 SVHN [Netzer et al., 201 1]. CIFAR-10 (Krizhevsky, 2009], and CIFAR-100. Here, the goal is condense the train dataset down to a learned dataset of size 1, 10, or 50 images per class. We consider a variety of hyperparameter settings (image preprocessing method, whether augment target data, and whether train the support labels for KIP), the full details of which are described in SA For space reasons, we show a subset of our results in Table 1, with results corresponding the remaining set of hyperparameters left to Tables 13-A10. We highlight here that a crucial ingredient for our strong results in the RGB dataset setting the use of regularized ZCA preprocessing. Note the variable effect that our augmentations have on performance (see the last two columns of Table I): they typically only provides a benefit for sufficiently large support set. This result is consistent with Zhao al. [2021]. Zhao and Bilen [2021]] in which gains from augmentations are also generally obtained from larger support sets. We tried varying the fraction (0.25 and 0.5) of each target batch that augmented at each training step and found that while 10 images still did best without augmentations, 100 and 500 images typically did slightly better with some partial augmentations (versus none or full). For instance, for 500 images on CIFAR-10, 10, we obtained 81.1% %% test accuracy using augmentation rate 0.5. Thus, our observation is that given that larger support sets can distill larger target datasets, as the former increases in size, the latter can be augmented more aggressively for obtaining optimal generalization performance.

---

Korne Samp NN Trander (NSE)
Kome Sampling NN Transter (XENT)
Teat Acturacy
0.4
0.5
Conoliet
Cony-Voc
KIP Traning Stepe
KIP Traning Steps
0.3
Condiet Traiing NN Transfer (MSE)
Conyler Traiing NN Transkrr (XENT)
0.2
Train kemnls
Toul Amuuvey
0.4
Conwitet
Conn-Vec3
0.0
ConviNet
Voca
Venv
Test kernel
Traning
Training Stops
Figure 1: KIP with kernel sampling individual kernels. Left: Evaluation of three kernels, ConvNet, Conv-Vec3. Conv-Vect for KRR with respect to four train settings: sampling KIP ("all") which uses all the kernels else KIP trained with the individual kernels. For all three kernels, "all" is a close second place, outperformed only if the kernel used for training is exactly the same as the one used for testing. Right: We take the learned images of the four train settings described above and transfer them to finite-width neural networks corresponding to ConvNet, Conv-Vec3. Conv- Vec8. Each point is a neural network trained on specified KIP learned checkpoint. Top row sampling KIP images and bottom row is the baseline using just ConvNet for KIP. These plots indicate that sampling KIP improves performance across the architectures that are sampled, for both MSE and cross entropy loss. Settings: CIFAR-10, 100 images, no augmentations, no 7CA, no label learning.
Remarkably, our results in Table outperform all prior baselines across all dataset settings. Our results are especially strong in the small support size regime, with our 1 image per class results for KRR outperforming over 100 times as many natural images (see Table A1). We also obtain significant margin over prior art across all datasets, with our largest margin being a 37% absolute gain in test accuracy for the SVHN dataset.
3.2 Kernel Transfer
In $3.1, we focused obtaining state-of-the-art dataset distillation results for image classification using a specific kernel (ConvNet). Here, we consider the variation of KIP in which we sample from a family of kernels (which we call sampling KIP) We validate that sampling KIP adds robustness to the learned images in that they perform well for the family of kernels sampled during training.
In Figure 1, we plot test performance of sampling KIP when using the kernels ConvNet, Conv-Veck and Conv-Vec8 (denoted by "all") alongside KIP trained with just the individual kernels. Sampling KIP performs well at test time when using any of the three kernels, whereas datasets trained using single kernel have significant performance drop when using a different kernel.
3.3 Neural Network Transfer
In this section, we study how our distilled datasets optimized using KIP and L. transfer to the setting of finite-width neural networks. The main results are shown in Table 2. The third column shows the best neural network performance obtained by training with respect to some choice of KIP and neural network training hyperparameters (see ŠA for details). Since the datasets optimized for kernel ridge-regression and not for neural network training itself, we expect some performance loss when transferring to finite-wdtth networks, which we record the fourth column. Remarkably, the drop due to this transfer is quite moderate or small and sometimes the transfer can even lead to gain in performance (see LS for SVHN dataset with 10 images per class).
KIP dataset of the corresponding size
Overall, our transfer to finite-width networks outperforms prior art based on DC/DSA [Zhao et al., 2021, Zhao and Bilen, 2021] in the image per class setting for all the RGB datasets (SVHN, CIFAR-10. 10, CIFAR- 100). Moreover, for CIFAR-10, outperform DC/DSA in all settings.
5

---

Table 2: Transfer of KIP and LS to neural network training. Datasets obtained from KIP and LS using the ConvNet kernel are optimized for kernel ridge-regression and thus have reduced performance when used for training the corresponding Unite-width ConvNet neural network. Remarkably, the loss performance is mostly moderate and even small in many instances. Grayscale datasets use standard channel -wise preprocessing while RGB datasets use regularized ZCA preprocessing. The KIP datasets used here can have augmentations or no augmentations and, unlike those in Table I, can have either fixed or learned labels. * denotes best chosen transfer is obtained with learned labels.
Perf. change
Perf. change
Imgs/Class
DC/DSA
KTP to NN
LS to NN
91.7+
90.1+0.1
-5.5
71.0+0.2
-2.4
MNIST
10
97.8+0.1
97.5+0.0
-1.1
95.2+0.1
-1.2
50
99.240.1
98.3+0.1
-0.8
97.9+0.0
-0.4
70.640.6
73.5
-9.8
61.2+0.1
-4.1
Pashion-MNIST
10
84.610.3
86.810.1
-1.3
79.7+0.1
-1.2
50
88.7+0.2
88.0+0.1
-4.5
85.0+0.1
-1.8
3..2+1.4
57.3+0.1"
-8.3
23.8+0.2
-0.2
SVHN
10
79.2
75.0+0.1
-1.6
53.240.3
0.4
50
84.410.4
80.5+0.1
-1.0
76.5+0.3
-0.4
28.8+0.7
49.910.2
.9.2
24.7#0.1
-1.4
CIFAR-10
10
52.1
62.7+0.3
-4.6
49.3+0.1
-4.3
50
60.6+0.5
68.640.2
-4.5
62.0+0.2
-3.9
13.940.3
15.7+0.2
-18.1
11.8+0.2
-12.0
CIFAR-100
32.3+0.3
28.3+0.1
-17.4
25.0+0.1
-14.2
Figures 2 and 3 provide closer look at KIP transfer changes under various settings. The first of these tracks how transfer performance changes when adding additional layers as function the number of KIP training steps used. The normalization layers appear to harm performance for MSE loss, which can be anticipated from their absence in the KIP and LS optimization procedures. However they appear to provide some benefit for cross entropy loss. For Figure training progresses, the downstream finite-width network's performance also improves in general. A notable exception is observed when learning the labels in KTP, where longer training steps lead to deterioration of information useful to training finite-width neural networks. We also observe that as predicted by infinite-widde theory [Jacot al., 2018, Lee et al., 2019], the overall gap between KIP or LS performance and finite-width neural network decreases as the width increases. While our best performing transfer is obtained with width 1024, Figure (middle) suggest that even with modest width of 64, our transfer can outperform prior art of 60.6% by Zhao and Bilen [2021].
observe that KIP
Finally, in Figure 4, investigate the performance of KIP images a single run, as compared to natural images, over range of hyperparameters. We find the outper- formance of KIP images above natural images consistent across hy perparammesess and checkpoints. This suggests that our KIP images may also be effective for accelerated hyperparameter search, an application of dataset distillation explored in Zhao al. [2021], Zhao and Bilen [2021].
the course of training of
Altogether, we find our neural network training results encouraging. First, it validates the applicability of infinite-width methods to the setting of finite width [Huang and Yau, 2020, Dyer and Gur-Ari, 2020, Andreassen and Dyer, 2020, Yaida, 2020, Lee et al., 2020]. Second, we find some of the transfer results quite surprising, including efficacy of label solve and the use of cross-entropy for certain settings (see SA for the full details).
Understanding KIP Images and Labels
natural question to consider is what causes KIP to improve generalization performance. Does simplify support images, removing noise and minor sources of variation while keeping only the core features shared between many target images of a given class? does it make them more complex by producing outputs that combine characteristies of many samples in single resulting collage? While

---

Compare ONN Archiletures NN Thansfer (MSE)
Compare Archileck.es: NN (XENT)
0.75
Text Antuney
0.05
0.60
0.50
Base
Batch-Nam
LayerNorm Norm
Myrtie
Instance Norm
KIP Width
10
10
10'
10
10P
10"
10"
KIP Training Staps
KIP Training Steps
Figure 2: Robustness to neural network variations. KIP ConvNet images (trained with fixed labels) are tested on variations of the ConvNet neural network. including those which have vari- ous normalization layers (layer, instance, batch). A similar architecture to ConvNet, the Myrtle5 architecture (without normalization layers) [Shankar et al., 2020], which differs from the ConvNet architecture by having an additional convolutional layer at the bottom and global average pooling that replaces the final local average pooling at the top, is also tested. Finally, mean-square error is compared with cross-entropy loss (left versus right), Settings: CIFAR-10, 500 images, ZCA, no label learning.
Comnect CIFAR-10
Conviet CIFAR110 10 Translervs Staddadd
Conshet CIRAR-10
Eiaiiing Transfer
Labo Soleo Transfor v Standard
0.3
1024
KP Steps
Plumbr Channel))
(Number Charnel))
Figure 3: Variations for neural network transfer. Left: Plot of transfer performance a function of KIP training steps across various train settings. Here (a) denotes augmentations used during KIP training and (a+l) denotes that additionally the labels were learned. MSE and XENT denote mean-square-error and cross entropy loss for the neural network, where for the case of XENT and (a+l), the labels for the neural network are the argmax of the learned labels. Middle: Exploring the effect of width on transferability of vanilla KIP data. Right: The effect of width on the transferability of label solved data. Settings: CIFAR-10. 10, 500 images, ZCA.
0.7
CIFAR 10 suppont S.Ne 50rthags:
CIRAR-O ogeer 002
OA 15 support 9ondrrig 57
CIFAP- SOPAP-O-AA wwp.rrpeerooooo..oo.oooooooo caN (2eibogg 112
0.6
0.5-
0.4
x NNNeRc66 KDNT
xom
0.3
0.4
0.6
07
0s
04
04
07 63
0.
05 64
Nezural TeetAooray
hakore Acuuaay
Natusl Arcuacyy
Naaur AestAcuuyy
C1744-10 suport pp
CIPAB-12 support tbe Soodheipi 1000
C CFA Stodholgt 5000
CI suppor Scolltiii 50000
0.4
0.5-
MEP No ALD. 5002 yot MiriM DUB
Morppo Shtera RLB
UPP por 3Nimral 253)
03
04
0.5 Noosa AocAAooett
0.5 06
04 05 Natusl Arcuray
07
04
Natura Teceaayy
Natural Tor.Acouscs
Figure 4: Hyperparameter robustness. In the above, KIP images across eight different checkpoints are used to train the ConvNet neural network. Each point in each plot is a neural network training with different hyperparameter, and its location records the final test accuracy when training natural images versus the KIP images obtained from initializing from such images. For both MSE and cross entropy loss, KIP images consistently exceed natural images across many hyperparameters. Settings: CIFAR-10 10, 500 images, ZCA, no augmentations, no label learning.
7

---

CIFAR-100
apple
aquarlum_ fish
baby
bear
beaver
bed
bee
beetle
blcycle
bottle
Trained Init
Figure 5: Examples of learned images. Images are initialized from natural images in the top row (Init) and converge to images in the bottom row (Trained). Settings: 100 images distilled, no ZCA, no label training, augmentations.
properly answering this question is subject to precise definitions of simplicity and complexity, we find that KIP tends to increase the complexity of pictures based on the following experiments:
Visual analysis: qualitatively, KIP images tend to be richer in textures and contours.
Dimensional analysís: KIP produces images of higher intrinsic dimensionality (i.e. an estimate of the dimensionality of the manifold on which the images lie) than natural images.
Spectral analysis:
unlike natural images, for which the bulk generalization performance is explained by a small number of top few cigendirections, KIP images leverage the whole spectrum much more evenly.
Combined, these results let us conclude that KIP usually increases complexity, integrating features from many target images into much fewer support images.
Visual Analysis. A visual inspection of our learned data leads to intriguing observations in terms of interpretability. Figure 5 shows examples of KTP learned images from CIFAR-100. The resulting images are heterogeneous in terms of how they can be interpreted as distilling the data. For instance, the distilled apple image seems consist of many apples nested within possibly larger apple, whereas the distilled bottle image starts off as two bottles and before transforming into one, while other classes (like the beaver) are altogether visually indistinct. Investigating and quantifying aspects that make these images generalize so well is a promising avenue for future work. We show examples from other datasets in Figure A2.
In Figure 6, we compare MNIST KIP data learned with and without label learning. For the latter case with images and labels optimized jointly, while labels become more informative, encoding richer inter-class information, the images become less interpretable. This behavior consistently leads to superior KRR results, but appears not be leveraged as efficiently in the neural network transfer setting (Table 2). Experimental details can be found in SA.
Dimensional Analysis. We study the intrinsic dimension of KIP images and find that they tend to grow. Intrinsic dimension (ID) was first defined by Bennett [1969] as "the number of free parameters required in hypothetical signal generator capable of producing a close approximation to each signal in the collection". In our context, it is the dimensionality of the manifold embedded into the image space which contains all the support images. Intuitively, simple datasets have low ID as they be described by small number of coordinates on the low-dimensional data manifold.
ID can be defined and estimated differently based on assumptions on the manifold structure and the probability density function of the images on this manifold (see [Camastra and Staiano, 2016] for review). We the "Two-NN" method developed by [Facco et al., 2017], which makes relatively few assumptions and allows to estimate ID only from two nearest-neighbor distances for each datapoint.
Figure shows that the ID is increasing for the learned KIP images function of the training step across a variety of configurations (training with or without augmentations/llael learning) and datasets. One might expect that a distillation procedure should decrease dimensionality. On the other hand, Ansuini et al. [2019] showed that ID increases the earlier layers of a trained neural network. remains to be understood if this latter observation has any relationship with our increased ID. Note that ZCA preprocessing, which played an important role for getting the best performance for our RGB datasets, increases dimensionality of the underlying data (see Figure A4).

---

MNIST
Label Imaages
3
Init
Trained Trained images only
labels and images
0123436789
0123456789
0123456759
0123156789
0123456789
Figure 6: Dataset distillation with trainable and non-trainable labels. Top row: initialization of support images and labels. Middle row: trained images if labels remain fixed. Botton row: trained images and labels, jointly optimized. Settings: 100 images distilled, no augmentations.
Dataset
Images intrinatc Dimension
GIFARTO
FASHION MNIST
MNIST
SVHN GROPPED
60-
Data Augmentation
20-
305
5,000
335 6,000
335 6,000
335 6,000
Training step
Training step
Training stop
Training
Figure 7: Intrinsic dimension of a learned datasets grows during training. As training progresses, intrinsic dimension of the learned dataset grows, indicating that training non-trivially transforms the data manifold. See Figures A2 and 6 for visual examples of learned images, and Figures A3 and A4 for similar observations using other metrics and settings. Settings: 500 images distilled, no ZCA.
Spectral Analysis. Another distinguishing property of KIP images is how their spectral components contribute to performance. In Figure 8, we spot how different spectral bands of KIP learned images affect test performance compared to their initial natural images. Here, and ConvNet architectures. We note that for natural images (light bars), most of their performance captured by the top 20% of eigenvalues. For KIP images, the performance is either more evenly distributed across the bands (FC and Conv-Vec8) or else is skewed towards the tail (ConyNet).
the FC2, Conv-Vecte
Related Work
Dataset distillation was first studied in Wang al. [2018]. The work of Sucholutsky and Schonlau [2019], Bohdal et al. [2020] build upon it by distilling labels. Zhao et al. [2021] proposes condensing training set by harnessing a gradient matching condition. Zhao and Bilen [2021] takes this idea further by applying a suitable augmentation strategy. Note that while Zhao and Bilen [2021] is limited augmentation expressiveness (they have to apply single augmentation per training iteration), we can sample augmentations independently per image in our target set per train step. Our work together with Nguyen et al. [2021] are, to the best of our knowledge, the only works using kernel-based methods for dataset distillation image classification datasets.

---

Relative Performance using Banded Eigenvectors
1.0
FCNaturrl
FCXIP
Relative Performance
Con- Nactauralllural
0.8
Con-VeecKPP
Natural
0.6
KiP
0.4
0.2
0.0
ToP 0-20%
Top 20400%
Top 40-60%
Top 80-80%
80- 80-100%
Figure 8: Spectral contribution to test accuracy shifts to the tail. By setting the param-er to zero kernel-ridge regression and composing Kx,x, with the spectral projection onto various eigenspaces, we can explore how different spectral bands affect test accuracy of kernel ridge- regression. We plot the relative change in test accuracy using contiguous bands of 20% of the eigenvalues. Settings: CIFAR-10, 500 images. Further details in $A
Our use kernels stems from the correspondence between infinitely-wide neural networks and kernel methods [Neal, 1994, Lee et al., 2018, Matthews et al., 2018, Jacot et al., 2018, Novak et al., 2019, Garriga-Aonso et al., 2019, Arora et al., 2019, Yang. 2019a,b, Hron et al., 2020] and the extended cor- respondence with the finite width corrections [Dyer and Gur-Ari, 2020, Huang and Yau, 2020, Yaida, 2020]. These correspondences underlic the transferability of our KRR results to neural networks, and have been utilized in understanding trainability [Xiao et al., 2020], generalizations [Adlam and Pennington, 2020], training dynamics [Lewkowycz et al., 2020, Lewkowycz and Gur-Ari, 2020], uncertainty [Adlam et al., 2021], and demonstrated their effectiveness for smaller datasets [Arora et al., 2020] and neural architecture search [Park et al., 2020, Chen et al., 2021].
6 Conclusion
We performed an extensive study of dataset distillation using the KIP and LS algorithms applied to convolutional architectures, obtaining SOTA results variety of image classification datasets. In some cases, our learned datasets were more effective than a natural dataset two orders of magnitude larger in size. There are many interesting followup directions and questions from work:
First, integrating efficient kernel- aamnolllppooiiiioonn methods into our algorithms, such as those of Zandich et al. [2021], will reduce computational burden and enable scaling up to larger datasets. In this direction, the understanding when optimizing for neural network performance has received significant attention machine learning models continue to stretch computational limits [Hestness et al., 2017, Rosenfeld et al., 2020, Kaplan al., 2020, Bahri et al., 2021]. Developing understanding of how to harness smaller, yet more useful representations data would aid in such endeavors. In particular, it would be especially interesting to explore how well datasets be compressed they scale up in size.
how various resources (e.g. data, parameter count, compute) scale
Second, LS and KIP with label learning shows that optimizing labels a very powerful tool for dataset distillation. The labels we obtain are quite far away from standard, interpretable labels and we feel their effectiveness suggests that understanding of how to optimally label data warrants further study.
Finally, the novel features obtained by our learned datasets, and those of dataset distillation methods in general, may reveal insights into interpretability and the nature of sample-effe -efilelfffcientttt representations. For instance, observe the bee and bicycle images Figure 5: the bee class distills into what appears to be spurious visual features (e.g. pollen), while the bicycle class distills to the essential contours of a typical bicycle. Additional analyses and explorations of this type could offer insights into the perennial question of how neural networks learn and generalize.
10

---

Acknowledgments We would like to acknowledge special thanks to Samuel S. Schoenholz, who proposed and helped develop the overall strategy for our distributed KIP learning methodology. We are also grateful to Ekin Dogus Cubuk and Manuel Kroiss for helpful discussions.
References
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training deep bidirectional transformers for language understanding, arXiv preprint arXiv:18/0.04905, 2018.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers image recognition scale. arXiv preprint arXiv:2010.1192) 2020.
Tongzhou Wang, Van Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.1095, 2018.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006. 08572, 2020.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, 2021.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features, 2019.
Minyoung Huh, Pulkit Agrawal, and Alexei Efros. What makes imagenet good for transfer learning? arXiv preprint 1608 08614, 2016.
Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006.124: 12433, 2020.
Zalán Borsos, Mojmir Mutný, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming, arXiv preprint 2r06v:2006. 03875, 2020.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang. Jiri Hron, Daniel A. Abolafia. Jeffrey Pennington, and Jascha Dickksien.. Bayesian deep convolutional networks with many channels are gaussian processes. In International Conference on Learning Representations, 2019.
Adrià Marriga-Alonso Laurence Aitchison, and Carl Edward Rasmussen. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. In Advances in Neural Information Processing Systems, pages 8141-8150. Curran Associates, Inc., 2019.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjeev Arora. Enhanced convolutional neural tangent kemnlss. arXiv preprint arXiv: 1000009, 2019.
Vaishaal Shankar, Alex Chengyu Fang, Wenshuo Guo, Sara Fridovich-Kae Ludwig Schmidt, Jonathan Ragan- Kelley, and Benjamin Recht. Neural kernels without tangents. In International Conference on Machine Learning, 2020.
Alberto Bietti. Approximation and learning with deep convolutional models: a kemel perspective. arXiv preprint axXiv:2102.10032. 2021.

---

Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets small-data lasks. In International Conference on Learning Representations, 2020. URL
Jachoon Lee, Samuel S Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: empirical study. In Advances Neural Information Processing Systems, 2020.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks python. International Conference on Learning Representations, 2020. URL https: //
//tttp://github /ttphbb.ttttuut/g/ cww/cooo/ooo/.ooo grogle/neural. neural- -taggetts.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milnee JAX: composable transformations Python+NumPy programs, 2018. URL http://githuue com/ google/jaz.
Yann LeCun, Corinna Cortes, and Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: 2, 2010.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemell: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenbolz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advonces in Neural Information Processing Systems, 2019.
Jiaoyang Huang and Horn--Tzar Yau. Dynamies of deep neural networks and neural tangent hierarchy. In International Conference on Machine Learning, 2020.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. In International Confer- ence on Learning Representations, 2020. URL https://openrevieg
Anders Andreassen and Ethan Dyer. Asymptotics of wide convolutional neural networks. arxiv preprint arXiv:2008. 08675, 2020.
Sho Yaida. Non-Gaussian processes and neural networks at finite widths. Mathematical and Scientific
Machine Learning Conference, 2020.
R. Bennett The intrinsic dimensionality signal collections. IEEE Transactions on Information Theory, 15(5): 517-525, 1969. doi: 10.1109/TTT.1961 1054365.
Francesco Camastra and Antonino Staiano. Intrinsic dimension estimation: Advances and open problems. Information Sciences, 328:26-41, 2016. ISSN 0020-0255. doi: https///doi.rr//1..000jjjji in..::m................... URL https: hwwp://wwwww./
Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):18 8, 2017.
A. Ansuini, Laio, Macke, and Zoccolan. Intrinsic dimension of data representations in deep neural networks. In NeurlPS, 2019.
Ilia Sucholutsky and Matthis Schonlau. Soft-label dataset distillation and text dataset distillation. arXiv preprint arXiv:1910.02551. 2019.
Radford M. Neal. Priors for infinite networks (tech. rep. no. crg 94-11), University of Toronto, 1994.
Jachoon Lee, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl-dickstein. Deep neural networks gaussian processes. In Internutional Conference Learning Representations. 2018.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations. 2018.
12

---

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural langent kernel derivation. arXiv preprint arXiv:1902.04760, 2019a.
Greg Yang. Wide feedforward recurrent neural networks of any architecture gaussian processes. In Advances in Neural Information Processing Systems, 2019b.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: NNGP and NTK for deep attention networks. In International Conference Machine Learning, 2020.
Lechao Xiao, Jeffrey Pennington, and Samuel S Schoenholz. Disentangling trainability and generalization in deep learning. In International Conference Machine Learning, 2020.
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference on Machine Learning, pages 74-84. PMLR, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase deep learning: catapult mechanism. arXiv preprint arXiv:2003.022 0.20000,.222,,... 2020.
Aitor Lewkowycz and Guy Gur-Ari. On the training dynamics of deep networks with 1_2 regularization. In Advances Neural Information Processing Systems, volume 33, pages 4790-4799, 2020.
Ben Adlam, Jachoon Lee, Lechao Xiao, Jeffrey Pennington, and Jasper Snoek. Exploring the uncertainty properties neural networks' implicit priors infinite-width limit. International Conference on Learning Representations, 2021.
Daniel Park, Jaehoon Lee, Daiyi Peng. Yuan Cao, and Jascha Sohl-Dickstein. Towards ungp-guided neural architecture search. arXiv preprint arXiv:201 2020.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet four (gpu) hours: A theoretically inspired perspective. In International Conference on Learning Representations, 2021.
Amir Zandieh, Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, Jinwoo Shin. Scaling neural tangent kernels via sketching and random features. In Advances in Neural Information Processing Syxtems, 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/1712.00409, 2017. URL http: /ttp///axvvvv.v. org/abs/ / 111.1.11.000 00409.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference Learning Representations, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXtv:2 102006701, 2021.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jascha
Roman Novak, Samuel Schoenholz, and Jachoon Lec. On the infinite width limit of neural networks with standard parameterization. arXiv preprint arXiv:2001.07300 2020.
13


Reviews:
[{"review_id": "yzi-0XdwROT", "reviewer": "Reviewer_zgVX", "summary": "This paper performs dataset distillation to improve the deep learning training efficiency. By using the algorithm made by [1] and some expanded methods, the authors achieved the state-of-the-arts accuracy on CIFAR-10, MNIST, Fashion-MNIST, CIFAR10, CIFAR100, and SVHN datasets.\n\n[1] DATASET META-LEARNING FROM KERNEL RIDGE REGRESSION, ICLR 2021", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This is an expanded work of the previously developed method, but the authors expanded it well and achieved state-of-the-art results.\n\nQuality: The submission is technically sound overall. \n\nClarity: The submission is well written and very clear. The authors make it clear what has been done, what is typically done in this field of data distillation, and what has been done, and why it is novel/different.\n\nSignificance:  The significance of the work is moderate. On some small datasets, the authors can attain state-of-the-art results, but on the extremely large datasets such as ImageNet. Exploring this further would be interesting and valuable for the optimization community.\n\n\n- Overall the paper is well written, quite thorough, and well-cited. In particular, the authors did an excellent job in the introduction; it was a concise, detailed yet intuitive presentation of data distillation.\n\n- In line 139, the authors mentioned that \"sometimes the transfer can even lead to gain in performance (LS for SVHN)\". Why and how this gain can be observed?\n\n- The size of the datasets used in the experiments is very small, such as MNIST, CIFAR, and SVHN. As the authors mentioned one of the motivations of this study is increasing the training efficacy, it would be nice to have an experiment on extremely large datasets such as the ImageNet.\n\n- The init point of the training is from a random example of the training data, how much does the initial point affect the performance? Also, what happens if we start from a completely random image? \n\n-I just wonder that the model trained with the proposed method and the model trained on the original data could be equally strong to adversarial attacks? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "TvEncrJxJ0G", "reviewer": "Reviewer_AopF", "summary": "This paper extends the Kernel Inducing Point (KIP) Label Solve (LS) algorithms from Nguyen et al. to support infinitely wide conv nets, and convnets of finite width, to obtain an impressive performance boost. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-----------------------------------PROS\n- the performance gain is quite impressive on all datasets, both compared to the original KIP paper that didn't use convolutions and competing methods.\n\n-----------------------------------CONS\n\nMAJOR\n- The structure of the paper is poor and it makes your contribution hard to grasp, or worse, look like it is a very minor implementation tweak to an existing method (change fully connected layers of KIP to conv layers). Section 2 is called \"Set up\" even though it's almost entirely background information, and your method should be explained and put in contrast with the work of Nguyen2021 before the experimental section. Your experimental section takes most of the paper and includes experiments that are more appropriate for appendices (it feels like the main contribution was a bit light and therefore you added lots of side experiments which aren't any more relevant for your method than any other method). This leaves you too little room for the Related Work section and Method section which are respectively too short and non-existent.\n- Issues of clarity. Figure 4: I don't understand the value/meaning of checkpoints here? line 183: you need to explain what intrinsic dimension is and what it represents/ why it's meaningful in this context, since it's not a very famous concept. \n\n\nMINOR\n- The most important setting to report in dataset distillation is whether or not you use a fixed initialization or a random initialization. I don't see that being reported anywhere, did I miss it? You would need to include both settings or the random-init setting only. I suspect you did the latter based on your performances. This setting changes the results entirely and changes how relevant your distilled images are.\n- You're missing 50 images per class for CIFAR100 in Table 1, is it because 5000 distilled images is too much in memory?\n- l210 typo \"between\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-----------------------------------QUESTIONS\n- Have you tried continual learning experiments with your images? Are the gains as good as for test accuracy?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "69107upbRt4", "reviewer": "Reviewer_UNoV", "summary": "The paper performs thorough analysis and exploration on the implication of the proposed KIP and LS methods by Nyugen 2021 for dataset distillation. KIP and LS are based on the inf-width network limit as kernels, and devised based on kernel Ridge regression, and has been shown in prior work to have good dataset distillation performance. This paper greatly extends the analysis in Nyugen 2021, by investigating how KIP and LS can be used to achieve SOTA dataset distillation results in a kernel setting, and also sometimes in a neural network (NN) transfer setting. Furthermore, the authors provide impressive and insightful analyses on (1) how the dataset distillation performance behaves as knobs of the algorithms are tuned (2) what are the properties (spectral, intrinsic dim., visual) of the distilled data. Not every paper needs to propose a new method or new theory. Here, thorough analysis and understanding of an existing good method should still be appreciated.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I very much like the paper overall. It is not a groundbreaking paper, but provides deep understanding into an existing method. It is clearly written, and addresses some of my questions I had when reading Nyugen 2021. The analysis are thorough, and I particularly liked the comparison between initialization and learned images, since dataset distillation methods can suffer from spurious minima that prevents the images to be optimized freely.\n\nOverall, I think this is good paper. My questions and comments fall into the following three categories:\n\nClarity:\n1. What's the dataset Fig. 2 is based on?\n2. For NN transfer results in Table 2, why are the numbers better than Table A6 of Nyugen 2021 which used the same method? Is it due to using different kernels.\n\nUnanswered questions:\n1. How well can KIP results generalize to NN of *different* architectures?\n2. From your analysis, what do you think are the limitations of KIP and LS approaches? What could be the directions for improvement? \n3. Are there any scalability analyses for KIP and LS?\n\n~~Missing computation cost information:~~\n1. ~~The authors claim to provide computation cost information in the appendix. However, the appendix lacks the number of GPUs, the types of GPUs, the time (wall clock / GPU time) needed, and cluster information.~~\n\n\n~~To me, the paper should have rate 6. However, due to the misinformation in the checklist, I am currently giving a rating of 5, and will update when the authors provide the missing information during rebuttal period.~~\n\n--- \nUpdate (8/12): The authors have provided my requested information.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mostly interpret limitations as the sensitivity of KIP and LS w.r.t. hyperparameters and settings. That topic is important, and well discussed in the paper indeed. I, however, would also love to see discussion on the potential limitations of such kernel-based approaches overall. In my opinion, this paper doesn't have potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers had some mixed feelings concerning this paper. On one hand, the reviewers felt the paper direction is important and well justified. On the other hand, some large scale results are missing and would overall make the claims of this paper stronger. However, the main concern was raised by Reviewer AopF: I believe the paper structure is quite difficult to follow (the "Set up" section looks more like a Related Work section and the method is never clearly specified as the next section is the Experimental section) and a significant rewriting is needed before accepting this paper. Thus I recommend the authors to take this in account and to resubmit this paper in another venue.
Decision: Accept (Poster)