Dataset Distillation with Infinitely Wide Convolutional Networks

Abstract: The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data. As model and dataset sizes increase, dataset distillation methods that compress large datasets into significantly smaller yet highly performant ones will become valuable in terms of training efficiency and useful feature extraction. To that end, we apply a novel distributed kernel-based meta-learning framework to achieve state-of-the-art results for dataset distillation using infinitely wide convolutional neural networks. For instance, using only 10  datapoints (0.02% of original dataset), we obtain over 64% test accuracy on CIFAR-10 image classification task, a dramatic improvement over the previous best test accuracy of 40%. Our state-of-the-art results extend across many other settings for MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100, and SVHN. Furthermore, we perform some preliminary analyses of our distilled datasets to shed light on how they differ from naturally occurring data.


Reviews and author responses:

 Dear all, 

In our submission, we mentioned that one of our contributions would be that we would open source the datasets we constructed (using 1000s of GPU hours). We are happy to report that the data is now available (over 760 checkpoints, spanning 38 different sets of hyperparameters, hosted through GitHub / GCS) and ready to be shared with the NeurIPS audience once the paper is de-anonymized. We feel it is worth pointing out that this is a valuable contribution to the research community by enabling the study of dataset distillation on data that would otherwise be inaccessible to those without large computational resources.

Best,
Authors

 Thank you for your comments. Since reviewers cannot see an updated manuscript, it's hard to accept a paper with so many issues even when there seems to be a clear path for improving it. I would suggest including all those changes and resubmitting. On top of the major changes you need to do (add a methodology section, clarify your contribution etc.,) I would also suggest including a discussion of computational cost since that is a con of your method.

 Firstly, we would like to acknowledge your comments are very helpful in revising our manuscript.

> To clarify, the point about too many experiments was to say that you need to relate each experiment with your specific proposed method... Otherwise you have a set of orthogonal observations that aren't related to one another.

* If you mean that our image analysis is orthogonal to their SOTA performance, then we partially agree, in that we don’t know whether intrinsic dimension (ID) or other metrics are important for DD or generalization in general. Investigating whether these properties have a causal influence on generalization is an interesting direction, but certainly out of scope for this work. For this reason we did not think comparing these simple metrics between our and other works will be insightful, since no matter how they compare, we can’t be sure that  they can explain generalization sufficiently to draw any conclusions (For example, as we discuss in the paper, our images are less interpretable than those of DC [3] / DSA [4]; however, this property is indeed “orthogonal” to their performance, and we don’t claim that our results outperform SOTA _because_ they are less interpretable).

* However, if you mean that our image analysis observations are orthogonal _to each other_, then we respectfully disagree. As we describe above in 2.b), all these experiments are related as they analyze image complexity from different angles. Yet establishing a causal relationship between dataset complexity measures and how they can predict generalization is an entire subdomain of ML (see e.g. [1]).

Nevertheless, we decided to do a preliminary investigation of ID on prior art as compared with our work. In the table below we have evaluated the ID of vanilla KIP [2] and DC/DSA. While our work tends to produce images with higher ID, this doesn’t always happen (see SVHN). Ignoring training hyperparameters (e.g. use of augmentations) there doesn’t seem to be a strong correlation between ID and test accuracy (though this preliminary investigation suggests that by excluding augmentations, i.e. excluding DSA, results in ID being potentially positively correlated with test accuracy)

| Dataset       	| Baseline 	| FC-KIP [2] 	| Ours     	| DC [3]   	| DSA [4] 	|
|---------------	|----------	|------------	|----------	|----------	|---------	|
| CIFAR-10      	| 22.3     	| 37.5       	| **51.9** 	| 42.3     	| 23.4    	|
| Fashion-MNIST 	| 13.1     	| 22.1       	| **27.2** 	| 23.6     	| 17.2    	|
| MNIST         	| 12.8     	| 29.5       	| **38.6** 	| 19.8     	| 20.4    	|
| SVHN          	| 17.2     	| 28.8       	| 34.4   	| **37.8** 	| 21.3    	|


It is also interesting that all distillation techniques considered increase the ID compared to natural images, and we would be happy to include this observation and evaluation of other metrics in our next revision. 

To conclude, while we agree that our analysis provides only a glimpse into the nature of KIP-distilled images, we argue that (1) it is coherent and provides a specific qualitative insight with practical consequences, (2) it is much more comprehensive than any of [2, 3, 4], and (3) a more thorough investigation, especially one that covers all prior DD techniques, is an ambitious effort worthy of a separate paper.

> Secondly, while your results are impressive it's simply not enough if I don't understand what part of your method leads to such improvements. 

Our results are SOTA due to using KIP [2] in combination with deep infinitely wide convolutional networks _with pooling_. Other aspects like ZCA and data augmentation are beneficial but not necessary to obtain a large improvement over prior SOTA as demonstrated in ablation Table A3 (as long as one is willing to consider kernels vs neural network methods).

To our knowledge prior SOTA on the tasks considered were FC-KIP [2], DC [3], and DSA  [4].

* **Why better than FC-KIP [2]?** We perform much better due to using convolutional networks _with pooling_. While this may seem like a small change, it actually plays an enormous role in the infinite width limit. In short, infinite width vanilla CNNs coincide with the infinite width limit of locally connected networks (CNNs with no weight sharing) and thus ignore correlation among pixels with different spatial locations. This means that infinite width vanilla CNNs are unable to capture translation invariance. In contrast, infinite width CNNs with pooling _are_ translation invariant, and hence much more powerful image models than those without, and this is the reason we can obtain such strong results. For details see [5, Figure 1, section 5], [6, Table 1].

* **Why better than DC [3] / DSA [4]?** Both approaches seek to synthesize a dataset such that _training_ on it will produce gradients close to those produced on natural images. However, this is a surrogate objective. In our work, we can easily backpropagate through the _generalization loss_, since in the infinite width limit generalization to unseen images becomes the explicit, closed-form expression in Equation 1! Therefore, our optimization objective is better aligned with the end goal (generalization), and this is likely the reason we outperform DC/DSA.

> Why does your model require more compute than other models? 
* **Why more than vanilla KIP [2]?** Just like pooling layers dramatically increase performance of our models, they also dramatically increase the compute cost (see e.g. [5, section 3/4], [6, section 4]). Roughly, the cost is increased by a factor of `number of pixels`, which is ~1000-fold (!) for CIFAR10. Hence the necessity and value of our distributed framework, allowing us to perform computations 3 orders of magnitude more costly.

* **Why more than DC [3] / DSA [4]?** 
Per the same references as above, computation with infinite networks with pooling is quadratic in the product of `number of pixels` and `dataset size` (due to them being kernel methods). DC/DSA, as is common for neural networks, scale linearly with these factors.

> If you train other models for longer do they do better? 
* **Vanilla KIP [2]** - No, it would not benefit from more training, since authors train their models to convergence / flattening [2, section B], and we see no other straightforward way to leverage more compute with their method.

* **DC [3] / DSA [4]** - It is unclear how this method would benefit from additional compute: the authors also point out that having more training iterations does not lead to better performance [3, Figure F6]. Likewise, they mention that having a larger batch size does not help [3, Section A] (unlike for our method). 

> All of those questions would be easier to answer with a method section. If your main contribution was months of development to scale up KIP, it should be in the main text along with why your model is more expensive etc.

We agree, and your feedback has helped us a lot in revising the paper! While we can’t upload revisions, we hope that the above replies clarify our methodology. 

> Finally, if the ultimate point of your paper is better DD performance, it is very hard to avoid continual learning benchmarks compared to other DD methods. This is because if you're not contributing a new algorithm per say it has to be really clear that the performance gain in DD is useful for something.

We are happy to report that based on some quick initial experiments, we have obtained promising results for KIP data for use as a memory replay buffer for (task incremental) continual learning. Specifically, using the split CIFAR-100 benchmark (twenty disjoint 5-way classification tasks with task boundaries and separate heads for each task), we achieve the following mean test accuracy over 17 train tasks at the end of training (3 were used as validation for tuning the learning rate): 

| Method   | Baseline       | Multitask      | Replay (natural) | Replay (KIP)   |
|----------|----------------|----------------|------------------|----------------|
| Mean test accuracy, % | $36.6 \pm 0.8$ | $\boldsymbol{{81.2} \pm 0.3}$ | $73.1 \pm 0.9$   | $74.8 \pm 0.6$ |


Here 
* baseline refers to training one task after another with no replay buffer, 
* multitask refers to training on all 17 tasks simultaneously, and 
* replay (KIP/natural) involves using a growing memory buffer of 10 images per class for previous task (thus 50 images per task), where “natural” involves using randomly sampled images from the dataset while “KIP” involves using KIP learned data trained beforehand (where all 2500 train images from a 5-class task are distilled down to 50 support images). The ConvNet architecture was used for the neural network training (and KIP algorithm). Numbers are reported with respect to 5 seeds.


As expected, memory replay mitigates catastrophic forgetting (natural replay beats baseline), but we also see a meaningful improvement of KIP data over natural data. We think the positive outcome of this quick test experiment makes for promising future work in studying KIP as an effective algorithm for producing replay data for continual learning. In particular, it validates the versatility of KIP data for training networks that are not necessarily initialized from scratch.

We would appreciate the reviewer considering a higher score for our paper if we have adequately addressed the concerns.


**References:**

[1] [Fantastic Generalization Measures and Where to Find Them](https://arxiv.org/abs/1912.02178)

[2] [Dataset Meta-Learning from Kernel Ridge-Regression](https://arxiv.org/abs/2011.00050)

[3] [Dataset Condensation with Gradient Matching](https://arxiv.org/abs/2006.05929)

[4] [Dataset Condensation with Differentiable Siamese Augmentation](https://arxiv.org/abs/2102.08259)

[5] [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://arxiv.org/abs/1810.05148)

[6] [On Exact Computation with an Infinitely Wide Neural Net](https://arxiv.org/abs/2102.08259)


The paper performs thorough analysis and exploration on the implication of the proposed KIP and LS methods by Nyugen 2021 for dataset distillation. KIP and LS are based on the inf-width network limit as kernels, and devised based on kernel Ridge regression, and has been shown in prior work to have good dataset distillation performance. This paper greatly extends the analysis in Nyugen 2021, by investigating how KIP and LS can be used to achieve SOTA dataset distillation results in a kernel setting, and also sometimes in a neural network (NN) transfer setting. Furthermore, the authors provide impressive and insightful analyses on (1) how the dataset distillation performance behaves as knobs of the algorithms are tuned (2) what are the properties (spectral, intrinsic dim., visual) of the distilled data. Not every paper needs to propose a new method or new theory. Here, thorough analysis and understanding of an existing good method should still be appreciated.  I very much like the paper overall. It is not a groundbreaking paper, but provides deep understanding into an existing method. It is clearly written, and addresses some of my questions I had when reading Nyugen 2021. The analysis are thorough, and I particularly liked the comparison between initialization and learned images, since dataset distillation methods can suffer from spurious minima that prevents the images to be optimized freely.

Overall, I think this is good paper. My questions and comments fall into the following three categories:

Clarity:
1. What's the dataset Fig. 2 is based on?
2. For NN transfer results in Table 2, why are the numbers better than Table A6 of Nyugen 2021 which used the same method? Is it due to using different kernels.

Unanswered questions:
1. How well can KIP results generalize to NN of *different* architectures?
2. From your analysis, what do you think are the limitations of KIP and LS approaches? What could be the directions for improvement? 
3. Are there any scalability analyses for KIP and LS?

~~Missing computation cost information:~~
1. ~~The authors claim to provide computation cost information in the appendix. However, the appendix lacks the number of GPUs, the types of GPUs, the time (wall clock / GPU time) needed, and cluster information.~~


~~To me, the paper should have rate 6. However, due to the misinformation in the checklist, I am currently giving a rating of 5, and will update when the authors provide the missing information during rebuttal period.~~

--- 
Update (8/12): The authors have provided my requested information. The authors mostly interpret limitations as the sensitivity of KIP and LS w.r.t. hyperparameters and settings. That topic is important, and well discussed in the paper indeed. I, however, would also love to see discussion on the potential limitations of such kernel-based approaches overall. In my opinion, this paper doesn't have potential negative societal impact.

 Thank you for clarification! My questions are mostly answered and I have raised my score accordingly.

 Thanks for getting back to me.

To clarify, the point about too many experiments was to say that you need to relate each experiment with your specific proposed method. For instance, if you think intrinsic dimension is a relevant metric to consider for dataset distillation (DD), it should be evaluated on different DD methods (perhaps your method performs better BECAUSE it increases the intrinsic dimension more that other methods). Otherwise you have a set of orthogonal observations that aren't related to one another. 

Secondly, while your results are impressive it's simply not enough if I don't understand what part of your method leads to such improvements. Why does your model require more compute than other models? If you train other models for longer do they do better? All of those questions would be easier to answer with a method section. If your main contribution was months of development to scale up KIP, it should be in the main text along with why your model is more expensive etc.

Finally, if the ultimate point of your paper is better DD performance, it is very hard to avoid continual learning benchmarks compared to other DD methods. This is because if you're not contributing a new algorithm per say it has to be really clear that the performance gain in DD is useful for something. 

 We thank the reviewer for their time and providing helpful feedback! Here are our responses to specific comments, especially on the significance, from the reviewer.

> Transfer leading to gain in performance

This fact was also pleasantly surprising for us. In principle, there’s no reason the kernel's performance upper bounds the finite width neural network transfer. In most cases, since we directly optimized validation performance with kernels, typically there is performance loss during transfer. However, since we are training neural networks beyond the lazy training setting (high learning rate, L2 regularization, learning rate decay), the transfer could potentially outperform kernel cases (as commonly observed for undistilled full images on CNNs; see [1], [2])

>  datasets used in the experiments is very small

We agree that scaling up to larger and more realistic datasets beyond academic benchmark is an important future direction. Note that the fields of dataset distillation and infinite-width networks are still young and so far most prior work benchmarked at most on datasets we’ve considered. For KIP using ConvNet, currently there are limitations for naively scaling up to ImageNet due to high image resolution and sheer extent of dataset size. Solving those challenges would be an exciting future direction.

One interesting idea for scaling to larger datasets includes using faster kernel approximation and inference such as sketching [3]. Such an approach can speed up kernel computation by a factor of several hundred. 

Another possibility is to divide and conquer to handle larger support sets. For instance running KIP/LS on disjoint partitions of the target dataset and then taking the union of the distilled dataset as the distilled version of the target dataset. This would ameliorate the costs of kernel-ridge regression arising from cubic scaling in the support set size.


> effect of initial point

The exact initialization does not affect performance much (as is consistent with [4]). We used a fixed seed for random subsampling for each run (for simplicity), but we also tried random noise images and obtained similar results. Note that this is discussed in the SM, but thanks to the reviewer's suggestion, we plan to make them more prominent in main for the next revision. 

> robustness to adversarial attacks

This is a great question. We have not explored this direction, but given that we are open sourcing our datasets, this is also a question that the research community can explore.

> on originality

Our work stands apart from [4] in several ways:

1) While our main algorithm is directly from [4], the engineering novelty of our work is significant and cannot be overlooked. It was only through both the coordination of hundreds of GPUs and the use of `jax.vjp` to make the backward differentiation efficient that the KIP training could be done feasibly with ConvNet. This took months of engineering and was not a straightforward implementation of [4], in which a single machine is sufficient and communication between workers is not required. Such engineering effort is justified by the fact that our work often improved upon prior work by surprisingly large margins (up to 37% gain in absolute test acc; SVHN 1img/cls). A modern theme in machine learning is to explore the benefits that come with scaling up (and implicitly, how they compare to algorithmic developments at a fixed scale). We believe our margin of improvement well exceeds the bar for how much either scaling up or algorithmic innovation improves performance, and thus worth sharing with the NeurIPS audience. 
2) Our analysis of the KIP images reveals quantitative insights into what distinguishes distilled data from naturally occurring data. In particular, our observation that intrinsic dimension increases and that performance concentrates around low eigenvalues of the kernel matrix (instead of high ones) are surprising and suggests that KIP images are more complex than natural images in ways that deserve further study. Note that such increase in complexity is a novel unexpected finding. 


We hope we have addressed the reviewer’s concerns and are happy to answer any remaining concerns or questions. 

References:

[1] [Arora et al., On Exact Computation with an Infinitely Wide Neural Net, NeurIPS 2019](https://arxiv.org/abs/1904.11955)

[2] [Lee et al., Finite Versus Infinite Neural Networks: an Empirical Study, NeurIPS 2020](https://arxiv.org/abs/2007.15801)

[3] [Zandieh et al., Scaling Neural Tangent Kernel via Sketching and Random Features.](https://arxiv.org/abs/2106.07880)

[4] [Nguyen et al., Dataset Meta-learning from Kernel Ridge-Regression, ICLR 2021](https://arxiv.org/abs/2011.00050)



 Thank you very much for your thoughtful review! Responses to your specific points below:

#### Clarity:

> 1. What's the dataset Fig. 2 is based on?

Figure 2 is CIFAR-10 500 images with ZCA. This will be stated explicitly in the next revision.

> 2. For NN transfer results in Table 2, why are the numbers better than Table A6 of Nyugen 2021 which used the same method? Is it due to using different kernels.

Correct, unlike [1], we consider deep convolutional networks _with pooling_, which makes these models translation-invariant and is a major source of improved performance in both finite- and infinite-width settings (see e.g. [3] and [4] for discussion).


#### Unanswered questions:

> 1. How well can KIP results generalize to NN of different architectures?

We provide some preliminary results for this in Figures 1 and 2. Specifically, Figure 2 shows how minor variations to the NN architecture such as using different normalization layers still leads to major improvements in generalization. More substantial changes such as removing pooling (Figure 1, right, bottom row) or inserting local average pooling (`Myrtle5` in Figure 2) still lead to improvements, but admittedly of smaller magnitude. We believe that kernel sampling (section 3.2) should make KIP images even more robust to such architectural changes and would like to investigate this direction in future work.

> 2. From your analysis, what do you think are the limitations of KIP and LS approaches?

Both methods have compute costs that scale unfavorably with the dataset sizes, as is common for kernel methods. Precisely, assuming target set size $T$ and support set size $S$, kernel ridge-regression has complexity $\mathcal{O}\left(S \times T + S^3\right)$ time, while LS has complexity $\mathcal{O}\left(S^3 + S \times T^2\right)$ time. Further, for convolutional architectures _with pooling_, both methods are also quadratic in the number of pixels, as we discussed in section B.1. This is why we have developed our distributed computation framework (section B.2). We will include a more detailed discussion of the compute costs in the main text in the next revision.

>  What could be the directions for improvement?

Scaling up our approach would be an exciting future direction.

One interesting idea for scaling to larger datasets includes using faster kernel approximation and inference such as sketching [2]. Such an approach can speed up kernel computation by a factor of several hundred. 

Another possibility is to divide and conquer to handle larger support sets. For instance running KIP/LS on disjoint partitions of the target dataset and then taking the union of the distilled dataset as the distilled version of the target dataset. This would ameliorate the costs of kernel-ridge regression arising from cubic scaling in the support set size.

> 3. Are there any scalability analyses for KIP and LS?

We are uncertain what scalability analysis refers to in the question. The algorithms have two principal costs: the cost coming from kernel ridge-regression each training step and the cost from computing individual kernel elements. The first is well-known (cost of inverting a matrix / doing SVD) and the second is discussed in section B.1 but also discussed in prior works on convolutional infinite-width kernels, e.g. [3, section 3] and [4, section 4]. See also our discussion above, which we plan to incorporate into the next revision. If this does not answer your question, please clarify.


#### Missing computation cost information:

> The authors claim to provide computation cost information in the appendix. However, the appendix lacks the number of GPUs, the types of GPUs, the time (wall clock / GPU time) needed, and cluster information.

In the Supplementary Material, Section B.1 we discuss our GPU type (Tesla V100) and some back of the envelope style computations for kernel computation. For the remaining requested data, our runs typically used 100-400 GPUS (depending on server availability) per training run. For a forward pass, an 18x18 block of a ConvNet kernel matrix takes 80 ms to compute during the forward pass and 250 ms to compute during the backward pass.

So as an explicit example, for support size equal to 100 images and target batch size of 5K, this is roughly 500K kernel elements per train step, which is roughly 1.5K 18x18 kernel blocks to compute. This is 480s worth of compute per train step. Hence a single train step with 100-400 GPUS working in parallel takes on the order of a few seconds. When training for 50K iterations, the wall clock time for training works out to several hundred GPU-days.

A similar computation shows that training 10 images and 500-1000 images takes around several GPU-weeks and several hundred GPU-weeks, respectively. 

Servers usually were run on a machine with a single GPU, at least 50 Intel CPU cores and at least 150 Gb of RAM. Workers (clients) were run on machines with up to 8 GPUs, having at least 10 CPUs per GPU and at least 32 Gb of RAM per GPU.

We will provide these more explicit details in the next revision.

#### Limitations And Societal Impact:

> I, however, would also love to see discussion on the potential limitations of such kernel-based approaches overall. 

Thank you for bringing this up - as discussed above, kernel methods indeed have significant scaling challenges, and we will include a more detailed discussion on this in the next revision.

> To me, the paper should have rate 6. However, due to the misinformation in the checklist, I am currently giving a rating of 5, and will update when the authors provide the missing information during rebuttal period.

We are glad that you liked our paper. We hope we have addressed all your questions / provided missing info above, and would appreciate it if you could revise your score accordingly. If you need any further details, please let us know and we'll do our best to provide them.

References:

[1] [Nguyen et al., Dataset Meta-learning from Kernel Ridge-Regression, ICLR 2021](https://arxiv.org/abs/2011.00050)

[2] [Zandieh et al., Scaling Neural Tangent Kernel via Sketching and Random Features. Arxiv:2106.07880](https://arxiv.org/abs/2106.07880)

[3] [Novak et al., Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes, ICLR 2019](https://arxiv.org/abs/1810.05148)

[4] [Arora et al., On Exact Computation with an Infinitely Wide Neural Net, NeurIPS 2019](https://arxiv.org/abs/1904.11955)


 We thank the reviewer for their time and feedback.
1) _Major #1:_ 

    a) _“structure of the paper is poor”_:  We have tried our best to improve our structure in the upcoming revision. If the reviewer has specific concerns about the writing,  we are happy to make further changes - note that the other reviewers (see reviews by UNoV, zgVX) deemed our paper as “well” and “clearly” written. 

    b) _“very minor implementation tweak”_: While our main algorithm is directly from [1], the engineering novelty of our work is significant and cannot be overlooked. It was only through both the coordination of hundreds of GPUs and the use of `jax.vjp` to make the backward differentiation efficient that the KIP training could be done feasibly with ConvNet. This took months of engineering and was not a straightforward implementation of [1], in which a single machine is sufficient and communication between workers is not required. Such engineering effort is justified by the fact that our work often improved upon prior work by surprisingly large margins (up to 37% gain in absolute test acc; SVHN 1img/cls). A modern theme in machine learning is to explore the benefits that come with scaling up (and implicitly, how they compare to algorithmic developments at a fixed scale). We believe our margin of improvement well exceeds the bar for how much either scaling up or algorithmic innovation improves performance, and thus worth sharing with the NeurIPS audience.

    c) _“experimental section takes most of the paper, … are more appropriate for appendices”_:  We respectfully disagree on this point. We believe that empirical and scientific characterization of distilled dataset is an important contribution.  Our analysis of the KIP images reveals quantitative insights into what distinguishes distilled data from naturally occurring data. In particular, our observation that intrinsic dimension increases and that performance concentrates around low eigenvalues of the kernel matrix (instead of high ones) are surprising and suggests that KIP images are more complex than natural images in ways that deserve further study. 

    d) _“Methods”_: We agree that moving the discussion of the engineering in the appendix to the main body would improve the paper and plan to include this change in the next revision. We thank the reviewer for the suggestion!

    e) _“Related Work”_: While our related work section is intentionally brief, we believe they do cover the existing literature fairly broadly. If there’s concrete related work we are missing or misrepresenting, we are happy to incorporate those changes. 

2) _Major #2:_

    a) _“Checkpoints”:_ Each checkpoint refers to the number of KIP training steps. For each such step, there is a resultant learned dataset which is then transferred to a neural network training. By producing plots for many different checkpoints we demonstrate that KIP improvement in accuracy is robust to this hyperparameter as well (with the trend being that as KIP training progresses and checkpoint number increases, the gap between KIP and natural images widens further). The choice of checkpoints is explained in the SM. 

    b) _“Intrinsic dimension”:_ Intrinsic dimension (ID) is the dimension of the underlying input data manifold. Intuitively, simple datasets have a low ID as they can be described by a small number of coordinates on the low-dimensional data manifold. ID can be defined precisely and estimated differently depending on the assumptions on the manifold structure and the probability density function of the data. We use the “Two-NN” estimator from [2], which has been adopted in several notable machine learning papers (e.g. [3], [4]). The method is based on nearest neighbor distance statistics, and is somewhat involved to repeat the entire setup in our present paper, so we refer the reader to [2] for details. 
\
&nbsp;
\
&nbsp;
    The reason to look at ID is to better understand what KIP does to images: does it simplify them, by removing the noise and only keeping the core features present in many different images of the target set, or does it rather complexify them, by composing features from many targets together into a single collage? Increasing ID, together with other experiments (visual inspection and spectral analysis) indicates that KIP images become more complex than their natural counterparts. 
\
&nbsp;
\
&nbsp;
    All these findings can have practical consequences as well, for example the spectral analysis counter-indicates performing PCA dimensionality reduction on KIP images.
\
&nbsp;
\
&nbsp;
    We will make sure to improve exposition of this aspect in the next revision.

3) _Minor #1:_ We discuss initialization in experimental details in SM. It turns out the precise initialization has little effect (subset or random pixels do equally well). Both performance and visual appearance are quite similar.

4) _Minor #2:_ 5K images can be optimized for CIFAR-100 but prior art did not compute the number and it would computationally be ~5x slower to train than 1K images (time is the issue, not memory). Hence, we omitted this training. 

We hope we have addressed the reviewer’s concerns and would appreciate it if the reviewer could consider revising their score accordingly.

References:

[1] [Nguyen et al., Dataset Meta-learning from Kernel Ridge-Regression, ICLR 2021](https://arxiv.org/abs/2011.00050)

[2] [Facco et al., Estimating the intrinsic dimension of datasets by a minimal neighborhood information, Sci Rep 7, 12140 (2017)](https://www.nature.com/articles/s41598-017-11873-y)

[3] [Ansuini et al., Intrinsic dimension of data representations in deep neural networks, NeurIPS 2019](https://arxiv.org/abs/1905.12784)

[4] [Sharma & Kaplan, A Neural Scaling Law from the Dimension of the Data Manifold, 2020, arXiv:2004.10802](https://arxiv.org/abs/2004.10802)

This paper extends the Kernel Inducing Point (KIP) Label Solve (LS) algorithms from Nguyen et al. to support infinitely wide conv nets, and convnets of finite width, to obtain an impressive performance boost. 
  -----------------------------------PROS
- the performance gain is quite impressive on all datasets, both compared to the original KIP paper that didn't use convolutions and competing methods.

-----------------------------------CONS

MAJOR
- The structure of the paper is poor and it makes your contribution hard to grasp, or worse, look like it is a very minor implementation tweak to an existing method (change fully connected layers of KIP to conv layers). Section 2 is called "Set up" even though it's almost entirely background information, and your method should be explained and put in contrast with the work of Nguyen2021 before the experimental section. Your experimental section takes most of the paper and includes experiments that are more appropriate for appendices (it feels like the main contribution was a bit light and therefore you added lots of side experiments which aren't any more relevant for your method than any other method). This leaves you too little room for the Related Work section and Method section which are respectively too short and non-existent.
- Issues of clarity. Figure 4: I don't understand the value/meaning of checkpoints here? line 183: you need to explain what intrinsic dimension is and what it represents/ why it's meaningful in this context, since it's not a very famous concept. 


MINOR
- The most important setting to report in dataset distillation is whether or not you use a fixed initialization or a random initialization. I don't see that being reported anywhere, did I miss it? You would need to include both settings or the random-init setting only. I suspect you did the latter based on your performances. This setting changes the results entirely and changes how relevant your distilled images are.
- You're missing 50 images per class for CIFAR100 in Table 1, is it because 5000 distilled images is too much in memory?
- l210 typo "between"
 -----------------------------------QUESTIONS
- Have you tried continual learning experiments with your images? Are the gains as good as for test accuracy?

This paper performs dataset distillation to improve the deep learning training efficiency. By using the algorithm made by [1] and some expanded methods, the authors achieved the state-of-the-arts accuracy on CIFAR-10, MNIST, Fashion-MNIST, CIFAR10, CIFAR100, and SVHN datasets.

[1] DATASET META-LEARNING FROM KERNEL RIDGE REGRESSION, ICLR 2021  Originality: This is an expanded work of the previously developed method, but the authors expanded it well and achieved state-of-the-art results.

Quality: The submission is technically sound overall. 

Clarity: The submission is well written and very clear. The authors make it clear what has been done, what is typically done in this field of data distillation, and what has been done, and why it is novel/different.

Significance:  The significance of the work is moderate. On some small datasets, the authors can attain state-of-the-art results, but on the extremely large datasets such as ImageNet. Exploring this further would be interesting and valuable for the optimization community.


- Overall the paper is well written, quite thorough, and well-cited. In particular, the authors did an excellent job in the introduction; it was a concise, detailed yet intuitive presentation of data distillation.

- In line 139, the authors mentioned that "sometimes the transfer can even lead to gain in performance (LS for SVHN)". Why and how this gain can be observed?

- The size of the datasets used in the experiments is very small, such as MNIST, CIFAR, and SVHN. As the authors mentioned one of the motivations of this study is increasing the training efficacy, it would be nice to have an experiment on extremely large datasets such as the ImageNet.

- The init point of the training is from a random example of the training data, how much does the initial point affect the performance? Also, what happens if we start from a completely random image? 

-I just wonder that the model trained with the proposed method and the model trained on the original data could be equally strong to adversarial attacks? 
 -

Meta-review: Reviewers had some mixed feelings concerning this paper. On one hand, the reviewers felt the paper direction is important and well justified. On the other hand, some large scale results are missing and would overall make the claims of this paper stronger. However, the main concern was raised by Reviewer AopF: I believe the paper structure is quite difficult to follow (the "Set up" section looks more like a Related Work section and the method is never clearly specified as the next section is the Experimental section) and a significant rewriting is needed before accepting this paper. Thus I recommend the authors to take this in account and to resubmit this paper in another venue.

Decision: accept