Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work the authors extend the Kernel Inducing Points (KIP) method for dataset distillation to a new set of kernel functions and achieve new state of the art results on dataset compression. Since the KIP algorithm is a key algorithmic mechanism in this work, I will summarize it: The KIP method optimizes what can be seen as a test residual of a target dataset fitted by a kernel regression model that is created using a support dataset. This means that given a target dataset distribution, one can differentiate through the kernel computations to evolve the support data points in the support dataset freely in the input space and optimize them. After optimization the support dataset (Which is usually taken to be much smaller than the target distribution) can be used as a distilled version of the dataset. This compressed dataset enables classification and training without training with extensively large datasets. In this work the authors use the KIP algorithm using a family of kernels corresponding to infinitely wide neural networks that were discovered in the realm of NTK. Such infinite NTK convolution kernels are known to provide state of the art results (when compared with other kernels) on classic classification tasks such as MNIST, CIFAR, etc. The biggest issue of such infinite-network kernels is that they require immense amounts of computation to construct the kernel. In this work, the authors construct such kernels at each training step (!) and then back-propagate through the kernels (!!) to optimize the datapoint. This impressive feat is made possible by considering 3 layer convolutional neural networks and working with a relatively small number of data-points (5K as opposed to the whole dataset). This is still highly computationally challenging and the authors create a careful implementation including an orchestration system working with 100s of GPUs for optimizing the support dataset. With this new kernel family the authors achieve state of the art results on dataset compression and analyze the compressed dataset using high dimensional statistics tools.
main_review: Significance:

This work presents new state of the art results for dataset distillation in the cases of simple classification datasets. Dataset distillation is an important topic for neural networks as it can enable various knowledge transfer methods via a shorter training loop. In addition the authors demonstrate some results on actual neural network transfer, since this is an important contribution, can the authors clarify and add in the main text what neural network architectures were used as the networks trained with the distilled datasets? Dataset distillation is most useful when the distilled dataset can be used among many different architectures. Can the authors share about experiments of using the distilled dataset to train different neural network architectures?

Since the authors utilize infinite-width networks kernels, the required computation for the method becomes very large and makes it naively infeasible. One of the key contributions of this work is the engineering component of this work which is highly non-trivial and is interesting and significant by itself.

Quality and clarity:

Overall this submission is very high quality with broad literature review, thorough experimental setup and in-depth analysis of the properties of the distilled datasets. Further the attached supplement provides ample details on the experimental setup in the appendix in addition to the new client-server set-up used for the large experiments.
Overall this is an interesting and well delivered work on dataset distillation but at the same time I am a bit torn due to some weaknesses. Some of the weaknesses of the work are the apparent lack of novelty of the idea, and irreproducibility (stemmed from the large amount of computation required). Further given the very large computational requirements of this method on simple datasets such as MNIST, and CIFAR I am concerned about the applicability of this method to larger datasets of interest. At this time I am going to assign this a score of (6) but if the authors can clarify my questions with regards to the transferability of the distilled datasets to different neural architectures I am willing to improve my score.

limitations_and_societal_impact: Large computational costs and focus on smaller datasets.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Paper mainly extends the dataset distillation algorithms KIP and LS (Nguyen et al. 2021) to infinitely wide neural networks. To that end, authors present a distributed framework that draws upon huge hardware resources and show improvement on the distillation performance. Paper also analyzes the synthesized data samples (images, labels) via multiple studies.
main_review: - Main drawback of the paper is its limited technical contributions. Authors extend the existing distillation methods to (primarily) infinite width neural networks.
- Also, while the proposed framework obtains SOTA results on multiple datasets, the transfer results (to finite width NNs) suffer performance drop and result in non SOTA results in 50% of the cases. However, the results are still very strong.
- The distributed meta-learning framework (one of the claimed contributions) developed for scaling the distillation algorithms to complex neural networks has only been discussed in appendix.
- Despite the above issues, the paper presents an important result of adapting the successful distillation algorithms to crucial neural network configurations. Further, the experimental analysis and studies performed on the distilled samples add strong value to the paper.
limitations_and_societal_impact: I do not see any potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors proposed a dataset distillation method using Kernel Inducing Points (KIP) and Label Solve (LS). The proposed method was designed for infinitely wide convolutional networks, however, when training a finite-width neural network model with the compressed dataset, the performance drop is small or moderate.
main_review: 1. The proposed method is very neat, as it is basically optimising the support set (training data for the training kernel matrix) for a kernel ridge regression problem.

2. The performance is reasonably good, even it is applied to the finite-width neural networks.

3. The discussion for the choice of lambda appears to be missing.

4. Line 235 -- 236, ", distributed metalearning framework that leverages hundreds of GPUs per training run" is confusing, can author elaborate which part of the model needs hundreds of GPUs and why?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes to use infinitely wide CNNs to perform dataset distillation – obtaining a tiny dataset that enables high accuracy. The approach is an extension of a prior work (Kernel Inducing Points (KIP), etc.). Results are shown on MNIST, Fashion-MNIST, SVHN, CIFAR-10/100 datasets.
main_review: The paper has following strengths:
1.	Results for ConvNet (3-layer ReLU CNNs) outperform prior models.
2.	Results for transferring to NN show good results for 1-sample distilled datasets.
3.	Preliminary but interesting study of generated images is presented.

The paper has several major weaknesses:
1.	It is unclear if this proposed method will lead to any improvement for hyper-parameter search or NAS kind of works for large scale datasets since even going from CIFAR-10 to CIFAR-100, the model's performance reduced below prior art (if #samples are beyond 1). Hence, it is unlikely that this will help tasks like NAS with ImageNet dataset.
2.	There is no actual new algorithmic or research contribution in this paper. The paper uses the methods of [Nguyen et al., 2021] directly. The only contribution seems to be running large-scale experiments of the same methods. However, compared to [Nguyen et al., 2021], it seems that there are some qualitative differences in the obtained images as well (lines 173-175). The authors do not clearly explain what these differences are, or why there are any differences at all (since the approach is identical). The only thing reviewer could understand is that this is due to ZCA preprocessing which does not sound like a major contribution.
3.	The approach section is missing in the main paper. The reviewer did go through the “parallelization descriptions” in the supplementary material but the supplementary should be used more like additional information and not as an extension to the paper as it is.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference on Learning Representations, 2021.



--------------------
Update: Please see my comment below. I have increased the score from 3 to 5.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper introduces a distributed kernel-based meta-learning framework for dataset distillation. The paper extends previous work Nguyen et al. [2021] to a large-scale training setting. 

The paper has received mixed reviews. The reviewers appreciate the significant improvement of dataset distillation performance on MNIST and CIFAR. However, several reviewers are concerned about the technical novelty compared to [Nguyen et al., 2021],  the scalability to large-scale datasets and large batch sizes, and the performance of downstream tasks. The authors have provided a rebuttal and addressed some of the concerns. The AC considers the improvement of dataset distillation on MNIST and CIFAR is significant compared to prior works. Before this paper, it’s hard to imagine that one can get 64.3% performance on CIFAR with 1 image per class for any machine learning algorithm. The AC also appreciates the engineering efforts and recommends that the authors move the distributed meta-learning algorithm from the appendix to the main text. The AC understands that the current method cannot handle ImageNet-level datasets. But given the low performance (29%) of previous dataset distillation methods even on CIFAR-10, this paper has presented a meaningful intermediate step towards the goal of dataset distillation. How to make it more efficient could be left as future work. Therefore,  the AC recommended accepting the paper. 



Decision: Accept (Poster)