Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a set of changes/improvements to a prior SOTA dataset distillation algorithm (KIP), that allows to achieve comparable or better results at a fraction (e.g. 1/100) of the compute cost.

The changes are:
- Using finite-width NNGP at training time (vs the infinite-width NTK), allowing a dramatic compute improvement.
- Using the infinite-width NNGP at test time (vs the infinite-width NTK).
- Using cross-entropy loss with platt-scaling and a trainable temperature parameter.
- Representing the optimized support set as a matrix product (trainable preprocessing) instead of a single matrix.
- Using centering and label scaling for RFAD -> NN transfer.

strengths_and_weaknesses: The paper clearly demonstrates a dramatic speedup in achieving comparable to prior SOTA results, which is why I recommend acceptance. However, it does so with a moderately-sized bag of various tricks, and I find the analysis/interpretation/ablations of the results somewhat lacking. Therefore my recommendation is only weak.

## Strengths:

- Clearly demonstrated comparable or better to SOTA results at a fraction of the compute cost.
- The paper overall is clear and easy to follow.
- Code is released.

## Weaknesses:

- The compute cost analysis ($|S|$ vs $|S|^2$; section 4.1, abstract, conclusion, etc) appears vague and/or potentially wrong. 
	* Firstly, the asymptotic inference costs of both KIP and RFAD are the same due to using the infinite-width kernel; RFAD could be a constant 2x faster than KIP due to using NNGP vs NTK. For a test set of size $T$, this inference cost would be something like $|S|^3 + (|S|^2 + |S||T|)H^2 W^2 D$, i.e. the cost of the support set matrix inversion plus the cost of computing the support/test set kernels.
	* At training, cost of a single KIP iteration should be same as above (where $T$ is the training set size; I think complexity on line 105 is incorrect), while RFAD would be something like $|S|^3 + (|S|^2 + |S||T|) N M + (|S|+|T|)N M^2D$, if $M$ is the number of channels (i.e. the cost of support set matrix inversion plus the cost of computing the NNGP kernels, plus the cost of computing the features for the NNGP kernel). The source of the speedup might be the relatively small $N M$ compared to $H^2 W^2 D$; but IIUC for large $|T|$, both methods remain linear in $|S|$. So I don't see immediately why KIP has different scaling with $|S|$ that your method, and how exactly you plot the KIP curve in Figure 2.
    * My quick and rough math above might be wrong, but either way the authors should provide a more rigorous complexity analysis of both methods, and revise or substantiate their claims.

- No ablations of RFAD with and without Platt scaling cross-entropy loss, and with/without trainable matrix product (as described in the appendix).

## Minor:

- Some minor typos, e.g. Line 691: "Empirical NTK" -> "Empirical NNGP". Line 676 - capitalize "Table 5". Line 671 - "our times", line 149: "Probabilistic interpretation of MSE for classification?" etc.

- Could be useful to have an ablation table with results obtained from doing inference on the test set with the same exact NNGP as used during training (I assume you have these results from the appendix figures).

- Would be interesting to also compare against the DC/DSA compute costs.

- Codebase uses two frameworks (JAX and PyTorch), which is slightly inconvenient.
questions: - In Table 2, are random seeds used to initialize RFAD, and those used to initialize NNs, completely disjoint? To make sure there is no benefit gained due to seed reuse.

- Line 619 "this does not add any extra variables to the coreset" - why not, doesn't it add new $(CHW)^2$ trainable parameters?

- Why do you present results with centering and label scaling in Table 2? Wouldn't it be more fair to not use these settings, since I assume they weren't used in KIP or DC/DSA (and/or rerun those methods with centering and label scaling), to which you compare? This way we could better disentangle the benefits of the images themselves vs these optimization settings.

limitations: This work will help lower the carbon footprint of the industry, and make dataset distillation algorithms more accessible to the broader community, so I believe it only has a positive societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed an approximation algorithm to accelerate the Kernel Inducing Points (KIP) method with neural kernels for dataset distillation problem, which is mainly based on the random feature approximation of the Neural Network Gaussian Process (NNGP) kernel. To save the time cost of the KIP algorithm with neural kernels, the paper focused on approximating the NNGP kernel instead of the Neural Tangent Kernel (NTK), and proposed a simple approximation algorithm with empirical NNGP features which are easily generated from finite-width random neural networks. Besides, the paper changed the square loss to cross-entropy loss when learning the inducing points. In experiments, the efficiency superiority of the proposed method RFAD is verified with limited loss on performance. In addition, despite training KIP on the NNGP kernel which is naturally friendly for computation, the obtained distilled data could also apply to NTK and finite-width neural networks.
strengths_and_weaknesses: Stengths:

1. The efficiency superiority of the proposed method RFAD is impressing. The reduction on the computational cost for KIP on NNGP over MNIST and CIFAR is significant compared to prior works [Nguyen et al., 2021], and succeed to make the KIP with neural kernels more practical for real scenarios. The engineering efforts should be admired.

2. The paper is technically sound and well presented with broad literature review. Extensive experiments could support the effectiveness of the proposed method.

Weakness:

1. The main weakness of the paper is: the novelty is limited, especially compared with the previous work [Nguyen et al., NeurIPS'21; Nguyen et al., ICLR'21]. The approximation scheme is direct, and theoretically guaranteed, and already verified in the experiments of existing work. Besides, the improvement on MNIST and CIFAR is significant in [Nguyen et al., 2021], and before that we can not imagine that one can get 64.3% performance on CIFAR with only 1 image per class. However, from training KIP with KRR on NTK, to training KIP with KRR on NNGP, and then to training KIP with KRR on random finite-width neural networks and cross-entropy loss. The theoretical rigor is getting weaker and weaker.

2. The Platt scaling attributed highly to the performance of RFAD, but not compared separately.
questions: 1. What does it mean by "n=4"|"n=12" in the tables?
2. What is the difference between fixed and learned label configurations? For fixed label configuration, do you mean the label solving (LS) method?
3. As claimed in the experiments section, the performance gain is primarily attributed to the use of Platt scaling. Does this mean that if using cross-entropy loss, original KIP will perform even better than the results in tables?
4. It looks that replacing the loss function is independent to the speeding up of the algorithm, thus a comparison with RFAD without cross-entropy loss, or KIP with cross-entropy loss is necessary.
5. What are the technical challenges in the implementation? They should be written in the paper.

minor issues:

Page 8 and Page 4. The subscripts of the letters in the formulas are too large.

Besides, compared to many related work about coresets, Reduced Set Construction is more similar to this paper and dataset distillation, which was studied in:

Scholkopf, B.; Mika, S.; Burges, C. J. C.; Knirsch, P.; Muller, K.; Ratsch, G.; and Smola, A. J. 1999. Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks 10(5): 1000–1017.
limitations: NA.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a data distillation inspired by KIP. While KIP is the current state-of-the-art data distillation model, the complexity of such an algorithm becomes impractical when dealing with large models as well as large datasets. The authors refer to the use of an approximated variant of the NNGPs via Random Features to accelerate the KIP algorithm while maintaining comparable results.
strengths_and_weaknesses: * Strengths:
    1) The paper presents a practical algorithm for dataset distillation.
    2) It was a great idea to introduce an alternative loss function, replacing the MSE to obtain better results.
    3) The authors shed light on the applicabilities of their methods, from privacy, interpretability, and data corruption.
    4) The paper is clear and properly written.

* Weaknesses:
    1) From a theoretical point of view, what is the approximation factor of the approximated NNGP kernel matrix with respect to the NNGP kernel matrix? Please do shed some light on this.
    2) Similar to KIP, only coresets of size 10 were computed when handling the CIFAR-100 dataset. See Questions.
questions: 1) Using a different loss than the MSE loss function is indeed clever. However, would it be better to use a more robust loss function than that of the MSE, for example, $\ell_1$-loss for example? 
2) Is it possible to run your algorithm with coreset_size=50 with CIFAR-100? If not, is it related to insufficient memory?
3) is the proposed method applicable when using the Imagenet dataset or a more complex deep learning model, e.g., ResNet50?
 
limitations: The authors clearly stated their limitations.
review_title: null

Meta-review: metareview: The paper provides a novel and practical algorithm for dataset distillation. The paper is clearly written and the reviewers felt that this is a nice contribution to the field. The results demonstrate a clear improvement over state of the art and the experiments are sound. The reviewers raised concerns about limited novelty of the paper but, after a fruitful discussions, agreed that the paper should be accepted.  
confidence: Certain
Decision: Accept