Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new data distillation approach based on neural feature regression that is similar to a truncated backgprop through time using a pool of models. The approach sets a new state-of-the-art results both in terms of accuracy and training efficiency.  Ample experiments showcase the advantages of the proposed approach.
strengths_and_weaknesses: Strengths:
- State-of-the-art results while being significantly more efficient
- Well written and structured with in depth ablation studies

Weaknesses & Q:
- L173, why no augmentation is applied during training? Shouldn't this prevent overfitting too?
- How does this approach fair in comparison with state of the art when combined with few shot learning approaches (that were targeted for real data)? 
- Since the paper mentions the computational advantages, perhaps this information could be added in the tables too. Thought figure 3 addresses this in part.
- Given the difficulty on modeling similar classes, it would be interesting to see how such method fair on fine grained classification (e.g. on CUB-200)
questions: Please see the Strengths and Weaknesses section
limitations: It would be helpful if the authors would have specified in the form where each content is addressed instead of simply stating Yes/No.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new method of dataset distillation  called “Neural Feature Regression with Pooling.” This work avoids using a surrogate objective  like previous works (DSAm DM, MTT) while also circumventing the computational constraints of other methods that optimize the true objective (DD, KIP). Like other methods, FRePo trains teacher networks from which to obtain “meta-gradients.” Unlike other methods, FRePo obtains the meta gradient by finding the closed form solution of the Kernel Ridge Regression problem posed by the final layer (linear classifier) of the teacher network. This method shows state-of-the-art results, outperforming previous methods in nearly every setting. Two other applications in continual learning and membership inference defense are also explored.

strengths_and_weaknesses: Strengths

The paper is very well written with very few grammatical errors (e.g., subject-verb agreement). All figures and tables are formatted well and clearly communicate the authors’ ideas.

As for the method itself, FRePo clearly outperforms the other methods in the explored settings. Another dataset (ImageNet1k resized) is also introduced and will likely be a new evaluation metric for future dataset distillation works.

Extensive visualizations are also included in the appendix.

--------

Weaknesses

I was disappointed to see that the fact that a new model was used (with respect to previous dataset distillation works) was not mentioned until deep into the appendix. While advocating for the adoption of a new backbone model is fine, this should be made very clear in the body of the paper. I understand that the authors re-evaluated the previous methods using this new architecture in Table 1, but the fact that a new architecture was used was not clear at all from the body alone. The authors should have also included results from their method on the original architecture. If this yields poor results, then this is a significant limitation of the method and should be addressed as such (but by no means detracts from the merit of this new contribution).

Furthermore, even given the appendix, it still remains unclear if these re-evaluations of previous methods use an architecture identical to the new one used by FRePo. The methods should have been re-evaluated with batch-norm, and results for FRePo using instance norm should have also been included. It is unclear how much of the improvement over previous state of the art is due to the algorithm, normalization type, or number of channels.

Similarly, Table 2 should also \textit{at least} include an additional row for FRePo trained using Conv-IN.

Also, if FRePo is faster and uses less memory than MTT, DSA, and DM, why were results for CIFAR100 and T-ImageNet not included here when they were included in previous works?

One other tiny thing: your labels for “Samoyed” and “Golden Retriever” seem to be swapped in all of your ImageWoof figures :P
questions: While my questions should be easy to answer, I think they are quite critical to the validity of this work. I will happily adjust my rating if they are addressed.

1. How does FRePo perform on the original architecture? (I believe you call it DC-Conv in the appendix). Even if these results are significantly lower, they should be included. They can even be left to the appendix as long as it is made VERY clear in the body that a new, wider architecture is being used and the reasons for this change are properly explained. (You mentioned in the appendix that the reason is to ensure the KRR behaves well. This explanation makes sense, but it is still a limitation of this work.)

2. Are the re-evaluations in Table 1 done using the \emph{exact} same architecture used to get the FRePo numbers? (i.e., is the same normalization type being used?) If not, how do the other algorithms perform using BatchNorm.

3. Regardless of the answer to the previous question, how does FRePo perform using InstanceNorm?

4. How is FRePo’s cross-architecture performance when trained on Conv-IN like all the other methods in Table 2 (aside from KIP)?

5. If FRePo is faster and uses less memory than MTT, DSA, and DM, why were results for CIFAR100 and T-ImageNet at 50 img/cls not included here when they were included in previous works?

-----

Edit: All my questions and concerns have been addressed.
limitations: The limitations on the size of the teacher model seem unique to this method among dataset distillation works and should be clearly addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to use Kernel Ridge Regression on the last layer of a slowly updated feature extractor to perform Dataset Distillation. There are several key contributions in this paper, including demonstrating KRR can work well in DD, using model pools, fast training speed and state-of-the-art results on DD benchmarks. The experiment section shows results from both DD and other applications including continual learning and membership inference attacks.
strengths_and_weaknesses: Overall, this paper presents a practical method that performs well both in terms of performance and training time. 

Strengths:
+ The paper contributed a new algorithm that uses KRR on the last layer and slowly updates the feature backbone to perform dataset distillation.
+ The paper well presents the algorithm and the related techniques and tricks for training
+ More stable performance on cross-architecture transfer is shown in table 2
+ The authors show that the algorithm can be used for higher-resolution training such as 128x128 images and 64x64 ImageNet dataset.

Weakness
- I'm curious how the authors handled the kernel computation in equation 2. If not mistaken, it seems the algorithm needs to compute the kernel for the full real dataset? Do you use minibatches instead?

Minor (no need to compare)
There are two recent new works on the same task, it might be worth adding them to the discussion/related work part.
1. Dataset Condensation via Efficient Synthetic-Data Parameterization
2. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks


questions: The paper nicely contributes to the DD task. I have two questions:
1. How does the algorithm compute the kernel matrix(see above)
2. Have the authors considered using implicit differentiation[24] instead, which also has low memory requirements? What would be the advantage of using KRR?
limitations: This work has potential benefits in improving privacy and understanding how neural networks perceive a task.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an efficient method for dataset distillation -- the goal is to extract/synthesize a small set of images that represent a dataset, so that deep learning models trained on the small set can achieve high (as high as possible) classification accuracy. The core contribution of this ppaer is to simplify and accelerate the conventional two-stage optimization procedure with TBPTT, where the gradients need to be propagated through a long chain. Instead, a kernel is used for computing the loss function, minimizing which leads to a single-step optimization approach which is highly efficient. The method is evaluated over a series of popular datasets including ImageNet which was never studied before.
strengths_and_weaknesses: Strengths

1. The studied problem is important and interesting.

2. The proposed approach is sound and effective. The idea is reasonable and the experimental results are good.

3. The paper is well presented.

Weaknesses

I did not see significant weaknesses of this paper. Although the solution is still preliminary (in the context of the large CV community) and difficult to scale up to larger datasets, the proposed method is indeed a very good progress on this direction. There are some limitations that I hope the authors to address or explain, which I will write down in the following part.
questions: 1. What is the major difficulty that prevents the proposed method from being applied to large images, e.g. 256x256 images? If one uses a small resolution setting to extract synthesized images (e.g. 64x64) and using techniques like GAN to up-sample it larger (e.g. 128x128), does it bring good performance on large datasets (e.g. ImageNet)?

2. What is the time and memory consumption for the ImageNet experiments, in particular, how long did the program run, and how many GPUs  (or other device) did it use? In addition, if one hopes to extract more training instances from ImageNet (e.g. 5, 10, 20, ... images per class), what is the computational cost w.r.t. the number of training instances?

3. If one has extracted, say, 10 images per class, and hopes to add 5 images per class for better performance, does the algorithm need to start from the very beginning (to extract 15 images per class), or can it start from the current point? In other words, what is the incremental performance of the proposed approach?

4. Are there any difficulties in applying the method to other vision problems, such as detection and segmentation? I do not mean to add experiments (though it would be great to do so), I just want to see the major difficulties of generalizing the method.
limitations: Please see the above comments.
review_title: null

Meta-review: metareview: The paper proposes a new algorithm for dataset distillation, based on two key ideas:  (1) train a linear layer given the fixed feature extractor, and (2) use a diverse set of modes as feature extractors. The paper has received overwhelmingly positive reviews. Many reviewers find the algorithm effective, the paper well-written, and the results compelling. The rebuttal further addressed the concerns regarding the backbone models and missing experiments as well as provided additional clarifications. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 
confidence: Certain
Decision: Accept