Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##########################################################################

Summary:
 
The paper proposes a novel dataset condensation technique that generates synthetic samples by matching model gradients with those obtained on the original input dataset. This technique is investigated empirically on several smaller datasets like MNIST, SVHN and CIFAR10. Two applications to continual learning and neural architecture search (NAS) are also explored and show some promising results.

##########################################################################

Reasons for score: 

Overall, I vote for accepting this paper. The technique is intuitive and well-justified. Experimental results seem to suggest that it produces a synthetic set that compares favorably to those obtained using alternative methods. Also, additional applications of this technique to continual learning and NAS appear to be quite promising.

##########################################################################

Pros: 

1. The paper is well written. The core idea is arrived at systematically and is carefully explained.

2. The paper does a good job referencing prior work (with most papers that I knew of being included) and empirical results obtained by the authors appear to compare very favorably to this existing prior work. I do not think that presented empirical results are exhaustive, but they are definitely very promising.
 
##########################################################################

Cons.

I did not see any major problems with the paper. But wanted to make a few comments that could potentially be addressed:

1. I found a sentence "Note that each real and synthetic batch sampled from T and S contains samples from a single class and the synthetic data for each class are separately updated at each iteration" a bit confusing. Specifically, does this mean that all samples in T and S have the same label? If so, does this mean that in the case when we have only one sample per class, S contains essentially the same input sample (with or without image augmentations depending on the experiment)?

2. While S is built to match the gradients on the original architecture, it is a little counterintuitive that such a small training set would not cause dramatic overfitting on other (possibly "heavier") architectures. Do we need to use multiple synthetic sets in practice, or rely on heavy data augmentation to avoid overfitting? Or just limiting the number of training steps would be enough? It would be interesting to see a more detailed discussion of this aspect. While I am encouraged by NAS results, this remains one of my concerns.

3. It would be interesting to see results on larger datasets like ImageNet or at least CIFAR-100. I understand that this exploration would be quite computationally intensive, but this would make the results much more convincing.

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above. (I will update the score depending on the authors reply.)

#########################################################################

Post-rebuttal.

Thanks for a detailed response that clarified some questions and concerns that I previously had. I think the updated paper is stronger and I am inclined to raise the score to 8.

review_title: Interesting idea with promising empirical results

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper presents a method for generating synthetic datasets from the large realworld datasets. The CNN trained on such synthetic dataset supposed to have similar accuracy on the realworld data, as trained on the original one. The benefit of a such procedure is reduced model training time and storage space (for data).


The method is built on the idea that the gradients of the network being trained on the real images should be similar to gradients, which were obtained by the training on the synthetic images.
   
The method is validated on MNIST, SVHN, FashionMNIST and CIFAR-10 on several different architectures: MLP, AlexNet, VGG-like and ResNet architectures. 

Moreover, the paper compares the proposed method vs  many other baselines, e.g. methods, which select "representative" image from the dataset (coreset methods), as well as Dataset Distillation and cGAN. 

*****
Overall I like the paper a lot. The method is well-motivated, shows good (sota) results and also often (for MNIST, SVHN, FashionMNIST)  produces human-recognizable examples, although there is no term/regularization directly encouraging this. 

The paper also studies how architectural choices like normalization, pooling, etc. influence the generated samples and how samples generated for the architecture A are suitable for training architecture B.

I don't see any major weakness in the paper.


Questions and comments.

- In Figure 5, IMO it is bad practice to fit linear regression into blob-like data. (minor)
- Have you tried to add some term, encouranding the diversity for the different synthetic samples belonging to the same class? 

********

Post-rebuttal. 

The rebuttal didn't raised any concerns and made the paper even stronger, thus I am keeping my score.

review_title: Good paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper tackles the challenging dataset condensation problem. The goal is to learn to synthesize a small dataset, so that a neural network trained on the small synthetic dataset can have similar performance as a network trained on the full dataset. The proposed method tackles the problem by gradient matching. The proposed method achieves state-of-the-art performance, and shows promising results on two other downstream tasks, continual learning and neural architecture search.

Strength:
+ Comparing to existing approaches, the proposed method is efficient and effective, achieving the state-of-the-art performance
+ The authors use the synthetic dataset for two other downstream tasks and achieve promising results
+ The authors conduct extensive experiments to study and analyze the proposed method
+ The synthetic dataset trained on one architecture can be also used to train any other networks with different architectures, which makes to method more applicable 

Weakness:
- In Section 2.3 “Gradient matching loss”, the authors claim that the proposed distance metric is “a better distance”. It is probably better to use experiments or results to support this claim.

Other comments:
* Currently, the loss for the target task is cross-entropy loss only (Figure 1(b)). I wonder if this method can be used for other loss functions as well. Also, I wonder if the proposed method can be used for self-supervised tasks as well. Can the authors comment on these?
* It seems that the authors do not have any special designs to make the condensed synthetic dataset cross-architecture generalizable. I wonder why the proposed method has such a good cross-architecture generalization.
* In Figure 3, it seems that the performance on CIFAR10 saturates quickly as the number of images per class grows. Just curious, will the relative performance eventually reach 80~90% as in other datasets? If so, then what is the data ratio?

---- Post-rebuttal comments----

Thanks for the response. After reading other reviews' comments and the rebuttal, I think this paper is in a good shape now. Thus, I am willing to increase my score to 8 and recommend acceptance.
review_title: Good paper with extensive experiments

Meta-review: The paper introduces a novel dataset condensation technique that generates synthetic samples (images) by matching model gradients with those obtained on the original input samples (images). The authors also show that these synthetic images  are not architecture dependent and can be used to train different deep neural networks. The approach is validated on several smaller datasets like MNIST, SVHN and CIFAR10. This work is well-motivated and the methodological contributions convincing. All reviewers were enthusiastic and indicated that there were no flaws in this work. The rebuttal clarified outstanding questions and made the paper stronger.
Decision: Accept (Oral)