Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of data free knowledge distillation, wherein access to a (teacher) model trained on some data is assumed, however the training data is not available. DFKD tries to train a student model using just the teacher model. The paper focuses on one method for DFKD which trains a generator network to generate synthetic examples for distilling the knowledge of the teacher. The main contribution of the paper is to use the generator along with an exponential moving average of the generator’s weights for producing samples. The paper claims that this improves training stability, since the data distribution that the student network sees does not change abruptly. This helps limit catastrophic forgetting of the student network on data from generators at previous iterations. The paper has a thorough empirical evaluation of the proposed method on multiple image benchmarks.
strengths_and_weaknesses: Strengths - 
1. The paper proposes a simple trick to prevent catastrophic forgetting which does better than memory store techniques with lower overhead. The compute overhead is also lower than other DFKD methods.
2. The paper does extensive empirical evaluation of their method, along with a sensitivity analysis to various components of the approach.
3. The related works section is very well written.

Weaknesses - 
1. The empirical evaluation is over a single run, so it is hard to ascertain the significance of the results. Adding error bars would definitely help here.
2. The claim of the student undergoing catastrophic forgetting is not sufficiently explored. While the experiments show that having a higher momentum (and weight for the ema generator’s images) is better to an extent, a class wise breakdown of where such models do better, and how this improvement is related to catastrophic forgetting could help.
3. The method is not interpretable. This is a general shortcoming of such adversarial methods, and should not be taken as a very specific shortcoming of this work.

questions: 1. Can the authors provide more evidence around how EMA helps with the catastrophic forgetting of the student? An ablation comparing the training and test accuracy curves with and without EMA could help

2. Can this trick of using an additional generator be plugged into other DFKD methods which rely on synthetic data? If so does it always help, or are other methods of making training more stable better?

3. How would the performance change if the train and test set have some distribution shift within themselves? In that case, would EMA help?

4. Can EMA also be applied on the student to stabilize the generator's training?

5. While EMA might mitigate catastrophic forgetting to some extent, it is also possible that there is a large shift within the EMA generator as well, leading to some amount of catastrophic forgetting. While this effect can be controlled by the alpha parameter of decay, an alternative could be to maintain an EMA of the EMA generator and so on. Do the authors have any insights around this?

limitations: Some of the limitations are listed in the weaknesses above. More analysis of the generators and their impact on the student would be appreciated.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work focuses on Data-free Knowledge Distillation (DFKD). The main idea of existing DFKD is to use a generator to synthesize data for training the student. The authors propose Momentum Adversarial Distillation (MAD) to avoid forgetting the knowledge it acquired at the previous steps.
strengths_and_weaknesses: Pros:

Many existing data-free knowledge distillation methods adopt an adversarial training paradigm, where the generator is adversarial trained for distillation, making the data distribution different from the earlier. This work focuses on the important yet understudied distribution shift problem in DFKD and propose a reasonable method to solve the problem.

The paper is overall well written and easy to follow. The experiments are well designed and the results well demonstrate the superiority of the proposed method.

Cons:

The experiments of this work have already addressed most my concerns about the paper. The only one remaining concern is that the difference or superiority of the proposed method from some highly related works should be clarified. For example, MosaicKD [1] also studies the domain shift problem in knowledge distillation, the authors should make the differences from the work clearer or make some comparisons if possible.

[1] Fang G, Bao Y, Song J, et al. Mosaicking to distill: Knowledge distillation from out-of-domain data[J]. Advances in Neural Information Processing Systems, 2021, 34: 11920-11932.
questions: Please see the weakness.
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper shows that momentum adversarial distillation can help data-free knowledge distillation by solving the distribution shift problem. Extensive experiments are conducted on six datasets to demonstrate the effectiveness of the proposed method, including: CIFAR10, CIFAR100, Tiny-ImageNet, ImageNet, Places365 and Food101. 
strengths_and_weaknesses: Strengths:

-	The proposed method is simple and seems effective, also easy to follow for other research.
-	The paper is generally well-written with sufficient experiments on six datasets.

Weaknesses:

-	The momentum updating method itself is not novel and has been used in many problems, such as semi-supervised learning [1], self-supervised learning [2], etc. I’m a little bit concerned about the novelty.

	[1] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).

	[2] He, Kaiming, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. "Momentum contrast for unsupervised visual representation learning." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738. 2020.

-	It seems the performance of the proposed method is not competitive if compared to other existing DFKD methods, as shown in Table 1. The comparison with other DFKD methods in Table 2 is also not sufficient.

-	There can be more analyses and visualizations to show how momentum update helps alleviate the distribution shift issue, not just the synthetic samples.

questions: -	Clarify the key contribution and novelty of the proposed method since the momentum updating method is not a new technique and has been widely studied.

-	Provide more explanations on how momentum update handles the distribution shift issue in the proposed framework.

-	The performance in the paper is not competitive and the comparison is not sufficient in the experiments.

limitations: This paper has discussed the limitations and negative social impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In data-free knowledge distillation, a generator is used to synthesize data for training the student. This paper found that the distribution of synthetic data will change when the generator gets updated.  Thus, this paper proposes a new method called  Momentum Adversarial Distillation for data-free knowledge distillation. Experiments are done on six benchmark datasets to show the effectiveness of the proposed method. 

strengths_and_weaknesses: Strengths
1. The presentation of this work is clear and well-written.
2. The idea of the proposed method is straightforward.
3. I appreciate the sufficient experiments on different datasets and study on CIFAR100. They deliver the strong message that the proposed method is efficient.

Weakness
1. In figure5,  it is hard to conclude that the generated images by EMA Generator are better than the Generator. Could you use a metric to show how the difference for example the FID score?

2. In numerical experiments,  the results seems better than baselines. Results are mainly given the teacher and student are similar networks, how about the performance diverse architecture (resnet as teacher, alexnet as student)?

After rebuttal:
The reviewers have addressed my concerns. I insist my previous score after considering other reviewers' comments.
questions: Using EMA to stabilize the model training is a common technique in existing works. This paper applies it to the generator of images for data-free knowledge distillation. Thus the novelty of the proposed method is limited. Another concern is from Figure5, the EMA generator fails to improve the quality of generated images, which makes me concerned about the cause of the improvement of student performance. How about adding EMA to the student?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper trains a generator to produce synthetic data for knowledge distillation from a teacher model, thus allowing distillation without the need of the original training data. The reviewers generally liked and had positive things to say about the method as well as the presentation, and the discussion was mostly around clarification and having better comparisons. This seemed to have satisfied the reviewers who responded to the rebuttals (not all of them did), and from my reading the authors did a good job at responding to concerns of already mostly positive reviews. The one negative review, which I felt was a little off-the-mark, I feel was addressed well by the rebuttals, but the reviewer dropped out afterwards and did not back up their ongoing criticisms.

I therefore recommend acceptance of this paper to NeurIPS.

Overall, discussion was rather limited, but this could be that the reviewers didn't have and serious concerns from the start and discussion was straightforward. I wish tq3D had contributed a little more as it would have been nice to arrive to a consensus.
confidence: Certain
Decision: Accept