Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a super straightforward approach for data-free knowledge distillation (KD). Instead of generating pseudo samples to simulate the original training images, it uses Gaussian noise samples as the input and proposes to use current statistics rather than running statistics in the BatchNorm layers for knowledge transfer, which greatly improves the performance. The proposed approach is evaluated with several image classification datasets and the results indicates that it outperforms other noise input-based data-free KD approaches.
strengths_and_weaknesses: Pros:
1. Using noise inputs from a Gaussian distribution is a more efficient way for data-free KD since it saves a lot of computation resources during the pseudo data generation process. 
2. The proposed approach is surprisingly easy.
3. Results look good compared to other noise input-based data-free KD approaches. 
4. The paper is well written and easy to follow.


Cons:
1. Some of the recent related works of data-free KD are missing:
[1] Nayak, Gaurav Kumar, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. "Zero-shot knowledge distillation in deep networks." In International Conference on Machine Learning, pp. 4743-4751. PMLR, 2019.
[2] Wang, Zi. "Data-free knowledge distillation with soft targeted transfer set synthesis." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, pp. 10245-10253. 2021.

2. Although the proposed approach is surprisingly simple and I agree with the authors that using current statistics is much better than running statistics. However, it is not quite intuitive to imagine why the outputs with current statistics are very close to those from the original training samples because the inputs are quite different. It could be better if the authors could provide more evidence in the ablation study.

3. I see that the performance of the proposed approach performs better than other works that use noise inputs as the baseline, but there is still room for improvement compared to pseudo image synthesized methods [1,2,3].
[3] Yin, Hongxu, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. "Dreaming to distill: Data-free knowledge transfer via deepinversion." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8715-8724. 2020.

4. As also discussed in the paper, there are some limitations of the proposed approach: (1) it only works when there are BatchNorm layers in the model, and (2) a large batch needs to be fed into the networks in order to get the statistics. These limitations slightly weaken the contribution of the paper.
questions: Overall, I think this is an interesting paper that proposes a super simple but effective way that uses noise inputs for data-free KD. Compared to traditional KD which puts a lot of effort into sample generation, the proposed approach saves tremendous computation resources. I'd like to see how the authors respond to the cons that I'm concerned about and will consider raising my score accordingly.
limitations: The authors have adequately discussed the limitations in the manuscript.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies data-free knowledge transfer, specifically a new frontier using only Gaussian Noise as inputs. As known to literature, noise-based data-free KD imposes critical shift in intermediate distributions, and results in very limited efficacy in distillation. The paper proposes a very simple trick to overcome the issue using current batch mean and variances to normalize, as opposed to running stats, and the method seems to work surprisingly well. Method has been explored on CIFAR10, CIFAR100, SVHN, and Food101 datasets over changing network architectures.
strengths_and_weaknesses: Strengths

+ The problem of using Gaussian noise only to data-free distillation is very interesting, as it bypasses the need to synthesize lots of data or train auxiliary networks.
+ The approach is rather simple though, as the main trick is to replace the running stats with batch-wise stats, and has to slightly change the inference flow, that results in small degradation in accuracy. But it's pretty impressive observation that simply doing this trick allows for significant boost in distillation accuracy.
+ I think this paper can inspire deeper analysis of the potential of noise-based KD. The analysis from various fronts are informative.
+ Codes are provided.

Weekness
- It would be very beneficial to observe whether this holds for ImageNet1K-level KD, as the batch-wise stats and running stats discrepancy does impose performance drop, and it would be beneficial to see if the method works for large-scale high-resolution datasets. This can significantly improve the quality of paper.
questions: - How will this method scale to deeper networks, say ResNet-50, that has bottleneck blocks instead of basic blocks. Will this cause a difference?
- How will the method scale to ImageNet level analysis?
- How are the related work baselines in Table 1 implemented? Providing more details here can be helpful.
limitations: In my opinion the paper addresses these fronts well.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The article proposes a method for distillation using Gaussian noise by modifying the distribution of BN layers. The article describes why the accuracy of the previous method is severely degraded and how the distribution can be adjusted to improve the distillation performance.
strengths_and_weaknesses: $\textbf{Strengths}$

1.It proposes a method for distillation using Gaussian noise by modifying the distribution of BN layers；
2. Significantly better than the experimental results of other methods.

$\textbf{Weaknesses}$
1. Fewer other methods of comparison in the experiments;
2. It should try other network structures;
3. Similar to the evaluation phase, if some Gaussian noise is first input to the teacher model before training, is it not necessary to modify the mean and standard deviation of the BN layer?
4. Some experimental details are missing and there are some minor errors in the article that need to be fixed.

questions: See the weaknesses.
limitations: See the weaknesses.
review_title: null

Meta-review: metareview: Reviewers agree that the results presented in this paper are quite solid.  While some reviewers would still like to see a wider range of experiments on different architectures, the consensus seems to be that this paper provides a solid proof of concept for a solution to a very difficult problem.
confidence: Certain
Decision: Accept