Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the question of ensemble function diversity and how to preserve it on ensemble distillation to a single model. The authors use input perturbations that make the models, especially well trained models that are near 0 loss on the training set, disagree on the input, and then use knowledge distillation from individual members of the ensemble to individual subnetworks of the distilled model. The authors show that without this approach, the distilled models suffer from lack of function diversity, and that their approach enhances it and in accordance with their theory enhances performance as well.
main_review: I like the paper. It is well written, deals with an important question that is of interest to the community, and presents its approach in an easy to understand way. A small caveat is that despite following the subfield pretty closely I might not be able to judge the originality of the approach to the full extent and will likely defer to other reviewers if they have substantial points on this.

STRONG ASPECTS:
1) the paper is very clearly presented and was easy to read and understand
2) the problem the authors are looking at and addressing is important and broadly practically relevant
3) the technique is simple and could be implemented relatively easily (maybe apart from the gradient-informed input perturbation, which requires access to the models in the ensemble to be distilled)
4) I like the gradient matching perspective on why this technique could be working well

QUESTIONS AND CLARIFICATIONS:
1) what other approach to making sure that function diversity is transferred to the distilled model are there? did you compare your technique to them?
2) In Figure 2 and in the subsequent text it seems to be assumed that you are distilling an ensemble of models to a single model that still has a distinct number of subnetworks but is cheaper (for example the batch ensemble you are discussing). Could you approach work for actually distilling to a single model without this explicit multiple-subnetworks structure.
3) What is the advantage of using the gradient-informed perturbations in Equation 5 over the uniform perturbations? You mention that the random perturbations will be bad if the underlying data is low dimension,which I understand. This paper https://openreview.net/forum?id=XMoyS8zm6GA seems to suggest that they are actually very high dimension, which might make the random perturbations similarly good?
4) Could you perhaps use a MIMO-like architecture to be distilled into as well? I found this one by Havasi et al to be pretty high performance https://arxiv.org/abs/2010.06610 (only if you feel this might be a good idea, I know this is beyond the scope of your paper)
5) Aren't the perturbations in Eq. 5 pretty similar to adversarial perturbations, just with a target "label" being the random vector w?

WEAKER ASPECTS:
1) the experimental verification would be more convincing if you also looked at ImageNet. I know that this can be though, and I won't be decreasing my score based on it, but it could help a lot with the future adoption of your technique.
2) In Table 1 the models on CIFAR-100 reach only low 80% test accuracy. I think it would be good to show that your conclusions also work for more powerful SOTA models like the ViT that easily reach 90%+ test.

Overall, I like the paper!

UPDATE AFTER REBUTTAL:
I am very happy with the authors' detailed response and would like to increase my score from 6 to 7 to reflect that. Thank you very much and great job on the paper!


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper attempts to reduce the performance gap between an ensemble and a model that was distilled from it. The main idea is that, when distilling, the model should “absorb as much function diversity inside the ensemble as possible”. To do that, the paper proposes using ODS perturbation strategy, that samples diverse predictions for a given input form the ensemble, to use as label during distillation.

main_review: The ideas proposed are neat and well motivated by previous literature finding that prediction diversity is important for ensemble performance. I found particularly interesting the results that ODS sampling is approximately gradient matching, and that by doing gradient matching, the distilled model is encouraged to behave like the teachers on “nearby points”.

My main concern with the paper is that most of the experiments are performed on CIFAR-10 and CIFAR-100. I would like to see experiments on a more realistic benchmark, such as ImageNet, to assess the generality of the conclusions.

I would also be interested in seeing other ensembling methods tested, such as EMA.

Otherwise, I found the paper to be easy to read, well explained, and the analyses to be insightful.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to improve Knowledge Distillation (KD) for ensemble models. The authors apply ODS [Tashiro et.al, 2020] to each teacher model in the ensemble to obtain perturbed data. Thereafter, the perturbed data is used to train student models, each corresponding to one teacher model. The authors argue that ODS can improve the diversity among student models in the ensemble.
main_review: The main contribution is applying an existing technique, ODS [Tashiro et.al, 2020] to a non-trivial problem of ensemble model distillation. The abstract (line 12) promises to find input samples that teacher models' outputs disagree. Then I would expect one has to feed all teacher models' outputs into the process of seeking the perturbed inputs. But algorithm 1 treats each teacher model separately in the process. It seems to me that the algorithm 1 completely relies on the diversifying ability of ODS, and assumptions made in line 113-114. 

However, assumption in line 113-114 is quite counter-intuitive to me. Usually, a wise design of ensemble model is to include diversified member models. In this case, this means that the teacher models are very different from each other. If so, intuitively, a perturbation that maximumly change one teacher's decision does not necessarily change another's. For the similar reason, the assumption made on Jacobians (line 143) seems too strong, and most likely cannot hold in practice. This concern challenges the correctness of using one teacher model to derive perturbation vectors for all teacher models (line 110-111).

The paper has many clarity issues. Below is an incomplete list of them:

line 96-97: It is not very clear how ODS works. Which one is the perturbation vector? $w$ or $v_{ODS}$? I assume it is $v_{ODS}$, after checking the seminal work [Tashiro et.al, 2020]. Please improve the presentation, e.g., including equation (3) from the seminal paper.

Before Equation (9), please define $\epsilon$ as the perturbation in this specific context. Using multiple perturbation vectors, $v_{ODS}$ and $\epsilon$, may be hard for readers to understand.

Equation (10) is taken from equation (2) of [Srinivas and Fleuret, 2018]. It is therefore not this paper's contribution, and the authors should say that in the paper. Also, the notation is very informal. What does $\frac{J_s(x)}{S(x)}$ mean? a matrix divide by a vector?

Line 194-195: I cannot find a definition of "confidence value". Is it the predicted probability of the correct class label? Please define it first. Fig 3 seems to be very important, as it fulfills the promise made in abstract (line 12). However, the setup from line 192-198 is hard to read, which made it hard to judge the correctness and significance of fig.3.

Line 224: It is not clear what the ROC curve means. Also how is it obtained? I cannot find any description in the main paper or supplementary material.

Line 234-235, why higher SNR is more effective?

Overall, the paper does not make it crystal clear why diversity among teacher models are signified/promoted.

Through multiple rounds of Q&A, I think correctness is now justified, and I appreciate the authors' efforts in clarification. Nevertheless, the current draft needs to include all the key points during the discussion process. For example, the draft itself needs heavy revision to manifest how diversity among ensemble members is achieved. I now raise the score to 5, due to the fact that such a revision may require another round of reviewing.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on the topic of effectively distilling ensembles of models into a single model. Assuming that the distilled model should inherit the diversity of the ensemble, they attempt to make the distilled performance more comparable to the ensemble by using a (adversarial) perturbation strategy that directly identifies sources of diversity within the ensemble (i.e., where models within the ensemble disagree with each other). The authors demonstrate empirically that utilizing such a diversity-inducing objective within distillation improves the final performance of the distilled model in comparison to the ensemble on image classification experiments. They motivate this objective for distillation with intuitive descriptions of the importance of diversity within an ensemble, as well as by making connections to gradient matching between the student and teacher. 
main_review: I begin my review by emphasizing that I am completely open to authors' comments and feedback in regard to my review. My final score will be mostly based upon discussion with authors in regard to my questions and initial thoughts on the manuscript.

Pros:
- A reasonable explanation is provided for the gap in distilled network performance in comparison to the ensemble that is supported by evidence. I found this discussion to be intuitive and useful.
- This method reduces the significant overhead (memory and computation) of ensembles by only using a single model, but narrows the gap in performance between the model and the ensemble.
- Authors make a connection to gradient matching that is explained well and supported empirically. This is a nice interpretation/understanding of why the method works.
- The methodology seems to consistently improve the performance of distilled models in comparison to distillation with gaussian noise or normal distillation procedures.
- Paper is generally well-written, and it seems like the authors have spent time to polish it.

Cons:
- There is not much performance improvement on CIFAR10. It would be good to replace this with a larger dataset (or simply add a larger dataset).
- Only one method of input perturbation is attempted. It would be nice to have justification for why ODS is the best perturbation strategy and comparisons to other options in the experiments. I see that you show Gaussian perturbation does not induce diversity, but it seems that there should be numerous other perturbation strategies that would induce a similar result to ODS.
- No large-scale classification experiments (e.g., ImageNet). I believe running experiments on ImageNet for this work is important, as it is unclear from experiments whether your approach will scale to datasets with a significantly increased number of classes (e.g., 1000 instead of 100). This is especially true because a significant performance difference is seen between 10 and 100 class cases. It seems like ODS has been tested on ImageNet by Tashiro et. al., however. 
- The experiments focus on one knowledge distillation strategy (one-to-one batch ensemble distillation). Is it reasonable to test some other methods of distillation with a similar ODS approach?


Questions:
- Are adversarial perturbations the most appropriate kind to use within this methodology? I am not fully convinced that adversarial perturbations will capture diversity in the ensemble that has tangible, semantic meaning, but rather small/imperceptible changes that drastically modify network output. 
- Assumption that diversifying directions transfer between teachers is fully justified. (But, how do we choose which teacher to use? Does it matter?)

Small Comments:
- You probably need a section on adversarial attacks on neural networks in the related work, as the perturbation strategy for this work is inherited from an adversarial attack paper. 
- It would be nice to provide some clearer explanation of why you choose the exact distillation strategy that you use. For me, it is not immediately clear whether using batch ensemble and one-to-one distillation is a reasonable choice (i.e., is this widely considered the SOTA approach?)

limitations_and_societal_impact: I address limitations within my main review (authors also address limitations in the paper directly). There is no negative societal impact.
review_title: null

Meta-review: The authors propose a method for distilling an ensemble to a single model by training on inputs which are perturbed to make ensemble member outputs disagree. Most reviewers (1b8g, 7xWm, z2qY) seemed to agree that the method was sensible, elegant, and clearly-presented, and that the gradient-matching intuition was clever and well-supported. Reviewer p6SF raised significant concerns about correctness of the authors' transferability assumption, but after rebuttal and discussion these were mostly addressed. p6SF also raised concerns about clarity, but these too were partially addressed and not in my opinion sufficient to justify rejection. Another common concern (1b8g, 7xWm, z2qY) was the lack of large-scale experiments, but the authors address this with TinyImageNet experiments in rebuttal to the satisfaction of multiple reviewers (1b8g, z2qY). Therefore I recommend acceptance.
Decision: Accept (Poster)