Venue: UAI 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: This paper looks at distilling predictive distributions arising from uncertainty estimation methods to improve deployment time latency. The paper also provides an approach to generate ensembles from the student model for better uncertainty estimation. 
Q3 Main strengths: The paper is nicely written, and the important aspects of the modeling adequately described. The paper looks at an important problem of distilling uncertainty representations to reduce runtime latency, and consists of experiments on a range of open source models + datasets.
Q4 Main weakness: I am conflicted about the novelty of this particular paper, given that the underlying idea of distilling in a Dirichlet has been done before by the EnD2 paper (Malinin et al., 2020).

References

1. Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales. "Ensemble distribution distillation." ICLR (2020).
Q5 Detailed comments to the authors: - There's missing discussion around related prior work [1] which looks at the same problem by directly distilling uncertainty metrics into a student model. 

- Why is the computational cost same for S2D and the ensemble based methods? Shouldn't the cost be lower for S2D, as it only requires a single forward pass?

- Given the overlap with prior work by Malinin et al., 2020 [2], can the authors help contrast their contributions with the former? This is the  point when it comes to the novelty, and hence I believe it's important to address this. 


References

1. Vadera, Meet, Brian Jalaian, and Benjamin Marlin. "Generalized bayesian posterior expectation distillation for deep neural networks." UAI (2020).

2. Malinin, Andrey, Bruno Mlodozeniec, and Mark Gales. "Ensemble distribution distillation." ICLR (2020).
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: My score is borderline reject as I have concerns around the novelty of the paper. However, I am open to changing my score based on authors' responses and discussion with the other reviewers.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: The authors focus on model distillation from ensembles. 

Ensembles capture uncertainty whilst improving performance but come at a large computational cost. While model distillation from ensembles is typically successful, the authors note that the student network is trained to match the average predictive dist of the ensemble, however, this overlooks the diversity across sub-models in the ensemble. 

To address this the authors proposed self dist distillation.  
Q3 Main strengths: The authors note that whilst distillation of ensembles can be effective, as the student network is trained to match the average outputs of the ensemble, its likely that the student will not the predictive uncertainty encoded in the ensemble. 

To try and address this, the authors propose self-distribution distillation. The main idea appears to be to add a second network to predict the distribution over the teacher predictions. As such, the setup is related to model distillation but has some important differences, primarily that the student network is not intended to be used independently (as would typically be the case in model distillation) but in conjunction with an efficient teacher network. For this reason the authors propose to use stochastic regularization to generate multiple predictions efficiently in the forward pass (in section 4 the authors also consider using traditional ensembles if there are no computational constraints).  
Q4 Main weakness: Overall I found the paper to be well written and clearly communicated. 
Q5 Detailed comments to the authors: Main comments

A further exciting avenue the authors could consider is combining their work with the recently proposed Model Soups: https://arxiv.org/abs/2203.05482
This could be alternative approach to obtain efficient networks at runtime whilst retaining the benefits of ensembles.



Minor comments

1. Whilst ensembles do help with uncertainty estimation, they do typically improve performance overall as well and this can serve as an additional motivation for this work. 
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: The paper proposes an intuitive idea to enjoy benefits of ensembles (with regards to model uncertainty) whilst reducing the overall computational costs. Experiments appear comprehensive, including reported run times and parameter counts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: The paper proposes a new training approach, referred to as self-distribution distillation (S2D). S2D serves the purpose to obtain single model, that learns to estimate uncertainties by distilling information from a teacher model ensemble. Different distribution distillation techniques are investigated an additional ensembles of these student models are investigated, leading to a hierarchy over uncertainty predictions. 
Q3 Main strengths: * The paper is very well written and structured. In particular the introduction, motivation and background sections were enjoyable to read. The present state is clearly illustrated and remaining challenges are pointed out.
*  To the best of my knowledge, the experimental evaluation is sound. The contrasted methods are reasonable, but more importantly, the experiments go beyond a simple “ours is better” discussion of results and adequately present S2D as a general technique to fit into prior works (like MCD, MIMO, SWAG). The reporting of computational cost and inference time, in addition to accuracy and losses, are appreciated. 
* The appendix is comprehensive and provides many additional clarifications and experiments of interest. 
Q4 Main weakness: The main weakness of the paper could be a potentially limited generality. To say this upfront, I do not believe this to be a dealbreaker necessarily, especially given the nevertheless interesting experimental results, but the proposed technique is at the same time somewhat incremental. However, much of the discussion around utility, inference time, inexpensive training seems to be a direct result of the assumption that a fixed feature extractor (see e.g. figure  1 or 2) is sufficient and that stochasticity can reside largely in later/select layers in the deep hierarchy. In a sense, the “full” Bayesian view outlined in the background section is therefore not completely adopted, and a trade-off is conducted with respect to empirical feasibility.  This reminds me of how Monte-Carlo Dropout is often employed only in late stages in works that attempt to get uncertainty estimates in encoder-decoder architectures for semantic segmentation. 
Q5 Detailed comments to the authors: As noted beforehand, the benefits of the presented approach are presented sufficiently well (despite above main weaknesses), especially on top of existing techniques, so I do believe the contribution to be sufficient. A potential improvement could be to place more emphasize here and provide a deeper discussion. 

In addition to the main weakness outlined above, I have the following suggestions and believe that some of the statements in the manuscript are not very well supported and would require additional clarifications/explanations:
* I do not believe the argument on “Note, calibration error and negative log-like hood can easily be reduced post-training by temperature scaling predictions” is very convincing. Naturally, much of the challenge of OOD scenarios is that we do not know what is actually all in and out of distribution. Thus we cannot possibly always rely on post-training calibration given some explicit data. I may be missing something here and would encourage the authors to explain.  
* Gaussian noise: “This form was chosen due to simplicity of sampling and possible ensemble diversity” -> explain the latter, particularly since the choice is rather specific and would thus intuitively rather limit diversity? 
* “Simply generating multiple teacher prediction by propagating through the last layer will not limit the ability of the model” -> it is unclear why this is the case. Certainly a fully Bayesian view of distribution on parameters across all layers is more general? 
* “This choice of integrating ensemble teacher training and distribution distillation into a single entity utilizing parameter tying also serves as a regulariser” -> Intuitively yes, but also rather hand-wavy and not clear what the benefit/role of this statement is in. 
* “Additionally stochasticity in the teacher forces the outputs to have some diversity, mildly limiting overconfidence”. -> This sounds like a soft version of my main point on the limitation of a fixed deterministic encoder. Is this the limitation that is being highlighted here? In either way, a more detailed description would be necessary here, perhaps even a small section on limitations/assumptions.  
* As a minor suggestion, and not a point of criticism, if more space is required to accommodate some of the changes, I believe it would be very easy and sufficient to merge figures 1 and 2, given their high content overlap. 
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: The merits become sufficiently clear and the paper is easy enough to follow. Limitations and assumptions could perhaps be discussed in more detail, especially with respect to the detailed comments above on the presumption that a fixed feature extractor is adequate/sufficient.
review_title: null

Meta-review: Meta Review: This paper introduces an efficient single network, based on knowledge distillation from the ensemble method, to perform both prediction and uncertainty quantification. It receives three weak accept reviews; reviewers believe the addressed problem is important, the proposed method is technically novel, and the experiments are extensive, which show strong performance of their method.  The main concerns include incremental contribution due to it’s similarity to an existing work and the method’s limited generality.  During the rebuttal, authors alleviate some of these concerns.  
Decision: Accept (Poster)