Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces several distillation-based uncertainty estimation methods with distribution-based outputs and losses for the student network: (1) distilling from an ensemble of point-estimation/regular sub-network to a distribution-based Dirichlet sub-network (S2D); (2) distilling from an ensemble of Dirichlet networks to a single network (H2D) that either predicts a Dirichlet distribution (H2D-Dir) or a Gaussian distribution over multiple Dirichlet parameters (H2D-Gauss). Experimental results show that the proposed methods outperform conventional MC and deep ensemble methods on several tasks.  
main_review: Strengths
- The paper is pretty solid in its technical idea contributions. It explores the (likely previously unexplored) space of modeling uncertainty with distillation/self-distillation very thoroughly, although using Dirichlet-based output for modeling uncertainty is not new.
- The writing is generally good. The good story flow in the paper makes it easy to read and follow.
- The proposed methods achieve good results compared to the baselines.

Weaknesses
- The paper mentions that ensembles suffer from high computational requirements and they propose S2D as an alternative to that. However, S2D Individual still has some pretty large performance gaps to Deep Ensemble, especially in Table 1 and 2. The methods that outperform Deep Ensemble still need heavy computations.
- The paper only includes EnD, traditional MC, and Deep Ensembles as the methods for experimental comparison. There are many more existing works that do uncertainty estimation. One example is `Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. CVPR 2019.` that distills confidence/uncertainty knowledge to a student that does single-shot confidence estimation.
- En2D is introduced as an existing method that can distill the distribution knowledge to the student in Sec. 2.2 but it is not included in experimental comparison. Is EnD actually En2D?
- The tables do not show the computational (memory, inference time) requirements of each method/row. This makes it hard to make a good comparison between the methods.
- What are the differences between "Best in category" and "Best overall"?
- In the Introduction section, there is this statement "inputs it is expected that the trained model parameters can return reliable predictions." that does not make any sense.
summary_of_the_review: This paper introduces some nice ideas for uncertainty estimation with distillation, but it fails to make comparison to many more related existing works and it has some clarity issues. It is hard to know how it stands when it sufficiently considers the existing methods for comparison.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an ensemble distillation approach, in which one network is used as feature extractor and "two heads" (two networks representing teacher and student) are added to the network for self-distillation. The multiple teacher predictions can be generated through by adding multiplicative Gaussian noise. The distillation approach follows Malinin et al. [1] who proposes to model the predictive distribution with a Dirichlet and predicts the parameter of the Dirichlet distribution (instead of predicting the Categorial distribution). This approach can be both used for model distillation and ensemble distillation. The authors evaluated their models on CIFAR-100, LSUN, SVHN w.r.t. classification performance, calibration and out-of-distribution detection.
main_review: *Strengths*

* It is a simple model that requires only one network and two "heads" (linear layers at the end of the network) for ensemble distillation. 
* The method is able to match the deep ensemble's performance for Cifar-100 (classification, calibration), and LSUN and SVHN (OOD deteection).

*Weaknesses*

* Clarity: The notion could be improved and does not contribute to understanding the work. E.g., what's $M$, $m$, $\alpha$, $\delta$?,  There are several typos and missing words (e.g., "viewed from a Bayesian.", "over categoricals:", "There are a wide range of"). Please proofread the paper and fix these.
* Significance: The distillation itself is very similar to [1] and the authors could make a better job in detailing the difference of it. The difference is about using the self-distribution distillation, however, the authors fail to compare with existing methods to compare whether their methods outperform other distillation or ensembling methods.
* Insufficient evaluation: There are NO (!) comparisons to other distillation approaches or state-of-the-art (efficient) ensembling methods. Among others, works like [1, 2, 3, 4] can be compared to. Please look into these works for more models that can be compared to. Further, it is also good to not just compare the performance of the distilled model but also compare its efficiency (e.g., number of parameters, FLOPs).

*References*

[1] Malinin, A., Mlodozeniec, B. and Gales, M., 2019. Ensemble distribution distillation. arXiv preprint arXiv:1905.00076.

[2] Wenzel, F., Snoek, J., Tran, D. and Jenatton, R., 2020. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006.13570.

[3] Havasi, M., Jenatton, R., Fort, S., Liu, J.Z., Snoek, J., Lakshminarayanan, B., Dai, A.M. and Tran, D., 2020. Training independent subnetworks for robust prediction. arXiv preprint arXiv:2010.06610.
summary_of_the_review: I do think this is an interesting approach of using just one model for ensembling and distillation. However, at this stage, I found the paper lacks in clarity, significance and thorough evaluation (see weaknesses above for details). Thus I am currently recommending a weak reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a way to construct and train neural networks which represent uncertainty which is efficient with respect to compute and memory at execution-time.
They propose learning a shared feature representation with an ensemble-head which is distilled into a single head during training, with the ensemble-head thrown away for deployment.
main_review: Although this paper is a creative attack on a significant problem, I cannot recommend acceptance at this time.
My main reservations are to do with how the paper situates itself with respect to prior work and the theoretical representation of the ideas.

The prior work section could be improved.
1) Ensemble methods, variational inference, and SWAG are distinct methods. Ensemble methods (e.g., Lakshminarayanan et al. 2017) are as you describe. Monte Carlo dropout (MCDO) is a variational inference (VI) method, which makes a variational approximation. The predictive distribution of an MCDO variational approximation can be efficiently computed with a Monte Carlo expectation, which has some resemblances to an ensemble, but has a different principled justification. SWAG is an ammortized stochastic gradient Markov-chain Monte Carlo method, which is neither an ensemble nor a variational method.
2) You write that MCDO "generally does not perform as well" as deep ensembles, which should be supported by citation (and is a complicated claim which depends a lot on the resource constraints and problem definition).
3) Generally, the claim that different measures of uncertainty are grounded in different measures is best supported by some evaluation of where the noise comes from and what it might have to do with the prior which was chosen.
4) Minor points: Jordan et al. 1999 Introduction to Variational Inference is a better citation for variational inference; the Gal, 2016 thesis which you cite for uncertainty decomposition is less appropriate than Kendall and Gal 2017 "What Uncertainties Do We Need for Bayesian Deep Learning for Computer Vision"; the Amodei et al. 2016 paper you cite isn't really about fully autonomous driving and you'd be better off citing a paper that is in the opening paragraph

If I understand correctly, the main difference between your proposal and Ensemble Distribution Distillation is the choice to tie all parameters before the last layer?
You mention that this may have a beneficial regularizing effect.
Are you able to construct experiments that show that this is the source of the benefit?
Are there other elements of your contribution that might extend here also?

I would like to see more of a theoretical investigation of what kinds of uncertainty are actually represented by the estimates.
For example, it seems like the network is not able to express any uncertainty that corresponds to the fixed feature representation.

Reference results for DenseNet-BC are ~82% accuracy on CIFAR-10, which is significantly greater than any of the results you report.
This makes me wonder if there is a problem with tuning the baselines, which might make the rest of the results less reliable?
Do you know what might be going on there?
summary_of_the_review: I am not able to recommend acceptance. I think the paper could use a more careful grounding in prior work which investigates the contributions and their theoretical motivations more carefully. I also have questions about the implementation of the baselines and whether there might be a reason the performance of baselines falls below those reported for the architecture.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper contributes to neural network classifier training and uncertainty prediction.  It proposes a self-distribution distillation method that can train a single model to estimate uncertainties in an integrated training phase.  Also, it is flexible to be extended to build ensembles of models in the training phase and efficiently deployed in the test phase. Experiments are done on both classification and out-of-distribution detection tasks to show effectiveness.
main_review: Strengths:

This paper is clearly presented.

The proposed method is easy to be integrated into other methods and Experiments are done on different tasks to validate effectiveness.

weakness:

my only concern is the novelty of this method. S2D combines parameter sharing, stochastic regularization and distribution distillation.  These components exist in the literature. For example, parameter sharing is popular in online knowledge distillation methods like [1]; Also, in experiments, the authors did not compare with other distillation methods and did not show the model size and flops comparison. 
[1] Online Knowledge Distillation via Collaborative Learning
summary_of_the_review: This paper proposed an efficient uncertainty prediction. Experiments are done on two tasks to do evaluations. The paper is well written. However, since the proposed method is like a combination of existing techniques. The main concern is the novelty of the proposed method and whether it is fairly compared with SOTA.
review_title: null

Meta-review: This article presents novel distillation-based methods for neural network training and uncertainty estimation. While the idea is interesting, there is a general agreement amongst reviewers that the paper lacks clarity, adequate discussion of the relevant literature  and comparisons to existing work. Although the revision uploaded by the authors goes in the right direction by adding some experiments and clarifying some of the issues raised by the reviewers, further work is needed to make the submission stronger.
Decision: Reject