Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper mainly concerns the quality of in-domain uncertainty for image classification. After exploring common standards for uncertainty quantification, the authors point out pitfalls of existing metrics by investigating different ensembling techniques and introduce a novel metric called deep ensemble equivalent (DEE) that essentially measures the number of independent models in an ensemble of DNNs. Based on the DEE score, a detailed evaluation of modern DNN ensembles is performed on CIFAR-10/100 and ImageNet datasets.

Strengths:
The paper is well written and easy to follow. The relationship to previous works is also well described. Overall, I think this is a good paper, which gives a detailed overview of existing metrics for accessing the quality in in-domain uncertainty estimation. The idea behind the proposed DEE score is nice and simple, clearly showing the quality of different ensembling methods (in Fig. 3). Given the importance of uncertainty analysis to deep learning, I believe this work will have a positive impact on the community.

Weaknesses:
- On page 6, the authors mention that the prior in Eq. 3 is taken to be a Gaussian N(\mu, diag(\sigma^2)) for Bayesian neural networks, however, many other choices of a prior distribution are available in the literature. What is the impact of changing prior distributions on the quality of uncertainty estimates in the case of variational inference? 
- Data augmentation is commonly used for improving model performance. However, I find the results presented in Sect 4.3 are not clear enough, note that for a given ensembling method in Table 1, the negative calibrated log-likelihood may increase or decrease when using different networks (VGG, ResNet, etc.). I think it would be interesting to elaborate a bit more on the influence of model complexity.
- On page 15, in Eq. 12, the choice of the variance parameter \sigma_p^2=1/(N*wd) seems unclear and should be better explained.

Minor comments:
The size of some figures appears too small, for example Fig. 4 and Fig. 5, which may hinder readability.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors response on 13th nov regarding the main concerns I have are valid. They make sense. I thank the authors for the detailed explanations. I went back and did another thorough read of the work. As it stands, I am OK to change my review to weak accept. 

---------------------------------

The authors evaluate a variety of ensemble models in terms of their ability to capture in-domain uncertainity. A set of metrics are used to perform these evaluations and in turn, using deep ensenble as a reference, the authors study the behaviour/capacity of the rest of the methods. 

Although the motivation for the work is sensible, there are several critical issues with the paper and the summary is not necessarily conclusive in terms of gaining any new insights. 
1. Much of the evaluations rely on the choice/nature of the optimal temperature, which would be different for different models. The authors suggest to use the model-specific optimal values when comparing instead of fixing the temperature? Why is this the case? Further, if we take this into account (i.e., allow for comparing different temperatures) then much of the differences between DEE and others cannot be directly interpreted. This is the case when using log-likelihood and Brier scores. 
2. AUC can be transformed into a normalized probability distribution (CDF), and hence in principle it is model/hyperparameters agnostic. This is one of the reasons i is used as information criterion in Bayesian model selection. Area of AUC is a valid metric as well. To that end, why do the authors suggest that it cannot be used as criteria for comparison across models? 
3. From section 3.5 it is not clear how test time cross validation is tackling temperature scaling? 
4. In section 4.1, the hypothesis on #independent trained networks is great and it makes sense? How is this translating ito the evaluations? None of the results actually talk about this aspect directly? Or am I missing something here?
5. Setting the evaluations with DEE as reference is problematic because we already know from random sampling theory that deep ensemble is better than the normal ensembles (tech results on random sampling for model fitting and RL etc. optimization results on mode finding with single mode vs. multi model methods also say similar things) and in fact that was the main motivation. Also normal regularization (like dropout or K-facL are more towards overfitting than ensembling) are not really an ensemble. Putting these together, most of the conclusions and the lots (figure 3 in particular) is by definition true. Nothing surprising. 


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper provide an extensive review of current advances in uncertainty estimation in neural networks with the analysis of drawbacks of currently used uncertainty metrics and comparison on scale the recent method to estimate uncertainty. The paper covers a lot of uncertainty metrics and a wide range of methods. The paper focuses on in-domain uncertainty estimation complementing the recent similar review on out-of-domain uncertainty estimation.

It seems that the paper provides the analysis missing in the current literature. Whereas as mentioned Yukun Ding in a public comment there is a related work on identifying issues with popular uncertainty metrics, the mentioned paper is missing the through comparison of the methods for estimating uncertainty. 

Such kind of thorough analysis (especially performed on scale on large datasets) and comparison is of obvious interest to the community as well as objective comparison of the current state-of-the-art. 

The paper is clearly written and easy to follow.

Based on this, I believe this is a strong technical paper and it should be accepted. However, the analysis in the paper is not overwhelmingly exhaustive. Some of the arguments on that are listed below.

Below is the list of comments/thoughts for potential improvement of the paper:
1.	“In this case, a model is expected to provide correct probability estimates:” – may be not the best choice of words, because for out of domain uncertainty estimation we still expect a model to provide correct probability estimates
2.	The first paragraph on page 3 seems to better fit in Section 2, for example, on the very beginning of Section 2.
3.	“Comparison of the log-likelihood should only be performed at the optimal temperature.” and others alike – personally, I do not support this kind of formatting for a scientific paper
4.	“can produce an arbitrary ranking of different methods. <…> Empirically,” – in the current form it seems that the first statement is somehow theoretically justified and then additionally it is confirmed empirically in this paper. I believe that the authors use empirical observation itself as the justification of the first statement, if that the case it should be reworded here. For example, “can produce an arbitrary ranking of different methods as we show below/ as we show empirically. We demonstrate that the overall …” If my belief is incorrect and there are other grounds that justify the first statement that it is required a reference after this statement.
5.	Italic and non-italic LL usage is unclear
6.	Having “Brier score” emphasised as a paragraph, it seems that there should be a paragraph log-likelihood as well
7.	“In that case, both objects and targets of induced binary classification problems remain fixed for all models” – do the authors consider in this case all out-of-domain objects as having a positive class and all in-domain objects as having a negative class? Because the models are still going to make individual misclassification mistakes
8.	Figure 2 – legend occupies too much space of the plot occluding almost a third part of the plot. Maybe taking the legend out of the plot to the right and squeezing the plot to make a room for the legend would be a better solution
9.	In eq. (4) and (5) subscript DE is not defined
10.	“SSE and cSGLD outperform all other techniques except deep ensembles” –cSGLD was not applied on ImageNet, therefore this statement is a bit misleading
11.	Colour of SWAG in Figure 3 is not very clear. Only excluding other colours I can determine which line is SWAG. Similar to cSGLD, is seems that SWAG was not applied on ImageNet. Why is that if that is the case? And it should be clearly stated at least in experimental setup in Supplementary  
For colours in general, lines in legends are very thin and it is difficult to assess their colour. I appreciate the authors compare a lot of methods and therefore have to use a lot of colours, but it is quite difficult to assess them even on screen not to mention if the paper is printed out. Could the authors please use thicker lines in legends at least?
12.	“Being more “local” methods” – without any context in the main paper this referral to “local” methods is unclear. Also it is good to add a reference to Appendix review of the considered methods in the main text.
13.	Missing details of what kind of augmentation is used in Section 4.3. Is it the same as training augmentation specified in Supplementary? It would require a reference to Supplementary
14.	“(Figure 1, Table REF)” – missing number for Table
15.	“Our experiments demonstrate that ensembles may be severely miscalibrated by default while still providing superior predictive performance after calibration.” – unclear which experiments demonstrate this and superior in comparison to what
16.	The issues of uncalibrated log-likelihood and TACE are clearly shown in the paper, whereas the issues with misclassification detection are only verbally discussed. An illustrative example, at least a toy thought example, could really improve the paper here
17.	The chosen main performance metric is not very convincingly motivated. It is clear why it is based on calibrated log-likelihood, but it is not very convincing why one cannot just used calibrated log-likelihood as a performance metric, why one should base the metric on deep ensembles instead. Also from the long-term perspective, if the community comes up with methods clearly outperforming deep ensembles, the metric would need to be based on one of these new methods 
18.	There is an indirect uncertainty metric that is not mentioned in the paper – uncertainty used in active learning (see, e.g., Hernández-Lobato and Adams, 2015. Probabilistic backpropagation for scalable learning of Bayesian neural networks)
19.	Figures 4 and 5 are too small
20.	“the original PyTorch implementations of SWA” – SWA is not considered in the paper
21.	“hidden inside an optimizer … The actual underlying optimization problem” – it seems that the ICLR audience should be familiar with “actual optimization problems” rather than using blindly the optimizer. It is always good to explicitly write down an equation that is used in a paper, but this wording seems a bit off for ICLR 
22.	“\hat{p}(y^∗_i = j | x_i, w) denotes the probability that a neural network with parameters w assigns to class j when evaluated on object x_i” – it should be \hat{p}(y_i = j | x_i, w), y^*_i is observed
23.	
a.	Why was dropout applied only for limited number of architectures and not applied on ImageNet at all?
b.	Why wasn’t cSGLD applied on ImageNet
24.	“On CIFAR-10/100 parameters from the original paper are reused” – it is better to repeat the reference here


Minor:
1.	Font size in eq. (10) should be the same as the rest of the paper
2.	“Or models achived top-1 error of”: “Or” - ?, “achived” -> achieved 
3.	“for a 45 epoch form a per-trained model”: “form” -> “from”?
review_title: Official Blind Review #2

Meta-review: The paper points out pitfalls of existing metrics for in-domain uncertainty quantification, and also studies different strategies for ensembling techniques.

The authors also satisfactorily addressed the reviewers' questions during the rebuttal phase. In the end, all the reviewers agreed that this is a valuable contribution and paper deserves to be accepted. 

Nice work!
Decision: Accept (Poster)