Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents a number of empirical comparisons for the metrics on uncertainty estimation performance such as AUROC, ECE with 484 ImageNet classification models and has drawn some observations such as the superior properties of distillation over vanilla training, pre-training and adversarial learning, the superiority if ViT over other networks, and so on. 
main_review: Strengthes:
- This work has extensive numerical studies on the metrics of uncertainty estimation performance and has drawn some observations. 

Weaknesses:
- It is somewhat unclear to justify the usage of selective model for the evaluations of the uncertainty. The original work for selective model was using it when the task has the option of 'rejection' instead of performing 'prediction', but it is not clear why it is related to the measurement of uncertainty.
- There is no new method, no new metric to be proposed in this work. In other words, the contribution of this work may be purely empirical. 
- There are a number of concerns on the experiments in terms of fairness: 1) it is unclear what Figure 2 means (see the first point in the weakness). 2) There are so many factors to determine the performance of a network on ImageNet that were not considered in this work such as optimization related parameters and practices, random initialization leading to different performance, possible differences in loss functions used and so on. However, a number of observations seem to assume that these are similar among all 484 trained models.
- It is very hard to grasp the main theme of this work as a conference paper. Many observations were lined up, but what are the main contributions? It is not really surprising to see ViT performing better than EfficientNet-V2.

Comments:
- Some of the important contents are in the Appendices. Unfortunately, in this manuscript, missing some of them in the main text seems to make this work to be very hard to understand. I strongly recommend to revise the manuscript to be accessible to more readers by writing key ideas and contents in the main body.
- There are too many contents in the main body and it is very hard to grasp all of them. It will be great if the authors can write down clear contributions of this work in the beginning and some key conclusions / take-home messages of this work as remarks.
- In selective model, the definition of coverage does not contain f while the symbol contains it (phi(f, g)). In addition, how did the authors implement g in experiments? It seems that g is important to define in details. 
- It is clear how MC dropout measures the uncertainty, but it is not clear how selective model does. Please elaborate more on this.
- In Page 6, it says "While TS is usually beneficial, it could harm some models (see Figures 3 and 4)." How can we argue it since ECE is also measured empirically. In other words, the model could be ok while the measured ECE could contain some measurement errors.
- In Page 9, how can we conclude that "providing reliable uncertainty estimation"? In other words, how one can measure reliability of uncertainty estimation and how were they measured in this work. It is not easy to see the supporting data / experiments for this sentence.


summary_of_the_review: Some of the observations are interesting from extensive experiments, but there are a number of concerns on the metric to be used (e.g., selective model for uncertainty). There is no new method presented in this work, which makes this contribution somewhat weak for top conferences like ICLR (but could be an interesting workshop paper). The conclusions are not really surprising or are not easy to believe since there are other factors that were not considered in this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an empirical study and focuses on evaluation of uncertainty with respect to different model architectures and training schemes. Using metrics capturing calibration and accuracy, the authors conclude that distillation based schemes are efficient in uncertainty estimation and ViT performs the best uncertainty estimates among the architectures considered.
main_review: Positives:
- The paper focuses on uncertainty estimation in deep learning, which is an important research area that will enable risk aware and safe deployment of models.
- The paper conducts extensive experiments with 484 models to gather insights into training schemes and model architectures for capturing uncertainty.

Negatives:
- I overall disagree with the classification of uncertainty used in the paper. Uncertainty due to class-out-of-distribution should be grouped into epistemic uncertainty, which by definition describes the uncertainty stemming from lack of knowledge.
- Authors claim that ViT is a superior architecture for uncertainty quantification. I think this claim would have benefited by the distinction of aleatory and epistemic uncertainty. Aleatory uncertainty is related to the inherent randomness in data, and modeling choices do not affect it.
- Most of the analysis in the paper uses confidence score function \kappa to be the softmax values. This confidence score function will capture aleatory uncertainty but will not capture epistemic uncertainty. This distinction is important since there are evaluations performed with out of distribution data, which by definition stems from epistemic uncertainty. I believe the evaluations need to be consistent with these definitions.
- Distillation is observed to yield better uncertainty estimates. I think this result was not surprising given knowledge distillation's connections to ensembling, which is a state-of-art method for uncertainty quantification that captures epistemic uncertainty. I recommend adding a discussion on this point.
- Evaluations would benefit from reporting log likelihood numbers which is a common metric capturing the probabilistic distribution.

Minor comments:
- Typo in page 5. Use capital B in "both": "... tied values. both metrics are also closely .."
- Figure 4. For consistency, please report MC Dropout values here as well. 
- Figure 6. It is interesting to comment on the slope of the curves, indicating how quickly the performance degrades with respect to severity levels.
summary_of_the_review: Overall I recommend a rejection of the paper. I think the definitions of sources of uncertainty are important in terms of what we are able to quantify and those definitions need to align with the evaluations, and I believe there are some mismatches in this paper. I think this distinction will immensely help the empirical evaluations performed in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an evaluation of different models in their capacity to reflect epistemic and aleatoric uncertainty and reviews different methods for uncertainty performance measurement. Models are classifiers trained on ImageNet. The number of models reported is 484.  The analysis also considers the cases when the data is in-distribution and out-of-distribution. The conclusions lead to a group of models (trained with distillation), improving uncertainty estimation performance, and the vision transformers architecture having the best uncertainty estimation performance.   

main_review: The capability of models to reflect a calibrated uncertainty is interesting for different applications that require a reliable confidence indicator for the model predictions. However, some points regarding clarity on the document can be commented on. 
 
The paper makes many references to figures, and sections of the appendix making the discussion of the work feel split between both documents. The appendix should be complementary and the paper should be self-explanatory, providing only complementary information like demonstrations or additional results. In this regard, it is recommended to review the information and results included in the main paper, compared with the content of the appendix, and include the necessary introduction to the concepts and definition, making it less for readers necessary to jump into the appendix and back to the paper. 

More details about how the severity level can be included, even though the details are included in one section of the appendix (L), some introductory details can be added to the main paper. 

Similarly, indicate some details about the models used, categories, criteria for selection. 
summary_of_the_review: In general, the document presents an interesting review about uncertainty estimating performance related to the models, and training strategies, however, perhaps due to space limitations, the main paper can present a general description of the work and rely on the appendix to give details. This can make the paper complicated to follow since in some cases, it might be necessary to jump into the appendix before continuing with the lecture of the main document.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors make use of 484 models pre-trained on ImageNet to assess both in- and out-of-distribution uncertainties along a number of metrics.
They not only provide a critical evaluation of the different metrics and their task-dependent strengths and weaknesses, but also discover previously unknown empirical patterns in different architectures and techniques.
main_review: I really loved this paper.
My only real criticism is that I felt like the paper could use some extra editing and structuring to make sure it has the impact it deserves to.
The abstract felt slightly too long and didn't draw focused enough attention to the main contributions of the paper, dwelling too much on details.
The introduction dives into specific trade-offs for various metrics before the metrics have been properly introduced. I would recommend moving the second half of the third paragraph and the fourth  and fifth paragraphs to a later section *after* the metrics have been defined.
I'd be tempted to slightly adjust the structure to: introduce the metrics; discuss the relative merits of the metrics; in-dist performance; introduce the novel OOD evaluation (which is a great contribution); ood performance; concluding remarks.

I really like the example in paragraph 2. I've not seen this made so explicit before and I think it does a great job motivating the work.

I also really like this paper as an example of how to make use of libraries of pre-trained models but nevertheless do extremely thorough empirical work.

Do you think the architecture-dependent correlations between ECE and AUROC are 'real'? Would this extend to other datasets? It makes me wonder if there's something unreliable going on there. In general it might be good to comment more on the limitation of relying on ImageNet, and add some consideration of how this might differ for other datasets/data modalities.

`both metrics are also' should be capitalized.

Title of Appendix G should be `Effects', not `Affects'
summary_of_the_review: This is a great paper. I have some quibbles about structure, but the basic results are extensive, interesting, and important. I would like to see this paper highlighted at the conference and would be disappointed if it were rejected (unless another reviewer points out to me prior work that I am not aware of).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides an empirical comparison of uncertainty estimates obtained from 484 deep neural networks (DNN), trained for image classification tasks on the ImageNet dataset. They compared uncertainty estimation performance of different architectures and training strategies (knowledge distillation) on many quantitative metrics such as AUC-ROC on classification tasks, Area under the risk coverage curve (AURC), etc. The authors finally summarize their findings on what architectures are best at providing uncertainty estimations and how to compare different methods on OoD detection and uncertainty in in-distribution samples.

main_review: ### Strength

1. The authors provided a thorough comparison of different architectures and training strategies. 
2. The study compares 484 deep networks.
3. Their main strength is the observations on knowledge distillation and how providing extra information in any form either as pre-training or learning from unlabelled data helps in getting better uncertainty estimates.
4.  The authors compared different networks in OoD detection tasks of varying severity levels. 
5.  They provide a comprehensive experimental design that considers OoD as found in real-applications that are not necessarily very far from training distribution.
6.  The authors demonstrate the strengths of the temperature scale (TS) method in proving ECE and uncertainty estimates.
7.  The paper demonstrates the strength of ViT architecture in providing better uncertainty estimates and predictive performance. 


### Weakness
1.  A comparison with existing strategies of improving uncertainty estimates for a single-pass neural network is missing.
Many methods have been proposed which improves the uncertainty quantification for any network architecture. For examples

*  Mix-up learning: It is a training strategy to improve uncertainty estimates and model calibration [1].
* Strategies that impose a bi-Lipschitz constraint on feature extractors to learn a distance-aware feature representation for better OoD detection. A two-sided gradient penalty can be added to a network as an extra soft constraint [2]. 
*  Spectral normalization can be ​​combined with any architecture which has residual connections, resulting in better uncertainty estimates from that architecture [3].  
* Adding extra regularization to learn an ordinal ranking relationship between different samples for better selective prediction [4]. 
* Temperature scaling as explored by the study

2. Experiments in Figure 3 and 4 show an improvement in ECE and AUC-ROC. This improvement over vanilla training came at a cost, such as the cost of training on unlabelled data as in semi-supervised learning. 

3. Previous work, [4] has shown that Softmax entropy cannot capture epistemic uncertainty. Hence, the final output layer of the classifier, irrespective of the architecture, is usually not sufficient to capture epistemic uncertainty arising from OoD test samples.

4.  In section 4, the authors used the average confidence given by κ to identify OoD samples. Average confidence is a measure of softmax entropy and hence, is not sufficient to capture epistemic uncertainty.

5.  The figures 1, 5 and 7  are not much readable. 

6.  In Figure 2, it’s not clear how to choose the sample Sm, so that we get different coverage. For example, most of the models report results on 100% coverage over the test sets of benchmark datasets. Should we mix some OoD samples in Sm so that we expect the model to have less than 100% coverage?  Or given any Sm, we partially rank samples based on confidence score, and then use different thresholds of confidence score to obtain a specific coverage level?
In this case, what if the model makes overconfident predictions, and hence confidence score is not sufficient to obtain the desired coverage level?


### Actions

1.  I would like to see quantitative comparison among 
*  baseline models, 
*  baseline model + [extra constraints for improved uncertainty [1-4], 
*  Models with improved architecture
	
This quantitative comparison will help me understand what part of the improvement in uncertainty quantification can be achieved from a given baseline model (+ extra constraints), before adapting to a more complex architecture. 

2. I would like to see extended results in Fig. 3 and 4 to include quantitative comparison among 
* baseline models, 
* baseline model + [extra constraints for improved uncertainty [1-4]], 
* Baseline model with different KD methods.
	
This quantitative comparison will help me understand what part of the improvement in uncertainty quantification can be achieved from a given baseline model (+ extra constraints), before adding KD.

3. For OoD detection, I would like to see a comparison of 
* baseline models such as ResNet
* Models with advance architectures (such as ViT)
*Feature-density based models that use data likelihood estimates to identify OoD samples.
	
This quantitative comparison will help me understand if advanced architectures reduce the need for extra feature-density based head such as gaussian mixture model to capture epistemic uncertainty.

4. For results in Fig.3 and 4, the authors should add analysis to quantify the added cost of knowledge distillation vs gain in ECE/Accuracy.
Report comparison on metrics such as extra training/pre-processing time.

5. I would like to see a through discussion on the results, summarizing the key findings in the form of clear guidelines: 
* A guideline to follow when deciding for model architecture for improved uncertainty estimates.
* A guideline to consider when choosing different knowledge distillation (KD) methods for learning better classifiers. Do KD methods perform superior to baseline models [extra constraints for improved uncertainty]?

6. The authors should consider adding more text to explain the results in Figure 2.
7. The authors should consider adding a table to summarize the different architectures explored in the study. 
Provide additional details on how different models with the same baseline architectures but different sizes or losses or knowledge distillation methods are considered. This table will greatly improve the readability of the current draft.


[1] On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

[2] Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation
using a single deep deterministic neural network. In International Conference on Machine
Learning, 2020.

[3] Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neural Information Processing Systems 33, 2020.

[4] Confidence-Aware Learning for Deep Neural Networks

[5] Deterministic Neural Networks with Inductive Biases Capture Epistemic and Aleatoric Uncertainty



summary_of_the_review: Overall the paper provides a thorough comparison of different networks on their uncertainty quantification. The paper largely did not consider the vast literature on improving uncertainty quantification of any given architecture.
review_title: null

Meta-review: As an empirical paper, this paper studies uncertainty estimations with respect to various architectures and learning schemes. Three reviewers suggested acceptance based on the strength of the paper (fairly extensive experiments were conducted, and some new observations were discovered, such as the superiority of ViT). On the other hand, two reviewers proposed rejection due to lack of rigor in writing and lack of novelty. No consensus was reached through additional discussion. In particular, the reviewer's point that the experiment was not well controlled-different models were trained with different hyperparameters etc- seems quite important, and it weakens the significance of the contribution of the paper. 

All reviewers agreed that it is a potentially interesting and important paper. I encourage the authors to resubmit in the future after carefully addressing the reviewers' concerns.
Decision: Reject