Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about quality of confidence predictions made by a bunch of pretrained models on ImageNet 1K, in terms of selective prediction (measured by the accuracy vs confidence plot), ranking (measured by AUROC), and calibration (using ECE).

The authors claim to evaluate 523 pretrained models from PyTorch and timm in terms of the above mentioned metrics, and explore issues like ranking and selective prediction performance (where a classifier can abstrain from making a prediction if its uncertainty is too high). There are some important conclusions and insights that can be drawn from this study that are summarized below and in the strengths.

The contributions are:
- A large scale study of pretrained models on ImageNet 1K in terms of the quality of their predictive uncertainty.
- Conclusion that knowledge distillation improves uncertainty estimation best as compard with other training regimes.
- Conclusion that temperature scaling also improves AUROC, meaning better selective and ranking performance.
- Finding that some vision transformer architectures, particularly ViT, have better uncertainty estimation than other comparable architectures.
- Information about large scale correlations between AUROC, ECE, and model parameters like accuracy, number of parameters. There are good conclusions that not much can be inferred in terms of correlations from these metrics.

strength_and_weaknesses: Strengths
- The paper is very well written and presented, I have no complaints about writing or presentation except some details on minor issues.
- It makes sense to make a large scale analysis of pretrained models on ImageNet about their (uncalibrated) uncertainty quantification capabilities, specially considering the recent advances in the last years with vision transformers. While I have my reservations about metrics and models with uncertainty, at least this is not a bad idea to start with.
- I like the analysis of risk-coverage, I have not seen this before at this level of depth, with so many models, and I believe this could be a novelty in this paper. The conclusion here is that some vision transformers produce better ranking and selective prediction, as measured by the coverage-accuracy plot.
- I like the conclusion that some vision transformers have better quality of uncertainty, but I think this is also not new since there is previous research that shows vision transformers improve on out of distribution detection, which is also closely related to uncertainty estimation. For this you can view the paper "Exploring the Limits of Out-of-Distribution Detection" by Stanislav Fort at NeurIPS 2021. I do not think this completely invalidates the results but at least it is something to acknowledge.
- One of the major results of this paper is an evaluation of different training regimes, including adversarial training, pretraining on ImageNet21K, semi-supervised learning, distillation, temperature scaling, and MC-dropout, on different uncertainty quantification metrics. Distillation seems to have the largest impact in improving AUROC and ECE over their standard variations. I believe this is a novel finding as I think it is not known at this scale.
- Additionally, temperature scaling also has a good improvement on AUROC and ECE and since it is easy to implement, it is a worthy information for practitioners to try to improve basic uncertainty quantification of their models.
- A good conclusion is that some particular vision transformers architectures (like the original ViT) clearly outperform all other architectures in terms of accuracy vs coverage, which I also think it is a novel finding.
- Another good analysis is correlation between AUROC and ECE with other parameters, they could be positive, negative, or zero, so not many conclusions can be drawn from this. Note that I have some criticism in weaknesses about this, specially since correlation is linear and there could be non-linear effects between these metrics.

Weaknesses
- The major problem with this paper is that all the pretrained models evaluated in this paper have been trained only with empirical risk minimization (except the ones with alternate training methods), meaning they are not really tuned to produce high quality uncertainty. I recognize that the authors used some models that use monte carlo dropout, but this is a very basic method for uncertainty estimation, a much better comparison would have been using bayesian neural networks and similar methods aimed to produce high quality uncertainty (ensembles, flipout, etc).
- The argument that ECE does not always point to the best model is well known, the ECE can be zero in degenerate cases like predicting mean accuracy as confidence, the ECE is not a proper scoring rule, so I think this is argument is not something new and severely weakens the paper. There are other calibration concepts that could have been used instead like group or adversarial calibration, for this see https://arxiv.org/abs/2006.10288

~~- In Figures 4 and 5, there is a large variation in the number of samples for each category of comparison (adv training, semi supervised, ImageNet21K, etc), with temperature scaling having the largest number of samples by far, I think the paper should comment on this as the sample size has an effect on the quality of the comparison and the conclusions that follow it, ideally all methods should use similar samples. Here I also assume that each data point is one different model being used, the caption does not make this clear. And there seems to be no explanation why different number of models were used in each of these experiments, particularly for MC-dropout for example.~~

~~- In Section 3, subpoint 4, the authors mention correlation of AUROC/ECE with several metrics are zero. The issue is the claim that this contradicts previous smaller scale studies, to me it is not clear which studies, and which exact claims. The paper later cites [Guo et al. 2017], but this paper does not evaluate AUROC, as that paper is about calibration and uses ECE for most experiments, so it is not clear which study claims to have found other correlations between AUROC and model parameters. This needs to be strongly clarified.~~

~~- While I understand the idea of using the AUROC to gauge ranking of predicted probabilities, the paper makes references to AUROC for binary classification (this is the most common use), but the results are for multi-class classification, and the paper does not mention which formulation is used for this case, as it is not trivial to extend binary classification AUROC into multi-class AUROC, so I would expect that the authors describe this in some level of detail, currently it is completely missing.~~

Minor Issues
- For Reproducibility, I would expect a list or at least some summary or information on which ones are the 523 models used in the evaluation of this paper, the authors refer to versions of the timm library but from the documentation (at https://rwightman.github.io/pytorch-image-models/models/ ) it is not clear where the figure of 523 models comes from.
- In Figure 8, what is the "various" models plotted with circles? This is not clear from the caption and should be clarified.
clarity,_quality,_novelty_and_reproducibility: About clarity and quality, I believe this is high quality work and it is mostly clear clear, there are some issues in presentation of results (missing information that I mention in weaknesses and minor issues) that hinder the interpretability of results. There is a good chunk of missing information like an exact list of the models architectures that were tried for each experiment.

About novelty, the paper has some novel aspects, in particular findings about uncertainty (minus my comment about out of distribution detection) for vision transformers are novel, the analysis of risk/coverage also I believe is novel, but the largest novel contribution is the analysis of different training regimes, which has a strong conclusion that knowledge distillation improves uncertainty quantification considerably, and temperature scaling also has an important effect, both on ECE and AUROC. 
There are some claims that are not novel or well known knowledge from statistics, like that ECE is not a proper measure for model selection and there are ways to trick it.

Reproducibility of this paper is slightly weak, the authors use many pretrained models from a clear source, but some plots are not reproducible due to lack of information about which exact models were used to produce those plots, I document these issues above.

summary_of_the_review: I have mixed feelings about this paper, in one side there are some good contributions and conclusions, this is a large scale study which can have a major impact in future research, specially since it uses ImageNet and many models across different architectural patterns, but in the other side there are some known or dubious claims that weaken the paper, but the major issue is the lack of information and clarity about how the experiments were performed, particularly in some figures, and selection of models for each experiment. This paper has potential but needs some strong clarifications (these are documented in weaknesses and minor issues).

~~I believe that right now this paper is borderline, this could definitely change with author feedback. My major issue is clarification about claims, previous work, and definition of AUROC.~~

After the rebuttal, I can recommend this paper for acceptance if the clarifications made by the authors during rebuttal are implemented in the final version.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies the ***relationship*** between ***deep models*** and their corresponding ***selective prediction and uncertainty estimation performance***. Specifically, this work considers several uncertainty estimation metrics, including AUROC, ECE, AURC and coverage for selective accuracy constraint. Using 523 models, this work provides some observations. For example, distillation-based training regimes consistently yield better uncertainty estimations; a subset of ViT models that outperform any other models in terms of uncertainty estimation performance.
strength_and_weaknesses: **[Strength]**
- Studying uncertainty estimation of deep models is important
- This work considers commonly used uncertainty metrics. Also, diverse models with various architectures, training regimes, and learning schedules are included in the study.

**[Weaknesses]**

- The major question is: this work conduct analysis on ImageNet validation set, which is an in-distribution dataset. When testing models on ***out-of-distribution (OOD)*** datasets (e.g., ImageNet-R, ImageNet-S, and ObjectNet), the observations in this work ***might not hold***. In real-world applications, the dataset distribution typically changes. Thus, I would like to see the study on OOD datasets. Moreover, the models with the same accuracy can have significantly different performances on OOD datasets. Then, in addition to Figures 3 and 4, reporting the accuracy of models on OOD datasets would be more helpful to understand the effect of different methods. 

- Section 3 uses a condition that the models have the same classification accuracy. This seems reasonable. How about different architectures with the same accuracy? I would like to check the effect of architecture on uncertainty estimation accuracy. 

- Figure 6 (Comparing teacher models to their KD students ) is a little confusing. The student models have different accuracy from their teachers. In this case, it might not reasonable to compare them. Please clarify this.

- Please clarify on which dataset the temperature scaling was learned.

clarity,_quality,_novelty_and_reproducibility: - This work is well written. It clearly introduces the involved uncertainty estimation metrics. Moreover, the analysis seems interesting.

- The models are provided by TIMM and Pytorch, so this work is reproducible.
summary_of_the_review: **I am around the borderline**. 
- Overall, this work is well-written and easy to follow. The study on the relationship between deep models and uncertainty estimation performance is useful.
- While the analysis is interesting, I expect to see the observations on out-of-distribution datasets.
- Some observations need more clarification (e.g., Figure 6).

***------Post-Rebuttal------***

The replies did not answer initial concerns. I have been waiting for the required clarifications. If not offered, I would reject this paper and be willing to fight for my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an extensive empirical study on the selective prediction and uncertainty estimation. The experiments include a large number of pretrained models and several metrics. Although there is no novel algorithmic novelty, the results presented are very intersting.
strength_and_weaknesses: Strength:
- The paper is well-written
- The insights presented are very interesting 
- The experiments are extensive including several metrics and pretrained models
- The community can benefit from the presented results by investigating why some architectures have low uncertainty and this can open the door designing better methods.

Weaknesses:
- The main point is the lack of algorithmic novelty 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear.
Novelty: The presents very interesting empirical results but lacks algorithmic novelty 
Reproducibility: It is not clear if the code will be released
summary_of_the_review: Overall, the paper presents an interesting empirical study on selective classification and uncertainty estimation. Although the paper lacks algorithmic novelty, the results presented 
It could have been better if the authors analyzed one of the presented insights and tried to find intuition for it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work examines the uncertainty calibration of various image classifiers. A fairly comprehensive list of classification models, calibration techniques, and metrics are examined experimentally. From experimental results, this work draws several observations and recommendations in both model type and training techniques. 

strength_and_weaknesses: From my understanding, this work relies on trained models released by various previous works, which precludes the authors from controlling optimization parameters such as learning rate, regularization parameters, and training time; these hyperparameters may have significant impact on calibration due to their effects on overfitting. Furthermore, the analysis in this work is limited to imagenet (and 1 plot with cifar 10?), which may limit the transferability of the observations made in this paper. 

The way in which results are presented in the paper is also problematic at times. For example, the box plots in figure 4 and figure 5 did not include the number of pairs in each column, thereby limiting the reader’s ability to interpret the statistical significance of the results illustrated. In another example, while the authors repeatedly stated that they evaluated 523 models, not a figure, table, or graph in the supplementary materials explicitly lists what these 523 models consist of. For the sake of reproducibility, a machine-readable but human friendly list of models should accompany the supplementary materials (so that readers can see which models are used without prowling the html source). Since the authors have already grouped these models into tens of categories for plotting (different marker types in figure 1), tables summarizing the various models within each category would be very informative in the appendix. Furthermore, a distilled version of table 2 with the number of models within each category would be beneficial to understanding in the main text. The paper is also not fully self-contained; a review of related works that measure and benchmark uncertainty estimation and calibration is missing. This makes it difficult to discern which results in this paper are in accordance with current knowledge, and which results are surprising. Also missing are descriptions of the different methods applied to the models (e.g. adversarial training, KD, temperature scaling etc.). A concise and mathematical description of each method should at least be included in the appendix. The omission of this information diminishes the persuasiveness of the paper’s message. 

Nonetheless, this work presents a trove of quantitative results from which interesting conclusions can be drawn. The six conclusions drawn from these results are salient and actionable. As such, I would recommend for acceptance, as I believe that the publication of this work will benefit the collective knowledge of our field. 

One question I have is whether the observations from this work can transfer to any other dataset, using a more selective subset of models that had shown significant variation in performance on imagenet. 

Other minor notes:
There’s a kappa on page two that is not rendered correctly as the symbol. 
Figure placement and captioning can be significantly tightened (e.g. figure 4 and 5 has basically the same caption). 
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors present a (very thorough!) empirical study on the selective prediction and uncertainty estimation. Being an empirical study, the work does not present a novel technical/algorithmic solution. However, Multiple interesting insights and resulting actions can be deduced from the work. As such, I judge the work as quite interesting for the community. The reviewers mainly agree on these points -- thus, I recommend acceptance.
justification_for_why_not_higher_score: Due to the nature of the paper, no novel technical insight are presented. Thus, I feel that a poster presentation would be the perfect fit.
justification_for_why_not_lower_score: A rejection would not be suited for the paper since the work is well executed and provides interesting insights.
Decision: Accept: poster