Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides a systematic benchmark of recent deep classifiers in terms of the level of calibration.

On the calibration side, this paper focus on the definition of top-label (argmax, confidence) calibration. As a result, the main evaluation metrics are based on the top-label (confidence) ECE. The authors also include accuracy, the proper scoring rules BS and NLL and visualise the reliability diagrams. Temperature scaling is selected as the post-hoc approach.

Various model architectures and training strategies are applied on the imagenet datasets, with additional experiments on out-of-distribution tasks. 

Other than using many popular recent approaches, the authors report findings over the relationships on (1) model size and in-distribution calibration, (2) model size and out-of-distribution calibration, (3) accuracy and top-label calibration, (4) model size, training settings and calibration. 
main_review: As suggested by the summary above, this paper is in the position of benchmarking recent deep approaches in terms of calibration. As at the moment there are many ongoing works on the calibration of deep models, it is indeed interesting to have such a paper to provide a systematic comparison. In particular, the reasons for learning uncalibrated results still require further understand, so this paper should provide some empirical results that could be of good impact to other researchers.  

That being said, the quality of this paper, therefore, depends on the comprehensiveness of the benchmarks and empirical findings. 

On the comprehensiveness side, I am impressed by the selection of deep models and training strategies, and I agree this should bring some nice insights to the deep learning community. Regarding the dataset, while it is mainly designed with the imagenet (hence CV domain), it is understandable given the number of experiments required for such benchmarks. 

However, I am less impressed by the choice of calibration definition hence some of the associated metrics. As indicated in the paper, this paper mainly investigates the definition of top-label calibration. While I agree this is the most popular definition since Guo et al 2016, there are indeed some pitfalls regarding that definition. For instance, there will not be any uncalibrated top-label probability smaller than $\frac{1}{K}$ where $K$ is a number of classes. One can easily argue that you should not only focus on the biggest probability values as other small values also represent useful information on the label uncertainty.

This choice of definition further leads to an issue over the empirical findings. One of the contributions claimed by the authors is that the calibration and accuracy are correlated. I tend to think this only holds for the definition of top-label calibration, as accuracy is also a metric about the top-label. For multi-class calibration (df.1) or class-wise calibration, I don't think we can obtain the same results. Therefore, I would be more satisfied if the authors consider explicitly stating they are talking about top-label calibration in both the title and through the main texts. And it would be more than welcome if the authors can include experiments on the aforementioned definitions of calibration (multi-class, class-wise) as well, but it is understandable if the authors prefer not as these experiments are somehow expensive.

On the evaluation metric side, I am happy with the discussion on ECE and including metrics like BS and NLL. Although, there is also a trend of performing statistical tests over the hypothesis of calibration, and report the p-value. It would be nice to see at least some discussion over such methods (https://arxiv.org/abs/1910.11385).

Regarding the post-hoc calibration approaches, while I would like to see more methods, it can easily make the number of experiments grow rather fast(N-models * N-calibration * N-datasets). But one thing I would appreciate is to consider adding (https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf), which suggests they can improve the level of calibration during the training time.

In general, I think this paper has a good value to be seen by the ML community. I am currently giving a 6 mainly due to the point that the paper doesn't differentiate top-label calibration / general calibration in terms of the correlation on accuracy. I am happy to change the score if upon a discussion with the authors.

=============After author feedback============================

The authors addressed some of my concerns in the response, I am happy to increase the score to 7.

(1) In response to my comments on the selected definition of calibration (hence ECE), the authors add some experiments to further justify their empirical findings.

(2) The authors acknowledge some of the related work and indicate the benchmark should be accessible for others to add additional experiments.

At the moment I am not giving any higher score (e.g. 8) due to the fact that the overall paper still mainly reports empirical findings. But I agree it should provide a good reference for the community.
limitations_and_societal_impact: Limitation: The authors discuss the limitations of the measures of ECE. So it might be good to include a discussion on calibration tests as mentioned above. It would also be nice to further mention that post-hoc calibration is limited as it depends on a good uncalibrated model. That is, if a model simply outputs the marginal distribution of the target, this isn't much a calibration method can do.

Social impact: I agree with the authors, poorly calibrated models can have a negative impact if applied in society. For instance, to have calibrated probability on things like credit card approval is non-trivial.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper systematically evaluates the relation between model calibration and accuracy. The main finding is that the most recent models, notably those not using convolutions, are among the best calibrated. In addition, the analysis shows that model size and amount of pretraining do not explain these differences. Experimental evaluation is extensive and very carefully performed.
main_review: The paper focuses on an important and timely question, namely the relation between model architecture, calibration and accuracy. The empirical evaluation performed in the paper is great and very insightful. The hypothesis that convolutions lead to poorly calibrated models is very interesting. Unfortunately, it is not further analysed. Also the paper is mainly empirical, a more theoretical analysis of the problem / interpretation of the results is lacking.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper evaluates numerous recent models in terms of uncertainty calibration. The models evaluated are recent advances in relation to a previous seminal paper on the topic. The findings of this work indicate that previous trends leading to overconfidence with larger models may not apply to the current state-of-the-art models.  
main_review: # Pros

- Compares many SOTA models which have not been evaluated together in terms of uncertainty thus far. 
- Utilizes types of distributionally shifted data which are not considered in most previous calibration works.
- The experiments are thorough, informative, and show that newer families of large non-convolutional models are not overconfident.  
- The paper is clearly written and well designed. 
- The paper highlights relevant problems with Expected Calibration Error. The authors highlighted the positive bias on ECE of finite samples within bins. To my knowledge, this has not been discussed before.

# Cons

- As there are many different models evaluated, It would be beneficial to see some comparison of the total number of parameters across model families vs. calibration performance of each model. 
- Section 5 highlights problems with ECE, which is a crucial axis for all results in the main text and then only refers the reader to the appendix for other measures. The experiments in the appendix (Figure 16) should probably be moved to the main text to provide a measure besides only ECE.  
- In the appendix (Figure 16), there is no ImageNet-C dataset. I would like to see the results for NLL and Brier score on this dataset, as it is included in the other results of the main text. The performance at each corruption level may be informative, given that ECE could be biased. 
- I am confused by the verbiage in a sentence of Figure 16. How exactly were the 3rd and 5th rows of residual errors calculated? 
    - "we also provide the residuals of NLL and Brier score after regressing out
classification error (third and fifth row)"

# Addendum:

My final score derives from the following:

- In terms of originality, the setting is not novel, but the results and conclusions are because the authors seem to have shown that the commonly held belief of DNN overconfidence may not be true anymore for recent models.
- The paper is high quality, very clear and easy to read.
- In terms of significance, I think this paper contributes in both upending the common mantra of overconfidence, and I also think the results here are immediately applicable to practitioners. Some of the models trained here are quite heavy and expensive, so being able to learn from these experiments presents an opportunity to both achieve better calibration and avoid costly and time consuming experimentation.  
- The authors have adequately answered any concerns I raised.

limitations_and_societal_impact: Yes, they discussed the pitfalls of one of the main metrics used in the text. I am not aware of any potential negative societal impacts.
review_title: null

Meta-review: This paper studies the question of calibration in neural network image classifiers. Prior and widely cited empirical work in this area showed that state-of-the art models (at the time) could exhibit quite poor calibration properties. This paper empirically demonstrates that these same issues appear to be much less of a concern with recent, high-performing deep architectures including the MLP-Mixer and Vision Transformers in both the in-distribution setting and the out-of-distribution setting. The primary novelty in this work comes from re-visiting the question of calibration in light of recent advances in model architectures. The empirical results have strong practical significance as they indicate that past findings about calibration in convolutional models appear not to apply to recent deep architectures. The experiments conducted and metrics chosen are sound. Following the author response and discussion, the consensus is that the paper should be accepted. In revising the manuscript, the authors should include the additional results described during the discussion and take the reviewers' suggestions on the partitioning of material between the main body of the paper and the supplemental material into account. 
Decision: Accept (Poster)