Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Thanks for the interesting paper!

Summary

The authors focus on the important problem of improved calibration measures as compared to the (now fairly standard) expected calibration error (ECE). More specifically, they define a new "Uncertainty Calibration Error" (UCE) metric based on the normalized entropy of the predictive distribution, rather than the max probability (as in ECE). The metric still uses fixed-width binning (as in ECE), and they motivate the interpretation w.r.t. perfect calibration based on a theoretical limit. They provide a set of experiments to show the differences in model ranking, sensitivity to number of bins, etc. between UCE, ECE, etc. on various models.

Strengths

- As noted in previous literature (and referenced in this paper), improved measures of calibration is an important research area.
- The authors provide a great background section to place their research within the broader area of uncertainty research.
- The experiments on sensitivity to the number of bins is informative and highly relevant in the context of previous literature (e.g., Nixon et al. (2019)) where it has been shown that ECE is particularly sensitive to this setting.

Weaknesses

As I have noted in more detail down below, I believe this paper suffers from a few weaknesses. Overall, at the end of the paper as a reader, I'm still left with questions of whether UCE is truly a better calibration metric. As noted above, the insensitivity to number of bins is great and an improvement on ECE and ACE. However, I don't believe the remaining experiments make a strong case that the metric (1) provides a better measure of calibration, (2) yields consistently improved model performance when used as a regularizer (though it's interesting that it can be used as one!), or (3) allows for improved model selection. Additionally, I believe its interpretability is limited. As noted below, the experiments have mixed results or make comparison claims that detract from the overall message, which I find troubling from an experimental rigor standpoint. Furthermore, I think this paper would benefit from experiments that are set up such that they can directly measure and compare the ability of the metrics to measure calibration error.

Recommendation

Given the above strengths and weaknesses, I'm currently inclined to suggest rejection of the paper in its current form. However, I think this could be a great paper and as a community I don't believe we have yet to devise the perfect calibration metric -- perhaps this could be it! I would highly recommend the authors push on the points above.

Additional comments

- p. 3, 4: It could be informative to includes notes about NLL and Brier score being strictly proper scoring rules (Gneiting & Raftery, 2007; Parmigiani & Inoue, 2009) that theoretically should be maximized only when the forecaster emits the distribution they believe to be true, and thus should, in theory, be well-calibrated asymptotically. However, we indeed know from Guo et al. (2017) that empirically, models can still overfit, leading to poor calibration.
- p. 4: The definition of perfect calibration can be traced back to Brier (1950), and, unlike ECE, is not limited to only the max predicted probability. Rather, for any predicted probability $p_k$ for class $k$, the probability that the true class is class $k$ should be equal to $p_k$ for all $p_k$ and all $k$. That is, $\mathbb{P}(Y = k | P_k = p_k) = p_k, \forall p_k \in [0, 1], \forall k \in [1, K]$.
- p. 5: UCE is based on an argument that normalized entropy approaches the top-1 error in the limit of number of class going to infinity. While this is interesting theoretically, this assumption seems too strong for empirical settings, and I think this affects the interpretability of the metric as claimed in the conclusion.
- p. 6, 7, Section 5.1: This section (and Figures 4 & 5 in the appendix) make claims that NLL and Brier score "fail at comparing calibration of models with different accuracy, as the metrics are always lower for models with better accuracy". I find this argument both surprising and confusing in terms of motivation. As strictly proper scoring rules, they should indeed have lower values for better probabilistic models. Although accuracy is a non-proper scoring rule, it should still correlate well with those strictly proper rules, so it is expected that the better models with lower NLL / Brier score will (typically with some variance) have higher accuracy (variance being due to the non-proper nature of accuracy). All strictly proper scoring rules can be decomposed into calibration and refinement terms (Parmigiani & Inoue, 2009; DeGroot & Fienberg, 1983), but in the non-decomposed setting, it is not expected that these rules would directly measure calibration. Therefore, given the focus on calibration measures, I'm confused as to the motivation behind comparing to NLL & Brier score directly (beyond the overconfidence analysis from Guo et al. (2017)) as a means of motivating the usefulness of UCE.
- p. 8, 12: In the regularization experiment, different regularization approaches are being compared in terms of calibration, but it's difficult to assess the results. In Table 2 (which needs an additional entry for the non-regularized result from Table 1), UCE regularization appears to improve accuracy, NLL, and Brier score over the non-regularized baseline. Interestingly though, NLL, Brier score, ECE, ACE, UCE, and MMCE (i.e, all metrics other than accuracy) point towards entropy-regularization being superior. It does result in a lower accuracy than UCE reg and the baseline, but by the other metrics (including the strictly proper scoring rules NLL and Brier score), it produces a better probabilistic model. For CIFAR-10 UCE reg is worse than the other regularization methods and the baseline.
- p. 8: Rejection & OOD Detection: This has been studied previously for unnormalized entropy, which should yield the same results. See, e.g., Malinin & Gales (2018), Ren et al. (2019).

Minor

- p. 3: s/as non Bayesian/as a non-Bayesian/
- p. 7: Figure 1 is too small.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update: After reading the other reviews and responses, and in light of the authors' updates to the paper, I have increased my score to a 6.

This paper proposes a new metric for uncertainty calibration, based on comparing the entropy of the marginal class probabilities conditioned on predicted class with the entropy of the predicted probabilities. The metric avoids the failure mode of ECE, where predicting the relative frequencies of classes results in perfect calibration, and can be used as a regularizer in a loss function. The paper demonstrates that regularization with UCE yields better-calibrated uncertainty on CIFAR predictions without sacrificing accuracy.

The paper is well-written and well-motivated. I’m uncertain as to its novelty. In particular, entropy as a basis for uncertainty estimation is well-explored (and was used as a baseline in (Lakshminarayanan et al., 2017) as well as Jie Ren et al., “Likelihood ratios for out-of-distribution detection,” NEURIPS 2017). It’s unclear what the results in Figure 3 contribute in light of these baselines (besides the normalization by the constant C).

Which loss function was used to produce the results in Table 1? If it’s the loss in (25), it would also be useful to see calibration metrics for NLL loss alone.

Figure 2 shows strong sensitivity of ACE to the number of bins. Quantile ECE (an ECE metric with bins defined by quantiles instead of fixed-width) often shows less sensitivity -- was this metric considered as well?

review_title: Well-motivated metric for uncertainty calibration; novelty is unclear

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a new calibration error measurement named UCE (Uncertainty Calibration Error) for deep classification models. It consists in doing a calibration in order to achieve "perfect calibration" (i.e., the uncertainty provided is equivalent to the classification error at all levels in [0, 1]), relying on normalized entropy for multiclass classification. This UCE is well justified for classification problems with several classes to process, where the entropy is demonstrated to be asymptotically equivalent to the classification (top-1) error. A point with this UCE metric is that is has some interpretability properties in terms of its value, and is said to be robust to the number of bins used.

The proposed metric is well explained, and justified, although I am wondering how well stands the assumption that the normalized entropy approaches the top-1 error for reasonable number of classes (e.g. C=10, as with CIFAR-10, or C=100, as with CIFAR-100). The properties presented are interesting.

If found the experiments to be well aligned to evaluate the approach, although limited in terms of dataset used (only CIFAR-10 and CIFAR-100), a greater variety of datasets would be more convincing in the of overall good performances of the approach, especially if datasets with a varied number of classes can be tested. Moreover, looking at the results in detail (Table 1), UCE does not appear to be particularly strong, having a worse calibration than ECE and ACE on CIFAR-10, but slightly better on CIFAR-100, assuming that we want it to be increased to reach the real error rate obtained. Moreover, the presentation of the results in Table 1 is messy: it gets difficult to match the calibration error with the accuracy, providing the classification error instead of accuracy would help to make a direct comparison with calibration error. Moreover, why the last two columns in Table 1 (Brier and NLL) are provided as floating-point values instead of percentages as with the other columns. That's unnecessary confusion that should be fixed.

Overall, I found the paper to be correct, relatively well written. I think that more room should have been given to experimentations, like with other datasets and with more space for OoD rejection and detection. Conversely, I am not sure of the relevance of providing all the detailed information on Bayesian methods in the second part of Sec. 2. It can be presented in a more concise way, as it uses a lot of space to explain well-known approaches.

In terms of potential impact of that paper, I still need to be convinced. What can tell me that this is just not yet another calibration metric. I think that the paper can have been made stronger on that aspect.

review_title: Good proposal but an enhanced set of experiments are required

final_score: rating: 5: Marginally below acceptance threshold
review_content: The work addresses an important problem in the study of uncertainty estimation: how does one compare model uncertainty at differing accuracy levels? The work proposes a novel uncertainty metric, relates this to existing methods and provides robust evaluation of the various merits of this approach. The paper is easy to follow. 

I have the following concerns with the work:
1. Regarding the use of UCE as a regularizer: how is the described behaviour of UCE different from the NLL loss? The NLL loss should penalize highly confident incorrect predictions and strives for confident predictions in high accuracy. What does the UCE regularizer add here?  Can table 2 include a non-regularized baseline as well to study this? In appendix A.3. it is said that UCE performs on par without regularization; then what is the point of proposing UCE as a regularizer?
2. What is the point of proposing the use of the normalized entropy as a thresholding factor for OOD detection? It seems that vanilla entropy would behave exactly the same. Is this considered to be a novel contribution in this work? 
3. Why is figure 2 (right) completely flat for UCE? Are there values not shown here where calibration error does change? Perhaps this should be included in the plot. 
4. Am I correct to say that max-p-based metrics might be preferable in very large-class problems such as language models? The paper does not discuss the computational tradeoffs of the method, and I believe this should be included. 
5. It appears that the experiment section does not provide much evidence  that this metric is favourable in selecting the best model for a downstream task where uncertainty is needed. This could be evaluated by e.g. an active learning problem. I believe it makes sense to include such an experiment. Right now, Table 1 and the accompanying discussion does not convince me that UCE is somehow more beneficial.

Overall, the work has merit and of interest to the community. However, the proposal of the use of the metric as a regulariser and a OOD scoring function seems unproductive and if so, distracts from the core contribution. This core contribution is understudied in the work. The work would benefit from more analysis into the computational tradeoffs, and evaluation of the signal that the proposed metric provides on model selection for downstream uncertainty tasks.

Nitpick:
- Table 1 could benefit from a vertical bar between the two datasets to clarify that the numbers are not comparable. 

Update:
Although the paper has improved, I still vote for rejection. The new insight of binary-classwise v/s multiclass UCE as a regularizer seems poorly explored in the paper and would benefit from closer study. This appears to be the basis of the improved results in table 1.
review_title: Reviewer3

Meta-review: This work proposes a novel metric for measuring calibration error in classification models.

Pros:
* Novel calibration metric addressing limitations of previously used metrics such as ECE

Cons:
* Limited experimental validation on CIFAR-10/CIFAR-100 only
* Unclear impact beyond proposing a new calibration metric
* Unclear value of using the proposed UCE metric for regularization and OOD detection

All reviewers appreciate the aim of the work to produce a calibration metric that addresses shortcomings of commonly used existing metrics such as expected calibration error (ECE), which is known to be sensitive to discretization choices.  However, all reviewers remain in doubt whether the proposed metric (uncertainty calibration error, UCE) is truly a better metric of calibration than previous proposals.  This doubt comes from two sources: 1. limited experiments that do not convincingly show the usefulness of UCE; and 2. interpretability of UCE not being as intuitive to the reviewers.  The experiments also use UCE as regularizer but the benefit of doing so over simple entropy regularization is not clear.

Overall the work is well-motivated and written and the proposed UCE measure is interesting.  However, the reviewers remain unconvinced of the claimed benefits and the potential impact for measuring or improving calibration.
Decision: Reject