Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper highlights a major flaw with the commonly-used $ECE_\text{BIN}$ calibration metric, namely that it is biased for perfectly-calibrated models. Through a large number of empirical experiments (including through simulation), the authors show that a newly proposed metric, $ECE_\text{SWEEP}$ is able to produce less biased estimates of calibration error.

### Paper Strengths

1. The paper has many empirical experimental and simulation data to support the claim that $ECE_\text{SWEEP}$ is a useful calibration metric.

2. The paper points drawbacks of commonly-used calibration metrics such as $ECE_\text{BIN}$, namely that it is biased for perfectly-calibrated models, and suggests alternatives.

### Major Concerns

1. In the theoretical discussion, the paper assumes binary classification. However, the empirical experiments are conducted on multi-class classification datasets. I did not find any mention of how the binary classification discussion is generalized to the multi-class setting.

2. The discussion about how the simulations were conducted was difficult to follow. Are the $m$ simulated datasets essentially subsets of the full CIFAR-10/100 or ImageNet datasets? (This seems to be implied, but is never made explicit. Otherwise, it can read as if these datasets are synthetic images.) And how do you draw samples such that "the confidence score distribution matches the neural model's best fit Beta distribution and the true calibration curve matches the neural model's best fit GLM"?

3. The equation for $ECE_\text{SWEEP}$ seems misleading. Are you really trying to maximize the quantity in parentheses over all possible values of $b$? And if so, why does that mean that this maximization will result in the largest number of bins, under the monotonicity constraint?

4. Is it always possible to satisfy the monotonicity constraint for $ECE_\text{SWEEP}$? Some more theoretically-sound discussion of $ECE_\text{SWEEP}$ would be much appreciated.

5. The discussion on Page 3 about Figure 2 can be expanded upon. Why does the "optimal bin count grow with the sample size"? This is stated as fact without explanation.

6. In general, while the experiments are mostly convincing, it would be useful to have some theoretical notions for why/when $ECE_\text{SWEEP}$ is less biased than $ECE_\text{BIN}$.


### Minor Concerns

1. There are many typos, grammatical errors, and mis-referenced figures throughout the manuscript. Please correct them.

2. Running experiments on new datasets is understandably time-consuming. However, given that this paper is heavily dependent on empirical evidence, it would be helpful to see experiments on datasets beyond image classification.

### Original Rating

**Rating** - 5: Marginally below acceptance threshold

**Confidence** - 3: The reviewer is fairly confident that the evaluation is correct

### Post-Rebuttal Update

I applaud the authors for providing detailed responses to my (and other reviewers') questions and for updating the manuscript appropriately. Several follow-up thoughts to my original questions:

1. **Binary classification.** I think I understand what you mean now. Basically you are still performing multi-class classification, but then you treat the calibration problem as a binary: is the top-1 model output correct or not? I think this still can be made clearer in the manuscript.

2. **Simulation procedure**: Thank you for the clarifications.

3. **$\text{ECE}_\text{SWEEP}$ equation**: I am still concerned that the equation given for $\text{ECE}_\text{SWEEP}$ differs from Algorithm 1. (Thanks for moving the algorithm into the main manuscript!) The issue is that I don't think Algorithm 1 is taking the maximum over the quantity in the parentheses of the $\text{ECE}_\text{SWEEP}$ equation. Algorithm 1 is not actually computing $\max_b g(b)$ for some function $g(b)$. Instead, it is first finding the maximum $b$ that satisfies some criterion, then calculating $g(b)$ at that selected $b$.

  If the equation and the algorithm are the same, it would imply that increasing the number of bins will increase the estimated calibration error. However, this does not seem to be true. Consider a binary dataset that is perfectly balanced: $\bar{y_1} = \frac{1}{n} \sum_{i=1}^n y_i = 0.5$, and consider a constant model, i.e. $\forall x: f(x) = 0.6$. Then for the case $b=1$, $ECE = 0.1$. But for the case $b=2$, $ECE = 0$ (assuming equal-width binning).

4. **Monotonicity constraint**: Thanks for pointing out the trivial constraint satisfaction.

5. **"optimal bin count grows with the sample size"**: I understand the empirical and intuitive reasoning for why this should be true. However, I still wish that this notion could be made more formal.

6. **Theoretical notions for why/when $\text{ECE}_\text{SWEEP}$ is less biased than other estimators of calibration error**: I am now more convinced that this is difficult to show, and I understand that a lot of the literature is based on empirical evidence. However, given that the results are empirical, I still would like to see experiments are non-image datasets.

**Updated Rating** - 6: Marginally above acceptance threshold
review_title: Improved Calibration Metric Based on Empirical Evidence

final_score: rating: 7: Good paper, accept
review_content: Post Rebuttal:

Clearly I was too cavalier in suggesting alternative bin selection methods, as the authors have clarified. I think the paper is much improved by the comparison to other techniques such as ECE_debiased. The rebuttal also made me re-read with the view of seeing the demonstration of how bias varies with sample size as a major part of the contribution. In addition, the concrete demonstration of the relevance of the discussion via fig. 2 is nice.

On the whole, my opinion has shifted to be much more positive. I have been rating the paper somewhere between 6 and 7, and in such a case I would rather err on the side of acceptance. 

Two asides:
1. I only realised this now, but I had misentered my confidence - it should have been 3 and not 5! I have corrected this now.

2. Of course, actual constants matter in practice, but one way to select number of bins by considering errors in $\overline{y}_k$ is that the squared error in each of these scales as $B/n,$ which, when added over $B$ bins induces error at scale $B^2/n$. With the same logic as the maximum number of bins in ECE_sweep, this suggests one way to set $B$ to be as $B = \lfloor \sqrt{n} \rfloor,$ which controls this error term. Of course, this is only a heuristic, and the analysis is incomplete because it's not accounting for correlations between these errors and $(f(x_i) - \overline{y}_k).$
In a larger sense, I still think that just looking at $15$ bins because that's all that has been looked at in the literature is not quite the right thing - basically I think that when studying an autotuning method, such as ECE_sweep, it is important to illustrate how it performs with respect to simple strategies based on some basic heuristics. This was the reason I wanted to score the paper at $6$, but I didn't bring this up when the authors asked for references and so felt it would be unfair to penalise on this basis.

-----
Summary of claims:

The paper studies the estimation of the calibration error of models, which is a problem of increasing relevance due to the rich settings that modern ML systems are being employed in. In particular, the focus is on mitigating the _bias_ in the commonly used binned estimators. The authors make the case that the biases of the binned estimators vary with the number of bins and the number of samples in a nontrivial manner, which is detrimental to point estimation. As a form of autotuning the number of bins that should be used, the  authors use the fact that the true calibration curve should be monotone increasing to propose choosing the number such that the resulting estimates of the average values of the $y$s in the resulting bins are monotonic. The bias of this estimator is then demonstrated on a number of simulations for which the laws of the classifier outputs and the calibration curves are obtained by fitting to models trained on standard datasets.

----

Strengths:

The problem being studied is certainly relevant, and the work is well contexualised. I found the idea of using the monotonicity of the $\{\\overline{y}_k\}$ as a criterion to choose an appropriate binning scale to be clever. I also like the problem of testing if the true calibration error is large or not. Finally, the method of section 5.1 is a nice way of generating realistic laws and calibration curves, and adds meaningful depth to the simulations.

----

Weaknesses: 


1\) The nature of the contribution with respect to ECE_sweep is not clearly described in the text. Concretely, this amounts to a way to choose the number of bins using data (i.e., autotuning a hyperparameter in the estimate). While this, of course, leads to a different estimator, this is not something fundamentally different. I would much rather that the paper was upfront about the contribution. (In fact, I was pretty confused about the point the paper was making until I realised this).

2\) I don't think the baseline comparisons made in the experiments are appropriate. The proposal is a method to choose the appropriate number of bins in the estimate, and should be compared to other methods to do so instead of to an arbitrary choice of number of bins as is done in section 5.2. Without this comparison, I have no way to judge if this is a good autotuning method or not. Reasonable comparisons could be, e.g., choosing $b$ by cross validation, or, in equal mass binning, choosing $b$ so that each bin has a reasonable number of samples for the error $\overline{y}_k$ to not be too large.

3\) While the focus of the paper is on bias, it should be noted that by searching over many different bin sizes, the variance of  ECE_sweep may be inflated. If this is to such an extent that the gains in bias relative to other autotuning methods are washed out, then this estimator would not be good. To judge this requires at least that the variances for ECE_sweep are reported, but these are never mentioned in the main text.

4\) Choice of law in simulation in section 3, which are used to illustrate the dependence of bias on the number of bins, not aligned with the laws/curves in figure 3. Taking the latter as representative of the sort of laws and calibration curves that arise in practice, there are two issues:

4a\) The pdfs of $f$ tend to be a lot more peaked near the end than the one explored in section 3 - this is borne out by the values of $\alpha, \beta$ in the fits in Table 1. Beta(1.1,1) is remarkably flat compared to the curves in Fig 3.

4b\) There seem to be a few different qualitative properties of the calibration curves - monotone but with a large intercept at 0; those with an inflection point in the middle; and those with the bulk lying below the $y=x$ line. In particular, all of them tend to have at least some region above the $y=x$ line. The choice of curve $c^2$ in section 3 doesn't completely align with any of these cases, but even if we make the case that it aligns with the third type, this leaves two qualitative behaviours unexplored. 

In fact, the choice of laws is such that the error of the hard classifier that thresholds $f$ at $1/2$ is $26\\%$. I don't think we're usually interested in the calibration of a predictor as poor as this in practice.

All of this makes me question the relevance of this simulation. Is the dependence of the bias on the number of bins as strong for the estimated laws as it is for these? Seeing the the equivalents of figs 7 and 8 for the laws from section 5 would go a long way in sorting this out.

5\) Experiments: As I previously mentioned, I don't think the correct baselines are compared to. Instead of posing the method against other autotuning schemes, just one choice of the number of bins is taken. This already makes it near impossible to judge the efficacy of this method.

Despite this, even the data presented does not make a clear case for ECE_sweep. In Fig. 4 we see that the bias of EW_sweep is even worse than EW. This already means that the sweep estimate doesn't fix the issues of ECE_bin in all contexts. It _is_ the case that EM_sweep has better bias than EM, but again, for samples large enough for the variances to be in control, it seems like these numbers are both converging to the same, so I don't see any distinct advantage when it comes to estimation. (of course, this is moot because this isn't the right comparison anyway)

Also, Fig. 5 is flawed because it compares EW and EM_sweep. It should either compare EM and EM_sweep, or EW and EW_sweep, I don't see why EW and EM_sweep are directly comparable.


Minor issues:

a\) Algorithm (1) and the formula for ECE_sweep in section 4 don't compute the same thing. In algorithm (1), you find the largest $b$ such that the resulting $\{\overline{y}_k\}$ is a monotone sequence, and return the ECE_bin for this number of bins. In the formula, you maximise the ECE_bin for all b that yield a monotone $\{\overline{y}_k\}$. From the preceding text, I assumed that the quantity in Algorithm (1) is intended. 

b\) Why is the $L_p$ norm definition of the ECEs introduced at all? In the paper only $p = 2$ is used throughout. I feel like the $p$ just complicates things without adding much - even if you only present the $L_2$ definition, the fact that a generic $p$ can be used instead should be obvious to the audience.

c\) Design considerations for ECE_sweep - it is worth noting that accuracy is not all that we want in an estimate of calibration error. For instance, one might reasonably want to add this as a regulariser when training a model in order to obtain better calibrated solutions. One issue with ECE_sweep is that it introduces a problem in that how the number of bins in the ECE_sweep estimate changes with a small change in model parameters seems very difficult to handle, which makes this a nondifferentiable loss. Broader issues of this form, and a discussion of how they may be mitigated, could lead to a more well rounded paper.

----

Comments:

a\) Exact monotonicity in the ECE_sweep proposal - I find the argument stemming from the monotonicity of the true calibration curve, and the idea to use this to nail down a maximum binning size interesting. However, why should we demand exact monotonicity in the bin heights? Each $\\overline{y}_k$ will have noise at the scale of  roughly $\sqrt{b/n},$ (for equal mass binning with $b$ bins), and in my opinion, violation of monotonicity at this scale should not be penalised. Also, what if a few $\overline{y}_k$s decrease but most are increasing (i.e., the sequence has a few falling regions, but the bulk is increasing)? Perhaps instead of dealing with this crudely, the error of a shape constrained estimator may serve as a better proxy.

b\) Isn't the procedure for parametrically fitting the pdf of $f$, and $\\mathbb{E}[Y|f(X)],$ and then integrating the bias a completely different estimator for TCE of a model? In fact, if the laws are a good fit, as is claimed in section 5.1, then this plug in estimator might do well simply because the integration is exact. In fact, since the fit is parametric, this can further be automatically differentiated (if, say, $f$ were a DNN), and thus used to train.

c\) It would be interesting to see what number of bins are ultimately adopted in the ECE_sweep computations that are performed. 

----

Overall opinion: The lack of comparison to appropriate baselines makes it near impossible for me to judge the validity of the proposed estimator. I feel like this is a deep methodological flaw when it comes to evaluating the main proposal of the paper. This is a real pity because I quite like some of the ideas in the paper.

Due to the inability to evaluate the main contribution of the paper, i am rating it a strong reject. I'd be completely open to re-rating it if appropriate comparisons are performed, and the case for the method is properly made.
review_title: Interesting ideas but baselines in experiments are deeply flawed.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper introduces an estimation approach for the true calibration error (TCE). TCE is compared to the binning-based estimator of calibration error (ECE) to measure bias.  Also, a maximum estimator of calibration error ECE_sweep is introduced and compared against standard binning ECE_bin across image datasets in classification tasks. 

**Strengths**
-  With extensive sensitivity analysis, the paper illustrates the importance of  non-biased ${\rm ECE}$  calibration evaluation and model selection

**Weakness**

The reviewer appreciates the extensive sensitivity analysis. However, the contributions seem incremental in terms of methodology.

*Clarity*:
- The writing could be improved by the following:
1) Simulation steps for estimating bias in section 3  can be written in algorithm form
2) Computation or approximation of TCE described in section 5 is difficult to follow
3) Sections arranged in terms of concepts, e.g., TCE is introduced in section 2 and an approximation approach for TCE described in section 3 and section 5
4) The proposed ${\rm ECE}_{\rm sweep}$ estimator moved from the appendix to the main section
5) What is the motivation for the proposed ${\rm ECE}_{\rm sweep}$

*Strong technical assumptions*:
- Estimation of TCE relies on parametric formulations for confidence distribution and the true calibration curve;  such parametric assumptions are violated in practice
- The paper assumes monotonicity in the true calibration curve, which is ill-justified 
- Provided that the proposed solutions rely on TCE, which is unknown. To evaluate bias, the proposed solution requires the above assumptions to hold

*Additional experiments*:
- How does the proposed solution compare to non-binning based approaches?
- Sensitivity analysis of top-r estimates, where r is the rank?
 
review_title: Interesting sensitivity analysis but strong technical assumptions

final_score: rating: 4: Ok but not good enough - rejection
review_content: In the manuscript, "Mitigating bias in calibration error estimation", the authors examine the problem of estimating the 'calibration error' of a binary classifier; the problem context being that addressed by techniques such as Platt scaling.  Here the class of estimators considered are what I would call 'bin smoothers', namely function estimators constructed from piecewise constant sub-estimators.  That these estimators suffer biases in the finite sample regime---and indeed asymptotically if the number of bins is not allowed to increase with sample size---does not seem remarkable to me (even without reference to the extensive literature on this class of problem), and likewise is noted (though not numerically explored) by previous authors considering the calibration error topic (e.g. Nixon et al. 2019).  The question then is whether this bias can be shown to have meaningfully affected the conclusions of previous studies of calibration error corrections (in particular, overturning or questioning the results of studies that were themselves important in the field).  From the experimental section presented here it is not clear that this is the case.  On the other hand, if the value of the 'sweep' method is to be the focus then a comparison should be made against a selection of competitive methods for bin smoothing in contemporary use (e.g. moving windows, adaptive bandwidths) rather than simply the equal width and equal mass bins of fixed number.  
review_title: A fair point but hard to gauge the importance of the problem

Meta-review: The paper proposes an adjustment to the ECE metric to make it less biased in the small sample case by including the assumption that the confidence output by a classifier is monotonic with the true correctness probability.  The main idea is to successively make finer bins until a non-monotonicity is observed.  The paper is interesting, but the magnitude of the contribution would be just enogh for a short paper if such a track existing in ICLR.  

Reviewers have raised concerns about the discrepancy between their revised ECE formula and the Algorithm accompanying it, although that has been fixed through the author feedback phase.  Another concern is that for a paper whose core technical contribution is a revised metric for measuring calibration, a more thorogh empirical study over larger datasets is required.
Decision: Reject