Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a framework of proper calibration errors proposing an upper bound as a reliable estimator for recalibration. The manuscript provides a comprehensive overview of calibration error and post-hoc calibration literature. This work also shows the popularly used calibration estimators are highly sensitive to the test-set size, while the proposed upper bound is robust to changes in data size.
strengths_and_weaknesses: Strengths:
* A strong related work section with comprehensive literature review on calibration error and model recalibration.
* The experiments show the proposed upper bound is stable and not sensitive to test dataset size.

Weaknesses:
* There is no theoretical justification why the existing calibration estimators are sensitive to test-set data size if they are i.i.d. and drawm uniformly in random from test data distribution.
* Given the Brier score is in range (-\infty, \infty), it is not clear how can square root of negative number (not real number) can be used as proper calibration error.  Or is there any approximation or rescaling performed to the upper bound?
* The term “root calibration upper bound (RBS)” for the proposed method is introduced in the Experiments section but could not find it in Section 4 where proper calibration errors are discussed. I suggest the paper introduce this term and its properties in Section 4.
* Reliability diagram and miscalibration area can be a better evaluation method to compare different calibration estimators to align with the actual definition of calibration. 

questions: * In Figure 1 and Figure 3, is the validation set size that is used to recalibrate the model is constant irrespective of test-test size? Is the test-set i.i.d and uniformly drawn from test data distribution for evaluation in sensitivity experiments (Fig 3)?
* Can the authors explain the basis for how models with perfect calibration, mediocre calibration and bad calibration defined in ECE estimator simulation in Figure 2?
*  What is the ResNet-Wide-32 model’s accuracy on CIFAR100 used for evaluation of calibration estimator sensitivity to test-set size in Figure 3? I suggest accuracy with different test-set size is plotted in Fig 3 to compare how well the calibration estimators correlate with the model accuracy.  
limitations: not applicable.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: . The paper makes a theoretical treatment of how calibration error is estimated, how biased these estimations are, proposes a taxonomy relating most calibration errors, argue that most calibration errors are lower bounds of the true calibration error, and propose upper bounds that can be estimated efficiently. The paper additionally proposed the concept of proper calibration errors, which are related to proper scoring rules, and are proper in terms that a zero proper calibration error results in a calibrated model.

The contributions are:
- A taxonomy and categorization of calibration errors and how they relate to each other.
- A definition of proper calibration errors, considering that most well known calibration errors used to measure miscalibration, are not proper.
- A proposal for an estimable upper bound for proper calibration error, since most proper calibration errors are intractable to compute.
- Experimental results that shows how practice matches theory, particularly about properly estimating calibration error without bias.
strengths_and_weaknesses: Strengths
- This paper touches an important topic, we usually use ECE and other calibration errors as metrics, but these are biased (they have parameters that can be tricked like number of bins), and this paper provides a good theoretical analysis and formulation on how calibration errors are related, and the fact that zero calibration error does not guarantee a calibrated model due to calibration error estimation biases and other issues. This work makes a good theoretical analysis of how these calibration errors are related, and some possible reasons on why these calibration errors are biased in estimating the true calibration error.
- This paper proposes the concept of proper calibration errors, which I find a good generalization from proper scoring rules, to have calibration errors with guarantees that the model will be calibrated. I believe this is a good contribution to the state of the art and is novel.
- Proper calibration errors are untractable to compute, and this paper proposes the calibration upper bound which is asymptotically unbiased and tractable to compute given some assumptions.
- The evaluation is made with a good selection of calibration errors from the literature, on a good selection of models (ResNet, Wide ResNet), and datasets/tasks, mainly on CIFAR10/100 and ImageNet for classification, and UCI Residential Building for regression. I like that the paper has results on both classification and regression.
- This work evaluates how the calibration error and its bias changes with the size of the test set, this is often not evaluated, and only a single test set size used, which I believe is a good contribution to the literature, and clearly shows the estimation bias of calibration errors.
- The supplementary material is pretty long/detailed and has all the proofs for the lemmas/theorems in the main paper, as well as additional results and reproducibility details. I have not checked the proofs in the supplementary.
- The paper is well written and is good but not so easy to read. It is heavy on statistical details, but overall it is readable and understandable, I only have some minor details on notation in Minor Issues.

Weaknesses
- I believe that the only Weaknesses of the paper is some missing notation and that the mathematical details might be a bit too much for machine learning practitioners, see Minor Issues for more details. Having a section like takeaways for practitioners would be the best (for example, how to compute upper bound calibration error in an algorithmic way for some commonly used proper calibration errors).

Minor issues

~~- What exactly is the RBS? The paper defines it as root of U_BS, but I believe U_BS is not completely defined, my guess is that it is the upper bound of the brier score? Clarification is needed.~~

- Some notation required to read the paper is available in the supplementary,  it would be best to have some of this notation in the main paper for easier readability, for example P_{Y | f(X)} and others.
- One possible suggestion about the literature is the paper [1] which shows expected calibration error with variations of the training set size, which can be compared with this papers' variations of the test set, it is not the same but can be related. I do not suggest this as a reference but only as further research and/or general interest.
- The figures in the paper look blurry in HiDPI screens and with zooming, I think this is because the figures have been produced as raster images (PNG or JPG), if you produce the figures directly as PDF, they will be vector based and easily zoomable and higher quality.

[1]: https://arxiv.org/abs/2111.0980
questions: - What is RBS in detail? I believe that it is not explicitly described in the paper (other than Line 292).
limitations: The paper does not describe negative societal impacts and/or limitations, and I do not believe this is required. There are positive societal impact that comes with better calibrated models and uncertainty quantification in general.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the reliability of common calibration estimators used to assess quality of the uncertainty estimates of machine learning models. Using the proposed framework of “proper calibration errors”, the authors create a taxonomy of calibration estimators, providing bounds on their performance.  Specifically, the framework defines proper calibration error as the expected divergence between the model predictions and true target distribution. Here, the divergence comes from a popper scoring rule. The suggested improvements based on this framework are shown to be effective on several practical applications.
strengths_and_weaknesses: Strengths

- The paper addresses a very important issue of reliability of calibration metrics especially in low data regime. It is shown that in extreme cases, a model with perfect calibration metric value can be severely miscalibrated. Overly optimistic estimation of calibration error can lead to false trust in the model’s uncertainty estimation quality and can lead to errors that can be quite risky in critical applications.
- The paper is easy to read. The taxonomy of calibration estimators is quite useful.
- Interesting theoretical analysis that connects various calibration estimators to proper scores. This connection helps provide some guarantees on the calibration estimation and helps generalize the estimator to non-classification settings.
- Tackles the more stricter notion of calibration than the typical top-class calibration in the classification setting.

Weaknesses 

In critical applications, apart from calibration over a test set, we are interested in the reliability of the confidence score at individual instance level. It is not clear if the upper-bound to the proper calibration error is particularly useful for this. How well would this metric correlate with selective prediction performance or be suitable to assess out-of-distribution detection performance?

questions: See Strengths And Weaknesses section.
limitations: See Strengths And Weaknesses section.
review_title: null

Meta-review: metareview: This paper addresses issues with the reliability of calibration estimators by introducing “proper calibration errors” - low variance estimators that are upper bounds of calibration error. The paper has a strong theoretical grounding and addressees an important issue that is relevant to the machine learning community at large. Though the original draft had some missing notation and definitions, the authors promise to fix these issues in the revision. Overall, this paper is well suited for publication at NeurIPS and will be a nice addition to the program.
confidence: Certain
Decision: Accept