Title: Minimum-Risk Recalibration of Classifiers
Venue: NeurIPS 2023
Accepted: yes
Keywords: probability calibration

optimal number of bins

label shift adaptation
Abstract: Recalibrating probabilistic classifiers is vital for enhancing the reliability and accuracy of predictive models. Despite the development of numerous recalibration algorithms, there is still a lack of a comprehensive theory that integrates calibration and sharpness (which is essential for maintaining predictive power). In this paper, we introduce the concept of minimum-risk recalibration within the framework of mean-squared-error (MSE) decomposition, offering a principled approach for evaluating and recalibrating probabilistic classifiers. Using this framework, we analyze the uniform-mass binning (UMB) recalibration method and establish a finite-sample risk upper bound of order $\tilde{O}(B/n + 1/B^2)$ where $B$ is the number of bins and $n$ is the sample size. By balancing calibration and sharpness, we further determine that the optimal number of bins for UMB scales with $n^{1/3}$, resulting in a risk bound of approximately $O(n^{-2/3})$. Additionally, we tackle the challenge of label shift by proposing a two-stage approach that adjusts the recalibration function using limited labeled data from the target domain. Our results show that transferring a calibrated classifier requires significantly fewer target samples compared to recalibrating from scratch. We validate our theoretical findings through numerical simulations, which confirm the tightness of the proposed bounds, the optimal number of bins, and the effectiveness of label shift adaptation.

Minimum-Risk Recalibration of Classifiers

Abstract

Recalibrating probabilistic classifiers is vital for enhancing the reliability and accuracy of predictive models. Despite the development of numerous recalibration algorithms, there is still a lack of a comprehensive theory that integrates calibration and sharpness (which is essential for maintaining predictive power). In this paper, we introduce the concept of minimum-risk recalibration within the framework of mean-squared-error (MSE) decomposition, offering a principled approach for evaluating and recalibrating probabilistic classifiers. Using this framework, we analyze the uniform-mass binning (UMB) recalibration method and establish a finite-sample risk upper bound of order ˜ O ( B/n +1 /B 2 ) where B is the number of bins and n is the sample size. By balancing calibration and sharpness, we further determine that the optimal number of bins for UMB scales with n 1 / 3 , resulting in a risk bound of approximately O ( n -2 / 3 ) . Additionally, we tackle the challenge of label shift by proposing a two-stage approach that adjusts the recalibration function using limited labeled data from the target domain. Our results show that transferring a calibrated classifier requires significantly fewer target samples compared to recalibrating from scratch. We validate our theoretical findings through numerical simulations, which confirm the tightness of the proposed bounds, the optimal number of bins, and the effectiveness of label shift adaptation.

1 Introduction

Generating reliable probability estimates alongside accurate class labels is crucial in classification tasks. A probabilistic classifier is considered "well calibrated" when its predicted probabilities closely align with the empirical frequencies of the corresponding labels [9]. Calibration is highly desirable, particularly in high-stakes applications such as meteorological forecasting [32, 34, 10, 15], econometrics [16], personalized medicine [25, 24], and natural language processing [37, 7, 11, 53]. Unfortunately, many machine learning algorithms lack inherent calibration [18].

To tackle this challenge, various methods have been proposed for designing post hoc recalibration functions. These functions are used to assess calibration error [52, 42, 17, 4], detect miscalibration [30], and provide post-hoc recalibration [50, 51, 18, 48, 29, 21]. Despite the rapid development of recalibration algorithms, there is still a lack of a comprehensive theory that encompasses both calibration and sharpness (retaining predictive power) from a principled standpoint. Furthermore, existing methods often rely on diverse calibration metrics [17, 4], and the selection of hyperparameters is often based on heuristic approaches [44, 20] without rigorous justifications. This highlights the

need for identifying an optimal metric to evaluate calibration, which can facilitate the development of a unified theory and design principles for recalibration functions.

In addition, the deployment of machine learning models to data distributions that differ from the training phase is increasingly common. These distribution shifts can occur naturally due to factors such as seasonality or other variations, or they can be induced artificially through data manipulation methods such as subsampling or data augmentation. Distribution shifts pose challenges to the generalization of machine learning models. Therefore, it becomes necessary to adapt the trained model to these new settings. One significant category of distribution shifts is label shift, where the marginal probabilities of the classes differ between the training and test sets while the class conditional feature distributions remain the same. With calibrated probabilistic predictions, label shift can be adjusted assuming access to class marginal probabilities [12]. However, miscalibration combined with label shift is common and remains a challenging problem [2, 14, 47, 41].

In this paper, we aim to address these issues in a twofold manner. Firstly, we develop a unified framework for recalibration that incorporates both calibration and sharpness in a principled manner. Secondly, we propose a composite estimator for recalibration in the presence of label shift that converges to the optimal recalibration. Our framework enables the adaptation of a classifier to the label-shifted domain in a sample-efficient manner.

1.1 Related work

Recalibration algorithms. Recalibration methods can be broadly categorized into parametric and nonparametric approaches. Parametric methods model the recalibration function in a parametric form and estimate the parameters using calibration data. Examples of parametric methods include Platt scaling [40], temperature scaling [18], Beta calibration [28], and Dirichlet calibration [27]. However, it has been reported that scaling methods are often less calibrated than supposed, and quantifying the degree of miscalibration can be challenging [29]. In contrast, nonparametric recalibration methods do not assume a specific parametric form for the recalibration function. These methods include histogram binning [50], isotonic regression [51], kernel density estimation [52, 42], splines [23], Gaussian processes [49], among others. Hybrid approaches, integrating both parametric and nonparametric techniques, have also been proposed. For instance, Kumar et al. [29] combine nonparametric histogram binning with parametric scaling to reduce variance and improve recalibration performance. Nevertheless, this hybrid approach is biased when its parametric assumptions fail. In this work, we consider a nonparametric histogram binning method called uniform-mass binning (UMB), which is asymptotically unbiased.

Histogram binning method. Histogram binning methods are widely used for recalibration due to their simplicity and adaptability. The binning schemes can be pre-specified (e.g., uniformwidth binning [18]), data-dependent (e.g., uniform-mass binning [50]), or algorithm-induced [51]. When selecting a binning scheme, it is crucial to consider the trade-off between approximation and estimation. Coarser binning reduces estimation error (variance), leading to improved calibration, but at the expense of increased approximation error (bias), which diminishes sharpness. Thus, determining the optimal binning scheme and hyperparameters, such as the number of bins ( B ), remains an active area of research. [36] proposes a Bayesian binning method, but verifying the priors is often challenging. [44] suggests choosing the largest B that preserves monotonicity, which is heuristic and computationally inefficient. [20] offers a heuristic for choosing the largest B subject to a calibration constraint, lacking a quantitative characterization of sharpness. Our work builds upon the existing upper bounds for calibration risks of binning methods [29, 20] and derived upper bounds for a complementary risk component known as the sharpness risk. We quantitatively characterize the calibration-sharpness tradeoff, which yields an optimal choice for the number of bins that achieves the minimum risk.

Adaptation to label shift. Label shift presents a challenge in generalizing models trained on one distribution (source) to a different distribution (target). As such, adapting to label shift has received considerable attention in the literature [12, 45, 31, 3, 2, 14]. In practical scenarios, it is common to encounter model miscalibration and label shift simultaneously [47, 41]. Empirical observations have highlighted the crucial role of probability calibration in label shift adaptation [2, 13], which is justified by subsequent theories [14]. However, to the best of our knowledge, there has been no prior work that specifically addresses the recalibration with a limited amount of labeled data from the target distribution. Our theoretical analysis points out that using only target labels achieves risk bounds of the same order as the methods using only target features [31, 3, 14].

1.2 Contributions

This paper contributes to the theory of recalibration across three key dimensions.

Firstly, we develop a comprehensive theory for recalibration in binary classification by adopting the mean-squared-error (MSE) decomposition framework commonly used in meteorology and space weather forecasting [5, 33, 8, 46]. Our approach formulates the probability recalibration problem as the minimization of a specific risk function, which can be orthogonally decomposed into calibration and sharpness components.

Secondly, utilizing the aforementioned framework, we derive a rigorous upper bound on the finitesample risk for uniform-mass binning (UMB) (Theorem 1). Furthermore, we minimize this risk bound and demonstrate that the optimal number of bins for UMB, balancing calibration and sharpness, scales on the order of n 1 / 3 , yielding the risk bound of order n -2 / 3 , where n denotes the sample size.

Lastly, we address the challenge of recalibrating classifiers for label shift when only a limited labeled sample from the target distribution is available, a challenging situation for a direct recalibration approach. We propose a two-stage approach: first recalibrating the classfier on the abundant sourcedomain data, and then transfering it to the label-shifted target domain. We provide a finite-sample guarantee for the risk of this composite procedure (Theorem 2). Notably, to control the risk under ε , our approach requires a much smaller sample size from the target distribution than a direct recalibration on the target sample ( Ω( ε -1 ) vs. Ω( ε -3 / 2 ) , cf. Remark 4).

1.3 Organization

This paper is organized as follows. In Section 2, we introduce notation and provide an overview of calibration and sharpness. Section 3 introduces the notion of minimum-risk recalibration by defining the recalibration risk that takes into account both calibration and sharpness. In Section 4, we describe the uniform-mass histogram binning method for recalibration and provide a risk upper bound with rate analysis. We extend our approach to handle label shift in Section 5. To validate our theory and framework, we present numerical experiments in Section 6. Finally, in Section 7, we conclude the paper with a discussion and propose future research directions.

2 Preliminaries

2.1 Notation

Let N and R denote the set of positive integers and the set of real numbers, respectively. For n ∈ N , let [ n ] := { 1 , . . . , n } . For x ∈ R , let ⌊ x ⌋ = max { m ∈ Z : m ≤ x } . For any finite set S = { s i : i ∈ [ n ] } ⊂ R and any k ∈ [ n ] , we let s ( k ) denote the k -th order statistic, which is the k -th smallest element in S .

Let (Ω , E , P ) denote a generic probability space. For an event A ∈ E , the indicator function 1 A : Ω →{ 0 , 1 } is defined such that 1 A ( x ) = 1 if and only if x ∈ A . We write A happens P -almost surely if P ( A ) = 1 . For a probability measure P , define E P as the expectation, with subscript P omitted when the underlying probability measure is clear. For a probability measure P and a random variable X : Ω → R d , let P X := P ◦ X -1 be the probability measure induced by X .

Letting f, g : R → R , we write f ( x ) = O ( g ( x )) as x →∞ if there exist M > 0 and x 0 > 0 such that | f ( x ) | ≤ Mg ( x ) for all x ≥ x 0 . Likewise, we write f ( x ) = Ω( g ( x )) if g ( x ) = O ( f ( x )) . We write f ( x ) ≍ g ( x ) if f ( x ) = O ( g ( x )) and g ( x ) = O ( f ( x )) . We write f ( x ) = ˜ O ( g ( x )) if there is k ≥ 1 such that f ( x ) = O ( g ( x ) log k ( g ( x ))) .

2.2 Calibration and sharpness

Consider the binary classification problem; let X ∈ X and Y ∈ Y := { 0 , 1 } denote the feature and label random variables. Letting P denote a probability measure, we want to construct a function f : X → Z = [0 , 1] that estimates the conditional probability, i.e. ,

Since estimating the probability conditioned on the high dimensional X is difficult, the notion of calibration captures the intuition of (1) in a weaker sense [33, 9, 19].

Definition 1. A function f : X → Z is (perfectly) calibrated with respect to probability measure P , if

Calibration itself does not guarantee a useful predictor. For instance, a constant predictor f ( X ) = E Y is perfectly calibrated, but it does not change with the features. Such a predictor lacks sharpness [26], also known as resolution [35], another desired property which measures the variance in the target Y explained by the probabilistic prediction f ( X ) .

Definition 2. The sharpness of a function f : X → Z with respect to probability measure P refers to the quantity

The following decomposition of the mean squared error (MSE) suggests why it is desirable for a classifier f to be calibrated and have high sharpness; note that Var[ Y ] is a quantity intrinsic to the problem, unrelated to f .

3 Optimal recalibration

For an arbitrary predictor f : X → Z , the aim of recalibration is to identify a post-processing function h : Z → Z such that h ◦ f is perfectly calibrated while maintaining the sharpness of f as much as possible. The calibration and sharpness can be evaluated using the following two notions of risks. We suppress the dependency of risks on f and P when it leads to no confusion.

Definition 3. Let f : X → Z and h : Z → Z . The calibration risk of h over f under P is defined as

Definition 4. Let f : X → Z and h : Z → Z . The sharpness risk of h over f under P is defined as

Note that the calibration risk R cal ( h ; f ) = 0 if and only if h ◦ f is perfectly calibrated, cf. Definition 1. The sharpness risk R sha ( h ; f ) quantifies the decrement in sharpness of f incurred by applying the recalibration map h , and R sha ( h ; f ) = 0 when h is injective [29].

Next we define a comprehensive notion of risk that we will use to evaluate recalibration functions.

Definition 5. Let f : X → Z and h : Z → Z . The recalibration risk of h over f under P is defined as

The following proposition shows that the recalibration risk can be decomposed into calibration risk and sharpness risk. The proof is deferred to Appendix A.

Proposition 1 (Decomposition of recalibration risk) . For any f : X → Z and any h : Z → Z ,

Note that R P ( h ) = 0 if and only if R cal P ( h ) = 0 and R sha P ( h ) = 0 . This happens if and only if h ◦ f is calibrated, and the recalibration h preserves the sharpness of f in predicting Y .

If R ( h ) = 0 , then we call h an optimal recalibration function (or minimum-risk recalibration function ) of f under P . Let h ∗ f,P : Z → Z be the function

Then R ( h ∗ f,P ) = 0 . Indeed, h is a minimum-risk recalibration function of f if and only if h = h ∗ f,P P Z -almost surely.

Problem 1 (Recalibration) . Suppose that we have a measurable function f : X → Z and a dataset D = { ( x i , y i ) : i ∈ [ n ] } that is an independent and identically distributed (IID) sample drawn from P . The goal of recalibration is to estimate a recalibration function ˆ h ≈ h ∗ f,P using f and D .

Problem 1 can be viewed as a regression problem, where we estimate the function form of E [ Y | Z ] from data { ( z i , y i ) : i ∈ [ n ] } , where z i = f ( x i ) , ∀ i ∈ [ n ] .

4 Recalibration via uniform-mass binning

4.1 Uniform-mass binning algorithm for recalibration

Given a dataset of prediction-label pairs { ( z i , y i ) ∈ Z × Y : i ∈ [ n ] } , the histogram binning calibration method partitions Z = [0 , 1] into a set of smaller bins, and estimate E [ Y | Z ] by taking the average in each bin. We consider the uniform mass binning, constructed using quantiles of predicted probabilities.

Definition 6 (Uniform mass binning) . Let S = { z i ∈ [0 , 1] : i ∈ [ n ] } . A binning scheme B = { I 1 , I 2 , . . . , I B } is the uniform-mass binning (UMB) scheme induced by S if

For our subsequent discussions, we make the following assumptions on the distribution of Y and Z :

(A1) The cumulative distribution function of Z , denoted by F Z , is absolutely continuous.

(A2) h ∗ f,P , defined in (7), is monotonically non-decreasing on supp P Z .

(A3) There exists K > 0 such that if z 1 ≤ z 2 , then h ∗ f,P ( z 2 ) -h ∗ f,P ( z 1 ) ≤ K · ( F Z ( z 2 ) -F Z ( z 1 ) ) .

Assumption (A1) is made for the sake of analytical convenience without loss of generality; see discussions in [20, Appendix C]. An important implication of Assumption (A1) is that all intervals in B are non-empty P Z -almost surely if S = { Z i : i ∈ [ n ] } are IID under P Z . Assumption (A2) makes sure Z is informative to preserve the rankings of P [ Y | Z ] [51]. Lastly, Assumption (A3) posits that Z is sufficiently informative that P [ Y = 1 | Z = z ] does not change too rapidly in any interval I where P Z ( I ) is small. This assumption is mild but not trivial; see Appendix B.2.

Now we describe how to construct a recalibration function ˆ h using UMB.

Algorithm.

Given { ( z i , y i ) : i ∈ [ n ] } , and B ∈ N , let B = { I 1 , I 2 , . . . , I B } be the UMB scheme of size B induced by { z i : i ∈ [ n ] } .

Let ˆ h = ˆ h B : Z → Z be a function such that

4.2 Theoretical guarantees

Here we establish a high-probability upper bound on the recalibration risk R ( ˆ h ) for the UMB estimator, ˆ h defined in (9), which converges to 0 as the sample size n increases to ∞ .

Theorem 1. Let P be a probability measure and f : X → Z be a measurable function. Suppose that (A1) & (A2) hold. Let B be the UMB scheme induced by an IID sample of f ( X ) , and let ˆ h = ˆ h B be the recalibration function based on B , cf. (9) . Then there exists a universal constant c > 0 such that for any δ ∈ (0 , 1) , if n ≥ c · |B| log ( 2 |B| δ ) , then with probability at least 1 -δ ,

Remark 1. We note that the upper bound on R cal ( ˆ h ) in Theorem 1 coincides with the result presented in [20] up to a constant factor in the failure probability. However, Theorem 1 provides an additional upper bound on R sha ( ˆ h ) , thereby effectively managing the overall recalibration risk R ( ˆ h ) .

Optimal choice of the number of bins |B| . Based on Theorem 1, for any fixed sample size n , as the number of bins B = |B| increases, the calibration risk bound increases, while the sharpness risk bound decreases. This trade-off suggests we may get an optimal number of bins B by minimizing the upper bound for the overall risk, R ( ˆ h ) = R cal ( ˆ h ) + R sha ( ˆ h ) .

For the simplicity of our analysis, we assume n is divisible by B , and Assumption (A3) holds. Firstly, observe that n/B ≥ 2 , and thus, n/B -1 ≥ n/ (2 B ) . Also, since B ≥ 1 and δ < 1 , log( 4 B δ ) ≥ 1 , it follows that 1 n/B ≤ √ 1 2( n/B -1) log ( 4 B δ ) ≤ √ 1 n/B log ( 4 B δ ) . Thus, we obtain a simplified risk bound R ( ˆ h ) ≤ ζ ( B ; n, δ ) where ζ ( B ; n, δ ) := 4 B n log ( 4 B δ ) + 8 K 2 B 2 . With log B terms ignored, minimizing ζ ( B ; n, δ ) over B yields optimal B ∗ and resulting risk bounds, respectively:

Furthermore, the asymptotic risk bound in (11) implies that R ( ˆ h ) , R cal ( ˆ h ) , and R sha ( ˆ h ) are all bounded by ε with high probability if the sample size

Comparison with the hybrid method [29]. Kumar et al. [29] proposed a hybrid recalibration approach that involves fitting a recalibration function in a parametric family H , which is then discretized using UMB. Note that the hypothesis class H may or may not include the optimal recalibration function h ∗ f,P . If h ∗ f,P ∈ H , using a similar analysis as above, we derive their high probability risk bound as O ( 1 n log B δ + 1 B 2 ) under Assumption (A3) , which achieves ˜ O ( n -1 ) when B ∗ ≍ √ n , exhibiting a faster decay than our R ( ˆ h ) = O ( n -2 / 3 ) . While the faster rate is anticipated from employing parametric methods, we note that when h ∗ f,P / ∈ H , their method exhibits inherent bias (approximation error) induced by parametric function fitting, whereas our method is asymptotically unbiased. This distinction is corroborated by numerical simulations in Appendix E.2.

Proof sketch of Theorem 1. First, we demonstrate that the uniform-mass binning scheme B = { I b } B b =1 satisfies two regularity conditions with high probability, when the sample size n is not too small. Specifically, we show that (i) B is 2 -well-balanced [29] with respect to f ( X ) , resulting in B bins having comparable probabilities (Lemma 3); and that (ii) the empirical mean in each bin of B uniformly concentrates to the population conditional mean of Y conditioned on f ( X ) being contained within the bin (Lemma 4). Thereafter, we prove that if B satisfies these two properties, then the calibration risk R cal and the sharpness risk R sha can be upper bounded as stated in Theorem 1; see Lemmas 5 and 6 (or Lemma 7 when (A3) holds). The detailed proof is in Appendix B.

5 Recalibration under label shift

This section extends the results from Section 4 to address label shift. In Section 5.1, we introduce the label shift assumption (Definition 7) and reframe the recalibration problem accordingly. We show that the optimal recalibration function in this context can be expressed as a composition of the optimal recalibration function (cf. Section 3) and a shift correction function. Building on this observation, we propose a two-stage estimator in Section 5.2, where each stage estimates one of the component functions. The composite estimator's overall performance is supported by theoretical guarantees.

5.1 Revisiting the problem formulation

Let P and Q denote the probability measures of the source and the target domains, respectively. We assume P and Q satisfy the label shift assumption defined below.

Definition 7 (Label shift) . Probability measures P and Q are said to satisfy the label shift assumption if the following two conditions are satisfied:

According to Condition (B1) , the class conditional distributions remain the same, while the marginal distribution of the classes may change. Condition (B2) requires all classes to be present in the source population, which is a standard regularity assumption in the discussion of label shift [31, 14]; it also posits the presence of every class in the target population.

Optimal recalibration under label shift. Under the label shift assumption between P and Q , we define the label shift correction function g ∗ : Z → Z such that

The conditional probabilities under P and Q can be related [45] as follows:

Recall that the optimal recalibration function for a predictor f : X → Z under probability measure P is defined as h ∗ f,P ( z ) = P [ Y = 1 | f ( X ) = z ] ; see (7). In the presence of a label shift between P and Q , we may write the optimal recalibration function for f under Q as

because h ∗ f,Q ( z ) ( a ) = Q [ Y = 1 | f ( X ) = z ] ( b ) = g ∗ ( P [ Y = 1 | f ( X ) = z ]) ( c ) = ( g ∗ ◦ h ∗ f,P ) ( z ) , where (a) and (c) follows from the definition of h ∗ f,P and (b) is due to (14).

Recalling the definition of the risk R P ( h ; f ) from (5), we observe that R Q ( h ∗ f,Q ; f ) = 0 , which is consistent with the risk characterization of the optimal recalibration. Our goal is to estimate the optimal recalibration function h ∗ f,Q = g ∗ ◦ h ∗ f,P from data.

Problem 2 (Recalibration under label shift) . Suppose that we have a measurable function f : X → Z and two IID datasets D P = ( x i , y i ) n P i =1 ∼ P and D Q = ( x ′ i , y ′ i ) n Q i =1 ∼ Q . The goal of recalibration under label shift is to estimate ˆ h ≈ h ∗ f,Q using f , D P and D Q .

Remark 2. The source (training) dataset D P may not be accessible due to privacy protections, proprietary data, or practical constraints, as is often the case when recalibrating a pre-trained black box classifier to new data. In these cases, it suffices to have estimates of the recalibration function h ∗ f,P and the marginal probabilities P [ Y = k ] , k ∈ { 0 , 1 } under P , for our method and analysis.

5.2 Two-stage recalibration under label shift

Method. We propose a composite estimator of h ∗ f,Q = g ∗ ◦ h ∗ f,P , which comprises two estimators ˆ g ≈ g ∗ and ˆ h P ≈ h ∗ f,P . Here we describe a procedure to produce this composite estimator.

Use D P to construct ˆ h P : Z → Z , the estimated recalibration function (9) (for f under P ).

Use D P and D Q to construct ˆ g : Z → Z such that

where ˆ P [ Y = k ] := 1 |D P | ∑ |D P | i =1 1 [ y i = k ] and ˆ Q [ Y = k ] := 1 |D Q | ∑ |D Q | i =1 1 [ y ′ i = k ] are the empirical estimates of the class marginal probabilities.

Let

Note that the recalibration estimator ˆ h P (Step 1) remains the same with that in Section 4.1. Furthermore, the shift correction estimator ˆ g (Step 2) is a plug-in estimator of the label shift correction function g ∗ in (14) based on the estimated weights, ˆ w 1 and ˆ w 0 .

Theory. We present a recalibration risk upper bound for the proposed two-stage estimator. We let p k := P [ Y = k ] , q k := Q [ Y = k ] , and w ∗ k = q k p k for k ∈ { 0 , 1 } . Moreover, we let p min := min k p k , q min := min k p k , w ∗ min := min k w ∗ k and w ∗ max := max k w ∗ k .

Theorem 2 (Convergence of ˆ h Q ) . Let P, Q be probability measures and let f : X → Z be a measurable function. Let D P ∼ P be an IID sample of size n P and D Q ∼ Q be an IID sample of size n Q . Suppose that Assumptions (B1) & (B2) hold. Let B be the UMB scheme induced by D P . Let ˆ h P = ˆ h P, B be the recalibration function (9) based on B , and let ˆ g denote the shift correction function as defined in (16) . Then

Furthermore, suppose that (A1) , (A2) & (A3) hold. Then there exists a universal constant c > 0 such that for any δ ∈ (0 , 1) , if

then with probability at least 1 -δ ,

Remark 3. Note that ρ k → 1 as n P , n Q → ∞ , and thus, the upper bound (18) reduces to 2 w ∗ max 3 w ∗ min 2 · R P ( ˆ h P ; f ) . Moreover, when P = Q , we have w ∗ min = w ∗ max = 1 , and this further simplifies to the recalibration risk without label shift, up to multiplicative constant 2 .

Remark 4 (Target sample complexity) . Assume that n P ≥ n Q and the number of bins satisfies |B| ≍ n 1 / 3 P . Then (19) implies R Q ( ˆ g ◦ ˆ h P ; f ) = O ( n -2 / 3 P + n -1 Q ) . This result indicates that the proposed recalibration method using (17) requires a significantly smaller target sample size n Q = Ω( ε -1 ) to control the risk, as compared to n Q = Ω( ε -3 / 2 ) in (12) .

Remark 5 (Comparison with label shift using unlabeled target data) . When the source sample size n P is sufficiently large, we achieve a risk of R Q (ˆ g ◦ ˆ h P ) = O ( n -1 Q ) with high probability. It is important to note that in this scenario, we only utilize the labels from the target sample to address label shift. Remarkably, the same rate applies when employing the algorithms proposed in [31, 3, 14], which solely rely on features from the target sample. For a proof sketch, please refer to Appendix D.

6 Numerical experiments

In this section, we present the results of our numerical simulations conducted to validate and reinforce the theoretical findings discussed earlier. The simulations are based on a family of recalibration functions called beta calibration [28]: H beta = { h beta ( z ; a, b, c ) : a ≥ 0 , b ≥ 0 , c ∈ R } , where h beta ( · ; a, b, c ) : [0 , 1] → [0 , 1] is defined as

In addition, consider the joint distributions D ( π ) of X and Y , where Y ∼ Bernoulli ( π ) , X | Y = 0 ∼ N ( -2 , 1) , and X | Y = 1 ∼ N (2 , 1) , and a pre-trained probabilistic classifier f ( x ) = σ ( x ) := 1 / (1 + e -x ) . To accommodate the limitations of space, we summarize the results in Figure 1, 2, 3, and Table 1, 2, providing a concise overview. Detailed information about the simulation settings, implementation details, and further experimental results and discussions can be found in Appendix E. Our simulation code is available at https://github.com/ZeyuSun/calibration_label_ shift .

10 3

10 5

10 7

n

10

7

10

5

10

3

10

1

Calibration Risk

B=6

B=34

B=184

B=1000

(a)

R

cal

( ˆ

h

)

vs.

n

10 1

10 2

10 3

B

10

7

10

5

10

3

10

1

Calibration Risk

n=1.0e+02

n=4.6e+03

n=2.2e+05

n=1.0e+07

(b)

R

cal

( ˆ

h

)

vs.

B

10 1

10 2

10 3

B

10

5

10

3

10

1

10 1

Sharpness Risk

n=1.0e+02

n=4.6e+03

n=2.2e+05

n=1.0e+07

(c)

R

sha

( ˆ

h

)

vs.

B

10 1

10 2

10 3

B

10

5

10

3

10

1

10 1

Risk

n=1.0e+02

n=4.6e+03

n=2.2e+05

n=1.0e+07

(d)

R

( ˆ

h

)

vs.

B

10 3

10 5

10 7

n

0

500

1000

1500

2000

Optimal B

experiment

theory

10 3

10 5

10 7

n

10 1

10 2

10 3

Optimal B

experiment

theory

n

1/3

6

19

59

184

569

B

1.0e+02

4.6e+03

2.2e+05

1.0e+07

n

10

4

10

2

Risk

(a) Correct parametric assumption

(b) Misspecified parametric assumption

0.0

0.5

1.0

Predicted Probability

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Frequency

h

*

Platt

Hybrid

UWB

UMB

(a) Correct parametric assumption

0.0

0.5

1.0

Predicted Probability

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Frequency

h

*

Platt

Hybrid

UWB

UMB

(b) Misspecified parametric assumption

7 Discussion

This paper presents a comprehensive theory for recalibration, considering both calibration and sharpness within the mean-squared-error (MSE) decomposition framework. We use this framework to quantify the optimal calibration-sharpness balance and establish a rigorous upper bound on the finite-sample risk for uniform-mass binning (UMB). Additionally, we address the challenge of recalibration under label shift with limited access to labeled target data. Our proposed two-stage approach effectively estimates the recalibration function using ample data from the source domain and adjusts for the label shift using target domain data. Importantly, our findings suggest that transferring a calibrated classifier requires a significantly smaller target sample than recalibrating from scratch on the new domain. Numerical simulations confirm the tightness of the finite sample bounds, validate the optimal number of bins, and demonstrate the effectiveness of the label shift adaptation.

In concluding this paper, we identify several promising directions for future research.

Relaxation of the assumptions It would be worthwhile to explore whether or not the assumptions made in our analysis could be relaxed. For instance, the widely adopted monotonicity assumption (A2) and its variants [51, 52, 43, 44] may not hold in real-world settings. Thus, exploring potential relaxations of this assumption is valuable. In addition, a mild but non-trivial smoothness assumption (A3) (c.f. Remark 7 and 8) is introduced to obtain a tight sharpness risk upper bound (c.f. Remark 9); investigating its practical implications and exploring potential relaxations could be interesting future work.

Calibration-sharpness framework analysis Applying our framework to analyze recalibration methods beyond UMB, such as isotonic regression [51] and kernel density estimation [52], can offer further insights into their performance and properties, providing guidance to practitioners in selecting suitable algorithms based on specific conditions and requirements.

Multiclass probability recalibration The concept of calibration considered in this work extends to multi-class classification settings, known as canonical calibration [48]. Weaker, but more tractable notions of multi-class calibration have also been explored in the literature [48, 21]. While partitionbased methods, as multi-class extensions of binning methods, are known to have consistency [48] and vanishing calibration error [41], establishing upper bounds for their sharpness risk remains difficult. Additionally, designing a partition scheme in a multidimensional space is a challenging task [21]; the interplay between calibration and sharpness could potentially guide the development of partition strategies that balance both aspects in multi-class classification.

Applications to real-world data Investigating the calibration-sharpness tradeoff in real-world applications, which often involve multiple classes, presents an interesting challenge. It is crucial to develop effective estimators for both calibration risk and sharpness risk in such scenarios. While estimators for calibration risk exist (e.g., binning-based and KDE-based) [6, 29, 52, 42] and a lower bound for sharpness risk has been established [39], a direct estimator of sharpness risk is still lacking.

Acknowledgments and Disclosure of Funding

The research in this paper was partially supported by ARO grants W911NF-23-1-0343 and W911NF19-1-0269 and by DOE grant DE-NA0003921.

References

John Aitchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika , 67(2):261-272, 1980.

Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with biascorrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning , pages 222-232. PMLR, 2020.

Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations , 2018.

Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , pages 1727-1740, 2023.

Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly Weather Review , 78(1):1-3, 1950.

Jochen Bröcker. Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. Climate dynamics , 39:655-667, 2012.

Dallas Card and Noah A Smith. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1636-1646, 2018.

Misty D Crown. Validation of the NOAA Space Weather Prediction Center's solar flare forecasting look-up table and forecaster-issued probabilities. Space Weather , 10(6), 2012.

A Philip Dawid. The well-calibrated Bayesian. Journal of the American Statistical Association , 77(379):605-610, 1982.

Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician) , 32(1-2):12-22, 1983.

Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 295-302, 2020.

Charles Elkan. The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence , volume 17, pages 973-978, 2001.

Andrea Esuli, Alessio Molinari, and Fabrizio Sebastiani. A critical reassessment of the SaerensLatinne-Decaestecker algorithm for posterior probability adjustment. ACM Transactions on Information Systems (TOIS) , 39(2):1-34, 2020.

Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems , 33:3290-3300, 2020.

Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods. Science , 310(5746):248-249, 2005.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007.

Sebastian Gruber and Florian Buettner. Better uncertainty calibration via proper scores for classification and beyond. Advances in Neural Information Processing Systems , 35:8618-8632, 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , pages 1321-1330. PMLR, 2017.

Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. Distribution-free binary classification: Prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems , 33:3711-3723, 2020.

Chirag Gupta and Aaditya Ramdas. Distribution-free calibration guarantees for histogram binning without sample splitting. In International Conference on Machine Learning , pages 3942-3952. PMLR, 2021.

Chirag Gupta and Aaditya Ramdas. Top-label calibration and multiclass-to-binary reductions. In International Conference on Learning Representations , 2022.

Chirag Gupta and Aaditya Ramdas. Online Platt scaling with calibeating. In International Conference on Machine Learning , 2023.

Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Conference on Learning Representations , 2020.

Yingxiang Huang, Wentao Li, Fima Macheret, Rodney A Gabriel, and Lucila Ohno-Machado. A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association , 27(4):621-633, 2020.

Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association , 19(2):263-274, 2012.

Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. Advances in Neural Information Processing Systems , 28, 2015.

Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in Neural Information Processing Systems , 32, 2019.

Meelis Kull, Telmo M Silva Filho, and Peter Flach. Beyond sigmoids: How to obtain wellcalibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics , 11:5052-5080, 2017.

Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems , 32, 2019.

Donghwan Lee, Xinmeng Huang, Hamed Hassani, and Edgar Dobriban. T-Cal: An optimal test for the calibration of predictive models. arXiv preprint arXiv:2203.01850 , 2022.

Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International Conference on Machine Learning , pages 3122-3130. PMLR, 2018.

Allan H Murphy. A new vector partition of the probability score. Journal of Applied Meteorology and Climatology , 12(4):595-600, 1973.

Allan H Murphy and Edward S Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology and Climatology , 6(5):748-755, 1967.

Allan H Murphy and Robert L Winkler. Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society Series C: Applied Statistics , 26(1):41-47, 1977.

Allan H Murphy and Robert L Winkler. A general framework for forecast verification. Monthly weather review , 115(7):1330-1338, 1987.

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Khanh Nguyen and Brendan O'Connor. Posterior calibration and exploratory analysis for natural language processing models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 1587-1598, 2015.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Alexandre Perez-Lebel, Marine Le Morvan, and Gaël Varoquaux. Beyond calibration: Estimating the grouping loss of modern neural networks. In The Eleventh International Conference on Learning Representations , 2023.

John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers , 10(3):61-74, 1999.

Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. In Uncertainty in Artificial Intelligence , pages 844-853. PMLR, 2021.

Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable l p canonical calibration error estimator. Advances in Neural Information Processing Systems , 35:7933-7946, 2022.

Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. Intra order-preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems , 33:13456-13467, 2020.

Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C Mozer. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics , pages 4036-4054. PMLR, 2022.

Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation , 14(1):21-41, 2002.

Zeyu Sun, Monica G Bobra, Xiantong Wang, Yu Wang, Hu Sun, Tamas Gombosi, Yang Chen, and Alfred Hero. Predicting solar flares using CNN and LSTM on two solar cycles of active region data. The Astrophysical Journal , 931(2):163, 2022.

Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re-calibration for imbalanced datasets. Advances in Neural Information Processing Systems , 33:8101-8113, 2020.

Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 3459-3467. PMLR, 2019.

Jonathan Wenger, Hedvig Kjellström, and Rudolph Triebel. Non-parametric calibration for classification. In International Conference on Artificial Intelligence and Statistics , pages 178-190. PMLR, 2020.

Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning , pages 609-616, 2001.

Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 694-699, 2002.

Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning , pages 11117-11128. PMLR, 2020.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning , pages 12697-12706. PMLR, 2021.

A Proof of Proposition 1

Proof of Proposition 1. First of all, we recall the definition of the two risks from (3) and (4):

To keep our notation concise, we use Z = f ( X ) as a shorthand notation, and also let Y Z := E [ Y | Z ] and Y h ( Z ) = E [ Y | h ( Z )] throughout this proof. We can decompose the recalibration risk from Definition 5:

B Proof of Theorem 1

In this section, we present a proof of Theorem 1. Let ( Y, Z ) ∈ Y × Z be random variables that admits a joint distribution P Y,Z , which we assume to be fixed throughout this section. Let S = { ( y i , z i ) ∈ Y × Z : i ∈ [ n ] } and let B = { I 1 , I 2 , . . . , I B } be the uniform-mass binning scheme (cf. Definition 6) of size B induced by ( z i 's in) S . Note that if S is a random sample from P Y,Z , then the binning scheme B induced by S is also a random variable following a derived distribution. To facilitate our analysis, we introduce the notion of well-balanced binning.

Definition 8 (Well-balanced binning; [29]) . Let B ∈ N , let Z be a random variable that takes value in [0 , 1] , and let α ∈ R such that α ≥ 1 . A binning scheme B of size B is α -well-balanced with respect to Z if

In addition, we define two (parameterized families of) Boolean-valued functions Φ balance and Φ approx as follows: for any binning scheme B ,

where 1 ( A ) = 1 if and only if the predicate A is true, and for each interval I ∈ B ,

Note that if Φ balance ( B ; α ) = 1 for α ≥ 1 , then B is α -well-balanced with respect to Z (cf. Definition 8). Also, if Φ approx ( B ; ε ) = 1 for ε ≥ 0 , then the conditional empirical mean of Y in each bin I ∈ B approximates the conditional expectation with error at most ε , uniformly for all bins.

The rest of this section is organized as follows. In Section B.1, we ensure that for an appropriate choice of α, ε ∈ R , it holds with high probability (with respect to the randomness in B ) that Φ balance ( B ; α ) = Φ approx ( B ; ε ) = 1 . In Section B.2, we establish upper bounds on the reliability risk R cal and the sharpness risk R sha under the premise that Φ balance ( B ; α ) = Φ approx ( B ; ε ) = 1 . Finally, in Section B.3, we conclude the proof of Theorem 1 by combining these results together.

B.1 High-probability certification of the conditions

Well-balanced binning scheme. First of all, we observe that the uniform-bass binning scheme B induced by an IID random sample from P Y,Z is 2-well-balanced with high probability, if the sample size is sufficiently large. Here we paraphrase a result from [29] in our language.

Lemma 3 ([29, Lemma 4.3]) . Let S = { Z i : i ∈ [ n ] } be an IID sample drawn from P Z and let B be the uniform-mass binning scheme of size B induced by S . There exists a universal constant c ′ > 0 such that for any δ ∈ (0 , 1) , if n ≥ c ′ · B log( B/δ ) , then Φ balance ( B , 2) = 1 with probability at least 1 -δ .

Lemma 3 states that

While the value of the universal constant c was not specified in the original reference [29], we remark that one may set, for example, c ′ = 2420 , which can be verified by following their proof with c ′ kept explicit.

The proof of Lemma 3 in [29] relies on a discretization argument that considers a fine-grained cover of Z = [0 , 1] consisting of disjoint intervals-namely, { I ′ j : j ∈ [10 B ] } such that P [ Z ∈ I ′ j ] = 1 10 B for all j ∈ [10 B ] -and then approximates each I b by a subset of the cover. As the authors of [29] remarked, this argument provides a tighter sample complexity upper bound than naïvely applying Chernoff bounds or a standard VC dimension argument, which would yield an upper bound of order O ( B 2 log ( B δ )) . We omit the proof of Lemma 3 and refer interested readers to the referenced paper [29] for more details.

Uniform concentration of bin-wise means. Next, we argue that for the uniform-mass binning scheme B induced by an IID sample, the conditional empirical means of each bin concentrates to the population conditional expectation, uniformly for all bins in B . Here we restate a result from [20].

Lemma 4 ([20, Corollary 1]) . Let P Z be an absolutely continuous probability measure on Z = [0 , 1] , and S = { Z i : i ∈ [ n ] } be an IID sample drawn from P Z . Let B ∈ N such that B ≤ n 2 and B be the uniform-mass binning scheme of size B induced by S . Then for any δ ∈ (0 , 1) ,

Lemma 4 states that under the mild regularity condition of P Z being absolutely continuous, the uniform-mass binning accurately approximates all bin-wise conditional means as long as there are at least two samples per bin in the sense that

B.2 Conditional upper bounds on reliability risk and sharpness risk

In this section, we establish upper bounds on the reliability risk R cal and the sharpness risk R sha for ˆ h under the premise that Φ balance ( B ; α ) = 1 and Φ approx ( B ; ε ) = 1 for appropriate parameters α, ε ∈ R .

Preparation. To avoid clutter in the lemma statements to follow, here we recall our problem setting and set several notation that will be used throughout this section. Recall that P = P X,Y is a joint distribution on X × Y and let f : X → Z is a measurable function. In addition, we let ˜ S = { ( x i , y i ) ∈ X × Y : i ∈ [ n ] } be an IID sample drawn from P , and let S = { ( z, y ) ∈ Z × Y : ( x, y ) ∈ ˜ S and z = f ( x ) } . Let B be the uniform-mass binning scheme induced by ( z 's in) S , and let ˆ h = ˆ B : Z → Z be the recalibration function derived from B as we described in Section 4.1; see (9). The dependence among P, f, ˜ S, S, B , and ˆ h are summarized by a diagram in Figure 4.

Furthermore, we define the index function for a binning scheme to facilitate our analysis.

P

˜

S

f

S

B

ˆ

h

Definition 9. Let B be a binning scheme. The index function for B is the function β : Z → [ |B| ]

such that

Remark 6. Note that β is a measurable function and defines an index function that identifies which bin of B the argument z ∈ [0 , 1] belongs to. Specifically, suppose that B = { I 1 , . . . , I B } for some B ∈ N and there exists u 0 , u 1 , . . . , u B ∈ [0 , 1] such that (i) 0 = u 0 < u 1 < · · · < u B = 1 and (ii) I b = ( u b -1 , u b ] for all b ∈ [ B ] \ { 1 } and I 1 = [ u 0 , u 1 ] . Then β ( z ) = b if and only if z ∈ I b .

B.2.1 Calibration risk upper bound

We observe that if a binning scheme B produces empirical means ˆ µ I that approximate the true means µ I with error at most ε , then the calibration risk is upper bounded by ε 2 .

Lemma 5 (Calibration risk bound) . For any ε ≥ 0 , if Φ approx ( B ; ε ) = 1 , then

Proof of Lemma 5. To begin with, we recall the definition of the calibration risk (Definition 3), and let Z = f ( X ) . Then we may write

Note that if Φ approx ( B ; ε ) = 1 , then max I ∈B ( ˆ µ I -µ I ) 2 ≤ ε 2

We remark that the proof of Lemma 5 is a simple application of applying Hölder's inequality. Also, we note that a similar argument was considered in [20, Proposition 1] to establish the inequalities between the L p -counterparts of the calibration risk, which they call the ℓ p -expected calibration error (ECE). In this work, we focus on the case p = 2 .

B.2.2 Sharpness risk upper bound

Next, we present an upper bound for the sharpness risk that diminishes as the binning scheme B becomes more balanced.

Lemma 6 (Sharpness risk bound) . Suppose that the optimal post-hoc recalibration function h ∗ f,P , cf. (7) , is monotonically non-decreasing. Let α ∈ R such that α ≥ 1 . If Φ balance ( B , α ) = 1 , then

Proof of Lemma 6. Letting Z = f ( X ) , we can write the sharpness risk of ˆ h over f with repsect to P as

We recall the definition of the index function β for B (Definition 9) and observe that

The inequality in the last line follows from the facts that (i) I ∈ B are mutually exclusive and (ii) h ∗ f,P ( z ) ∈ [0 , 1] and h ∗ f,P is monotone non-decreasing.

Our proof of Lemma 6 relies on similar techniques that are used in [29, Lemmas D.5 and D.6]. However, we note that we obtain an improved constant - 1 as opposed to 2 in [29, Lemma D.6] with a more refined analysis.

An improved rate with additional assumptions. It is possible to improve the rate of the sharpness risk upper bound from O ( |B| -1 ) to O ( |B| -2 ) with an additional regularity assumption on h ∗ f,P .

Recall that we assumed in (A3) that there exists K > 0 such that if z 1 ≤ z 2 , then h ∗ f,P ( z 2 ) -h ∗ f,P ( z 1 ) ≤ K · ( F Z ( z 2 ) -F Z ( z 1 ) ) , that is, h ∗ f,P is K -smooth with respect to F Z . This posits that the conditional probability P [ Y = 1 | Z = z ] of the target variable Y given a forecast variable Z cannot vary too much in regions where the density of Z is low, or where the forecast is rarely issued. This is a reasonable assumption because if P [ Y = 1 | Z ] changes too rapidly with respect to Z , then it suggests that we need additional information about Y beyond what Z can provide in order to improve the quality of forecasts. We remark that (A3) is indeed a fairly mild assumption to impose on, however, is not a trivial one.

Remark 7 (Mildness of (A3) ) . Suppose that Z = f ( X ) has a density p Z that is uniformly lower bounded by ϵ on the support of Z . If h ∗ f,P is L -Lipschitz, then h ∗ f,P is ( L/ϵ ) -smooth with respect to F Z . This also provides a sufficient condition to verify (A3) in practice.

Remark 8 (Non-triviality of (A3) ) . Notice that even if F Z is absolutely continuous and h ∗ f,P is continuous, the smoothness constant K could become large if the prediction Z is heavily miscalibrated. For instance, in Figure 6, h ∗ f,P ( z ) is changing fast in the interval [0 . 5 , 0 . 75] where p Z ( z ) is small, which results in a larger value of K that can even diverge if p Z ( z ) → 0 .

Here we define the notion of ψ -smoothness to formalize Assumption (A3) , and then present an improved upper bound for the sharpness risk.

Definition 10 ( ψ -smoothness) . Let K ∈ R + and ψ : [0 , 1] → [0 , 1] be a monotone non-decreasing function. A function ϕ : [0 , 1] → [0 , 1] is K -smooth with respect to ψ if for any z 1 , z 2 ∈ [0 , 1] such that z 1 ≤ z 2 ,

Lemma 7 (Improved sharpness risk bound) . Suppose that the function h ∗ f,P ( z ) defined in (7) is monotonically non-decreasing and K -smooth with respect to F Z for some K ≥ 0 , where F Z is the cumulative distribution function of Z = f ( X ) . If Φ balance ( B , α ) = 1 , then

Proof of Lemma 7. Let Z = f ( X ) and B = |B| . For each b ∈ [ B ] , we let z b, max := sup I b and z b, min := inf I b . Then we have

B.3 Completing the proof of Theorem 1

Proof of Theorem 1. For given δ ∈ (0 , 1) , let δ 1 = δ 2 = δ/ 2 . Then we observe that

where c ′ > 0 is the universal constant that appears in Lemma 3 and

Observe that δ 1 = δ 2 < 1 2 and |B| ≥ 1 , and thus, log ( |B| δ 1 ) ≥ log 2 . Letting c := max { c ′ , 2 log 2 } and applying the union bound, we have

Next, we observe that if Φ balance ( B , 2) = 1 and Φ approx ( B , ε δ 2 ) = 1 , then

Additionally, if the Assumption (A3) also holds, then we obtain a stronger upper bound on R sha ( ˆ h ; f, P ) by Lemma 7:

C Proof of Theorem 2

This section contains a proof of Theorem 2. Prior to the proof, in Section C.1, we provide several lemmas that will be useful in our proof. Thereafter, we present a proof of Theorem 2 in its entirety in Section C.2.

C.1 Useful lemmas

C.1.1 Concentration of ˆ w k to w ∗ k

First of all, we recall the binomial Chernoff bound, which is a classical result about the concentration of measures that can be found in standard textbooks on probability theory.

Lemma 8 (Binomial Chernoff bound) . Let X i be IID Bernoulli random variables with parameters p ∈ (0 , 1) , and let S n := 1 n ∑ n i =1 X i . Then for any δ ∈ R such that 0 < ε < 1 ,

It follows from Lemma 8 that for any ε, δ ∈ (0 , 1) ,

Let P, Q be two distributions on Y = { 0 , 1 } , and let D P ∼ P , D Q ∼ Q denote IID samples of size n P , n Q , respectively. Recall from (13) and (16) that for each k ∈ { 0 , 1 } , we define

Then, we let

Now we define another parameterized family of Boolean-valued functions Φ ratio ( D P , D Q ; β ) as follows. Given D P ∼ P , D Q ∼ Q , and β ∈ R such that 1 < β ≤ 2 ,

Corollary 9. Let P, Q be two distributions on Y = { 0 , 1 } , and let D P ∼ P , D Q ∼ Q denote IID samples of size n P , n Q , respectively. For each k ∈ { 0 , 1 } , let p k := P P [ Y = k ] and q k := P Q [ Y = k ] . Likewise, we let ˆ p k = 1 n P ∑ y i ∈D P 1 { y i = k } and ˆ q k = 1 n Q ∑ y i ∈D Q 1 { y i = k } . For any δ ∈ (0 , 1) and any β ∈ (1 , 2] , if

then

Proof of Corollary 9. Let ε = β -1 3 . Since 1+ x 1 -x ≤ 1 + 3 x for all x ∈ [0 , 1 / 3] , we have 1 β ≤ 1 -ε 1+ ε < 1+ ε 1 -ε ≤ β . Then it follows from (27) that for each k ∈ { 0 , 1 } ,

Applying the union bound, we obtain the following implication:

C.1.2 Regularity of the Shift Correction Function

Lemma 10. Let w = ( w 0 , w 1 ) ∈ R 2 such that w 0 , w 1 > 0 and w 0 + w 1 = 1 . The function g w : [0 , 1] → [0 , 1] such that g w ( z ) = w 1 z w 1 z + w 0 (1 -z ) is L -Lipschitz where L = max { w 1 w 0 , w 0 w 1 } .

Proof of Lemma 10. First of all, consider the first-order derivative of g w :

We observe that g w is monotone increasing as d dz g w ( z ) > 0 for all z ∈ [0 , 1] . Next, we consider the second-order derivative of g w :

Therefore,

Lemma 11. Let P, Q be joint distributions of ( X,Y ) ∈ X × { 0 , 1 } , and let w k = P [ Y = k ] Q [ Y = k ] for k ∈ { 0 , 1 } . If P, Q satisfy the label shift assumption (Definition 7), i.e., if Assumptions (B1) and (B2) hold, then for any measurable function f : X → R , the following two-sided inequality holds:

Proof of Lemma 11. First of all, we observe that

)

|

Y

]

.

by the law of total expectation by definition of

w

k

&the label shift assumption

Thus, it follows that min k w k · E P [ f ( X )] ≤ E Q [ f ( X )] ≤ max k w k · E P [ f ( X )] .

C.2 Completing the proof of Theorem 2

Proof of Theorem 2. This proof is presented in four steps. In Step 1, we establish a simple upper bound for the risk R Q ( ˆ h Q ; f ) that consists of two error terms: the first term quantifies the error introduced by the estimated label shift correction, ˆ g , while the second term quantifies the error due to the estimated recalibration function, ˆ h P . In Steps 2 and 3, we derive separate upper bounds for these two error terms. Finally, in Step 4, we combine the results from Steps 1-3 to obtain a comprehensive upper bound for R Q , which concludes the proof.

Step 1. Decomposition of R Q . Recalling the definition of the risk R Q , cf. (5), we obtain the following inequality:

2 2 2

where (a) follows from the simple inequality ( a + b ) ≤ 2( a + b ) for all a, b ∈ R . In Step 2 and Step 3 of this proof, we establish separate upper bounds for the two terms, T 1 , T 2

Step 2. An upper bound for T 1 . Recall from (13) and (16) that

Let

Then we observe that for any z ∈ (0 , 1) ,

It remains to establish probabilistic tail bounds for ρ 0 , ρ 1 , which we will accomplish in Step 4 of this proof.

.

Step 3. An upper bound for T 2 . We observe that

Step 4. Concluding the proof. For given δ ∈ (0 , 1) , let 2 δ 1 = δ 2 = δ/ 4 and δ 3 = δ/ 2 . We observe that

where c ′ > 0 is the universal constant that appears in Lemma 3 and

Furthermore, assuming

we may define β δ 3 as a function of n P , n Q and δ 3 such that

Then it follows from Corollary 9 that

Observe that δ 1 = δ 4 < 1 4 and |B| ≥ 4 , and thus, log ( |B| δ 1 ) ≥ log 16 ≥ 2 . Let c = c ′ . Since c ′ ≥ 1 and log ( |B| δ 1 ) ≥ log ( 16 δ ) = log ( 8 δ 3 ) , we notice that

In summary, we obtain that for any given δ ∈ (0 , 1) ,

Conditioned on the event Φ balance ( B , 2) = 1 & Φ approx ( B , ε δ/ 4 ) = 1 & Φ ratio ( D P , D Q ; β δ/ 2 ) = 1 , T 1 ≤ (∣ ∣ ρ 0 -ρ 1 ∣ ∣ ρ 0 + ρ 1 ) 2 ≤ ( β δ/ 2 -1 β δ/ 2 β δ/ 2 + 1 β δ/ 2 ) 2 ≤ ( β δ/ 2 -1 ) 2 , ∵ (34) ; also, see (29) T 2 ≤ w ∗ max 3 w ∗ min 2 · R P ( ˆ h P ; f ) ≤ w ∗ max 3 w ∗ min 2 · ( ε 2 δ/ 4 + 2 |B| ) . ∵ proof of Theorem 1; Lemmas 5 & 6

Note that if Assumption (A3) holds, then we additionally have

Inserting these upper bounds for T 1 and T 2 into (31), (32) and recalling the expression for β in (35), we complete the proof.

D Proof sketch of the argument in Remark 5

Recall our composite recalibration function,

does not use the features in D Q . Specifically, ˆ g is parameterized by ˆ w = ( ˆ w 0 , ˆ w 1 ) , which can be estimated using only labels in D P and D Q , cf. (16). According to Theorem 2, R Q ( ˆ h Q ) = O ( n -1 Q ) with high probability for sufficiently large n P .

Now suppose we are given an unlabeled target sample with unknown label shift. We can estimate w using the target features via a maximum likelihood label shift estimation approach [14], yielding ˆ w ML = ( ˆ w ML 0 , ˆ w ML 1 ) . and the calibrated classifier ˆ h ◦ f . This results in a different composite recalibration function than Equation (37),

where ˆ g ML : [0 , 1] → [0 , 1] is defined as ˆ g ML ( z ) = ˆ w ML 1 z/ ( ˆ w ML 1 z + ˆ w ML 0 (1 -z )) . We claim in Remark 5 that, for sufficiently large n P , the composite recalibration function in Equation (38) achieves R Q ( ˆ h ML Q ) = O ( n -1 Q ) with high probability, enjoying the same convergence rate as ˆ h Q (Equation 37). Here we give a proof sketch.

Proof sketch. Suppose we use the maximum likelihood approach in [14] to estimate w . We want to show R Q ( ˆ h ML Q ) = O ( n -1 Q ) with high probability for sufficiently large n P . Recall R Q ( ˆ h Q ) ≤ 2( T 1 + T 2 ) according to Equation (31) and (32), and label shift estimation error only affects T 1 , so it is sufficient to show T 1 = O ( n -1 Q ) with high probability.

For sufficiently large n P , ˆ h P ◦ f is sufficient calibrated, so ∥ ˆ w -w ∥ 2 2 = O ( n -1 Q ) by Theorem 3 in [14]. Since

we have ρ k ∈ [1 -α, 1+ α ] for k ∈ { 0 , 1 } , where α = ∥ ˆ w -w ∥ 2 w ∗ min > 0 . For sufficiently small ∥ ˆ w -w ∥ 2 2 , we can control α < 0 . 5 , which bounds T 1 in (34):

The rest of the proof are the same with Appendix C.2.

E Details on the experiments

In Section E.1 and E.3, we consider a family of joint distributions D ( π ) of X and Y , where Y ∼ Bernoulli ( π ) , X | Y = 0 ∼ N ( -2 , 1) , and X | Y = 1 ∼ N (2 , 1) . Suppose we are given f ( x ) = σ ( x ) := 1 / (1 + e -x ) , for x ∈ R , as a probabilistic classifier. The optimal recalibration function can be derived as

In Section E.2, we consider a parametric family of recalibration functions called beta calibration [28]: H beta = { h beta ( · ; a, b, c ) : a ≥ 0 , b ≥ 0 , c ∈ R } , where h beta ( · ; a, b, c ) : [0 , 1] → [0 , 1] is defined as

In addition, consider a subfamily H logit-normal ⊂ H beta defined as H logit-normal = { h logit-normal ( · ; a, c ) := h beta ( · ; a, a, c ) = σ ( aσ -1 ( · ) + c ) : a ≥ 0 , c ∈ R } 3 . Apparently, the optimal recalibration function in Equation (41), h ∗ f,P ∈ H logit-normal .

E.1 Verifying results for UMB

First, we recalibrate f on data distributed as D (0 . 5) using UMB.

Verifying the risk convergence in Theorem 1 We vary n ∈ [10 2 , 10 7 ] and B ∈ [6 , 10 3 ] in the log scale. For each combination of ( n, B ) , we use UMB to recalibrate f on data generated from D (0 . 5) , and compute quadrature estimates of population R cal ( ˆ h ) , R sha ( ˆ h ) , and R ( ˆ h ) , as well as their high probability bounds based on Theorem 1. The constant K in Assumption (A3) is selected by numerical maximization as

Figure 1 shows the bounds follow the same trends as their associated population quantities, providing valid upper bounds in all cases.

Verifying the optimal choice of the number of bins. We find empirically optimal B ∗ experiment that achieves the minimal risk for each choice of n . We compute the theoretically optimal choice of the number of bins, B ∗ theory , by minimizing the finite-sample upper bounds. Figure 2 shows B ∗ experiment follows the same trend with B ∗ theory , both scales in O ( n 1 / 3 ) .

E.2 Comparing recalibration methods

To highlight the benefits and drawbacks of UMB's nonparametric nature, we compare UMB with (semi-)parametric recalibration methods in scenarios where the parametric assumption is correct and where it is misspecified. We compare the method under study, uniform-mass binning (UMB), with 3 other recalibration methods: uniform-width binning (UWB) [18], Platt scaling [40] 4 , and a hybrid parametric-binning method [29]. Note that Platt scaling and the hybrid method adopt the parametric assumption h ∗ ∈ H logit-normal .

For the first setting, we construct optimal recalibration function h ∗ ∈ H logit-normal so that the parametric assumption of Platt scaling and the hybrid method holds. In particular, we consider the distribution Z ∈ Uniform [0 , 1] and Y | Z ∼ Bernoulli ( h logit-normal ( Z ; a, c )) with a = 4 and c = 0 . For the second setting, we construct h ∗ ∈ H beta but h ∗ / ∈ H logit-normal so that the parametric assumption fails. In particular, we consider the distribution Z ∼ Uniform [0 , 1] and Y | Z ∼ Bernoulli ( h beta ( z ; a, b, c )) with a = 0 . 1 , b = 4 , and c = 0 . For each setting, we fix calibration sample size to be n = 5000 .

10

2

5

100

2

5 1000 2

5

100μ

2

5

0.001

2

5

0.01

2

5

1μ

100μ

5

100μ

2

5

0.001

2

5

0.01

2

5

Platt

Hybrid

UWB

UMB

B

R

R

sha

R

cal

10

2

5

100

2

5 1000 2

2

5

0.001

2

5

0.01

2

5

1μ

10μ

100μ

0.001

100μ

0.001

0.01

0.1

Platt

Hybrid

UWB

UMB

B

R

R

sha

R

cal

(a) Correct parametric assumption

(b) Misspecified parametric assumption

Risks as functions of the number of bins B We traverse the number of bins B ∈ [10 , 2000] in the log scale and compare how each method behaves as B changes. When the parametric assumption is correct, the hybrid method achives significantly lower R cal and overall R than UMB and UWB for sufficiently large number of bins (Figure 5a), an advantage highlighted in [29]. In contrast, when the parametric assumption fails, the binning methods UMB and UWB has better performance with the optimal number of bins (Figure 5b). This is because Platt scaling and hybrid methods are intrinsicly biased when h ∗ / ∈ H logit-normal, as noted in Section 4.2.

Quantitative results of risks under optimal B For each setting, we fix B that achieves low recalibration risk for UWB and UMB in Figure 5. Specifically, we choose B = 2 ⌊ n 1 / 3 ⌋ = 34 for the correct parametric assumption setting, and B = ⌊ n 1 / 3 ⌋ = 17 for the misspecified parametric assumption setting. Then, for each setting, we compare the 90% quantiles of risks of each recalibration method fitted on 100 random replicates of calibration datasets of size n = 5000 .

Table 1 quantititively verifies that Platt and the Hybrid method achieves lower R cal and overall R if the parametric assumption is correct, and UWB and UMB achieves lower R cal and overall R when the parametric assumption fails.

Visualization of calibration curves We fix the calibration dataset and visualize the calibration curves for all methods under the two settings. Figure 3 shows that the binning methods (UWB and UMB) closely track the optimal recalibration function h ∗ in both settings. In contrast, the hybrid approach follows the Platt scaling estimates, leading to an inherent bias from h ∗ when the parametric assumption is invalid (Figure 3b).

E.3 Comparing recalibration schemes under label shift

We consider the label shift with source distribution D (0 . 5) and target distribution D ( π Q ) , where π Q varies in { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } . The results where π Q > 0 . 5 can be inferred by symmetry and hence not experimented. We vary n P in { 10 , 10 3 , 10 5 , 10 7 } and n Q in { 10 , 10 3 , 10 5 } .

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Probability

0.0

0.2

0.4

0.6

0.8

1.0

Empirical Frequency

h

*

Q

hP

h target

Q

g

hQ

0.0

0.2

0.4

0.6

0.8

1.0

Predicted Probability

z

0

1

2

3

4

5

Density

pQ

(

z

)

Aside from our proposed recalibration function ˆ h Q = ˆ g ◦ ˆ h P (17), referred to as COMPOSITE, we consider three other calibration approaches as baselines: (1) SOURCE, denoted as ˆ h P , which is only calibrated on the source data, (2) LABEL-SHIFT, denoted as ˆ g , which performs label shift correction without calibration, and (3) TARGET, denoted as ˆ g target Q , which is only calibrated on the target data. The number of bins B are chosen to be n 1 / 3 for COMPOSITE and SOURCE, and n 1 / 3 for TARGET.

P Q

Table 2 shows the risks for different approaches with π Q = 0 . 1 , n P = 10 3 , and n Q = 10 2 . In terms of R cal , COMPOSITE performs the best, as it is calibrated to the target distribution by taking advantage of the abundant source data. In terms of R sha , LABEL-SHIFT achieves R sha = 0 due to the strictly increasing ˆ g , but it suffers from high R cal . COMPOSITE and SOURCE achieve smaller R sha than TARGET, as a result of using more bins on a larger sample. Considering the combined impact of calibration and sharpness, our approach COMPOSITE attains the lowest overall recalibration risk R as well as MSE.

Figure 6 shows the optimal recalibration function h ∗ and the recalibration functions for the four approaches. It can be seen that COMPOSITE best estimates h ∗ with the highest resolution.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBackground:\nGenerating reliable probability estimates alongside accurate class labels is crucial in classification tasks. Calibration, which refers to the alignment between predicted probabilities and empirical frequencies of labels, is highly desirable in various applications. However, many machine learning algorithms lack inherent calibration.\n\nThis paper aims to address these issues in two ways:\n1. Development of a unified framework for recalibration that incorporates both calibration and sharpness in a principled manner.\n2. Proposal of a composite estimator for recalibration under label shift, which converges to the optimal recalibration and enables sample-efficient adaptation of classifiers to label-shifted domains.\n\nSTRENGTHS:\n1. This paper is exceptionally well-written, with a clear and logical structure, making it easily understandable to a wide audience.\n2. The theoretical foundation of this paper is solid, the proof process is very detailed, making it easy for people to follow.\n3. The algorithm proposed by the author is simple and efficient, making it easy to implement.\n\nWEAKNESSES:\n1. Although the author provides solid theoretical analysis, this paper lacks enough experiments to demonstrate the effectiveness of their method.\n2. The author employs three assumptions, and in my view, assumption 2 is a rather strong one, which is difficult to guarantee in practical applications.\n3. The experiments in this paper were conducted primarily on toy datasets, and the author do not provide any experiments on how to apply Recalibration in real-world applications.\n\nQUESTIONS:\n1. In Eq. (9), I understand that the function $\\hat{h}(z)$ calculates the expected label within the bin that $z$ belongs to. So, why is $\\hat{h}$ a monotonically increasing function of $z$ and its growth pattern similar to the cumulative distribution function (CDF) of $z$? Although you have made Assumption 2, can I understand that this assumption implies that $y$ must follow a specific distribution form in order to satisfy the monotonicity of $\\hat{h}$?\n2. In the experimental section, the author only presents a very simple toy scenario, and I am quite curious about the performance of the author's algorithm in more complex scenarios. For instance, in real-world classification datasets like cifar-10-long-tail or cifar-100-long-tail, how does the recalibration effect of more complex models, such as neural networks or decision trees?\n3. In Fig. 1 (a), as the value of $n$ increases, regardless of the magnitude of $B$, $R^{cal}(\\hat{h})$ exhibits a noticeable decrease. Does this imply that when the dataset size $n$ becomes sufficiently large, $R^{cal}(\\hat{h})$ has already reached a level where recalibration is unnecessary?\n\nIf the author can answer my question, I will change my score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a very relevant problem of post-hoc calibration in probabilistic classifiers. There is a great body of work on calibrating probabilistic classifiers so that the predicted probabilities match the empirical label frequencies in the popular machine learning literature. However, calibrating probabilistic classifiers so that they retain their sharpness / refinement is not studied much. The paper focuses on the post-hoc calibration with retained sharpness properties from first principles. Its entry point is the classical decomposition of mean squared error of the classifier into sharpness and (mis)calibration. Traditionally, both the measures were important, but the works in machine learning literature has mostly focussed on (mis)calibration only. There are important contributions in this paper: a) proposing recalibration risk measure which attains its minimum value only when both sharpness and the calibration requirements are met. b) uniform mass binning algorithm to approximate the optimal recalibration map. c) Theoretical results on the binning scheme and binning scheme to bound both the calibration risk and the sharpness risk. d) Formal result on the trade-off between calibration and sharpness. e) recalibration studies in a simple (but important) distribution shift with label shift. All of these results are for binary classification setting.\n\nSTRENGTHS:\n1. It is clear that the paper makes theoretical contributions to an important problem. It has been observed that major recalibration algorithms in machine learning literature not just ignore the sharpness measure, but they degrade it [1]. There is a trade-off between these quantities, and this paper formally state this with some actionable insights to balance this trade-off.\n2.  Theorem 1 generalises a previous result in the literature with an additional insight on the sharpness risk. Generalisations and connections to previous results is a sign of great work. \n3. While the results on label shift are easy consequences of the general recalibration results presented earlier in the paper, it is still great to see this extension. \n4. In my opinion, the paper is significant and relevant to the machine learning community. I believe the paper will spur more work on post-hoc calibration methods. \n5. The paper is excellently written, easy to read and understand.\n\n\n\n\n\n[1] Aditya Singh et al. On The Dark Side Of Calibration For Modern Neural Networks. (2021) (http://www.gatsby.ucl.ac.uk/~balaji/udl2021/accepted-papers/UDL2021-paper-074.pdf)\n\nWEAKNESSES:\nI do not have major concerns with this paper. I have some questions though (please see below):\n\nQUESTIONS:\n1. The optimal recalibration function is same as the canonical calibration map defined in [2] (Section 3.1). While it is clearly interesting to see that a canonical calibration map is the optimal recalibration map, could authors comment on this connection? The optimal recalibration map is certainly a calibrated function (as the Proposition 1 in [2] states), I am curious about some insights on why it would also be a map to minimise the sharpness risk.\n\n2. Additionally, [2] also propose a sort of general methodology to estimate this canonical calibration map (Section 4.1). While they do not comment on the partitioning scheme, it would be interesting to draw insights on general partitioning schemes and the bound on the recalibration risk. \n\n3. While the authors mention that extending the results presented in the paper to multi-class setting is a future research direction, could authors see that the estimator provided in [2] could be useful for multi class case (as [2] do not restrict itself to binary classification)?  \n\n\n[2]  Juozas Vaicenavicius et al. Evaluating model calibration in classification. (2019)\n\nLIMITATIONS:\nObviously, there are limitations to this work, as there are with any works. However, the authors have been very clear in stating them.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of minimum risk recalibration, utilizing Mean Squared Error (MSE) decomposition. The authors provide justification for their approach by demonstrating that minimizing the proposed risk yields simultaneous minimization of the calibration risk while preserving the sharpness of the probability forecaster. Furthermore, the authors employ the MSE decomposition to analyze the recalibration method known as UMB. Theoretical analysis reveals that selecting an appropriate number of bins allows for achieving a balance between the calibration risk and sharpness. Expanding on their findings, the authors apply their methodology to address the problem of recalibration in label shift. They provide theoretical and experimental validation for their approach in the context of label shift, further supporting the effectiveness of their proposed method.\n\nSTRENGTHS:\n1. The authors provide a rigorous introduction to key statistical measures essential for the recalibration task, such as recalibration/calibration risk and sharpness risk. They demonstrate that minimizing these measures can result in a well-calibrated forecaster.\n\n2. Building upon the introduced statistics and their properties, the authors enhance the analysis of UMB. They unveil a tradeoff between calibration risk and the preservation of sharpness by establishing a high-probability error bound. Furthermore, they leverage this upper bound to guide the selection of the number of bins in the UMB method, highlighting the practical significance of the proposed bound and decomposition.\n\n3. The authors extend the application of their proposed method to address the challenge of label shift, showcasing the validity of their work in practical downstream tasks.\n\n4. The paper thoroughly discusses the mildness of the assumptions made and experimentally validates the non-trivial nature of the proposed bounds and schemes.\n\nWEAKNESSES:\nWhile the outcome of this study is fruitful, it is encouraged that the authors delve into a further discussion regarding the potential extension of their framework to the multi-class scenario. This extension would provide additional intuitive information and enhance the applicability of their proposed methodology.\n\nQUESTIONS:\nPlease refer to the Weakness part.\n\nLIMITATIONS:\nThe authors have analyzed the limitation of the used method in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

comments: SUMMARY:
This work proposes a calibration method for probabilistic classifiers. It is known that most machine learning models produce predictions with high confidence that yield a distribution different than the underlying true label distribution. The proposed method focuses on calibration without a loss on the prediction performance and is claimed to adjust for label shift.

STRENGTHS:
- This paper is written clearly with good and easy-to-follow notations. The different definitions are well-placed and help the reader to follow the flow of the paper. Overall good structure of the paper. 
- The equal focus during the calibration on both calibration and sharpness goals is well-motivated and importantly this setting. The theory is developed rigorously and the different choices and assumptions are justified.
- This paper extends the proposed work to label-shfit setting and provides a good discussion on "recalibration under label shift" .
- This paper provides experimental results that show the validity of the theoretical work proposed.

WEAKNESSES:
I think this paper lacks the following two points:

- A discussion in section 1.1 highlighting to the reader how this paper approaches the recalibration problem differently than existing work. Only the work in [26] has been addressed.
- Section 6 does not include a discussion about the empirical results of previous works.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work looks at methods for calibrating probabilistic classifiers for Mean Squared Error by decomposing it into calibration error and sharpness errors. It gives finite sample error guarantees on both of these for the Uniform Mass Binning method. By balancing sharpness and calibration error, they also propose the optimal number of bins to use. Additionally, they look at the problem of calibrating classifiers in the case of label distribution shift between train and test. Their results show that transferring a calibrated classifier requires significantly fewer target samples compared to recalibrating from scratch. They validate our theoretical findings through numerical simulations.\n\nSTRENGTHS:\n- Existing works have looked at finite sample bounds for the calibration error but not the sharpness (as mentioned in this paper). This is the first method to look at both sharpness and calibration together which is an important criterion. \n- The paper is generally well written.\n\nWEAKNESSES:\n- The main contribution is to give risk bounds for sharpness along with the calibration error. In my opinion, this is not a significant contribution. There is not any particular strategy proposed using this analysis but only choosing the number of bins.\n- The second problem of using this method for label distribution shift also seems straightforward and I have also asked a related question below.\n\nQUESTIONS:\n- Why did the authors choose to analyze only Uniform Mass Binning and not other methods like Uniform width binning?\n- For the label distribution shift problem, is the proposed method equivalent to just scaling all the predicted probablities according to the target distribution and then using the uniform binning technique?\n- In definition 4, the sharpness risk of h over f, is this the sharpness risk of h.f - sharpness risk of f?\n- Why is Assumption 3 needed for the label distribution case?\n- Just to confirm, is this the first sharpness risk bound?\n- In figure 2a, why is the gap increasing between the theoretical and empirical bound as n increases?\n- In table 1, the different methods compared are not clear to me. Can the authors please explain?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}