SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison with the hybrid method [29]. Kumar et al. [29] proposed a hybrid recalcibration approach that involves fitting a recalibration function in a parametric family  $\mathcal{H}$ , which is then discretized using UMB. Note that the hypothesis class  $\mathcal{H}$  may or may not include the optimal recalibration function  $h_{f,P}^{*}$ . If  $h_{f,P}^{*} \in \mathcal{H}$ , using a similar analysis as above, we derive their high probability risk bound as  $O\left(\frac{1}{n}\log \frac{B}{\delta} + \frac{1}{B^2}\right)$  under Assumption (A3), which achieves  $\tilde{O}(n^{-1})$  when  $B^{*} \asymp \sqrt{n}$ , exhibiting a faster decay than our  $R(\hat{h}) = O(n^{-2/3})$ . While the faster rate is anticipated from employing parametric methods, we note that when  $h_{f,P}^{*} \notin \mathcal{H}$ , their method exhibits inherent bias (approximation error) induced by parametric function fitting, whereas our method is asymptotically unbiased. This distinction is corroborated by numerical simulations in Appendix E.2.
Proof sketch of Theorem 1. First, we demonstrate that the uniform-mass binning scheme  $\mathcal{B} = \{I_b\}_{b=1}^B$  satisfies two regularity conditions with high probability, when the sample size  $n$  is not too small. Specifically, we show that (i)  $\mathcal{B}$  is 2-well-balanced [29] with respect to  $f(X)$ , resulting in  $B$  bins having comparable probabilities (Lemma 3); and that (ii) the empirical mean in each bin of  $\mathcal{B}$  uniformly concentrates to the population conditional mean of  $Y$  conditioned on  $f(X)$  being contained within the bin (Lemma 4). Thereafter, we prove that if  $\mathcal{B}$  satisfies these two properties, then the calibration risk  $R^{\mathrm{cal}}$  and the sharpness risk  $R^{\mathrm{sha}}$  can be upper bounded as stated in Theorem 1; see Lemmas 5 and 6 (or Lemma 7 when (A3) holds). The detailed proof is in Appendix B.
5 Recalibration under label shift
This section extends the results from Section 4 to address label shift. In Section 5.1, we introduce the label shift assumption (Definition 7) and reframe the recalibration problem accordingly. We show that the optimal recalibration function in this context can be expressed as a composition of the optimal recalibration function (cf. Section 3) and a shift correction function. Building on this observation, we propose a two-stage estimator in Section 5.2, where each stage estimates one of the component functions. The composite estimator's overall performance is supported by theoretical guarantees.
5.1 Revisiting the problem formulation
Let  $P$  and  $Q$  denote the probability measures of the source and the target domains, respectively. We assume  $P$  and  $Q$  satisfy the label shift assumption defined below.
Definition 7 (Label shift). Probability measures  $P$  and  $Q$  are said to satisfy the label shift assumption if the following two conditions are satisfied:
(B1)  $P[X \in B \mid Y = k] = Q[X \in B \mid Y = k]$  for all  $k \in \{0, 1\}$  and all  $B \in \mathcal{B}(\mathcal{X})$ .
(B2)  $P[Y = 1] \in (0,1)$  and  $Q[Y = 1] \in (0,1)$ .
According to Condition (B1), the class conditional distributions remain the same, while the marginal distribution of the classes may change. Condition (B2) requires all classes to be present in the source population, which is a standard regularity assumption in the discussion of label shift [31, 14]; it also posits the presence of every class in the target population.
Optimal recalibration under label shift. Under the label shift assumption between  $P$  and  $Q$ , we define the label shift correction function  $g^{*}: \mathcal{Z} \to \mathcal{Z}$  such that
$$
g ^ {*} (z) = \frac {w _ {1} ^ {*} z}{w _ {1} ^ {*} z + w _ {0} ^ {*} (1 - z)} \quad \text {w h e r e} \quad w _ {k} ^ {*} = \frac {Q [ Y = k ]}{P [ Y = k ]}, \forall k \in \{0, 1 \}. \tag {13}
$$
The conditional probabilities under  $P$  and  $Q$  can be related [45] as follows:
$$
Q [ Y = 1 \mid X \in B ] = g ^ {*} \left(P [ Y = 1 \mid X \in B ]\right), \quad \forall B \in \mathcal {B} (\mathcal {X}). \tag {14}
$$
Recall that the optimal recalibration function for a predictor  $f: \mathcal{X} \to \mathcal{Z}$  under probability measure  $P$  is defined as  $h_{f,P}^{*}(z) = P[Y = 1 \mid f(X) = z]$ ; see (7). In the presence of a label shift between  $P$  and  $Q$ , we may write the optimal recalibration function for  $f$  under  $Q$  as
$$
h _ {f, Q} ^ {*} = g ^ {*} \circ h _ {f, P} ^ {*} \tag {15}
$$
because  $h_{f,Q}^{*}(z) \stackrel{(a)}{=} Q[Y = 1 \mid f(X) = z] \stackrel{(b)}{=} g^{*}\left(P[Y = 1 \mid f(X) = z]\right) \stackrel{(c)}{=} \left(g^{*} \circ h_{f,P}^{*}\right)(z)$ , where (a) and (c) follows from the definition of  $h_{f,P}^{*}$  and (b) is due to (14).
Recalling the definition of the risk  $R_{P}(h;f)$  from (5), we observe that  $R_{Q}(h_{f,Q}^{*};f) = 0$ , which is consistent with the risk characterization of the optimal recalibration. Our goal is to estimate the optimal recalibration function  $h_{f,Q}^{*} = g^{*}\circ h_{f,P}^{*}$  from data.
Problem 2 (Recalibration under label shift). Suppose that we have a measurable function  $f: \mathcal{X} \to \mathcal{Z}$  and two IID datasets  $\mathcal{D}_P = (x_i, y_i)_{i=1}^{n_P} \sim P$  and  $\mathcal{D}_Q = (x_i', y_i')_{i=1}^{n_Q} \sim Q$ . The goal of recalibration under label shift is to estimate  $\hat{h} \approx h_{f,Q}^*$  using  $f$ ,  $\mathcal{D}_P$  and  $\mathcal{D}_Q$ .
Remark 2. The source (training) dataset  $\mathcal{D}_P$  may not be accessible due to privacy protections, proprietary data, or practical constraints, as is often the case when recalibrating a pre-trained black box classifier to new data. In these cases, it suffices to have estimates of the recalibration function  $h_{f,P}^{*}$  and the marginal probabilities  $P[Y = k]$ ,  $k\in \{0,1\}$  under  $P$ , for our method and analysis.
5.2 Two-stage recalibration under label shift
Method. We propose a composite estimator of  $h_{f,Q}^{*} = g^{*} \circ h_{f,P}^{*}$ , which comprises two estimators  $\hat{g} \approx g^{*}$  and  $\hat{h}_P \approx h_{f,P}^*$ . Here we describe a procedure to produce this composite estimator.
1. Use  $\mathcal{D}_P$  to construct  $\hat{h}_P: \mathcal{Z} \to \mathcal{Z}$ , the estimated recalibration function (9) (for  $f$  under  $P$ ).
2. Use  $\mathcal{D}_P$  and  $\mathcal{D}_Q$  to construct  $\hat{g}:\mathcal{Z}\to \mathcal{Z}$  such that
$$
\hat {g} (z) = \frac {\hat {w} _ {1} z}{\hat {w} _ {1} z + \hat {w} _ {0} (1 - z)} \quad \text {w h e r e} \quad \hat {w} _ {k} = \frac {\hat {Q} [ Y = k ]}{\hat {P} [ Y = k ]}, \forall k \in \{0, 1 \}, \tag {16}
$$
where  $\hat{P}[Y = k] := \frac{1}{|\mathcal{D}_P|} \sum_{i=1}^{|\mathcal{D}_P|} \mathbb{1}[y_i = k]$  and  $\hat{Q}[Y = k] := \frac{1}{|\mathcal{D}_Q|} \sum_{i=1}^{|\mathcal{D}_Q|} \mathbb{1}[y_i' = k]$  are the empirical estimates of the class marginal probabilities.
3. Let
$$
\hat {h} _ {Q} = \hat {g} \circ \hat {h} _ {P}. \tag {17}
$$
Note that the recalibration estimator  $\hat{h}_P$  (Step 1) remains the same with that in Section 4.1. Furthermore, the shift correction estimator  $\hat{g}$  (Step 2) is a plug-in estimator of the label shift correction function  $g^{*}$  in (14) based on the estimated weights,  $\hat{w}_{1}$  and  $\hat{w}_{0}$ .
Theory. We present a recalibration risk upper bound for the proposed two-stage estimator. We let  $p_k \coloneqq P[Y = k]$ ,  $q_k \coloneqq Q[Y = k]$ , and  $w_k^* = \frac{q_k}{p_k}$  for  $k \in \{0,1\}$ . Moreover, we let  $p_{\min} \coloneqq \min_k p_k$ ,  $q_{\min} \coloneqq \min_k p_k$ ,  $w_{\min}^* \coloneqq \min_k w_k^*$  and  $w_{\max}^* \coloneqq \max_k w_k^*$ .
Theorem 2 (Convergence of  $\hat{h}_Q$ ). Let  $P, Q$  be probability measures and let  $f: \mathcal{X} \to \mathcal{Z}$  be a measurable function. Let  $\mathcal{D}_P \sim P$  be an IID sample of size  $n_P$  and  $\mathcal{D}_Q \sim Q$  be an IID sample of size  $n_Q$ . Suppose that Assumptions (B1) & (B2) hold. Let  $\mathcal{B}$  be the UMB scheme induced by  $\mathcal{D}_P$ . Let  $\hat{h}_P = \hat{h}_{P,\mathcal{B}}$  be the recalibration function (9) based on  $\mathcal{B}$ , and let  $\hat{g}$  denote the shift correction function as defined in (16). Then
$$
R _ {Q} (\hat {g} \circ \hat {h} _ {P}) \leq 2 \left\{\left(\frac {\rho_ {0} - \rho_ {1}}{\rho_ {0} + \rho_ {1}}\right) ^ {2} + \frac {w _ {\max } ^ {*}}{w _ {\min } ^ {* 2}} \cdot R _ {P} (\hat {h} _ {P}; f) \right\} \quad \text {w h e r e} \quad \rho_ {k} := \frac {\hat {w} _ {k}}{w _ {k} ^ {*}}, k \in \{0, 1 \}. \tag {18}
$$
Furthermore, suppose that (A1), (A2) & (A3) hold. Then there exists a universal constant  $c > 0$  such that for any  $\delta \in (0,1)$ , if
$$
n _ {P} \geq \max  \left\{c, \frac {2 7}{p _ {\operatorname* {m i n}}} \right\} \cdot | \mathcal {B} | \log \left(\frac {4 | \mathcal {B} |}{\delta}\right) \quad a n d \quad n _ {Q} \geq \frac {2 7}{q _ {\operatorname* {m i n}}} \log \left(\frac {1 6}{\delta}\right),
$$
then with probability at least  $1 - \delta$
$$
\begin{array}{l} R _ {Q} \left(\hat {g} \circ \hat {h} _ {P}\right) \leq 2 \frac {w _ {\operatorname* {m a x}} ^ {*}}{w _ {\operatorname* {m i n}} ^ {*}} ^ {3} \cdot \left\{\left(\sqrt {\frac {1}{2 \left(\lfloor n _ {P} / | \mathcal {B} | \rfloor - 1\right)} \log \left(\frac {8 | \mathcal {B} |}{\delta}\right)} + \frac {1}{\lfloor n _ {P} / | \mathcal {B} | \rfloor}\right) ^ {2} + \frac {8 K ^ {2}}{| \mathcal {B} | ^ {2}} \right\} \tag {19} \\ + 5 4 \max  \left\{\frac {1}{p _ {\operatorname* {m i n}} \cdot n _ {P}}, \frac {1}{q _ {\operatorname* {m i n}} \cdot n _ {Q}} \right\} \cdot \log \left(\frac {1 6}{\delta}\right). \\ \end{array}
$$
Remark 3. Note that  $\rho_{k} \to 1$  as  $n_{P}, n_{Q} \to \infty$ , and thus, the upper bound (18) reduces to  $2\frac{w_{\mathrm{max}}^{*}}{w_{\mathrm{min}}^{*}}\cdot R_{P}\left(\hat{h}_{P};f\right)$ . Moreover, when  $P = Q$ , we have  $w_{\mathrm{min}}^{*} = w_{\mathrm{max}}^{*} = 1$ , and this further simplifies to the recalibration risk without label shift, up to multiplicative constant 2.
Remark 4 (Target sample complexity). Assume that  $n_P \geq n_Q$  and the number of bins satisfies  $|\mathcal{B}| \asymp n_P^{1/3}$ . Then (19) implies  $R_Q\big(\hat{g} \circ \hat{h}_P; f\big) = O\big(n_P^{-2/3} + n_Q^{-1}\big)$ . This result indicates that the proposed recalibration method using (17) requires a significantly smaller target sample size  $n_Q = \Omega(\varepsilon^{-1})$  to control the risk, as compared to  $n_Q = \Omega(\varepsilon^{-3/2})$  in (12).
Remark 5 (Comparison with label shift using unlabeled target data). When the source sample size  $n_P$  is sufficiently large, we achieve a risk of  $R_{Q}(\hat{g} \circ \hat{h}_{P}) = O(n_{Q}^{-1})$  with high probability. It is important to note that in this scenario, we only utilize the labels from the target sample to address label shift. Remarkably, the same rate applies when employing the algorithms proposed in [31, 3, 14], which solely rely on features from the target sample. For a proof sketch, please refer to Appendix D.
In this section, we present the results of our numerical simulations conducted to validate and reinforce the theoretical findings discussed earlier. The simulations are based on a family of recalibration functions called beta calibration [28]:  $\mathcal{H}_{\mathrm{beta}} = \{h_{\mathrm{beta}}(z;a,b,c):a\geq 0,b\geq 0,c\in \mathbb{R}\}$ , where  $h_\mathrm{beta}(\cdot ;a,b,c):[0,1]\to [0,1]$  is defined as
$$
h _ {\text {b e t a}} (z; a, b, c) = \frac {1}{1 + 1 / \left(e ^ {c} \frac {z ^ {a}}{(1 - z) ^ {b}}\right)}. \tag {20}
$$
In addition, consider the joint distributions  $\mathcal{D}(\pi)$  of  $X$  and  $Y$ , where  $Y \sim \text{Bernoulli}(\pi)$ ,  $X \mid Y = 0 \sim N(-2,1)$ , and  $X \mid Y = 1 \sim N(2,1)$ , and a pre-trained probabilistic classifier  $f(x) = \sigma(x) := 1/(1 + e^{-x})$ . To accommodate the limitations of space, we summarize the results in Figure 1, 2, 3, and Table 1, 2, providing a concise overview. Detailed information about the simulation settings, implementation details, and further experimental results and discussions can be found in Appendix E. Our simulation code is available at https://github.com/ZeyuSun/calibration_labelhift.
(a)  $R^{\mathrm{cal}}(\hat{h})$  vs.  $n$
(b)  $R^{\mathrm{cal}}(\hat{h})$  vs.  $B$
(c)  $R^{\mathrm{sha}}(\hat{h})$  vs.  $B$
(d)  $R(\hat{h})$  vs.  $B$
Figure 2: The optimal number of bins  $B$  for different sample sizes  $n$  plotted in linear scale (left) and log scale (middle), and the population risk for various combinations of  $n$  and  $B$ , with the optimal  $B$  marked by black dots (right). Note that the risk surface is relatively smooth around its minimum, suggesting the robustness of the optimal  $B$ .
Figure 1: Medians (solid lines) and 10-90 percentile ranges (shaded areas) of quadrature estimates of population risks over 10 realizations and theoretical risk upper bounds ( $\delta = 0.1$ ) (dashed lines) for various  $n$  and  $B$ . (a)-(c) The empirical rates,  $R^{\mathrm{cal}} = O(n^{-0.99}B^{0.98})$  and  $R^{\mathrm{sha}} = O(B^{-1.83})$ , align with theoretically predicted rates,  $\tilde{O}(B/n)$  and  $O(B^{-2})$ , in Thm. 1. (d) The empirically observed  $R$  and our upper bound exhibit similar trends as a function of  $B$ .
Table 1:  $90\%$  -quantiles of the risks of Platt scaling [40], the hybrid method [29], uniform-width binning (UWB) [18], and uniform-mass binning (UMB) over 100 random calibration datasets drawn from  $Z \sim$  Uniform[0,1], and (a)  $Y \mid Z \sim$  Bernoulli  $(h_{\mathrm{beta}}(z;4,4,0))$ , or (b)  $Y \mid Z \sim$  Bernoulli  $(h_{\mathrm{beta}}(z;0.1,4,0))$ . While Platt scaling and the hybrid method achieve lower  $R$  under the correct parametric assumption, UWB and UMB may outperform when the parametric assumption fails.
(a) Correct parametric assumption
Metric (×10-3)	Rcal	Rsha	R	MSE
Platt	0.122	0.000	0.122	10.338
Hybrid	0.119	0.212	0.315	10.532
UWB	0.661	0.194	0.855	11.071
UMB	0.647	0.212	0.839	11.055
(b) Misspecified parametric assumption
Metric (×10-3)	Rcal	Rsha	R	MSE
Platt	3.682	0.000	3.682	21.994
Hybrid	3.117	0.251	3.360	21.672
UWB	0.572	0.238	0.810	19.122
UMB	0.560	0.251	0.797	19.109
Table 2: Risks under label shift from  $\mathcal{D}(0.5)$  to  $\mathcal{D}(0.1)$ , with  $n_P = 10^3$  and  $n_Q = 10^2$ . Standard deviations are computed from 10 random realizations. LABEL-SHIFT, only applying an injective  $\hat{g}$ , achieves  $R^{\mathrm{sha}} = 0$  but incurs high  $R^{\mathrm{cal}}$ . SOURCE, recalibrated with  $B = n_P^{1/3}$  on  $\mathcal{D}_P$ , incurs high  $R^{\mathrm{cal}}$ . TARGET, recalibrated with  $B = n_Q^{1/3}$  on  $\mathcal{D}_Q$ , incurs low  $R^{\mathrm{cal}}$ . Our proposed COMPOSITE achieves the lowest nonzero  $R^{\mathrm{sha}}$  and lowest  $R^{\mathrm{cal}}$ .
Method	Estimator	Rcal	Rsha	R	MSE
SOURCE	hP	0.016±0.005	0.0032±0.0014	0.019±0.006	0.029±0.006
TARGET	hQtarget	0.0020±0.0025	0.049±0.006	0.051±0.006	0.060±0.006
LABEL-SHIFT	g	0.026±0.006	0	0.026±0.006	0.035±0.006
COMPOSITE	hQ	0.00019±0.00017	0.0032±0.0014	0.0034±0.0013	0.0127±0.0013
(a) Correct parametric assumption
(b) Misspecified parametric assumption
Figure 3: Optimal recalibration function  $h^*$  and recalibration function estimates by Platt Scaling [40], the hybrid method [29], UWB, and UMB when the parametric assumption is (a) correct and (b) misspecified. UMB traces the  $h^*$  in both cases, whereas the hybrid method traces Platt scaling, exhibiting an intrinsic bias from  $h^*$  in (b).
This paper presents a comprehensive theory for recalibration, considering both calibration and sharpness within the mean-squared-error (MSE) decomposition framework. We use this framework to quantify the optimal calibration-sharpness balance and establish a rigorous upper bound on the finite-sample risk for uniform-mass binning (UMB). Additionally, we address the challenge of recalibration under label shift with limited access to labeled target data. Our proposed two-stage approach effectively estimates the recalibration function using ample data from the source domain and adjusts for the label shift using target domain data. Importantly, our findings suggest that transferring a calibrated classifier requires a significantly smaller target sample than recalibrating from scratch on the new domain. Numerical simulations confirm the tightness of the finite sample bounds, validate the optimal number of bins, and demonstrate the effectiveness of the label shift adaptation.
In concluding this paper, we identify several promising directions for future research.
Relaxation of the assumptions It would be worthwhile to explore whether or not the assumptions made in our analysis could be relaxed. For instance, the widely adopted monotonicity assumption (A2) and its variants [51, 52, 43, 44] may not hold in real-world settings. Thus, exploring potential relaxations of this assumption is valuable. In addition, a mild but non-trivial smoothness assumption (A3) (c.f. Remark 7 and 8) is introduced to obtain a tight sharpness risk upper bound (c.f. Remark 9); investigating its practical implications and exploring potential relaxations could be interesting future work.
Calibration-sharpness framework analysis Applying our framework to analyze recalibration methods beyond UMB, such as isotonic regression [51] and kernel density estimation [52], can offer further insights into their performance and properties, providing guidance to practitioners in selecting suitable algorithms based on specific conditions and requirements.
Multiclass probability recalibration The concept of calibration considered in this work extends to multi-class classification settings, known as canonical calibration [48]. Weaker, but more tractable notions of multi-class calibration have also been explored in the literature [48, 21]. While partition-based methods, as multi-class extensions of binning methods, are known to have consistency [48] and vanishing calibration error [41], establishing upper bounds for their sharpness risk remains difficult. Additionally, designing a partition scheme in a multidimensional space is a challenging task [21]; the interplay between calibration and sharpness could potentially guide the development of partition strategies that balance both aspects in multi-class classification.
Applications to real-world data Investigating the calibration-sharpness tradeoff in real-world applications, which often involve multiple classes, presents an interesting challenge. It is crucial to develop effective estimators for both calibration risk and sharpness risk in such scenarios. While estimators for calibration risk exist (e.g., binning-based and KDE-based) [6, 29, 52, 42] and a lower bound for sharpness risk has been established [39], a direct estimator of sharpness risk is still lacking.
The research in this paper was partially supported by ARO grants W911NF-23-1-0343 and W911NF-19-1-0269 and by DOE grant DE-NA0003921.
[1] John Aitchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-272, 1980.
[2] Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning, pages 222-232. PMLR, 2020.
[3] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2018.
[4] Jarosław Błasiok, Parikshit Gopalan, Lunjia Hu, and Preetum Nakkiran. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, pages 1727-1740, 2023.
[5] Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950.
[6] Jochen Bröcker. Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. *Climate dynamics*, 39:655–667, 2012.
[7] Dallas Card and Noah A Smith. The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1636-1646, 2018.
[8] Misty D Crown. Validation of the NOAA Space Weather Prediction Center's solar flare forecasting look-up table and forecaster-issued probabilities. *Space Weather*, 10(6), 2012.
[9] A Philip Dawid. The well-calibrated Bayesian. Journal of the American Statistical Association, 77(379):605-610, 1982.
[10] Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12-22, 1983.
[11] Shrey Desai and Greg Durrett. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295-302, 2020.
[12] Charles Elkan. The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence, volume 17, pages 973-978, 2001.
[13] Andrea Esuli, Alessio Molinari, and Fabrizio Sebastiani. A critical reassessment of the Saerens-Latinne-Decaestecker algorithm for posterior probability adjustment. ACM Transactions on Information Systems (TOIS), 39(2):1-34, 2020.
[14] Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 33:3290-3300, 2020.
[15] Tilmann Gneiting and Adrian E Raftery. Weather forecasting with ensemble methods. Science, 310(5746):248-249, 2005.
[16] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-378, 2007.
[52] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International Conference on Machine Learning, pages 11117-11128. PMLR, 2020.
[53] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR, 2021.
Proof of Proposition 1. First of all, we recall the definition of the two risks from (3) and (4):
$$
R ^ {\mathrm {c a l}} (h; f) = \mathbb {E} \left[ \left(f (X) - \mathbb {E} \left[ Y \mid f (X) \right]\right) ^ {2} \right]
$$
$$
R ^ {\mathrm {s h a}} (h; f) = \mathbb {E} \left[ \left(\mathbb {E} \left[ Y \mid h \circ f (X) \right] - \mathbb {E} \left[ Y \mid f (X) \right]\right) ^ {2} \right].
$$
To keep our notation concise, we use  $Z = f(X)$  as a shorthand notation, and also let  $Y_Z \coloneqq \mathbb{E}[Y \mid Z]$  and  $Y_{h(Z)} = \mathbb{E}[Y \mid h(Z)]$  throughout this proof. We can decompose the recalibration risk from Definition 5:
$$
\begin{array}{l} R (h) = \mathbb {E} [ (h (Z) - Y _ {Z}) ^ {2} ] \\ = \mathbb {E} \left[ \left(h (Z) - Y _ {h (Z)} + Y _ {h (Z)} - Y _ {Z}\right) ^ {2} \right] \\ = \mathbb {E} [ (h (Z) - Y _ {h (Z)}) ^ {2} ] + \mathbb {E} [ (Y _ {h (Z)} - Y _ {Z}) ^ {2} ] + 2 \mathbb {E} [ (h (Z) - Y _ {h (Z)}) (Y _ {h (Z)} - Y _ {Z}) ] \\ = \mathbb {E} [ (h (Z) - Y _ {h (Z)}) ^ {2} ] + \mathbb {E} [ (Y _ {h (Z)} - Y _ {Z}) ^ {2} ] + 2 \mathbb {E} [ \mathbb {E} [ (h (Z) - Y _ {h (Z)}) (Y _ {h (Z)} - Y _ {Z}) \mid h (Z) ] ] \\ = \mathbb {E} [ (h (Z) - Y _ {h (Z)}) ^ {2} ] + \mathbb {E} [ (Y _ {h (Z)} - Y _ {Z}) ^ {2} ] + 2 \mathbb {E} [ (h (Z) - Y _ {h (Z)}) (Y _ {h (Z)} - \mathbb {E} [ Y _ {Z} \mid h (Z) ]) ] \\ = \mathbb {E} [ (h (Z) - Y _ {h (Z)}) ^ {2} ] + \mathbb {E} [ (Y _ {h (Z)} - Y _ {Z}) ^ {2} ] + 2 \mathbb {E} [ (h (Z) - Y _ {h (Z)}) (Y _ {h (Z)} - Y _ {h (Z)}) ] \\ = \underbrace {\mathbb {E} [ (h (Z) - Y _ {h (Z)}) ^ {2} ]} _ {R ^ {\operatorname {c a l}} (h)} + \underbrace {\mathbb {E} [ (Y _ {h (Z)} - Y _ {Z}) ^ {2} ]} _ {R ^ {\operatorname {s h a}} (h)}. \\ \end{array}
$$