SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} n _ {P} \geq \frac {3}{\varepsilon^ {2} \min  \{p _ {0} , p _ {1} \}} \log \left(\frac {8}{\delta}\right) \text {a n d} n _ {Q} \geq \frac {3}{\varepsilon^ {2} \min  \{q _ {0} , q _ {1} \}} \log \left(\frac {8}{\delta}\right) \\ \Longrightarrow \qquad P \left(\max  _ {k \in \{0, 1 \}} \frac {\left| \hat {p} _ {k} - p _ {k} \right|}{p _ {k}} > \varepsilon \text {o r} \max  _ {k \in \{0, 1 \}} \frac {\left| \hat {q} _ {k} - q _ {k} \right|}{q _ {k}} > \varepsilon\right) \leq \delta \\ \Rightarrow \quad P \left(\max  _ {k \in \{0, 1 \}} \rho_ {k} > \frac {1 + \varepsilon}{1 - \varepsilon} \text {o r} \min  _ {k \in \{0, 1 \}} \rho_ {k} <   \frac {1 - \varepsilon}{1 + \varepsilon}\right) \leq \delta \\ \Rightarrow \qquad P \left(\max  _ {k \in \{0, 1 \}} \rho_ {k} > \beta \text {o r} \min  _ {k \in \{0, 1 \}} \rho_ {k} <   \frac {1}{\beta}\right) \leq \delta . \\ \end{array}
$$
C.1.2 Regularity of the Shift Correction Function
Lemma 10. Let  $w = (w_0, w_1) \in \mathbb{R}^2$  such that  $w_0, w_1 > 0$  and  $w_0 + w_1 = 1$ . The function  $g_w : [0,1] \to [0,1]$  such that  $g_w(z) = \frac{w_1z}{w_1z + w_0(1 - z)}$  is L-Lipschitz where  $L = \max \left\{\frac{w_1}{w_0}, \frac{w_0}{w_1}\right\}$ .
Proof of Lemma 10. First of all, consider the first-order derivative of  $g_{w}$ :
$$
\frac {d}{d z} g _ {w} (z) = \frac {w _ {1} \cdot [ w _ {1} z + w _ {0} (1 - z) ] - w _ {1} z \cdot (w _ {1} - w _ {0})}{[ w _ {1} z + w _ {0} (1 - z) ] ^ {2}} = \frac {w _ {1} w _ {0}}{[ w _ {1} z + w _ {0} (1 - z) ] ^ {2}}.
$$
We observe that  $g_{w}$  is monotone increasing as  $\frac{d}{dz} g_{w}(z) > 0$  for all  $z \in [0,1]$ . Next, we consider the second-order derivative of  $g_{w}$ :
$$
\frac {d ^ {2}}{d z ^ {2}} g _ {w} (z) = \frac {2 w _ {0} w _ {1} \cdot (w _ {0} - w _ {1})}{\left[ w _ {1} z + w _ {0} (1 - z) \right] ^ {3}} \left\{ \begin{array}{l l} > 0, \forall z \in [ 0, 1 ] & \text {i f} w _ {0} > w _ {1}, \\ = 0, \forall z \in [ 0, 1 ] & \text {i f} w _ {0} = w _ {1}, \\ <   0, \forall z \in [ 0, 1 ] & \text {i f} w _ {0} <   w _ {1}. \end{array} \right.
$$
$$
\sup_{z\in [0,1]}\frac{d}{dz} g_{w}(z) = \left\{ \begin{array}{ll}\frac{d}{dz} g_{w}(z)\bigg|_{z = 1} = \frac{w_{0}}{w_{1}} & \text{if} w_{0} > w_{1},\\ \frac{d}{dz} g_{w}(z)\bigg|_{z = 0} = \frac{w_{1}}{w_{0}} & \text{if} w_{0}\leq w_{1}. \end{array} \right.
$$
Lemma 11. Let  $P, Q$  be joint distributions of  $(X, Y) \in \mathcal{X} \times \{0, 1\}$ , and let  $w_k = \frac{P[Y=k]}{Q[Y=k]}$  for  $k \in \{0, 1\}$ . If  $P, Q$  satisfy the label shift assumption (Definition 7), i.e., if Assumptions (B1) and (B2) hold, then for any measurable function  $f: \mathcal{X} \to \mathbb{R}$ , the following two-sided inequality holds:
$$
\min  _ {k \in \{0, 1 \}} w _ {k} \leq \frac {\mathbb {E} _ {Q} [ f (X) ]}{\mathbb {E} _ {P} [ f (X) ]} \leq \max  _ {k \in \{0, 1 \}} w _ {k}. \tag {30}
$$
Proof of Lemma 11. First of all, we observe that
$$
\mathbb {E} _ {Q} \left[ f (X) \right] = \mathbb {E} _ {Q} \left[ \mathbb {E} _ {Q} \left[ f (X) \mid Y \right] \right]
$$
by the law of total expectation
$$
\begin{array}{l} = \sum_ {k = 0} ^ {1} P _ {Q} [ Y = k ] \cdot \mathbb {E} _ {Q} [ f (X) \mid Y ] \\ = \sum_ {k = 0} ^ {1} \left(w _ {k} \cdot P _ {P} [ Y = k ]\right) \cdot \mathbb {E} _ {P} \left[ f (X) \mid Y \right]. \quad \text {b y} \\ \end{array}
$$
Thus, it follows that  $\min_k w_k\cdot \mathbb{E}_P[f(X)]\leq \mathbb{E}_Q[f(X)]\leq \max_k w_k\cdot \mathbb{E}_P[f(X)].$
C.2 Completing the proof of Theorem 2
Proof of Theorem 2. This proof is presented in four steps. In Step 1, we establish a simple upper bound for the risk  $R_{Q}(\hat{h}_{Q};f)$  that consists of two error terms: the first term quantifies the error introduced by the estimated label shift correction,  $\hat{g}$ , while the second term quantifies the error due to the estimated recalibration function,  $\hat{h}_P$ . In Steps 2 and 3, we derive separate upper bounds for these two error terms. Finally, in Step 4, we combine the results from Steps 1-3 to obtain a comprehensive upper bound for  $R_{Q}$ , which concludes the proof.
Step 1. Decomposition of  $R_{Q}$ . Recalling the definition of the risk  $R_{Q}$ , cf. (5), we obtain the following inequality:
$$
\begin{array}{l} R _ {Q} (\hat {h} _ {Q}; f) = \mathbb {E} _ {Q} \left[ \left(\hat {h} _ {Q} \circ f (X) - \mathbb {E} _ {Q} [ Y | f (X) ]\right) ^ {2} \right] \\ = \mathbb {E} _ {Q} \left[ \left(\hat {g} \circ \hat {h} _ {P} \circ f (X) - g ^ {*} \circ \hat {h} _ {P} \circ f (X) + g ^ {*} \circ \hat {h} _ {P} \circ f (X) - \mathbb {E} _ {Q} [ Y | f (X) ]\right) ^ {2} \right] \\ \stackrel {(a)} {\leq} 2 \cdot \left\{\underbrace {\mathbb {E} _ {Q} \left[ \left(\hat {g} \circ \hat {h} _ {P} \circ f (X) - g ^ {*} \circ \hat {h} _ {P} \circ f (X)\right) ^ {2} \right]} _ {=: T _ {1}} \right. (31) \\ + \underbrace {\mathbb {E} _ {Q} \left[ \left(g ^ {*} \circ \hat {h} _ {P} \circ f (X) - \mathbb {E} _ {Q} [ Y | f (X) ]\right) ^ {2} \right]} _ {=: T _ {2}} \Bigg \}, (32) \\ \end{array}
$$
where (a) follows from the simple inequality  $(a + b)^2\leq 2(a^2 +b^2)$  for all  $a,b\in \mathbb{R}$
In Step 2 and Step 3 of this proof, we establish separate upper bounds for the two terms,  $T_{1}, T_{2}$ .
Step 2. An upper bound for  $T_{1}$ . Recall from (13) and (16) that
$$
g ^ {*} (z) = \frac {w _ {1} ^ {*} z}{w _ {1} ^ {*} z + w _ {0} ^ {*} (1 - z)} \qquad \text {w h e r e} \qquad w _ {k} ^ {*} = \frac {Q [ Y = k ]}{P [ Y = k ]}, \forall k \in \{0, 1 \},
$$
$$
\hat {g} (z) = \frac {\hat {w} _ {1} z}{\hat {w} _ {1} z + \hat {w} _ {0} (1 - z)} \qquad \text {w h e r e} \qquad \hat {w} _ {k} = \frac {\hat {Q} [ Y = k ]}{\hat {P} [ Y = k ]}, \forall k \in \{0, 1 \}.
$$
$$
\rho_ {0} := \frac {\hat {w} _ {0}}{w _ {0} ^ {*}} \quad \text {a n d} \quad \rho_ {1} := \frac {\hat {w} _ {1}}{w _ {1} ^ {*}}. \tag {33}
$$
Then we observe that for any  $z\in (0,1)$
$$
\begin{array}{l} | \hat {g} (z) - g ^ {*} (z) | = \left| \frac {\hat {w} _ {1} z}{\hat {w} _ {1} z + \hat {w} _ {0} (1 - z)} - \frac {w _ {1} ^ {*} z}{w _ {1} ^ {*} z + w _ {0} ^ {*} (1 - z)} \right| \\ = \left| \frac {\left(\hat {w} _ {1} w _ {0} ^ {*} - w _ {1} ^ {*} \hat {w} _ {0}\right) \cdot z (1 - z)}{\left[ \hat {w} _ {1} z + \hat {w} _ {0} (1 - z) \right] \cdot \left[ w _ {1} ^ {*} z + w _ {0} ^ {*} (1 - z) \right]} \right| \\ \leq \left| \frac {\left(\hat {w} _ {1} w _ {0} ^ {*} - w _ {1} ^ {*} \hat {w} _ {0}\right) \cdot z (1 - z)}{\left(\hat {w} _ {1} w _ {0} ^ {*} + w _ {1} ^ {*} \hat {w} _ {0}\right) \cdot z (1 - z)} \right| \\ = \left| \begin{array}{c} \hat {w} _ {1} w _ {0} ^ {*} - w _ {1} ^ {*} \hat {w} _ {0} \\ \hat {w} _ {1} w _ {0} ^ {*} + w _ {1} ^ {*} \hat {w} _ {0} \end{array} \right| \\ = \frac {\left| \rho_ {0} - \rho_ {1} \right|}{\rho_ {0} + \rho_ {1}}. \\ \end{array}
$$
Moreover,  $\hat{g} (0) = g^{*}(0) = 0$  and  $\hat{g} (1) = g^{*}(1) = 1$ . Letting  $Z_{\hat{h}}\coloneqq \hat{h}_P\circ f(X)$ , we obtain
$$
T _ {1} = \mathbb {E} _ {Q} \left[ \left(\hat {g} \left(Z _ {\hat {h}}\right) - g ^ {*} \left(Z _ {\hat {h}}\right)\right) ^ {2} \right] \leq \left(\frac {\rho_ {0} - \rho_ {1}}{\rho_ {0} + \rho_ {1}}\right) ^ {2}. \tag {34}
$$
It remains to establish probabilistic tail bounds for  $\rho_0, \rho_1$ , which we will accomplish in Step 4 of this proof.
Step 3. An upper bound for  $T_{2}$ . We observe that
$$
\begin{array}{l} T _ {2} = \mathbb {E} _ {Q} \left[ \left(g ^ {*} \circ \hat {h} _ {P} \circ f (X) - \mathbb {E} _ {Q} [ Y \mid f (X) ]\right) ^ {2} \right] \\ = \mathbb {E} _ {Q} \left[ \left(g ^ {*} \circ \hat {h} _ {P} \circ f (X) - g ^ {*} \left(\mathbb {E} _ {P} [ Y \mid f (X) ]\right)\right) ^ {2} \right] \quad \because \text {L a b e l s h i f t a s s u m p t i o n , c f .} \tag {14} \\ \leq \left(\frac {w _ {\operatorname* {m a x}} ^ {*}}{w _ {\operatorname* {m i n}} ^ {*}}\right) ^ {2} \cdot \mathbb {E} _ {Q} \left[ \left(\hat {h} _ {P} \circ f (X) - \mathbb {E} _ {P} [ Y \mid f (X) ]\right) ^ {2} \right] \quad \because g ^ {*} \text {i s} \frac {w _ {\operatorname* {m a x}} ^ {*}}{w _ {\operatorname* {m i n}} ^ {*}} \text {- L i p s c h i t z , c f . L e m m a 1 0} \\ \leq \left(\frac {w _ {\operatorname* {m a x}} ^ {*}}{w _ {\operatorname* {m i n}} ^ {*}}\right) ^ {2} \cdot w _ {\operatorname* {m a x}} ^ {*} \cdot \mathbb {E} _ {P} \left[ \left(\hat {h} _ {P} \circ f (X) - \mathbb {E} _ {P} [ Y \mid f (X) ]\right) ^ {2} \right] \quad \because \text {b y L e m m a 1 1} \\ = \frac {w _ {\operatorname* {m a x}} ^ {* 3}}{w _ {\operatorname* {m i n}} ^ {* 2}} \cdot R _ {P} \big (\hat {h} _ {P}; f \big). \\ \end{array}
$$
Step 4. Concluding the proof. For given  $\delta \in (0,1)$ , let  $^2\delta_1 = \delta_2 = \delta /4$  and  $\delta_{3} = \delta /2$ . We observe that
$$
\begin{array}{l l l l} n _ {P} \geq c ^ {\prime} \cdot | \mathcal {B} | \log \left(\frac {| \mathcal {B} |}{\delta_ {1}}\right) & \Longrightarrow & P \left[ \Phi_ {\text {b a l a n c e}} (\mathcal {B}, 2) = 1 \right] \geq 1 - \delta_ {1} & \text {b y L e m m a 3} \\ n _ {P} \geq 2 | \mathcal {B} | & \Longrightarrow & P \left[ \Phi_ {\text {a p p r o x}} (\mathcal {B}, \varepsilon_ {\delta_ {2}}) = 1 \right] \geq 1 - \delta_ {2} & \text {b y L e m m a 4} \end{array}
$$
Furthermore, assuming
$$
n _ {P} \geq \frac {2 7}{\min  \left\{p _ {0} , p _ {1} \right\}} \log \left(\frac {8}{\delta_ {3}}\right) \quad \text {a n d} \quad n _ {Q} \geq \frac {2 7}{\min  \left\{q _ {0} , q _ {1} \right\}} \log \left(\frac {8}{\delta_ {3}}\right),
$$
we may define  $\beta_{\delta_3}$  as a function of  $n_P, n_Q$  and  $\delta_3$  such that
$$
\beta_ {\delta_ {3}} = \beta_ {\delta_ {3}} \left(n _ {P}, n _ {Q}\right) := 1 + \sqrt {\max  \left\{\frac {1}{n _ {P} \cdot \min  \left\{p _ {0} , p _ {1} \right\}}, \frac {1}{n _ {Q} \cdot \min  \left\{q _ {0} , q _ {1} \right\}} \right\} \cdot 2 7 \log \left(\frac {8}{\delta_ {3}}\right)}. \tag {35}
$$
Then it follows from Corollary 9 that
$$
P \left(\Phi_ {\text {r a t i o}} \left(\mathcal {D} _ {P}, \mathcal {D} _ {Q}; \beta_ {0}\right) = 1\right) \geq 1 - \delta_ {3}.
$$
Observe that  $\delta_1 = \frac{\delta}{4} < \frac{1}{4}$  and  $|\mathcal{B}| \geq 4$ , and thus,  $\log \left(\frac{|\mathcal{B}|}{\delta_1}\right) \geq \log 16 \geq 2$ . Let  $c = c'$ . Since  $c' \geq 1$  and  $\log \left(\frac{|\mathcal{B}|}{\delta_1}\right) \geq \log \left(\frac{16}{\delta}\right) = \log \left(\frac{8}{\delta_3}\right)$ , we notice that
$$
\begin{array}{l} n _ {P} \geq \max  \left\{c, \frac {2 7}{\min  \left\{p _ {0} , p _ {1} \right\}} \right\} \cdot | \mathcal {B} | \log \left(\frac {4 | \mathcal {B} |}{\delta}\right) \\ \Longrightarrow \qquad n _ {P} \geq \max  \left\{c ^ {\prime} \cdot | \mathcal {B} | \log \left(\frac {| \mathcal {B} |}{\delta_ {1}}\right), 2 | \mathcal {B} |, \frac {2 7}{\min  \left\{p _ {0} , p _ {1} \right\}} \log \left(\frac {8}{\delta_ {3}}\right) \right\}. \\ \end{array}
$$
In summary, we obtain that for any given  $\delta \in (0,1)$
$$
\begin{array}{l} n _ {P} \geq \max  \left\{c, \frac {2 7}{\min  \left\{p _ {0} , p _ {1} \right\}} \right\} \cdot | \mathcal {B} | \log \left(\frac {4 | \mathcal {B} |}{\delta}\right) \quad \text {a n d} \quad n _ {Q} \geq \frac {2 7}{\min  \left\{q _ {0} , q _ {1} \right\}} \log \left(\frac {1 6}{\delta}\right) \\ \Longrightarrow \quad P \left[ \Phi_ {\text {b a l i a n c e}} (\mathcal {B}, 2) = 1 \& \Phi_ {\text {a p p r o x}} (\mathcal {B}, \varepsilon_ {\delta / 4}) = 1 \& \Phi_ {\text {r a t i o}} (\mathcal {D} _ {P}, \mathcal {D} _ {Q}; \beta_ {\delta / 2}) = 1 \right] \geq 1 - \delta . \tag {36} \\ \end{array}
$$
2We remark that our decomposition of  $\delta$  into  $\delta_1,\delta_2,\delta_3$  is arbitrary, and is intended to simplify the subsequent analysis.
Conditioned on the event  $\Phi_{\mathrm{balance}}(\mathcal{B},2) = 1$  &  $\Phi_{\mathrm{approx}}(\mathcal{B},\varepsilon_{\delta /4}) = 1$  &  $\Phi_{\mathrm{ratio}}(\mathcal{D}_P,\mathcal{D}_Q;\beta_{\delta /2}) = 1$
$$
T _ {1} \leq \left(\frac {\left| \rho_ {0} - \rho_ {1} \right|}{\rho_ {0} + \rho_ {1}}\right) ^ {2} \leq \left(\frac {\beta_ {\delta / 2} - \frac {1}{\beta_ {\delta / 2}}}{\beta_ {\delta / 2} + \frac {1}{\beta_ {\delta / 2}}}\right) ^ {2} \leq \left(\beta_ {\delta / 2} - 1\right) ^ {2}, \quad \because (3 4); a l s o, s e e (2 9)
$$
$$
\begin{array}{l} T _ {2} \leq \frac {w _ {\mathrm {m a x}} ^ {* 3}}{w _ {\mathrm {m i n}} ^ {* 2}} \cdot R _ {P} (\hat {h} _ {P}; f) \\ \leq \frac {w _ {\operatorname* {m a x}} ^ {*}}{w _ {\operatorname* {m i n}} ^ {* 2}} \cdot \left(\varepsilon_ {\delta / 4} ^ {2} + \frac {2}{| \mathcal {B} |}\right). \quad \because \text {p r o o f} \\ \end{array}
$$
Note that if Assumption (A3) holds, then we additionally have
$$
T _ {2} \leq \frac {w _ {\operatorname* {m a x}} ^ {*} {} ^ {3}}{w _ {\operatorname* {m i n}} ^ {* 2}} \cdot \left(\varepsilon_ {\delta / 4} ^ {2} + \frac {8 K ^ {2}}{| \mathcal {B} | ^ {2}}\right).
$$
Inserting these upper bounds for  $T_{1}$  and  $T_{2}$  into (31), (32) and recalling the expression for  $\beta$  in (35), we complete the proof.
D Proof sketch of the argument in Remark 5
Recall our composite recalibration function,
$$
\hat {h} _ {Q} = \hat {g} \circ \hat {h} _ {P}, \tag {37}
$$
does not use the features in  $\mathcal{D}_Q$ . Specifically,  $\hat{g}$  is parameterized by  $\hat{w} = (\hat{w}_0,\hat{w}_1)$ , which can be estimated using only labels in  $\mathcal{D}_P$  and  $\mathcal{D}_Q$ , cf. (16). According to Theorem 2,  $R_{Q}(\hat{h}_{Q}) = O(n_{Q}^{-1})$  with high probability for sufficiently large  $n_P$ .
Now suppose we are given an unlabeled target sample with unknown label shift. We can estimate  $w$  using the target features via a maximum likelihood label shift estimation approach [14], yielding  $\hat{w}^{\mathrm{ML}} = (\hat{w}_0^{\mathrm{ML}},\hat{w}_1^{\mathrm{ML}})$ . and the calibrated classifier  $\hat{h}\circ f$ . This results in a different composite recalibration function than Equation (37),
$$
\hat {h} _ {Q} ^ {\mathrm {M L}} = \hat {g} ^ {\mathrm {M L}} \circ \hat {h} _ {P}, \tag {38}
$$
where  $\hat{g}^{\mathrm{ML}}:[0,1]\to [0,1]$  is defined as  $\hat{g}^{\mathrm{ML}}(z) = \hat{w}_1^{\mathrm{ML}}z / (\hat{w}_1^{\mathrm{ML}}z + \hat{w}_0^{\mathrm{ML}}(1 - z))$ . We claim in Remark 5 that, for sufficiently large  $n_P$ , the composite recalibration function in Equation (38) achieves  $R_{Q}(\hat{h}_{Q}^{\mathrm{ML}}) = O(n_{Q}^{-1})$  with high probability, enjoying the same convergence rate as  $\hat{h}_Q$  (Equation 37). Here we give a proof sketch.
Proof sketch. Suppose we use the maximum likelihood approach in [14] to estimate  $w$ . We want to show  $R_{Q}(\hat{h}_{Q}^{\mathrm{ML}}) = O(n_{Q}^{-1})$  with high probability for sufficiently large  $n_P$ . Recall  $R_{Q}(\hat{h}_{Q}) \leq 2(T_{1} + T_{2})$  according to Equation (31) and (32), and label shift estimation error only affects  $T_{1}$ , so it is sufficient to show  $T_{1} = O(n_{Q}^{-1})$  with high probability.
For sufficiently large  $n_P, \hat{h}_P \circ f$  is sufficient calibrated, so  $\| \hat{w} - w \|_2^2 = O(n_Q^{-1})$  by Theorem 3 in [14]. Since
$$
\left\| \hat {w} - w \right\| _ {2} ^ {2} = \sum_ {k \in \{0, 1 \}} \left(\rho_ {k} - 1\right) ^ {2} w _ {k} ^ {2} \geq w _ {\min } ^ {*} ^ {2} \sum_ {k \in \{0, 1 \}} \left(\rho_ {k} - 1\right) ^ {2} \geq w _ {\min } ^ {*} ^ {2} \max  _ {k \in \{0, 1 \}} \left(\rho_ {k} - 1\right) ^ {2}, \tag {39}
$$
we have  $\rho_{k}\in [1 - \alpha ,1 + \alpha ]$  for  $k\in \{0,1\}$ , where  $\alpha = \frac{\|\hat{w} - w\|_2}{w_{\mathrm{min}}^*} >0$ . For sufficiently small  $\| \hat{w} -w\| _2^2$ , we can control  $\alpha < 0.5$ , which bounds  $T_{1}$  in (34):
$$
T _ {1} \leq \left(\frac {\rho_ {0} - \rho_ {1}}{\rho_ {0} + \rho_ {1}}\right) ^ {2} \leq \frac {2 \alpha}{2 - 2 \alpha} \leq 2 \alpha = 2 \frac {\| \hat {w} - w \| _ {2}}{w _ {\operatorname* {m i n}} ^ {*}} = O \left(n _ {Q} ^ {- 1}\right). \tag {40}
$$
The rest of the proof are the same with Appendix C.2.
E Details on the experiments
In Section E.1 and E.3, we consider a family of joint distributions  $\mathcal{D}(\pi)$  of  $X$  and  $Y$ , where  $Y \sim \text{Bernoulli}(\pi)$ ,  $X \mid Y = 0 \sim N(-2,1)$ , and  $X \mid Y = 1 \sim N(2,1)$ . Suppose we are given  $f(x) = \sigma(x) \coloneqq 1 / (1 + e^{-x})$ , for  $x \in \mathbb{R}$ , as a probabilistic classifier. The optimal recalibration function can be derived as
$$
h _ {f, P} ^ {*} (z) = P [ Y = 1 \mid f (X) = z ] = \sigma \left(4 \sigma^ {- 1} (z)\right). \tag {41}
$$
In Section E.2, we consider a parametric family of recalibration functions called beta calibration [28]:  $\mathcal{H}_{\mathrm{beta}} = \{h_{\mathrm{beta}}(\cdot ;a,b,c):a\geq 0,b\geq 0,c\in \mathbb{R}\}$ , where  $h_\mathrm{beta}(\cdot ;a,b,c):[0,1]\to [0,1]$  is defined as
$$
h _ {\text {b e t a}} (z; a, b, c) = \frac {1}{1 + 1 / \left(e ^ {c} \frac {z ^ {a}}{(1 - z) ^ {b}}\right)}. \tag {42}
$$
In addition, consider a subfamily  $\mathcal{H}_{\mathrm{logit-normal}} \subset \mathcal{H}_{\mathrm{beta}}$  defined as  $\mathcal{H}_{\mathrm{logit-normal}} = \{h_{\mathrm{logit-normal}}(\cdot; a, c) := h_{\mathrm{beta}}(\cdot; a, a, c) = \sigma(a\sigma^{-1}(\cdot) + c): a \geq 0, c \in \mathbb{R}\}$ . Apparently, the optimal recalibration function in Equation (41),  $h_{f,P}^{*} \in \mathcal{H}_{\mathrm{logit-normal}}$ .
E.1 Verifying results for UMB
First, we recalibrate  $f$  on data distributed as  $\mathcal{D}(0.5)$  using UMB.
Verifying the risk convergence in Theorem 1 We vary  $n \in [10^2, 10^7]$  and  $B \in [6, 10^3]$  in the log scale. For each combination of  $(n, B)$ , we use UMB to recalibrate  $f$  on data generated from  $\mathcal{D}(0.5)$ , and compute quadrature estimates of population  $R^{\mathrm{cal}}(\hat{h})$ ,  $R^{\mathrm{sha}}(\hat{h})$ , and  $R(\hat{h})$ , as well as their high probability bounds based on Theorem 1. The constant  $K$  in Assumption (A3) is selected by numerical maximization as
$$
K = \max  _ {0 \leq z _ {1} <   z _ {2} \leq 1} \frac {h ^ {*} (z _ {2}) - h ^ {*} (z _ {1})}{P [ Z \in [ z _ {1} , z _ {2} ] ]}.
$$
Figure 1 shows the bounds follow the same trends as their associated population quantities, providing valid upper bounds in all cases.
Verifying the optimal choice of the number of bins. We find empirically optimal  $B^{*}$  experiment that achieves the minimal risk for each choice of  $n$ . We compute the theoretically optimal choice of the number of bins,  $B^{*}$  theory, by minimizing the finite-sample upper bounds. Figure 2 shows  $B^{*}$  experiment follows the same trend with  $B^{*}$  theory, both scales in  $O(n^{1/3})$ .
E.2 Comparing recalibration methods
To highlight the benefits and drawbacks of UMB's nonparametric nature, we compare UMB with (semi-)parametric recalibration methods in scenarios where the parametric assumption is correct and where it is misspecified. We compare the method under study, uniform-mass binning (UMB), with 3 other recalibration methods: uniform-width binning (UWB) [18], Platt scaling  $[40]^4$ , and a hybrid parametric-binning method [29]. Note that Platt scaling and the hybrid method adopt the parametric assumption  $h^* \in \mathcal{H}_{\mathrm{logit-normal}}$ .
For the first setting, we construct optimal recalibration function  $h^* \in \mathcal{H}_{\mathrm{logit-normal}}$  so that the parametric assumption of Platt scaling and the hybrid method holds. In particular, we consider the distribution  $Z \in \mathrm{Uniform}[0,1]$  and  $Y \mid Z \sim \mathrm{Bernoulli}(h_{\mathrm{logit-normal}}(Z; a, c))$  with  $a = 4$  and  $c = 0$ . For the second setting, we construct  $h^* \in \mathcal{H}_{\mathrm{beta}}$  but  $h^* \notin \mathcal{H}_{\mathrm{logit-normal}}$  so that the parametric assumption fails. In particular, we consider the distribution  $Z \sim \mathrm{Uniform}[0,1]$  and  $Y \mid Z \sim \mathrm{Bernoulli}(h_{\mathrm{beta}}(z; a, b, c))$  with  $a = 0.1, b = 4$ , and  $c = 0$ . For each setting, we fix calibration sample size to be  $n = 5000$ .
<sup>3</sup>We say  $Z \sim \mathrm{Logit - Normal}(\mu, \tau^2)$  if  $\sigma^{-1}(Z) \sim N(\mu, \tau^2)$  [1]. Similar to beta calibration [28], we adopt the name "logit-normal calibration" after a simple example: if  $Y = \mathrm{Bernoulli}(0.5)$ ,  $Z \mid Y = i \sim \mathrm{Logit - Normal}(\mu_i, \tau_i^2)$  for  $i \in \{0, 1\}$ , then the optimal recalibration function  $\mathbb{E}[Y \mid Z = z] = h_{\mathrm{logit - normal}}(z; a, c)$  for some  $a, c$  depending on  $\mu_i$ 's and  $\tau_i$ 's.
4The original Platt scaling operates on outputs of real-valued SVM outputs [40]. For probabilistic classifiers, we follow [38, 29, 22] and implement Platt scaling by first transforming probabilities onto the real line via the logit transform  $\sigma^{-1}$ .
(b) Misspecified parametric assumption
Figure 5: Risks vs. number of bins  $B$ .
Risks as functions of the number of bins  $B$  We traverse the number of bins  $B \in [10, 2000]$  in the log scale and compare how each method behaves as  $B$  changes. When the parametric assumption is correct, the hybrid method achieves significantly lower  $R^{\mathrm{cal}}$  and overall  $R$  than UMB and UWB for sufficiently large number of bins (Figure 5a), an advantage highlighted in [29]. In contrast, when the parametric assumption fails, the binning methods UMB and UWB has better performance with the optimal number of bins (Figure 5b). This is because Platt scaling and hybrid methods are intrinsically biased when  $h^{*} \notin \mathcal{H}_{\mathrm{logit-normal}}$ , as noted in Section 4.2.
Quantitative results of risks under optimal  $B$ . For each setting, we fix  $B$  that achieves low recalibration risk for UWB and UMB in Figure 5. Specifically, we choose  $B = 2\lfloor n^{1 / 3}\rfloor = 34$  for the correct parametric assumption setting, and  $B = \lfloor n^{1 / 3}\rfloor = 17$  for the misspecified parametric assumption setting. Then, for each setting, we compare the  $90\%$  quantiles of risks of each recalibration method fitted on 100 random replicates of calibration datasets of size  $n = 5000$ .
Table 1 quantititively verifies that Platt and the Hybrid method achieves lower  $R^{\mathrm{cal}}$  and overall  $R$  if the parametric assumption is correct, and UWB and UMB achieves lower  $R^{\mathrm{cal}}$  and overall  $R$  when the parametric assumption fails.
Visualization of calibration curves We fix the calibration dataset and visualize the calibration curves for all methods under the two settings. Figure 3 shows that the binning methods (UWB and UMB) closely track the optimal recalibration function  $h^*$  in both settings. In contrast, the hybrid approach follows the Platt scaling estimates, leading to an inherent bias from  $h^*$  when the parametric assumption is invalid (Figure 3b).
E.3 Comparing recalibration schemes under label shift
We consider the label shift with source distribution  $\mathcal{D}(0.5)$  and target distribution  $\mathcal{D}(\pi_Q)$ , where  $\pi_Q$  varies in  $\{0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5\}$ . The results where  $\pi_Q > 0.5$  can be inferred by symmetry and hence not experimented. We vary  $n_P$  in  $\{10, 10^3, 10^5, 10^7\}$  and  $n^Q$  in  $\{10, 10^3, 10^5\}$ .
Figure 6: Left: calibration curves of for COMPOSITE  $\hat{h}_Q$ , SOURCE  $\hat{h}_P$ , TARGET  $\hat{h}_Q^{\mathrm{target}}$ , and LABEL-SHIFT  $\hat{g}$ . Right: the marginal density of  $Z = f(X)$  under  $Q$ .
Aside from our proposed recalibration function  $\hat{h}_Q = \hat{g} \circ \hat{h}_P$  (17), referred to as COMPOSITE, we consider three other calibration approaches as baselines: (1) SOURCE, denoted as  $\hat{h}_P$ , which is only calibrated on the source data, (2) LABEL-SHIFT, denoted as  $\hat{g}$ , which performs label shift correction without calibration, and (3) TARGET, denoted as  $\hat{g}_Q^{\mathrm{target}}$ , which is only calibrated on the target data. The number of bins  $B$  are chosen to be  $n_P^{1/3}$  for COMPOSITE and SOURCE, and  $n_Q^{1/3}$  for TARGET.