SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present a proof of Theorem 1. Let  $(Y,Z)\in \mathcal{V}\times \mathcal{Z}$  be random variables that admits a joint distribution  $P_{Y,Z}$ , which we assume to be fixed throughout this section. Let  $S = \{(y_i,z_i)\in \mathcal{V}\times \mathcal{Z}:i\in [n]\}$  and let  $\mathcal{B} = \{I_1,I_2,\dots ,I_B\}$  be the uniform-mass binning scheme (cf. Definition 6) of size  $B$  induced by  $(z_{i}$ 's in  $S$ . Note that if  $S$  is a random sample from  $P_{Y,Z}$ , then the binning scheme  $\mathcal{B}$  induced by  $S$  is also a random variable following a derived distribution. To facilitate our analysis, we introduce the notion of well-balanced binning.
Definition 8 (Well-balanced binning; [29]). Let  $B \in \mathbb{N}$ , let  $Z$  be a random variable that takes value in  $[0,1]$ , and let  $\alpha \in \mathbb{R}$  such that  $\alpha \geq 1$ . A binning scheme  $\mathcal{B}$  of size  $B$  is  $\alpha$ -well-balanced with respect to  $Z$  if
$$
\frac {1}{\alpha B} \leq P [ Z \in I _ {b} ] \leq \frac {\alpha}{B}, \quad \forall b \in [ B ].
$$
In addition, we define two (parameterized families of) Boolean-valued functions  $\Phi_{\mathrm{balance}}$  and  $\Phi_{\mathrm{approx}}$  as follows: for any binning scheme  $\mathcal{B}$ ,
$$
\forall \alpha \in \mathbb {R}, \quad \Phi_ {\text {b a l a n c e}} (\mathcal {B}; \alpha) := \mathbb {1} \left\{\frac {1}{\alpha | \mathcal {B} |} \leq P [ Z \in I ] \leq \frac {\alpha}{| \mathcal {B} |}, \forall I \in \mathcal {B} \right\}, \tag {21}
$$
$$
\forall \varepsilon \in \mathbb {R}, \quad \Phi_ {\text {a p p r o x}} (\mathcal {B}; \varepsilon) := \mathbb {1} \left\{\max  _ {I \in \mathcal {B}} | \hat {\mu} _ {I} - \mu_ {I} | \leq \varepsilon \right\}, \tag {22}
$$
where  $\mathbb{1}(A) = 1$  if and only if the predicate  $A$  is true, and for each interval  $I\in \mathcal{B}$
$$
\hat {\mu} _ {I} = \frac {\sum_ {i = 1} ^ {n} y _ {i} \cdot \mathbb {1} _ {I} \left(z _ {i}\right)}{\sum_ {i = 1} ^ {n} \cdot \mathbb {1} _ {I} \left(z _ {i}\right)} \quad \text {a n d} \quad \mu_ {I} = \mathbb {E} _ {(Y, Z) \sim P _ {Y, Z}} [ Y \cdot \mathbb {1} _ {I} (Z) ]. \tag {23}
$$
Note that if  $\Phi_{\mathrm{balance}}(\mathcal{B};\alpha) = 1$  for  $\alpha \geq 1$ , then  $\mathcal{B}$  is  $\alpha$ -well-balanced with respect to  $Z$  (cf. Definition 8). Also, if  $\Phi_{\mathrm{approx}}(\mathcal{B};\varepsilon) = 1$  for  $\varepsilon \geq 0$ , then the conditional empirical mean of  $Y$  in each bin  $I \in \mathcal{B}$  approximates the conditional expectation with error at most  $\varepsilon$ , uniformly for all bins.
The rest of this section is organized as follows. In Section B.1, we ensure that for an appropriate choice of  $\alpha, \varepsilon \in \mathbb{R}$ , it holds with high probability (with respect to the randomness in  $\mathcal{B}$ ) that  $\Phi_{\mathrm{balance}}(\mathcal{B}; \alpha) = \Phi_{\mathrm{approx}}(\mathcal{B}; \varepsilon) = 1$ . In Section B.2, we establish upper bounds on the reliability risk  $R^{\mathrm{cal}}$  and the sharpness risk  $R^{\mathrm{sha}}$  under the premise that  $\Phi_{\mathrm{balance}}(\mathcal{B}; \alpha) = \Phi_{\mathrm{approx}}(\mathcal{B}; \varepsilon) = 1$ . Finally, in Section B.3, we conclude the proof of Theorem 1 by combining these results together.
B.1 High-probability certification of the conditions
Well-balanced binning scheme. First of all, we observe that the uniform-bass binning scheme  $\mathcal{B}$  induced by an IID random sample from  $P_{Y,Z}$  is 2-well-balanced with high probability, if the sample size is sufficiently large. Here we paraphrase a result from [29] in our language.
Lemma 3 ([29, Lemma 4.3]). Let  $S = \{Z_{i} : i \in [n]\}$  be an IID sample drawn from  $P_Z$  and let  $\mathcal{B}$  be the uniform-mass binning scheme of size  $B$  induced by  $S$ . There exists a universal constant  $c' > 0$  such that for any  $\delta \in (0,1)$ , if  $n \geq c' \cdot B\log(B / \delta)$ , then  $\Phi_{\mathrm{balance}}(\mathcal{B},2) = 1$  with probability at least  $1 - \delta$ .
Lemma 3 states that
$$
n \geq c ^ {\prime} \cdot B \log \left(\frac {B}{\delta}\right) \quad \Longrightarrow \quad P [ \mathcal {B} \text {i s 2 - w e l l - b a n c e d w i t h r e s p e c t t o} P _ {Z} ] \geq 1 - \delta .
$$
While the value of the universal constant  $c$  was not specified in the original reference [29], we remark that one may set, for example,  $c' = 2420$ , which can be verified by following their proof with  $c'$  kept explicit.
The proof of Lemma 3 in [29] relies on a discretization argument that considers a fine-grained cover of  $\mathcal{Z} = [0,1]$  consisting of disjoint intervals—namely,  $\left\{I_j':j\in [10B]\right\}$  such that  $P[Z\in I_j'] = \frac{1}{10B}$  for all  $j\in [10B]$ —and then approximates each  $I_{b}$  by a subset of the cover. As the authors of [29] remarked, this argument provides a tighter sample complexity upper bound than naively applying Chernoff bounds or a standard VC dimension argument, which would yield an upper bound of order  $O\left(B^{2}\log \left(\frac{B}{\delta}\right)\right)$ . We omit the proof of Lemma 3 and refer interested readers to the referenced paper [29] for more details.
Uniform concentration of bin-wise means. Next, we argue that for the uniform-mass binning scheme  $\mathcal{B}$  induced by an IID sample, the conditional empirical means of each bin concentrates to the population conditional expectation, uniformly for all bins in  $\mathcal{B}$ . Here we restate a result from [20].
Lemma 4 ([20, Corollary 1]). Let  $P_Z$  be an absolutely continuous probability measure on  $\mathcal{Z} = [0,1]$ , and  $S = \{Z_i : i \in [n]\}$  be an IID sample drawn from  $P_Z$ . Let  $B \in \mathbb{N}$  such that  $B \leq \frac{n}{2}$  and  $\mathcal{B}$  be the uniform-mass binning scheme of size  $B$  induced by  $S$ . Then for any  $\delta \in (0,1)$ ,
$$
P \left[ \Phi_ {\text {a p p r o x}} (\mathcal {B}; \varepsilon_ {\delta}) = 1 \right] \geq 1 - \delta \quad \text {w h e r e} \quad \varepsilon_ {\delta} = \sqrt {\frac {1}{2 (\lfloor n / B \rfloor - 1)} \log \left(\frac {2 B}{\delta}\right)} + \frac {1}{\lfloor n / B \rfloor}. \tag {24}
$$
Lemma 4 states that under the mild regularity condition of  $P_Z$  being absolutely continuous, the uniform-mass binning accurately approximates all bin-wise conditional means as long as there are at least two samples per bin in the sense that
$$
n \geq 2 B \quad \Longrightarrow \quad P \left[ \sup  _ {b \in [ B ]} | \hat {\mu} _ {b} - \mu_ {b} | \leq \sqrt {\frac {1}{2 (\lfloor n / B \rfloor - 1)} \log \left(\frac {2 B}{\delta}\right)} + \frac {1}{\lfloor n / B \rfloor} \right] \geq 1 - \delta .
$$
B.2 Conditional upper bounds on reliability risk and sharpness risk
In this section, we establish upper bounds on the reliability risk  $R^{\mathrm{cal}}$  and the sharpness risk  $R^{\mathrm{sha}}$  for  $\hat{h}$  under the premise that  $\Phi_{\mathrm{balance}}(\mathcal{B};\alpha) = 1$  and  $\Phi_{\mathrm{approx}}(\mathcal{B};\varepsilon) = 1$  for appropriate parameters  $\alpha ,\varepsilon \in \mathbb{R}$ .
**Preparation.** To avoid clutter in the lemma statements to follow, here we recall our problem setting and set several notation that will be used throughout this section. Recall that  $P = P_{X,Y}$  is a joint distribution on  $\mathcal{X} \times \mathcal{Y}$  and let  $f: \mathcal{X} \to \mathcal{Z}$  is a measurable function. In addition, we let  $\tilde{S} = \{(x_i, y_i) \in \mathcal{X} \times \mathcal{Y} : i \in [n]\}$  be an IID sample drawn from  $P$ , and let  $S = \{(z, y) \in \mathcal{Z} \times \mathcal{Y} : (x, y) \in \tilde{S} \text{ and } z = f(x)\}$ . Let  $\mathcal{B}$  be the uniform-mass binning scheme induced by (z's in)  $S$ , and let  $\hat{h} = \hat{\mathcal{B}}: \mathcal{Z} \to \mathcal{Z}$  be the recalibration function derived from  $\mathcal{B}$  as we described in Section 4.1; see (9). The dependence among  $P, f, \tilde{S}, S, \mathcal{B}$ , and  $\hat{h}$  are summarized by a diagram in Figure 4.
Furthermore, we define the index function for a binning scheme to facilitate our analysis.
Figure 4: Stochastic dependence among  $P, f, \tilde{S}, S, \mathcal{B}$ , and  $\hat{h}$ .
Definition 9. Let  $\mathcal{B}$  be a binning scheme. The index function for  $\mathcal{B}$  is the function  $\beta : \mathcal{Z} \to [|\mathcal{B}|]$  such that
$$
\beta (z) = \sum_ {I \in \mathcal {B}} \mathbb {1} _ {(0, \sup  I ]} (z). \tag {25}
$$
Remark 6. Note that  $\beta$  is a measurable function and defines an index function that identifies which bin of  $\mathcal{B}$  the argument  $z\in [0,1]$  belongs to. Specifically, suppose that  $\mathcal{B} = \{I_1,\dots ,I_B\}$  for some  $B\in \mathbb{N}$  and there exists  $u_{0},u_{1},\ldots ,u_{B}\in [0,1]$  such that (i)  $0 = u_{0} < u_{1} < \dots < u_{B} = 1$  and (ii)  $I_{b} = (u_{b - 1},u_{b}]$  for all  $b\in [B]\setminus \{1\}$  and  $I_{1} = [u_{0},u_{1}]$ . Then  $\beta (z) = b$  if and only if  $z\in I_b$ .
B.2.1 Calibration risk upper bound
We observe that if a binning scheme  $\mathcal{B}$  produces empirical means  $\hat{\mu}_I$  that approximate the true means  $\mu_I$  with error at most  $\varepsilon$ , then the calibration risk is upper bounded by  $\varepsilon^2$ .
Lemma 5 (Calibration risk bound). For any  $\varepsilon \geq 0$ , if  $\Phi_{\mathrm{approx}}(\mathcal{B};\varepsilon) = 1$ , then
$$
R ^ {\mathrm {c a l}} (\hat {h}; f, P) \leq \varepsilon^ {2}.
$$
Proof of Lemma 5. To begin with, we recall the definition of the calibration risk (Definition 3), and let  $Z = f(X)$ . Then we may write
$$
= \mathbb {E} \left[ \mathbb {E} \left[ \left(\hat {h} (Z) - \mathbb {E} [ Y \mid \hat {h} (Z) ]\right) ^ {2} \mid \beta (Z) \right] \right] \quad \because \text {t h e l a w o f t o t a l e x p e c t a t i o n}
$$
$$
\begin{array}{l} R ^ {\mathrm {c a l}} \big (\hat {h}; f, P \big) = \mathbb {E} \left[ \left(\hat {h} (Z) - \mathbb {E} [ Y \mid \hat {h} (Z) ]\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\hat {\mu} _ {I _ {\beta (Z)}} - \mu_ {I _ {\beta (Z)}}\right) ^ {2} \right] \tag {23} \\ \leq \max _ {I \in \mathcal {B}} \left(\hat {\mu} _ {I} - \mu_ {I}\right) ^ {2}. \\ \end{array}
$$
Note that if  $\Phi_{\mathrm{approx}}(\mathcal{B};\varepsilon) = 1$ , then  $\max_{I\in \mathcal{B}}\left(\hat{\mu}_I - \mu_I\right)^2\leq \varepsilon^2$
We remark that the proof of Lemma 5 is a simple application of applying Hölder's inequality. Also, we note that a similar argument was considered in [20, Proposition 1] to establish the inequalities between the  $L^p$ -counterparts of the calibration risk, which they call the  $\ell_p$ -expected calibration error (ECE). In this work, we focus on the case  $p = 2$ .
B.2.2 Sharpness risk upper bound
Next, we present an upper bound for the sharpness risk that diminishes as the binning scheme  $\mathcal{B}$  becomes more balanced.
Lemma 6 (Sharpness risk bound). Suppose that the optimal post-hoc recalibration function  $h_{f,P}^{*}$ , cf. (7), is monotonically non-decreasing. Let  $\alpha \in \mathbb{R}$  such that  $\alpha \geq 1$ . If  $\Phi_{\mathrm{balance}}(\mathcal{B},\alpha) = 1$ , then
$$
R ^ {\mathrm {s h a}} (\hat {h}; f, P) \leq \frac {\alpha}{| \mathcal {B} |}.
$$
Proof of Lemma 6. Letting  $Z = f(X)$ , we can write the sharpness risk of  $\hat{h}$  over  $f$  with respect to  $P$  as
$$
R ^ {\mathrm {s h a}} (\hat {h}; f, P) := \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ]\right) ^ {2} \right].
$$
We recall the definition of the index function  $\beta$  for  $\mathcal{B}$  (Definition 9) and observe that
$$
\mathbb {E} \left[ \left(\mathbb {E} \left[ Y \mid \hat {h} (Z) \right] - \mathbb {E} [ Y \mid Z ]\right) ^ {2} \right]
$$
$$
\leq \mathbb {E} \left[ \left| \mathbb {E} \left[ Y \mid \hat {h} (Z) \right] - \mathbb {E} [ Y \mid Z ] \right| \right]
$$
$$
\because \left| \mathbb {E} \left[ Y \mid \hat {h} (Z) \right] - \mathbb {E} \left[ Y \mid Z \right] \right| \leq 1
$$
$$
= \sum_ {I \in \mathcal {B}} \mathbb {E} \left[ \left| \mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ] \right| \cdot \mathbb {1} _ {I} (Z) \right]
$$
$$
= \sum_ {I \in \mathcal {B}} \mathbb {E} \left[ \mathbb {E} \left[ \left| \mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ] \right| \cdot \mathbb {1} _ {I} (Z) \mid \beta (Z) \right] \right]
$$
the law of total expectation
$$
= \sum_ {I \in \mathcal {B}} P [ Z \in I ] \cdot \mathbb {E} \left[ \mathbb {E} \left[ | \mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ] | \mid Z \in I \right] \right]
$$
Remark 6
$$
\leq \sum_ {I \in \mathcal {B}} P [ Z \in I ] \cdot \left(\sup  _ {z \in I} h _ {f, P} ^ {*} (z) - \inf  _ {z \in I} h _ {f, P} ^ {*} (z)\right)
$$
$\because$  by definition of  $h_{f,P}^{*}$ ; cf. (7)
$$
\leq \sum_ {I \in \mathcal {B}} \frac {\alpha}{| \mathcal {B} |} \cdot \left(\sup  _ {z \in I} h _ {f, P} ^ {*} (z) - \inf  _ {z \in I} h _ {f, P} ^ {*} (z)\right)
$$
$$
\because \Phi_ {\text {b a l a n c e}} (\mathcal {B}, \alpha) = 1
$$
$$
\leq \frac {\alpha}{| \mathcal {B} |}.
$$
The inequality in the last line follows from the facts that (i)  $I \in \mathcal{B}$  are mutually exclusive and (ii)  $h_{f,P}^{*}(z) \in [0,1]$  and  $h_{f,P}^{*}$  is monotone non-decreasing.
Our proof of Lemma 6 relies on similar techniques that are used in [29, Lemmas D.5 and D.6]. However, we note that we obtain an improved constant — 1 as opposed to 2 in [29, Lemma D.6] — with a more refined analysis.
An improved rate with additional assumptions. It is possible to improve the rate of the sharpness risk upper bound from  $O(|\mathcal{B}|^{-1})$  to  $O(|\mathcal{B}|^{-2})$  with an additional regularity assumption on  $h_{f,P}^{*}$ .
Recall that we assumed in (A3) that there exists  $K > 0$  such that if  $z_{1} \leq z_{2}$ , then  $h_{f,P}^{*}(z_{2}) - h_{f,P}^{*}(z_{1}) \leq K \cdot \left(F_{Z}(z_{2}) - F_{Z}(z_{1})\right)$ , that is,  $h_{f,P}^{*}$  is  $K$ -smooth with respect to  $F_{Z}$ . This posits that the conditional probability  $P[Y = 1|Z = z]$  of the target variable  $Y$  given a forecast variable  $Z$  cannot vary too much in regions where the density of  $Z$  is low, or where the forecast is rarely issued. This is a reasonable assumption because if  $P[Y = 1|Z]$  changes too rapidly with respect to  $Z$ , then it suggests that we need additional information about  $Y$  beyond what  $Z$  can provide in order to improve the quality of forecasts. We remark that (A3) is indeed a fairly mild assumption to impose on, however, is not a trivial one.
Remark 7 (Mildness of (A3)). Suppose that  $Z = f(X)$  has a density  $p_Z$  that is uniformly lower bounded by  $\epsilon$  on the support of  $Z$ . If  $h_{f,P}^*$  is  $L$ -Lipschitz, then  $h_{f,P}^*$  is  $(L / \epsilon)$ -smooth with respect to  $F_Z$ . This also provides a sufficient condition to verify (A3) in practice.
Remark 8 (Non-triviality of (A3)). Notice that even if  $F_Z$  is absolutely continuous and  $h_{f,P}^*$  is continuous, the smoothness constant  $K$  could become large if the prediction  $Z$  is heavily miscalibrated. For instance, in Figure 6,  $h_{f,P}^*(z)$  is changing fast in the interval [0.5, 0.75] where  $p_Z(z)$  is small, which results in a larger value of  $K$  that can even diverge if  $p_Z(z) \to 0$ .
Here we define the notion of  $\psi$ -smoothness to formalize Assumption (A3), and then present an improved upper bound for the sharpness risk.
Definition 10 (  $\psi$  -smoothness). Let  $K\in \mathbb{R}_{+}$  and  $\psi :[0,1]\to [0,1]$  be a monotone non-decreasing function. A function  $\phi :[0,1]\rightarrow [0,1]$  is  $K$  -smooth with respect to  $\psi$  if for any  $z_{1},z_{2}\in [0,1]$  such that  $z_{1}\leq z_{2}$
$$
\left| \phi \left(z _ {2}\right) - \phi \left(z _ {1}\right) \right| \leq K \cdot \left(\psi \left(z _ {2}\right) - \psi \left(z _ {1}\right)\right). \tag {26}
$$
Lemma 7 (Improved sharpness risk bound). Suppose that the function  $h_{f,P}^{*}(z)$  defined in (7) is monotonically non-decreasing and  $K$ -smooth with respect to  $F_{Z}$  for some  $K \geq 0$ , where  $F_{Z}$  is the cumulative distribution function of  $Z = f(X)$ . If  $\Phi_{\mathrm{balance}}(\mathcal{B},\alpha) = 1$ , then
$$
R ^ {\mathrm {s h a}} \leq \frac {K ^ {2} \alpha^ {3}}{B ^ {2}}.
$$
Proof of Lemma 7. Let  $Z = f(X)$  and  $B = |\mathcal{B}|$ . For each  $b \in [B]$ , we let  $z_{b,\max} := \sup I_b$  and  $z_{b,\min} := \inf I_b$ . Then we have
$$
\begin{array}{l} R ^ {\mathrm {s h a}} (\hat {h}; f, P) \\ = \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ]\right) ^ {2} \right] \\ = \mathbb {E} \left[ \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ]\right) ^ {2} \mid \beta (Z) \right] \right] \\ = \sum_ {b = 1} ^ {B} P [ Z \in I _ {b} ] \cdot \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid \hat {h} (Z) ] - \mathbb {E} [ Y \mid Z ]\right) ^ {2} \Big | \beta (Z) = b \right] \\ \leq \sum_ {b = 1} ^ {B} P [ Z \in I _ {b} ] \cdot \left(h _ {f, P} ^ {*} \left(z _ {b, \max }\right) - h _ {f, P} ^ {*} \left(z _ {b, \min }\right)\right) ^ {2} \quad \because h _ {f, P} ^ {*} \text {i s n o n - d e c r e a s i n g} \\ \leq \sum_ {b = 1} ^ {B} P [ Z \in I _ {b} ] \cdot \left(K \cdot \left(F _ {Z} \left(z _ {b, \max }\right) - F _ {Z} \left(z _ {b, \min }\right)\right)\right) ^ {2} \quad \because h _ {f, P} ^ {*} \text {i s K - s m o o t h w . r . t .} F _ {Z} \\ = \sum_ {b = 1} ^ {B} K ^ {2} \cdot P [ Z \in I _ {b} ] ^ {3} \\ \leq K ^ {2} \sum_ {b = 1} ^ {B} \left(\frac {\alpha}{B}\right) ^ {3} \quad \because \Phi_ {\text {b a l a n c e}} (\mathcal {B}, \alpha) = 1 \\ = \frac {K ^ {2} \alpha^ {3}}{B ^ {2}}. \\ \end{array}
$$
Remark 9 (Tightness of the rate  $O(B^{-2})$ ). The asymptotic rate  $R^{\mathrm{sha}} = O(B^{-2})$  is tight and cannot be further improved without additional assumptions. For instance, let's consider a uniform-mass binning of size  $B$  on  $Z \sim \text{Uniform}[0,1]$ . In the population limit, each bin has width  $1 / B$  and within-bin variance  $1 / (12B^{2})$ . Thus, the sharpness risk, obtained by taking expectation of the conditional variance (per each bin), is  $1 / (12B^{2})$ , attaining the rate  $B^{-2}$ .
B.3 Completing the proof of Theorem 1
Proof of Theorem 1. For given  $\delta \in (0,1)$ , let  $\delta_1 = \delta_2 = \delta /2$ . Then we observe that
$$
n \geq c ^ {\prime} \cdot | \mathcal {B} | \log \left(\frac {| \mathcal {B} |}{\delta_ {1}}\right) \quad \Longrightarrow \quad P \left[ \Phi_ {\text {b a l a n c e}} (\mathcal {B}, 2) = 1 \right] \geq 1 - \delta_ {1} \quad \text {b y}
$$
$$
n \geq 2 | \mathcal {B} | \quad \Longrightarrow \quad P \left[ \Phi_ {\text {a p p r o x}} \left(\mathcal {B}, \varepsilon_ {\delta_ {2}}\right) = 1 \right] \geq 1 - \delta_ {2} \quad \text {b y}
$$
where  $c' > 0$  is the universal constant that appears in Lemma 3 and
$$
\varepsilon_ {\delta_ {2}} = \sqrt {\frac {1}{2 (\lfloor n / | \mathcal {B} | \rfloor - 1)} \log \left(\frac {2 | \mathcal {B} |}{\delta_ {2}}\right)} + \frac {1}{\lfloor n / | \mathcal {B} | \rfloor}.
$$
Observe that  $\delta_1 = \frac{\delta}{2} < \frac{1}{2}$  and  $|\mathcal{B}| \geq 1$ , and thus,  $\log \left(\frac{|\mathcal{B}|}{\delta_1}\right) \geq \log 2$ . Letting  $c \coloneqq \max \{c', \frac{2}{\log 2}\}$  and applying the union bound, we have
$$
n \geq c \cdot | \mathcal {B} | \log \left(\frac {2 | \mathcal {B} |}{\delta}\right) \quad \Longrightarrow \quad P \left[ \Phi_ {\text {b a l a n c e}} (\mathcal {B}, 2) = 1 \text {a n d} \Phi_ {\text {a p p r o x}} (\mathcal {B}, \varepsilon_ {\delta / 2}) = 1 \right] \geq 1 - \delta .
$$
Next, we observe that if  $\Phi_{\mathrm{balance}}(\mathcal{B},2) = 1$  and  $\Phi_{\mathrm{approx}}(\mathcal{B},\varepsilon_{\delta_2}) = 1$ , then
$$
R ^ {\text {c a l}} (\hat {h}; f; P) \leq \left(\varepsilon_ {\delta_ {2}}\right) ^ {2} \quad \text {b y}
$$
$$
R ^ {\mathrm {s h a}} (\hat {h}; f, P) \leq \frac {2}{| \mathcal {B} |}, \quad \text {b y}
$$
Additionally, if the Assumption (A3) also holds, then we obtain a stronger upper bound on  $R^{\mathrm{sha}}(\hat{h};f,P)$  by Lemma 7:
$$
R ^ {\mathrm {s h a}} (\hat {h}; f, P) \leq \frac {8 K ^ {2}}{| \mathcal {B} | ^ {2}}.
$$
This section contains a proof of Theorem 2. Prior to the proof, in Section C.1, we provide several lemmas that will be useful in our proof. Thereafter, we present a proof of Theorem 2 in its entirety in Section C.2.
C.1 Useful lemmas
C.1.1 Concentration of  $\hat{w}_k$  to  $w_k^*$
First of all, we recall the binomial Chernoff bound, which is a classical result about the concentration of measures that can be found in standard textbooks on probability theory.
Lemma 8 (Binomial Chernoff bound). Let  $X_{i}$  be IID Bernoulli random variables with parameters  $p \in (0,1)$ , and let  $S_{n} \coloneqq \frac{1}{n}\sum_{i = 1}^{n}X_{i}$ . Then for any  $\delta \in \mathbb{R}$  such that  $0 < \varepsilon < 1$ ,
$$
P \left[ S _ {n} \geq (1 + \varepsilon) p \right] \leq \exp \left(- \frac {\varepsilon^ {2} p}{3} n\right),
$$
$$
P \left[ S _ {n} \leq (1 - \varepsilon) p \right] \leq \exp \left(- \frac {\varepsilon^ {2} p}{2} n\right).
$$
It follows from Lemma 8 that for any  $\varepsilon, \delta \in (0,1)$ ,
$$
n \geq \frac {3}{\varepsilon^ {2} p} \log \left(\frac {2}{\delta}\right) \quad \Longrightarrow \quad P \left(\frac {\left| S _ {n} - p \right|}{p} > \varepsilon\right) \leq \delta . \tag {27}
$$
Let  $P, Q$  be two distributions on  $\mathcal{Y} = \{0,1\}$ , and let  $\mathcal{D}_P \sim P$ ,  $\mathcal{D}_Q \sim Q$  denote IID samples of size  $n_P, n_Q$ , respectively. Recall from (13) and (16) that for each  $k \in \{0,1\}$ , we define
$$
w _ {k} ^ {*} = \frac {P _ {Q} [ Y = k ]}{P _ {P} [ Y = k ]}, \qquad \text {a n d} \qquad \hat {w} _ {k} = \frac {P _ {\mathcal {D} _ {Q}} [ Y = k ]}{P _ {\mathcal {D} _ {P}} [ Y = k ]}.
$$
$$
\rho_ {0} := \frac {\hat {w} _ {0}}{w _ {0} ^ {*}} \quad \text {a n d} \quad \rho_ {1} := \frac {\hat {w} _ {1}}{w _ {1} ^ {*}}. \tag {28}
$$
Now we define another parameterized family of Boolean-valued functions  $\Phi_{\mathrm{ratio}}(\mathcal{D}_P,\mathcal{D}_Q;\beta)$  as follows. Given  $\mathcal{D}_P\sim P,\mathcal{D}_Q\sim Q$  , and  $\beta \in \mathbb{R}$  such that  $1 <   \beta \leq 2$
$$
\Phi_ {\text {r a t i o}} \left(\mathcal {D} _ {P}, \mathcal {D} _ {Q}; \beta\right) := \mathbb {1} \left\{\frac {1}{\beta} \leq \rho_ {k} \leq \beta , \forall k \in \{0, 1 \} \right\}. \tag {29}
$$
Corollary 9. Let  $P, Q$  be two distributions on  $\mathcal{Y} = \{0,1\}$ , and let  $\mathcal{D}_P \sim P$ ,  $\mathcal{D}_Q \sim Q$  denote IID samples of size  $n_P$ ,  $n_Q$ , respectively. For each  $k \in \{0,1\}$ , let  $p_k := P_P[Y = k]$  and  $q_k := P_Q[Y = k]$ . Likewise, we let  $\hat{p}_k = \frac{1}{n_P} \sum_{y_i \in \mathcal{D}_P} \mathbb{1}\{y_i = k\}$  and  $\hat{q}_k = \frac{1}{n_Q} \sum_{y_i \in \mathcal{D}_Q} \mathbb{1}\{y_i = k\}$ . For any  $\delta \in (0,1)$  and any  $\beta \in (1,2]$ , if
$$
n _ {P} \geq \frac {2 7}{(\beta - 1) ^ {2} \min \{p _ {0} , p _ {1} \}} \log \left(\frac {8}{\delta}\right) a n d n _ {Q} \geq \frac {2 7}{(\beta - 1) ^ {2} \min \{q _ {0} , q _ {1} \}} \log \left(\frac {8}{\delta}\right),
$$
$$
P \left(\Phi_ {\text {r a t i o}} (\mathcal {D} _ {P}, \mathcal {D} _ {Q}; \beta) = 1\right) \geq 1 - \delta .
$$
Proof of Corollary 9. Let  $\varepsilon = \frac{\beta - 1}{3}$ . Since  $\frac{1 + x}{1 - x} \leq 1 + 3x$  for all  $x \in [0,1/3]$ , we have  $\frac{1}{\beta} \leq \frac{1 - \varepsilon}{1 + \varepsilon} < \frac{1 + \varepsilon}{1 - \varepsilon} \leq \beta$ . Then it follows from (27) that for each  $k \in \{0,1\}$ ,
$$
n _ {P} \geq \frac {3}{\varepsilon^ {2} p _ {k}} \log \left(\frac {8}{\delta}\right) \quad \Longrightarrow \quad P \left(\frac {\left| \hat {p} _ {k} - p _ {k} \right|}{p _ {k}} > \varepsilon\right) \leq \frac {\delta}{4},
$$
$$
n _ {Q} \geq \frac {3}{\varepsilon^ {2} q _ {k}} \log \left(\frac {8}{\delta}\right) \quad \Longrightarrow \quad P \left(\frac {| \hat {q} _ {k} - q _ {k} |}{q _ {k}} > \varepsilon\right) \leq \frac {\delta}{4}.
$$
Applying the union bound, we obtain the following implication: