SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeyu Sun*
zeyusun@umich.edu
dogyoons@umich.edu
Alfred Hero
hero@eecs.umich.edu
Recalibrating probabilistic classifiers is vital for enhancing the reliability and accuracy of predictive models. Despite the development of numerous recalibration algorithms, there is still a lack of a comprehensive theory that integrates calibration and sharpness (which is essential for maintaining predictive power). In this paper, we introduce the concept of minimum-risk recalibration within the framework of mean-squared-error (MSE) decomposition, offering a principled approach for evaluating and recalibrating probabilistic classifiers. Using this framework, we analyze the uniform-mass binning (UMB) recalibration method and establish a finite-sample risk upper bound of order  $\tilde{O}(B / n + 1 / B^2)$  where  $B$  is the number of bins and  $n$  is the sample size. By balancing calibration and sharpness, we further determine that the optimal number of bins for UMB scales with  $n^{1/3}$ , resulting in a risk bound of approximately  $O(n^{-2/3})$ . Additionally, we tackle the challenge of label shift by proposing a two-stage approach that adjusts the recalibration function using limited labeled data from the target domain. Our results show that transferring a calibrated classifier requires significantly fewer target samples compared to recalibrating from scratch. We validate our theoretical findings through numerical simulations, which confirm the tightness of the proposed bounds, the optimal number of bins, and the effectiveness of label shift adaptation.
Generating reliable probability estimates alongside accurate class labels is crucial in classification tasks. A probabilistic classifier is considered "well calibrated" when its predicted probabilities closely align with the empirical frequencies of the corresponding labels [9]. Calibration is highly desirable, particularly in high-stakes applications such as meteorological forecasting [32, 34, 10, 15], econometrics [16], personalized medicine [25, 24], and natural language processing [37, 7, 11, 53]. Unfortunately, many machine learning algorithms lack inherent calibration [18].
To tackle this challenge, various methods have been proposed for designing post hoc recalibration functions. These functions are used to assess calibration error [52, 42, 17, 4], detect miscalibration [30], and provide post-hoc recalibration [50, 51, 18, 48, 29, 21]. Despite the rapid development of recalibration algorithms, there is still a lack of a comprehensive theory that encompasses both calibration and sharpness (retaining predictive power) from a principled standpoint. Furthermore, existing methods often rely on diverse calibration metrics [17, 4], and the selection of hyperparameters is often based on heuristic approaches [44, 20] without rigorous justifications. This highlights the
need for identifying an optimal metric to evaluate calibration, which can facilitate the development of a unified theory and design principles for recalibration functions.
In addition, the deployment of machine learning models to data distributions that differ from the training phase is increasingly common. These distribution shifts can occur naturally due to factors such as seasonality or other variations, or they can be induced artificially through data manipulation methods such as subsampling or data augmentation. Distribution shifts pose challenges to the generalization of machine learning models. Therefore, it becomes necessary to adapt the trained model to these new settings. One significant category of distribution shifts is label shift, where the marginal probabilities of the classes differ between the training and test sets while the class conditional feature distributions remain the same. With calibrated probabilistic predictions, label shift can be adjusted assuming access to class marginal probabilities [12]. However, miscalibration combined with label shift is common and remains a challenging problem [2, 14, 47, 41].
In this paper, we aim to address these issues in a twofold manner. Firstly, we develop a unified framework for recalibration that incorporates both calibration and sharpness in a principled manner. Secondly, we propose a composite estimator for recalibration in the presence of label shift that converges to the optimal recalibration. Our framework enables the adaptation of a classifier to the label-shifted domain in a sample-efficient manner.
Recalibration algorithms. Recalibration methods can be broadly categorized into parametric and nonparametric approaches. Parametric methods model the recalibration function in a parametric form and estimate the parameters using calibration data. Examples of parametric methods include Platt scaling [40], temperature scaling [18], Beta calibration [28], and Dirichlet calibration [27]. However, it has been reported that scaling methods are often less calibrated than supposed, and quantifying the degree of miscalibration can be challenging [29]. In contrast, nonparametric recalibration methods do not assume a specific parametric form for the recalibration function. These methods include histogram binning [50], isotonic regression [51], kernel density estimation [52, 42], splines [23], Gaussian processes [49], among others. Hybrid approaches, integrating both parametric and nonparametric techniques, have also been proposed. For instance, Kumar et al. [29] combine nonparametric histogram binning with parametric scaling to reduce variance and improve recalibration performance. Nevertheless, this hybrid approach is biased when its parametric assumptions fail. In this work, we consider a nonparametric histogram binning method called uniform-mass binning (UMB), which is asymptotically unbiased.
Histogram binning method. Histogram binning methods are widely used for recalibration due to their simplicity and adaptability. The binning schemes can be pre-specified (e.g., uniform-width binning [18]), data-dependent (e.g., uniform-mass binning [50]), or algorithm-induced [51]. When selecting a binning scheme, it is crucial to consider the trade-off between approximation and estimation. Coarser binning reduces estimation error (variance), leading to improved calibration, but at the expense of increased approximation error (bias), which diminishes sharpness. Thus, determining the optimal binning scheme and hyperparameters, such as the number of bins  $(B)$  remains an active area of research. [36] proposes a Bayesian binning method, but verifying the priors is often challenging. [44] suggests choosing the largest  $B$  that preserves monotonicity, which is heuristic and computationally inefficient. [20] offers a heuristic for choosing the largest  $B$  subject to a calibration constraint, lacking a quantitative characterization of sharpness. Our work builds upon the existing upper bounds for calibration risks of binning methods [29, 20] and derived upper bounds for a complementary risk component known as the sharpness risk. We quantitatively characterize the calibration-sharpness tradeoff, which yields an optimal choice for the number of bins that achieves the minimum risk.
Adaptation to label shift. Label shift presents a challenge in generalizing models trained on one distribution (source) to a different distribution (target). As such, adapting to label shift has received considerable attention in the literature [12, 45, 31, 3, 2, 14]. In practical scenarios, it is common to encounter model miscalibration and label shift simultaneously [47, 41]. Empirical observations have highlighted the crucial role of probability calibration in label shift adaptation [2, 13], which is justified by subsequent theories [14]. However, to the best of our knowledge, there has been no prior work that specifically addresses the recalibration with a limited amount of labeled data from the target distribution. Our theoretical analysis points out that using only target labels achieves risk bounds of the same order as the methods using only target features [31, 3, 14].
This paper contributes to the theory of recalibration across three key dimensions.
Firstly, we develop a comprehensive theory for recalibration in binary classification by adopting the mean-squared-error (MSE) decomposition framework commonly used in meteorology and space weather forecasting [5, 33, 8, 46]. Our approach formulates the probability recalibration problem as the minimization of a specific risk function, which can be orthogonally decomposed into calibration and sharpness components.
Secondly, utilizing the aforementioned framework, we derive a rigorous upper bound on the finite-sample risk for uniform-mass binning (UMB) (Theorem 1). Furthermore, we minimize this risk bound and demonstrate that the optimal number of bins for UMB, balancing calibration and sharpness, scales on the order of  $n^{1/3}$ , yielding the risk bound of order  $n^{-2/3}$ , where  $n$  denotes the sample size.
Lastly, we address the challenge of recalibrating classifiers for label shift when only a limited labeled sample from the target distribution is available, a challenging situation for a direct recalibration approach. We propose a two-stage approach: first recalibrating the classifier on the abundant source-domain data, and then transferring it to the label-shifted target domain. We provide a finite-sample guarantee for the risk of this composite procedure (Theorem 2). Notably, to control the risk under  $\varepsilon$ , our approach requires a much smaller sample size from the target distribution than a direct recalibration on the target sample  $(\Omega(\varepsilon^{-1})$  vs.  $\Omega(\varepsilon^{-3/2})$ , cf. Remark 4).
This paper is organized as follows. In Section 2, we introduce notation and provide an overview of calibration and sharpness. Section 3 introduces the notion of minimum-risk recalibration by defining the recalibration risk that takes into account both calibration and sharpness. In Section 4, we describe the uniform-mass histogram binning method for recalibration and provide a risk upper bound with rate analysis. We extend our approach to handle label shift in Section 5. To validate our theory and framework, we present numerical experiments in Section 6. Finally, in Section 7, we conclude the paper with a discussion and propose future research directions.
Let  $\mathbb{N}$  and  $\mathbb{R}$  denote the set of positive integers and the set of real numbers, respectively. For  $n\in \mathbb{N}$ , let  $[n]\coloneqq \{1,\ldots ,n\}$ . For  $x\in \mathbb{R}$ , let  $\lfloor x\rfloor = \max \{m\in \mathbb{Z}:m\leq x\}$ . For any finite set  $\mathcal{S} = \{s_i:i\in [n]\} \subset \mathbb{R}$  and any  $k\in [n]$ , we let  $s_{(k)}$  denote the  $k$ -th order statistic, which is the  $k$ -th smallest element in  $\mathcal{S}$ .
Let  $(\Omega, \mathcal{E}, P)$  denote a generic probability space. For an event  $A \in \mathcal{E}$ , the indicator function  $\mathbb{1}_A: \Omega \to \{0, 1\}$  is defined such that  $\mathbb{1}_A(x) = 1$  if and only if  $x \in A$ . We write  $A$  happens  $P$ -almost surely if  $P(A) = 1$ . For a probability measure  $P$ , define  $\mathbb{E}_P$  as the expectation, with subscript  $P$  omitted when the underlying probability measure is clear. For a probability measure  $P$  and a random variable  $X: \Omega \to \mathbb{R}^d$ , let  $P_X := P \circ X^{-1}$  be the probability measure induced by  $X$ .
Letting  $f, g: \mathbb{R} \to \mathbb{R}$ , we write  $f(x) = O(g(x))$  as  $x \to \infty$  if there exist  $M > 0$  and  $x_0 > 0$  such that  $|f(x)| \leq M g(x)$  for all  $x \geq x_0$ . Likewise, we write  $f(x) = \Omega(g(x))$  if  $g(x) = O(f(x))$ . We write  $f(x) \asymp g(x)$  if  $f(x) = O(g(x))$  and  $g(x) = O(f(x))$ . We write  $f(x) = \tilde{O}(g(x))$  if there is  $k \geq 1$  such that  $f(x) = O(g(x) \log^k (g(x)))$ .
2.2 Calibration and sharpness
Consider the binary classification problem; let  $X \in \mathcal{X}$  and  $Y \in \mathcal{Y} \coloneqq \{0,1\}$  denote the feature and label random variables. Letting  $P$  denote a probability measure, we want to construct a function  $f: \mathcal{X} \to \mathcal{Z} = [0,1]$  that estimates the conditional probability, i.e.,
$$
f (X) \approx P [ Y = 1 \mid X ]. \tag {1}
$$
Since estimating the probability conditioned on the high dimensional  $X$  is difficult, the notion of calibration captures the intuition of (1) in a weaker sense [33, 9, 19].
Definition 1. A function  $f: \mathcal{X} \to \mathcal{Z}$  is (perfectly) calibrated with respect to probability measure  $P$ , if
$$
f (X) = P \left[ Y = 1 \mid f (X) \right] \quad P - a l m o s t s u r e l y.
$$
Calibration itself does not guarantee a useful predictor. For instance, a constant predictor  $f(X) = \mathbb{E}Y$  is perfectly calibrated, but it does not change with the features. Such a predictor lacks sharpness [26], also known as resolution [35], another desired property which measures the variance in the target  $Y$  explained by the probabilistic prediction  $f(X)$ .
Definition 2. The sharpness of a function  $f: \mathcal{X} \to \mathcal{Z}$  with respect to probability measure  $P$  refers to the quantity
$$
\operatorname {V a r} \left(\mathbb {E} [ Y \mid f (X) ]\right) = \mathbb {E} \left[ \left(\mathbb {E} [ Y \mid f (X) ] - \mathbb {E} [ Y ]\right) ^ {2} \right].
$$
The following decomposition of the mean squared error (MSE) suggests why it is desirable for a classifier  $f$  to be calibrated and have high sharpness; note that  $\mathrm{Var}[Y]$  is a quantity intrinsic to the problem, unrelated to  $f$ .
$$
\operatorname {M S E} (f) := \underbrace {\mathbb {E} \left[ \left(Y - f (X)\right) ^ {2} \right]} _ {\text {m e a n - s q u a r e d e r r o r}} = \operatorname {V a r} [ Y ] - \underbrace {\operatorname {V a r} \left(\mathbb {E} \left[ Y \mid f (X) \right]\right)} _ {\text {s h a r p n e s s}} + \underbrace {\mathbb {E} \left[ \left(f (X) - \mathbb {E} [ Y \mid f (X) ]\right) ^ {2} \right]} _ {\text {l a c k o f c a l i b r a t i o n}} \tag {2}
$$
3 Optimal recalibration
For an arbitrary predictor  $f: \mathcal{X} \to \mathcal{Z}$ , the aim of recalibration is to identify a post-processing function  $h: \mathcal{Z} \to \mathcal{Z}$  such that  $h \circ f$  is perfectly calibrated while maintaining the sharpness of  $f$  as much as possible. The calibration and sharpness can be evaluated using the following two notions of risks. We suppress the dependency of risks on  $f$  and  $P$  when it leads to no confusion.
Definition 3. Let  $f: \mathcal{X} \to \mathcal{Z}$  and  $h: \mathcal{Z} \to \mathcal{Z}$ . The calibration risk of  $h$  over  $F$  under  $P$  is defined as
$$
R ^ {\operatorname {c a l}} (h) = R _ {P} ^ {\operatorname {c a l}} (h; f) := \mathbb {E} _ {P} \left[ \left(h \circ f (X) - \mathbb {E} _ {P} [ Y \mid h \circ f (X) ]\right) ^ {2} \right]. \tag {3}
$$
Definition 4. Let  $f: \mathcal{X} \to \mathcal{Z}$  and  $h: \mathcal{Z} \to \mathcal{Z}$ . The sharpness risk of  $h$  over  $f$  under  $P$  is defined as
$$
R ^ {\mathrm {s h a}} (h) = R _ {P} ^ {\mathrm {s h a}} (h; f) := \mathbb {E} _ {P} \left[ \left(\mathbb {E} _ {P} [ Y \mid h \circ f (X) ] - \mathbb {E} _ {P} [ Y \mid f (X) ]\right) ^ {2} \right]. \tag {4}
$$
Note that the calibration risk  $R^{\mathrm{cal}}(h; f) = 0$  if and only if  $h \circ f$  is perfectly calibrated, cf. Definition 1. The sharpness risk  $R^{\mathrm{sha}}(h; f)$  quantifies the decrement in sharpness of  $f$  incurred by applying the recalibration map  $h$ , and  $R^{\mathrm{sha}}(h; f) = 0$  when  $h$  is injective [29].
Next we define a comprehensive notion of risk that we will use to evaluate recalibration functions.
Definition 5. Let  $f: \mathcal{X} \to \mathcal{Z}$  and  $h: \mathcal{Z} \to \mathcal{Z}$ . The recalibration risk of  $h$  over  $F$  under  $P$  is defined as
$$
R (h) = R _ {P} (h; f) := \mathbb {E} _ {P} \left[ \left(h \circ f (X) - \mathbb {E} _ {P} [ Y \mid f (X) ]\right) ^ {2} \right]. \tag {5}
$$
The following proposition shows that the recalibration risk can be decomposed into calibration risk and sharpness risk. The proof is deferred to Appendix A.
Proposition 1 (Decomposition of recalibration risk). For any  $f: \mathcal{X} \to \mathcal{Z}$  and any  $h: \mathcal{Z} \to \mathcal{Z}$ ,
$$
R _ {P} (h; f) = R _ {P} ^ {\operatorname {c a l}} (h; f) + R _ {P} ^ {\operatorname {s h a}} (h; f). \tag {6}
$$
Note that  $R_P(h) = 0$  if and only if  $R_P^{\mathrm{cal}}(h) = 0$  and  $R_P^{\mathrm{sha}}(h) = 0$ . This happens if and only if  $h \circ f$  is calibrated, and the recalibration  $h$  preserves the sharpness of  $f$  in predicting  $Y$ .
If  $R(h) = 0$ , then we call  $h$  an optimal recalibration function (or minimum-risk recalibration function) of  $f$  under  $P$ . Let  $h_{f,P}^{*}: \mathcal{Z} \to \mathcal{Z}$  be the function
$$
h _ {f, P} ^ {*} (z) = \left\{ \begin{array}{l l} \mathbb {E} _ {P} [ Y \mid f (X) = z ], & \text {i f} z \in \operatorname {s u p p} P _ {f (X)}, \\ 0, & \text {i f} z \notin \operatorname {s u p p} P _ {f (X)}. \end{array} \right. \tag {7}
$$
Then  $R(h_{f,P}^{*}) = 0$ . Indeed,  $h$  is a minimum-risk recalibration function of  $f$  if and only if  $h = h_{f,P}^{*}$ .  $P_{Z}$ -almost surely.
Problem 1 (Recalibration). Suppose that we have a measurable function  $f: \mathcal{X} \to \mathcal{Z}$  and a dataset  $\mathcal{D} = \{(x_i, y_i): i \in [n]\}$  that is an independent and identically distributed (IID) sample drawn from  $P$ . The goal of recalibration is to estimate a recalibration function  $\hat{h} \approx h_{f,P}^*$  using  $f$  and  $\mathcal{D}$ .
Problem 1 can be viewed as a regression problem, where we estimate the function form of  $\mathbb{E}[Y\mid Z]$  from data  $\{(z_i,y_i):i\in [n]\}$ , where  $z_{i} = f(x_{i}),\forall i\in [n]$ .
4 Recalibration via uniform-mass binning
4.1 Uniform-mass binning algorithm for recalibration
Given a dataset of prediction-label pairs  $\{(z_i, y_i) \in \mathcal{Z} \times \mathcal{Y} : i \in [n]\}$ , the histogram binning calibration method partitions  $\mathcal{Z} = [0, 1]$  into a set of smaller bins, and estimate  $\mathbb{E}[Y \mid Z]$  by taking the average in each bin. We consider the uniform mass binning, constructed using quantiles of predicted probabilities.
Definition 6 (Uniform mass binning). Let  $S = \{z_{i} \in [0,1] : i \in [n]\}$ . A binning scheme  $\mathcal{B} = \{I_1, I_2, \ldots, I_B\}$  is the uniform-mass binning (UMB) scheme induced by  $S$  if
$$
I _ {1} = \left[ u _ {0}, u _ {1} \right], \quad \text {a n d} \quad I _ {b} = \left(u _ {b - 1}, u _ {b} \right] \forall b \in [ B ] \backslash \{1 \}, \tag {8}
$$
where  $u_0 = 0$ ,  $u_B = 1$ , and  $u_b = z_{(\lfloor nb / B \rfloor)}$  for  $b \in [B - 1]$ .
For our subsequent discussions, we make the following assumptions on the distribution of  $Y$  and  $Z$ :
(A1) The cumulative distribution function of  $Z$ , denoted by  $F_{Z}$ , is absolutely continuous.
(A2)  $h_{f,P}^{*}$ , defined in (7), is monotonically non-decreasing on  $\operatorname{supp} P_Z$ .
(A3) There exists  $K > 0$  such that if  $z_{1}\leq z_{2}$  , then  $h_{f,P}^{*}(z_{2}) - h_{f,P}^{*}(z_{1})\leq K\cdot \bigl (F_{Z}(z_{2}) - F_{Z}(z_{1})\bigr)$
Assumption (A1) is made for the sake of analytical convenience without loss of generality; see discussions in [20, Appendix C]. An important implication of Assumption (A1) is that all intervals in  $\mathcal{B}$  are non-empty  $P_Z$ -almost surely if  $S = \{Z_i : i \in [n]\}$  are IID under  $P_Z$ . Assumption (A2) makes sure  $Z$  is informative to preserve the rankings of  $P[Y \mid Z]$  [51]. Lastly, Assumption (A3) posits that  $Z$  is sufficiently informative that  $P[Y = 1 \mid Z = z]$  does not change too rapidly in any interval  $I$  where  $P_Z(I)$  is small. This assumption is mild but not trivial; see Appendix B.2.
Now we describe how to construct a recalibration function  $\hat{h}$  using UMB.
1. Given  $\{(z_i, y_i) : i \in [n]\}$ , and  $B \in \mathbb{N}$ , let  $\mathcal{B} = \{I_1, I_2, \ldots, I_B\}$  be the UMB scheme of size  $B$  induced by  $\{z_i : i \in [n]\}$ .
2. Let  $\hat{h} = \hat{h}_{\mathcal{B}}: \mathcal{Z} \to \mathcal{Z}$  be a function such that
$$
\hat {h} (z) = \sum_ {I \in \mathcal {B}} \hat {\mu} _ {I} \cdot \mathbb {1} _ {I} (z) \quad \text {w h e r e} \quad \hat {\mu} _ {I} := \frac {\sum_ {i = 1} ^ {n} y _ {i} \cdot \mathbb {1} _ {I} \left(z _ {i}\right)}{\sum_ {i = 1} ^ {n} \mathbb {1} _ {I} \left(z _ {i}\right)}, \forall I \in \mathcal {B}. \tag {9}
$$
4.2 Theoretical guarantees
Here we establish a high-probability upper bound on the recalibration risk  $R(\hat{h})$  for the UMB estimator,  $\hat{h}$  defined in (9), which converges to 0 as the sample size  $n$  increases to  $\infty$ .
Theorem 1. Let  $P$  be a probability measure and  $f:\mathcal{X}\to \mathcal{Z}$  be a measurable function. Suppose that (A1) & (A2) hold. Let  $\mathcal{B}$  be the UMB scheme induced by an IID sample of  $f(X)$ , and let  $\hat{h} = \hat{h}_{\mathcal{B}}$  be the recalibration function based on  $\mathcal{B}$ , cf. (9). Then there exists a universal constant  $c > 0$  such that for any  $\delta \in (0,1)$ , if  $n\geq c\cdot |\mathcal{B}|\log \left(\frac{2|\mathcal{B}|}{\delta}\right)$ , then with probability at least  $1 - \delta$ ,
$$
R ^ {\mathrm {c a l}} (\hat {h}) \leq \left(\sqrt {\frac {\log (4 | \mathcal {B} | / \delta)}{2 (\lfloor n / | \mathcal {B} | \rfloor - 1)}} + \frac {1}{\lfloor n / | \mathcal {B} | \rfloor}\right) ^ {2}, \quad \text {a n d} \quad R ^ {\mathrm {s h a}} (\hat {h}) \leq \left\{ \begin{array}{l l} \frac {2}{| \mathcal {B} |}, \\ \frac {8 K ^ {2}}{| \mathcal {B} | ^ {2}} \end{array} \right. \quad i f (A 3) h o l d s.
$$
Remark 1. We note that the upper bound on  $R^{\mathrm{cal}}(\hat{h})$  in Theorem 1 coincides with the result presented in [20] up to a constant factor in the failure probability. However, Theorem 1 provides an additional upper bound on  $R^{\mathrm{sha}}(\hat{h})$ , thereby effectively managing the overall recalibration risk  $R(\hat{h})$ .
Optimal choice of the number of bins  $|\mathcal{B}|$ . Based on Theorem 1, for any fixed sample size  $n$ , as the number of bins  $B = |\mathcal{B}|$  increases, the calibration risk bound increases, while the sharpness risk bound decreases. This trade-off suggests we may get an optimal number of bins  $B$  by minimizing the upper bound for the overall risk,  $R(\hat{h}) = R^{\mathrm{cal}}(\hat{h}) + R^{\mathrm{sha}}(\hat{h})$ .
For the simplicity of our analysis, we assume  $n$  is divisible by  $B$ , and Assumption (A3) holds. Firstly, observe that  $n / B \geq 2$ , and thus,  $n / B - 1 \geq n / (2B)$ . Also, since  $B \geq 1$  and  $\delta < 1$ ,  $\log \left(\frac{4B}{\delta}\right) \geq 1$ , it follows that  $\frac{1}{n / B} \leq \sqrt{\frac{1}{2(n / B - 1)} \log \left(\frac{4B}{\delta}\right)} \leq \sqrt{\frac{1}{n / B} \log \left(\frac{4B}{\delta}\right)}$ . Thus, we obtain a simplified risk bound  $R(\hat{h}) \leq \zeta(B; n, \delta)$  where  $\zeta(B; n, \delta) := \frac{4B}{n} \log \left(\frac{4B}{\delta}\right) + \frac{8K^2}{B^2}$ . With  $\log B$  terms ignored, minimizing  $\zeta(B; n, \delta)$  over  $B$  yields optimal  $B^*$  and resulting risk bounds, respectively:
$$
B ^ {*} \asymp n ^ {1 / 3} \cdot \log^ {- 1 / 3} (1 / \delta), \tag {10}
$$
$$
R (\hat {h}) \asymp R ^ {\operatorname {c a l}} (\hat {h}) \asymp R ^ {\operatorname {s h a}} (\hat {h}) = O \left(B ^ {* - 2}\right) = O \left(n ^ {- 2 / 3} \cdot \log^ {2 / 3} (1 / \delta)\right). \tag {11}
$$
Furthermore, the asymptotic risk bound in (11) implies that  $R(\hat{h})$ ,  $R^{\mathrm{cal}}(\hat{h})$ , and  $R^{\mathrm{sha}}(\hat{h})$  are all bounded by  $\varepsilon$  with high probability if the sample size
$$
n = \Omega \left(\varepsilon^ {- 3 / 2}\right). \tag {12}
$$