SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xingdong Feng $^{1}$ , Xin He $^{1}$ , Caixing Wang $^{1,*}$ , Chao Wang $^{1}$ , Jingnan Zhang $^{2}$
$^{2}$ International Institute of Finance, School of Management,
{feng.xingdong, he.xin17}@mail.shufe.edu.cn
{wang.caixing, wang.chao}@stu.shufe.edu.cn
jnzhang@ustc.edu.cn
Covariate shift occurs prevalently in practice, where the input distributions of the source and target data are substantially different. Despite its practical importance in various learning problems, most of the existing methods only focus on some specific learning tasks and are not well validated theoretically and numerically. To tackle this problem, we propose a unified analysis of general nonparametric methods in a reproducing kernel Hilbert space (RKHS) under covariate shift. Our theoretical results are established for a general loss belonging to a rich loss function family, which includes many commonly used methods as special cases, such as mean regression, quantile regression, likelihood-based classification, and margin-based classification. Two types of covariate shift problems are the focus of this paper and the sharp convergence rates are established for a general loss function to provide a unified theoretical analysis, which concurs with the optimal results in literature where the squared loss is used. Extensive numerical studies on synthetic and real examples confirm our theoretical findings and further illustrate the effectiveness of our proposed method.
Covariate shift is a phenomenon that commonly occurs in machine learning, where the distribution of input features (covariates) changes between the source (or training) and target (or test) data, while the conditional distribution of output values given covariates remains unchanged (Shimodaira, 2000; Pan & Yang, 2010). Such a phenomenon is illustrated in Figure 1 where the learned predictive function from the source data may significantly differ from the true function. Thus, the prediction performance can be largely degraded since the predictive function has not been trained on data that accurately represents the target environment. Covariate shift can arise in a variety of domains, including medicine and healthcare (Wei et al., 2015; Hajiramezanali et al., 2018), remote sensing (Tuia et al., 2011), and natural language and speech processing (Yamada et al., 2009; Fei & Liu, 2015). Various factors contribute to covariate shift, such as data sampling bias (e.g., patient demographics in healthcare applications), changes in the studied population (e.g., vocabulary evolution in natural language processing), or measurement and observational errors (e.g., sensor noise or calibration errors in the sensor domain). Compared to the well-studied supervised learning without such a distribution mismatch (Györfi et al., 2002), there still exists some gap in both theoretically and numerically understanding the influence of the covariate shift under various kernel-based learning problems.
*Caixing Wang is the corresponding author and all the authors contributed equally to this paper and their names are listed in alphabetical ordering.
This paper provides a unified analysis of the kernel-based methods under covariate shift. Moreover, a general loss function is considered, which is allowed to belong to a rich loss function family and thus includes many commonly used methods as special cases, such as mean regression, quantile regression, likelihood-based classification, and margin-based classification. This paper considers two types of covariate shift problems in which the importance ratio is assumed to be uniformly bounded or to have a bounded second moment. A unified theoretical analysis has been provided that the sharp convergence rates are established for a general loss function under two evaluation metrics, which concurs with the optimal results in Ma et al. (2023) where the squared loss is used. Our theoretical findings are also validated by a variety of synthetic and real-life examples.
(b)
Figure 1: (a) The probability density functions of normal distributions with  $\mu_1 = 0$ ,  $\sigma^2 = 0.4$  that the source data is driven from and  $\mu_1 = 1.5$ ,  $\sigma^2 = 0.6$  that the target data is driven from, respectively; (b) the learned function trained by using the source data and the true mean regression function. Note that the considered example serves as an illustration that satisfies case (ii) in Section 2.3.
Our contributions. The contributions of this paper are multi-fold. We propose a unified analysis of the kernel-based methods under covariate shift, which provides an insightful understanding of the influences of covariate shift on the kernel-based methods both theoretically and numerically. By utilizing empirical process in learning theory, we investigate the theoretical behaviors of several estimators under various conditions, with particular emphasis on different importance ratio cases between the source and target distributions. Specifically, we show that the unweighted estimator achieves the optimal learning rates in terms of both  $\mathcal{L}^2$ -error and the excess risk with respect to the target distribution under the uniformly bounded case. Yet, the unweighted estimator is suboptimal under the bounded second moment case. Then, we construct a weighted estimator by using an appropriate truncated ratio, which again attains a sharp convergence rate. Unlike kernel ridge regression (KRR), the estimator for the general loss does not have an explicit form, making many classical analysis techniques inapplicable (Lian, 2022) and in our technical proofs, the theoretical derivation is much more involved due to the considered general loss function. Numerous experiments on synthetic data and multi-source real data with various loss functions confirm our theoretical findings. To some extent, this paper provides a comprehensive study of the numerical performance of the kernel-based methods under various scenarios with covariate shift.
Related work. Some most related works including covariate shift adaptation and importance ratio estimation are presented below.
Covariate shift adaptation. Shimodaira (2000) investigates the impact of covariate shift in parametric regression with maximum likelihood estimation and proposes an importance-weighting (IW) estimator, which has a significant improvement when the misspecification does exist. Sugiyama & Müller (2005b) further extend this work by analyzing an unbiased estimator for  $\mathcal{L}^2$ -error. These fundamental works also motivate a variety of follow-up studies under the parametric setting Sugiyama & Storkey (2006); Yamazaki et al. (2007); Wen et al. (2014); Lei et al. (2021). Beyond the parametric setting, Kpotufe & Martinet (2021) consider the nonparametric classification problem and provide a new minimax result that concisely captures the relative benefits of source and target labeled data under covariate shift. Focusing on the Hölder continuous regression function class, Pathak et al. (2022) propose a new measure of distribution mismatch between the source and target distributions. Ma et al. (2023) and Gogolashvili et al. (2023) establish the optimal learning rates for KRR over RKHS. Recently, covariate shift in overparameterized models, such as high-dimensional models and neural networks, has also drawn tremendous attention from relevant researchers (Hendrycks & Dietterich, 2019; Byrd & Lipton, 2019; Hendrycks et al., 2021; Tripuraneni et al., 2021).
Importance ratio estimation. A straightforward approach to estimating the importance ratio is separately estimating the source and target densities by kernel density estimation (Sugiyama & Müller, 2005a; Baktashmotlagh et al., 2014) and then computing the ratio. In practice, it is more efficient to directly estimate the importance ratio by minimizing some discrepancy measures between distributions, including kernel mean matching (Huang et al., 2006; Gretton et al., 2009), Kullback-Leibler divergence (Sugiyama et al., 2007a,b, 2012) and non-negative Bregman divergence (Kato & Teshima, 2021).
Notation. In this paper, we use  $C$  to denote a universal constant that may vary from line to line. For two sequence  $a_{n}$  and  $b_{n}$ , we say  $a_{n} \asymp b_{n}$  if and only if  $a_{n} = O(b_{n})$  and  $b_{n} = O(a_{n})$  hold.
2.1 Learning within a rich family of loss functions
Suppose that the random pair  $(\mathbf{x},y)$  is drawn from some unknown distribution  $P_{\mathbf{x},y}$  with  $\mathbf{x} = (x_1,\ldots ,x_p)^\top \in \mathcal{X}$ , where  $\mathcal{X}\subset \mathcal{R}^p$  is a compact support and  $y\in \mathcal{V}\subset \mathcal{R}$ . In the literature of machine learning, the true target function  $f^{*}$  is often defined as the minimizer of the expected risk with a pre-specified loss function  $L(\cdot ,\cdot):\mathcal{Y}\times \mathcal{R}\to \mathcal{R}^{+}$  that
$$
f ^ {*} := \operatorname {a r g m i n} _ {f} \mathcal {E} ^ {L} (f) = \operatorname {a r g m i n} _ {f} E [ L (y, f (\mathbf {x})) ]. \tag {1}
$$
Throughout this paper, we consider a general loss function belonging to a rich loss function family that  $L(y,\cdot)$  is assumed to be convex and locally  $c_{L}$ -Lipschitz continuous (Wainwright, 2019; Dasgupta et al., 2019), that is for some  $V\geq 0$ , there exists a constant  $c_{L} > 0$  such that  $|L(y,\omega) - L(y,\omega^{\prime})|\leq c_{L}|\omega -\omega^{\prime}|$  holds for all pairs  $\omega ,\omega^{\prime}\in [-V,V]$  and  $y\in \mathcal{V}$ . A variety of popularly used loss functions satisfy these two conditions, including
- Squared loss:  $L(y, f(\mathbf{x})) = (y - f(\mathbf{x}))^2$  with  $c_{L} = 2(M_{y} + V)$ , for any  $|y| \leq M_{y}$  with constant  $M_{y} > 0$ ;
- Check loss:  $L(y, f(\mathbf{x})) = (y - f(\mathbf{x})) \left( \tau - I_{\{y \leq f(\mathbf{x})\}} \right)$  with  $c_L = 1$  and quantile level  $\tau$ ;
- Huber loss:  $L(y, f(\mathbf{x})) = (y - f(\mathbf{x}))^2$ , if  $|y - f(\mathbf{x})| \leq \delta$ ;  $\delta |y - f(\mathbf{x})| - \frac{1}{2} \delta^2$ , otherwise, with  $c_L = \delta$
- Logistic loss:  $L(y, f(\mathbf{x})) = \log (1 + \exp (-yf(\mathbf{x}))) / \log 2$  with  $c_{L} = (\log 2)^{-1}e^{V} / (1 + e^{V})$ ;
- Hinge loss:  $L(y, f(\mathbf{x})) = (1 - yf(\mathbf{x}))_{+}$  with  $c_{L} = 1$ .
Note that the explicit form of  $f^{*}$  may differ from one loss to another. For example, for the squared loss,  $f^{*}(\mathbf{x}) = E[y|\mathbf{x}]$ ; for the check loss,  $f^{*}(\mathbf{x}) = Q_{\tau}(y|\mathbf{x})$  with  $Q_{\tau}(y|\mathbf{x}) = \inf \{y: P(Y \leq y|\mathbf{x}) \geq \tau\}$ ; for the hinge loss,  $f^{*} = \mathrm{sign}(P(y = 1|\mathbf{x}) - 1/2)$  with  $\mathrm{sign}(\cdot)$  denoting the sign function. Moreover, we require  $f^{*} \in \mathcal{H}_{K}$ , where  $\mathcal{H}_{K}$  denotes the RKHS induced by a pre-specified kernel function  $K(\cdot, \cdot): \mathcal{X} \times \mathcal{X} \to \mathcal{R}$ . In practice, one of the most important tasks in machine learning is to learn  $f^{*}$  from the source data and use the estimated predictive function  $\widehat{f}$  for prediction in the target data.
2.2 Measurement under source and target distributions
Classical learning problems often explicitly or implicitly assume that the source and target data are drawn from the same distribution. Precisely, the source data comes from a joint distribution  $P_{S} = P_{y|\mathbf{x}}P_{\mathbf{x}}^{S}$  where  $P_{y|\mathbf{x}}$  is the conditional distribution and  $P_{\mathbf{x}}^{S}$  is the input distribution with density function  $\rho_{\mathbf{x}}^{S}$ . Then, the performance of the estimated predictive function  $\hat{f}$  is usually evaluated by the  $\mathcal{L}^2 (P_{\mathbf{x}}^{S})$ -error or the excess risk error with respect to  $P_{S}$  that
$$
\| \widehat {f} - f ^ {*} \| _ {S} ^ {2} = E _ {\mathbf {x} \sim S} \big [ \big (\widehat {f} (\mathbf {x}) - f ^ {*} (\mathbf {x}) \big) ^ {2} \big ], \quad \mathcal {E} _ {S} ^ {L} (\widehat {f}) - \mathcal {E} _ {S} ^ {L} (f ^ {*}) = E _ {S} \big [ L (y, \widehat {f} (\mathbf {x})) - L (y, f ^ {*} (\mathbf {x})) \big ],
$$
where  $E_{\mathbf{x} \sim S}$  and  $E_S$  are the expectation over  $P_{\mathbf{x}}^S$  and  $P_S$  conditioning on the observed data, respectively. When covariate shift occurs, the target data may come from some totally different joint
distribution  $P_{T} = P_{y|\mathbf{x}}P_{\mathbf{x}}^{T}$  in the sense that although  $P_{y|\mathbf{x}}$  representing the regression or classification rule remains invariant,  $P_{\mathbf{x}}^{T}$  differs significantly from  $P_{\mathbf{x}}^{S}$ . We also assume that  $P_{\mathbf{x}}^{T}$  has density function  $\rho_{\mathbf{x}}^{T}$ . Obviously, it is reasonable to evaluate  $\widehat{f}$  under  $P_{T}$  instead of  $P_{S}$ . Then, our primary interest is the  $\mathcal{L}^2 (P_{\mathbf{x}}^T)$ -error or the excess risk error with respect to  $P_{T}$  that
$$
\| \widehat {f} - f ^ {*} \| _ {T} ^ {2} = E _ {\mathbf {x} \sim T} \big [ \big (\widehat {f} (\mathbf {x}) - f ^ {*} (\mathbf {x}) \big) ^ {2} \big ], \quad \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) = E _ {T} \big [ L (y, \widehat {f} (\mathbf {x})) - L (y, f ^ {*} (\mathbf {x})) \big ],
$$
where  $E_{\mathbf{x} \sim T}$  and  $E_T$  are the expectation over  $P_{\mathbf{x}}^T$  and  $P_T$  conditioning on the observed data, respectively. In the rest of this paper,  $f^*$  is defined as the optimal function under the target distribution such that  $f^* = \operatorname{argmin}_f E_T[L(y, f(\mathbf{x}))]$ .
2.3 Kernel-based estimation under covariate shift and importance ratio correction
Suppose that random sample  $\mathcal{Z}_n^S = \left\{\left(\mathbf{x}_i^S,y_i^S\right)\right\}_{i = 1}^n$  are i.i.d. drawn from the source distribution  $P_{S}$ . We consider the classical nonparametric estimation problem in RKHS (Vapnik, 1999) that
$$
\widehat {f} := \underset {f \in \mathcal {H} _ {K}} {\operatorname {a r g m i n}} \frac {1}{n} \sum_ {i = 1} ^ {n} L \left(y _ {i} ^ {S}, f \left(\mathbf {x} _ {i} ^ {S}\right)\right) + \lambda \| f \| _ {K} ^ {2}, \tag {2}
$$
where  $\lambda$  is the regularization parameter. Without covariate shift, the first term on the right side of (2) is an unbiased estimator of  $\mathcal{E}_T^L(f)$ . However, it becomes biased when covariate shift occurs, and thus may lead to inaccurate predictive estimator. To tackle this issue, we consider the importance ratio measuring the discrepancy between distributions that is  $\phi(\mathbf{x}) = \rho_{\mathbf{x}}^T(\mathbf{x}) / \rho_{\mathbf{x}}^S(\mathbf{x})$ , for any  $\mathbf{x} \in \mathcal{X}$ , and we notice that  $\int \int L(y, f(\mathbf{x})) \rho_{\mathbf{x}}^T(\mathbf{x}) dP_{y|\mathbf{x}} d\mathbf{x} = \int \int \phi(\mathbf{x}) L(y, f(\mathbf{x})) \rho_{\mathbf{x}}^S(\mathbf{x}) dP_{y|\mathbf{x}} d\mathbf{x}$ . Inspired by this, the weighted version of (2) can be used, which leads to an importance ratio weighted (IRW) estimator that
$$
\widetilde {f} ^ {\phi} := \underset {f \in \mathcal {H} _ {K}} {\operatorname {a r g m i n}} \frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(\mathbf {x} _ {i} ^ {S}\right) L \left(y _ {i} ^ {S}, f \left(\mathbf {x} _ {i} ^ {S}\right)\right) + \lambda \| f \| _ {K} ^ {2}. \tag {3}
$$
Throughout this paper, we focus on two types of the importance ratio that
(i)  $\phi (\mathbf{x})$  is  $\alpha$  -uniformly bounded that is  $\sup_{\mathbf{x}\in \mathcal{X}}\phi (\mathbf{x})\leq \alpha$  , for some positive constant  $\alpha$
(ii)  $\phi (\mathbf{x})$  's second moment is bounded that is  $E_{\mathbf{x}\sim S}[\phi^2 (\mathbf{x})]\leq \beta^2$  , for some constant  $\beta^2\geq 1$
Note that case (i) reduces to the classical case without covariate shift if  $\alpha = 1$ . Yet, the bounded case is somewhat restrictive, and a much weaker condition is considered in case (ii). It is also clear that case (i) can be regarded as a special case of case (ii) by taking  $\beta^2 = \alpha$ . To see this, we have  $E_{\mathbf{x}\sim S}[\phi (\mathbf{x})^2 ] = E_{\mathbf{x}\sim T}[\phi (\mathbf{x})]\leq \alpha$ . It is interesting to notice that these two cases are related to Renyi divergence (Renyi, 1961) between  $\rho_{\mathbf{x}}^{S}$  and  $\rho_{\mathbf{x}}^{T}$ . Specifically, the conditions in cases (i) and (ii) are equivalent to requiring  $D_{\infty}(\rho_{\mathbf{x}}^{T}||\rho_{\mathbf{x}}^{S})$  and  $D_{2}(\rho_{\mathbf{x}}^{T}||\rho_{\mathbf{x}}^{S})$  being bounded (Cortes et al., 2010), respectively. Moreover, as pointed out by Ma et al. (2023), (3) may result in significant inflation of the variance, largely due to the unbounded importance ratio. Thus, it is natural to consider a truncated importance ratio with a pre-specified threshold value  $\gamma_{n} > 0$  (Ma et al., 2023; Gogolashvili et al., 2023). Precisely, we consider the following truncation
$$
\phi_ {n} (\mathbf {x}) := \min  \left\{\phi (\mathbf {x}), \gamma_ {n} \right\}.
$$
Accordingly, the following truncated important ratio weighted (TIRW) estimator is considered that
$$
\widehat {f} ^ {\phi} := \underset {f \in \mathcal {H} _ {K}} {\operatorname {a r g m i n}} \frac {1}{n} \sum_ {i = 1} ^ {n} \phi_ {n} \left(\mathbf {x} _ {i} ^ {S}\right) L \left(y _ {i} ^ {S}, f \left(\mathbf {x} _ {i} ^ {S}\right)\right) + \lambda \| f \| _ {K} ^ {2}, \tag {4}
$$
where the theoretical suggestion of the choice of  $\gamma_{n}$  is provided in Section 3.
In this section, we provide a unified theoretical analysis of the unweighted estimator in (2) and the TIRW estimator in (4) under two types of importance ratio. Let  $\mathcal{H}_K$  denote the RKHS induced by a
symmetric, continuous, and positive semi-definite kernel function  $K(\cdot, \cdot): \mathcal{X} \times \mathcal{X} \to \mathcal{R}$ . Under some regularity conditions, Mercer's theorem (Mercer, 1909) guarantees that  $K$  has an eigen-expansion of the form
$$
K (\mathbf {x}, \mathbf {x} ^ {\prime}) = \sum_ {j = 1} ^ {\infty} \mu_ {j} \psi_ {j} (\mathbf {x}) \psi_ {j} (\mathbf {x} ^ {\prime}),
$$
where  $\{\mu_j\}_{j\geq 1}$  are the eigenvalues and  $\{\psi_j\}_{j\geq 1}$  are the orthonormal eigenfunctions of  $\mathcal{L}^2 (\mathcal{X},P_{\mathbf{x}}^T) =$ $\left\{f:\int_{\mathcal{X}}f^{2}(\mathbf{x})\rho_{\mathbf{x}}^{T}(\mathbf{x})d\mathbf{x} <   \infty \right\}$ . We expand the kernel in  $\mathcal{L}^2 (\mathcal{X},P_{\mathbf{x}}^T)$  for deriving bounds under the target distribution. For any  $f\in \mathcal{H}_K$ , we have  $f = \sum_{j = 1}^{\infty}a_{j}\psi_{j}$  with  $a_{j} = \int_{\mathcal{X}}f(\mathbf{x})\psi_{j}(\mathbf{x})\rho_{\mathbf{x}}^{T}(\mathbf{x})d\mathbf{x}$  and the RKHS-norm is  $\| f\| _K^2 = \sum_{j = 1}^\infty a_j^2 /\mu_j <   \infty$ . The kernel complexity function of  $\mathcal{H}_K$  is then given as  $R(\delta) = \sqrt{\frac{1}{n}\sum_{j = 1}^{\infty}\min(\delta^2,\mu_j\|f^*\|_K^2)}$ . In literature,  $R(\delta)$  is important to quantify the localized Rademacher complexity of  $\mathcal{H}_K$ , which helps to build tight bounds in kernel-based methods (Mendelson, 2002; Koltchinskii & Yuan, 2010). For theoretical simplicity, we assume  $\| f^{*}\|_{K} = 1$  in the rest of this paper, which is commonly considered in machine learning literature (Yang et al., 2017; Li et al., 2021). The following technical assumptions are required for the theoretical analysis.
Assumption 1: There exists some constant  $\kappa >0$  such that  $\sup_{\mathbf{x}\in \mathcal{X}}|K(\mathbf{x},\mathbf{x})|\leq \kappa$ , and the eigenfunctions  $\{\psi_j\}_{j\geq 1}$  are uniformly bounded such that  $\sup_{\mathbf{x}\in \mathcal{X}}|\psi_j(\mathbf{x})|\leq 1$  for all  $j\geq 1$ .
Assumption 2: For some sufficiently small constant  $u > 0$ , we assume  $E_{I}[L(y,f(\mathbf{x}))] - E_{I}[L(y,f^{*}(\mathbf{x}))]\geqslant c_{0}\| f - f^{*}\|_{I}^{2}$  holds with some constant  $c_{0} > 0$ , for all  $\| f - f^{*}\|_{I}\leq u$ , where  $I\in \{S,T\}$ .
Assumption 1 imposes a boundedness condition on the kernel function and its corresponding eigenfunctions, which is commonly considered in the literature and satisfied by many popularly used kernel functions with the compact support condition (Smale & Zhou, 2007; Steinwart et al., 2009; Mendelson & Neeman, 2010). Assumption 2 is a local  $c_{0}$ -strong convexity condition of the expected loss function with respect to  $\mathcal{L}^2(\mathcal{X}, P_{\mathbf{x}}^S)$  and  $\mathcal{L}^2(\mathcal{X}, P_{\mathbf{x}}^T)$  at  $f^{*}$ . Similar assumptions are also considered by Steinwart & Christmann (2008); Wainwright (2019); Li et al. (2021); Lian (2022). Note that many popularly used loss functions, including squared loss, check loss, Huber loss, logistic loss, and hinge loss, satisfy this assumption. More detailed discussions on Assumption 2 are deferred to Section A.7 of the supplementary material.
3.1 Convergence rate for the uniformly bounded case
We begin to establish the convergence rate of the unweighted estimator  $\widehat{f}$  under the uniformly bounded case with covariate shift.
Theorem 1. Under Assumptions 1-2, if the importance ratio is  $\alpha$ -uniformly bounded, let  $\lambda > c_0 \delta_n^2 / 4$  with  $\delta_n$  being the smallest positive solution to  $C \sqrt{\log n} R (\sqrt{\alpha} \delta) \leq c_0 \delta^2 / 2$ , then for some constant  $c_1 > 0$ , with probability at least  $1 - n^{-c_1}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \leq \alpha \left(\delta_ {n} ^ {2} + 2 c _ {0} ^ {- 1} \lambda\right). \tag {5}
$$
Furthermore, based on (5), we have
$$
\left. \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} \left(f ^ {*}\right) \leq c _ {L} \alpha \left(\delta_ {n} ^ {2} + 2 c _ {0} ^ {- 1} \lambda\right) ^ {1 / 2}. \right. \tag {6}
$$
Note that the existence and uniqueness of  $\delta_{n}$  in Theorem 1 is guaranteed for any kernel class (Bartlett et al., 2005). To facilitate the comprehension of Theorem 1, we define an index as  $d(\delta) = \min \{j\geq 1|\mu_j\leq \delta^2\}$  for a given target error level  $\delta >0$ . It is known that the kernel with eigenvalues satisfying that  $\sum_{j = d(\delta) + 1}^{\infty}\mu_j\leq Cd(\delta)\delta^2$  is referred to as the regular kernel class (Yang et al., 2017). From the definition of  $d(\delta)$  and regular kernel, we can get that  $\sum_{j = 1}^{\infty}\min (\delta^{2},\mu_{j})\asymp d(\delta)\delta^{2}$ , thus the inequality  $C\sqrt{\log n} R(\sqrt{\alpha}\delta)\leq c_0\delta^2 /2$  in Theorem 1 can be directly simplified as  $\sqrt{\alpha\log nd(\sqrt{\alpha}\delta) / n}\leq C\delta$  under the assumption that  $\| f^{*}\|_{K} = 1$ . Consequently, we can rewrite (5) as
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \leq C \alpha \left(\delta^ {2} + \alpha \frac {\log n}{n} d (\sqrt {\alpha} \delta)\right), \tag {7}
$$
where  $\delta$  satisfies  $\alpha \frac{\log n}{n} d(\sqrt{\alpha}\delta) \leq C\delta^2$  and  $\lambda \asymp \delta^2$  (see Section C.4 of the supplementary material for the detailed proof). The bound in (7) controls a type of bias-variance tradeoff by the choice of  $\delta$ , and hence  $\lambda$ . Under the uniformly bounded case, Ma et al. (2023) has established a minimax rate of the order  $\alpha \inf_{\delta > 0} \left\{ \delta^2 + \frac{\sigma^2 d(\sqrt{\alpha}\delta)}{n} \right\}$  for the squared loss function. It is worth noting that the result in (7) can also attain this lower bound up to a logarithmic factor under some weaker conditions. Particularly, the assumption of Ma et al. (2023) that the noise terms are sub-Gaussian is no longer needed in (7). Although the convergence rate of (6) is sub-optimal for general loss function  $L$  considered in Section 2.1, it becomes optimal at the rate of  $c_0' \alpha (\delta_n^2 + 2c_0^{-1}\lambda)$  if the following assumption holds.
Assumption 3: For some sufficiently small constant  $u > 0$ , we assume  $E_T[L(y, f(\mathbf{x}))] - E_T[L(y, f^*(\mathbf{x}))] \leq c_0' \|f - f^*\|_T^2$  holds with some constant  $c_0' > c_0 > 0$ , for all  $\|f - f^*\|_T \leq u$ .