SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In fact, Assumption 3 is a mild condition that can be satisfied by many commonly used losses. For instance, for the squared loss, the equality always holds with  $c_0' = 1$ ; for the check loss, Assumption 3 is satisfied if the conditional density of the noise term is uniformly bounded (Zhang et al., 2021; Lian, 2022). Moreover, the regular kernel class includes kernels with either finite rank (i.e., linear or polynomial kernels), polynomially decayed eigenvalues (i.e., Sobolev kernels), or exponentially decayed eigenvalues (i.e., Gaussian kernels). Corollary 1 provides the convergence rate of  $\widehat{f}$  over these three specific kernel classes.
Corollary 1. Under Assumptions 1-3, if the kernel has a finite rank  $D$ , and let  $\lambda = C\frac{\alpha D\log n}{n}$ , then with probability at least  $1 - n^{-c_1}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} \left(f ^ {*}\right) \leq C \frac {\alpha^ {2} D \log n}{n}, \tag {8}
$$
and if the eigenvalues of the kernel decay polynomially such as  $\mu_j \leq Cj^{-2r}$  with some constant  $r > 1/2$  for  $j = 1, 2, \ldots$ , and let  $\lambda = C\alpha^{\frac{2r - 1}{2r + 1}}\left(\frac{\log n}{n}\right)^{\frac{2r}{2r + 1}}$ , then with probability at least  $1 - n^{-c_1}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) \leq C \left(\frac {\alpha^ {2} \log n}{n}\right) ^ {\frac {2 r}{2 r + 1}}, \tag {9}
$$
and if the eigenvalues of the kernel decay exponentially such as  $\mu_j \asymp e^{-Cj\log j}$ , and let  $\lambda = C\frac{\alpha\log^2n}{n}$  then with probability at least  $1 - n^{-c_1}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) \leq C \frac {\alpha^ {2} \log^ {2} n}{n}, \tag {10}
$$
Note that these bounds in (8)-(10) reduce to the known minimax lower bounds (Yang et al., 2017) for the squared loss without covariate shift (i.e.,  $\alpha = 1$ ). As the uniform boundedness implies the boundedness of the second moment, the convergence rate of the TIRW estimator  $\widehat{f}^{\phi}$  for the uniformly bounded case is similar to Theorem 3 in the next section by replacing  $\alpha$  with  $\beta^2$ .
3.2 Convergence rate for the second moment bounded case
The optimality for the  $\alpha$ -uniformly bounded importance ratio condition relies on the inequality that  $\| \widehat{f} - f^{*} \|_{T}^{2} \leq \alpha \| \widehat{f} - f^{*} \|_{S}^{2}$ . Yet, such a desired relation is not guaranteed for the second moment bounded case. Theorem 2 shows that the unweighted estimator  $\widehat{f}$  is still consistent, but not optimal.
Theorem 2. Under Assumptions 1-2, if the importance ratio satisfies that  $E_{\mathbf{x} \sim S}[\phi^2(\mathbf{x})] \leq \beta^2$ , let  $\lambda > c_0 \delta_n^2 / 4$  with  $\delta_n$  being the smallest positive solution to  $C \sqrt{\log n} R((c_0^{-1} c_L \sqrt{\beta^2 \delta})^{1/2}) \leq c_0 \delta^2 / 2$ , then for some constant  $c_2 > 0$ , with probability at least  $1 - n^{-c_2}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \leq c _ {0} ^ {- 1} c _ {L} \sqrt {\beta^ {2}} \left(\delta_ {n} ^ {2} + 2 c _ {0} ^ {- 1} \lambda\right) ^ {1 / 2}. \tag {11}
$$
Furthermore, based on (11), we have
$$
\mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) \leq c _ {L} \sqrt {\beta^ {2}} \left(\delta_ {n} ^ {2} + 2 c _ {0} ^ {- 1} \lambda\right) ^ {1 / 2}. \tag {12}
$$
To see the bound in (11) is sub-optimal, we consider the kernel with finite rank  $D$ , we can show that  $\delta_n \asymp \left(\frac{\sqrt{\beta^2} D \log n}{n}\right)^{1/3}$ , and if we take  $\lambda \asymp \left(\frac{\sqrt{\beta^2} D \log n}{n}\right)^{2/3}$ , the unweighted estimator satisfies that  $\| \widehat{f} - f^* \|_T^2 = O_P\left(\left(\frac{\beta^4 D \log n}{n}\right)^{1/3}\right)$ , which is far from the optimal rate (see Section C.4 of the supplementary material for examples of finite rank, polynomially and exponentially decay kernel classes). To deal with the sub-optimality, we consider the importance ratio correction ensuring that  $E_S[\phi(\mathbf{x}) L(y, f(\mathbf{x}))] = E_T[L(y, f(\mathbf{x}))]$ . The following theorem shows that the TIRW estimator  $\widehat{f}^\phi$  can again reach a sharp convergence rate up to logarithmic factors under some mild conditions.
Theorem 3. Under Assumptions 1-2, if the importance ratio satisfies that  $E_{\mathbf{x} \sim S}[\phi^2(\mathbf{x})] \leq \beta^2$ , let  $\lambda > c_0 \delta_n^2 / 4$  with  $\delta_n$  being the smallest positive solution to  $C \sqrt{\beta^2} \log nR(\delta) \leq c_0 \delta^2 / 2$ , and set the truncation level  $\gamma_n = \sqrt{n \beta^2}$ , then for some constant  $c_3 > 0$ , with probability at least  $1 - n^{-c_3}$ , we have
$$
\left\| \widehat {f} ^ {\phi} - f ^ {*} \right\| _ {T} ^ {2} \leq \delta_ {n} ^ {2} + 2 c _ {0} ^ {- 1} \lambda . \tag {13}
$$
Furthermore, based on (13), we have
$$
\mathcal {E} _ {T} ^ {L} \left(\widehat {f} ^ {\phi}\right) - \mathcal {E} _ {T} ^ {L} \left(f ^ {*}\right) \leq \frac {1}{2} c _ {0} \delta_ {n} ^ {2} + 2 \lambda . \tag {14}
$$
Similar to (7), when dealing with regular kernel classes, the bounds in (13) and (14) become
$$
\left\| \widehat {f} ^ {\phi} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} \left(\widehat {f} ^ {\phi}\right) - \mathcal {E} _ {T} ^ {L} \left(f ^ {*}\right) \leq C \left(\delta^ {2} + \beta^ {2} \frac {\log^ {2} n}{n} d (\delta)\right), \tag {15}
$$
where  $\delta$  is any solution of  $\beta^2\frac{\log^2n}{n} d(\delta)\leq C\delta^2$  and  $\lambda \asymp \delta^2$ . Since the second moment boundedness can be implied by the uniform boundedness, it can be concluded that  $\widehat{f}^{\phi}$  also reaches the minimax lower bound for the square loss up to logarithmic factors (with  $\alpha$  substituted by  $\beta^2$ ). For three specific kernel classes, we also have the following Corollary for the TIRW estimator  $\widehat{f}^{\phi}$ .
Corollary 2. Under Assumptions 1-2, if the kernel has a finite rank  $D$ , and let  $\lambda = C\frac{\beta^2D\log^2n}{n}$ , then with probability at least  $1 - n^{-c_3}$ , we have
$$
\left\| \widehat {f} ^ {\phi} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} \left(\widehat {f} ^ {\phi}\right) - \mathcal {E} _ {T} ^ {L} \left(f ^ {*}\right) \leq C \frac {\beta^ {2} D \log^ {2} n}{n}, \tag {16}
$$
and if the eigenvalues of the kernel decay polynomially such as  $\mu_j \leq C j^{-2r}$  with a constant  $r > 1/2$  for  $j = 1, 2, \ldots$ , and let  $\lambda = C \left( \frac{\beta^2 \log n}{n} \right)^{\frac{2r}{2r + 1}}$ , then with probability at least  $1 - n^{-c_3}$ , we have
$$
\left\| \widehat {f} ^ {\phi} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} (\widehat {f} ^ {\phi}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) \leq C \left(\frac {\beta^ {2} \log^ {2} n}{n}\right) ^ {\frac {2 r}{2 r + 1}}, \tag {17}
$$
and if the eigenvalues of the kernel decay exponentially such as  $\mu_j \asymp e^{-Cj\log j}$ , and let  $\lambda = C\frac{\beta^2\log^3n}{n}$ , then with probability at least  $1 - n^{-c_3}$ , we have
$$
\left\| \widehat {f} - f ^ {*} \right\| _ {T} ^ {2} \asymp \mathcal {E} _ {T} ^ {L} (\widehat {f}) - \mathcal {E} _ {T} ^ {L} (f ^ {*}) \leq C \frac {\beta^ {2} \log^ {3} n}{n}, \tag {18}
$$
For easy reference, we summarize some important theoretical results that have been established by us in table 1.
Table 1: Established convergence rates under different cases
Kernel class	Uniformly bounded case	Moment bounded case
Unweighted estimator	TIRW estimator	Unweighted estimator	TIRW estimator
Finite rank D	OP(α2D log n/n)	OP(αD log2n/n)	OP((β4D log n/n)1/3)	OP(β2D log2n/n)
Polynomial decay	OP((α2 log n/n)2r/2r+1)	OP((α log2n/n)2r/2r+1)	OP((β4 log n/n)6r+1)	OP((β2 log2n/n)2r/2r+1)
Exponential decay	OP(α2 log2n/n)	OP(α log3n/n)	OP((β4 log2n/n)1/3)	OP(β2 log3n/n)
In this section, we validate our theoretical analyses by performing numerical experiments on both synthetic data and real applications. To estimate the importance ratios, we apply the Kullback-Leibler importance estimation procedure (KLIEP) (Sugiyama et al., 2007b), and the estimation details are provided in the supplementary material. For brevity, we only report the performance of kernel-based quantile regression (KQR) where the check loss is used in synthetic data analysis, and the performance of kernel support vector support machine (KSVM) with hinge loss in the real data analysis; completed results for other loss functions as well as the detailed settings and results for the other examples, including for the multi-dimensional cases, can be found in Section A of the supplementary material. In our experiments, we consider the RKHS induced by Gaussian kernel in all the examples, and all the experiments are replicated 100 times in the synthetic data analysis.
4.1 Synthetic data analysis
We investigate the performance of KQR under covariate shift with the following generating model
$$
y = f _ {0} (x) + \left(1 + r (x - 0. 5) ^ {2}\right) \sigma (\varepsilon - \Phi^ {- 1} (\tau)),
$$
where  $f_0(x) = \sin(\pi x)$  with  $x \in \mathcal{R}$ ,  $\Phi$  denotes the CDF function of the standard normal distribution and  $\varepsilon \sim N(0,1)$ . We consider  $\rho_{\mathbf{x}}^{S} \sim N(\mu_{1},\sigma_{1}^{2})$  and  $\rho_{\mathbf{x}}^{T} \sim N(\mu_{2},\sigma_{2}^{2})$  with  $\mu_{1} = 0, \sigma_{1}^{2} = 0.4, \mu_{2} = 0.5, \sigma_{2}^{2} = 0.3$  for the uniformly bounded case and  $\mu_{1} = 0, \sigma_{1}^{2} = 0.3, \mu_{2} = 1, \sigma_{2}^{2} = 0.5$  for the moment bounded case, respectively. Moreover, we set  $r = 0$  and  $\sigma = 0.5$  for the homoscedastic case, and  $r = 1$  and  $\sigma = 0.3$  for the heteroscedastic case, respectively. Note that the simulation results for the case that  $\tau = 0.3$  and  $r = 1$  are presented in Figure 2 under the uniformly bounded case and in Figure 3 under the moment bounded case; completed results for other cases are provided in the supplementary material. We compare the averaged mean square error (MSE) and empirical excess risk of the unweighted estimator and weighted estimator, either with true or estimated weights across different choices of regularization parameter  $\lambda$ , source sample size  $n$ , and target sample size  $m$ .
(f)
Figure 2: Averaged MSE and empirical excess risk for unweighted KQR, TIRW KQR with true weight and estimated weight, respectively. Note that in (a) and (d), the curves are plotted with respect to  $\log_{10}\lambda$  with  $n = 500$ ,  $m = 1000$ ; in (b) and (e) the curves are plotted with respect to  $n$  with fixed  $m = 1000$ ,  $\lambda = 10^{-4}$ ; in (c) and (f), the curves are plotted with respect to  $m$  with fixed  $n = 500$ ,  $\lambda = 10^{-4}$ .
From (a) and (d) in Figure 2, we can conclude that the error of the unweighted estimator is very close to that of the weighted estimator for the uniformly bounded case, which is consistent with our theoretical findings in Section 3. For the moment bounded case as demonstrated in (a) and (d) of Figure 3, the weighted estimator consistently outperforms its unweighted counterpart for all choices of  $\lambda$ . Even when  $\lambda$  is far from its optimal choice, the weighted estimator still maintains a lower
(f)
Figure 3: Averaged MSE and empirical excess risk for unweighted KQR, TIRW KQR with true weight and estimated weight, respectively. Note that in (a) and (d), the curves are plotted with respect to  $\log_{10}\lambda$  with  $n = 500$ ,  $m = 1000$ ; in (b) and (e) the curves are plotted with respect to  $n$  with fixed  $m = 1000$ ,  $\lambda = 10^{-4}$ ; in (c) and (f), the curves are plotted with respect to  $m$  with fixed  $n = 500$ ,  $\lambda = 10^{-4}$ .
error level with significant improvement over the unweighted estimator whose error is on occasion extremely high for small choices of  $\lambda$ . Additionally, it is clear from Panels (b) and (e) of Figures 2 and 3 that in the moment bounded case, the error curves for the unweighted estimator have some significant gaps with those of the weighted estimator, while in the bounded case, these curves tend to coincide as  $n$  grows. It is also clear from the table 1 that the trends in Figure 2 (b)-(e) and Figure 3 (b)-(e) almost agree with the explicit convergence rate where the Gaussian kernel with exponential decay is used. This phenomenon is consistent with our theoretical conclusion that the unweighted estimator can only achieve sub-optimal rates when the importance ratio is moment bounded and attain optimal rates when uniformly bounded. Finally, we note that the target sample size has a subtle influence on all the estimators, as demonstrated in Panels (c) and (f) of Figures 2 and 3.
4.2 Real case study
We consider the binary classification problem on the Raisin dataset, which is available in https://archive.ics.uci.edu/ml/datasets.php. The dataset contains 900 instances and 7 attributes. After standardization, the data are first randomly split into source and target datasets. We introduce a binary random variable  $s$  that serves as a labeling indicator and assign it to the source dataset if  $s = 0$ . To ensure that the conditional distribution of different datasets remains invariant, we require  $s$  to be conditionally independent with the response  $y$  given the covariate  $\mathbf{x}$  (Zadrozny, 2004; Quinonero-Candela et al., 2008). We implement KSVM in which the covariate shift only exists in the first covariate. Specifically, for  $\ell > 0$ , we conduct the splitting rule as  $P(s_i = 1 \mid \mathbf{x}_i) = \min(1, (x_{i,0} - c)^2 / \ell)$ , where  $x_{i,0}$  is the first element of  $\mathbf{x}_i$  and  $c = \min_i\{x_{i,0}\}$ . We refer to  $\ell$  as the shift level that a smaller value of  $\ell$  favors a greater distinction between source and target datasets. We set the turning parameter  $C_\lambda = (n\lambda)^{-1}$ . For the TIRW estimator, we use importance weighted cross validation (IWCV) (Sugiyama et al., 2007a) to tune the truncation parameter  $\gamma_n$ . We refer to Section A in the supplementary material for the details of the adopted method.
Figure 4 summarizes the accuracy rate and standard error in different settings. As shown in this figure, the unweighted estimator has poor performance on target data, even for the optimal choice of  $C_{\lambda}$ . Nevertheless, the weighted estimator has a significant improvement in performance. Surprisingly, the weighted estimator is relatively stable with respect to the choices of  $C_{\lambda}$ , which is consistent with our results on synthetic data. Moreover, we can find that a proper truncation slightly improves the accuracy rate of the weighted estimator while reducing its variation.
(a)  $\ell = 6$
(b)  $\ell = 8$
Figure 4: Averaged accuracy rate for unweighted KSVM, IRW KSVM and TIRW KSVM with different shift levels  $\ell$ , respectively.
(c)  $\ell = 10$
In this work, we propose a unified analyzing framework for kernel-based methods under covariate shift and show that the predictive function can be better estimated by using the distribution information of the covariates from both aspects of theory and numerical performance. The established theoretical results fulfill the gap in understanding the influence of the covariate shift under various kernel-based learning problems, both researchers and practitioners may benefit from our theoretical findings. Extensive numerical studies also provide empirical evidence and confirm the established theoretical results. Note that it is very interesting to test if there exists a distribution shift in practical applications and if the distribution shift satisfies the uniformly bounded or moment bounded assumptions. Unfortunately, the relevant approaches have remained lacking to our best knowledge. And it is interesting to note that as shown in our real applications, the TIRW estimator always outperforms the unweighted estimator, and thus we suggest using the TIRW estimator to analyze the real-life dataset. Moreover, in the theory, we directly use the true importance ratio, and it is difficult to derive the consistency of the estimated ratio and plug it into our theoretical analysis. In the machine learning literature, there exist several measures to quantify the divergence between two distributions, including but not limited to  $f$ -divergence, Wasserstein distance and kernel mean matching. It is still an open problem if it still works when we use other measures rather than the importance ratio. We leave such promising and interesting topics as potential future work.
Xingdong Feng's research is supported in part by NSFC-12371270 and Shanghai Science and Technology Development Funds (23JC1402100). Xin He's research is supported in part by NSFC-11901375 and Program for Innovative Research Team of Shanghai University of Finance and Economics. This research is also supported by Shanghai Research Center for Data Science and Decision Technology and the Fundamental Research Funds for the Central Universities.
Baktashmotlagh, M., Harandi, M. T., Lovell, B. C., & Salzmann, M. (2014). Domain adaptation on the statistical manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2481-2488).
Bartlett, P. L., Bousquet, O., & Mendelson, S. (2005). Local rademacher complexities. The Annals of Statistics, 33, 1497-1537.
Byrd, J., & Lipton, Z. (2019). What is the effect of importance weighting in deep learning? In Proceedings of the 36th International Conference on Machine Learning (pp. 872-881). PMLR volume 97.
Cortes, C., Mansour, Y., & Mohri, M. (2010). Learning bounds for importance weighting. Advances in Neural Information Processing Systems, 23, 442-450.
Dasgupta, S., Goldberg, Y., & Kosorok, M. R. (2019). Feature elimination in kernel machines in moderately high dimensions. The Annals of Statistics, 47, 497-526.
Yamada, M., Sugiyama, M., & Matsui, T. (2009). Covariate shift adaptation for semi-supervised speaker identification. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 1661-1664). IEEE.
Yamazaki, K., Kawanabe, M., Watanabe, S., Sugiyama, M., & Müller, K.-R. (2007). Asymptotic Bayesian generalization error when training and test distributions are different. In Proceedings of the 24th International Conference on Machine Learning (pp. 1079-1086).
Yang, Y., Pilanci, M., & Wainwright, M. J. (2017). Randomized sketches for kernels: Fast and optimal nonparametric regression. The Annals of Statistics, 45, 991 - 1023.
Zadrozny, B. (2004). Learning and evaluating classifiers under sample selection bias. In Proceedings of the 21th International Conference on Machine Learning (pp. 903-910).
Zhang, F., Li, R., & Lian, H. (2021). Approximate nonparametric quantile regression in reproducing kernel Hilbert spaces via random projection. Information Sciences, 547, 244-254.