Title: Computational Efficiency under Covariate Shift in Kernel Ridge Regression
Venue: NeurIPS 2025
Accepted: yes
Keywords: statistical learning theory

kernel methods

covariate shift

random projections
Abstract: This paper addresses the covariate shift problem in the context of nonparametric regression within reproducing kernel Hilbert spaces (RKHSs). Covariate shift arises in supervised learning when the input distributions of the training and test data differ, presenting additional challenges for learning. Although kernel methods have optimal statistical properties, their high computational demands in terms of time and, particularly, memory, limit their scalability to large datasets. To address this limitation, the main focus of this paper is to explore the trade-off between computational efficiency and statistical accuracy under covariate shift. We investigate the use of random projections where the hypothesis space consists of a random subspace within a given RKHS. Our results show that, even in the presence of covariate shift, significant computational savings can be achieved without compromising learning performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUnder the learning setting where the covariate shift presents, this manuscript investigates the trade-off between computational efficiency and statistical accuracy in kernel ridge regression. Specifically, the authors show the employing random projection techniques, e.g., Nystrom method, in KRR can achieve optimal convergence rates while dramatically reducing computational costs. Studies are extended to unknown IW functions.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n1. clear writing, easy to follow. \n2. The authors honestly state limitations of their studies, such as the misspecified case they considered, although slightly more general, but conceptually align with well-specified case.\n3. Clear discussions about related works, including how assumptions are different and what results are (non-)comparable with existing results.\n4. The simulation setting mimics the misspecified setting.\n\n# Weakness:\n\n1. The biggest incentive of consider random projection technique to build KRR estimators is the necessity to compute the inverse of the sample covariance operator, or specifically the matrix in eq. (6). However, KRR is well-known to be a special case of spectral regularized algorithms, including KRR, kernel gradient descent or gradient flow, etc. Therefore, if one generates the estimator via KGD (or others), one can at least avoid the computational time bottleneck of KRR’s matrix inversion.\n2. It is interesting to see the strong moment assumption (Assumption 3) over the weight function is crucial for applying Nystron method. While technically explanations are provided in Appendix C, it would be more reader-friendly to provide some intuitive explanations.  \n3. While having the eigenvalue of covariance operator decay polynomially satisfy Assumption 6, applying Gaussian kernel will not satisfy this assumption. Using Matern kernel with finite order seems more reasonable and align with the assumption. \n4. For those who are not familiar with random projection technique in kernel methods, lines 150 - 154 could be a bit overwhelming. That being said, authors still provided enough details in the Appendix. If there is more room allowed, it can be further detailed. Also, missing argmin at the r.h.s. of eq. (18).\n\nQUESTIONS:\nIntuitively, why is Assumption 3 crucial to apply Nystrom method? Do authors think it is necessary? Or it can be relaxed further with different mathematical tools.\n\nLIMITATIONS:\nYes. Authors provide a paragraph in introduction to describe the limitations in terms of the theoretical assumptions and their results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to reduce the computational cost of RKHS-based non-parametric regression under covariate shift between the training and testing distributions. Traditionally, this problem is addressed using kernel ridge regression with importance weighting (IW) correction, which has a computational complexity of $O(n^3)$, making it impractical for large datasets. To address this, the authors propose a Nyström-based approximation method that restricts the search for the minimizer to a low-dimensional random subspace of the RKHS. They prove that the proposed method achieves the optimal convergence rate when the importance weights are known, and they extend the results to the case of unknown weights. Empirical results demonstrate both the efficiency and accuracy of the approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The use of Nyström approximation in the context of non-parametric regression under covariate shift is both relevant and innovative. The paper provides strong statistical guarantees, including non-asymptotic results and effective dimension.\n* The related work on kernel ridge regression and its approximations is generally well-situated. However, some notable methods, such as Random Features and SGD, which could also be applicable in this context (especially from a computational standpoint), are omitted.\n* The extension of the method to scenarios with unknown IW is interesting.\n\n**Weaknesses**\n\n* Based on Section 6, it remains unclear whether knowledge of the IW  leads to improved convergence rates. Clarifying this would strengthen the theoretical contributions.\n* Some important assumptions, such as the requirement that the test distribution is dominated with respect to the training distribution, are not sufficiently highlighted. This assumption rules out support shift and is critical to the theoretical guarantees.\n* The misspecified setting (addressed in Section 6) is treated as the harder case, yet there is no discussion of it in Section 5. This separation is confusing, as it leaves the reader uncertain whether the guarantees proven earlier apply only in the well-specified case.\n\n**Writing Suggestions**\n\n* There is redundancy in the presentation of equations, especially in Section 4. Additionally, the definition of the Nyström ALS method should appear in the main. Readers unfamiliar with Nyström methods may find it difficult to understand the approach without it.\n* Notation could be made more consistent—for instance, both $\\int$ and $\\mathbb{E}$ are used for same integration (with $\\mu$) in Section 2. Some definitions, such as the norm $\\Vert \\cdot \\Vert_\\rho$, are also missing.\n\nIn summary, I appreciate the paper's solid mathematical foundation, the clever approach to addressing the problem, and the empirical validation. However, there is room for improvement in terms of clarity and overall soundness.\n\nQUESTIONS:\nYou could address the three points raised under 'Weaknesses' to improve your rating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of computational efficiency of kernel methods in out-of-distribution (OOD) scenarios. The authors consider importance-weighting (IW) correction to handle distribution mismatch and random projections (the Nystrom method) to improve efficiency, and rigorously analyze the interplay between the two. They extend the previous literature on random-projection methods to the covariate-shift case and propose an efficient algorithm. On realistic datasets, the algorithm is shown to run much faster while achieving similar prediction accuracy to the naive kernel regression.\n\nI could not follow the entire mathematical framework, but I think the general idea is to constrain the solution space by considering only a subset of training data to reduce computational overhead and to apply the IW framework on the restricted space. The authors then prove excess-risk bounds for this algorithm and find that the Nyström method does not really decrease OOD generalization with IW.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. The manuscript is clearly written and well organized; the necessary theoretical concepts are introduced in a clean way.\n2. The paper studies two important problems in classical machine learning—efficiency and OOD generalization—simultaneously.\n3. The assumptions and limitations are described clearly.\n\n### Weaknesses\n\n1. It was hard for me to follow the analysis after Section 5.\n2. The theory assumes ALS to subsample inputs, but its definition is missing. The authors do not discuss how ALS compares to naive subsampling. The choice of subset directly affects IW correction, and it is not clear to me whether ALS is optimal.\n3. The paper only considers IW to treat OOD generalization and does not discuss alternative methods.\n\n**Some minor comments/suggestions**\n\n* It may help the reader to increase the font size in the figure.\n* In line 300: W-KRR → IW-KRR.\n* You could provide a step-by-step description of the algorithm for practitioners.\n\nQUESTIONS:\nOne could alternatively approximate the kernel with random features without subsampling training inputs. How does your method compare to that?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the covariate shift problem in nonparametric regression within reproducing kernel Hilbert spaces (RKHSs). While kernel methods offer strong statistical guarantees, their high computational and memory costs limit their scalability. To address this, the authors propose using random projections to define a random subspace of the RKHS, aiming to balance computational efficiency and statistical accuracy under covariate shift. The study demonstrates that this approach can yield substantial computational savings without sacrificing learning performance, even when training and test distributions differ.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe authors derived excess risk bounds for the Nyström predictor defined in Eq. (12), and show that the Nyström approximation does not compromise the state-of-the-art convergence rates, while significantly reducing the computational burden.\n\nWeaknesses \n\n1. I am not sure whether the provided analysis could shed light on practical applications (see questions), and also  in interpreting the empirical results reported in Table~1 where the MSE of Nystrom method in KRR is even lower than without this approximate sampling scheme.\n\n2. The authors employ a more sophisticated sampling strategy using approximate leverage scores (ALS) instead of uniform random sampling. However, in many cases, random sampling can be both more accurate and significantly cheaper than ALS (see related questions for details). In this regard, the analysis in this paper may be less generalizable or practically advantageous than it appears. (see more details in questions).\n\nQUESTIONS:\nIt is well known that the Nyström method can be used to scale up a wide range of computations involving kernel matrices. I am curious how the statistical guarantees presented in this work offer new insights into the practical application of the Nyström method under covariate shift—particularly regarding the choice of sampling scheme, determination of sampling size, and the trade-offs in performance compared to exact computations.  Specifically, while statistical guarantees are valuable, do they shed light on why subsampling often leads to improved performance in practice especially in case of large sample size in terms of predictive accuracy?\n\nI would like to clarify whether all the empirical and theoretical results in this paper are based solely on the approximate leverage score (ALS) sampling scheme. If so, I would encourage the authors to include a comparison with uniform random sampling, particularly in the empirical evaluations. In many cases, random sampling is not only significantly cheaper but can also yield comparable or even better accuracy than ALS (see, for example, Figure 1, left panel in [1]). Including results based on random sampling would provide valuable insights and broaden the practical relevance of the work.\n\n[1] Petros Drineas et al.,  An Experimental Evaluation of a Monte-Carlo Algorithm for Singular Value Decomposition.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}