Title: Kernel Regression in Structured Non-IID Settings: Theory and Implications for Denoising Score Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: kernel ridge regression

generalization

sample complexity

diffusion model
Abstract: Kernel ridge regression (KRR) is a foundational tool in machine learning, with recent work emphasizing its connections to neural networks. However, existing theory primarily addresses the i.i.d. setting, while real-world data often exhibits structured dependencies - particularly in applications like denoising score learning where multiple noisy observations derive from shared underlying signals. We present the first systematic study of KRR generalization for non-i.i.d. data with signal-noise causal structure, where observations represent different noisy views of common signals. Under standard spectral decay assumptions, we develop a novel blockwise decomposition method that enables precise concentration analysis for dependent data. Using this technique, we derive excess risk bounds for KRR that explicitly depend on: (1) the kernel spectrum, (2) causal structure parameters, and (3) sampling mechanisms (including relative sample sizes for signals and noises). We further apply our results to denoising score learning, establishing generalization guarantees and providing principled guidance for sampling noisy data points. This work advances KRR theory while providing practical tools for analyzing dependent data in modern machine learning applications.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considered the model $y_{i}^{j} = f^*(g_{i}^{j}) + \\epsilon_{i}^{j}$, $i \\leq n$, $j \\leq k$. The authors further assumed that there exists a function $g$, such that for each group $j$, the inputs are generated as $g_{i}^{j} = g( x_{i}, u_{i}^{j})$, where i.i.d. signals $\\{x_i\\}$ are independent of i.i.d. noises $\\{u_{i}^{j}\\}$.\n\nThis paper then obtained an upper bound on the convergence rate of learning curve (i.e., the excess risk w.r.t. different choices of regularization parameter $\\lambda$) of KRR under common assumptions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- i) The authors considered the non-i.i.d. data settings, which is new within the KRR literature. They also provide two real-world examples (Examples 3.1 and 3.2) to illustrate the relevance of this setting.\n- ii) The paper is carefully written and overall enjoyable to read.\n- iii) The proof is novel compared with existing works. For example, the authors used a blockwise decomposition method for $k$-gap independent random sequence as an alternative to Bernstein-type concentration inequalities typically used for i.i.d. data (e.g., Lemma B.4 in [31]).\n- iv) The results in this paper recovered known results in the i.i.d. case when $k = 1$.\n\nWeaknesses:\n- i) The authors assume that the target function $f^* \\in L^2$ lies in the interpolation space $[\\mathcal{H}]^s$ with $s \\geq 1$. In particular, when $s = 1$, this reduces to $f^* \\in \\mathcal{H}$, i.e., the RKHS itself. This assumption may be restrictive when dimension $d$ is large. For example, \n    - Consider the Sobolev RKHS $W^{m,2}(\\mathcal{X})$ with $\\mathcal{X} \\subset \\mathbb{R}^{d}$ and $m > d/2$. Its interpolation space with $s>0$ is given by $ [W^{m,2}(\\mathcal{X})]^s \\cong W^{m s , 2}(\\mathcal{X})$, and functions in $W^{ms,2}(\\mathcal{X})$ are weak differentiable of order $m s$.\n    - On one hand, if we assume that $f_{\\star} \\in \\mathcal{H}$, then it requires that $f_{\\star}$ is weak differentiable of order $m>d/2$, which becomes increasingly unrealistic as $d$ grows. \n    - On the other hand, if we consider the interpolation space with $s>0$, then we can adopt a weaker assumption $f_{\\star} \\in W^{m s , 2}(\\mathcal{X})$, allowing the results to cover less smooth functions.\n\n    It would be helpful if the authors could discuss the challenges associated with analyzing the case $s < 1$, and whether these challenges are analogous to those in the i.i.d. setting.\n\n- ii) The experiments involve a three-layer ReLU MLP. Since there is currently no theoretical approximation framework between MLPs and kernel gradient flow (KGF) / KRR in the non-i.i.d. setting, it is better to use KRR instead.\n\nQUESTIONS:\n- 1) What is the motivation of setting $r \\leq d$ in line 172? I notice that $r$ systems are trained independently, and both the theoretical analysis and the example in Section 4.1 consider the special case $r = 1$.\n\n- 2) There might be typos in (2) and possibly other equations: The bias convergence rate should be $n^{-\\min (s, 2)\\theta}$; and the $\\Theta$ notation should be $O$ unless both upper and lower bounds are established.\n\n- 3) The paper would benefit from a more comprehensive discussion of relevant prior work in several areas: \n    - (i) Optimal rate of KRR and kernel gradient flow (deep neural networks, that the authors mentioned, are more closely related to kernel gradient flow). E.g., Optimal rates for the regularized least-squares algorithm; On the optimality of misspecified kernel ridge regression; Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay.\n    - (ii) saturation effect of KRR (the appearance of $\\min$\\{$s$, 2\\} in Theorem 4.1 is related to it). E.g., Optimal rates for the regularized least-squares algorithm; ON THE SATURATION EFFECT OF KERNEL RIDGE REGRESSION.\n    - (iii) high-dimensional KRR and KGF. E.g., Just interpolate: Kernel “Ridgeless” regression can generalize; linearized two-layers neural networks in high dimension; on the Saturation Effects of Spectral Algorithms in Large Dimensions; On the Pinsker bound of inner product kernel regression in large dimensions\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes upper bounds on the risk of scalar kernel ridge regression in the misspecified setting (meaning that the optimal regressor is in a specific power space of the RKHS) with block-independent data. The first theorem (4.1) provides the upper bound, showing polynomial decrease rates of the population risk if the regularization parameters decreases slowly enough. In contrast, the second theorem establishes that the risk is lower bounded by a positive constant if the regularization parameter decreases too sharply, establishing the absence of generalization (which I understand as consistency of the estimator).\nThe paper then applies the obtained rates to conclude on the successful learning of the denoising score algorithm under theoretical assumptions. This use case is the motivational application for the theory, and is illustrated in experiments on a toy problem.\nOn a technical level, the main novelty seems to reside in a block argument to recover the case of iid variables (cf. Section 5.1).\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is well-motivated and generally well-written (modulo the issues raised below). It proposes interesting ideas, including in the extensive appendix. However, I have significant concerns regarding several aspects: the completeness of the related work discussion, the meaningfulness of the theoretical setup, the adequacy of the assumptions relative to the claimed contributions, and the validity of the application to denoising score learning. I also raise concerns about some aspects of the writing.\n\nGiven the scope and depth of these issues, I believe that fully addressing them within the rebuttal period may be challenging. That said, I remain open to discussion and would welcome a constructive exchange.\n\n1. **Related work** The paper does not sufficiently discuss its relation to the following bodies of work: \n\t1. Classical literature on statistical learning of support vector machines (SVM) [R1]: a central goal in statistical learning is to derive generalization guarantees in the form of high confidence bounds on the excess risk. While the paper does acknowledge that it is a central question in KRR (which is a special case of SVMs), it does not acknowledge the classical theory like that of [R1]\n\t2. Learning with SVMs from non-iid data [R2,R3,R4]: there is a rich literature on learning from non-iid data, either asymptotically for general losses [R2,R3] or in the finite-sample case with mixing assumptions [R4,R5] or without (under assumptions on the loss and kernel) [R6,R7]. I believe the results on mixing are of particular interest to the authors, because block-independent processes _are_ mixing. This connects to one of my concern below about the fact that the non-iid-ness considered in this work is mild.\n\t3. Time-uniform concentration inequalities in bandits and Bayesian optimization: there is a class of concentration results bounding the error between the target function and the KRR estimate pointwise in the input [R8, Corollary 3.17], [R9, R10]. While these results originate from a different community and are not immediately applicable to the present use case (well-specified case, no asymptotic analysis), they should be mentioned in the related work and may be of interest to the authors.\n2. **Setup** The non-iid-ness considered in the general paper is mild, in the sense that simple subsampling recovers iid data, or more generally that it implies any form of mixing, which is a very standard assumption to handle dependent data in learning theory. \nIn particular, what seems to be the main novelty in the proof (decomposing the data in independent blocks) reminds me of standard arguments in $\\beta$-mixing tracing back to [R11] (see [R12] for a more recent reference on this applied to learning theory, and [R13] for an application to kernel methods). The more restricted setup considered in the paper may be sufficient for denoising score matching, but I would encourage the authors to justify why they consider this particular form of non-iid-ness compared to a form of mixing.\n3. **Assumptions** \n\t1. The paper provides insufficient details on the assumptions imposed on the kernel function. In particular, the assumptions for RKHS elements to be square-integrable and for the Mercer decomposition to exist, be well-behaved, and yield an ONB of $\\mathcal H$ instead of an ONS [48] are not stated. Furthermore, the domain and range of the operator $T$ (and related operators) need to be specified. Are they operators on $L_2$? On $\\mathcal H$? Similarly, the paper should make explicit at the beginning of 3.2 already what set $f_\\rho$ is assumed to lie in, as this may impose further restrictions on the kernel (e.g., if $f_\\rho$ is assumed square integrable at this point). Additionally, the space $[\\mathcal H]^t$ (which the paper calls \"interpolation spaces\" but is actually the power RKHS, cf. [48, Section 4]) is not always well-defined; only the actual interpolation space is [48, Eq. (36)] (and it is not a space of functions, but one of equivalence classes). I do not think these imprecisions impact the results, but they need to be addressed.\n\t2. I would like to emphasize that the classical convention [48] in the Mercer decomposition is that the family $(\\lambda^{1/2}e_i)_i$ is an ONS in $\\mathcal H$, and **not** the family $(e_i)$ (which is an ONS when embedded in $L_2$, but is not normalized in $\\mathcal H$). A different convention is used in the main text (l. 178). This is not critical as it is ultimately a convention, and I also believe this is a typo and that the authors meant to use the classical convention (given comments such as l.183). I will assume this from now on, as it is important for my next point.\n\t3. More importantly, I have significant concerns regarding the first two points of Assumption 1. Indeed, I claim that the combination of the first and second points imply that $f_\\rho\\in\\mathcal H$ if $s>1$ (see below). As a result, Assumption 1 imposes a **very strong** smoothness condition on $f_\\rho$ (RKHS membership), which directly contradicts the storyline and claims of the paper and makes unnecessary all the refinements leveraging interpolation spaces. Technically, the membership results from the additional term $i^{-1/2}$ assumed in the expansion of $f_\\rho$, which deviates from standard source conditions in the misspecified setting for KRR (interpolation space membership, which is usually expressed via the expansion in Assumption 1 without the $i^{-1/2}$ factor and assuming that $a_i$ is square summable instead of bounded above and below, cf. [48, Eq. (36)]). Finally, the assumption that $a_i\\geq c>0$ is also nonstandard, and further restricts possible values for $f_\\rho$ (in particular, $f_\\rho$ cannot be $0$, as no coefficient may be $0$ in the expansion). I encourage the authors to revise this assumption or the claims of the paper.\n\t4. I am also surprised by the form of the polynomial eigenvalue decay in Assumption 1.\n\tIn particular, I have only encountered the condition $c_\\beta i^{-\\beta}\\leq \\lambda_i$ to show _lower bounds_ on the excess risk [R14, Section 5]. A discussion of why this lower bound is needed for the present results would be welcome.\n4. **Application** The application to denoising score learning in Theorem 4.4 reduces to applying Theorem 4.1 with a particular form of measurement model ($g$). Yet, the classical method (and the present numerical experiment) relies on a neural network for the regressor, and _not_ on KRR. In other words, Theorem 4.1 is only valid if the chosen neural network can represent all functions of the RKHS $\\mathcal H$ (and only those); i.e., if it behaves as KRR. This is highly unrealistic, and yet the paper does not discuss this aspect, and instead claims that the result can be applied for practitioners to perform rigorous kernel engineering. I believe this is an overclaim, as it would require an analysis of how to design the neural network should be chosen to approximate the RKHS and of the incurred error due to the approximation. I am not convinced such an analysis would be interesting, however, as one would certainly be better off using KRR rather than approximating it with a neural network. Summarizing, I believe that the claims on the relevance of Theorem 4.4 in practice and the discussion of its limitations should be adapted in light of this discussion.\n\n**Proof of $f_\\rho\\in\\mathcal H$** Define $a_i := a_i^{(r)}$, and assume that $(\\lambda_{i}^{1/2}e_{i})$ is an ONB of $\\mathcal H$ and that $s>1$. Then, the series used to define $f_\\rho$ converges in $\\mathcal H$. Indeed: $$\n\\begin{split}\\sum_{i}\\lVert a_i \\lambda_i^{s/2} e_i\\rVert^2_k\n&= \\sum_i a_i^2\\lambda_i^si^{-1}\\lambda_i^{-1}\\\\\n&\\leq \\max(c^2,C^2)\\sum_i \\lambda_i^{s-1}i^{-1}\\\\\n&\\leq \\max(c^2,C^2)\\cdot C_\\beta^{s-1}\\cdot \\sum_i i^{-\\beta(s-1)-1}.\\end{split}\n$$\nThe RHS is finite as long as $\\beta(s-1)+1>1$, which holds since $s>1$.\nAs a result, the general term of the series is square summable, implying that the series converges in norm in $\\mathcal H$ and that $f_\\rho\\in\\mathcal H$.\n\n\n**Minor comments**\n* The terminology \"the signal\" to designate the input is nonstandard; I recommend using \"the input\"\n* I recommend avoiding self-congratulatory expressions like \"advanced technique\" (l. 67)\n* I am not fond of notations like $R \\leq \\Theta(\\dots)$, $R \\geq \\Omega(\\dots)$, or $R\\leq \\mathcal O(\\dots)$. Indeed, the symbols $\\Theta$, $\\Omega$, and $\\mathcal O$ are _already_ inducing orderings, as $f\\geq g$ and $g = \\mathcal O(h)$ implies that $f = \\mathcal O(h)$. While this is merely style for $\\mathcal O$ and $\\Omega$, it becomes misleading for $\\Theta$, as the notation $f \\leq \\Theta(g)$ may have a different meaning than $f=\\Theta(g)$. I recommend using the equality symbol everywhere.\n* You should avoid informal statements like \"roughly\" (l.71) in formal statements if they are not rigorously defined/made precise later.\n* The notation $\\mu_\\mathcal G$ (l. 141) is undefined (I suspect it should be $\\rho$). I recommend redefining $\\mu_\\mathcal G:= \\rho$, however.\n* The change of notations in Example 3.2 seems unnecessary\n* It would be helpful to clarify why KRR is applied independently per dimension rather than using a multi-output variant (possibly with a diagonal kernel if the proofs don't generalize).\n* l.203: I believe that \"$f_\\rho^{(r)}\\in\\mathcal H$\" is a typo, as the paper tries to address the misspecified case (up to the points raised in my comment 3.2). This is also present l.813\n* Other undefined symbols: $\\lambda$ in Theorem 1 (undefined at this point) and $\\sigma_\\epsilon^2$,  $\\sigma_{\\epsilon_{1,2}}$, $\\sigma$, and $\\sigma_G$ ll.191--193\n* The notation $g$ is overloaded and ambiguous. It is sometimes used as a random variable on which expectations are taken (cf. (1)), sometimes as a function (l.145), sometimes as an element of $\\mathcal X\\times\\mathcal U$, and sometimes as multiple of these options (cf. the equation ll.192--193). I encourage the authors to clarify this.\n* l.141: I don't understand what is meant by $\\mu_\\mathcal G(g(x,u)) = \\mu_\\mathcal X(x)\\mu_\\mathcal U(u)$, as measures should take sets as inputs.\n\n[R1] Steinwart and Christmann, Support vector machines, 2008\n\n[R2] Steinwart et al., Learning from dependent observations, 2009\n\n[R3] Massiani et al., On the consistency of kernel methods with dependent observations, 2024\n\n[R4] Steinwart and Christmann, Fast learning from non-iid observations, 2009\n\n[R5] Hang and Steinwart, Fast learning from $\\alpha$-mixing observations, 2014\n\n[R6] Ziemann and Tu, Learning with little mixing, 2022\n\n[R7] Simchowitz et al., Learning without mixing (...), 2018\n\n[R8] Abbasi-Yadkori, Online Learning for Linearly Parametrized Control Problems, 2013\n\n[R9] Chowdhury and Gopalan, On kernelized multi-armed bandits, 2017\n\n[R10] Chowdhury and Gopalan, No-regret algorithms for multi-task bayesian optimization, 2021\n\n[R11] Yu, Rates of convergence for empirical processes of stationary mixing sequences, 1997\n\n[R12] Mohri and Rostamizadeh, Stability Bounds for Stationary ϕ-mixing and β-mixing Processe, 2010\n\n[R13] Kuznetsov and Mohri, Generalization bounds for non-stationary mixing processes, 2017\n\n[R14] Li et al., Optimal learning rates for regularized conditional mean embedding, 2022\n\nQUESTIONS:\nMy questions are detailed in the section \"Strengths and weaknesses\". In particular:\n1. How do your results relate to the works mentioned?\n2. Is there a reason to considering the \"mild\" non-iid-ness of block-independent data compared to mixing data, and how do your proof techniques relate to those for mixing data?\n3. Can you clarify Assumption 1 and the assumptions of $f_\\rho$? In particular, is it assumed to lie in $\\mathcal H$ and, if not, how is this consistent with Assumption 1?\n4. Can you provide guidelines or related works on how to design neural networks to abide by the assumptions of Theorem 4.4?\n\nLIMITATIONS:\nThe paper does not satisfyingly identify the limitations of its Assumption 1, leading to an overblown claim of lack of assumptions on the ground truth, and the limitations of Theorem 4.4 of the difficulty/impossibility for neural networks to abide by the assumptions of the theorem, leading to an overblown claim of practicality.\nOther than that, the authors adequately address the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes kernel ridge regression under a structured non-i.i.d. setting where each latent signal generates multiple noisy observations.  A k-block model captures blockwise intra-dependencies with inter-block independence.  The authors derive excess risk bounds using a blockwise concentration approach.  The results describe how sampling and dependency affect generalization in denoising score learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper provides an explicit excess risk bound for kernel ridge regression under a structured non-i.i.d. block setting, which clarifies how the causal data model and sampling scheme affect generalization.\n2. The analysis connects directly to denoising diffusion probabilistic models, showing how the optimal noise sampling ratio should depend on the time-varying noise-to-signal ratio, which is practically relevant.\n3. The paper develops a Bernstein-type concentration inequality tailored for block-dependent samples, offering a reusable tool for analyzing learning with dependent data.\n\n**Weaknesses：**\n\n1. The paper does not propose a new algorithm but focuses mainly on theoretical bounds, which is not a novel direction by itself given that excess bias-variance bounds for kernel methods have been widely studied.\n2. The new Bernstein-type concentration inequality mainly builds on existing techniques, including the Intrinsic Dimension Lemma and standard $\\phi(a) = e^a - 1$ tricks from related literature, which limits its originality.\n3. In Assumption 1, the notation for sub-Gaussian variance proxies is not used consistently: $\\sigma^2_\\epsilon$, $\\sigma^2_{\\epsilon_{1,2}}$, $\\sigma^2$, and $\\sigma^2_G$ appear without clear distinction. Since this assumption underpins the main risk bound, clearer notation or explicit definitions for each variance term would improve clarity and rigor.\n4. The paper states “Yes” for open access to data and code, but only synthetic data generation details are described; no actual code link or repository is provided, which limits full reproducibility.\n\nQUESTIONS:\n1. The theoretical contribution mainly extends known bias-variance analysis to a blockwise non-i.i.d. setting. Could the authors clarify what key steps or insights make this extension substantially novel compared to existing kernel generalization theory?\n\n2. The paper’s findings on the optimal noise sampling schedule provide some guidance for diffusion models. Could the authors comment on how these results might inspire practical training or data usage strategies more broadly?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops excess‐risk bounds for kernel ridge regression when the training data consist of multiple noisy observations of shared latent signals, a setting that breaches the usual i.i.d. assumption. A new blockwise decomposition technique yields a Bernstein‐type concentration inequality for k-gap independent sequences, leading to bounds that expose how the number of noises per signal k and a relevance parameter r jointly govern variance. The theory is specialized to denoising score learning in diffusion models, where it predicts an optimal k that grows as noise dominates, and small simulations corroborate the trend.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: proofs appear rigorous, with explicit assumptions and detailed appendices, and the novel concentration inequality is broadly useful. Clarity: notation is heavy but consistent; proofs are sketched in the main text with full details deferred, and key intuitions are conveyed. Significance: the work fills a gap by treating structured dependence that arises in modern generative modeling, giving actionable guidance for noise‑pairing schedules. \nOriginality: the blockwise Bernstein bound and its application to kernel regression under causal noise structure are new. \n\nMain weaknesses are strong spectral and Hölder continuity assumptions, reliance on equal k across signals, no comparison with alternative dependent‑data analyses, and only toy simulations on a 1‑D mixture that limit empirical support. Practical impact on diffusion training remains speculative without large‑scale experiments.\n\nQUESTIONS:\nPlease clarify whether the blockwise concentration still holds if k varies per signal or if gaps are only approximately independent. \n\nCould the relevance parameters be estimated from data to guide k adaptively in practice? \n\nHow sensitive are the bounds to the Hölder index p; can slower continuity be handled? \n\nAn ablation on real image diffusion training would strengthen claims; what computational overhead would varying k incur?\n\nLIMITATIONS:\nThe authors include a brief limitations section but should discuss the restrictiveness of the equal‑k assumption and the challenge of estimating r and spectral decay in high dimensions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}