Title: Regularized least squares learning with heavy-tailed noise is minimax optimal
Venue: NeurIPS 2025
Accepted: yes
Keywords: kernel methods

nonparametric regression

kernel ridge regression
Abstract: This paper examines the performance of ridge regression in reproducing kernel Hilbert spaces 
in the presence of noise that exhibits a finite number of higher moments. We establish excess
risk bounds consisting of subgaussian and polynomial terms based on the well known integral
operator framework. The dominant subgaussian component allows to achieve convergence rates
that have previously only been derived under subexponential noise—a prevalent assumption in
related work from the last two decades. These rates are optimal under standard eigenvalue decay
conditions, demonstrating the asymptotic robustness of regularized least squares against heavy-
tailed noise. Our derivations are based on a Fuk–Nagaev inequality for Hilbert-space valued
random variables.


Peer reviews:
comments: SUMMARY:
This paper studies the generalization performance of ridge regression under heavy-tailed noises,
showing optimal rates similar to that derived unde subexponential noises in the literature.

STRENGTHS AND WEAKNESSES:
## Strengths

* This paper is well written and easy to read.
* This paper extends the analysis of kernel ridge regression under general heavy-tailed noise, which is novel.  By showing the same optimal convergence rate as the sub-Gaussian noise case, the paper also shows the robustness of regularized least squares against heavy-tailed noise.
* In terms of technical contribution, the application of Fuk–Nagaev inequality in this paper can also be useful in other kernel-related problems.

## Weakness

* Numerical simulations could be provided to validate the theoretical results and further demonstrate the methods robustedness, particularly for noises with different tail behaviors.

* This paper does not provide practical ways of choosing the hyperparameter $\alpha$ under different confidence levels, limiting the paper's real-world implications.

QUESTIONS:
1. Can you discuss the requirement of $q \geq 3$ in Assumption 2.2? Can it be improved to $q > 2$?

2. As mentioned in the paper, the typical assumption that the kernel is bounded leads to subgaussian concentration of the embedded covariates. Can the techiques in this paper be further applied to "heavy-tailed" kernels that is unbounded?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the performance of ridge regression in reproducing kernel Hilbert spaces (RKHS) when the observational noise has heavy tails, meaning it lacks exponential tail decay. Unlike prior work assuming subexponential noise (e.g., satisfying a Bernstein condition), the authors only require a finite number of conditional moments. They prove that ridge regression remains statistically robust under these weaker assumptions by using a Hilbert-space version of the Fuk–Nagaev inequality. They show that the convergence rates match known optimal rates under standard eigenvalue decay assumptions, thereby establishing that regularized least squares is minimax optimal even under heavy-tailed noise. The work generalizes existing theory and demonstrates that tight learning guarantees are achievable without strong tail assumptions, making kernel methods more broadly applicable in practice.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the paper is clear and well structured, results are explained and motivated precisely. The assumed setting is meaningful and tries to solve an interesting problem. The paper brings technical novelty to widespread ridge regression and the level of technical sophistication of the tools and analysis is high.\n\nWeaknesses: no important ones. Despite the paper being clearly theoretical, it would have been interesting to see some practical application or simulation under this heavy-tailed noise.\n\nQUESTIONS:\n- Have you considered or do you plan to validate your theoretical results with empirical experiments — particularly comparing performance under subgaussian vs. heavy-tailed noise?\n\n- The choice of the squared loss is not deeply discussed in the paper. Do you think your analysis—particularly the use of Fuk–Nagaev inequalities in Hilbert spaces—can be extended to learning problems with general convex loss functions? For instance, would similar results be attainable for regularized learning algorithms such as SVMs or other kernel-based methods involving Lipschitz or piecewise-linear losses?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the performance of ridge regression in reproducing kernel Hilbert spaces under heavy-tailed noise conditions, where the noise has only finite higher moments. The authors derive excess risk bounds for ridge regression consisting of subgaussian and polynomial terms, demonstrating that the dominant subgaussian term allows for convergence rates previously only achieved under subexponential noise assumptions. The results show that the derived rates are minimax optimal under standard eigenvalue decay conditions, proving the robustness of regularized least squares against heavy-tailed noise. The analysis relies on a Fuk-Nagaev inequality for Hilbert-space valued random variables.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The use of the Fuk-Nagaev inequality is innovative and well-justified.\n\n2. The results cover both capacity-independent and capacity-dependent settings, offering a comprehensive view of the problem.\n\n3. The work has significant implications for robust statistical learning, as it shows that optimal rates can be achieved even with heavy-tailed noise.\n\nWeaknesses\n\n1. While the theoretical contributions are strong, the paper lacks empirical validation. \n\n2. The comparison to related work may lack sufficient detail (e.g. [37-38]) and may be unclear to non-experts.\n\n3. A discussion of practical implications is absent. How do the results inform the choice of loss functions in the presence of heavy-tailed noise?\n\nQUESTIONS:\n1.\tWhy isn’t least squares commonly used in practice under heavy-tailed conditions? Providing some insightful remarks would be beneficial.\n\n2.\tHow do the derived rates compare to those of robust regression methods (e.g., Huber loss) under the same noise conditions? \n\n3.\tThe distinction between subgaussian and polynomial regimes is insightful. Are there practical guidelines for choosing the regularization parameter in each regime?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the behavior of ridge regression in reproducing kernel Hilbert spaces when the noise is heavy-tailed (only finite qth moment for q>= 3). They discuss the excess risk bound by using vectorized version of standard Fuk-Nagaev inequality used for heavy tails. Further the authors claim that ridge regression achieves minimax optimal sample complexity. Hence, as compared to existing methods, the paper is giving theoretical guarantees for minimax optimal results for heavy tailed noise which was assumed to be sub-exponential before.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths: \n- Achieving the minimax optimal guarantees \n- Generalizing the assumption to heavy tailed noise instead of sub-exponential  distribution. \n- The generalized extension of Fuk-Nagaev inequality with decomposition of tail behavior into sub-Gaussian and polynomial term is novel. It can be used for other works. \n\n## Weakness: \n- There is no empirical evidence of the proposed theoretical claims. \n- Heavy tailed noise can be motivated more in the introduction section. \n- What specific kernels follow assumption 2.1 or can you examples of kernels which do not follow assumption 2.1? Do the polynomial kernels in https://proceedings.neurips.cc/paper_files/paper/2015/file/f7f580e11d00a75814d2ded41fe8e8fe-Paper.pdf follow Assumption 2.1?\n\nQUESTIONS:\n1) Is the computational complexity affected by heavy tailed noise? If yes, what is the specific relation?\n2) What will be the performance of other regularization methods such as Lasso for heavy tailed noise? Can the minimax results be claimed for other regularization methods?  \n3) In Assumption 2.2, the authors assume q >= 3. Can the analysis be extended for q >=2? The authors discuss existing works in line 93 to have results for q > 0. \n4) Can the authors clarify their contributions as compared to [51]? In line 94, the authors say that [51] follows a similar approach.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}