Title: Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limits
Venue: NeurIPS 2025
Accepted: yes
Keywords: hessian clipping

gradient clipping

optimal complexity

stochastic

second-order optimization

heavy tailed noise

normalization
Abstract: Heavy-tailed noise is pervasive in modern machine learning applications, arising from data heterogeneity, outliers, and non-stationary stochastic environments. While second-order methods can significantly accelerate convergence in light-tailed or bounded-noise settings, such algorithms are often brittle and lack guarantees under heavy-tailed noise—precisely the regimes where robustness is most critical. In this work, we take a first step toward a theoretical understanding of second-order optimization under heavy-tailed noise. We consider a setting where stochastic gradients and Hessians have only bounded $p$-th moments, for some $p\in (1,2]$, and establish tight lower bounds on the sample complexity of any second-order method. We then develop a variant of normalized stochastic gradient descent that leverages second-order information and provably matches these lower bounds. To address the instability caused by large deviations, we introduce a novel algorithm based on gradient and Hessian clipping, and prove high-probability upper bounds that nearly match the fundamental limits. Our results provide the first comprehensive sample complexity characterization for second-order optimization under heavy-tailed noise. This positions Hessian clipping as a robust and theoretically sound strategy for second-order algorithm design in heavy-tailed regimes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of second-order stochastic optimization (SOSO) under heavy-tailed noise, a setting where both stochastic gradients and Hessians only have bounded p-th moments, which is common in modern machine learning tasks with outliers or data heterogeneity. The paper establishes tight minimax lower bounds on the sample complexity for any second-order optimization algorithm under heavy-tailed noise. The results demonstrate that second-order methods can still achieve provable improvements over first-order methods, even in heavy-tailed regimes. The paper proposes a normalized SGD with Hessian correction algorithm that incorporates second-order information and provably achieves the established lower bond for finding approximate stationary points. The paper also introduces Hessian clipping, an extension of gradient clipping to handle extreme noise in Hessian estimates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper is well-written and well-structured, easy to read and follow.\n- This paper provides rigorous theoretical contribution.\n- The analysis is technically sound and comprehensive, covering both in-expectation and high-probability results and addressing the impact of gradient and Hessian noise.\n- The originality of this paper is clear and solid.\nWeaknesses:\n- The experimental validation is minimal, consisting only of toy synthetic experiments in low dimensions. There is no evaluation on real-world machine learning tasks to demonstrate practical effectiveness.\n- The proposed methods require assumptions on hyperparameters and question setup, which may limit their practical usability.\n- While theory is strong, there is no discussion of computational costs for Hessian-vector products or clipping in large-scale models.\n\nQUESTIONS:\nPlease refer the the weaknesses section.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies second-order optimization algorithms with heavy-tailed noise. A lower bound on the sample complexity for second-order algorithms is established along with an algorithm to match it in terms of first moment error. A second algorithm with clipping with high probability guarantees for the estimation error is proposed and shown to achieve near-optimal sample complexity.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper has interesting contributions which, as far as I am concerned, are indeed novel. The assumptions, contributions and setting of the paper are clearly identified early on, making it easy to read. I was not able to check the proofs in detail.\n\nRegardless, I would encourage the authors to proof-read their paper as I found a few typos while reading it. (e.g. E[  || \\nabla f(x,\\xi) || ]^p \\leq B when talking about bounds on pth moment).\n \nOne weakness of this submission is the lack of numerical studies to illustrate the theory developed in the paper. While, the authors provide two plots in page 3, very little discussion is provided regarding the setup of their experiments and how these plots are related to the theorems in their paper. I would encourage the authors to include a simple toy problem illustrating their sample complexity bound. Figure 1 shows a plot of the sample complexity, but not details on this experiment are provided.\n\nIn terms of clarity, I also think that there is some work needed. For example, the authors should explicit say which algorithm (i.e. first equation of section B) they focus on in the very beginning of the paper to engage the reader and avoid confusion. Also, the authors reference Table 1 in the main text, but do not mention this table is in the supplementary material, which might create confusion.\n\nI think a statement on how the choice of g_0 changes the sample complexity of algorithm 2 should be included in the main text. Fine details do not need to be included, but the fact that it only changes a constant on the sample complexity and not the dependency on \\epsilon.\n\nQUESTIONS:\n- Is \\xi assumed to be i.i.d.? If not, is the expectation taken at steady-state (if \\xi is Markovian for example)?The authors need to explicitly state their assumptions on the noise. \n\n- Is the BCM assumption that much weaker than the p-moment bound assumption of [Zhang et al., 2020]? Why is the difference between these assumptions that important?\n\n-What is the difference between O(.) and \\Omega(.)? Why the need for two different notations? Also, Thm 1 uses \\Theta(.), what is that?\n\nLIMITATIONS:\nThe assumptions of this paper are clearly identified and the authors provide a discussion on thee limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors present a theoretical analysis of second-order optimization in the presence of heavy-tailed noise, addressing the common issue that such methods often underperform and lack guarantees in this setting. They consider scenarios where stochastic gradients and Hessians have only bounded first- and second-order moments, and derive tight lower bounds on the sample complexity for any second-order method. To complement their analysis, the authors propose a normalized stochastic gradient descent algorithm that leverages second-order information and matches the established lower bounds. They also introduce a novel gradient and Hessian clipping technique, along with high-probability upper bounds to support its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper makes several high-quality theoretical contributions, summarized as follows:\n\n1. It establishes a lower bound on the sample complexity for any second-order stochastic optimization algorithm under the p-BCM noise model, showing that using higher-order derivatives does not necessarily improve the convergence rate under the same noise assumptions.\n\n2. It proposes a stochastic gradient descent method that leverages second-order Hessian information and proves that its sample complexity matches the previously established lower bound.\n\n3. It introduces a gradient and Hessian clipping technique and demonstrates that this method can achieve near-optimal sample complexity with high confidence.\n\nThese theoretical advances contribute significantly to the optimization community’s understanding of second-order stochastic methods in high-noise settings.\n\nThe main weakness of this submission is the lack of sufficient empirical results. The paper should include more numerical experiments evaluating the proposed Normalized SGD with Hessian correction algorithm and the gradient and Hessian clipping methods. Presenting empirical performance would help verify whether these novel algorithms align with the theoretical predictions.\n\nFor instance, since the normalized SGD with Hessian correction combined with clipping requires a clipping threshold parameter $\\lambda$, the authors should explore the impact of varying $\\lambda$ values on the algorithm’s performance. This would help assess whether the clipping technique enhances robustness. Incorporating such numerical experiments would significantly strengthen the paper and, if the empirical results support the theory, greatly increase its publication potential.\n\nQUESTIONS:\n1. From assumptions one to four, this analysis seems to be independent of the convexity of the objective function. All results are applicable in the non-convex setting. If we add the assumption of convexity or even strong convexity, how will these convexity condition improve the lower bound and the sample complexity of the proposed algorithms? All assumptions from this submission are relevant to the smoothness or Lipshiz continuity of both gradients and Hessian. If we add some lower bounds for the smallest eigenvalue of the Hessian, how will the the number of oracle queries required to find an ε-stationary point be reduced? Will this be some future research direction?\n\n2. I'm not sure whether the proposed Normalized stochastic gradient decent with Hessian correction (with gradient clipping and Hessian clipping) belong to the class of second-order optimization algorithms. In fact this proposed algorithm utilize the second-order Hessian matrix information, but in general second-order optimization algorithm correspond to Newton type methods such as Newton's method, quasi-Newton method (BFGS, DFP ,SR1) or L-BFGS with iterations like: $x_{t + 1} = x_t - B_t \\nabla{f(x_t)}$ where $B_t$ is some matrix that approximates the exact Hessian matrix $\\nabla^2{f(x_t)}$. This Hessian approximation matrix $B_t$ is usually constructed using some first-order gradient informations.  However, the proposed variant SGD method has no such iteration updating forms. Hence, strictly speaking, the proposed Normalized SGD with Hessian correction belong to the first-order optimization algorithms which utilized some second-order Hessian queries. This method don't belong to the formal \"second-order\" optimization class.\n\n3. From Theorem 2 and Theorem 3, both the step size $\\gamma$ from the normalized SGD and the clipping threshold $\\lambda$ from the clipping technique requires the estimate the initial sub-optimality $\\delta = f(x_0) - f(x_*)$. How to estimate this initial sub-optimality in practice? Theorem 2 and 3 presented that $\\gamma$ and $\\lambda$ need to be set by values of this initial sub-optimality $\\delta$ to reach the corresponding sample complexity. Hence, the proposed methods are impractical since it is in general very difficult to estimate the initial sub-optimality accurately.\n\nLIMITATIONS:\nThe authors have presented all the limitations of this work in the last section of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper studies nonconvex stochastic optimization for smooth functions where both stochastic gradients and Hessians have $p$-th bounded central moment for $p \in (1, 2]$. The authors establish minmax lower bounds on the sample complexity of second order stochastic methods for the considered problem class and zero-respecting algorithms. A near optimal algorithm, i.e., Normalized SGD with Hessian correction, is proposed and shown to achieve in-expectation upper bounds that almost matches with the lower bound. In addition, the, a clipped variant of the previous algorithm, i.e., Normalized SGD with Hessian correction and clipping, is shown to achieve nearly optimal high-probability convergence.

STRENGTHS AND WEAKNESSES:
Strengths: 
1. This paper for the first time proposes SOSO methods to address heavy-tailed noise. 
2. The minmax lower bound is shown to be tight in several special cases. 
3. Near-optimal algorithms are designed to match the lower bounds in the in-expectation, and high-probability sense with logarithmic failure probability dependence, respectively. 

Weakness:
1. There is a lack of compelling numerical or analytical examples showcasing the advantages of SOSO over first-order methods.

QUESTIONS:
1. In line 160, is it $q$-th order?
2. In definition 1, can the authors provide some examples on algorithms that belong to the defined algorithm family?

LIMITATIONS:
N.A.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03