Title: Tight High-Probability Bounds for Nonconvex Heavy-Tailed Scenario under Weaker Assumptions
Venue: NeurIPS 2025
Accepted: yes
Keywords: weaker assumptions

heavy-tailed noise

(L_0

L_1)-smoothness

optimization

generalization
Abstract: Gradient clipping is increasingly important in centralized learning (CL) and federated learning (FL). Many works focus on its optimization properties under strong assumptions involving Gaussian noise and standard smoothness. However, practical machine learning tasks often only satisfy weaker conditions, such as heavy-tailed noise and $(L_0, L_1)$-smoothness. To bridge this gap, we propose a high-probability analysis for clipped Stochastic Gradient Descent (SGD) under these weaker assumptions. Our findings show a better convergence rate than existing ones can be achieved, and our high-probability analysis does not rely on the bounded gradient assumption. Moreover, we extend our analysis to FL, where a gap remains between expected and high-probability convergence, which the naive clipped SGD cannot bridge. Thus, we design a new \underline{Fed}erated \underline{C}lipped \underline{B}atched \underline{G}radient (FedCBG) algorithm, and prove the convergence and generalization bounds with high probability for the first time. Our analysis reveals the trade-offs between the optimization and generalization performance. Extensive experiments demonstrate that \methodname{} can generalize better to unseen client distributions than state-of-the-art baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides the convergence bounds and the generalization error bounds for clipped-SGD, considering the $(L_0,L_1)$-smoothness and the heavy-tailed noise assumption. The authors also extend the clipped-SGD to federated learning and provide both the convergence bounds and the experimental results to illustrate the effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\na. The theoretical results are new and interesting, which are beneficial for the optimization community. There are no convergence bounds and generalization error bounds for the $(L_0,L_1)$-smoothness and the heavy-tailed noise assumption. Also, the improvement of logarithm order in the convergence bound is interesting.\n\nb. The proof looks sound. Although the proof borrows some techniques from the convergence analysis of heavy-tailed noise and $(L_0,L_1)$-smoothness, it's OK. Also, there are some new proof techniques.\n\nc. It's also interesting to study the federated learning scenario, which is rarely considered in the classical optimization field.\n\n**Weaknesses**:\n\na. Although the results are new, the contribution is a bit incremental. It's well known that the convergence results for clipped-SGD have been derived under standard $L$-smoothness and the heavy-tailed noise assumption. When relaxing to $(L_0,L_1)$-smoothness, the additional gradient inside the smooth parameter can be cancelled by a proper selection of the step-size. One of the central proof techniques still relies on the classical results for the heavy-tailed noise, such as the result in [29] to bound the two types of noise variance, as indicated in Eq. (15) on Page 16. \n\nb. If there are some new estimation techniques due to the $(L_0,L_1)$-smoothness, it would be better to elaborate more in the proof sketch. The proof sketch lacks details.\n\nc. The results for FL seem not very novel in comparison with the ones for CL. I will regard the theoretical results for CL as the major contribution in this paper, while the ones for FL as extension results. If there are some essential differences in the results or the proof techniques, it may be better to present them clearly. Otherwise, the part for FL can be compressed or moved to the appendix while adding more proof details for the results of CL in the main body of the paper.\n\nd. Although there is an improvement on the logarithm order in the convergence bound, the improvement can be considered a bit weak since the dominated order is the polynomial dependency on $T$ instead of the logarithm order.\n\nQUESTIONS:\nIs it possible to obtain a convergence bound with $\\log(T/\\delta)$ order, as the one in [1]? Or is it possible to drop the logarithm factor and match the lower bound?\n\n[1] Ashok Cutkosky and Harsh Mehta. High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems, 34:4883–4895, 2021.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the Clipped SGD method under more relaxed assumptions of $(L_0,L_1)$-smoothness and heavy-tailed noise for nonconvex functions. Current literature tackles only one of the cases but two simultaneously. The authors extend the results to a Federated setting, providing the FedCBG method with similar convergence guarantees. On top of the convergence results, the authors provide generalization bounds for the Clipped SGD and FedCBG methods. Importantly, the convergence results are optimal in the number of iterations $T$.\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths:***\n\n- Strong theoretical guarantees for the Clipped SGD algorithm with an optimal dependency on the number of iterations $T$, only with a logarithmic factor depending on the failure probability $\\delta$.\n\n- I went through the proofs, and they look correct to me. The idea is to inductively bound the martingale differences.\n\n- The analysis does not depend on the gradient bound, which is an improvement over some earlier works. \n\n- Extension to the federated setting and an algorithm FedCBG, which also converges under relaxed smoothness and heavy-tailed noise with a competitive performance in practice.\n\n***Weaknesses:***\n\n- The results seem to be suboptimal in $L_0$ and $L_1$. It looks like we would get the same rates when using SGD with a fixed stepsize. In other words, there is no advantage to using the Clipped SGD.\n\n- The related works section can be improved further. Several important works that also study heavy-tailed noise assumption are not discussed, e.g., [Sadiev et al. \"High-probability bounds for stochastic optimization and variational inequalities: the case of unbounded variance.\" ICML, 2023.], [Gorbunov et al. \"High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise.\" ICML 2024], one of them studies the federated setting as well. \n\n- The current theoretical results in the main paper are hard to parse. The dependency on other problem parameters, e.g., $L_0, L_1$ is missing. Several quantities that are used in the theorems, e.g., $\\Delta_1$ are not defined, or it is not clear how they affect the rate, e.g., $R$ or $\\mathcal{R}$. Similar claims are applied to the results presented in the table, to the proof sketches (what are the constants $A_{1,2,3}$ ?), and to the rate of FedCBG.\n\n- The convergence is given for a local version of $(L_0, L_1)$-smoothness, i.e. the smoothness holds when $\\|x-y\\|\\le 1/L_1$ (assymetric $(L_0,L_1)$-smoothness). It would be interesting to see the convergence results when a more general class of symmetric $(L_0,L_1)$-smooth functions is considered, i.e., for all $x,y\\in\\mathbb{R}^d$ we have $\\\\|\\nabla f(x)-\\nabla f(y)\\\\| \\le (L_0+L\\_1\\max\\_{u\\in[x,y]}\\\\|\\nabla f(u)\\\\|)\\cdot \\\\|x-y\\\\|$.\n\nQUESTIONS:\n- I suggest moving assumptions 2 and 3 to the introduction, as they are used a lot before formally providing them. This would improve the clarity of the presentation.\n\n- There is a concurrent work [Chezhegov, Savelii, et al. \"Convergence of Clipped-SGD for Convex $(L_0, L_1) $-Smooth Optimization with Heavy-Tailed Noise.\" arXiv preprint arXiv:2505.20817 (2025)]. They consider the same setting assumptions but in the convex regime. It would be worth mentioning the paper and comparing the results, though it is not necessary. Interestingly, they obtain the optimal rate only after $\\mathcal{O}(1/\\delta)$ iterations. There is no such restriction in this work. Can the authors provide an intuition why it is so if possible?\n\n- There is no gradient-similarity-type assumption in the theory of the FedCBG method, while vanilla FedAvg's rate does depend on such a quantity. How is it avoided here? Is it hidden somewhere?\n\n- Is the power of $\\log$ terms in the rates optimal, or can they be improved further?\n\n- What is the difference between tables 1 and 4?\n\n- How is the power of $\\log$ terms improved? Does it only come from a more careful choice of the hyperparameters, like stepsize?\n\n- How are problem parameters like $L_0, L_1, \\Delta_1$ present in the rates? Can the authors provide explicit rates? Right now, I suspect that the rates are worse than what I expect in the nonconvex regime (see, e.g., [Hübler et al. \"Parameter-agnostic optimization under relaxed smoothness.\" AISTATS, 2024].\n\n- What is the value of $p$ in the experiments? Can the authors estimate it to ensure that the noise is indeed heavy-tailed with $p\\in(1,2]$?\n\n- Table 3 contains the results on CIFAR10, which is not heavy-tailed as far as I know. Therefore, the caption of Table 3 should be clarified.\n\n- What algorithm is used to obtain Figure 2? \n\n- What does \"recursive momentum\" mean?\n\n- What is the difference between the settings of the two left and two right plots in Figure 2?\n\nLIMITATIONS:\nThe authors provided some discussion in section 7, though it is very limited. In particular, it is not clear what \"recursive momentum\" means and which part of the complexity results they aim to reduce. I suggest also including \n\n- Generalization to symmetric $(L_0, L_1)$-smoothness.\n\n- Dependency on the problem parameters $L_0,L_1$.\n\nin the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive high-probability analysis of clipped stochastic gradient methods under weaker assumptions: (L_0, L_1)-smoothness and heavy-tailed gradient noise. The authors claim that they prove tighter high-probability convergence rates than existing work. Moreover, they design FedCBG, and establish its optimization and generalization bounds in high probability, a first in the FL literature under such weak assumptions. Experiments on CIFAR and Shakespeare datasets empirically validate both convergence and generalization improvements over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper makes solid theoretical advances in stochastic optimization under realistic conditions. Rigorous theoretical development with carefully designed proofs removing strong assumptions (e.g., bounded gradients), supported by martingale and inductive analyses.\n2. Despite the complex technical content, the structure is well-organized, and the derivations and algorithm descriptions are clear.\n\nWeaknesses:\n1. This paper omits some key details, leading to potential ambiguity. For example, the definition of $\\Delta_t$ is not formally given, but the main theoretical results, such as Theorem 1, rely on it. Therefore, it is recommended that the authors complete the key notation description that may be omitted.\n2. This paper has typos or inappropriate descriptions. For example, in line 158, please confirm whether the bound is obtained based on the Jensen's inequality or should be the Cauchy-Schwarz inequality. In line 8 of Algorithm 2, $\\tilde{\\Delta}_t$ should have a subscript about $i$. There is a bracket error in the value of T in Theorem 4 on line 294. And so on.\n3. The main theoretical results of this work rely heavily on [29] which even considers momentum optimization, but the difference between the two is not fully discussed in the main text. It is recommended that (1) the differences and advantages and disadvantages of the two theoretical results be compared in detail; (2) the difficulties in deriving the theoretical results based on [1] be explained, and specifically which results are innovative and which follow existing work.\n\n[1]Liu Z, Zhang J, Zhou Z. Breaking the lower bound with (little) structure: Acceleration in non-convex stochastic optimization with heavy-tailed noise[C]//The Thirty Sixth Annual Conference on Learning Theory. PMLR, 2023: 2266-2290.\n\nQUESTIONS:\n1. The main theoretical results of this work rely heavily on [29], which even considers momentum. However, the distinctions between this work and [29] are not sufficiently discussed in the main text. To improve the clarity and originality of the contribution, it is recommended that the authors: (1) Provide a detailed comparison between the theoretical results of this work and those in [1], including both the differences and the respective advantages and limitations; (2) Clearly explain the specific challenges involved in extending or adapting the theoretical framework of [1] to the current setting; (3) Explicitly identify which theoretical results in this paper are novel contributions and which are direct extensions or applications of existing results from [1].\n\n[1]\tLiu Z, Zhang J, Zhou Z. Breaking the lower bound with (little) structure: Acceleration in non-convex stochastic optimization with heavy-tailed noise[C]//The Thirty Sixth Annual Conference on Learning Theory. PMLR, 2023: 2266-2290.\n\n2. It is recommended that the authors carefully review recent advances in related work. For instance, in lines 39-40, the author claims that \"only one work focuses on the heavy-tail noise problem in federated learning\", but in fact, there are other works that also focus on this. For example, the work of Tao et al. [2] focuses on Byzantine-resilient federated learning in the presence of heavy-tail noise. This suggests that the related work section may not be sufficiently comprehensive and should be revised to reflect the current state of research more accurately.\n\n[2] Tao Y, Cui S, Xu W, et al. Byzantine-resilient federated learning at edge[J]. IEEE Transactions on Computers, 2023, 72(9): 2600-2614.\n\n3. This paper omits some important details and contains notational and typographical issues that may affect readability. For example, key symbols such as $\\Delta_t$ are used in the main theorems without formal definitions. Additionally, there are problems with notational rigor, bracket mismatches, and unclear references to mathematical inequalities. It is recommended that the authors complete missing definitions, standardize the notation, and thoroughly proofread the manuscript to improve clarity, precision, and overall rigor.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}