Title: Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
Venue: NeurIPS 2025
Accepted: yes
Keywords: Non-Euclidean

Frank-Wolfe

deep learning

clipping
Abstract: This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces generalized gradient norm clipping (GGNC), a hybrid optimization method that combines steepest descent and conditional gradient approaches. It extends the concept of gradient norm clipping to non-Euclidean spaces and incorporates weight decay while maintaining convergence guarantees. The proposed method achieves a convergence rate of $\\mathcal{O}(n^{-1/4})$ in stochastic settings. Experimental results on image classification and language modeling demonstrate that GGNC outperforms the Adam optimizer in terms of both performance and convergence speed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1）The proposed GGNC effectively integrates steepest descent and conditional gradient approaches, extending gradient norm clipping to non-Euclidean settings and even constrained problems.\n\n2）In the deterministic case, GGNC is proven to be a descent method, under the generalized $(L_0, L_1)$-smoothness condition.\n\n3）In stochastic settings, GGNC achieves an order-optimal $\\mathcal{O}(n^{-1/4})$ convergence rate by utilizing a momentum-based gradient estimator.\n\n**Weaknesses:**\n\n1）Although the numerical results, especially the result in Fig. 2, are interesting, more experimental results on large-scale model training (e.g., the GPT-2 model series) and the ablation study are lacking and would be valuable.\n\nQUESTIONS:\nQ1. Can the bounded variance assumption used for the stochastic convergence results in Section 4.2 be relaxed?\n\nQ2. Why is the generalized $(L_0, L_1)$-smoothness condition a suitable condition to study for neural networks, as claimed in Conclusion?\n\nQ3. Can GGNC be compared with the Muon optimizer [Jordan et al., 2024b], for example, on NanoGPT?\n\nBy the way, there is a typo on Line 290, where Figure 2 should be replaced with Figure 1.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces a training algorithm that combines norm-constrained linear minimization oracles (LMOs) with gradient clipping. The authors assume $(L_0,L_1)$-smoothness in their theoretical framework and prove an optimal convergence rate of $O(n^{-1/4})$. The performance of their algorithm is demonstrated through experiments on CIFAR-10 and NanoGPT.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well-written with clearly presented theoretical results.
2. The authors extend the understanding of clipping benefits (previously limited to the Euclidean setting) to arbitrary norms, proving descent under a novel non-Euclidean $(L_0,L_1)$-smoothness condition.

Weaknesses:
1. The motivation for using gradient norm clipping is inadequately explained. The approach does not appear to improve the convergence rate.
2. The empirical performance of the proposed algorithm appears nearly identical to the unclipped version.

QUESTIONS:
1. Is the $(L_0,L_1)$-smoothness condition valid in non-Euclidean settings? Do you have empirical verification of this assumption?
2. Compared to Pethick et al. (2025), the optimal convergence rate remains the same in the stochastic case. Why is there no improvement despite additional smoothness assumptions?
3. Pethick et al. (2025) use LMOs to avoid computing the dual norm, but your paper requires dual norm computation for gradient norm clipping. Is this why your learning rate depends on the Lipschitz constant while Pethick et al. (2025) does not?
4. When obtaining $\text{lmo}(d)$, $x$ is already constrained within domain $D$. Why not simply control the norm-ball $D$ directly?

LIMITATIONS:
The theoretical contribution appears incremental, and there is no significant improvement in empirical performance.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new technique that generalizes gradient clipping to non-Euclidean ($\\ell_2$) norms. On the non-Euclidean norms, standard gradient descent is generalized to steepest descent with update $d^\\\\# = \\mathrm{argmax}\\_x \\langle d,x\\rangle - \\frac{1}{2}\\\\|x\\\\|^2$, while normalized gradient is generalized to conditional gradient via linear minimization oracle (lmo), that is $lmo(d) = \\mathrm{argmin}\\_{\\\\|x\\\\|=1} \\langle d,x\\rangle$. However, it was unclear what is the correct analogy of gradient clipping on non-Euclidean norms. To close this gap, this paper observes a key connection between the two methods, that is $lmo(d) = -d^\\\\# / \\\\|d\\\\|\\_* $, and proposes the generalized gradient clipping defined as $\\min\\\\{1, \\frac{\\rho}{\\\\|d\\\\|\\_*} \\\\} \\cdot d^\\\\#$.\n\nBased on this core idea, this paper provides two variants of gradient clipping, for unconstrained and constrained problems. In both settings, the authors provided convergence analysis in both deterministic and stochastic setting, and proved that the proposed gradient clipping algorithm converges to the optimal rate. In addition, the authors also validated the algorithm on both image classification and nanoGPT pre-training tasks, and showed that the proposed algorithm achieves better performance compared to the Adam baseline and Scion, an algorithm based on conditional gradient (analogous to normalized gradient on carefully chosen norms) instead of gradient clipping. These empirical results demonstrate the practical benefits of the new algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper has several notable strengths. First of all, it introduces a novel technique that extends gradient clipping to non-Euclidean norms. By connecting steepest descent update and lmo update in conditional gradient, the authors managed to define a meaningful clipping on general norms.\n\nAdditionally, the paper makes a solid contribution to the literature by addressing the previously unresolved question of how to implement gradient clipping for general non-Euclidean norms. Prior to this work, the correct analogy for gradient clipping in non-Euclidean norms was unclear, leaving a gap in the understanding of optimization techniques. The authors successfully fill this gap, providing a valuable resource for researchers and practitioners looking to improve their optimization strategies.\n\nFinally, the proposed algorithm is supported by solid evidences, both in theoretical analysis and empirical experiments. The authors demonstrate that their proposed gradient clipping algorithm converges to the optimal rate in both deterministic and stochastic settings, and for both constrained and unconstrained problems. This theoretical foundation is complemented by empirical validation on tasks of image classification and nanoGPT pre-training, where the algorithm outperforms existing methods like the Adam baseline and Scion. These results demonstrate the empirical effectiveness of gradient clipping with general norms.\n\nOverall, I think this paper has solid results, thus recommend acceptance.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Generalized Gradient Norm Clipping (GGNC), a hybrid first-order method that interpolates between steepest descent (SD) and an unconstrained conditional-gradient (uCG) step by clipping the dual gradient norm with respect to an arbitrary norm. Under a generalized (L₀, L₁)-smoothness assumption, the authors prove:\n\n1. A descent lemma and an O(n⁻¹) deterministic rate to a neighborhood followed by O(n⁻¹/²) convergence near stationary points.\n2. An order-optimal O(n⁻¹/⁴) stochastic rate via a momentum estimator.\n3. An interpretation of weight decay as a clipped Frank-Wolfe \"short step,\" along with the Stochastic Short-Step Conditional Gradient (S3CG) algorithm.\n4. Instantiations for ℓ∞ (\"Clipped Sign\"), spectral, and product norms, accompanied by small-scale experiments on CIFAR-10 and NanoGPT, suggesting faster early-stage progress compared to unclipped methods.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n\nStrengths:\n\n* Rigorous extension of the (L₀, L₁) framework to arbitrary norms; proofs are mostly complete and technically competent.\n* The connection between clipping and Frank-Wolfe short steps is elegant and may interest the optimization community.\n\nWeaknesses:\n\n* Theory largely recycles existing proof templates for Euclidean clipping; novelty is modest.\n* Crucial practical aspects—computational cost of the sharp operator and lmo for deep networks, choice of radius ρ, and stability when the dual norm is ill-conditioned—are not analyzed.\n* Stochastic guarantees require known L₀ and L₁ to set stepsizes, conflicting with the \"parameter-agnostic\" narrative in the introduction.\n\nClarity\n\nStrengths:\n\n* High-level motivation is clear and algorithms are presented in pseudocode (Algorithms 1–2).\n\nWeaknesses:\n\n* Notation is heavy, and several definitions (e.g., \"product norm\" radii) are buried mid-paragraph, making reproduction difficult.\n* Experimental section omits many details (network width, regularization, hardware, etc.); Appendix C only lists a subset.\n* Figures 1–2 plot clipped and unclipped variants but omit error bars for Adam, hindering interpretation.\n\nSignificance\n\nStrengths:\n\n* May stimulate further work on non-Euclidean clipping and constrained optimization links.\n\nWeaknesses:\n\n* Empirical impact is unconvincing: gains on CIFAR-10 are approximately 1–2 percentage points and vanish once a learning-rate schedule is introduced.\n* Experiments are confined to toy-scale models; no evidence on large-scale vision or language tasks where clipping is commonly used.\n* Competing adaptive methods (AdamW, Lion, RMSProp) are not compared, limiting relevance for practitioners.\n\nOriginality\n\nStrengths:\n\n* The specific synthesis of SD, uCG, and Frank-Wolfe short steps under (L₀, L₁)-smoothness appears new.\n\nWeaknesses:\n\n* Core idea—clip the gradient in a dual norm—has been folklore; the paper formalizes but does not fundamentally change practice.\n* Prior concurrent works (Gorbunov et al., Vankov et al.) already extend clipping under relaxed smoothness; the non-Euclidean lift is incremental.\n\nQUESTIONS:\n1. Computational overhead: For spectral or product norms, the lmo requires an SVD per update; how does GGNC scale to ImageNet-size models? Please provide wall-clock comparisons.\n\n2. Sensitivity to ρ: Figures use a fixed clipping threshold, but how robust is performance to mis-specifying ρ by ±2×? A sweep would strengthen claims.\n\n3. Adaptive stepsizes: Your theory requires γ ≤ 1/L₀. In practice, L₀ is unknown. Can backtracking or AdaGrad be combined with GGNC while retaining descent? Empirical evidence would increase the quality score.\n\n4. Baselines: Why are AdamW, Lion, and normalized SGD with momentum absent? Demonstrating advantages over these would affect my significance assessment.\n\n5. Assumption tightness: Is (L₀, L₁) actually satisfied for deep nets in the modular norm? Please supply empirical Lipschitz estimates or counterexamples.\n\nI would consider increasing my score if robust large-scale experiments with adaptive stepsizes and stronger baselines show clear benefits, or if you provide a cheap approximation of the lmo/sharp operator that preserves convergence.\n\nLIMITATIONS:\nThe paper includes a \"Limitations\" checklist entry but does not discuss computational cost, hyperparameter sensitivity, or failure modes in depth. These should be added.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}