Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
Authors: Thomas Pethick
Wanyun Xie
Mete Erdogan
Kimon Antonakopoulos
Tony Silveti-Falls
Volkan Cevher
Venue: NeurIPS 2025 oral ()
Keywords: Non-Euclidean
Frank-Wolfe
deep learning
clipping
Primary area: optimization

Abstract: This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Generalized Gradient Norm Clipping & Non-Euclidean (Lo, Li)Smmohhhsss
Thomas Pethick'
Wanyun Xie EPFL (LIONS)
Mete Erdogan
EPFL (LIONS) thomas. cehhickepffcc
EPFL (LIONS) mete mete. dette.erdogan@epfl.cd
wanyun.xie@epfl.ch
Kimon Antonakopoulos
Antonio Silveti-Falls Université Paris-Saclay (CVN)
EPFL (LIONS)
kimon, k sttnon..nttonakooouossssep......
tonys. falmsma.llm@maii..com
Volkan Cevher
EPFL (LIONS)
volkan volkan.ccvhereeeee cewher@epfl.ccc
Abstract
This work introduces hybrid non-Euclidean optimization method which gen- eralizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion decay is incorporated in principled manner by identifying connection the Frank-Wold short step. In the stochastic case, we show an order optimal 0nr-1/4 convergence rate by leveraging momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties image classification and language modeling. The code is available at https: //gi thub. -Ctp//LLOOSPELLCCI.IIpP
(Lo.LL)-smmothnesss. Weight
1 Introduction
Recent work [Pethick al., 2025] has shown that conditional gradient methods?. traditionally used for constrained optimization, can also solve unconstrained problems Soflems-offering alternative to steepest descent. From their analysis it becomes apparent that the two methods have distinct properties: whereas steepest descent requires the stepsize for L-smooth objectives to be taken y 2/1, conditional gradient methods have no such requirement, thus allowing for large stepsizes, while remaining stable.
The price pay for the stability that conditional gradient based methods are not descent methods and thus eventually needs a diminishing stepsize to converge, even in the deterministic case. The problem becomes very apparent the iterates are close to the solution, since the iterates always move by fixed magnitude and are thus pushed away from the solution. Steepest descent does not suffer from the same problem since the effective stepsize automatically becomes smaller the iterates approach a solution. This observation naturally raises the following question:
'Equal contribution.
BB conditional gradient based methods, we mean those methods which leverage linear minimization oracle Imo d) = arg xx) when updating their parameters with an open-loop stepsize.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Special instantiations of Algorithm 1 according to different choices of norm. Control on the
norm of the parameters is guaranteed by the constrained variant the method (Algorithm 2).
Method
Norm type
Norm hall
Imo(d)
Md]
Reference
Clipped GD
Vector
Euclidean Ip-bbll
[4]
[Miknlov al., 2012]
Clipped
Vector
Max-norm
lu-hall
signtd)
This paper
Clipped Spectral
Matrix
Spectral noem I Is -ball
-/VI
This paper
Clipped Scion (Algorilims 3 and 4)
Product
Max-noom ball over layers
In Imio (dolhapp)
E, Imvid))
This paper
The reduced SVD is given U diag(or)V".
Can we combine the two methods and get the best of both worlds? That is, does stable method exist which takes large steps initially but adapts the stepsize when near a solution?
In this paper we answer the above in the affirmative by considering hybrid method that combines a conditional gradient method with steepest descent. The proposed method generalizes gradient norm clipping [Mikolov et al., 2012] beyond the Euclidean case. In practice, gradient norm clipping has been widely adopted to stabilize training of recurrent neural networks (RNNs), Transformers and diffusion models, especially in large-scale settings. Theoretically, precise characterization of the benefits has emerged under the (Lo. Lmmooothsssssss assumption [Zhang et al., 2019, 2020, Koloskova et al., 2023]. Expanding on this, we show that these benefits of clipping can be made compatible with non- -uncldeid methods. Besides clipping, we provide a novel analysis of conditional gradient methods without clipping under these same smoothness assumptions.
Concretely, we make the following contributions:
(i) We introduce hybrid method between conditional gradient method and steepest descent (Algorithm 1), which in the Euclidean case recovers gradient norm clipping. The benefit of the hybrid method is made precise by showing descent property under generalized (Lo, smmothhessss condition.
(ii) In the stochastic case we show an order optimal Onn-1/4) rate by leveraging a momentum estimator. Convergence for clipped algorithm with stochastic feedback appears to be new even in the Euclidean case.
(iii) We establish a connection between clipping and the short step from the Frank Woolee literature, which similarly enjoys descent property. The connection enables us to combine clipping with weight decay in a principled manner that maintains convergence guarantees. We propose stochastic variant of the short step (Algorithm 2) and establish O(11 rate.
(iv) We explicitly instantiate the algorithms for deep learning through a product norm over layers (Algorithms and 4) and demonstrate their properties through experiments on image classification and language modeling.
Preliminaries
Given a continuously differentiable objective function f X - R. the classical gradient descent method (GD) with a stepsize y > 0 can be written as
arg min y(V fx)),,) H|lx x11 - ((x))
(GD)
xeX
The normalized gradient descent method with radius 0 is. in comparison, defined as follows
x = arg min (Vf(())))) = x² S
arg min y (VV = -
PTR
(Normalized GD)
hybrid variant is much more popular in practice,
arg min (Vf(()))))) x) + -|lx - x'11 = x min|1, po vV((+))
(Clipped GD)
which we notice can be rewritten by combining GD and Normalized GD, Indeed, all three of these algorithms correspond to minimizing
R(x)

---

for different choices of R. For GD, R(x) =|| x11 while for Normalized GD, R(x) = 40(X x²). the indicator function for Euclidean ball 2) = [x: | Σ 1] scaled by the radius p; Clipped GD combines both by taking R(x) >|,x x11 + 1, - r²). This results in the iterates of Clipped GID being generated by the update in Normalized GD II f(x is large, but reducing to the update in GD when ||V f(x Ill is small enough.
Observation Our first observation is that both GD and Normalized GD can be generalized to the non-Fuclidean case. Define the sharp-operator [Nesterov, 2012, Kelner et al., 2014].
d' arg max( x) - -||x||²)
xeX
Then, we can write the (possibly am--uulidean) steepest descent method (SD) as follows
++1
yVV
(SD)
Observe that recover GD when choosing the Euclidean l norm.
Generalizing Normalized GD to non-Euclidean norms is possible by noticing that the normalization can be written in terms of the linear minimization oracle (Imo)
Imo(d) E arg min (d,
xED
where the constraint is a (now assumed to be non- -on--uclidean) norm-ball D == (x ||x|| s 1). By choosing the norm ball, Normalized GD can be seen as an instance of the so-called unconstrained conditional gradient method (uCG) [Pethick et al., 2025].
(uCG)
Observation II
Our second central observation is that uCG can in general be considered normal- ized version of steepest descent. This relationship follows from noticing that the sharp operator and Imo can be defined in terms of each other. Specifically, we have that
Imo(d)
d [d.
equivalently,
= Imo(d).
In the following section use this observation generalize Clipped GD the non-Euclidean case.
3 Method
We propose the generalized gradient norm clipping method (GGNC)
= =
yr with T min(1, (....)
(GGNC)
There is freedom in how to compute the dual norm || due to the following equivalence property for the sharp operator, ||s|l2 ||5 = (s, st). This form is useful, e.g., in the Euclidean case where the sharp-operator is readily available, since then [=+1==dd
For norm choices where the Imo is naturally available we equivalently write GGNC as
++1 x + ym Imo(d") with 7 := min(p, [[d
We have that ||d|||- = -d" Imo(d")) due to the definition of the dual norm and the optimality of Imo(d"). So, provided that imo has been computed, we can obtain ||d' with very little overhead. From this rewriting we also see that p can also be interpreted as the radius of the norm-ball constraint over which we compute the Imo.
The GGNC update rule can be seen as the solution to the following optimization problem:
A+1 E arg min y (d" x +|Ix x1112
The objective is the same quadratic approximation that gives rise to SD, but the iterates are further constrained to trust-region of radius p in the chosen norm, in uCG.
3

---

Algorithm Generalized Gradient Norm Clipping (GGNC)
Input: Horizon n, init. x¹ X, d O, momentum ag (0, I], stepsize E (0, 1)
1: for k II...., n do
2: Sample & p
3: de - ag Vxx E1) + (1 ad
4: v Imo(d")
5:
Th - mintp, (d" vt
6:
x x yme
7: Choose X uniformly at random from (x'.......
Return x
Equivalently step 4-6: r - If y with Tı min(1, (
and 1A [[d
Algorithm 2 Stochastic Short Step Conditional Gradient CSC
Input: Horizon n, init. x² E BD = (x E X: ||x|| s B), d = 0, momentum og E (0, 1J. stepsize
y (0, ball radius /
1: for I....., Sample & P
2:
3:
dt + -x f(x (((x)E))))) + (1 - a()
4:
y² v -Blmo(d
5:
Variant 1: 17 min(p,
6: Variant 2: 1/2 t min(p.
7: et1 x x *
8: Choose x² uniformly at random from (x) .x!."....")
x*)
Return F
Stochastic case In the deterministic case we can simply take the direction to be d' = f((*). In the stochastic case, one has proceed with more care, since Imo(d") can be biased even when d' is unbiased, due to its potential nonlinearity. With as E (0, 1], we define the momentum based gradient estimator
d = (1 - ar ax &E).
The final algorithm involving the momentum based gradient estimator is presented in Algorithm 1.
Weight decay & constrained problems Weight decay is very popular technique, both regularizer to avoid overfitting and for ensuring numerical stability. A precise characterization exists for weight decay when combined with the conditional gradient based schemes like uCG, since the resulting update reduces to the classical conditional gradient method (a.k.. Frank- Wolfe) designed for solving constrained problems [Chen al., 2023, D' Angelo al., 2023, Xie and Li, 2024, Pethick et al., 2025).
k+1 = (1 7x)x (mo(m()
(CG)
where B > O is the radius of norm- ball constraint and ys > 0 is some stepsize to be defined. The simplicial combination ensures that the iterates remain within the constraint set BD and, result, ensure that ||**1|| s / for all
The CG method is not necessarily a descent method. For the classical open-loop stepsize choice Y 3/4+2, is possible to step too far the direction given by the Imo, since the stepsize does not decrease near critical point. Naively adopting the adaptive stepsize choice from GGNC does not seem appropriate the constrained case, since ||/|]|, might not necessarily zero at a solution. Instead, we will argue that the correct analog of clipping in the constrained setting corresponds to a clipped version of the Frank- Wolfe short step. Like GGNC, this stepsize ensures an analogous descent property.

---

The short step is almost an immediate consequence of the L-smoothness descent lemma, from which we have
1( ((
(lmo(V + 2 (lmo
(2)
By optimizing this bound with respect to ys. we arrive at two variants of the short step
Y min[1, "T 211--plmaVV(e"l
y min "9fo""a'-8lmoxV/
41,
where the second variant is useful when the norm || || expensive to compute. What is particularly noteworthy of these stepsize choices is that they lead to descent, i.e., f(x*+1) s f(x"), by construction.
We extend these stepsize choices to the stochastic case with Algorithm 2, where we propose slightly different parameterization given by
(f. WftttBBlootd T| [f---olmodd III
?e min(p,
or m minlp,
48"
careful reader might have noticed the similarity between the short step Algorithm 2 and gradient clipping in Algorithm 1. These schemes are indeed equivalent when v is appropriately modified in Algorithm 2 to Imo(dd)) This connection motivates our parameterization of the updates in Algorithm 2, which are scaled by Bym,, so that the following holds
Byn: = By min(p,
-dd (ddtBllmmddd)) [8lmo(d)))
A| = By min(p. im xx By minlp,
Alrll minlp, [|.")
#
The modified Step 7 of Algorithm 2 then becomes
x = + y minlp, Imo(d")
which is exactly what is used in GGNC.
3.1 Norm choices
Algorithm and Algorithm 2 crucially generalize beyond the Euclidean case of Clipped GD, The following section focuses on the unconstrained variant (Algorithm 1) for simplicity, but its constrained counterpart follows in straightforward way through Algorithm 2.
Sign A simple non non-Euclidean example is the to vector norm for which GGNC reduces to sign-based update
x = x ym sign(d")
(Clipped Sign)
where Th == minlp, [[] The update is dense in the sense that each coordinate undergoes the same magnitude change.
Spectral The matrix analog of the too norm is the Schatten-oo matrix norm, a.k.a. the spectral norm, which induces the following update
- ynp U*
(Clipped Spectral)
where the reduced singular value decomposition (SVD) is given d U* diag(o (V The dual norm can be computed given the Imo as ||o-5|| = ||d'lss = - (d Imo(d')) = tr(lmo(dd)) T flatten(Imo(d") flatten(d)), where ||- lls, is the Schatten-1 norm, a.k.a. the nuclear norm. This scheme is a clipped variant of the stochastic spectral descent method [Carlson et al., 2015b,a].
Product norm The neural networks in deep learning consist of multiple layers and will therefore be useful to consider what we will call product norm. Consider x (W,..., Wp). A norm of can be composed using norms on [Wilepp)
|x]|| IKC (IWill .... " IWllll,
for radius parameters F7 0. Notable choices of || llx include the fn-nomm [Flynn, 2017] and the Eas fas-norm choice made by the modular norm [Large et al., 2024]. Interestingly, if |f llx is the max-norm, || Ilx = || I|.., then:
(i) The Imos can be computed separately as Imox(x) = (ri Imow. (W1), TD Imow,
ii The dual norm requires summing over all elements, i.c., ||x||+ ΣP, AllWillw...
5

---

As particular example, consider the LARS optimizer [You et al., 2017], which performs normalized SGD layer-wise. The update rule can be written in terms of the Imo-based scheme uCG with the norm choice ||^|| maxll|Willr. Writing the analog sharp-operator based scheme (i.e., SD), we see that it does not correspond to simply removing the normalization as for the & norm. Instead, using the relationship (1), we see that the correct form for the hybrid GGNC method is
W = W y minlp, 2,, [ld))=)) Tof
V [D]
where d = ..., de) and y 0 is the stepsize. Through this duality, see that while the Imo only requires local information, the dual norm computation (and consequently also the sharp-operator in SD) requires global information.
In Algorithms 3 and 4 of the appendix we specialize Algorithms 1 and 2 the particular case where I| Ilx is the max-norm. The resulting algorithms can be seen as clipped variants of the (unconstrained) Scion algorithm [Pethick al., 2025] so we refer to them (UNCONSTRAINED) CLIPPEDSCION.
Analysis
Why might be useful to consider hybrid of SD and uCG? As will see, the convergence properties of the two methods are complementary. One can show for SD under L-smoothness that
f(x²*1) S f(x) - y(1 - //DNIV
In other words, SD is descent method in the sense that it decreases the function value f(x²) at every iteration. The price we pay for this descent that the stepsize needs to be taken sufficiently small, specifically as y < 2/1..
On the other hand, under the same L-smoothness assumption, uCG instead satisfies
)(+))) s f(x)) ->pl//)>>>llVV--- f(x")l.
1y +2
Notice that this is not descent method, due to the positive contribution of Ly'p", 2_ However, there are no restrictions on the stepsize, and we can in fact show a fast rate of O(1)s) for the norm of the gradient with a constant stepsize (as opposed to 00/ vk) of SD), albeit only to neighborhood whose radius is proportional to yp, as we formalize the following result.
Proposition 4.1. Suppose is L-smooth with respect to || ll. and denote f inf seX f(x). Then, the iterates (x keNN of uCG satisfy, for all n E N
min fV Ml. s E || Iskea
f0'')-1- ypn 2
Lye
Recall that GGNC reduces to uCG when the gradient norm is large, so we can expect in the early Typ phase GGNC will converge rapidly to neighborhood of size 2- If the gradient norm is small in this region, then GGNC reduces SD, which converges to an exact critical point even with constant stepsize and which can adapt to the loss landscape through the gradient norm.
We can make this intuition precise by analyzing these algorithms under the following generalization of (Lo, L1) smoothhess to arbitrary norms.
Assumption 4.2. The gradient f is said to be LL -smooth with Lo, L E (0, if. for all x. e with [|x - yll < T it holds
f(x vf(yil. S (Lo + -Ill -ll.
(4)
4.1 Deterministic case
We now proceed to generalizing Koloskova al. [2023, Thm. 2.1] in the deterministic case. The main argument relies on establishing that GGNC (Algorithm 1) is descent method even under the generalized (La, Li)-smoothhesss assumption, which enables the scheme to converge even for a fixed, horizon-independee
stepsize y. For the remainder of the paper, we will always denote f* := inf,ex (xx (where is understood this infimum is taken over (2 for constrained problems) and A =

---

Theorem 4.3. Suppose Assumption 4.2 holds and let E N'. Consider |x lsska generated by GGNC with d= Vf(('), and y S 1/Lo-p))). Then, the following holds
min /V
24
ypu
Specifically, with p Li
La and Ln we have
min |IV f(' 1skgn
LaA 24
Remark 4.4, Note that the condition || x I 1/4, of Assumption 4.2 required in the proof is always satisfied, since YP s Vı, holds for any p. We note that descent can also be established for SD with an adaptive stepsize Yv: = 1/ (see e.g., Balles et al. [2020, C.2.2], which uses a definition of (Lo, smoothness based the Hessian).
In contrast with GGNC, uCG is not descent method and requires diminishing stepsize to converge as suggested by the following theorem. The uCG method trades off the descent property with being agnostic to the Lipschitz constant Lo-
Theorem 4.5. Suppose Assumption 4.2 holds and let E E.. Consider (s"Isssn generated by uCG with yp 1/21. Then, the following holds
min ISASH
7p 2LOyp.
Remark 4.6. The assumption that YP 1/21, can be relaxed to yp < 1/4, while still ensuring convergence, modulo different constant the convergence rate.
Let us now turn the constrained case. The following theorem establishes convergence for Algorithm in the deterministic setting, i.e., with = =Vf(x). The convergence rate is established for quantity called the Wolfe-gap.
max V (xx,xxx-x) MERD
which, when equal to certifies that critical point for the constrained problem. It is the equivalent of the dual norm of the gradient but for constrained problems, since the gradient might not vanish a critical point in the constrained setting. The theorem also includes an assumption that L-smooth rather than (Lo, -smoohhh Because the iterates of Algorithm 2 are guaranteed to never leave the compact set BB, L-smoothness implied by (Lo, sm)smohtsssss here.
Theorem 4.7. Suppose is L-smooth and let N Consider (sshs generated by Algorithm 2 with d f(((), s 1. and S so that yp S 1. Then, for all u E BDD the following holds
min (Vx(x) n) 2B 1kke
24
4.2 Stochastic case
consider the following standard assumption about the bias and variance of the stochastic oracle. Assumption 4.8. For the stochastic gradient estimator v((EE) Rd the following holds.
(i) Unbiased: B& [V)( = vx(() Vx
(ii) Bounded variance: Be [VV V S(OI
[ o
VreX.o20
In order to establish convergence in what follows, important quantity to introduce is the error produced by the stochastic estimator d², which we denote by 2 := d -v((xt)
We establish the following order optimal convergence guarantee for GGNC under (Lo, L,) smoothnesss using a momentum based estimator. These convergence results for clipping with momentum appear to be new, even in the Euclidean case.
Theorem 4.9. Suppose Assumptions 4.2 and 4.8 hold and let n N. Consider the iterates {x 1s)ss generated by Algorithm with constant stepsize S 1/L, and yp S 1/21, Then,
B(IV)(("").]
A VON
c
8c.
ypn

---

where == f(x'- and 6, : Ei== O( 111 + 111
Furthermore, assuming f is L-smoothh and taking a = 1/ vi, y 1/ voto and p 14/20/11, such that yp L,, we have that
/LI & O (
Remark 4.10. For case of exposition, the guarantee is presented with horizon horizon-dependent parameter choices, but the result can be extended an any time guarantee in straightforward manner by choosing the parameters as a function of & instead of n and modifying the proofs accordingly.
In the constrained case, we have the following convergence guarantee for SCG with a clipped short step (Algorithm 2) using momentum based estimator. To the best of our knowledge, this is the first convergence proof using the short step in the stochastic setting.
Theorem 4.11. Suppose Assumptions 4.2 2 and 4.8 hold and let E N'. Consider the iterates (s generated by Algorithm 2 2( (Variant 1) with a constant stepsize s 1/t and p S Ynl1. Then, for all u E BD,
B(V X 4)]
4v V
3A 7pm
86.
where A f(x)- f and == Ei=i O V 1 + 12)
Furthermore, laking o 1/ y 1/a. and p 1/ such that yp 1/(ne we have that
(((V)(²") x .)] = ((t))).
We additionally provide an identical guarantee for Algorithm 2 (Variant 2) in the appendix.
5
Related work
(Lo, smmoohhhessssss An (Lo, -mmoothhsssssss condition was introduced based on the Hessian in [Zhang al., 2019] and later generalized the first-order notion that extend the non-Euclidean case [Zhang et al., 2020]. (Lo.Li)-smoothee
was used to analyze signSGD under heavy-tailed noise assumptions in Kornilov al. [2025]. A coordinate-wise (La.oii-ssooohnness condition has also been considered for analyzing a generalized version of signSGD [Crawshaw et al., 2022].
In the Euclidean casc, a descent property under (Lo, -mmoothhessss clipping [Zhang et al., 2020, Koloskova et al., 2023] and gradient descent with an appropriate adaptive stepsize studied in the two concurrent works Gorbunov et al. [2024] and Vankov et al. [2024].
shown for both gradient
Parameter-agnostic the deterministic case, gradient descent with backtracking line-search was shown to converge under (La. L1) ss)ssmotsssss without knowledge of the Lipschitz constants [Hübler et al., 2024]. For (star)-conve problems, an interesting connection was established between gradient norm clipping and the Polyak stepsize in Takezawa et al. [2024] and further analyzed in Gorbunov ct al. [2024] and Vankov et al. [2024]. The adaptive stepsize removes the need for knowing both Lo and L, Unfortunately, the Polyak stepsize is deeply tied to the Euclidean and (star)-conve structure and thus does not seem to be directly extendable to our more general setting.
In the stochastic case, the current best known parameter-agnostie method introduces an undesirable exponential dependency on L1 in the complexity [Hübler et al., 2024]. However, knowledge of Lo can be removed without such issues through either an AdaGrad type stepsize [Wang et al., 2023, Faw et al., 2023] or normalized gradient descent with momentum [Cutkosky and Mehta, 2020] as shown in Hübler et al. [2024]. This mirrors results from the online learning community where both AdaGrad and gradient normalization are known to adapt to Hölder smoothness [Orabona, 2023].
Short step In contrast with gradient descent, the Frank- algorithm [Frank et al., 1956] does not ensure descent with an open-loop stepsize even in the deterministic setting. Descent can be ensured by an adaptive stepsize known as the short step, originally introduced by Frank & Wolfe [Frank et al., 1956] and extended by Rubinov & Dem't yanov [Dem't yanov and Rubinov, 1968]. See Pokutta [2024] for expository treatment.
OOll local Lipschitz is needed in the sense that the condition only needs to hold for (x.y) satisfying [Xx yll s 1/4 It is also possible to replace L with L, := maxes Lo + L,Il(( I.. e.g., as is done in [Koloskova al., 2023, Thm. 2.3].
8

---

Spectral norm methods Clipped Spectral can be viewed as hybrid method between the stochastic spectral descent [Carlson al., 2015b] and the Muon optimizer [Jordan et al., 2024b], with some crucial differences.
Muon builds the gradient estimator de differently. Specifically they take d' = =V((. Bd- if
Nesterov momentum is disabled. This is equivalent to our choice d' = aV f(x.fffffx)oo . (1 a)d for LMO-based schemes, since the LMO scale-invariant (i.e., Imo(a s) = Imo(s) for
[Pethick
et al., 2025]. However, for SD this equivalence no longer holds (in fact we have [a s] a[s] for a E R). The appropriate choice of d", which generalizes to SD and GGNC, turns out to be the convex combination.
Stochastic spectral descent [Carlson et al., 2015b] does not construct a gradient estimator and instead takes d² = [V Ex). This restricts their convergence result to the case of (mild) relative noise.
In this sense, Clipped Spectral could just as well be called Clipped Muon (not to be confused with the unrelated MuonClip [Team et al., 2025]) but we prefer Clipped Spectral as the algorithm itself is not tied to momentum nor to Newton-Schulz, as the name Muon fundamentally is.
Tuddenham al. [2022] also studied an optimization algorithm focused on orthogonalization, however they orthogonalize before doing the momentum step. Pethick et al. [2025] analyzed more general algorithm called Averaged LMO Directional Descent which admits as a special case the algorithm studied in Tuddenham et al. [2022]; their empirical and theoretical findings found this algorithm to be worse than orthogonalization after the momentum step, e.g., the Scion family of algorithms [Pethick et al., 2025].
We note that many works have recently analyzed the convergence behavior of algorithms using spectral LMOs like Muon and Scion, starting first with Pethick et al. [2025], Li and Hong [2025] and then Kovalev [2025], Sfyraki and Wang [2025], but always under L-smoothness assumptions, in contrast to this work.
Modular norm [Large et al., 2024] introduced a norm choice for neural networks and established smoothness condition for given neural network provided the parameter remains bounded. The dual norm computation needed in GGNC is particularly easy to implement in the accompanying Modula software package since [[d]| := flatten(Imo(d)) flatten(d), which in Modula code reads as dual_norm=-summ (mooee.. duwliialaaii ze (d) *d).
Weight decay Weight decay [Pratt, 1992] is a crucial component in deep learning and has become standard in training modern neural networks through its integration with Adam [Loshchilov and Hutter, 2017]. When combined with LMO based updates such as sign descent and the normalized gradient descent the resulting methods can be seen instantiations of the conditional gradient method for constrained optimization problems [Chen et al., 2023, D' Angelo et al., 2023, Xie and Li, 2024, Pethick et al., 2025]. Our adaptive stepsize in Algorithm 2 effectively scales the weight decay as well as the update. This is similar to scheduled weight decay [Xie et al., 2023] which uses the adaptive stepsize in Adam to also scale the weight decay parameter.
Experiments
For the norm choice of Scion and ClippedScion use the (Sign - Spectral - Sign) and (Spectral - +Spectral + Sign) configurations for language modeling and image classication respectively (see Pethick et al. [2025, Tbl. 2-4] for the associated scaling factors). To compute the spectral Imo use the efficient implementation provided in Jordan et al. [2024b] of the Newton-Schultz iteration proposed in Bernstein and Newhouse [2024]. There have been recent efforts move beyond the "N" (Newton-Schulz) in Muon, the most popular algorithm computing the spectral LMO, through alternative subroutines; our algorithm compatible with these alternatives, like the optimized PolarExpress routine [Amsel et al., 2025] or power iterations [Ahn et al., 2025, Vogels et al., 2019], although we do not explore them here.
Image classification We test on a convolutional neural network (CNN) on the CIFAR10 dataset. Hyperparameters can found in Table 2 in Appendix C. We consider both a fixed stepsize setting and stepsize scheduling using linear rampdown to investigate if the theoretical results are predictive of practice. We report the experimental results in Figure where mean and standard deviation are computed over 5 independent runs.

---

unesnominnd Cipoadidan
1000
Claain trrshels p
2500
2300
0.3
Urovstiieee Unorssaated
500
60
250
500
1000
3230
1500
3750 2000
Figure I: For CIFARIO experiments with fixed stepsize clipping leads to a substantial improvement.
Unconctrs ned Clopertiiin plipging thrsehold
7030
6003
5000
Adam
Uncoricio Soan
Uncersvia Chanedtonn
Haration
Figure 2: For fixed stepsize comparison clipping improves over Scion by than 10% speedup NanoGPT (1B). We observe similar gains on the smaller 124M parameter model size (ef. Appendix C).
We find that clipping can substantially improve the test accuracy in the fixed stepsize setting, when the gradient norm (i.e. ||d"1l. = (d" is decreasing. This separation is agreement with the theoretical separation between Theorem 4.3 and Theorem 4.5 fixed stepsizes. In the constrained case (Algorithm 2) we surprisingly find that (d r²) is increasing (cf. Figure in Appendix C) which requires further investigation. With stepsize scheduling ClippedScion) and normalization (i.e., Unconstrained Scion) achieve similar performance, which aligns with the matching theoretical rates of GGNC (Theorem 4.9) and uSCG (Pethick al. [2025, Thm. 5.4]) in the stochastic case when stepsizes are taken decreasing.
observe that clipping (i.e., Unconstrained
We also evaluate the unconstrained case (Algorithm 1) using Vision Transformers (VIT) the ImageNet dataset. We train a DeiT-base model using the DeiT codebase [Touvron et al., 2021] with replacing LayerNorm by RMS norm following [Pethick et al., 2025]. Table 3 in Appendix C contains the hyperparameter details. As shown in Figure (Appendix C). Unconstrained ClippedScion achieves 11% speedup over Unconstrained Scion, even though its gradient norm (IId"I)) is increasing. This obscrvation requires further exploration.
NanoGPT We additionally test NanoGPT Karpathy [2023] in Figure 2 with modernizations following [Jordan et al., 2024a]: rotary embeddings are used instead of positional embeddings, RMS norm is used instead of LayerNorm, and the ReI.²² [So al., 2021] instead of GELU activation function. All methods are trained for 5100 iterations with a batchsize of 512 and context length of 1024 on the FineWeb dataset (see Table 4 Appendix C for further details). The empirical observations matches those for CIFAR1O experiments.
7 Conclusion
We have shown that clipping can be extended to non-Fuclidean settings and even constrained problems by establishing a precise connection the Wolfe short step. descent property was established under a generalized notion of (Lo, L) smoothnesss which opens up a range of interesting directions:
The descent property both in the unconstrained and constrained case enables integration with adaptive stepsize choices such as AdaGrad and backtracking line-search.
The non- Euclideen notion of (Lo, L -smoothhess we introduce might be suitable condition to study for neural networks. Large et al. [2024] showed that neural networks are smooth in the modular norm provided that the parameters are constrained. However, in practice, violating the constraints seem to be unproblematic for optimization, which suggests that a looser smoothness assumption might hold such Assumption 4.2.
10

---

Acknowledgment
This work was supported as part of the Swiss Al Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 Alps. This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_205011 I. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). Research was sponsored by the Army Research Office and was accomplished under Grant Number W91 24-1-0048.
References
Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint N0Xfv:2504.05295, 2025.
Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm. arXiv preprint arXiv:2505.16932, 2025.
Lukas Balles, Fabian Pedregosa, and Nicolas Le Roux. The geometry of sign gradient descent. arXiv preprint arXiv:2002. 08,55, 2020.
Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv preprint 2xXi:2449.200325 2024.
David Carlson, Volkan Cevher, and Lawrence Carin. Stochastic spectral descent for restricted boltzmann machines. Artificial Intelligence and Statistics, pages 111-119. PMLR, 2015a.
David Carlson, Ya-Ping Hsich, Edo Collins, Lawrence Carin, and Volkan Cevher. Stochastic spectral descent for discrete graphical models. IEEE Journal of Selected Topics in Signal Processing, 10 (2):296-311, 2015b.
Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As Iyapunov predicts. arXiv preprint arXiv:2310.0523 2023.
Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang. Robustness to unbounded smoothness of generalized signsgd. Advances in neural information processing systems, 35:9955-9968, 2022.
Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International comfertt on machine learning. pages 2260-2268. PMLR, 2020.
Francesco D)' Angelo, Maksym Andriushchenko, Aditya Varre, and Nicolas Flammarion. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:2310004415. 2023.
VF Dem't yanov and AM Rubinov. Minimization of functionals in normed spaces. SIAM Journal on Control, 6(1):73-88, 1968.
Matthew Faw, Litu Rout, Constantine Caramanis, and Sanjay Shakkottai. Beyond uniform smooth- ness: A stopped analysis of adaptive sgd. In The Thirty Sixth Annual Conference Learning Theory, pages 89-160. PMLR, 2023.
Thomas Flynn. The duality structure gradient descent algorithm: analysis and applications to neural networks. arXiv preprint arXiv: 108.00708..00233 2017.
Marguerite Frank, Philip Wolfe, et al. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-100, 1956.
Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury, Alen Aliev, Peter Richtárik, Samuel Horváth, and Martin Takáč. Methods for convex (1_0, 11 smoohh optimization: Clipping, acceleration, and adaptivity. arXiv preprint 24X.:22409.14988 2024.

---

Florian Hübler, Junchi Yang, Xiang Li, and Niao He. Parameter agnostic optimization under relaxed smoothness. International Conference on Artificial Intelligence and Statistics, pages 4861-4869. PMLR, 2024.
Keller Jordan. Cifar-10 airbench, 2024. Affarl0-airbence
URL http::///itttuo.cc////lllllorooo
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fermbear.sskysscciall Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @ @Grad62309977. modded-nanogpt Speedrunning the nanogpt baseline, 2024a. URL https://st// gith......... com
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https: //kellerjordan. gi gitb. hotbii////sss/moo//.
Andrej Karpathy. nanoGPT. https://github..c.m hatphubbcccooaaappattyaaaanrrr 2023. Accessed: 2025- 01-25.
Jonathan A Kelner, Yin Tat Lee, Lorenzo Orecchia, and Aaron Sidford. An almost-linear-te algorithm for approximate max flow undirected graphs, and its multicommodity generalizations. In Proceedings of the twenty-fitth annual ACM-SIAM symposium on Discrete algorithms, pages 217-226. SIAM, 2014.
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U Stich. Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In International Conference on Machine Learning, pages 17343-17363. PMLR, 2023.
Nikita Kormilov, Philip Zmushko, Andrei Semenov. Mark Ikonnikov, Alexander Gasnikov, and Alexander Beznosikov. Sign operator for coping with heavy-tailed noise in non-convex optimiza- tion: High probability bounds under 2025.
arXiv preprint arXiv:2502.07923,
Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. arXiv preprint wrXiv:2503.22655 2025.
Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. arXiv preprint arXiv:2405.14813. 2024.
Jiaxiang Li and Mingyi Hong. A note on the convergence of muon and further. arXiv e-prints, pages arXiv-2502, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101,,20 2017.
Tomáš Mikolov al. Statistical language models based on neural networks. 2012.
Aryan Mokhtari, Hamed Hassani, and Amin Karbasi. Stochastic conditional gradient methods: From сonvex minimization to submodular maximization. Journal of machine learning research, 21(105): 1-49, 2020.
Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-32,, 2012.
Francesco Orabona. Normalized gradients for all. arXiv preprint UrXiv:2308.05621, 2023.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained Imos. arXiv preprint arXiv:2502.07522, 2025.
Sebastian Pokutta. The Tank-wolfe algorithm: short introduction. Jahresbericht der Deutschen Keathematikerr-eriiiiiurgr 126(1):3-35. 2024.
Lorien Pratt. Non-literal transfer among neural network learners. Colorado School of Mines: MCS-92-04, 1992.
12

---

Maria-Eleni Styraki and Jun-Kun Kun Wang. Lions and muons: Optimization via stochastic frank-wolfe. arXiv preprint arXiv:2506.04193 2025.
David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoe V Le. Searching for efficient transformers for language modeling. Advances in neural information processing systems, 34:6010-6022, 2021.
Yuki Takezawa, Han Bao, Ryoma Sato, Kenta Niwa, and Makoto Yamada. Polyak meets parameter- free clipped gradient descent. arXiv preprint arXiv:2405.15010, 2024.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20554, 2025.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
Mark Tuddenham, Adam Prügel-Bennee and Jonathan Hare. Orthogonalsing gradients to speed up neural network optimisation. arXiv preprint 200002220. 2022.
Daniil Vankov, Anton Rodomanov, Angelia Nedich, Lalitha Sankar, and Sebastian U Stich. Op- timizing (-soo_)) functions by gradient methods. arXiv preprint arXiv:2410. 10800, 2024.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Bohan Wang, Huishuai Zhang, Zhiming Ma, and Wei Chen. Convergence of adagrad for non-convex objectives: Simple proofs and relaxed assumptions, In The Thirty Sixth Annual Conference Learning Theory, pages 161-190. PMLR, 2023.
Shuo Xie and Zhiyuan Li. Implicit bias of AdamW: Co norm constrained optimization, arXiv preprint arXiv:2404. 044544 2024.
Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, and Masashi Sugiyama. On the overlooked pitfalls of weight decay and how to mitigate them: A gradient-norm perspective. Advances in Neural Information Processing Systems, 36: 16:1-201222, 2023.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 7008. 03888, 2017.
Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. Advances in Neural Information Processing Systems, 1-55211,, 2020.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint 1905 1881, 2019.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes we generalize clipping and (Lo, Lms-moothhess
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See experimental section and conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
All theorem statements explicit states assumptions and proof (in the appendix).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: For hyperparameters see tables in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide an implementation of Algorithms and 4.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c.
aublic//uides/Coeeeeee wwt.:/uuumissiiiinnii ssionPolccy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nis. ctcepuulicggguieeeec des/CodeSSummis wwtpsuiiiissisooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See experimental sections and associated appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: CIFAR10 experiments are particularly noisy and multiple random seeds used.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ https//:///erisseeeem httpuiipp..cpplllittttticcciiiio ines?
Answer: [Yes]
Justification:
We have read the guidelines and do conform.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work is foundational research.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww /dttastssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Human subjects are not used.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http:://eerrrps.cc//ofeeeeeeeeee for what should or should not be described.
20

---

Appendix
Table of Contents
A Preliminaries
22
Proofs for Section (Analysis)
22
C Experiments
36
21

---

Algorithm 3 Unconstrained ClippedScion
Input: Horizon n, init. x² =W] D), dl = 0, momentum ag E (0, I], stepsize y E (0, 1), radii
R,, and O.
1: for k= ..... do
2: Sample & 9
3: de ag f Ex) (1 ai xt
4: vi = -n Hhw (dp) V1 VIsIsD
5:
N - min/p. DP (d 1/
6:
xtl - x² -na -maa**
7: Choose X² uniformly at random from .x'
Return I"
Algorithm ClippedScion
Input: Horizon n, init. x! (W).... p..... wh) 11 xrDDp. d stepsize y (0. 1),
momentum E (0, 1]
1: for k 1. do
2: Sample & p
3: dh - a (x Ex) (1 ar)elll1
4:
v - of Imo (4t) V1 SISD
Eie (o maxffr Ilg
5:
Variant 1: 1h minlp,
6: Variant 2: 1h min(p,
7:
x+11 - x ymxx**
8: Choose X uniformly at random from (x!....,.*)
Return X
A Preliminaries
Throughout, L-smoothness is defined follows. Definition A.1. A gradient mapping Vf X - Rd is said to be L-smooth with L E (0, oo) for all E holds that,
V fx) - ff(i)l, = L||x - yll.
(5)
The sharp operator has the following properties
(s. s+) || ||s||
(6)
See Kelner et al. [2014, App. A.1] for the proof.
B Proofs for Section 4 (Analysis)
Proposition 4.1. Suppose f is L-smooth with respect to || ll. and denote f = inf f(x). Then, the iterates heer of uCG satisfy, for all n N
min f Il. $ Σ IV & f6''-1 Lskis
apa + Lyp
Proof. By the descent lemma for L-smooth functions applied at the points x² and x++ and the definition of l+l we have, for
f(x+1) f(x)) ( h.
Summing from to n, and dividing by n gives
E (V)( -
f('' Lyp 7pn
2
Remarking that the minimum summand is smaller than the average completes the proof.
22

---

B.1 Deterministic case
Recall the notation == f(x')-
Theorem 4.3, Suppose Assumption 4.2 holds and let E N'. Consider hiskn generated by GGNC with d: vf(x), and y s 1/(o=pla). Then, the following holds
min ISKSR
IV
+
yp
Specifically, with La and To we have In
min |V 2 Lar 21
lskke
Proof. For each k n, we can write the formula for xf*1 as follows
*+1 = =x=x -rrcl((s(<>))
with 73 min(1, 1V
From (Lo, smootthesssss and properties of the sharp-operatee (s. 5t) ||s I|sl1.. we have
s(x $ - + Lot
= - MI - II
A useful observation is that by definition of we have
Tx Ill sp.
since if [[V then f/lll and if f[V z then 1. Thus we can upper-bound the term :z rx[[V[(x [l, by ماو in the quadratic part, yielding
f(x s f(x) - yTxllV f( (TLL PLIVV
$ f(x) -ITx(1 ILo +LLDIIVS(>)
s f(+( 1V
where the middle inequality uses that 7² since and the last inequality uses the stepsize
choice S Tomphi
There are now two cases to consider.
Case Clipping Active (IV ffff((llla p).
Here, we have T Ma
IV = s(+))-
Therefore, the descent inequality in this case reads
f(x 11
Case II No Clipping (I[V f(x))l p). In this regime, Ts so the inequality becomes
f(x²t)) s - f(b
This is the familiar descent guaranty for the classical steepest descent method with smooth functions. By combining the two cases and summing over all k until we obtain
i (p Exea [VV Il. + Σn ||V [|V f(x))
where A is the of indices where clipping is active (Case I). Since each is nonnegative, conclude that
Eien VV f(
20(()))-) ym
and Etem [Vf(x))) 20())1)
7pm
23

---

Recall the following inequality for real numbers: for all a. a.b. > 0, a² 2ab - b². Applying this with a = IV Il. and 0 gives
1 Exe IV f( 2 Exen(2bIV - -b²) [I 6'01-191)
Substituting this estimate into (7) and using the fact that [ n gives
(EmanlV f(++())).))
Fix-1990
200
Eee fV s
/'')-1" bya
+ b)
y
Then, choosing b
10''-/" ym
simplifies the above
EAeA [V
Now, we can combine both to bound the over S k n
E [V < Re'' 2
yn
Taking the minimum of the summand over 1 < k = on the left hand side gives the final result.
Theorem 4.5. Suppose Assumption 4.2 holds and let E N Consider (sshs generated by uCG
with yp 1/21.. Then, the following holds
min Isks
|V
2A yo + 2Loyp.
Proof. We begin by invoking the descent lemma for (Lo, --mothhth functions the points x4+1 and x", which is justified since ||x x²11 = yp I Then, applying the definition of k+1 we get, for
all 1 s sks
)(xt()) s(()) +
YPX +(a)))+++++((((( -f(x))-xplive Il. + + LIVVF(()IL)) =((0) 2o8 (L17p 1)xeIl
1t) + Y LIV
Rearranging the above yields
5 | (1 LYP)) 10b)-007) Loyp
where we have used the assumption that yp s M in the first inequality above. The desired claim immediately follows.
Theorem 4.7. Suppose ff L-smooth and let n N Consider (x²) liste generated by Algorithm with d vf(x), y z. and L so that yp S 1. Then, for all BBD, the following holds
min (V u) 2B Iksa
+ 24
yp"
Proof. We start by applying the descent lemma for L-smooth functions at the points g²+l and x to get, for all s k s n.
Now, we divide the analysis into two cases depending whether or not clipping is active.
24

---

Case Clipping active ( TPp
2 pi p).
In this case, we can use the fact that (Vʃ(x*), 14) 2 pllv~11? to get
1 A
f(x s f(x) ym
= =(()) xXVV(()) ,/ + P'
s - - (PAAS(A)) + p(V
s 5 2 > (t)
where the final inequality is due to the assumption that s 1.- Rearranging this gives
YP
//2 (V f(x)), V. s f(x)) f(x²t1).
Case II
No clipping
fo 16
S p: 1/:
When clipping is not active, 17x acts like short step which gives
L 2 n2llv=1
f(x S f(xt) (V vty +
s((()) - (VS())),)) L
(V (VI())) 1822
2
1|v+1p
||v²11
$ s(( - (Vf(x))
1|v11p
where the last inequality follows from the assumption that 1 Rearranging this gives
y (VS(), ,/ -f(t))
402
Combining both cases to n we find
Denoting A the set of indices where clipping is active and summing from
Σ
yp 2
Σ Y
(V((s)
A2 s f(x))-
+
4B2
KEST
REA
which, since each summand is nonnegative, implies that
vv 2(f(x')-f) and
My 4f(((x)))
(8)
ypn
yn
KEM
REA
Recall the following inequality for real numbers: for all a. 0, 2 2ab b². Applying this with a = (Vf((*), (4) and b > 0 0gives
1 Σ vv
(2 -
2b
Σ
b²(n - ())
KEM
Substituting this estimate into (8) and using the fact that | S n gives
2b
Σ (vrat))))))
b²in (A)
n
n
yn
which implies that
1 12
1 4B2 (fx()(x))) )r) + b)
<
71
2
byn
25

---

Thus, choosing
simplifies the above to
4B2 4B-(((x)) Evschh
f(x)) - 1
2B
yn
KEA
Now, we can combine both cases to bound the sum over S S n1 as
Evrehh s 2B
f(x))-f* 2(f(x') -
yn
ypn
Finally, by lower bounding the left hand side by the minimal summand over 1 s k s and using the definition of y arrive, for all E BBD, at
-f(x))-*
((x))-(
min (Vf(a))) u) 5 Iskee
Vyn
ypn
B.2
Stochastic case
B.2.1
Convergence Analysis of uSCG
We now generalize the еπтоr control lemma Mokhtari et al. [2020, Lem. 6] to the (Lo, smooth case and modify it for the clipped algorithm Algorithm
Lemma B.1 (Linear recursive inequality for E 1| for GGNC). Suppose Assumptions 4.2 and 4.8 hold and let n N Consider the iterates (x hisss generated by Algorithm 1. Then, for all k nl,...,n, n), it holds
BI
az
4/'.'1 ac
4y f( II.
where &. = max [|].
Proof. The proof is straightforward adaptation of the arguments laid out in Mokhtari et al. [2020, Lem. 6), which in fact do not depend on convexity nor on the choice of stepsize. Let E N and k E 12..... n), then
| ||( 13
||v ar -1 (a-
=||0 | Vs (2) + (1 an) - (1- v
= V + an) [ -
+ +(1 ( 1/
+ 20 < - EE-)) V)(+) (
21 (x' V/(At &.), vl(+ -t-1
2(1 (v=)))) - " V
1
Taking the expectation conditioned on the filtration F2 generated by the iterates until k, i... the sigma algebra generated by (x' ........f. x4t, which we denote using [,[.], and using the unbiased property in Assumption 4.8, we get,
Bal Ba B. v ( (1- ( v
+ +1 az) 1 2(1 -
26

---

For brevity define 4. := L Il.. From the above expression we can estimate,
B+1|++1111 (n)
-1 (4 + (1- and 11-
+ 21 -(00-
0 + (1 - (x) 1/(11) - v 111 + (1- an) 11
+(1 (t.
/ 112 2 [|11-11]]] (13
19
02 + (1 ( 2.
7/CC y/p +(1 a² ||
+(1 (11-0) +(1-a)) (
- //(aa))):
|| [1]
a. + (1 - -1
10)
$ aj + +1 - .3 (1- + (1 - ax) ( + a 11-111
3 + (1 (1 a; -
|
_ 2200 + 2(1 + )1 - O6
+ 3 X1 on)) a3) ||
using the bounded variance property from Assumption 4.8 for (a), Young's inequality with parameter /x/2 > O for (b), 5. := max,ex Tr.
1ibe for (c), Assumption 4.2 for (d), the definition of from
Algorithm for (e), the fact that ag < for (f), and m. S p and Young's inequality on L for (g). To complete the proof, we note that, for all k E (I...., n), it holds
(1 2 X1 ax) 2 and (1-a,)(1 + 3
which, applied the previous inequality and taking total expectations, yields
4y (1) ((()).
Ed| s (1
ax 2 =| az a
Lemma B.2 (Bound on B1 with horizon-dependent a for GGNC). Suppose Assumptions 4.2 and 4.8 hold, f is L-smooth, and let n E N Consider the iterates (x* lisksn generated by Algorithm I with stepsize yp satisfying
YP < I
(9)
1 Moreover, consider a momentum ar = a = Va for all kE (I,....n). Then, for all (I..... n) the following holds
B1| s 2007+)000)0))
(10)
Proof. Let E (I.....n). We start from the recursive inequality obtained in Lemma B.1 for B
Since we are now assuming that f is L-smooth, this inequality is satisfied with Lo = L and L, 0. which gives
B| B|| = (1- =|I + a²02 + 4PP0
(11)
Now, we substitute the specific choice a Vor of momentum to find
E|| E|||~1 < (1 -
12.12 Vn.
(12)
Using the particular choice of yp. we have
B1||>11311 (1 - 2
1| 1 +
27

---

Let ue = 2
and b 2 CL.
Unrolling the recurrence relation u2 (1-a)us-1+*) +b,
2vn
we have
Up S (1 u) + - a)' b E/=o a)' = =b1-11-0 -اا1-=
Since <a we have a)x 1 for k Thus, 1-11 aa) <1. Therefore,
" %/.
(13)
Substituting the values for and b, we have
(14)
This concludes the proof.
Theorem 4.9. Suppose Assumptions 4.2 and 4.8 hold and let n N. Consider the iterates {x hssts generated by Algorithm with constant stepsize y I 1/4, and yP s 1/24, Then,
B[IV
4v4
Ba + VE, 8c.
where A f(x)x and 6, 1 O
1| + 111
Furthermore, assuming f is L-smooth' and taking a 1/ y VoLo and To/2a such that YP = 1/2, we have that
B[|V)((")II - 0 44
Proof. Given that ||x* - x1 s Y4, which we will ensure by choice of the stepsize yn and radius p. we have from Assumption 4.2 that
0 f( ((0)-f(((+)))) (Vf(x)),))))))o A+1 xt + / 1| 1 x²11
yn. f(²),, Imo(d")) )) + 2 Imo(d) I1I + // )I? yn<Vf(x)) (yn Imo(d")) ))=d))))+++ Lg 2 2 y f(2V
f(xx)- )
where we recall that 1/,- = min(p, [Id)
To treat the inner product we introduce the error X² := d f((+t) and proceed as follows (V = (V d Imo(d')) (d Im)(d"))
(V & Imoodd)) 1|d%11
= (Vf(x²) at, Imo(d")) )) jllc211 311c(11
(Triangle ineq.) s (vf(2²) - d,, Imo(d")) - =11K-111 -II + 111 (Cauchy-Schwarz) < 11 111./111 f(V 1111-111 211222122 Illad1ll }
where & := maxxex [Mbb
||L is the norm equivalence constant.
Combining the two inequalities we have
0s +
ym - 1IV 1/ 2
= -(x +
y LOYTs) 7m /
Case Clipping Active (p [|(+)
In this case we have that 17x- p. so
0 -(x()))((x+)))) y 2 (1 - 4ox) - f(xx)) - f(x + 211
- -1 Lixe))If(<>1)
where we have used y:
*Only local Lipschitz needed in the sense that the condition only needs to hold for (x.y) satisfying [Xx 1 yll x 1/4, It is also possible replace L with L, := max Lo + L,I.(() I.. e.g., as is done in [Koloskova al., 2023, Thm. 2.3].
28

---

Case II No Clipping (p 2 ||a*
Here have that 7)x |ld" so s f(x)) - f(x!=)) + r - - LOYe) - x2 (1 Limp)lld (I.1VV
Focusing the last term, have
| Il. = 1|d
(Triangle ineq.) 2 (
= - =
For the last term, using the triangle inequality, have
| MI- < 112"11-01//(++ d'1l 1ld%)..)
= -1221122 |
s 11 11
s 5112111 + 511,"112p.
By combining, we have
0 s(x) - >2 + 71 - 4 11-2"1120)
(1 Lors) - 7x - L>
- Lore) 7x - (1.x00ll)))VV((>>>
7)p((2 /
$ f(+)) - - - ((=>)
(2 + - -I
where the last inequality uses y: s +.
Combining both cases Introducing the set of iterates where clipping is active, A: == (k [m] [ we can take the expectation both sides and sum the two cases to find
1 LippMp E kE B[IV E f(>)
s f(x')- f + Exen 3 YeE YP
-
s s(((') + Eia YP5 _
|1| Eien (P ((p)
|1|
S sf(x)) f + - (P
( B11 + + L11p)E(112111
where the second to last inequality is due to Jensen's inequality and the last inequality uses that y S 1/L.. Using the stronger requirement that y 1/2,r, it follows that
tr ΣtE BUIVS(* ]1-] Σne BUIV ((* (1E)) < A + ye.
with == f(x) - and c == n Σi PC || + {(B1))1))) By nonnegativity of the summands, it follows that
EAER BlI |1
41 46. ypm
(15)
corresponding to the indices from Case I. Using Jensen's 's incquality, we similarly have
Enea BUV s ; Σn B[]
+ 4e, =: = 41 pm
(16)
corresponding to the indices from Case II. We will now use that 2az a² < 2 for any a. >>>>0 0. Pick z = B[JV [(.) then we have that
Exen = S $ Eien E
+
S
1
Exe
- +
29

---

Choosing = VA and using the triangle inequality we have
ExEn BlIV((+))))) VA x 245 + 2 VEn-
(17)
Summing the two cases, (15) and (17), we have
E BUV ((+
4vA VYn
8A Ypm
Ve, +
Sea
What remains to bound the error 6, that is due to the stochastic estimator. With the choice y s vila. yp 1/2a and or 1/ vn, invoke Lemma B.2 from which have
200 (224)) B.
B -
v/
It follows that
s p VB 1B
6,
and in turn the inequality B.2.1 simplifies
i # Σ BUl
1I.1
0
+ vp VB + {B + ol VLvTE
Vyn
00
VpCB!.t + V
CB p
ypa
where we have used the triangle inequality in the second inequality, Letting b 2(0 G.1. (1:) have with the choice volo and Lo/2al/1, that
$ Σ BU J M.J1 s of ( VALo AL,
VIL
Veb + < vb (bL1
NL Vilao 12
$ o (VLL + ALI
+ v5) (02 + G(IL)) AoAAAA
Velal vi++ Vo.42)
S o H VALO + AL1
Via
+ + Volo + 6.41.))
Lo
completts Noting that [[+))]] Σi B [[IV(((+)]II] completes the proof.
B.2.2 Convergence analysis of CC
Following the same outline as the convergence analysis for Algorithm given in the previous subsection, start with an control lemma in the vein of [Mokhtari al., 2020, Lem. that is compatible with our adaptive stepsize.
Lemma B.3 (Linear recursive inequality for B||ai[A for SCCG). Suppose Assumptions 4.2 and 4.8 hold and let e Nº. Consider the iterates (x* lisks generated by Algorithm with stepsize ym, p. Then, for all k E (I,...,n),
82LBBp2p2
E s (1
2 B | a²0²
a
where 5. = max
H+l [А]
Proof. The proof straightforward adaptation of the arguments laid out in Mokhtari al. [2020, Lem. 6], which in fact do not depend on convexity of the function ff nor the choice of stepsize
30

---

Mh. as long as it is in [0, 11. Let E N and k E (1, m). then
| ([v/())- - 12
||/(()) aV -
-I(((V/6)) VS( + (V/(2)) - -0) v )|(
-
= =''I\v((>) - Vf ( + + (1- a) [/(}) -
+(1 a) [(/((//-)
2a (xV)((+-))-)))o)))))) vf(*t -6-1), Vf(x)) (
2a(1 (x² - - E.), ) - d
2(1 - -010))(V/(()>) - V/(a-1),, ) d'11).
-
Taking the expectation conditioned on the filtration Fg generated by the iterates until k, i.e., the sigma algebra generated by (x' (x'....,.x) 1²), which we denote using Eg[-], and using the unbiased property in Assumption 4.8, we get,
11 + (1 - -
From the above expression we can estimate,
E. 1
- 1|
+21-087776)- -
V V i1ا + - a)² 1
+(--a) 4
112 2 -1
3/( + - a) ||
+(1 -0-00(3211716) 4 I1p - Il
8 200 -6 01 1
+(1 -11-,|((
S 2000 ", (2c + 3 -11)--111))
5 128-:44+3111088183397- -1) 11
$ 0000 + 4(1 70 +1++0000x22 +1 (1+++))(100) 111
using the bounded variance property from Assumption 4.8 for (a), Young's inequality with parameter [(s]: for (c). Assumption 4.2 for (d), the definition of x² from a/2 > O for (b), C. := max [[Í..
Algorithm 2 for (e), the fact that 1 - a < 1 for (f), and 1/ S and for (g). To complete the proof, we note that
(1 x1 a) s (1 ) and
s 2 (1 - (1 + +
which, applied to the previous inequality and taking total expectations, yields
822
B|||*1111 = (1- - + a²0²
31

---

Lemma B.4 (Bound on the gradient enror with horizon-dependent for S'CG), Suppose Assump- tion 4.8 holds, f is L-smooth with respect || ll.,, and let E N'. Consider the iterates |x hston generated by Algorithm 2 with stepsize yp satisfying
YP < Lo
(18)
Moreover, consider a constant momentum 0g = a E [I)....n) the following holds
1
for all k [l.....n.. Then, for all vi
B 11
20'+166888
(19)
Vir
Proof. Let E N' and k E (l.....m) n). We start from the recursive inequality obtained in Lemma B.3 for Bi| with the Lipschitz constant of the gradient over the compact D to get
=|| s (1 x 0² y
(20)
a
Now, we substitute the specific choice a +
1
σ² 8L.L'3''''''' Vn.
B1|
(21)
2 /n
n
Using the particular choice yp specified in the statement of the lemma, we have
1
B1|
! 1
2 + S2.
2 2
Let 11 a)ua=11 b, we have
I 2 and b |0² + 8L.B²). Unrolling the recurrence relation 44 I
S (1 - at'uo + - - a) = b - a) = = 1-1-aa bb = b 1-(1-at
Since 0 < 1, we have 0 < (1 -1- a) 1 for k 2 1. Thus, 1 - (1 a)* < 1. Therefore,
- b/a.
(22)
Substituting the values for a and b, we have
B B||
(23)
VR
This concludes the proof.
Theorem 4.11. Suppose Assumptions 4.2 and 4.8 hold and let E N.. Consider the iterates (ss)s
generated by Algorithm (Variant 1) with constant stepsize y S 1/Lvi and p S 1/ Then, for all E BD,
B[C R" 11}
4vA Vyn
SA yOR
+ 4 VEn
8
where A = f(x)) and e, := / Σx= O 11
B3)
Furthermore, taking a = vm, 1/L vi and = 2/ such that = 1/(22' we have that
BI(VF(E"), 4)) s 0 (at)
Proof. Note that, since ff continuously differentiable and D is compact, f must be Lipschitz-smooth on the scaled ball BD with respect to the norm || |l: call the Lipschitz constant L > We can therefore start with the descent lemma for f at the points r+1 and x' to find
0 f(+) f(x**(()=+++++== (V x - -x) + L 11a4+1-xAP1
2
-M / L
s
1.
$ f(x) - yn ((d* (Vf())) d,, d',
Now we can proceed case-by-case depending on whether clipping active or not.
32

---

Case Clipping Active (yT = yp: 6or.4)
11/T= p).
For all E B2 it holds,
= ((x5) yolx (1at 14) + (VS(+²) dt, ,4) - > n
1 (at, v1 5 (V/(x)) )x,, 4) 2 2( VS(A), 11) (VS(x)) - d,, -d..1)) -.2 1
1
$ f(x²) -(tttl) 7mh
(
2 ( v) + 2 (V(()) x - - 11181ballat - -ll2 -1221aa11*112
L n
s f(x)) - -((tt)) 7m
" f(()) f(x + +
L
2 2 ,²) yTh 1
3
(V x -2) ynk 2 11lt
2
3 ((²) - -(( + yp
1.
1 2 - 2xp ( ----
-1) yp 3 >
2
( f(x) - - + yp 2
3
1 ( u)
11.2112D2
where D2 = max ||x - yll is the diameter of the set BD in the Euclidean norm. The inequality (a)
x.
follows by Scharr,, (b) follows by using the diameter of B2D (c) follows since clipping is active, and (d) follows since L.y S 1. Finally, rearranging gives
yPVV x -4) s ( 3/Dyyplld'1b.
(24)
(.r Sot Case II No Clipping (y7 = = "P-F
S p).
DF
In this case, our stepsize acts like the short step. Starting with the previous inequality from the descent lemma we have, for all E BDD
0s 0(5( -(xtt)) y (et v4)
L
(d
||ui112 (att ,4 (Vx))((()))) d,, d.
11x0112
2
1|1112
s - -((tt)) y (dt 1422
(at 1892
pN(V &..') + LY 21115112
(25)
11,$112
sfa+) - f(xtt)) y (dt v432
2111*112 VVx+V _d.
where in the last inequality we have used that y S 1. Rearranging, estimate
(ctt, .422
0 5(+)) s(++t)
(V(V)(() d,
211
$ s(()) - (((
(V) - u0² - -2x(
1²²) (n( - 1A,
= -f(+)- - -
41111112 (Vf()),, -u)²+ Y (Vf(+)) -d 1/2 - d
Y
||~-112"
y 1) s(+) - s((t))-
Dÿy
11l1~112 (V)(tt,. uu² uu) 11~-1p
11a D2yp11-'112
@ 1688 (1(F) (((t)- -(( - -w) 4D3
+ (660)Byypp1
(26)
where (a) is due to Young's inequality, (b) is due to Cauchy-Schwaer and the definition of D2 as the diameter of BD in the Euclidean norm, and (c) follows by multiplying everything by 4|| and estimating. We can rearrange this to finally arrive at
(V - n1² 168² (f(x²) f(x++))) + 4D2 + 16D
Combining Both Cases Let M- max 16B 2 and M' max max(16)228,331) and let A c (1,2...., nt) denote the indices where clipping active. Let E N and denote
M M'PE11111a1 + Σ 4D3
E
33

---

Then, taking expectations, adding from k = 1 to n, and dividing by gives
Σ ypBKVf(x),x + ! Σ KEA
yBKV)()), -12 M (f(x') - -5t) yc.
YG.
(27)
We can lower bound the left hand side by the sum over A and divide by yp to get
MA 6, ypn p
BKV -0l
(28)
KEAT
Similarly, lower bounding the left hand side by the sum over the complement of A and dividing by y. we get by Jensen's 's inequality
Σ B(V -
BRV u)²1 s yn
MA
En.
(29)
We use the fact that 2az d² S x² for any O. Picking
f(=) ) x² - 1)). it follows that
2
Σ
(30)
π
2a
2a
2a
where A =
MA yn
+ En. Choosing a = VĀ and replacing z by B((V(())) x² 4)] we get
VMA
BKV 1)1s VĀ=
VEn
(31)
Vyn
Adding both of these we get
B(V - -11
VMA
+ Ve, +
MA
+ 6n
(32)
Vyn
ypn
p
20'+16 Let A
By lemma B.4, A 2 B| for n
Vii
Next, we can estimate
6x S M pA + 4
Ve, s pA, 2D2An
E s M' A, + D2 P
4
P
Substituting in the definition of An y, and p while also noting the definition of I", we get
MM
MA
E
+
VE, +
+
Vyn
ypn
P
MA
MA
4
M' + 2D2 +
+ M' A, + -
Vyn
ypn
p
M' (2 +663
VLMA
2Dz 20 16
n//4
n//4
n! 1//4
LMA
M' 2202 + 1622B
4D² (20 162,8²
+
+
1/4
1/4
n
which gives a big O rate
1
B(VV(("".F , =1) s
11/44
34

---

Theorem B.5, Suppose Assumptions 4.2 and 4.8 hold and let n E Nº. Consider the iterates {x") hssenn
generated by Algorithm 2 ( (Variant 2) with constant stepsize y S 1/L and p S 1/114. Then, for all E D,
BKVS(F"), u)]
4v Vyn
34 7pm
+ 4 VEn
84.
where A == f(x)) f and en := 1 2i.1 O 1 + 11) Furthermore, taking a = vn, vm)
vi) and p = 1/ such that yp 1/cnnity we have that
[((V)(²") x" .)1 = 0(
Proof. Note that, since f is continuously differentiable and 2 is compact, f must be Lipschitz-smooth on the scaled ball BD with respect to the norm || II; call the Lipschitz constant I., We can therefore start with the descent lemma for f at the points _ and x to find
0 0s f(*) - f(+tt)) (Vf(x)) A+1 - -x- + 2 -1 1 x112
L.
s f(+) - - ,/
2 " 111211111?
(33)
s f(²) f(x yr ((at M (V)(x))) - d" v)) 1/2 5 P
Now we can proceed case-by-case depending on whether clipping is active or not.
Case
Clipping Active (yT = yp;
(f 48 2 p).
0s f(xt) - f(+tt)) f(t)-f((tt))) - ym (Eat A + (Vf(tt) - ,
$ f(x)) - s(att1)- - ymh
2 (d vt 2 (V(xx), x² - u) + 2 2 - -Vf(C),
1
1
u) (VFCt)) -ot L +
ss(3) - -((+1) - 77k
1 1(t M) (V((t) - - 2
4)
11181le11a+== 4412 -11"%1a111111)
L.
ss(x) - -((+t) + +y7h
2 m 2 _(,)t11)) - -y11: 2 [((()) x - u1) + ym マ 21111120s
,8)
3
S s((+)) - -f((+t) yp (2LypB² 2088) yp (Vf(x)), u) + yp 3 s
11211111aa - 11 (V
(34)
in the second inequality we have used the fact that ||12112 s 21|x²1112 + 2116 Imo MI 48 and the fact that ||r*||2 5 D2. Finally, rearranging gives
ypV X -1) 2((())) + +3Dxyoll>*1
(35)
sutttt, sottrt s))
Case II No Clipping (yT7 = y
18 46
0 f(+)) - f(xtt)) y (d vt)
(dt,, via f(x+)) d 119 + 1. //22 (2/((at, (ct,
11
4B2
2
48²
s((+) - f(xttl) - y (dth 222z
M(VV - -d,,t) + Ly² (ct v892
(36)
4/2
862
s((+)) - s(1tt)) - (dt v422
(((()(()))) - -d.
802
35

---

where in the last inequality we have used that y s i. Rearranging,
(d 142
((t)
MM<V)((A) -
862
s $ -((4t))-
(VCSCC)) - - - -...t-u) 1 -
- d.
8/22
$(({) - )((A1)
(( - -))) Y (V - -1²² - -
802
4B
s $((3)-((**)) -1(A')
Y
(VS()),, - Y (V)(x) -d=,x u)² -YI(VS(E) d..
1682
4B
(Vf(x"), -1) D
s $(()) - - - 16B²
y
|1 + D2yplluB12e
4B²
s 166² (((x))((x( >>(s)(()))x+) u3² 4D3
(37)
The rest of the proof is exactly the same as it was for Variant so we omit it.
C Experiments
Our implementations follow Unconstrained ClippedScion and ClippedScion Algorithm 3 and Algo- rithm 4 (Variant 2), respectively. For simplicity, we absorb the latter's factor of 4 into the clipping threshold p. so both algorithms directly clip Σ1 (di, vf) at p.
CIFARIO 10 experiments are run a single A100 NVIDIA GPU, NanoGPT runs are run on 4 H100 NVIDIA GPUs, and ViT experiments use 16 x GH200 NVIDIA GPUs. Hyperparameters are provided in Tables 2 to 4.
SL.3
20.3
943
31.3
X2
913
$1.
$2.3
$3.4
963
942
15.3
$2.4
$2.3
53.3
934
943
$4.4
945
33.9
52.2
942
9233
943
33.9
214
$2.5
323
203
94.
S1.4
92.2
$3.3
206
903
53.7
K1.4
K2.
RX2
54.
99.
M4.0
92.8
89.3
943
RAB
51.0
$50
544
B2.7
95.4
92.2
83.0
A18
Figure 3: The optimal hyperparameters for Unconstrained ClippedScion on CIFAR1O for 80 epochs, (left) no stepsize decay (right) with stepsize decay. (indicated in red). The first row indicated with "No Clip" corresponds to Unconstrained Scion.
Olrpedidinn
Hearmaa
13402
B11
P0B
11-4
53.
664
15.4
33002
R55
300
93.3
50.5
85.
K0
123
452
50.
582
51.9
302
033
16022
4A-1
203
684
62.3
924
643
780
1900
27.2
303
48C
66.
11 Trareny Rate
Figure 4: The optimal hyperparameters for ClippedScion on CIFAR10 for 80 epochs, stepsize decay (indicated in red). The first row indicated with "No Clip" corresponds to Scion.
36

---

0.9
Clpping threshald
Unconstrained dispesScion
Gradient nerm
2000
0.5
3500
0.4
2000
Adam
Uncondtrained Scien
Unconderaindd Ciloodddcion
80
1250
1750
2000
Epoch
Iteration
Figure 5: For CIFARIO experiments with stepsize decay; Unconstrained Scion and Unconstrained ClippedScion achieve similar performance expected.
1x100
CtppedScion
Elppiig
0.8
5x10P
0.7
0.6
0.5
4x10"
0.4
2x10"
Sclunn
0.3
CllonesSdoo
Iteration
Figure 6: For CIFARIO experiments for constrained variant of the algorithms without stepsize decay; clipping less effective due to the surprising increase of (v* .d"). We observe that even the (deterministic) Wolfe gap is increasing, which is otherwise expected to go to zero.
0.90
Unconstraited Sción
Unconstrained ClppedScion
5000
Gradiant nomn [0
Test Accurssy
7000
0.76
5000
0.74
11% speedup
Uncomatrained Cippettcion
4000
Cliping threshold
0.70
25
175
Epoch
Epoch
Figure 7: Clipping improves over Scion by a 1 1% speedup on DeiT-base. Cosine learning rate schedule is used.
Unconstra CipocdEcion clipping
Gradient norm 10%
Villcation LOSS
3.7
600
Unconstrained Scion
Unconstrained ClepedScion
3.3
1000
2000
3000
4000
5000
1000
2003
3000
4000
5000
Iteration
Iteration
Figure 8: For fixed stepsize comparison clipping improves over Scion by more than 10% speedup on NanoGPT (124M).
37

---

unconstrained
3.7
Unconstrained Cippeedsion
Salidation Loss
3.6
3.3
3.
2000
3000
5000
Figure 9: NanoGPT (124M) with stepsize decay. Unconstrained Scion and Unconstrained Clipped- Scion similar performance for the final iterate as expected under stepsize decay.
3.8
1000
Sdon
Cippodicion
OtepedScion
Olppng tireshold
(olttation Locs
3..4
600
32g
Iteration
Iteration
Figure 10: NanoGPT (124M) for constrained variants of the algorithm with stepsize decay. An interesting observation, which requires further investigation, is that (1*, a,t) surprisingly increases during the linear stepsize decay.
3.3700
3.3675
Validation Loss
3.3625
3.000
3.3550
400 500 600 700 800
Figure 11: NanoGPT (124M) for Unconstrained ClippedScion with sweeping. The sweep range set according to the gradient norm from Figure (right). Both steepest descent (p oa) and conditional gradient (p - 0) perform worse than clipping.
38

---

Table 2: Hyperparameters for the CIFARIO experiments building on airbench [Jordan, 2024].
Hyperparameter
Adam
(Clipped)Sccin
Unconst. Scion
Unconst. ClippedScion
Block size (b1, b2, b3)
width factor x (64, 256, 256)
Activation function
GELU CIFARIO (50000 training examples)
Dataset batch size
2000 80
Epochs
Stepsize schedule
Linear decay y y -k/n)
Averaging parameter
0.9
0.5
Stepsize
le-3
2
2-3
2-2
Initial stepsize y for decay
2e-3
2-1
2-1
Clipping parameter p
12800
1600
Radius ri rr
1/5/2000
1/5 2/00 200
1/5/200 200
Table 3:
DeiT-hase hyperparameters following the tuned hyperparameters of Pethick et al. [2025]
Hyperparameter
Unconstrained Scion
Unconstrained ClippedScion
Layers Head dim
12
64
Activation function
v2. GELU (scaled to preserve variance)
Normalization function
RMSNorm
Sequence Length
197
Dataset
ImageNet-Ik 1kk
Stepsize schedule
Cosine decay
Max Ir
0,00024
Warmup epochs
End
10-7
Batch size
4096
Epochs
200
Averaging parameter
0.1
Radius Pr Pt.
25 / 25 500
Clipping parameter p
8000
Table 4: NanoGPT hyperparameters following the tuned hyperparameters of Pethick et al. [2025].
Hyperparameter
AdamW
(Unconstrained) Scion
(Unconstrained) ClippedScion
Layers Head dim
128
Activation function
2 ReLU(x)²
Vocabulary size
50304
Dataset
Fine Web
batch size
512
block size
1024
Iterations
5100
Warmdown
28.5% or 0%
y
if < m
Stepsize schedule
Constant then linear decay y/
y
(
ifk
Warmup Gradient clipping
5% Yes
0
No
Momentum B1 B2
0.9 0.95
Averaging parameter a
0.1
Stepsize yp
0.0018
0,00036
Clipping parameter p Radius rt / rp
600 for 124M model and 6000 for 1B model
/50 / 3000
39


Reviews:
[{"review_id": "QsDkZUQSl9", "reviewer": "Reviewer_eHBQ", "summary": "The paper proposes Generalized Gradient Norm Clipping (GGNC), a hybrid first-order method that interpolates between steepest descent (SD) and an unconstrained conditional-gradient (uCG) step by clipping the dual gradient norm with respect to an arbitrary norm. Under a generalized (L₀, L₁)-smoothness assumption, the authors prove:\n\n1. A descent lemma and an O(n⁻¹) deterministic rate to a neighborhood followed by O(n⁻¹/²) convergence near stationary points.\n2. An order-optimal O(n⁻¹/⁴) stochastic rate via a momentum estimator.\n3. An interpretation of weight decay as a clipped Frank-Wolfe \"short step,\" along with the Stochastic Short-Step Conditional Gradient (S3CG) algorithm.\n4. Instantiations for ℓ∞ (\"Clipped Sign\"), spectral, and product norms, accompanied by small-scale experiments on CIFAR-10 and NanoGPT, suggesting faster early-stage progress compared to unclipped methods.", "questions": "1. Computational overhead: For spectral or product norms, the lmo requires an SVD per update; how does GGNC scale to ImageNet-size models? Please provide wall-clock comparisons.\n\n2. Sensitivity to ρ: Figures use a fixed clipping threshold, but how robust is performance to mis-specifying ρ by ±2×? A sweep would strengthen claims.\n\n3. Adaptive stepsizes: Your theory requires γ ≤ 1/L₀. In practice, L₀ is unknown. Can backtracking or AdaGrad be combined with GGNC while retaining descent? Empirical evidence would increase the quality score.\n\n4. Baselines: Why are AdamW, Lion, and normalized SGD with momentum absent? Demonstrating advantages over these would affect my significance assessment.\n\n5. Assumption tightness: Is (L₀, L₁) actually satisfied for deep nets in the modular norm? Please supply empirical Lipschitz estimates or counterexamples.\n\nI would consider increasing my score if robust large-scale experiments with adaptive stepsizes and stronger baselines show clear benefits, or if you provide a cheap approximation of the lmo/sharp operator that preserves convergence.", "limitations": "The paper includes a \"Limitations\" checklist entry but does not discuss computational cost, hyperparameter sensitivity, or failure modes in depth. These should be added.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality\n\nStrengths:\n\n* Rigorous extension of the (L₀, L₁) framework to arbitrary norms; proofs are mostly complete and technically competent.\n* The connection between clipping and Frank-Wolfe short steps is elegant and may interest the optimization community.\n\nWeaknesses:\n\n* Theory largely recycles existing proof templates for Euclidean clipping; novelty is modest.\n* Crucial practical aspects—computational cost of the sharp operator and lmo for deep networks, choice of radius ρ, and stability when the dual norm is ill-conditioned—are not analyzed.\n* Stochastic guarantees require known L₀ and L₁ to set stepsizes, conflicting with the \"parameter-agnostic\" narrative in the introduction.\n\nClarity\n\nStrengths:\n\n* High-level motivation is clear and algorithms are presented in pseudocode (Algorithms 1–2).\n\nWeaknesses:\n\n* Notation is heavy, and several definitions (e.g., \"product norm\" radii) are buried mid-paragraph, making reproduction difficult.\n* Experimental section omits many details (network width, regularization, hardware, etc.); Appendix C only lists a subset.\n* Figures 1–2 plot clipped and unclipped variants but omit error bars for Adam, hindering interpretation.\n\nSignificance\n\nStrengths:\n\n* May stimulate further work on non-Euclidean clipping and constrained optimization links.\n\nWeaknesses:\n\n* Empirical impact is unconvincing: gains on CIFAR-10 are approximately 1–2 percentage points and vanish once a learning-rate schedule is introduced.\n* Experiments are confined to toy-scale models; no evidence on large-scale vision or language tasks where clipping is commonly used.\n* Competing adaptive methods (AdamW, Lion, RMSProp) are not compared, limiting relevance for practitioners.\n\nOriginality\n\nStrengths:\n\n* The specific synthesis of SD, uCG, and Frank-Wolfe short steps under (L₀, L₁)-smoothness appears new.\n\nWeaknesses:\n\n* Core idea—clip the gradient in a dual norm—has been folklore; the paper formalizes but does not fundamentally change practice.\n* Prior concurrent works (Gorbunov et al., Vankov et al.) already extend clipping under relaxed smoothness; the non-Euclidean lift is incremental.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, which has satisfactorily addressed my concerns. I'm increasing my score.", "rebuttal": "We thank the reviewer for the thorough feedback and address the remaining concerns and questions below.\n\n> I would consider increasing my score if robust large-scale experiments with adaptive stepsizes and stronger baselines show clear benefits, or if you provide a cheap approximation of the lmo/sharp operator that preserves convergence.\n\n**Large scale experiments**\nRegarding large scale experiments, the 10% speedup in Figure 2 is observed on a 124M parameter model (we will add the missing parameter count to the paper).\nWe ran additional experiments to check if the conclusion is robust to even larger model sizes, specifically a 1 billion parameter model:\n\n| Model size | Method | Final val. loss | speedup | optimal lr | lr sweep |\n|--|--|--|--|--|--|\n|124M | Adam| 3.43 | | $2^{-9}$ |$2^{-16},2^{-15},...,2^{-6}$ | \n|124M | Scion| 3.37 | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|124M | ClippedScion | **3.35** |10% over Scion | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|1B | Adam|3.14 || $2^{-11}$ | $2^{-16},2^{-15},...,2^{-6}$ | \n|1B | Scion|3.08 | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|1B | ClippedScion | **3.06** | 10% over Scion| $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n\nFor the above NanoGPT experiments we observe that:\n\n- The optimal learning rate transfers for ClippedScion (this is not the case for Adam).\n    We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.\n- Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.\n\nWe also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion.\nNote that this improvement is _without_ tuning $\\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h.\nWe will include this experiment in the final version of the paper.\n\n| Method | Test accuracy | speedup |\n|---|---|---|\n|Unconstrained Scion| 79.25% | |\n|Unconstrained ClippedScion| **79.46%** | 15% |\n\n\n**Cost of LMO computation/wall-clock time**\nRegarding the computation of the lmo/sharp operator, the only relevant norm-choice for which these objects cannot be computed in closed form is the spectral norm.\nFortunately, we can efficiently approximate the spectral LMO through a fast implementation of the Newton-Schultz iterations (see the experimental section). In practice, the Newton-Schultz iterations add a ~0.7% overhead, e.g., for NanoGPT (see [1], which our experiments are also in agreement with). This already small overhead shrinks proportionally with increasing bathsize and depth of the model.\n\nCompared with the speedup over Adam, the overhead is negligible. The wall-clock time speedup of Scion over Adam for NanoGPT in Figure 2 is 34%, while for ClippedScion it is 39%. We will include wall-clock time in the final version.\n\n[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse and Jeremy Bernstein, \"Muon: An optimizer for hidden layers in neural networks\"\n\n> Sensitivity to ρ: Figures use a fixed clipping threshold, but how robust is performance to mis-specifying ρ by ±2×? A sweep would strengthen claims.\n\nWe conduct additional experiments to test the sensititve to $\\rho$.\nThe sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\\rho$.\n\nNote that the two extreme cases of steepest descent ($\\rho=\\infty$) and unconstrained conditional gradient ($\\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method.\nWe will include a plot in the final version of the paper.\n\n|     $\\rho$   | $0$ | 400   | 500   | 600   | 700   | 800   | $\\infty$ |\n| -- |--| ----- | ----- | ----- | ----- | ----- | -- |\n| **Val loss** | 3.371 | 3.357 | **3.354** | **3.354** | 3.359 | 3.360 | 3.363 |\n\n\n> Notation is heavy, and several definitions (e.g., \"product norm\" radii) are buried mid-paragraph, making reproduction difficult.\n\nFor reproduction we have summerized the full algorithm in the appendix (Algorithm 3 and 4), which includes the per layer radii explicitly.\nThis should provide an overview of what steps are needed for the implementation. \nWe didn't include it in the main text due to space limitations, but we will consider pulling it up.\n\nThe supplementary also includes a compact pytorch implementation, which should make it straightforward to reproduce.\n\n> Experimental section omits many details (network width, regularization, hardware, etc.); Appendix C only lists a subset.\n\n- The network width is indeed missing from the hyperparameter table, thanks for catching that. \nThe NanoGPT width is 768 which makes it a 124 million parameter model, which is the default in the NanoGPT repository that we build on.\n- The cifar10 experiment uses the default data augmentation from the airbench repository that we build on. We will add a description to the paper.\n- The hardware used is specified in beginning of Appendix C.\n\n> Theory largely recycles existing proof templates for Euclidean clipping; novelty is modest.\n\nWe respectfully disagree with the framing of building on prior work as recycling existing proof templates. The way we are controlling the magnitude of the update is completely different to Euclidean clipping alone and so is the analysis that follows; prior works did not use non-Euclidean LMOs (e.g., spectral LMO) in combination with a clipped step size before our work, nor was it common in practice. \n\n> Prior concurrent works (Gorbunov et al., Vankov et al.) already extend clipping under relaxed smoothness; the non-Euclidean lift is incremental.\n> Core idea—clip the gradient in a dual norm—has been folklore; the paper formalizes but does not fundamentally change practice.\n\nOur change to a non-Euclidean setting goes much further than simply changing the norm in the clipping of the stepsize. The algorithm we are considering fundamentally alters the direction (and not just the magnitude) used to update the iterates through the use of the non-Euclidean LMO (a kind of nonlinear preconditioner). This significantly changes the algorithm, both in practice (we do not only clip the stepsize, we must also apply an LMO, e.g., Newton-Schulz iterations) and in its theoretical analysis (e.g., the direction of steepest descent used in the algorithm may not be unique, no inner product identities to rely on, etc).\n\n> Crucial practical aspects—computational cost of the sharp operator and lmo for deep networks, choice of radius ρ, and stability when the dual norm is ill-conditioned—are not analyzed.\n\nWe have elaborated on the cost of the sharp operator and LMO for deep networks in our previous responses but, to be clear, there is very little overhead coming from the computation of the sharp operator and LMO. Regarding stability when the dual norm is ill-conditioned, it's not clear what is meant here. There is no requirement on the dual norm besides needing the LMO to be accessible (either in closed-form or approximately using a subroutine like Newton-Schulz).\n\n> Assumption tightness: Is (L₀, L₁) actually satisfied for deep nets in the modular norm? Please supply empirical Lipschitz estimates or counterexamples.\n\nYes, there is evidence to support that.\nIn a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [2], the assumption is empirically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively).\nIt seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).\n\n[2] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik \"Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)\"\n\n> Baselines: Why are AdamW, Lion, and normalized SGD with momentum absent? Demonstrating advantages over these would affect my significance assessment.\n\nAdamW has been tested in this setting and was found to be worse than Adam. Therefore we do compare against Adam but not AdamW, which would be redundant for the modded-nanoGPT repository. We focus on comparing against one strong, well-tuned baseline rather than having many baselines: \n- The Adam optimizer is tuned (across 12 runs) and warmup is added to the learning rate schedule (in contrast to Scion/ClippedScion), which significantly boosts performance.\n- In addition to comparing to Adam, we compare against Scion since it can achieve state-of-the-art on NanoGPT experiments and since it provides a direct comparison to see the effect of clipping (no other changes are made simultaneously to the optimizer).\n\n> Adaptive stepsizes: Your theory requires γ ≤ 1/L₀. In practice, L₀ is unknown. Can backtracking or AdaGrad be combined with GGNC while retaining descent? Empirical evidence would increase the quality score.\n\nWe believe this is an interesting direction that we also mention in the conclusion, but it does not seem completely obvious how to generalize beyond the Euclidean case (despite our preliminary efforts). Besides this, the results we present are nonetheless novel (there were no previous results in this setting, even assuming knowledge of the constants) which is why we have left it for future work.\n\n> The paper includes a \"Limitations\" checklist entry but does not discuss computational cost, hyperparameter sensitivity, or failure modes in depth. These should be added.\n\nWe elaborate on the limitations in the checklist, thanks for pointing this out."}, {"review_id": "hPkcoZKjFu", "reviewer": "Reviewer_cpYw", "summary": "This paper proposes a new technique that generalizes gradient clipping to non-Euclidean ($\\ell_2$) norms. On the non-Euclidean norms, standard gradient descent is generalized to steepest descent with update $d^\\\\# = \\mathrm{argmax}\\_x \\langle d,x\\rangle - \\frac{1}{2}\\\\|x\\\\|^2$, while normalized gradient is generalized to conditional gradient via linear minimization oracle (lmo), that is $lmo(d) = \\mathrm{argmin}\\_{\\\\|x\\\\|=1} \\langle d,x\\rangle$. However, it was unclear what is the correct analogy of gradient clipping on non-Euclidean norms. To close this gap, this paper observes a key connection between the two methods, that is $lmo(d) = -d^\\\\# / \\\\|d\\\\|\\_* $, and proposes the generalized gradient clipping defined as $\\min\\\\{1, \\frac{\\rho}{\\\\|d\\\\|\\_*} \\\\} \\cdot d^\\\\#$.\n\nBased on this core idea, this paper provides two variants of gradient clipping, for unconstrained and constrained problems. In both settings, the authors provided convergence analysis in both deterministic and stochastic setting, and proved that the proposed gradient clipping algorithm converges to the optimal rate. In addition, the authors also validated the algorithm on both image classification and nanoGPT pre-training tasks, and showed that the proposed algorithm achieves better performance compared to the Adam baseline and Scion, an algorithm based on conditional gradient (analogous to normalized gradient on carefully chosen norms) instead of gradient clipping. These empirical results demonstrate the practical benefits of the new algorithm.", "questions": "NA", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper has several notable strengths. First of all, it introduces a novel technique that extends gradient clipping to non-Euclidean norms. By connecting steepest descent update and lmo update in conditional gradient, the authors managed to define a meaningful clipping on general norms.\n\nAdditionally, the paper makes a solid contribution to the literature by addressing the previously unresolved question of how to implement gradient clipping for general non-Euclidean norms. Prior to this work, the correct analogy for gradient clipping in non-Euclidean norms was unclear, leaving a gap in the understanding of optimization techniques. The authors successfully fill this gap, providing a valuable resource for researchers and practitioners looking to improve their optimization strategies.\n\nFinally, the proposed algorithm is supported by solid evidences, both in theoretical analysis and empirical experiments. The authors demonstrate that their proposed gradient clipping algorithm converges to the optimal rate in both deterministic and stochastic settings, and for both constrained and unconstrained problems. This theoretical foundation is complemented by empirical validation on tasks of image classification and nanoGPT pre-training, where the algorithm outperforms existing methods like the Adam baseline and Scion. These results demonstrate the empirical effectiveness of gradient clipping with general norms.\n\nOverall, I think this paper has solid results, thus recommend acceptance.", "was_revised": true, "final_justification": "Sorry for the late response, I wasn't aware of the new requirement of this year. I have read the responses from the authors and the comments from other reviewers, and I keep my current score as the recommended score.", "rebuttal": "Thank you for the positive review - we remain available if there are any further questions."}, {"review_id": "j8ozSja284", "reviewer": "Reviewer_wW69", "summary": "This paper introduces a training algorithm that combines norm-constrained linear minimization oracles (LMOs) with gradient clipping. The authors assume $(L_0,L_1)$-smoothness in their theoretical framework and prove an optimal convergence rate of $O(n^{-1/4})$. The performance of their algorithm is demonstrated through experiments on CIFAR-10 and NanoGPT.", "questions": "1. Is the $(L_0,L_1)$-smoothness condition valid in non-Euclidean settings? Do you have empirical verification of this assumption?\n2. Compared to Pethick et al. (2025), the optimal convergence rate remains the same in the stochastic case. Why is there no improvement despite additional smoothness assumptions?\n3. Pethick et al. (2025) use LMOs to avoid computing the dual norm, but your paper requires dual norm computation for gradient norm clipping. Is this why your learning rate depends on the Lipschitz constant while Pethick et al. (2025) does not?\n4. When obtaining $\\text{lmo}(d)$, $x$ is already constrained within domain $D$. Why not simply control the norm-ball $D$ directly?", "limitations": "The theoretical contribution appears incremental, and there is no significant improvement in empirical performance.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written with clearly presented theoretical results.\n2. The authors extend the understanding of clipping benefits (previously limited to the Euclidean setting) to arbitrary norms, proving descent under a novel non-Euclidean $(L_0,L_1)$-smoothness condition.\n\nWeaknesses:\n1. The motivation for using gradient norm clipping is inadequately explained. The approach does not appear to improve the convergence rate.\n2. The empirical performance of the proposed algorithm appears nearly identical to the unclipped version.", "was_revised": true, "final_justification": "I believe both the theoretical and empirical contributions of this paper are incremental, as explained in my earlier comments. Therefore, I tend to keep my score unchanged.", "rebuttal": "We thank the reviewer for the feedback and address the remaining concerns and questions below.\n\n> The empirical performance of the proposed algorithm appears nearly identical to the unclipped version.\n\nWe politely disagree. The difference in validation loss translates into a 10% speedup for NanoGPT, which we consider substantial.\nWe additionally carry out the following experiments in order to provide further evidence:\n\n- training a NanoGPT with 1B parameters in addition to the 124M parameter model in the paper\n- training a ViT on ImageNet\n- sensitivity of $\\rho$\n\n**NanoGPT 1B**:\n\n\n| Model size | Method | Final val. loss | speedup | optimal lr | lr sweep |\n| -------- | -------- | -------- | -- | -- | -- |\n|  124M    | Adam     |   3.43   | | $2^{-9}$ |  $2^{-16},2^{-15},...,2^{-6}$ | \n|  124M    | Scion     | 3.37 | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  124M    | ClippedScion    | **3.35** |  10% over Scion    | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  1B    | Adam     |  3.14   |  | $2^{-11}$ | $2^{-16},2^{-15},...,2^{-6}$ | \n|  1B    | Scion     |  3.08   | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  1B    | ClippedScion    | **3.06** | 10% over Scion  | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n\nFor the above NanoGPT experiments we observe that:\n\n- The optimal learning rate transfers for ClippedScion (this is not the case for Adam).\n    We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.\n- Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.\n\n**ViT on ImageNet**:\nWe also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion.\nNote that this improvement is _without_ tuning $\\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h.\nWe will include this experiment in the final version of the paper.\n\n| Method | Test accuracy | speedup |\n|---|---|---|\n|Unconstrained Scion| 79.25% | |\n|Unconstrained ClippedScion| **79.46%** | 15% |\n\n**Sensitivity to $\\rho$**:\nWe conduct additional experiments to test the sensitivity to $\\rho$.\nThe sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\\rho$.\n\nNote that the two extreme cases of steepest descent ($\\rho=\\infty$) and unconstrained conditional gradient ($\\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method.\nWe will include a plot in the final version of the paper.\n\n\n|     $\\rho$   | $0$ | 400   | 500   | 600   | 700   | 800   | $\\infty$ |\n| ------------ |--| ----- | ----- | ----- | ----- | ----- | -- |\n| **Val loss** | 3.371 | 3.357 | **3.354** | **3.354** | 3.359 | 3.360 | 3.363 |\n\n\n> The motivation for using gradient norm clipping is inadequately explained. The approach does not appear to improve the convergence rate.\n\nIndeed, the convergence rate we find is matching the one given in [1]. Ultimately, this is a worst-case convergence rate and, in practice, the behavior of the two algorithms is not always the same, even if there exists some pathological function for which both algorithms would have the same rate of convergence. This is evidenced by our accompanying numerical experiments, which confirm the benefits of clipping under constant learning rate in a practical setting like training a NanoGPT network.\n\n[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls and Volkan Cevher, \"Training Deep Learning Models with Norm-Constrained LMOs\"\n\n> Is the $(L_0, L_1)$-smoothness condition valid in non-Euclidean settings? Do you have empirical verification of this assumption?\n\nYes, there is evidence to support that.\nIn a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [1], the assumption is empirically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively).\nIt seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).\n\n[1] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik \"Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)\"\n\n> Compared to Pethick et al. (2025), the optimal convergence rate remains the same in the stochastic case. Why is there no improvement despite additional smoothness assumptions?\n\nThe $(L_0,L_1)$-smoothness assumption is a strictly _weaker_ assumption (i.e., more general, thus allowing for a broader class of functions).\n\n> Pethick et al. (2025) use LMOs to avoid computing the dual norm, but your paper requires dual norm computation for gradient norm clipping. Is this why your learning rate depends on the Lipschitz constant while Pethick et al. (2025) does not?\n\nYes ClippedScion needs to set the stepsize clipping factor $\\rho$, but can then converge faster in practice as seen in the experiments.\n\n> When obtaining $\\mathrm{lmo}(d)$, $x$ is already constrained within domain $\\mathcal{D}$. Why not simply control the norm-ball $\\mathcal{D}$ directly?\n\nDecreasing the radius is in some sense what Scion does when the stepsize is taken diminishing ($\\rho$ and $\\gamma$ plays the same role in the unconstrained case as e.g. seen in Theorem 4.5)."}, {"review_id": "ZQ1f7Fm8S3", "reviewer": "Reviewer_cFZM", "summary": "The paper introduces generalized gradient norm clipping (GGNC), a hybrid optimization method that combines steepest descent and conditional gradient approaches. It extends the concept of gradient norm clipping to non-Euclidean spaces and incorporates weight decay while maintaining convergence guarantees. The proposed method achieves a convergence rate of $\\mathcal{O}(n^{-1/4})$ in stochastic settings. Experimental results on image classification and language modeling demonstrate that GGNC outperforms the Adam optimizer in terms of both performance and convergence speed.", "questions": "Q1. Can the bounded variance assumption used for the stochastic convergence results in Section 4.2 be relaxed?\n\nQ2. Why is the generalized $(L_0, L_1)$-smoothness condition a suitable condition to study for neural networks, as claimed in Conclusion?\n\nQ3. Can GGNC be compared with the Muon optimizer [Jordan et al., 2024b], for example, on NanoGPT?\n\nBy the way, there is a typo on Line 290, where Figure 2 should be replaced with Figure 1.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1）The proposed GGNC effectively integrates steepest descent and conditional gradient approaches, extending gradient norm clipping to non-Euclidean settings and even constrained problems.\n\n2）In the deterministic case, GGNC is proven to be a descent method, under the generalized $(L_0, L_1)$-smoothness condition.\n\n3）In stochastic settings, GGNC achieves an order-optimal $\\mathcal{O}(n^{-1/4})$ convergence rate by utilizing a momentum-based gradient estimator.\n\n**Weaknesses:**\n\n1）Although the numerical results, especially the result in Fig. 2, are interesting, more experimental results on large-scale model training (e.g., the GPT-2 model series) and the ablation study are lacking and would be valuable.", "was_revised": true, "final_justification": "The responses addressed all of my concerns thoroughly.", "rebuttal": "We thank the reviewer for the positive feedback.\nThe remaining questions are addressed below.\n\n> Although the numerical results, especially the result in Fig. 2, are interesting, more experimental results on large-scale model training (e.g., the GPT-2 model series) and the ablation study are lacking and would be valuable.\n\nTo address the reviewer's comment, we carry out the following additional experiments:\n\n- training a NanoGPT with 1B parameters in addition to the 124M parameter model in the paper\n- training a ViT on ImageNet\n- a sensitivity study of the radius $\\rho$\n\n\n**NanoGPT 1B**:\n\n\n| Model size | Method | Final val. loss | speedup | optimal lr | lr sweep |\n| -------- | -------- | -------- | -- | -- | -- |\n|  124M    | Adam     |   3.43   | | $2^{-9}$ |  $2^{-16},2^{-15},...,2^{-6}$ | \n|  124M    | Scion     | 3.37 | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  124M    | ClippedScion    | **3.35** |  10% over Scion    | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  1B    | Adam     |  3.14   |  | $2^{-11}$ | $2^{-16},2^{-15},...,2^{-6}$ | \n|  1B    | Scion     |  3.08   | | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n|  1B    | ClippedScion    | **3.06** | 10% over Scion  | $2^{-12}$ | $2^{-17},2^{-16},...,2^{-7}$ | \n\nFor the above NanoGPT experiments we observe that:\n\n- The optimal learning rate transfers for ClippedScion (this is not the case for Adam).\n    We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.\n- Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.\n\n**ViT on ImageNet**:\nWe also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion.\nNote that this improvement is _without_ tuning $\\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h.\nWe will include this experiment in the final version of the paper.\n\n| Method | Test accuracy | speedup |\n|---|---|---|\n|Unconstrained Scion| 79.25% | |\n|Unconstrained ClippedScion| **79.46%** | 15% |\n\n**Sensitivity to $\\rho$**:\nWe conduct additional experiments to test the sensititve to $\\rho$.\nThe sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\\rho$.\n\nNote that the two extreme cases of steepest descent ($\\rho=\\infty$) and unconstrained conditional gradient ($\\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method.\nWe will include a plot in the final version of the paper.\n\n\n|     $\\rho$   | $0$ | 400   | 500   | 600   | 700   | 800   | $\\infty$ |\n| ------------ |--| ----- | ----- | ----- | ----- | ----- | -- |\n| **Val loss** | 3.371 | 3.357 | **3.354** | **3.354** | 3.359 | 3.360 | 3.363 |\n\n> Can the bounded variance assumption used for the stochastic convergence results in Section 4.2 be relaxed?\n\nYes, we expect that it should eventually be possible to relax the standard bounded variance assumption we have used, although we have not investigated this in detail. We are aware of several works that have relaxed this assumption in other algorithms (e.g., SGD) and it seems possible that the techniques used there could be combined with our analysis of GGNC to produce some results. Further studies on this, as well as clipping the gradient before adding it to the momentum estimator (which is known to be helpful for SGD with momentum in the case of heavy-tailed noise), are certainly fruitful directions to continue this work in.\n\n> Why is the generalized $(L_0,L_1)$-smoothness condition a suitable condition to study for neural networks, as claimed in Conclusion?\n\nThe best empirical evidence we are aware of is in a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [1] in which the assumption is emperically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively).\nIt seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).\n\n[1] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik \"Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)\"\n\n> Can GGNC be compared with the Muon optimizer [Jordan et al., 2024b], for example, on NanoGPT?\n\nThe main mechanism used in Muon is the spectral LMO, computed using the Newton-Schulz iterations. We compare in our paper to Scion, which we believe is a strong representative of a spectral LMO based method, since it performs slightly better than Muon on NanoGPT training and since it provides a way to directly see the effect of clipping. Between Scion and Clipped Scion, the only difference is the clipped stepsize. In contrast, if we tried to compare clipped Scion to Muon, we would have to account for the fact that Muon uses AdamW on the vector parameters and a different momentum, which would hinder interpretation.\n\n> By the way, there is a typo on Line 290, where Figure 2 should be replaced with Figure 1.\n\nThanks for catching the typo, we will correct it."}]

Author rebuttal:

Meta-review: **Summary.** This paper introduces Generalized Gradient Norm Clipping (GGNC), which extends gradient descent with Euclidean norm clipping to the non-Euclidean setting. The method interpolates between steepest descent and conditional gradient approaches. Under a newly defined non-Euclidean version of $(L_0, L_1)$-smoothness, the authors establish new convergence guarantees for GGNC in the non-convex case, generalizing prior results of Koloskova et al. (2023). The analysis is further extended to stochastic problems with bounded-variance gradient noise. In addition, the authors propose the Stochastic Short Step Conditional Gradient (S3CG) method, which combines clipping with weight decay and achieves the same convergence rate in the stochastic regime. Numerical experiments evaluating specific instances of GGNC and S3CG (Unconstrained Clipped Scion and Clipped Scion) demonstrate promising performance.

**Strengths.** The paper tackles an important problem: extending convergence analysis under generalized smoothness to non-Euclidean norms. This direction is strongly motivated by the recent success of LMO-based optimizers such as Muon and Scion in large-scale language model training. The contributions are therefore both theoretically and practically relevant. Furthermore, the work clarifies and unifies connections between different optimization methods, which is valuable for future research.

**Weaknesses.** The reviewers highlighted the following limitations: (1) modest empirical gains, (2) reliance on known proof techniques, (3) stochastic results requiring knowledge of $L_0$ and $L_1$, and (4) lack of clear motivation. The authors responded effectively with additional experiments and clarifications. In my view, point (2) is not a valid weakness - most theoretical works necessarily build on existing techniques, and novelty should not be reduced to technical tricks alone. Similarly, point (3) is a common assumption in related literature and acceptable here given this is the first work addressing the stated problem.

**Reviewers' consensus.** Reviewers cpYw, eHBQ, and cFZM expressed strong support (scores 5, 4, 5) and were satisfied with the rebuttal. Reviewer wW69 recommended rejection (score 2), raising concerns about novelty and significance in their review and during the internal AC-Reviewers discussion. I find these objections largely subjective. For example,
- Claiming the extension from the $\ell_2$ ball to the $\ell_p$ ball is “straightforward” is factually inaccurate since the authors extend to arbitrary norms, not merely $\ell_p$.
- Suggesting that $(L_0, L_1)$-smoothness reduces to standard smoothness because of clipping is incorrect, since the definition depends on the unclipped gradient.
- Questioning the motivation for clipping is subjective; in practice, clipping is a standard component of large-scale training, which itself justifies this study.

Overall, I do not find these concerns sufficient to outweigh the technical contributions and importance of the work.

**Final recommendation.** Considering the theoretical significance, the novelty of extending convergence guarantees to the non-Euclidean setting, and the positive reception from the majority of reviewers, I recommend **acceptance as an oral presentation**. The paper addresses an important and timely problem in optimization for machine learning, offers technically sound contributions, and presents encouraging empirical validation.
Decision: Accept (oral)