Training Deep Learning Models with Norm-Constrained LMOs
Authors: Thomas Pethick
Wanyun Xie
Kimon Antonakopoulos
Zhenyu Zhu
Antonio Silveti-Falls
Volkan Cevher
Venue: ICML 2025 spotlightposter (2025)
Keywords: non-euclidean
linear minimization oracle
deep learning
spectral norm
Primary area: deep_learning->algorithms

Abstract: In this work, we study optimization methods that leverage the linear minimization oracle (LMO) over a norm-ball. We propose a new stochastic family of algorithms that uses the LMO to adapt to the geometry of the problem and, perhaps surprisingly, show that they can be applied to unconstrained problems. The resulting update rule unifies several existing optimization methods under a single framework. Furthermore, we propose an explicit choice of norm for deep architectures, which, as a side benefit, leads to the transferability of hyperparameters across model sizes. Experimentally, we demonstrate significant speedups on nanoGPT training without any reliance on Adam. The proposed method is memory-efficient, requiring only one set of model weights and one set of gradients, which can be stored in half-precision.

Training Deep Learning Models with Norm-Constrained LMOs
Thomas Pethick'
Kimon Antonakopoulos Antonio Silveti-Falls Volkan Cevher'
Wanyun Xie
Zhenyu Zhu'
Abstract
Adaptation on-the-fly has been the defacto standard in this setting, with adaptive algorithms, such as Adam (Kingma, 2014), dominating the deep learning model training.
In this work, we study optimization methods that leverage the linear minimization oracle (Imo) over a norm-ball. We propose a new stochastic family of algorithms that uses the Imo to adapt to the geometry of the problem and, perhaps sur- prisingly, show that they can be applied to un- constrained problems. The resulting update rule unifies several existing optimization methods un- der a single framework. Furthermore, we pro- pose an explicit choice of norm for deep archi- tectures, which, as a side benefit, leads to the transferability of hyperparameters across model sizes. Experimentally,
One possible way for adaptation a priori, which we fo- cus on this work, to modify the underlying norm used to measure distances in the parameter space. There is precedence to our proposal, as the early work by Carl- son et al. (2015a;b;e) introduced the stochastic spectral de- scent method (SSD), which performs steepest descent in the spectral norm, and demonstrated that the method can substantially accelerate deep learning training.
The significance of the SSD approach has been very re- cently brought back attention by Bernstein & Newhouse (2024b), who showed that the Shampoo optimizer (Gupta et al., 2017)-winner of the external tuning track at the 2024 AlgoPerf: Training Algorithms competition (Dahl et al., 20n be viewed as SSD when a certain ac- cumulation is disabled. Moreover, Bernstein & Newhouse (2024b) introduced an efficient Newton-Schultz iteration to replace the approximate SVD calculations previously required. Jordan et al. (2024b) incorporated the Newton- Schultz iteration with additional momentum into SSD un- der the name Muon to achieve impressive results on the na architecture by applying it to the hidden layers.
demonstrate signifi- cant speedups on nanoGPT training using our al- gorithm, Scion, without any reliance on Adam. The proposed method is memory-efficient, re- quiring only one set of model weights and one set of gradients, which can be stored in half- precision. The code is available at https: / //github. www..////LIOO-ELLLLLL....
1. Introduction
Deep learning has greatly benefited from adaptive opti- mization methods such as RMSProp (Hinton et al., 2012), AdaGrad (Duchi et al., 2011; McMahan & Streeter, 2010), and Adam (Kingma, 2014), which dynamically change the geometry of the problem based on gradients encoun- tered on-the-fly during training. While these methods have demonstrated remarkable success, they fundamentally treat neural networks (NNs) as optimization problems where we lack any prior knowledge about their particular setting.
Contributions
This work focuses on developing an algo- rithmic framework that can exploit an appropriate choice of norm for the entire neural network with particular empha- sis on hyperparameter transfer across model sizes (Yang & Hu, 2021), convergence and practical performance.
To adapt to the geometry priori, we will build clas- sical (but unexpected) family of algorithms in contrast to the steepest descent methods, namely the ones involving the linear minimization oracle (Imo) over a norm-ball con- straint known as the Conditional Gradient (CG) methods.
However, NNs are far from being black boxes mexes-their struc- ture is not only known but they deliherately designed. This simple observation raises directly the question:
Is it more beneficial to adapt the optimizer priori, instead of exploring their respective geometries on-the-fly?
While classically being used for constrained problems, take the slightly unusual approach by showing that the Imos can be used even for unconstrained problems. The algorithm, dubbed as the unconstrained Stochastic Condi- tional Gradient method (uSCG), shows improvements both theoretically and practically when the norm-ball constraint matches the natural geometry of the problem.
'LIONS, EPFL CVV, Université Paris-Saclay Correspon- dence Thomas Pethick <thomas,, @eptl.ch>.
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Training Deep Learning Models with Norm-Constrained LMOs
Table 1. Special instantiations of uSCG according to different choices of norm. The reduced SVD is given as d U (o))
Weight decay is captured by SCG, which provides explicit control on the norm of the parameters.
Method
ax
Problem
Imo constraint set
Imo
Reference
Normalized SGD
1
Unconstrained
Euclidean
1l2lball
P Tdlfa
(Hazan et al., 2015)
Momentum Normalized SGD
[0, 1]
Unconstrained
Euclidean
||
baal
P Idlla
(Cutkosky & Mehta, 2020)
SignSGD
1
Unconstrained
Max-norm
Io ball
- sigg((d)
(Bemstein et al., 2018, Thm. 1)²
Signum
[0, 1]
Unconstrained
Max-norm
1 -bal
sign((d)
(Bemstein et al., 2018, Thm. 3)
Muon
[0,
Unconstrained
Spectral
lls. -ball
-ball
- -PVV
(Jordan et al., 2024b)
With non-Nesterov based momentum. The theoretical guarantee relies on increasing batch size.
In particular, we build on the Stochastic Conditional Gra- dient (SCG) method of Mokhtari et al. (2020) from the constrained setting, which provides explicit control on the norm of NN weight matrices. This is particularly relevant for robust image classification (Cisse al., 2017), gener- alization bounds (Bartlett al., 2017), Lipschitz control of generative adversarial networks (Arjovsky al., 2017; Miyato et al., 8), diffusion models (Karras et al., 2024, Sec. 2.3), and for ensuring Lipschitz continuity of NNs (Large et al., 2024).
An additional Imo-based algorithm ALMOND can be found in Appendix D.3, generalizing the Normalized SGD based method of Zhao et al. (2020), for training with large- batches. Key differences of ALMOND with uSCG and SCG are discussed to further motivate our algorithms.
2. Preliminaries
We are interested in solving the following general (possibly nonconvex) optimization problem
min f(x),
Concretely, we make the following contributions:
Theoretical rates: We introduce a new, stochastic Imo based family of algorithms uSCG, which can exploit the specific geometry of the problem. In doing so we achieve the (n -144 order optimal convergence rate under general nonconvexity and stochasticity for uSCG (Arjevani et al., 2022). Moreover, we provide a new analogous guarantee for the constrained case for SCG. A major benefit of both methods is that their stepsize is agnostic the Lipschitz constant, in contrast steepest descent which requires the stepsize to be taken small enough.
where / is smooth some not necessarily Euclidean norm and the problem is either unconstrained (e-g.. X IR:)) or constrained to X DD where D is the norm-ball defined as
D = {x |||||| S p}.
The central primitive in the algorithms considered in this work is the linear minimization oracle (Imo) defined as
(+) E arg min (8, x),
(2)
xED
Unification: Our Imo-based approach provides a unifying framework for various popular algorithms, based on the norm choice (see Table 1); as a byproduct we establish the first provable rate for the Muon optimizer with and without weight decay. More importantly, this generality allows to design method for deep learning based on operator norms called SCION (Algorithm 3), which enjoys zero-shot hyperparameter transferability (Yang et al., 2022), and can be implemented storing only one set of parameters and one gradient (stored in half- precisioc),,, economizing on mem- ory in large-scale training.
where we are particularly interested in the special case where the constraint set is a norm constraint || < p. for some p > 0 and some norm || . |]. which does not have to be the Euclidean norm. Examples of norm-constrained Imos are provided in Table and Table regarding operator norms. An important property of the Imo is that the opera- tor is scale invariant, i.e., Imo(a s) Imo(s) for > O, and in fact we have by construction under the norm con- straints that || Imo(s)) (+)] S p. Thus, it is only the direction of the input s that matters and not the magnitude.
Numerical validation: We carry exhaustive numerical evaluation of SCION ranging from small scale experiments on MLPs and CNNs to VIT on ImageNet and NanoGPT models with up to 3B parameters. We consistently observe the transfcrability properties across all settings for SCION. The scheme more tolerant to large batch sizes and ex- hibits superior performance due the a priori adaptation.
A classical method for solving the constrained variant of problem 1, when the Imo is available, is the Conditional Gradient method (CG) (Frank et al., 1956; Clarkson, 2010; Jaggi, 2013), which proceeds as follows with T E (0,
x k+1 = - 7h )a* + The Imo V (+)())))
(CG)
ensuring the feasibility of * via simplicial combination.
2

---

Training Deep Learning Models with Norm-Constrained LMOs
Algorithm Unconstrained SCG (uSCG)
Algorithm 2 Stochastic Conditional Gradient (SCG)
Input: Horizon n. initialization x' E X, dd = O, momen-
Input: Horizon initialization x² D, d = 0, momen-
tum 04 E (0, 1]. and stepsize "Ak E (0, 1)
tum og E (0, 11, and stepsize 7/k E (0,
1: for k k= l,..., do
1: for k = ,...,,"
.....,, , do
2: Sample &x ~ P
2:
Sample E ~ P
3: dh 4- V (x)++ +1 aag)d--
3: d t- 02 fx(= {&) + (1 - as )a
4:
x x + x² + +% lmo(d")
4:
x k+1 + (1 - 11++)))a )z) + 7 Imo(d")
5: Choose x" uniformly at random from {x ".)
5: Choose x" uniformly at random from {x² ,x',...,""" x")
Return nn
Return F²
Usually, the CG is attractive when the constraint set is an atomic set (e.g., the nonor ball) in which case each up- date may be efficiently stored. Our focus lies in the more unconventional cases of the vector & noom ball and spee- tral norm ball for which the updates Furthermore, we interested in the unconstrained case in addition to the constrained problem which CG solves.
The weight decay parameter μ E [0, 1] interpolates be- tween uSCG and SCG. If the weight decay is in (0, 1) then the algorithm is still an instance of SCG and thus solve constrained problem, but one with larger radius of p' with a stepsize chosen as 7/4 = 7n-p-
in contrast dense.
Therefore, all schemes in Table guarantees norm bound of on the parameters when combined with weight decay. The connection between weight decay and constrained op- timization, in the special case where Imo sign (when the norm- corm-constraint in (2) is the vector -norm) has also been observed Xie Li (2024); D' Angelo et al. (2023). Due to the fixed magnitude of the Imo both methods provides a guarantee on the maximum norm of the parameters.
In the stochastic regime, the analyzing Imo-based al- gorithms is involved. Even when the stochastic oracle fa, <) unbiased, the direction of the updates, as de- fined by Imo (m,(V.((x,))
not unbiased. To help overcome this difficulty, we will employ a commonly used trick of av- eraging past gradients with 0% (0, 1] (aka momentum),
dh = -
a1-a a ag ff(= Ec),
Insight 3.1. Both uSCG and SCG provide explicit control on the norm of the parameters:
which will rigorously help with algorithmic convergence.
SCG guarantees ||a=|| S p.
3. Our Methods
(ii) uSCG guarantees ||^|||| S S/psp0 Ek=1 7k-
For the unconstrained case we introduce a new method, dubbed the unconstrained SCG method (uSCG):
Norm control particularly useful for long runs (ef. Fig- ure 9) and avoid overfitting in multi-epoch training (cf. Figures 10 and 11 regarding CIFARIO experiments).
g+l = x² + +7% Im)(d")
(uSCG)
with stepsizes " E (0, 1). Instead of the convex combina- tion in CG, the update rule simply sums the lmos. In con- trast with e.g., gradient descent, the update always has the same magnitude regardless of the size of the gradient aver- age d The final algorithm is presented in Algorithm 1.
3.1. Choice of Norm Constraint
To choose an appropriate norm for deep learning, we build on the operator norm perspective of Large et al. (2024); Bernstein & Newhouse (2024a). To simplify the presenta- tion we will consider a linear MLP as a running example, but in Section 3.2, we point to our theoretical guarantees with activation functions.
For the constrained case, we revisit the SCG method of Mokhtari et al. (2020) and adopt it for the non-convex ob- jectives typically encountered deep learning model train- ing. This algorithm (Algorithm 2) proceeds as follows
Let us consider a linear MLP with the initial hidden layer defined as (2) = W, 2 + by and the remaining layers
x+1 (1 - 7*)a** k + Th Im)(d")
(SCG)
he(z) = Wehher(() + be E
with stepsizes 7 1).
with by = 0. We denote the global loss as L(hr (1), v) where is the loss function and is a 1-hot encoded target vector. We use the overloaded notation W, E Rdom xdia, where dout and dm implicitly have dependency on E and can thus be distinct across different layers.
Connection to weight decay For uSCG, weight decay has a very precise interpretation, since the method reduces to SCG. Consider the following variant of uSCG with weight decay gh+1 x + 7 Imo *
3

---

Training Deep Learning Models with Norm-Constrained LMOs
Table 2. Example operator norms and the associated Imos of a matrix A E Rdouu x xdia, The reduced SVD is given as A = U (ddiago)VV
sign acts elementwise, col, (A) := A. and (A) == As,. Note that this table is not exhaustive.
1 - RMS (ColNorm)
1 - xo (Sign) max,,; |
RMS - RMS (Spectral)
RMS + oc (RowNorm)
Norm
max, vaean
|col, (A) 12
V /an//dam ||
max, vdn row, (A)]22
LMO
(A) 1>
Voout (A)
Ar> (ign(A)
A v Alws Mhans//mmll UV
row;(A) H>
row, (A)
(Ailla
vid (A)]l
Table 3. The choice of lmo be different between layers and can depend the assumptions the input. For simplicity we overload notation and write the reduced SVD as W; = U diag(a)V) E uodo Xdi for all E [L].
Parameter
Wi (image domain)
[Wa)eeta...A-11
Wr
by
Norm
RMS RMS
RMS RMS
RMS RMS
RMS
RMS
LMO
max(1, / /was/d..)UUU
VVuuja.UUV
V /oos/d, UVT
row, row,(WL) H vdn I| now, (WE)
Tow;
di sign(Wr)
Telllnas
by
Init.
Semi-orthoonna
Semi-orthogonal
Semi-ootthgooal
Row-wise normalized Gaussian
Random sign
Fact 3.1 tells that we can bound operator norms using bounds the vector norms, i.c.
Table 4. Example Imo choices for 1-hot encoded inputs.
Parameter
Wi (1-hot encoded input)
Norm
2 RMS
1 RMS
1 1++ oc sign( Random sign
|| s 1= s || S vdll=laa s dli= Va E Rd
IMO
UVT
marttr (W)) - Mani (ill
(5)
Init.
Schhoontl
Column-wise normalized Guussian
We start by focusing on controlling the RMS norm, but will later consider other norms. There are three example opera- tor norms to consider for the MLP in consideration:
We need that none of the intermediary hidden states he(z) blows up by requiring one of the following norm bounds:
ii) Initial layer h1(:): [Willa]-RMM 1.
Lows [1)(2)l1
(the average entry is bounded)
(ii) Intermediary layers he(z): [We||RMS-RR S
(ii)
[[be(2) [RMS I 1
(the typical entry is bounded)
YE E (2, L 1).
(iii) [lbe(=)]lo 1
(the maximum entry is bounded)
(iii) Last layer (z)::): I|W, W 1
where || := la ||l|| for E Rdd Assuming the input to any given layer is bounded in some norm || lla. this requirement corresponds to placing an operator norm constraint on the weight matrices (Wehre(r.) and a norm constraint on the biases (be (be)ee[L-1)-
Note that the operator norm || ||RMS -RMS is a scaled spectral norm, i... || |RMSSSSSS / /un/dout |||||| 2-22 V / domee || Alls... for E Rdouxdia.
To concisely write the layerwise norm constraints in terms of norm constraint on the joint parameter æ (We, be JE(L), we can define the norm in the Imo (2) as
The operator norm in turn defined as follows
|Alla=
max
| 1
sup |=
|
(4)
||>||| max max |bell) < 2E[L] Pt
X EER" ,4300
where Pe is a layerwise scaling factor of the constraint ra- dius. What particularly convenient algorithmically for the -nom is that the layerwise Imos can be computed separately (ef. Algorithm 3). A norm choice across layers was made through the fnorm in Flynn (2017) and through the -norm with the modular norm (Large et al., 2024) and
Directly from the definition, we have that if the input a is bounded through ||*/|a < 1, then the output ||A=l|s will be bounded when || is bounded.
A collection of operator norms and their resulting Imos is provided in Table 2. It will be convenient to convert be- tween bounds on these different operator norm which the following fact makes precise.
(Balles et al., 2020; Yu et al., 2017; Ginsburg et al., 2019).
Fact 3.1. The operator norm satisfies for some p > O
A choice needs to be made for the input norm on and output norm 3L. which depends on the application:
(i) || < pllz|ler Vz E Rdd + ||All S AllAlla-oo
Input layer For image domains, usually the input is rescaled pixel-wise to e.g., ensure that z E [-1, 1] in which
(i) ||=||a 2 |llelle, eRd = ||Allla-3 S P||A|loas
4

---

Training Deep Learning Models with Norm-Constrained LMOs
case || 1 and the appropriate operator norm for the [
Recommendation 3.1. We refer to the instantia- tion of uSCG and SCG using operator norms as UNCONSTRAINED SCION and SCION respectively (cf. Algorithm 3), which stands for Stochastic Conditional Gradient with Operator Norms. We recommend the following configurations of the layer norms (First layer + Intermediary layers + Last layer):
first layer becomes || IIM = V /aa//ooe [ -. In order to deal with the case where din > dout. we choose the radius to be max(1, (dom/d..) (cf., Appendix B.1).
For language tasks, the input 2 is usually a 1-hot encoded vector in which case =0 = ||=1|2 = ||=||| = 1. In turn, [Willl-RMM = ||W [WiiII-RMS holds on this restricted domain, where we can freely pick the oper- ator norm that leads to the simplest update rule (Table 2).
[| -RMS
(i) image domains:
Spectral - Spectral + Sign
The simplest form for the Imo is arguably induced by || -III-RMMM since the Imo can be computed exactly, while from || -|-ll-- we can observe a more aggressive scaling factor in the Imo, Von than the - used in intermediate layers. The norm choice || Inn- was first proposed in Large al. (2024) for 1-hot encoded input. Through the above reasoning we see how the norm is equivalent to an appropriately scaled spectral norm.
(ii) 1-hot input:
ColNorm -> Spectral + Sign
iiii weight sharing:
Sign + Spectral + Sign
The Imo names are defined in Table 2 and weight sharing refers to parameter sharing between the first and last layer. Each layer should be scaled ap- propriately according to Tables 3 and 4.
Output layer For the final layer, we are not restricted to bounding the output in fRMS and can alternatively choose
Other norm choices So far our argument has been based on the invariance provided by || ||RMS-RMS for interme- diary layers: i.e., the RMS norm of the output of layer & is bounded, so the input of next layer L + 1 is also bounded in the RMS norm. Since the Imo of || ||RMs- can be computed efficiently we can directly use this norm choice for our update rule. However, it is possible to choose an- other norm such as || |la-RMMs. as long as the RMS norm guarantee on the output of Wr is converted into a guaran- tee on the fi-norm of the input of layer & + 1. Specif- ically, we have that || RMM--RM S d 1|1-RMS through Fact 3.1. Alternatively, we can rely the invari- ance provided by 11 for which Fact 3.1 tells us that I1-llx-+oo S | and 11o1loo->o S dll We obtain methods exclusively relying on RowNorm, Col- Norm and Sign as summarized in Table of Appendix B.
bounding the maximal entry through foo- Additionally, we can bound || S ___ || 1- by using (5) through Fact 3.1, which leads to a dimension scaled sign update rule for the last layer.
We summarize the different norm choices and their result- ing Imos in Tables 3 and 4. Table 3 provides an overview of norm choices of output layers, while Table 4 provides choices for input layers under 1-hot encoded input.
Provided that the input is bounded as described, each of the hidden states hz(2) and logits h2 (z) will be bounded in the RMS norm. In order to ensure feasibility of the initial- ization in the constrained case when employing SCG, we propose to initialize on the boundary similar to Large et al. (2024) (see Appendix B for details).
3.2. Hyperparameter Transfer
Insight 3.2.
The intuition behind why (UNCONSTRAINED) SCION may enjoy hyperparameter transfer is suggested by the spectral scaling rule of Yang et al. (2023), which states that feature learning may be ensured by requiring that, for MLPs with weight matrices We Rde-1xde the following holds:
(i) For 1-hot encoded input, ColNorm and Spec- tral are equivalent for the first layer, in which case ColNorm is favored since the Imo can be computed exactly.
(ii) Sign can be used both for the first and last layer which is crucial for weight sharing.
|[Welss.. - - (V (Valt and |l = de
de
(Vale, de
de
(iii) To transfer learning rate from proxy models when the width is smaller than the input di- mension it is important to rescale the Imo as max(1, / /dour/
where || Ils5 denotes the spectral norm and AWr is the update change. For uSCG, the update change is given by x*++1 ,² 7 lmo ), so the requirement is automati- cally satisfied by the spectral norm choice from Table 3.
We formalize this intuition in Lemma C.6 of Appendix C.2 following the proof technique of Yang et al. (2023), which holds for losses including logistic regression and MSE,
These observations leads to the recommendations below.
5

---

Training Deep Learning Models with Norm-Constrainedd LMOs
and activation functions including ReLU, GELU and Tanh. Specifically, we show that the so-called maximal update learning rate > (i.e., the learning rate that enables the hidden layer preactivations to undergo the largest possible change in a single update step) is independent of width. Our theoretical analysis is conducted under a simplified set- ting with momentum Org = in Algorithm 1, which we adopt to facilitate a clean derivation of the width-invariant property. Thus, a learning rate tuned on smaller model can be directly applied to wider model without compro- mising the maximal update property.
the Lipschitz constant L (cf. Theorems 5.4 to 5.7).
Unlike the Imn-based schemes, extending sharp-operator- based algorithms to handle constrained problems is non- trivial even in the vector case (El Halabi, 2018). Addition- ally, a practical concern of using specifically spectral norm projections in deep learning is that the model weights them- selves can be dense (so the required SVD would be expen- sive), while gradients used in the Imo are usually low-rank (allowing efficient SVD approximations).
Muon The Muon optimizer (Jordan et al., 2024b) is in- troduced as a steepest descent method. The implementation interestingly ignores the scaling || ||> appearing in the up- date (ef., (7)), so Muon is effectively using the Imo over the spectral norm instead of the sharp operator. Provided a stream of stochastic gradients (g keN and an initialization x0 E X the Muon optimizer can then be written as follows
4. Related Works
Hyperparameter transfer Yang & Hu (2021); Yang et al. (2022) showed that there exists parameterization (i.e.. a choice of initialization and layerwise stepsize scal- ing) for which the features in every single layer evolve in a catth-independent manner. The so-called Maximal Update Parametrization (nP) allows transferring optimal hyperpa- rameter from a small proxy model large model.
G ===9 + +BG* x+11
(Muon)
a*th + y Immo + gG*) if Nesterov
A relationship with the spectral norm was established in Yang et al. (2023). An operator norm perspective was taken in the modular norm framework of Large et al. (2024); Bernstein & Newhouse (2024a), which was used to show Lipschitz continuity with constants independent of width. We build on this perspective and propose the 1 -> oo op- erator norm and RMS + oo, which leads to sign update rule and row normalization respectively.
2 + --ylmo (mo
otherwise
where the Imo corresponds implicitly to the spectral norm.
The accumulation G can be written in terms of the aver- aged gradient d² in Algorithm 2. We have that d² = oG by picking a = (1 - 3). Since the Imo is scale invariant, d and G can be used interchangeably without changing the update. Thus, we can alternatively write Muon (with non-Nesterov based momentum) exactly as uSCG.
Steepest descent in a normed space Steepest descent in a possibly non-Euclidean space can be written in terms of the Imo (cf., Appendix A.I) provided stream of stochastic gradients (g*) kgN) and an initialization N² E X,
In practice Muon is only applied to hidden layers, thus ex- cluding the first layer and the last layer for which Adam (W) or SGD is used. In contrast, we apply uSCG and SCG to all layers and demonstrate transferability of the stepsize.
bb1 = x 7 = = + 211 lmo(g*),
(7)
Moonlight Since the first version of this paper appeared on arXiv on the 11th of February, Liu et al. (2025) also proposed integrating Muon with weight decay to control the norm of the parameters. They use the same scaling fac- tor of the Imo, / / dout J. as the Muon bascline we compare against (ef. Appendix E.4).
where [.] = arg max 2 (, - || is the sharp- operator (Nesterov, 2012; Kelner et al., 2014).
The deterministic case is analyzed in Nesterov (2012); Kelner et al. (2014), and was extended to the stochas- tic case in Carlson et al. (2015b), with a particular empirically focus on the spectral norm, named as (pre- conditioned) stochastic spectral descent (SSD) (Carlson et al., 2015a;b;c). Their SSD algorithm is an instance of the
MARS The MARS- MARS-Shampoo optimizer (Yuan et al., 2024, Alg. 4) can be seen as an instance of SCG with spectral norm constraints, but using the STORM gradient estimator (Cutkosky & Orabona, 2019) instead of (3).
(MM) algorithms (Lange, 2016), which iteratively minimizes a locally tight upper bound. The dualization in Bernstein & Newhouse (2024a) is also motivated by the sharp-operator.
Sign SignSGD and the momentum variant Signum were brought to prominence and further analyzed in Bernstein et al. (2018) motivated by efficient communication for dis- tributed optimization, while they are originally introduced with the dual norm scaling and used only for weight bias updates in Carlson et al. (2015a;b;c). These schemes are typically studied under the framework of steepest descent,
In contrast to (7), uSCG and SCG are invariant to the mag- nitude of the gradients and do not need to compute the dual norm || -I... which cannot be computed independently across layers. Intuitively, the scale invariance of the Imo al- lows convergence to be established without knowledge of
6

---

Training Deep Learning Models with Norm-Constrained LMOs
which results in the Ilgh stepsize scaling in (7) usually not present in practice as remarked in Balles al. (2020).
those methods, we do not keep track of an explicit precon- ditioner; we compute the Imo at each iteration instead.
Normalization The LARS optimizer (You et al., 2017) uses normalized gradient and was shown to be particu- larly useful for large batch settings. The method can be viewed as performing normalized SGD with momen- tum (Cutkosky& & Mehta, 2020) layerwise with a particular adaptive
Natural gradient Early work on non-Euclidean meth- ods considered measuring the "distance" between models through the Kullback-Leible output distributions of the models (Amari, 1998). Approxi- mating the KL divergence through a Taylor expansion leads to steepest descent method, which preconditions with the Fisher information matrix, known as the natural gra- dient method. Trust-region variants of the natural gradi- ent method were considered with TRPO (Schulman et al., 2015) similarly to how uSCG can be seen as a trust-region variant of steepest descent.
divergence between the
dependent stepsize.
The layerwise normalization can be captured by uSCG with the norm choice maxe [[Wlll.. The LAMB optimizer (You et al., 2019) incorporates the update into Adam-like structure. Zhao et al. (2020) considers averaging the nor- malized gradients rather than the gradients prior to normal- ization. The update can be written in terms of an Imo, with the (flattened) norm choice ||æ||22 which we generalize with a new algorithm in Appendix D.3 to arbitrary norms.
5. Analysis
We begin by presenting the two main assumptions we will make to analyze Algorithms and 2. The first is an as- sumption on the Lipschitz-conttnuity of Vf with respect to the norm I|- restricted X. We do not assume this norm to be Euclidean which means our results apply to the geometries relevant to training neural networks.
With zero initialization, xi = 0, and stepsize 7 y = 1/n, uSCG recovers the stochastic con- tinuous greedy method (Mokhtari et al., 2020; Vondrák, 2008), which can be used to solve DR-submodular maxi- mization problems under Matroid polytope constraints.
Continuous greedy
Assumption 5.1. The gradient f is L-Lipschitz with L (0, oo), i.e.,
LMO for deep learning SCG for training neural net- works has been suggested in Pokutta et al. (2020) and Lu et al. (2022). where optimization was specifically con- strained to the K-sparse polytope with increasing batch- size for handling stochasticity. Beyond these works, we provide convergence guarantees for SCG with constant batch-sizes and, introduce principled framework for se- lecting effective constraints based on the input and output space geometries of the layers of the network.
[Vf(2) (-(()) S L||T vl Va,yE X.
(8)
Furthermore, f is bounded below by f*
Remark 5.2. Strictly speaking, uSCG only needs Lipschitz continuity to hold locally within a radius YP, since the as- sumption is only invoked between two consecutive iterates.
Our second assumption is that the stochastic gradient oracle we have access to is unbiased and has a bounded variance, typical assumption in stochastic optimization.
The perturbation in the tharpness-aware minimization (SAM) has been interpreted as an Imo and generalized to arbitrary norms (Pethick et al., 2025), focusing on the max- norm over nuclear norms, max; |[W..s
Assumption 5.3. The stochastic gradient oracle ff({E) X + Rd satisfies.
Scion can also be seen an instantiation of the Lion-KC (Chen et al., 2023) algorithm with ØK chosen to be the Imo. In this case, however, IC is not smooth and the continuous- time analysis presented in Chen et al. (2023) and related works do not apply.
(i) Unbiased: Ee
Vo E.
(ii) Bounded variance:
E& [[V [ < o Vr VrXX,02 0.
With these assumptions, we can state our worst-case con- vergence rates, first for Algorithm I and then for Algo- rithm 2.
The SCG method can be seen as trust- region method with a linear surrogate. Usually, the surro- gate is taken to be quadratic (ef. Wright (2006, Ch. 4)). We refer to Conn et al. (2000) for an extensive overview of trust-region methods.
Trust-region
To bridge the gap between theory and practice, we inves- tigate these algorithms when run with constant stepsize 7. which depends on the specified horizon E N' and momentum which is either constant a E (0, 1) (except for the first iteration where we take a = by convention) or vanishing ag > 0. All results can be extended to any time guarantees in a straightforward manner by choosing 7K as
Preconditioned SGD We also recognize the spectral Imo used in SCION as being related to the Preconditioned SGD (PSGD) family of algorithms (Li, 2017; Pooladzandi & Li, 2024) in the sense of whitening the update. In contrast to
7

---

Then, it Training Deep Learning Models with Norm-Constrained LMOs
function of the iteration counter k instead of horizon n and modifying the proofs accordingly.
Theorem 5.6 (Convergence rate for SCG with constant o)). Suppose Assumptions 5.1 and 5.3 hold. Let N and consider the iterates {x* Jkm generated by Algorithm 2 with constant stepsize 7y = 1 and constant momentum a E (0, 1). Then, for all u DD holds that
The exact constants for the rates can be found in the proofs in Appendix D; we try to highlight the dependence on the parameters L and P. which correspond to the natural ge- ometry of f and D, explicitly here. Our rates are non- asymptotic and use big O notation for brevity.
vn
E[(V (FV)( F. u) =
Le vn
+).
Theorem 5.4 (Convergence rate for uSCG with constant a). Suppose Assumptions 5. and 5.3 hold. Lern E N and consider the iterates {x* Jk=1 generated by Algorithm and constant momentum with constant stepsize ? v/
Theorem 5.7 (Convergence rate for SCG with vanishing (g.). Suppose Assumptions 5.1 and 5.3 hold. Let e N' and consider the iterates {-²}]= generated by Algo- rithm with a constant stepsize y satisfying 2n374 < ? < n374 and vanishing momentum On
E (0, 1). Then, it holds that
The Then, for all
u D, it holds that
|[IVS("")ILL S S + +o). vn
Lp
E (VV 10:00000 w// + me/1)
Theorem 5.5 (Convergence rate for uSCG with vanish- ing OKg). Suppose that Assumptions 5.1 and 5.3 hold. Let n E N' and consider the iterates {=²))= generated by Algorithm with a constant stepsize y satisfying 2/3/4 <
Insight 5.2. For both algorithms, our worst-case analyses for constant momentum suggest that tun- ing a requires balancing two effects. Making a smaller helps eliminate constant term that is pro- portional to the noise level σ.. However, if a be- comes too small, it amplifies O(1/vn) term and an O(a/n) term. The stepsize y must also align with the choice of momentum a: for vanishing ay the theory suggests a smaller constant stepsize like ? 1(19/45 to ensure convergence.
< 3/4 and vanishing momentum as = Vi holds that
|[|Bs("")|l]) =0
-0
1374 +
Le n373
Insight 5.1. The uSCG algorithm remarkably does not require knowledge of the Lipschitz constant L, which is intuitively explained by viewing the method as a normalized version of steepest descent through the relationship Imo(-) izing gradients with the dual norm has also been used in the online learning community to adapt to (local) Hölder smoothness as a simple alternative to AdaGrad-Norm (Orabona, 2023).
6. Experiments
[]' Normal- I-ll-
For computing the Imo of layers using a spectral norm con- straint, we use the efficient implementation provided in Jor- dan et al. (2024b) of the in Bernstein & Newhouse (2024b). In this section, Muon (Jordan al., 2024b) refers to the version used in practice, which uses AdamW for the first layer and last layer and Nesterov type momentum.
iteration proposed
These results show that, in the worst-case, running Algo- rithm with constant momentum a guarantees faster con- vergence but to a nominndde region with radius pro- portional a. In contrast, running Algorithm 1 with van- ishing momentum og is guaranteed to make the expected dual norm of the gradient small but at a slower rate. Algo- rithm 2 exhibits the analogous behavior, as we show next.
GPT We build on the excellent moodeddnanognn code- base (Jordan al., 2024a), which makes the following modernizations to Karpathy (2023): rotary embeddings is used instead of positional embeddings, RMS norm is used instead of LayerNorm, and linear decay schedule instead of a cosine stepsize, and the ReLU² instead of GELU activa- tion function (scaled according to Appendix E.3). SCION and UNCONSTRAINED SCION use the (Sign + Spectral + Sign) configuration with scaling factors in accordance with Tables 3 and 4. We train for 5100 iterations with batch- size of 512 on the Fib dataset (see Table 7 regarding hyperparameters) In comparison with Adam, both Muon and (UNCONSTRAINED) SCION do not require learning rate warmup. We sweep over stepsizes and model width in Figure 1.
Before stating the results for Algorithm 2, we emphasize that they are with constant stepsize 7, which atypical for conditional gradient methods. However, like most condi- tional gradient methods, we provide convergence rate the so-called Frank-Wolfe gap which measures criticality for the constrained optimization problem over D.
Finally, we remind the reader that the iterates of Algo- rithm 2 are always feasible for the set D by the design of the update and convexity of the norm ball D.
8

---

Training Deep Learning Models with Norm-Constraine LMOs
14
34
Figure 1. Performance on NanoGPT with between 64M and IB parameters. The optimal learning rate of SCION is invariant to width.
Method
3B model Using the optimal configuration of the 124M parameter proxy model, we perform a large model exper- iment on a 3B parameter model, which also increases the depth. Specifically, we take the embedding dimension to be 2560 and the depth to be 36. We observe in Table 5 that UNCONSTRAINED SCION outperforms all other methods. The loss curve is provided in Figure 8 of Appendix E.
Adbee
Vuen
busn
Uscerstarnee Scie
Table 5. Validation loss 3B parameter GPT model.
Adam
Muon
UNCONSTRAINED SCION SCION
3.024
2.909
2.882
2.890
Figure 2. Batch size sensitivity on NanoGPT (124M). The gener-
alization of SCION is less sensitive to larger batches.
Large batches To test the effect of large batches we fix the total number of tokens for the 124M parameter model and sweep over the batch sizes while rescaling the total number of steps accordingly. The stepsize ? optimized over {2-17, 2 16 ..... 2 2- for each combination of batch size and optimizer. We observe that (UNCONSTRAINED) SCION is better at maintaining a low validation loss with increasing batch size than the baselines (ef., Figure 2).
For large batches, SCION achieves a significantly better validation loss than Muon. To assess the implication for training time, we perform additional experiment for the large batch size of 6144, where reduce the number of iterations until SCION matches the larger validation loss of Muon. We find that SCION can achieve the same validation loss as Muon with a 25% smaller wallclock time.
Adunti
Scion
Accurscy111
Figure 3. SCION leads to 30% fewer epochs VIT on ImageNet and 400% wallclock speedup due to larger critical batch size.
We provide two possible explanations for why SCION fa- vors large batches: The treatment of the noise is tied the Fuclidean geometry, in order to exploit the unbiasedness of the stochastic oracle, apparent from the analysis. Addi- tionally, in the extreme case of single sample, the gradi- ents of the linear layers are rank-1 and all Schatten norms become equivalent (e.g.. Frobenius and Spectral norm).
From Figure I, we observe that the optimal stepsize of SCION and UNCONSTRAINED SCION transfer across model width as oppose to Adam and Muon. The 124M model size configuration in Figure corresponds to one of the official speedrun entries of Jordan et al. (2024a), for which SCION has slightly lower validation loss (across 3 runs) than Muon. Even when Muon is tuned on the largest model size it achieves a validation loss of 2.988 in compar- ison with 2.984 of UNCONSTRAINED SCION.
Image classification We additionally test on vision trans- formers (ViT) on ImageNet and convolutional neural net- works (CNN) on the CIFARIO dataset using the configu- ration (Spectral Spectral - Sign). The explicit control on the norm provided by SCION circumvents the need for the Frobenius norm normalization of the weights present in the CIFAR10 implementation of Muon (Jordan, 2024). The results regarding ImageNet are shown in Figure (gf. Appendix E.4 for details and experiments on CIFAR10).
Our methods completely remove the need for using Adam otherwise present in the Muon implementation, which per- mits an implementation that only requires storing one set of weights and one set of gradient (stored in half- -elf-precision) across all layers (see Appendix E.2). The experiments ad- ditionally demonstrates that our method works for weight
sharing.

---

Training Deep Learning Models with Norm-Constrained LMOs
Impact Statement
Bernstein, J. and Newhouse, L. Old optimizer, new norm:
An anthology. arXiv preprint arXiv:2409.20322, 2024b.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bernstein, J., Wang, Y.X.,, Azizzadenesheli,, K., and Anandkumar, A. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learming, pp. 560-569, PMLR, 2018.
Acknowledgement
Carlson, D., Cevher, V., and Carin, L. Stochastic spectral descent for restricted Boltzmann machines. In Artificial Intelligence and Statistics, pp. 11 11-119.. PMLR, 2015a.
We thank Leena Chennuru Vankadara and Jeremy Bern- stein for helpful discussion. This work was supported as part of the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 Alps. This work was supported by the Swiss National Science Foundation (SNSF) under grant num- ber 200021 1220011 This work was supported by Hasler Foundation Program: Hasler Responsible Al (project num- ber 21043). Research was sponsored by the Army Re- search Office and was accomplished under Grant Number W91 INF-24-1-0045 ASF was supported by public grant from the Fondation Mathématique Jacques Hadamard.
Carlson, D., Hsich, Y.-P., Collins, E., Carin, L., and Cevher, V. Stochastic spectral descent for discrete graphical models. IEEE Journal of Selected Topics in Signal Pro- cessing, 10(2):296-311 2015b.
Carlson, D. E., Collins, E., Hsich, Y.P., Carin, L., and Cevher, V. Preconditioned spectral descent for deep learning. Advances in neural information processing sys- lems, 28, 2015c.
Chen, L., Liu, B., Liang, K., and Liu, Q. Lion secretly solves constrained optimization: As Iyapunov predicts. arXiv preprint arXiv:2310.0558 2023.
References
Amari, S.-I. Natural gradient works efficiently in learning. Neurul computation, 10(2):251-276 1998.
Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N. Parseval networks: Improving robustness adversarial examples. In International conference on machine learning, pp. 854-863. PMLR, 2017.
Arjevani, Y., Carmon, Y., Duchi, J. C.,, Foster, D. J., Sre- bro, N., and Woodworth, B. Lower bounds for non- convex stochastic optimization, 2022. URL https: //aaxiivoorg//e/.. abs/ / /ttpb///.11212.....11 02365.
Clarkson, K. L. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Trans. Algorithms, 6(4), September 2010. ISSN 1549-6325. doi: 10. 1 145/1824777.1824733 URL https: http///..ooo/////// org/
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gen- erative adversarial networks. International conference on machine learning, pp.
PMLR, 2017.
10. 01.115/1822777.182222
Balles, L., Pedregosa, F., and Roux, L. The geometry of sign gradient descent. arXiv preprint arXiv:2002.08056. 2020.
Conn, R., Gould, N. I., and Toint, P. L. Trust region methods. SIAM, 2000.
Cutkosky, A. and Mehta, H. Momentum improves
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. Spectrally-normaaized margin bounds for neural net- works. In Guyon, L. Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL https://prooeeedngg..o .ww.eeeddddinsn..nui...... neerips. ec cc/paper ce/paper_filesteeeeeeeeeeeeeseee http:////ppapeee//////eeeee
malized SGD. In International conference on machine learning, pp. 2260-2266 PMLR, 2020.
Cutkosky, A. and Orabona. F. Momentum-bass variance reduction in non-convex sgd. Advances in neural infor- mation processing systems, 32, 2019.
Dahl, G. E., Schneider, F., Nado. Z., Agarwal, N., Sastry, C. S., Hennig, P., Medapati, S., Eschenhagen, R., Kasim- beg, P., Suo, D., et al. Benchmarking neural network training algorithms. arXiv preprint arXiv: 2rXiv:2306.071799 2023.
pdf.
Bauschke, H. and Lucet, Y. What is a Fenchel conjugate. Notices of the AMS, 59(():44 46,:444466622 2012.
Angelo, F., Andriushchenko, M., Varre, A., and Flam- marion. N. Why do we need weight decay in modern deep learning? arXiv preprint arXiv:23 10. 04415, 2023.
Bernstein, J. and Newhouse. Modular duality in deep learning. arXiv preprint 12X2v.2410.21265. 2024a.
10

---

Training Deep Learning Models with Norm-Constrained LMOs
Defazio, A., Yang, X. A., Mehta, H., Mishchenko, K., Khaled, A., and Cutkosky, A. The road less scheduled, 2024. URL https://arxias org/ abs/ /20052405 15682.
Jordan, K., Bernstein, J., Rappazzo, B., @fern-
bear.bsky.social, Vlado, B., Jiacheng, Y., Cesista, F, Koszarsky, B., and @Grad62304977. modded- nanogpt: Speedrunning the nanogpt baseline, 2024a. URL https:/ //github Ktt.b..co/eelllerrrrrrddoo lerJordan/ modded-nanogpt.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgra- dient methods for online learning and stochastic opti- mization. Journal of Machine Learning Research, 12 (61):2121-2159, 2011. URL http://mmlrrrrrr html.
Jordan, K., Jin, Y., Boza, V., Jiacheng, Y., Cecista, F., New- house, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https: / l/kellerjordan github http://ospossoooooo...o
El Halabi, M. Learning with structured sparsity: From dis- crete to convex and back. Technical report, EPFL, 2018.
Karpathy, A. nanoGPT. https://githitt com/ karpathy hwn.oPTT 2023. Accessed: 2025-01-25.
Flynn, T. The duality structure gradient descent algorithm: analysis and upplications to neural networks. arXiv preprint 7708. 00523, 2017.
Karras, T., Aittll,, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamies of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Frank, M., Wolfe, P., et al. An algorithm for quadratic pro- gramming. Naval research logistics quarterly, 3(1-2): 95-110, 1956.
Kelner, J. A., Lee, Y. T., Orecchia, L., and Sidford, A. An linear-tmnr algorithm for approximate max flow in undirected graphs, and its multicommodity general- izations. In Proceedings of the twenty-fifth annual ACM- SIAM symposium on Discrete algorithms, pp. 217-226. SIAM, 2014.
Ginsburg, B., Castonguay, P., Hrinchuk, O., Kuchaiev, O., Lavrukhin, V., Leary, R., Li, J., Nguyen, H., Zhang, Y., and Cohen, J. M. Stochastic gradient methods with layer-wise adaptive moments for training of deep works. arXiv preprint arXiv: 11286, 2019.
Gupta, V., Koren, T., and Singer, Y. A unified approach to adaptive regularization in online and stochastic opti- mization. arXiv preprint
Kingma, D. P. Adam: A method for stochastic optimiza- tion. arXiv preprint arXiv:1412.6699 2014.
2017.
Lange, K. MM optimization algorithuss. SIAM, 2016.
Hazan, E. and Kale, Projection-free online learning. 2012.
Large, T., Liu, Y., Huh, M., Bahng, H., Isola, P., and Bern- stein, J. Scalable optimization in the modular norm. arXiv preprint arXiv:2405.148/1 2024.
arXiv preprint
Hazan, E., Levy, K., and Shalev-Shhe Shalev-Shwartz, Beyond con- vexity: Stochastic quasi-convex optimization. Advances in neural information processing systems, 28, 2015.
Li, X.-L. Preconditioned stochastic gradient descent. IEEE transactions on neural networks and learning systems, 29(5):1454-146. 2017.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance ima- genet classification. In Proceedings of the IEEE inter- national conference on computer vision, pp. 1026-1034, 2015.
Liu, J., Su, J., Yao, X., Jiang, 7., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., et al. Muon is scalable for Ilm training, arXiv preprint arXiv:2502. 16982, 2025.
Lu, M., Luo, X., Chen, T., Chen, W., Liu, D., and Wang, Z. Learning mouning-friendie networks via Frank-Wolfe: One-shot, any-sparsity, and no retraining. In Interna- tional Conference Learning Representations, 2022.
Hinton, G., Srivastava, N., and Swersky, K. Neural net- works for machine learning lecture 6a overview of mini- batch gradient descent. Cited on, 14(8):2, 2012.
Jaggi, M. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In International conference on ma- chine learning, pp. 427-435. PMLR, 2013.
McMahan, H. B. and Streeter, M. Adaptive bound opti- mization for online convex optimization, 2010. URL https: /ttss///axiiv.o
Jordan, K.
C0 airbench, 2024.
URL
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial net- works, arXiv preprint 102 2018
tttps:///githe wtwattuubcoooaaee htt.ruu/..omm/eeeeeedddaaa Kedlen cifar10-a a rbench.

---

Training Deep Learning Models with Norm-Constrained LMOs
Mokhtari, A., Hassani, H., and Karbasi, A. Stochastic con- ditional gradient methods: From convex minimization to submodular maximization. Journal of machine learning research, 21(105): 1-99, 2020.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint 2e.::220303666 2022.
Nesterov, Y. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Op- fimization, 22(2):341-362, 2012.
Yang, G., Simon, J. B., and Bernstein, J. A spec- tral condition for feature learning. arXiv preprint arXiv:2310.178111 2023.
Orabona, F. Normalized gradients for all. arXiv preprint (rXiv:2308.06620 2023.
You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint 2017.
Pethick, T., Raman, P., Minorics, L., Hong, M., Sabach, S., and Cevher, V. usam: Memory-efficient sharpness- aware minimization via nuclear norm constraints. Trans- actions Machine Learning Research, 2025.
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsich, C.- J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.0099 2019.
Pokutta, S., Spiegel, C., and Zimmer, M. Deep neu- ral network training with Prank-Wolfe arXiv preprint arXiv:2010.07243, 2020.
Pooladzandi, O. and Li, X.-L. Curvature-informed sgd via general purpose lie-group preconditioners. arXiv preprint arXiv:2402.0455. 042,5353, 2024.
Yu, W., Huang, L., Lin, Q.. Salakhutdinov, R., and Car- bonell, J. Block-normalized gradient method: An em- pirical study for training deep neural network. arXiv preprint arXiv: /ee. 2017.
Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
Yuan, H., Liu, Y., Wu, S., Zhou, X., and Gu, Q. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10453 2024.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Interna- tional conference machine learning, pp. 1889-1897. PMLR. 2015.
Zamani, M. and Glineur, F. Exact convergence rate of the last iterate in subgradient methods. arXiv preprint arXiv:2307.11134 2023.
Zhao, S.-Y., Xie, Y.-.., and Li, W.-. Stochastic normalized gradient descent with momentum for large batch train- ing. arXiv preprint arXiv:2007.13988 2020.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablay- rolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention. In Inter- national conference on machine learning. pp. 10347- 10357. PMLR, 2021.
Vershynin, R. High-dimensions probability: An introduc- fion with applications in data science, volume 47, Cam- bridge university press, 2018.
Vondrák, Optimal approximation for the submodular welfare problem in the value oracle model. In Proceed- ings of the fortieth annual ACM symposiun Theory of computing, pp. 67-74, 2008.
Wright, S. J. Numerical optimization, 2006.
Xie, S. and Li, Z. Implicit bias of AdamW: lo norm con- strained optimization. arXiv preprint arXiv:2404.0445 2024.
Yang, G. and Hu, E. Tensor programs iv: Feature learn- ing in infinite-width neural networks. International Conference on Machine Learning, pp. 11727-11737. PMLR, 2021.
12

---

Training Deep Learning Models with Norm-Constrained LMOs
Appendix
Table of Contents
A Preliminaries
14
A.1 Relationship between steepest descent and uSCG
14
B Method
14
B.1 Input radius scaling
14
B.2 Alternative norm choices
15
B.3 Boundary initialization
16
Proofs for Section (Our Methods)
16
C.1 Notation and detailed problem setting
16
C.2 Hyperparameter transfer
17
D Proofs for Section (Analysis)
20
D.1 Convergence analysis of uSCG
20
D.2 Convergence analysis of SCG
25
D.3 Averaged LMO Directional Descent (ALMOND)
28
D.4 Linear recursive inequalities
30
E
Experiments
31
E.1 Additional experiments
31
E.2
Implementation details
31
E,3 Scaled ReLU
32
E.4
Hyperparameters
32
13

---

Training Deep Learning Models with Norm-Constrained LMOs
A. Preliminaries
A.1. Relationship between steepest descent and uSCG
There are two prominent families of norm-based non-Euclidean method, namely the ones based on the Imo and the ones based on the sharp operator, both of which can be expressed in the terms of the Fenchel conjugate. The Fenchel conjugate of proper, convex, and lower semicontinuous function h X R U {oo} is defined as:
ht h*))) sup ((8,x) h(x))
zEX
where s E X. The subdifferential Oh* equivalent to the argmax of the conjugate operation, i.c.,
Oh (s) = argmax ((s, x) - h(x)).
#EX'
This follows from the Senchel-Young inequality (see e.g. Bauschke Lucet (2012)).
LMO
The Imo is special case when h is an indicator function of convex set, i.c.,
lmo Oh" (-s)
EED
with h(x) (x)
00
otherwise
The Imo is commonly used for constrained minimization in e.g., CG since the operator ensure feasibility on the constrained set D. When : {x || < p}. the Imo satisfies (s, lmo(s)) = - p||s|| which is central to the convergence proof of uSCG (see Lemma D.I).
Sharp operator
Another important example is the sharp operator (Nesterov, 2012; Kelner et al., 2014) defined as s arg max (s, 2) -
sEA
for some norm || I|. which can equivalently be written as
s Oh (s)
with h(x) |
The sharp operator satisfies (s, sF) || 1|
|| 11 (Kelner al., 2014, App. A.1).
The sharp operator and Imo can be defined in terms of each other when D = {or || s p), specifically
||||+ Imo(s)
From (9) we clear distinction between the Imo and the sharp operator, namely that, while the Imo is scale invariant (i.e. Imo(a s) = lmo for > 0) the sharp operator is not (since (a s)F = a(s) for R).
Steepest descent Steepest descent in a normed space can be written in terms of the sharp operator as follows
x b+1
= x [V
with a stepsize > 0. From (9) it becomes apparent that uSCG can be seen as a normalized variant of steepest descent
with momentum.
B. Method
B.1. Input radius scaling
Based on the spectral norn perspective (Yang et al., 2023), which requires that [Wells.. O(vdou//dm) might be inclined to pick the initialization such that Wells = Voou//d is ensured exactly. This argument indeed valid
14

---

Training Deep Learning Models with Norm-Constrained LMOs
Algorithm 3 (Unconstrained) Scion
Input: Horizon n, init. x' (W). .... W)), d0 = O, momentum aę E (0, 11, stepsize E (0, 1), radii Ps E R...
for ...... do
2: Sample Ea P
3: d' a& ffxx, [x) (1 as) a
4: k+1 xtt + 7pE lmo 1 (d))
if unconstrained
3A E [
7) TPc l - (dl)
else
Return "
SCG and uSCG with the layerwise norm choice from (6). For simplicity we ignore biases.
[Walls.
V
|| = max(1 v )
W; - ~M00 )
Layer
Layer
Layer 2
2"
2
2'
2
width
width
width
Figure 4. Coordinate check at initialization. Preactivations are not constant with spectral scaling
V
Laut
di
when dia >
dout
asymptotically, since input dimension is kept fixed. However, when the input dimension is larger than the output dimension, this does not lead to constant preactivations demonstrated through coordinate check (Yang & Hu, 2021) carried out in Figure 4.
Kaiming initialization (He et al., 2015) fortunately circumvents this problem. From random matrix theory we have that ||Al|s Re (Vdn + Vouu) for Aij ~ (O, o²) (Vershynin, 2018). So the Kaiming initialization, [Welis ~ N NN, 1/d..,, leads to ||Wells.. R 1 + /dow/dun which prevents the preactivation from going to zero as dout - 0. Alternatively, one can simply choose [|Wells.. mma(11, V ddos//mm).
Ensuring correct norm scaling is particularly important for SCG and uSCG, since the scaling not only affects initialization but also the update rule itself. Specifically, if the methods were run with the norm bound choice ||Wilss.. S /dowe/d. for the input layer, then the issue in Figure persists, due the Imo always lying on the boundary of the norm ball. The choice |[We = max(1, V doos/d... resolves this issue.
B.2. Alternative norm choices
The primary argument in Section 3.1 for the norm choice is based on the invariance provided by the RMS - RMS operator norm: i.e., the RMS norm of the output layer is bounded, so input of next layer + 1 is also bounded in the RMS norm. Since the Imo of RRM can be computed efficiently, we can directly use this norm choice for our update rule.
However, it is possible to choose another norm such as || |I.-RRM.. as long as the RMS norm guarantee on the output of We is converted into guarantee on the fn-mr of the input of layer f f+ 1. Specifically, we have that || < dmall III through Fact 3.1. Alternatively, we can rely on the invariance provided by || lloc- for which Fact 3.1 tells us that || 1loc I / dull ||RMS- and ||. Iloc-10 S dull ||1-+00- The resulting Imo choices for the three norm choices across all layers are summarized in Table 6.
15

---

Training Deep Learning Models with Norm-Constrained LMOs
Table 6. It is possible use the norm throughout the network if scaled appropriately. By not treating the network a flattened
vector, hyperparameter can transfer across model sizes (cf. Figure 6). We have made use of Fact 3.1 and (5) to derive the correct layerwise scaling. We assume that the image input dimension smaller than the dout the first layer (otherwise see Appendix B.
Weight (bias norm)
Wi (1-bot encoded)
W (image domain)
(Wa)eera.... 1-11
Wr.
bq
Spectral
RMS RMS (RMS)
Imo
Moue UV
v doux deux/ UV
Ty Jaas
ColNorm
1 + RMS (RMS)
Imo
col,(W1) dout col, (W)) co 11
col, Lunut
col (We)
The JuNS
RowNorm
RMS (RMS)
Imo
row; (W1)
WWllE
row;
(W:)
V/hnl now, Well
(We)
1h Inass
Sign
+ oo (Oc)
Imo
sign(W1)
(ign(We)
siggn( (br)
B.3. Boundary initialization
Semi-orthogonal
Following Saxe et al. (2013), perform QR decomposition of a random matrix
Gij 10, Vi,j
G = QR
Use Q' = sign(diag(RR) (R)) the semi-orthogonal matrix the initialization.
Column-wise normalized Gaussian
As proposed in Large et al. (2024), initialize each column as follows
Wiy NN(0 1), Vi
(W)) (W) col, (W)ll2
Vi
Row-wise normalized Gaussian
Initialize each row follows
Wy ~ (0, 1),
Vi,j
(W) = row, (W)
Vj
row; (W)ll2
Random sign
+1 with probability 0.5
Wi -1 with probability 0.5
Vi,j
Each initialization should be scaled by the corresponding scaling of the Imo elementwise.
C. Proofs for Section (Our Methods)
C.1. Notation and detailed problem setting
First, we introduce some notation and detailed problem settings. Consider an L-layer neural network with input dimension do hidden layer widths (d1, dd,...,d,.... dr-1), and output dimension d,. For any input data z E RRd,, the forward pass the network is defined as
p (z) w (z) = (z)
(() = w(0) (z) (o0)==(((())) ((F()())) =e= 2
(10)
(())))) (())))
Here, ( E Rid are the weight matrices of the network, and (r-) is an element-wise activation function. We denote (z) E ddde as the preactivation of the 1-th layer and h(0) E TRde as the corresponding postactivation. The network output (z) is calculated as a linear transformation of the final hidden layer representation.
Below, we provide a brief overview of the specific assumptions for the input data and initialization. Then we explicitly define the spectral norm choice used our analysis:
16

---

Training Deep Learning Models with Norm-Constrained LMOs
Assumption C.1 (Input data). Training samples (z, y) are drawn from distribution P, where E Rdo and y Rdi, We assume has bounded second moments, Le., ||2| < xo.
Assumption C.2 (Initialization schemes). The initialization satisfy the following condition for the stability of the maximal update learning rate (Yang et al., 2023):
|W =0(v 10
de_ )
de-1
Both semi-orthogonal initialization and (with high probability) Gaussian initialization satisfy this condition. Assumption C.3 (Spectral norm choice for lmo). We adopt the layer-wise linear maximization oracles (lmos) from Ta- bles and Specifically, for the first layer W(),, we use
Imo I WWO L) = max 1,
dout V
U ) V()T U
du
where W(1) (U)) is the reduced SVD of W()), For each intermediate layer w( l L}. similarly set
Imo woo L) dant U V
Remank C.4. In all cases, these choices are consistent with the (Spectral + Spectral + Spectral) configuration from Tables and 4. Our results will simultaneously hold for the (ColNorm - Spectral + Spectral) configuration, due to the equivalence under 1-hot encoded (ef. Insight 3.2).
To investigate how these neuron preactivations change after one step of Algorithm I, We analyze the update dynamics of the network parameters under general loss functions L, including but not limited to mean squared error (MSE) and logistic loss.
we consider a batch size setting.
We follow Yang & Hu (2021) which states that a good learning rate enables hidden layer preactivations to undergo the largest possible change in a single update step, while still avoiding divergence when the network width is large.
Let A (0) (z) be the change in the preactivation of the i-th neuron in the 1-th hidden layer after one step of Algorithm on (z, y). The so-called "maximal update" heuristic requires that:
The maximal update learning rate > := the learning rate for which [[ (2)) 1,
with the expectation again taken over the initialization distribution.
C.2. Hyperparameter transfer
To begin with, we prove the following lemma that derives the Imo of the gradient in an L-layer neural network using the spectral norm choice from Table 3.
Lemma C.5 (Spectral Imo for the gradient with respect to W()). Consider an L-layer neural network with input di- mension do, hidden layer widths {di, d2...... ., dL-1}, and output dimension dr, Training samples (z, y) are drawn from some distribution P. where
Rddo and y E Rd,, The network follows the forward pass (10). For convenience, we set f())= (0) 2, making the notation consistent for all layers. The linear maximization oracle (lmo) over the scaled spectral norm ball, D {W[ [[W]|s.. SV di for V Win L(z, denoted as lmo V wie f(z,, Y)), is given by de
(C((xy) df
(1-1)
de
Imo V (w,
de-
dC(zx)
[[(e-1]||2
dflt)
Proof. First, express the gradient with respect to (!!) By applying the chain rule we have
V C(i,)(zzy) = dd(zzy)
df(!) (
h 11-1)²
We immediately have that the Imo given as
17

---

Training Deep Learning Models with Norm-Constrained LMOs
dC(a,y) dF())
h((-1)'
dz
Imo ((e,y))) = V de-1
dC(z.y) dfL()
[|(t22)||2
Now, we are equipped to state and prove Lemma C.6. We follow the proof technique of Yang et al. (2023) in our derivations. Lemma C.6 (Width-invariance the maximal update learning rate). Consider an L-layer MLP with widths do, di,... dt, (where d1 2 do), and assume do a fixed constant. Let its activation function o have Lipschitz constant Lo and satisfy (0) 0 (e.g., ReI.U, Tanh, or GELUU. Suppose:
(i) The input data yz. meets the requirements in Assumption C.I.
(ii) The network is initialized according to Assumption C.2, and
(iit) Parameter updates use Algorithm with the spectral- nortrall-oor--oe nore---ae Imo described in Assumption C.3.
For various loss functions L (e.g. MSE, logistic), define the maximal update condition by
E
(2)) = for all & S I.,
where (z) is the change in the preactivation of the i-th neuron in the E-th hidden layer after one update step. Unless all activations are simultaneously zero during training (which highly unlikely in practice), the optimal learning rate " (under the setting or- in Algorithm 1) satisfying this condition is independent of the hidden-layer widths.
Proof. For Algorithm 1 with the setting 0 1, we have through the spectral norm choice Assumption C.3 and Lemma C.5 that
de-1
de-1
1 (z)
A
Σ
AW (e-1)
inj 11 (z)
Σ
W
Ah (e-
()) (z)
i.j
5=1
5=1
d
Σ
Imo V wio L(z,
h1
(z) + (h
7
j=1
i.j
de-1
(d((ny) dFll
(e- (2) (z) ((-1)((2)
de
= 7 Σ
11
(2) + (w("))h (C!))
V
de-
dC(zyy dPi
h 12)1
j=1
(d(axy)
dg
de(()
|| (1((1) + (W(0Ah(--))
V
de-1
d[(xxy)
diic)
Unless these terms perfect cancel each other out, we have
E de
||
(z)]l1
(2))
1=1
de
dd((zy) dfit)
2
de
E
de
(2)))(2)1
+
(w
Ah(--)1
de-1
dd((z.y) dPi4)
1=1
7 de
(e11)
+E
(e1)
d=-1
2
18

---

Training Deep Learning Models with Norm-Constrained LMOs
For the second term, have
(TAh(()((2)) I 12 ( 12
So
de de-1
E w
Ah
Ah
Ah
A
Recall that, under Assumption C.I, the input z has bounded second moments, ie., ||z|| < co. Consequently, we can treat || do and by as constants, Under these conditions, we have:
de E (1
E B[JAr 2 [Z)
7²
(x) +
de-
1f 10 112
de o
E w Ah
de-1
2 12)1 = o |
E Ah ((-1)
Recall that, by Assumption C.I, the input z has bounded second moments, i.c., || < co. Focusing on the case E = 1, we obtain
22 12
E[|A P (z)
7 711
do
Moreover, by the Assumption C.2 and leveraging the proof from Yang et al. (2023) (specifically, by Eq. (8) at initialization).
we have:
B (2) =((b) Ve E [L - -]. 1].
By induction, unless all activations are simultaneously zero during training (which unlikely in practice), we have:
[|| ( [z) ~ d2de.
By symmetry, we obtain:
[[IA l (6)
(2)
1 [ de
2 e (de),
where independent with the width for every hidden layer.
The maximal update condition ensures that the network operates in stable but maximally adaptive regime, balancing efficient learning and numerical stability.
Remark C.7.
Remark C.8. As the hidden layer widths d,,..., ......,d,,,1 increase, the learning rate required to maintain the maximal update property remains unchanged, demonstrating width invariance in deep networks. Consequently, a learning rate tuned on a smaller model can be directly applied to wider model without sacrificing training dynamics.
19

---

Training Deep Learning Models with Norm-Constrained LMOs
D. Proofs for Section 5 (Analysis)
In this section we present the proofs the main convergence results of the paper as well as some intermediary lemmas that we will make use of along the way. Throughout this section, we adopt the notation:
(stochastic gradient estimator error)
A d
fx²*)
(diameter of 'D in l norm)
D2 == max J.ED
| yl12
(radius of DD in le norm)
max .ED
|
|
(norm equivalence constant)
max =eX
[
(Lipschitz constant of f with respect to 11-112) L2: inf M 0: Va, & Vf(i)l22 S M | v1l2)
We analyze each algorithm separately, although the analysis is effectively unified between the two, modulo constants. This is done in Appendices D.1 and D.2, respectively. Our convergence analysis proceeds in three steps: we begin by establishing template descent inequality for each algorithm via the descent lemma. Next, we analyze the behavior of the second moment of the error [|| X [|]] under different choices for Then, combine these results to derive convergence rate. Finally, note that when analyzing algorithms with constant momentum, will still always take 1 on the first iteration k- = 1.
D.I. Convergence analysis of uSCG
We begin with the analysis of Algorithm 1 by establishing generic template inequality for the dual norm of the gradient at iteration k. This inequality holds regardless of whether the momentum is constant or vanishing, as long as it remains in (0, 1].
Lemma D.1 (uSCG template inequality). Suppose Assumption 5.1 holds. Let n. E N and consider the iterates {x 1n generated by Algorithm with a constant stepsize 7 > O. Then we have
1k=1
El((x*)
1
Lp7
P2
+
+
(11)
pm
2
k=1
Proof. Under Assumption 5.1, the descent lemma for the function f at the points with and wh+1 get, for all k E nL.
f(x*11) {f(x')++++++++ (Vf(z)) 2++1 - -² + & || - x11
= =((a²) + (V/(x*)) - -dtt x4+1 2²) ) + (dtt x - xn) + L ||a -1 - x012
(12)
= =((x²) + +>(f((z**) - -d dd",mmo(d")) 7dd Imo(dd")) + - ||lmod")))|
< 5((a)) + 702 ||**1|, + tdtt lm)(dd)) + Ln2
the final step employing Schwarz, the definition of X and the definition of p2 as the radius of in the |1-1|2 norm. By definition of the dual norm we have, for all u E
|u]|=
(u, v) max(u, Ju0) -(u,, Imm(u))
w: [
MED
which means that, for all l: E n..
radh lm) = 7p(at Immo(1*)) -
20

---

Training Deep Learning Models with Norm-Constrained LMOs
Plugging this expression for 7dt ))mo(d))) into (12) gives, for all & E n).
f(a - f(**)* + |^ - > || + L7² p²
((aa) + |^1 - v5(xa) + V5(2 |||- 1n22 - ((a²) +00 | + 70l11~11 - ~ ~ )l1 + +4,2p p² E f(xx)) + 7 + (p) - + p²
applying the reverse triangle inequality in (a) while (b) stems from the definition of Ç. By rearranging terms and taking expectations, we get
Lp²n² 2
Tp
Vf(* s B[((**) f(a*+1]]] + (0)
Summing this from k = 1 to n and dividing by ypn we get
[[> 1 a
- [(
B[f(=')- - f(
LPY
+
1
P2
+
Σ
Bill 1122
pin
2
n
p
( B[f(x') - --
+
Lip
+
P2
+
E A
pyn
2
P
km 1
(b)
-
Lpy
P2
+
C
B ++ 1
pyn
2
P
1=1
using the definition of 1 for (a) and Jensen's inequality for (b).
At this point, need to determine the growth of the induced error captured by the quantity | To estimate this, we first recursion relating E[]|>*11 and |E[||==110 1|2 adapted from the proof in Mokhtari et al. (2020, Lem. 6) and then prove a bound on the decay of || l| for Algorithm 1.
Lemma D.2 (Linear recursive inequality for IE [|2*113). Suppose Assumptions 5.1 and 5.3 hold. Let n E No and consider the iterates generated by Algorithm I with constant stepsize 7 >> 0. Then, for all k E [I,....n].
0%k - + 2LLPł>²
E 1|| s (1 -
+ az
Proof. The proof is a straightforward adaptation of the arguments laid out in Mokhtari et al. (2020, Lem. 6), which in fact do not depend on convexity nor on the choice of stepsize. Let n E No and k E (I,..., nz, then
11= ||V 1²112
= |/Vf(='') as Vf(=" (x) (1 ax kd
= =I|ax (V/(at) vf(x*,E,) (1 an) (Vi(2*) V)((a=11) (1 (1-ax) (d vf(x
= (\((=) vf(=*, 11)1 + a1 ||*) v
+ (1- - an / /
- d2-2113
+ (1 an) ( -11 1- v.(a ( 551), ()(xx-1))
+ 2ak(1 (1 - an) (V((²) - Vf(a*, (E)) vf(x - db-1)
+ 2(1 -2)) vs(xa-1), vf(x - d
21

---

Training Deep Learning Models with Norm-Constrained LMOs
Taking the expectation conditioned the filtration F& generated by the iterates until k, i.e., the sigma algebra generated by (.1)...,22)). which we denote using [R and using the unbiased property in Assumption 5.3, we get,
E x -11 af V5(**) V/(=", & + (1--z)² ( 111
+ (1- 1 + 2(1 -
- Vf((=* )
From this expression we can estimate,
Ex [x|[|A10
(a) 2 ai0² a1=0
| [1*)
V/(
1112 + (1 (1- ax) 11-111~1110 11-
+ 2(1
az)) V5 - ),
(b)
0 + (1 - =
V V/(0* 1>122 + 2112211)11111111111111 - -0x)))//1
+ (1 - ax)²
at
\ - V - 1012 a4 || ||a)
?
S 0200 + (1- )2)) 13 1² -1 + - (1-aa)) > 1/= (1 axx) (1-0))(((4)355 1|= -0 a/ 11--111))
2 20 + +11 (1- 2 + +(1 +1- 1|
|| + (1 - az) (14) + 2/ 1
2 2ggo - (1+ a1 + (1+ - (1 - ax)) ||
using the bounded variance property from Assumption 5.3 for (a), Young's inequality with parameter on > 0 for (b), the Lipschitz property of f under norm ll2 for (c), the update definition from Algorithm for (d), and the fact that 1 - 0% < 1 for (e). To complete the proof, we note that
(1 2 (1 axk) S (1 )+ and a1-as)(1 +
2
a4
which, applied the previous inequality and taking total expectations, yields
B 1| = (1 (1- a* - az0² 2222372
0
D.1.1. CONSTANT
Lemma D.3. Suppose Assumptions 5.1 and 5.3 hold. Let E No and consider the iterates fa Jk=1 generated by Algorithm I with constant stepsize Y >> 0 and constant momentum a E (0, 1) with the exception of the first iteration, where we take a = 1. Then, we have for all k E [1,...,n}
a m /BI < V2IapEY + (Va+ (VI-
2 o.
Proof. Letn E N ke E (L....... n). and invoke Lemma D.2 to get
El| P1||A=1112 2LLP3>²
B1||^*12 < (1 (1- α
+ a²00
2/
Applying Lemma D.9 with 8 and r 212027
a²0² gives directly
a
E |1 <
223
a02 + (1- a,* & E|
||
a²
233
a (1 a k) 0²
a²
after using Assumption 5.3 in the final inequality. Taking square roots and upper bounding then yields
V2L2p27
(va+ (v1 a #
) a.
1 <
2
22

---

Training Deep Learning Models with Norm-Constrainedd LMOS
Theorem 5.4 (Convergence rate for uSCG with constant o). Suppose Assumptions 5.1 and 5.3 hold. Let n No and consider the tterates {a*) ) 1 generated by Algorithm I with constant stepsize y yn Then, holds that
+ and constant momentum (0,1).
E || 0 tn +o).
Proof. Let n E N we will first invoke Lemma D.1 and then we will estimate the error terms inside using Lemma D.2 under Assumptions 5.1 and 5.3. As shown in Lemma D.I.
B[((=') [*]
Lpp 1 P2 2n
E V(
B B
(13)
pyn
n
p
k=1
By Lemma D.2 with Lemma D.9, get
(v-- k
v |
)
which, if we sum from to n, gives us
V2Tapr + [Iva+ v 1-9
E S n
6.
a
2
Plugging this estimate into Equation (13) gives
-(((a))-19
Lpy
1
P2 +c)
Σ
Bill ** 122
+
Pyn
2
n
p
k=1
Elf(x') 1*1
Lp
1
P2 +5)
V2L2p22 + (nva+
2
(14)
pmm
2
n
P
a
2
[[f(x*)- f+]
Lpy
P2 + c)
V2LapEY
+
(va+ n(1
0
pyn
2
p
a
/ -
Finally, by substituting y = v and noting f(zn+1)= 2 fr we arrive at
Bif(x") ) - f1
Lp 2ym
22 ts)
+ (va+
v - n11 /
g
Vnp
+
+
av
= =0 1
vn + -
1.1.2. VANISHING 0y
Lemma D.4 (Bound on the gradient error with vanishing a). Suppose Assumptions 5.1 and 5.3 hold. Let n No and consider the iterates {=x) generated by Algorithm 1 with constant stepsize 7 satisfying
1
1
3n3 <7
(15)
3//4
Moreover, consider momentum which vanishes ag
1. Then, for all k E (l,...,,.) n2 the following holds
Vk'
402 + 8L33p3
E||l
[11]]
(16)
VT.
23

---

Training Deep Learning Models with Norm-Constrained LMOs
Proof. Let l E (I...., n). then by invoking the recursive inequality obtained in Lemma D.2 for B1I| |+1] we have,
E | + az + 2L2p3r²
- (1
C+k 2
(17)
Using the particular choice y given in the statement of the lemma,
2n3/4 < 33/4
(18)
well the choice of ag and the fact that n 2 k. get
212
A
2
0kk
B1|A* R-1
1 + a +
2
agTi112
0kk
B|| 1M 1-1
1221
+
0200
2L3
2
ark31/2
0²
2T223
B|| k 1
h +
h
k
1
BI||A* -1 112)
+ 2L3P2
2/ k
Then, by applying Lemma D.10 with ut = |1||>* |||ا and c = 0² + +2L3,2 we readily obtain
4a² + 8T222
||||~11 v
(19)
Vk
since Q defined in Lemma D.10 is given by Q max X ||21,40 402 + 8L2P2) < 4o + 8L322, which concludes result.
Combining these results yields our accuracy guarantees for Algorithm with vanishing ag, presented in the next lemma.
Theorem 5.5 (Convergence rate for uSCG with vanishing ax). Suppose that Assumptions 5.1 and 5.3 hold. Let N
1= and consider the iterates { generated by Algorithm / with constant stepsize satisfying 2n374 < 2374 and vanishing momentum on VK. Then, it holds that
B|| Vf O
n:74
1374
Proof. Letn E No, E (1....,,)) .n.. by combining Lemma D.1 and Lemma D.4 we have
|||\|(>>>||]] (D.I) 2 S
2(p2 + (p) Eh
E 1
Lp
1/1
pn
33/4
(D.) 2x -
2(pz + (p) 40 + 8L2p3
Ek=
kT/4
Lp
(20)
1/1
pn
33/4
2x[f(x') - f³1
2(pz + (p) 402 8L2p3
Lp
k= p344
1/4
12/4"
pn
Using the integral test and noting that x + क is decreasing on R,, we can upper bound the sum in the right hand side as
Σ Am
k1/4
S 11+ 1"
2//4
dx 1
3
4 = =1 3
4 (n 1)
3
n
3/4
3
1
< m
3
3/4
24

---

Training Deep Learning Models with Norm-Constrained LMOs
Inserting the above estimation into (20) we arrive at
2F - + 8n (p2 (P) 102 8L3p3
Lp
E Vf
+
pm2/4
3p
n//
2B[f(x') -f1+ {p2 + (p) /p)/440 40 + 8L302
Lp
pm2/4
31311
Lp
1/4 +
n3/4
which is the claimed result.
D.2. Convergence analysis of SCG
In this section we will analyze the worst-case convergence rate of Algorithm 2. To do this, we will prove bounds on the ex- pectation of the so-called Frank- Wolfe gap, m fxx), (xx)), x u), which ensures criticality for the constrained optimization
uED
problem over 2, i.e., for x² E D
0= Vf())* Nn(=")
max max(V/ (V[("),, - u) 50
LED
where ND is the normal cone to the set convex D.
This next lemma characterizes the descent of Algorithm 2 for any stepsize y and momentum ag in (0, 1]. Lemma D.5 (Nonconvex analog Mokhtari al. (2020, Lem. 2)). Suppose Assumption 5.1 holds. Let No and consider the iterates {=m) generated by Algorithm 2 with constant stepsize 7 E (0, Then, for all k {I,..., n., for all
D, it holds
w)) (BI([[(((((])_"""""" [(aht()]] + D2ry E | 2T
(21)
Proof. Let n E N then by Assumption 5.1 we can apply the descent lemma for the function f at the points xt and ++1 to get, for all & E (1,...,n), n)
f(z ) S f(0²) + (Vf(=") 1, - 2²) + $ f11 - 121
= =f(x²) + (d x - x²)) + (2#, xht 11 - 2/ 1 |11 - 11112
f(xt) + +d l 2² (A*: Imo(d") )² + =
Imood $ x* 112
(
I f(x + +1dd" u u )²) + {A* Imo (d x² )+ + ? lmm (dd - x11
= f(a*) + + u -²) + >< + f( u- x² + +>* .lmoddd) - x² + 4 Imo - xx112
f(a + +V((((=),
72 lmo - u) + Imo **12
())
$ f(x²) (VV+++))) 72 Imo - u) 2Lp²?2
using the optimality of (d* for the linear minimization subproblem for (a) and the 2p upper bound on lmmo x² for (b). Rearranging and estimating we find, for all nn). for all D.
(a) 7< (( 2* u) S -f(t) + 7/1 ||2|| lmo(d u|12 & Imo(d*) *11
(b) s((at) - f(x*+1)+ D2y ||*1|2 + 22p22²
where we have used the Cauchy-Schward inequality in (a) and and bounded || [mo(dd)) - 20112 using the diameter of the set D with respect to the Euclidean norm, denoted D2. (b). Taking the expectation of both sides and applying Jensen's inequality we finally arrive, for all is E [1....,n] n). for all u E D.
>BKV * u]] S [B - f(x (000)-((((t))
+ D2> [1 + 2Lp²n²
s Elf( f(aht1)]] + DaTVE 1131 + 2LP²?²
25

---

Training Deep Learning Models with Norm-Constrained LMOs
1. SCG WITH CONSTANT
Lemma D.6. Suppose Assumptions 5.1 and 5.3 hold. Let n E N' and consider the iterates fa generated by Algorithm 2 with constant stepsize ? Vn and constant momentum E (0,1) with the exception of the first iteration. where we take a 1. Then we have
E E1| & 4L2D3 2 a² + 22 (1))(12))) ))
+2at
Proof. Under Assumptions 5.1 and 5.3, Lemma 1 in Mokhtari et al. (2020) yields, after taking expectations, for all k E .1.
B BlIA** ++ 1221 1 (1
a++1
E 1 + o² 2 + +LLD2 ??
2
ak+1
Taking 7 and a to be constant we get
E|IA*+112 11 S (1
a
+ + + +2L3D3
2
Applying Lemma D.9 to the above with w² 1BI1I++111 11),, 3 2.2 and // o²a² + 21
obtain
B|] S 200 + 41222
72
+ (1
a
B 11
a²
2
x² 41223
2a (1
k) o²
2
with the final inequality following by the variance bound in Assumption 5.3.
Theorem 5.6 (Convergence rate for SCG with constant a). Suppose Assumptions 5.1 and 5.3 hold. Let E NN and consider the iterates fx Vh=1 generated by Algorithm 2 with constant stepsize 7 tn and constant momentum (0, 1).
Then, for all u E DD it holds that
B (V -
u}} 0 =)]==
( t0).
Le² Vn
Proof. Let n E N and let l E nI. By Assumption 5.1, we can invoke Lemma D.5 to get, for all & E [l....,n.. for all u E D,
BB x² - 4)1 - B[f(=*) - f(a + Dary B|JA + 2Lp²?²
Since Assumption 5.3 holds, we can then invoke Lemma D.6 and apply this to the above. This gives, for all u e D
72 + (2a+ (1- 2) 02
(V(((t)) - u1l s ((f(t - [(ck+)]]++1p1p + Dayy ALLD3
a²
2
< <[(((*>) - 2T 2I 2 D2y V2a++ (V1- "
h
Summing from h: = 1 to then dividing by nry we find for all E D,
), .n -w)l
BKV [a x -w]
17
(a) [xf((* f(x
1
+ +2Lp²' + 22 + D2
V2a+
m
2
k=1
(22)
(b) B[f(x') - f(xx+1)]
+ +2Lp² + 22 a
D22
<2a
2
7n
11
7
11
10 E[f(x*) - f+1
2Lp² + 2L2 D3 + D2
V2u+
2
0,
7
n /
2
26

---

Training Deep Learning Models with Norm-Constrained LMOs
applying the subadditivity of the square root for (a), geometric series due v - (0, 1) for (b), and the definition of f for (c). Taking 7 = t
t then gives the final result, for all a E D,
B (Vf(""), -u)] w)) (lf()')- 14]
2L vn
2D2D2
=0 =0(4)) Lp² +o)
D2
V2a
vn
ayn
n
Vn
D.2.2. SCG WITH VANISHING O
We now proceed to analyze the convergence of Algorithm with vanishing og. The next lemma provides an estimation on the decay of the second moment of the noise X
Lemma D.7 (Bound on the gradient error with vanishing a Algorithm 2). Suppose Assumptions 5.1 and 5.3 hold. Let n N" and consider the iterates {au]k=1 generated by Algorithm 2 with a constant stepsize 7y satisfying
1
2n3 <7
(23)
33/4"
Moreover, consider vanishing momentum on VR. Then, for all k E n. the following holds
E|||^||21 4o + +8L3D2
(24)
Vk
Proof. Under Assumptions 5.1 and 5.3, have the following recursion from Lemma 1 in Mokhtari al. (2020) after taking expectations, for all & E N
B BIIAA* R+1
Ok+1 < (1
L + + +213D3
2
01++1
Comparing with the bound in Lemma D.4, we see the only difference the change of the constant DŁ by p3. Re- peating the argument in Lemma D.4, the desired claim is directly obtained with Dł in place of p2z, with constant Q [B]] [1g1 40 SL3D3) < 4o + SL3D2 since EL]|^* |2] S o² by Assumption 5.3.
Theorem 5.7 (Convergence rate for SCG with vanishing ag)) Suppose Assumptions 5.1 and 5.3 hold. Let n E N and consider the iterates {a*h Inn Ik=1 generated by Algorithm with a constant stepsize 7 satisfying 2n y < 3374 and vanishing momentum ag VT. Then, for all E D, it holds that
1
E (VV - u)] =0 nJ74 +
Lo² n374
Proof. Let n E N" and k E (I,...,n)
n). By Assumption 5.1, we can invoke Lemma D.5 to get,
EE((S((**, w\] < Els(**) 1(x ((a
D27 D B + 2Lp37²
Applying the estimate given Lemma D.7 to the above we get
402 + +DLDD3
[B(V((**) = E(f(2** - - D27
+2Lp2n2
Vk
= =E[[(=*) - - D2v
402 + D3
+ +L
k1/4
27

---

Training Deep Learning Models with Norm-Constrained LMOs
Summing from & and then dividing by ny find, for all D,
(VV 4)] - PKVS(2 -w]
(a) B[ [[]
D2 D2V 402
+8L3D2
+
+2Lp2r
k1/4
n
(Bf(x'") - f(xn+1)]
4D2 40 8L
-2Lp'
107
3n
BlS( [(xn11)]
4D2v
4a²
DL
+
2Lp²7
ny
3/1
using division by yn for (a) and the integral test with decreasing function
for (b). Using the definition of f* and 2
2/4 estimating ny and y nउ74 gives
[((V(""") -1)- w)) 2E(f(x*) f*]
4D2
2 + 8L2D2
2
+
1/4
3/4
3/4
Lp²
=
1/4
32/4
D.3. Averaged LMO Directional Descent (ALMOND)
In this section we present a variation Algorithm 1 that computes the Imo directly on the stochastic gradient oracle and then does averaging. This is in contrast to how we have presented Algorithm which first does averaging (aka momentum) with the stochastic gradient oracle and then computes the Imo. special case of this algorithm is the Normalized SGD based algorithm of Zhao et al. (2020) when the set with respect to the Euclidean norm. In contrast with Algorithm 1, the method relies on large batches, since the noise not controlled by the momentum parameter a due to the bias introduced by the Imo.
Algorithm Averaged LMO directioNal Descent (ALMOND)
Input: Horizon n, initialization x X, d 0, momentum E (0. 1), stepsize (0,
1: for & == 1,..., do
2: Sample &a P
de amo {k)) (1- a)
4:
xtt x + 7d
5: Choose H" uniformly at random from for (st,...,,"") x")
Return zn
Lemma D.8. Suppose Assumptions 5.1 and 5.3 hold. Let n E No and consider the iterates {ax generated by
_ Algorithm 4 with stepsize 7 Vn. Then, it holds
[[ B[f(x') f
4(1-app
Lp + 2uo =0 + 2po
avn + 2/n
PVn
where! max |=||_ >E |
Proof. Let N and denote n² 1z TI....,
1-0
k k-
with the convention that 20 Φ1 so that 21 D1 and, for all
a
a
a
1 a (7d - 7(1 - oa)d"=1) = lmo
x
Cx
AAternntively, instead of invoking the constant a we could make assumption that the gradient oracle has bounded variance measured the norm I|-
28

---

Training Deep Learning Models with Norm-Constrained LMOs
Applying the descent lemma for I at the points =h1 and ε* gives
1(*h+1) $(*))
= + L || ||
= 1
-s(* + (V]((== Imo(()))) + In2
||lmo{/0)1
L
- -1(*) +7 ((VI(=*) - V/((="), (l - (V -r Immo(g + (0" Immo(g
|
+
2
L
S(= ((V[(**) - V Imo(g (v g", Immo(= p (l) +
||lmo[g x*) }|
2
In²
f(( +2 ((IV)(2)) ff(=")) + / 90 I1.) || P -|19*1 11,)
|Imo{g")]l
Lp²ng
$(((+)++ +> () (1 - v +\ - 1l) - - -pl19* (1.)) +
2
{)(( + (0 (4 || ||||+|/[((**>>>> 11)) - -l9g (l.)+ + Lpēn²
2
(25)
applying Hölder's inequality with norm ||-11_ for (a), the radius p of 'D for (b), and Assumption 5.1 for (c). We note that
x yat y ((1 a)d a lmo(g"") an 11 (1" 7 (1=0a -"=x"=1)
which we can use bound
| k 11 S (1 - a)) || - || + az | || S (1 a) 1|* x011l + apr S apy
(1-a) a))
We then have by using the definition of the update and the Imo, which can be plugged into (25) to get
|| - x11| (1-a) x² - -
(1 (1-a)py
py 11 s $(*tt) f(2ht1) + 7p (I|| - +
- 1-11.) -
= || ( f(x²) -1(
+ L 1112* -1 x111 + |/ - 8/11, + Lpy
2
(26)
f(sh) f(aht))
+ L(1 L(L-a)pp [ - - + Lpy
87
a
2
= ||vf(>")1 10 (f(*h) - f(ght))
L(1 amm +2
Lpy \f(=*)) - 1/1|_ + 2
p7
where (a) is the result of dividing both sides by pr; (b) is the result of bounding | x² ||. and (c) follows by the reverse triangle inequality after adding and subtracting V (*(**) in the norm on the left hand side. Taking expectations, using
Assumption 5.3 and the constant μ max ||=||_ it holds
=Ex |mll
E| fx**) || Vf(x*) -d1|2|| s my EllV)((at)
-*1 s no
which we can sum from 1: = 1 to n to obtain
Vf ||.1 Elf(29) f(2n+1)]
nI(1 a)p
2nuoo + nLpr
P7
a
2
Diving both sides by n and then plugging in 7 = 1 yields the desired final result.
29

---

Training Deep Learning Models with Norm-Constrained LMOs
D.4. Linear recursive inequalities
We now present two elementary lemmas that establish bounds for linear recursive inequalities. These results are essential for analyzing the convergence behavior of our stochastic gradient estimator, particularly when examining the error term |1|
2
Lemma D.9 (Linear recursive inequality with constant coefficients). Let n > 1 and consider {u E R4 a sequence of nonnegative real numbers satisfying, for all (2,....;1
u (1 -3t +
with > 0 and B E (0. Then, for all lo E [2.....n] it holds
27 (1 B3 a s 3
Proof. We prove the claim by induction on k. For the base case k = 2 we find
2 S 3)+² 73 - B
7 + B)u²
since B < 1. Assume now for some k E f2..... tu that the claim holds. Then, by the assumed recursive inequality on
(u:) we have
N+1 < (1- (1+ B3)=²²))))))) +1- = 11-)-))t/ppp +/ + 1 8 + -1===1),, = (1 (1-3))+t)) 1 +
" 3
and thus the desired claim holds by induction.
The first lemma establishes a geometric decay bound for sequences with constant momentum. The following lemma extends this analysis to the case of variable coefficients, which we will use when we analyze Algorithm and Algorithm with vanishing momentum ag--
Lemma D.10 (Linear recursive inequality with vanishing coefficients). Let fur JKEN be a sequence of nonnegative real numbers satisfying, for all k E N*, the following recursive inequality
nh s (1
1
c
2
k
where c >> 0 is constant. Then, the sequence (u* KENN satisfies, for all ls E N
Q
2
k
with Q max 4c).
Proof. We prove the claim by induction. For k = 1 the inequality holds by the definition of Q, since
xt S Q
VT
Let & > and assume that
h-1 Q
Vk
1
Then, by the assumed recursive inequality for th we have
1
u
<
1
k-1
2
(27)
1
Q
<
1
2
Vk
k
30

---

Training Deep Learning Models with Norm-Constrained LMOs
Since k >
can estimate
VE
k
1
Vk-
Vk(k
vk
2(1: - 1)
which, when applied to (27), gives
u $ 1
2vk)
(1+
1
Q +
c
(28)
2(k
1))
vk
k
Furthermore, 1, we also have
(1- 2Vk)
(1
1
1
14
1
2(k
1)
4vk
Applying the above to (28) gives
Q
4V
Vk
k
Q
Q/4
+
Vk
k
Q
VT
with the last inequality following since Q 2 4c. The desired claim is therefore obtained by induction.
E. Experiments
E.1. Additional experiments
MLP We consider a 3-layer MI.P with ReL.U activations to demonstrate the various output layers in Table We consider the configuration (Spectral + Spectral + X) where X is the output layer. Hyperparameters are provided in Table We observe in Figure that the optimal learning rate transfers across model width for all output layer configurations.
Shallow GPT We consider a 3-layer GPT model (Karpathy, 2023) with the same modernizations as for the deep GPT in Section 6. We additionally remove the weight sharing between the first and last layer so that the various input layers from Table 4 can be investigated. We consider SCION and UNCONSTRAINED SCION with the configuration (X + Spectral + Sign) where sweeps over the possible input layer Imos. We additionally consider the variant of UNCONSTRAINED SCION using the configuration (Sign + Sign + Sign), which useful for distributed settings. The hyperparameters can be found in Table 8. We observe in Figure that all configurations exhibit transferability of the optimal stepsize across layer width.
E.2. Implementation details
It is possible implement SCG and uSCG, while only storing on set of parameter and one set of gradients (possibly stored in half- nalf-preciiion). For concreteness, focus SCG, but the reasoning applies to uSCG well. Due to the scale invariance of the Imo, uSCG can be equivalently written as
G² = (1-a) (1=
k++1 = + 7% lmo
By rearranging the update, suffice maintain only two states:
G G Vf (backpropagation)
- 1y Imo
G (1
Implementation wise this approach relies on storing the averaged gradient at the memory location where backpropagation accumulating the gradient. Thus, it is important not to zero out the gradient at any point during training. We provide a reference implementation PyTorch referred as ScionLi ght
31

---

Training Deep Learning Models with Norm-Constrained LMOs
E.3. Scaled ReLU
Large et al. (2024, App. B.2) introduces ScaledR)LU(()) == v2 (x) in order to preserve the variance of the input. Building on this we define ScaledReLU ScaledReLU(z)" 2 ReLU(z)² as heuristic.
E.4. Hyperparameters
For all hyperparameter configuration (Tables 10) we first tune the radius parameters in (6) on small proxy model, similar the input and output scaling factor in jPP (Yang & Hu, 2021). The parameters be tuned with suboptimal stepsize y. The radius P1 refers to the radius of the input layer, the radius pe refers to the radius scaling of the intermediary layers, while Pr. refers to the radius scaling of the last layer in the hyperparameter configuration tables Tables 7 to 11.
All experiments report the loss computed at the last iterate. A linear decay stepsize schedule is employed, which is theoretically motivated by the last iterate guarantee provided in Zamani & Glineur (2023). The linear decay schedule compatible with any algorithm having regret bounds (Defazio et al., 2024), which was established for SCG (Hazan & Kale, 2012).
NanoGPT For NanoGPT we specifically build on the version snapshot at: http:p///h...ncooocaaaaaa/a///aa htcerhaceeeerr222222222666644462
VIT We train a DeiT-base model on ImageNet using the DeiT codebase (Touvron et al., 2021) in order to reach the reported test accuracy of 81.8%. For the AdamW, we retain the optimized hyperparameters reported in the original code, for both SCION and UNCONSTRAINED SCION, we introduce several revisions.
Specifically, inspired by the NanoGPT architecture, we replace the LayerNorm layers DeiT-base with RMSNorm (Root Mean Square Normalization), implemented without learnable parameters. Our empirical results indicate that this modifi- cation significantly improves the performance of SCION, while it does not produce similar improvement for Adam
Furthermore, we disable the learning rate warmup for SCION and increase the batchsize. The complete set of hyperparam- eters can be found in Table ll and the results can be found in Figures and 12.
Scan Spectral Spectal is Spectra))
Seion [Spertral Specrnl Rowllom)
Seioe (Spectal Specinl Sigr)
122
10-1
10.
$12
512
234
2243
2046
CPs
10-
10-4
Inty Lecrnin
Figure 5. Hyperparameter transfer for all three last layer choices on MLP.
Signum
Unrosnttsined Stion (Sign throughout)
Unnoontrssiee Stion (RowNirm Broughhul))
Unconstrenee Scion (CalNom throughoul)
3.0
3.0
With
256
2.6
1004
2.4
204n
2.4
296
512
2.0
2.0
1024
2048
2
Lewerinn
Leaening
Lewering
Leseming
Figure 6. Hyperparameter transfer a 3-yayrr GPT using appropriately rescaled Sign, RowNorm and ColNorm (cf. Table 6).
32

---

Training Deep Learning Models with Norm-Constrained LMOs
VAdamW!
1Slgn Spocial Sirii Unconstralned Solon BignUnnsonsraneed (ColNorm Spectral Bign)
3.00
2.75
512
1024
1.50
2348
Lowering
Leaming nala
Leaering 16
Learring tala
Solon Bign)
Solon (Spectial Speciral Sign)
Spectral Signn
3.00
2.75
2,50
512
1034
2043
Loaming
Loarnin rale
Figure 7. Hyperparameter transfer 3-layer GPT three input layer norms.
Adam Huon
Sdan
5200
Gryss
Figure 8. Validation loss curve for NanoGPT 3B.
32
2353
Figure 9, (left.iddlee) Spectral norm of weight matrices on NanoGPT (124M) throughout one training run. Recall the linear stepsize decay starts at iteration 3650. As expected SCION leads to weights with smaller norms than UNCONSTRAINED SCION. (right) Long run on NanoGPT. The norm control the constrained algorithm SCION appears to particularly important long runs as also observed in Liu et al. (2025) regarding Muon {w weight decay.
33

---

Training Deep Learning Models with
LMOs
40-2
12 seS sir aix 4 da3
374 01-3
$5.0
$4.0
524
522
5L.
203
493
402
12.
400
44
13.77
423
o1e 963
535
525
0.3
062
00.6
273
265
15.3
142
73.0
724
65-4
402
064
665
945
05
GL
033
61.
761
166
163
76.6
265
12.5
01.
66
65
25
353
02.4
02.6
81.0
10
834
7ea
283
762
N67
155
144
765
745
87.
860
554
642
845
845
52.4
50.0
864
18.1
615
81.0
86.5
00.2
STA
856
558
960 360
033
035
115
333
533
002 360
893
085
352
315
07.
065
02.5
10.0
52.0
525
913
953
50.5
560
562
83.
-00.
966
291
998
361
36. 680
074
357
357
50
-227
525
99. 91.
513
514
91.6
907
5633
-2-3033
10L1
52.0
537
925
922
51.
02.0
93.
91.
-50
122
923
06.7 114
513
564
903
13.5
87 -332
333
V02
10.2
536
515
92.
925
52.3
500
10.
93.0
537
105 XM4
51.2
023
933
923
-00
7-005
92.
127
50.7 12.5
524
523
922
11.
-333
536
35.0
15,
53
535
15.6
35.3
53
153
9L1
541 543
143
553
550
957
537
7- 16T
89
912
13.5 21.9
503
523
923
31.9
33
00.
1033
533
5344
93.5
335
55.7
53.6
*
100
160
19 533
94. 960
541
542
935
948
26
15.0
2000
120
100 1801 1010 121
420
140
10:1
188
188 000
10.0
18.4
18:01
100
10.0 11:0
12
03
05
24
(ppocts=3)
509 e 404 LE5
455
447
44.
(37
42
421
557
05.
(3
595
570
554
53
90
70.
TE4
263
13
12.1
731
C
30
262
15.6
33
124
87. 0GA
223 123
70
70
50
53
50
40
30
2ia 921
-22
060
373
854
002
90
922 918 911
933 517 902
-25 10.7 923
927
525 9266 124
52.4
10
100
100
05
Figure 10. The optimal hyperparameters for (UNCONSTRAINED) SCION on the airbench setting with increasing total number epochs (indicated in red). SCION outperforms UNCONSTRAINED SCION, which is not surprising since norm control is important in the setting,
Layer
aediviaaaa
Layer
Layer
Layer
Layer
0.50
600
000
602
076-
602
045
330
2003
1500
2300
3003 2253
1750
Figure 11. mlcft,middle) Spectral norm of weight matrices on CIFARIO, while sweeping over total number of epochs. The spectral norm grows empirically vn for UNCONSTRAINED SCION with fixed stepsize 7, whereas the norm (provably) stays bounded for SCION. (right) The optimal stepsize transfers across width.
Accursoy
Sclan
Figure 12, Test. accuracy curve on the DeiT-hase model. The more stringent norm control SCION is beneficial, similar to what observed for the CIFAR10 experiments.
34

---

Training Deep Learning Models with Norm-Constrained LMOs
Table 7. NanoGPT hyperparameeerss.
Hyperparameter
AdamW
Muon
UNCONSTRAINED SCION
SCION
Layers
12
Head dim
128
Activation function
ReLU
ScaledReL (see Appendix E.3)
Vocabulary size
50304
Dataset
Feb
batch size
512
block size
1024
Iterations π Warmdown
5100
28.5%
if < n m
Stepsize schedule
Constant then linear decay "k
7
if
ž nl
Warmup Gradient clipping
5%
Yes 0.9 /.95
No
Momentum B3
Averaging parameter
0.1
Muon stepsize multiplier'
0.1
Nesterov
Yes
Boundary init.
No
Radius pe / pz.
/50 / 3000
Muon uses Adam for the first and last layer. The stepsize for the intermediary layers is multiplied by constant.
Table 8. Shallow GPT hyperparameters. We increase the batch size 32, which is the maximum allowed for model with embedding size of 4096 on an A100
Hyperparameter
Adam W
UNCONSTRAINED SCION
SCION
Layers
3
Head dim
64
Activation function
22. GELU (scaled to preserve variance)
Vocabulary size
64
Dataset
Shakespeare
batch size
32
block size
1024
Iterations n
122
Stepsize schedule
Linear decay
(1 ke/n)
Gradient clipping
Yes
No
31 / B3
0.9/0.95
Averaging parameter ak
0.1
Boundary init.
Yes
Radius P1 pe PL
1 3 10
35

---

Training Deep Learning Models with Norm-Constrained LMOs
Table Shallow MLP hyperparameters.
Hyperparameter
SCION
Layers
3
Activation function
ReLU
Dataset
CIFARTO (50000 training examples)
batch size
2048
Epochs
20
Stepsize schedule
Linear decay 7A 7 (1 - k/n)
Averaging parameter
0.1
Boundary init.
Yes
Radius P1 / Pe / PL
1 1024
Table 10. Hyperparameters for the CNN experiments building on the airhench codebase (Jordan, 2024). Batch norm parameters use the Euclidean & and shares scaling factor Pe with intermediary layers. further optimized configuration be found in the associated Github repository, where we also conduct speedrun matching Muon (with Frobenious normalization) with SCION. Hyperparameter
UNCONSTRAINED SCION
SCION
Block size (block 1, block 2, block 3)
width factor (64, 256, 256)
Activation function
GELU CIFAR10 (50000 training examples)
Dataset
batch size
2000
Epochs
8
Stepsize schedule
Linear decay 7 = Ty - (1-k/n)
Averaging parameter a
0.5
Boundary init.
Yes
Radius P Pe PL
1/1/20
1/1/100
Table DeiT-base base hyperparameters. SCION and UNCONSTRAINED SCION uses the configuration (Spectral - Spectral + Sign) with foos-norm constraints for the learnable positional embeddings and class tokens, sharing the scaling factor pe of the intermediary layers. SCION
Hyperparameter
AdamW UNCONSTRAINED SCION
Layers
12
Head dim
64
Activation function
GELU LayerNorm
V2. GELU (scaled to preserve variance)
Normalization function
RMSNorm
Sequence Length
197
Dataset
ImageNet-Ik 1k
Stepsize schedule
Linear warmup and then cosine decay
Max Ir
0.001
0.00024
0.0004
Warmup epochs
5
Start warmup Ir
10-6
End Ir
10-5
10-7
Batch size
1024
4096
Epochs
300
200
3, /
0.9 0.999
Averaging parameter o
0.1
Boundary init.
No
Radius p pt / Pt.
25 / 25 500
36


Reviews:
[{"review_id": "m5nFh1XfhD", "reviewer": "Reviewer_WPwo", "summary": "This paper proposes a new approach to optimizing DNNs that leverages the Linear Minimization Oracle (LMO) over norm-constrained sets. The core idea is to adapt the optimizer a priori to the geometry of the neural network, rather than relying solely on on-the-fly adaptive methods like Adam. The authors introduce a new stochastic algorithm, unconstrained Stochastic Conditional Gradient (uSCG), which surprisingly applies the LMO concept to unconstrained problems. They also adapt the existing constrained Stochastic Conditional Gradient (SCG) method for non-convex deep learning. The key to their approach is choosing a norm for the LMO that reflects the structure of the neural network, particularly operator norms between layers. The paper claims that this approach leads to improved performance, better batch size robustness, memory efficiency, and zero-shot hyperparameter transferability across models of different widths. The authors provide theoretical convergence guarantees and empirical results on nanoGPT language modeling and CIFAR-10 image classification tasks. The framework unifies several known optimization algorithms like Normalized SGD, SignSGD, and Muon.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and address the remaining concerns below.\n\n> \"The CIFAR-10 experiments show the transferability of the optimal stepsize, but they do not establish the method's competitiveness in image classification\"\n\nTo establish competitiveness on image classification we conduct a speedrun to achieve the 94% test accuracy target on the [airbench](https://github.com/KellerJordan/cifar10-airbench) codebase.\nScion achieves a 94.08% test accuracy (averaged over 200 runs) in the same wallclock time that the SOTA method Muon uses to achieve the target accuracy.\nThis provides a 13.6% speedup over the tuned baseline of SGD with momentum.\n\nNote that Scion achieves this SOTA performance without the ad-hoc weight normalization otherwise used in the airbench implementation of Muon.\n\n> \"The paper could benefit from a clearer explanation of the intuition behind why uSCG works for unconstrained problems.\"\n\nIt was indeed also surprising to us initially that we could show convergence for uSCG.\n\nAfter applying smoothness in the analysis there are mainly two terms to address: i) the _direction_ of the update (given by an inner product), and ii) the _magnitude_ of the update. \nFor simplicity, let us consider the deterministic case where we have\n\n$f(x^{k+1}) \\leq f(x^k) + \\gamma\\rho \\langle \\nabla f(x^k), \\operatorname{lmo}(\\nabla f(x^k)\\rangle + \\tfrac{\\gamma^2\\rho^2 L}{2}$\n\nThe main insight is the relationship between the LMO over a norm-ball and the definition of the dual norm. Specifically, the value that the LMO attains on the boundary of the norm-ball is exactly the dual norm of the gradient. So the _direction_ of the update (the inner product) is still informative and it in fact decreases the function value by the dual norm of the gradient.\nHowever, the _magnitude_ of the update (the last term) will be large even close to a solution, but this can be taken care of by the stepsize choice (as made precise by Lemma 5.3 and 5.4).\n\nNotice that we do not need to know the Lipschitz constant $L$ to select the stepsize $\\gamma$ as is otherwise the case in e.g. gradient descent. The intuitive reason for this is the scale invariance of the LMO.\nWe will remark on these intuitions in the final version.", "claims_and_evidence": "The claims are generally supported by evidence. However, while the claim of improved performance and hyperparameter transferability is well supported for language modeling, the evidence is weaker for image classification tasks, lacking comparison to other optimizers in this domain.", "methods_and_evaluation_criteria": "The core methodological idea of using norm-constrained LMOs, and particularly the uSCG algorithm, is novel and makes sense for adapting the optimizer to the network structure. The connection to operator norms is theoretically well-motivated. The nanoGPT experiments, including the use of a 3B parameter model, are appropriate for evaluating performance and hyperparameter transfer in language modeling. The CIFAR-10 experiments are insufficient for evaluating the method's general applicability. Comparison to other optimizers should be made on more diverse datasets.", "theoretical_claims": "I didn't thoroughly check the correctness of the proofs.", "experimental_designs_or_analyses": "The nanoGPT experiments are well-designed to test the core claims on language modeling. Varying model width, batch size, and comparing to Adam and Muon provides a strong evaluation. The CIFAR-10 experiments show the transferability of the optimal stepsize, but they do not establish the method's competitiveness in image classification.", "relation_to_broader_scientific_literature": "The paper connects to several areas of the literature, including conditional gradient methods and adaptive optimization.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "The paper could benefit from a clearer explanation of the intuition behind why uSCG works for unconstrained problems.", "questions_for_authors": "Comparison to other optimizers should be made on more diverse datasets", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjMKpTbx30", "reviewer": "Reviewer_c2Ed", "summary": "The authors propose a new stochastic family of algorithms exploiting the so-called Linear Minimization Oracle (LMO) to solve optimization problems. It provides a more general framework and unifies many other algorithms as special cases. Theoretical guarantees as well as significant speed-up on the training of models such as nanoGPT demonstrate the effectiveness of their proposal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback.\n\n> \"To be verified: In Lemma D.1, I think it is the right constant is $L_2$, and not $L$.\"\n\nThe right constant is $L$, since smoothness is taken w.r.t. the arbitrary norm $\\Vert\\cdot\\Vert$.\nThis is what eventually yields the dependency on $L$ seen in the main results of Section 5.\n\nWhat might have caused the confusion is that there is a minor typo in the statement of Lemma D.1, since the guarantee of Lemma D.1 should be stated in terms of $\\Vert\\nabla f(\\bar x^n)\\Vert_*$ instead of $\\Vert\\nabla f(\\bar x^n)\\Vert_2^2$.\nThis is just a typo though, that does not change the final result, since the following Lemma 5.3 is correctly stated in terms of the dual norm.\nWe will correct the typo in the final version.\n\n> \"One particular line of related research (that is quite old) is natural gradient descent [1]. It also exploits the geometry of the function by defining a pseudo-metric in the parameters space by KL-divergence. It is not very popular nowadays since: 1) Calculating the Fischer matrix and its inverse is quite expensive, and 2) The empirical performance in stochastic setting is not very impressive. However, I think it is worth mentioning because it is among the first papers stepping away from the Euclidean metric and proposing a new metric for optimization, to the best of my knowledge. \"\n\nThis is a good point. We will remember to cite and discuss natural gradient descent.\n\n> Geometry of activation functions\n\nIt might be possible to further improve the performance by taking the geometry of activation functions into account, but it seems challenging.\nOne place to maybe look for ideas is the vast literature on initialization.", "claims_and_evidence": "All the claims and experimental evidence are clearly presented and justified. I hope the code is publicly available once the paper is published.", "methods_and_evaluation_criteria": "The proposed framework is pretty interesting and well-presented. The experiments conducted look credible and show certain improvement over existing methods.", "theoretical_claims": "I found all theoretical claims sound. I admit that I did not check the proof very carefully in the Appendix. However, for the parts that I did check, there is not any mathematical error. \n\nTo be verified: In Lemma D.1, I think it is the right constant is $L_2$, and not $L$.", "experimental_designs_or_analyses": "I think the experimental designs are justified and credible.", "relation_to_broader_scientific_literature": "The paper provides a general framework to unify several methods in the literature.", "essential_references_not_discussed": "One particular line of related research (that is quite old) is natural gradient descent [1]. It also exploits the geometry of the function by defining a pseudo-metric in the parameters space by KL-divergence. It is not very popular nowadays since: 1) Calculating the Fischer matrix and its inverse is quite expensive, and 2) The empirical performance in stochastic setting is not very impressive. However, I think it is worth mentioning because it is among the first papers stepping away from the Euclidean metric and proposing a new metric for optimization, to the best of my knowledge. \n\n[1]: S. Amari, \"Natural Gradient Works Efficiently in Learning,\" in Neural Computation, vol. 10, no. 2, pp. 251-276, 15 Feb. 1998, doi: 10.1162/089976698300017746.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have one question: From what I understand, all the LMOs calculated in Tables 2, 3 and 4 do not take into account the non-linear activation function. If that is the case, do you think taking them into account can improve the optimization process? After all, the geometry of the data after the activation is what really matters, and not the pre-activated ones.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lAsrwuTWek", "reviewer": "Reviewer_81Lu", "summary": "The paper introduces the unconstrained Stochastic Conditional Gradient method (uSCG), which builds upon classical Conditional Gradient methods by utilizing linear minimization oracles (LMOs) even for unconstrained optimization problems. This approach leverages the geometry of the problem by assigning different norms for different layers, particularly in language modeling tasks. The authors provide convergence guarantees for uSCG with constant batch sizes and introduce a principled framework for selecting effective constraints based on the input and output space geometries of neural network layers. The paper also discusses the relationship between their method and hyperparameter transfer techniques, such as the Maximal Update Parametrization, which allows for transferring optimal hyperparameters from smaller proxy models to larger ones. The numerical experiments show that the optimal learning rate for the proposed method to minimize validation loss is independent with the model width.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback.\nWe believe there are important misunderstandings that we address below.\n\n> \"The algorithmic descriptions in algorithm 1 and 2 are same. Is this a typo?\"\n\nThis is not a typo: the two algorithms are distinct and solves different problems as detailed in Section 3.\nAlgorithm 1 solves an unconstrained problem, while Algorithm 2 solves a norm-ball constrained problem.\n\nFor deep learning this leads to a distinct behaviour of the norm of the weights between Algorithm 1 and 2 as illustrated in Fig. 8-9, since only Algorithm 2 ensures that the norm does not grow beyond the constraint radius $\\rho$.\nThis explicit norm control can be particularly important for numerical stability and to avoid overfitting, as e.g. seen in the CIFAR10 experiments were Algorithm 2 (Scion) works without the heuristic weight normalization otherwise present in the Muon implementation for this problem.\n\n\n> \"Both algorithm 1 and 2 are simply steepest descent methods with momentum\"\n\nNeither of our two proposed algorithms is steepest descent (Please see Section 4 were we explicitly compare with steepest descent).\nAlgorithm 2 is a Conditional Gradient (aka Frank-Wolfe) based algorithm which applies to constrained problems, while with Algorithm 1 we show that the LMO can even be used for unconstrained problems.\n\nOne important distinction with steepest descent is that the stepsize $\\gamma$ does not have to depend on the Lipschitz constant $L$ (see our main results Lemma 5.3-5.6).\nIntuitively, this is due to the scale invariance of the LMO, which makes the algorithm more stable (the algorithm is less sensitive to the curvature).\n\n\n> \"The experiments on batch size sensitivity does support that the proposed method has better validation loss compares to Muon when batch size is large. Nevertheless, the validation loss curves for all compared algorithms look to be increasing w.r.t. batch size\"\n\nThe loss is increasing with increasing batch size in the experiments because we keep the total number of tokens fixed.\nThe validation loss as a function of batch size is expected to be worse for any optimizer in this setting – the comparison criterion is instead the rate with which performance deteriorates.\nAs the reviewer points out, Scion have better validation accuracy than the baselines for larger batch sizes.\n\nOne way to quantify the validation loss improvement, is to ask how much quicker Scion can achieve the same validation accuracy as one of the baselines. \nTo this end we conduct an additional experiment for the large batch size of 6144, where we simply reduce the number of iterations until Scion matches the larger validation loss of Muon.\nWe find that Scion can achieve the same validation loss as Muon with a 25% smaller wallclock time.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Not applicable.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The theoretical novelty of this paper from section 5 is sound and concrete. \n\nNevertheless, the algorithmic contribution of this paper is unclear. Both algorithm 1 and 2 are simply steepest descent methods with momentum. One minor novelty can be that the authors use different norms for different layers described in section 3.1. Specifically, the authors use Sign in the output layers. According to the numerical experiment, this change does make the optimal learning rate of the proposed method to be independent with the model width but the validation loss improvement is negligible in Figure 1. The experiments on batch size sensitivity does support that the proposed method has better validation loss compares to Muon when batch size is large. Nevertheless, the validation loss curves for all compared algorithms look to be increasing w.r.t. batch size and with the smallest batch size of 512, the difference between proposed method and Muon are not significant.\n\nBesides, the numerical experiments in this paper are restricted to nanoGPT. It would be better to try different transformer architecture to show the applicability of the proposed method in LLM pre-training.", "other_comments_or_suggestions": "The algorithmic descriptions in algorithm 1 and 2 are same. Is this a typo?", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X5f3c4TZAR", "reviewer": "Reviewer_2qhG", "summary": "This paper develops an algorithmic framework that can exploit and appropriate choice of norm for entire neural network with emphasis on hyperparameter transfer across model sizes. The algorithm called uSCG, unconstraint stochastic conditional gradient method, shows improvements both theoretically and practically when the norm-ball constraint matches the natural geometry of the problem.\n\n## update after rebuttal\nMy concerns have been addressed. Good paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy that the reviewer found the paper insightful.\n\n> \"Since you build on top of modded-nanoGPT, I have to ask the question: what is your speedrun result against Muon?\"\n\nThe speedrun configuration is the 124M parameter model with 512 batchsize in Fig. 1.\nIn this setting Scion consistently across 3 runs has slightly lower validation loss than Muon (with the same wallclock time), which would translate into a slightly better wallclock time if the number of iterations is reduced to exactly match the validation loss of Muon.\n\nThe more substantial gain occurs at larger batch sizes.\nWe conduct an additional experiment for the batch size 6144 configuration of Fig. 2, where we simply reduce the number of iterations until Scion matches the larger validation loss of Muon.\nWe find that Scion can achieve the same validation loss as Muon with a 25% smaller wallclock time in this setting.", "claims_and_evidence": "the evidence presented supports the claims.", "methods_and_evaluation_criteria": "The methods and evaluation both make sense.", "theoretical_claims": "The theoretical claims are reasonable though I have not checked the proofs in details.\nI really like the part where different optimizers get organized under the lmo framework and presented in a clean and clear way.\nVery insightful paper!", "experimental_designs_or_analyses": "The experiments are strong enough to support the claims for a mainly theoretical paper.", "relation_to_broader_scientific_literature": "This work is broadly related to series of works that aim to beat AdamW with new optimizer proposal, such as Shampoo, SOAP, Muon. Usually it's under the context of large language model pretraining and finetuning.", "essential_references_not_discussed": "Not that i am aware of.", "other_strengths_and_weaknesses": "The insight and systemic analysis is quite interesting, unifying everything under lmo gives me a new perspective on looking at things.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Since you build on top of modded-nanoGPT, I have to ask the question: what is your speedrun result against Muon?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel stochastic optimization framework, Stochastic Conditional Gradient (SCG) and unconstrained SCG (uSCG), which leverages the Linear Minimization Oracle (LMO) to adapt to problem geometry in deep neural networks. The framework unifies existing methods like Normalized SGD, SignSGD, and Muon. By instantiating SCG/uSCG with operator norms (dubbed Scion/unconstrained Scion), the method achieves hyperparameter transferability across model sizes and competitive performance on tasks such as nanoGPT and CIFAR-10.

While many optimizers have claimed to beat Adam in recent years, but none of them actually did. Recent consensus suggests Muon  may achieve this.    This paper theoretically and empirically demonstrate that Scion/unconstrained Scion outperform Muon and Adam. Moreove, Scion/unconstrained Scion have another appealing character — zero-shot hyperparameter transferability across models of different widths,  which means small proxy models can tune hyperparameters for large models, significantly boosting LLM training efficiency. This positions Scion as a potential default optimizer for DNNs, akin to Adam’s current dominance.

Three reviewers strongly endorse the paper; one raised concerns, but the authors addressed these in the rebuttal. Given the method’s technical novelty, empirical success, and broad applicability, the AC strongly recommends acceptance.
Decision: Accept (spotlight poster)