Muon Optimizes Under Spectral Norm Constraints
Authors: Lizhang Chen
Jonathan Li
qiang liu
Venue: Accepted by TMLR ()

Abstract: The pursuit of faster optimization algorithms remains an active and important research direction in deep learning. Recently, the Muon optimizer has demonstrated promising empirical performance, but its theoretical foundation remains less understood. In this paper, we bridge this gap and provide a theoretical analysis of Muon by placing it within the Lion-$\mathcal{K}$ family of optimizers. Specifically, we show that Muon corresponds to Lion-$\mathcal{K}$ when equipped with the nuclear norm, and we leverage the theoretical results of Lion-$\mathcal{K}$ to establish that Muon (with decoupled weight decay) implicitly solves an optimization problem that enforces a constraint on the spectral norm of weight matrices. This perspective not only demystifies the implicit regularization effects of Muon but also leads to natural generalizations through varying the choice of convex map $\mathcal{K}$, allowing for the exploration of a broader class of implicitly regularized and constrained optimization algorithms.

Published in Transactions on Machine Learning Research (03/2026)
Muon Optimizes Under Spectral Norm Constraints
Lizhang Chen
Izchen Ces.eneessuttaaeee uleras. edu
Deportment of Computer Science University of Texos Austin
Jonathan Li
jbsBes.utezas. utexas, edu
Deportment of Computer Science University of Teros at Austin
Qiang Liu
lqiang@es.
gpiang@os.utezas.. uuteras.a.o. edu
Department of Computer Science University of Teros at Austin
Reviewed OpenReview: https: // openreview. net/ forum? id=Blz4hjntuuf
Abstract
The pursuit of faster optimization algorithms remains an active and important research direction in deep learning. Recently, the MUON optimizer (Jordan et al., 2024) has demon- strated promising empirical performance, but its theoretical foundation remains less under- stood. In this paper, we bridge this gap and provide theoretical analysis of MUON by placing it within the LION-K family of optimizers (Chen et al., 2024). Specifically, we show that MUON corresponds to LION-K when equipped with the nuclear norm, and we leverage the theorctical results of LION-I to establish that MUON (with decoupled weight docay) implicitly solves an optimization problem that enforces a constraint on the spectral norm of weight matrices. This perspective not only demystifies the implicit regularization effects of MUON but also leads to natural generalizations through varying the choice of convex map XC allowing for the exploration of a broader class of implicitly regularized and constrained optimization algorithms.
Introduction
Optimization remains an important research direction in deep learning, where the backpropagation algo- rithm (LeCun, 1989) enables efficient and scalable gradient-based training of neural architoctures. Among gradient-based optimizers, adaptive methods such as ADAGRAD (Duchi al., 2011), ADAM (Kingma & Ba, 2015), and A ADAMW (Toshchilov & Hutter, 2019) have become standard for training large-scale deep neural networks due to their ability to dynamically adjust learning rates based on first- and second-order moment estimates.
Recent advances in optimization algorithms have shown promising potential outperform traditional adap- tive gradient methods in training large-scale neural networks (Peng et al., 2024; Liang et al., 2024; Liu al., 2024b; Jordan et al., 2024; Yuan et al., 2025; Vyas al., 2025; Pooladzandi & Li, 2024; Li, 2022; Pethick et al., 2025; Xie et al, 2024). A noteworthy example is the LION optimizer (Chen et al., 2023), which discovered through symbolic scarch and has demonstrated competitive empirical performance across diverse tasks despite its simple update rule. A theoretical foundation for LION was established through the LION-K framework (Chen et al., 2024), which generalizes LION and unifies powerful optimization techniques such as mirror descent (Krichene et al., 2015; Bock Teboulle, 2003), Nesterov momentum (Sun et al., 2023; Nesterov, 1983), Hamiltonian descent (Maddison al., 2018), Frank-Wolfe algorithms (Pethick et al., 2025; Jaggi, 2013), and decoupled weight decay (Loshchilov & Hutter, 2019; Liu et al., 2025).

---

Published in Transactions on Machine Learning Research (03/2026)
The recently proposed MUON optimizer (Jordan et al., 2024) is another compelling development among emerging optimizers. MUON introduces orthogonalized gradient momentum updates via Newton-Schulz iteration (Bernstein & Newhouse, 2024), demonstrating promising empirical results and potential for efficient large-scale model training (Liu et al., 2025). However, its thcorctical underpinnings and connections to broader optimization techniques remain unclear.
In this paper, we bridge this gap by embedding MUON within the LION-K framework, providing not only theoretical explanation for MUON's empirical success but also unified perspective that enables natural generalizations and directions for future work. Specifically, we make the following key contributions, which are further outlined in Section 3.
Muon as a Lion-I optimizer. We interpret MUON as special case of the LION-K family of optimizers (generalized to matrix-valued parameters) when the convex function K. and its subgradient VX are chosen as the nuclear norm and matrix sign function, respectively. By situating MUON within the broader LION- IC framework, we naturally enable generalizations beyond the muclear norm, introducing a richer family of optimizers with various implicit regularization effects derived from alternative convex functions.
Implicit constrained optimization via Muon. Under standard assumptions, we show that MUON with decoupled weight decay (Loshchilov & Hutter, 2019; Liu et al., 2025) implicitly solves an optimization problem that enforces spectral norm constraints by converging to its set of Karush-Kan Mucher (KKT) points (Theorems 3 to 6). Theorems 5 and 6 notably provide convergence guarantees in the nonconstant, Robbins Monro-style step size regime, which has not been explored in prior work.
Convergence rates of Muon. We introduce the KKT score (Definition 3), a stationarity measure for the aforementioned constrained optimization problem, and provide convergence rates for the KKT score in both the deterministic (Theorem 3) and stochastic gradient (Theorem 4) settings. In particular, Theorem 4 improves upon Theorem B.14 in Chen et al. (2024) by removing the assumption that IC has Lipschitz weak gradient, which does not apply to MUON.
2 Related work
Steepest descent under norm constraints. Bernstein & Newhouse (2024) reinterprets popular opti- mizers such as ADAM (Kingma & Ba, 2015), SHAMPOO (Gupta et al., 2018), and MUON (Jordan et al., 2024) as instances of steepest descent under norm constraints. Similarly, Pethick al. (2025) proposes stochastic conditional gradient approach from norm-constraiit perspective. However, these analyses do not account for momentum, central component in practical implementations of these optimizers. As result, when momentum is introduced, these methods no longer strictly conform to the steepest descent interpretation, highlighting fundamental limitation of this perspective. Moreover, this interpretation does not naturally extend to optimizers that incorporate decoupled weight decay.
Decoupled weight decay. Weight decay is a widely used regularization technique in deep learning, tra- ditionally implemented as an & penalty directly coupled with gradient- based parameter updates (Loshchilov & Hutter, 2019). The concept of decoupled weight docay, introduced by ADAMW (Loshchilov & Hutter, 2019), separates regularization from adaptive gradient computations. Empirical evidence suggests that this decoupling enhances training stability and improves generalization, making it standard practice in modern adaptive optimizers (Chen et al., 2023; Liu et al., 2025). Recently, Xie & Li (2024) demonstrated that ADAMW implicitly solves constrained optimization problem given convergence. Chen et al. (2024) proved that optimizers with bounded updates and decoupled weight decay inherently correspond to constrained optimization formulations, even without requiring convergence assumptions.
Lyapunov analysis of optimizers. Hamiltonian dynamics provides a rigorous theoretical framework for understanding momentum-basod optimization (Nesterov, 1983; Sutskever al., 2013). Unlike standard gradient descent, which ensures a monotonic decrease in the objective function, momentum methods ex- hibit nonmonotonic behavior, requiring more advanced analytical tools for convergence analysis (Jin et al.,
2

---

Published in Transactions on Machine Learning Research (03/2026)
2018). Lyapunov-ased techniques (Shevitz & Paden, 1994; Krichene et al., 2015; Liu et al., 2024a; Chen et al., 2024) have since been developed to analyze the stability and convergence properties of optimization algorithms (Liang et al., 2024).
Concurrent work. Although several concurrent works have studied the convergence of MUON under various smoothness assumptions (Li & Hong, 2025; An et al., 2025; Kovalev, 2025; Shen al., 2025), none of them consider MUON with decoupled weight decay, despite it being the variant that has demonstrated the most promising empirical results (Liu et al., 2025). Sfyraki & Wang (2025) analyzes MUON with decoupled weight decay through Frank-Wii Wolfe perspective, whereas our work is the first to use the LION-K framework and to prove convergence with decreasing step sizes la Robbins Monro.
3
Main results
In this paper, we consider the optimization problem
min F(X) with F(X) = BE [F(X.
(1)
XEX
where X = Raxm is the space of real the expectation is taken over the data distribution D with independent and identically distributed samples ~ 'D. Given realization of the function F(X, {), the stochastic gradient V.F(X,E) is defined as the gradient of F(X,ç) with respect to the variable X.
matrices, F X + R is differentiable loss function, and
The MUON optimizer (Jordan et al., 2024) was recently proposed for solving (1). When equipped with Nesterov momentum and decoupled weight decay (Liu et al., 2025), it has the implicit update rule
M 3,, 11-3991 B.
M = 3,M, - (1 -1
(2)
X = X, Th (M+4) AX++1)
where X, represents the parameters, M, and M, represent momentum, G, is either the deterministic gradient VF(X,) or stochastic gradient V.F(X,, Gi), 7h > 0 is the learning rale, 81 32 E [0,1) are two momentum coefficients, À 2 0 is the weight decay coefficient, and msgn is known as the matrix sign function (see Definition 2).
LION-KC (Chen et al., 2024) is a family of optimizers originally developed as a generalization and theoretical foundation for the LION optimizer (Chen et al., 2023). It parameterized by a convex function K. - with a subgradient VK and has the implicit update rule
M = 32M, - (1-
M 3,M, - (1 -1-B3)
(3)
X = X, + TN (VK(()+)) -AX++1).
The update rule of MUON bears remarkable similarity to (3), and MUON can in fact be identified as the special case of LION-K with K(X) = || and VK(() msgn(X), where ||- denotes the nuclear norm
and msgn known to be a subgradient of II. Recall that | llr = = min((,)))) oX()), where (X( is the 7th largest singular value of X.
Perhaps surprisingly, due to decoupled weight decay, LION-K optimizers do not minimize the original loss function. Instead, they minimize the regularized objective
f(x)= = = x (*X)),
(4)
where 1C* denotes the convex conjugate of Leveraging this property of LION-K, we conclude that MUON is implicitly solving the constrained optimization problem
min F(X) s.t. |X op Xex
(5)
3

---

Published in Transactions on Machine Learning Research (03/2026)
Singular Value Trajectory
F(X)
Lyapunov Functions
||0||=
0.0
3.0
10-1
0.7
6'
0.4
10-1
0.3
1.0
4000
01
Iteration
Iteration
Iteration
Opt.
Init. X. (0ag(0.96, 0.96)
||o|= 1/1. = =.0
Ve
Muon from
Convergence behavior of MUON. Although the primary objective value F(X) exhibits nonmono- tonic fluctuations, the Lyapmuov functions V/ and Ve decrease monotonically within their respective domains VB when the trajectory is outside B, and Vi once the trajectory enters B.
Figure 1:
where the spectral norm I|-llop- defined as | = ( is known to be the dual norm of II-ltrr
Despite the general LION-K framework, MUON's use of the nondifferentiable nuclear norm casts unique challenges in providing convergence guarantees. this work, we provide an analysis tailored to MUON and rigorously establish that the iterates of (2) converge to the set of KKT points of (5).
To give quick overview of the results, we first note that the KKT points of (5) can be characterized by the KKT score function
S(X) == ||VF(X)) + (AX, VF(X))
(6)
We show in Proposition 2 that a point X is KKT point if and only if the KKT score is zero, i.e., S(X) 0, and the primal constraint || S 3 satisfied. We then identify two Lyapunov functions that are used
to verify convergence in terms of these conditions. For the constraint condition [X| S X use the Lyapunov function
Va max (IXIoop (1x100- -
|AX I 1}. Following the which measures the distance from X to the constraint ball IB = XX E X update (2), show that Va(X.) decays exponentially fast when m/ < x i.e.
/-- Vx(X,) < II [[1-moll ((a-nd)) nol) )a(X))
(X())
Hence, VB converges to at a linear rate, which implies that X; rapidly converges to B and never leaves it after entering. Inside the ball, we use a second Lyapunov function
Vx(X, M) F(X) F*
C A (II (AX, ()),
where e is appropriately defined scalar. We show that MUON (approximately) monotonically decreases Vic within the constraint set, which implies that the KKT score vanishes along the trajectory by generalization of LaSalle's invariance principle (LaSalle, 1960) for discrete-time stochastic processes.
Our main results are summarized by Figure and the following theorems.
Theorem (Informal, Theorems 3 and 4). When X,, M,, and M, are updated using (2) with T)t = 1j
e and Var(G.) Wbatch. have that 1 ΣT [[s(X,))=00 ( Auatch
a²
Theorem 2 (Informal, see Theorems 5 and 6). Assume π S Σi so, and E nr < co. Under
cerlain conditions, when X,, M., and M, are updated using (2), we have that X, converges to the set of KKT points of (5) a.s., regardless of initialization.
Precise statements and detailed proofs be found in Section

---

Published in Transactions on Machine Learning Research (03/2026)
4 Preliminaries
= Raxm denote the space of real m matrices, corresponding to weight matrices in neural networks. We demote matrices in capital boldface and vectors in lowercase boldface. We let denote the zero matrix of appropriate dimension. We let (X, Y) == TT(( =>Trx"y) Y) denote the Frobenius inner product. For a differentiable function X. X - [0o0,, we let VK denote the gradient of X. We say KC conver if for all X,Y Y X and À E [0, I].
General notation. We let :
KC(( + S ()K(()) MC(Y).
We say G is a subgradient of K X if for all X,
K(Y) X(X) (G, -X)
If KC convex and differentiable at X, then VK(X) is the unique subgradient of at X. If convex but nondifferentiable, we let OK(()) denote the set of subgradients of K. at X and overload VÆ((X) to denote element of OK(X). For function X, we let K denote the convex conjugate of K,, where
(X) sup (X, Y) - K(Y)).
YEX
From this definition, we immediately deduce the Fenchel- Young inequality
x(X)+ 2 YX.
We let (A) : [X E E K(X)= < oo} denote the effective domain of X. We say : X + I U (oc) is closed the set (X E dom(K) K(X) a} is closed for each a E R. We say IC: + R U (00} is proper if dom((C) is nonempty. The celebrated Fenchel Moreau theorem states that IC + U convex, closed, and proper, then K K. A corollary is that Y aK(() if and only if
K(X) + (X, Y).
T.e. the Fenchel- Fenchel-Young inequality holds with equality, and E OK (Y) (Y* nonempty. We let XB denote the characteristic function of set DD C X, where
X=(X = 0o iXXED
00 otherwise
For 1,22,..,miin(11. we let (X) denote the 1tth largest singular value of X.
Norms. For norm ||:|| on X, and 0, we let (r) = [X x | < r) denote the ball of radius
r. When IB C X and the norm is clear from context, we let d(X, ID) = inf X - Y|l denote the distance from X to D. We let I|. denote the dual norm of |1-1|, where
X Y)
X == sup
|
Y
It follows directly from this definition that (X, Y) < |X| ||Y|| and |X]]., |||] norms
We define the matrix
min(n.m)
|
X P
1XI1 = Σ ox()), |X|| = |X|| |X = i
where p E [1, oc]. 11-1|p is the entrywise E, norm, |l-Il,, is known as the trace norm or nuclear norm, 11.11p is known as the Probenius norm, and Il-Ilop is known as the spectral norm. The dual norm of ||-I1, is I1-Il,g, where
! 1, the dual norm of Ile is I1-llop: and Ie self-dual.
p 4
5

---

Published in Transactions on Machine Learning Research (03/2026)
Fact 1. Let |-|1 be norm X with dual norm 11-11a- ((() |X||, then
if | S
(X) = XB.
olherwise
We say that a function F: X + R is I-smooth if it is differentiable and
|VF - vF(X)| L -XIlp for all Y.
If is L-smooth, then
F()) + (VF(X), X) + - for all X, Y X.
X
For additional background on convex analysis, we refer to Rockafellar (1970).
4.1 Assumptions
Assumption 1. == infxex F(X) is finite, and there evists X X such that F(X*) Ft
Assumption is necessary for (1) to be well-posed. For our discrete-time analysis, we impose an additional smoothness assumption on F.
Assumption (L-smoothness). F + R. L-smooth.
We now define the variance of random matrices and introduce an assumption for the analysis of stochastic settings.
Definition 1 (Variance). The variance of X-valued random variable X is defined as
Var
X E
Assumption 3 (Bounded variance). The stochastic samples & D are independent and identically dis- tributed (i.id.). Additionally, the stochastic gradient VF(X,, Ee) satisfies
E (V)(X, VF(X))) and Var(VF(X....)) Abatch
where o² is a constant and Thaach denotes the batch size. Assumption (Tteration-wise bounded variance). The stochaslic grudient VF((;, salisfies
BE VF - and E.) <
Abatch
We remark that Assumptions 2 and 3 are standard in the literature for the analysis of stochastic optimization algorithms, c.g. Bernstein et al. (2018); Défosscz et al. (2022); Liu et al. (2024a); Yuan et al. (2025).
Since we work within the LION-K framework, our last assumption concerns the choice of K. Assumption 5. K. X - R is conver. This implies that K. is also closed and proper.
5 Background on Lion-K
LION-IC (Chen et al., 2024) is family of optimization algorithms developed to provide a theoretical founda- tion for the LION optimizer, which was originally discovered via symbolic search (Chen et al., 2023). Given convex function K. X R with subgradient VAC, the update rule for LION-I is given by (3). This update rule is equivalent to the original one given by Chen al. (2024), where the last update is
X X, Th (VK(M.))) AX,).

---

Published in Transactions on Machine Learning Research (03/2026)
under the reparameterization 1/2 1+
LION-K can be seen as mixing several fundamental design elements in optimization:
Polyak momentum M, which accumulates the exponential moving average of the gradients, con- trolled by the coefficient $2.
Nesterov momentum M, which introduces extra gradient components into the update, controlled by the coefficient $1 (sec Appendix B.2, Chen et al. (2024)).
Nonlinear preconditioning VK, which applies transformation to the momentum before is used to update the parameters. This is legitimate since VK is monotone map, meaning that (VK(()) VK(Y), X Y) 2 0, which follows from the convexity of K. (see Lemma 1).
Decoupled weight decay XX, which reduces the parameter magnitude in addition to the update VK(M). This introduces regularization effect (see Section 5.1) and is closely related to Frank- Wolfe style algorithms.
5.1 Effect of decoupled weight decay
Due to the interplay of decoupled weight decay and the VK mapping, LION-K. optimizers minimize the regularized objective (4). To gain a quick heuristic understanding of how the regularization term arises, we can simply examine fixed point of the optimizer. Using the original update rule (8), assume that the algorithm reaches fixed point, where have M4+1 M VFX,)) and VK(MM+1) AX, 0. This yields VK(-VF(X.)) - XX; = O. Since VK* is the inverse function of VK by convex conjugacy, we have
vf(x.) VF(X.) + VXX* 0.
This suggests that every fixed point of the algorithm must be a stationary point the regularized objective F.
5.2 Lyapunov function for Lion-XC
The fixed-point analysis alone does not guarantee the convergence of the algorithm. We give a full analysis using a Lyapunov function method, patterned off Chen al. (2024). To understand this, it helps to focus on the limit of small step sizes, where the dynamics of LION-K can be modeled by the ordinary differential equation (ODE)
M, -FF(X) - MM,
(9)
X, = VK. (M, (VF(X,, + M.)) AX,-
Here, the effect of Nesterov momentum is captured by e [0, 1].
It not immediately obvious why the LION-K ODE would serve to minimize F(X), the ODE does not necessarily guarantee a monotonic decrease in F(X). However, we show in Appendix A that the LION-K ODE minimizes the auxiliary function
H(X, = f(x) 1-6 (XX) K(M) (M, AX))
1
d H(,,, S 0 until a in the sense that it is monotonically decreasing along the ODE trajectories, iee de local minimum is achieved. In other words, H(X, M) admits a Lyapunov function of the LION-KC ODE (9). Here, H(X, M) a joint function position and momentum M. It can be interpreted as a Hamiltonian function in a physical metaphor, where F(X) is the potential energy, and the additional term represents a form of kinetic energy.
Morcover, minimizing H(X, M) equivalent to minimizing the regularized objective F(X). This can be seen by the Fenchel-Youn Young incquality, which ensures that K (*XX)) K(M) (M, AX) 2 0, with equality when M OKXX.

---

Published in Transactions on Machine Learning Research (03/2026)
5.3 Constrained optimization problem
I[ KC* lakes on posilive infinile values, LION-K elfectively solves the constrained optimization problem
min F(X) s.t. XX dom(K").
(10)
XEX
If the algorithm is initialized outside the effective domain, Chen et al. (2024) shows that in the continuous- time setting, X, rapidly driven into the effective domain and stays inside afterwards. Specifically, this process is guaranteed to be exponentially fast. time:
d(A,,, dom(IC*) &
dom(K()) for all t t 0.
(11)
Consequently, AX, rapidly converges to dom(XC)) and remains within this domain once it arrives, where the Lyapunov function is finite and decreases monotonically.
6 Muon meets Lion-K
We recall the MUON update rule (2) and formally define the matrix sign function. Definition (Matrix sign). Let UEVT be a singular volue decomposition of X X X. The matrix sign function, denoted msgn, is given by
msgm (X) := - sgm s
where (-)-3 denotes the Moore Penrose inverse of the matrix square root and sgn denotes the entrywise signum function.
The matrix sign of X is also known the Mahalanobis whitening zero-phase component analysis (ZCA) whilening, which stands as the optimal whitening procedure that minimizes the distortion with the original data. It is also closely related to the polar decomposition of X.
We now illustrate the connection between MUON and LION-K using the following well-known fact. Fact (Watson (1992); Candès & Recht (2009)). Let IC(X) = | and X E X with singular value decomposition UEVT. Then
aK(X) {U (U) +W Wew.U' 0, wv 0, Wllop S $1].
In particular, msgn(X (X) (X))
Hence, MCON can be interpreted LION-K with K(X) |X| and VK(X) msgn(X), corrosponding to matrix generalization of LION, which is LION-K with K((X) X||] and VK(X) sgn(X). Indeed, II-Il and Ill are precisely the Schatten 1- and oc-norms, which are the & and for norms on the singular values of matrix, respectively, and the suggestively named msgn function can be seen as a matrix analog of sgn.
6.1 Spectral norm constraint
By (10) and Fact 1, we conclude that MUON solves the constrained optimization problem (5). The bound 1 is determined solely by the weight decay coofficient X. Without weight decay (A = 0), we obtain the original unconstrained optimization problem.
In fact, the bound constraint arises from any update of the form
X = X, + (0, AX,),
where O, has bounded norm, regardless of how it is updated, although the solution may not necessarily minimize the objective within the constrained set. Because O, has bounded norm, the weight decay term dominates the update whenever the constraint not satisfied (i.c. ||XX,|| > 1), leading an exponential decrease in magnitude. This intuition is formalized by the following result, which is a discrete-time variant of (11).
8

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition 1. For any update of the form Xe+1 X, x m AX,) where ||-|| is any norm on X and is constant, we have
with ||O,]| s b, na 1, and A > 0,
b 2
(Ixxll (1x011-2)
X |1
II 1=
Proof. We have
b X4 (0 AX
b
b
X
W (1 m.) || + m
|
A
b
(1 mn:)) -
$ - (1- X neb
Applying this recursively yields the result.
For MUON, we have |lmsgu( llop < 1, so Proposition 1 applies.
Although the continuous- time results in Section 5 provide intuition on the dynamics of LION-K, the non- differentiability of the trace norm ultimately prevents us from dircctly applying these results in cstablishing the theoretical properties of MUON. Instead, we will resort to our discrele-time analysis in Section 7 rigorously prove the convergence and implicit bias of MUON.
6.2 Generalizations of Muon
Generalizing beyond the nuclear norm, can take K. to be a general convex spectral function, i.c.
min(n.m)
X(X)=
Σ
(x),
where o: o0) + R is convex scalar function. Because (x)
u.v. where u, and v, are the singular
8x
vectors associated with subgradient of K. above is given by
VK(()= = Udiag ((Volo))))VT,
where X has singular value decomposition UΣVT.
Assume Vø is upper bounded by b, i.e. sup.20 VΦ(x) = b. Then the update O, = VK(M++1) satisfies |[Olllp < b, which yields constraint of | < by Proposition 1.
In the practical implementation of MUON, & is effectively taken as a high-order polynomial inspired by Newton Schulz ileration for calculating the matrix sign.
Table summarizes several common convex matrix functions along with their key properties. Importantly, the convexity of IC ensures the convergence of the associated LION-K optimizers, as established through both continuous-time and discrete-time analyses (Section 7 and Appendix A). Consequently, our framework introduces large class of provably convergent optimization algorithms parameterized by convex function K.
More generally, the constrained optimization ion problem minxex F(X) such that X E D, where D is a convex set, can solved using LION-IC with XC* (X) = (X). As in the cases of LION and MUON, a particularly important instance when is norm ball Bi.1 (1). In this setting, VKC(X) corresponds to the solution of linear minimization oracle problem maxzex (X, Z) such that Z < Related discussions can be found,
for example, in Bernstein & Newhouse (2024) and Pethick et al. (2025).
9

---

Published in Transactions on Machine Learning Research (03/2026)
Summary of common convex matrix functions. Note that the nuclear norm is special case of the spectral sum with d(.) = I-
Table 1:
Function
Domain
Remarks
Squared Frobenius
|
All matrices
Smooth, strongly convex Lipschitz continuous, convex Lipschitz continuous, convex
Nuclear norm ||X||>
All matrices
Spectral norm |I
All matrices
Quadratic form Tr(X MX)
All matrices, M / O
Smooth quadratic form,
Spectral sum Σ, (X))
All matrices, convex @
General convex spectral functions
7 Convergence analysis of Muon
In this section, generalize the analysis of Chen al. (2024) to handle X-valued updates and leverage this result to provide convergence rates for the KKT score function (6) and prove the convergence of MUÓN to the set of KKT points of (5). Our strategy consists of three components:
In Section 7.1, show that X* is KKT point of (5) if and only if |XX S and the KKT score function (6) vanishes at X
We give a discrete-time analysis of matrix LION-K and show that, as a corollary, the KKT score function is O VT in the deterministic gradient setting (Section 7.2) and O vT Mbbate stochastic gradient setting (Section 7.3).
in the
In Section 7.4, we put together the previous results and conclude that MUON converges to the set of KKT points of (5) using LaSalle's invariance principle (LaSalle, 1960).
7.1 KKT points of spectral problems
We note that (5) is equivalent to
1 for all i (1,2, min(n, m)]
min F(X) s.t. (X) Xex
and define the KKT points this constrained optimization problem.
Definition 3 (KKT points). We say that X* E is a point that satisfies the Karush-Kune -uhnn- Kuhn-Tucker (KKT) conditions, or a KKT point, of (5) if there exists μ E mmin(n.m) such that the following conditions hold:
p.v.v = =0, where u, and v, are singular vectors corresponding (stationarity) VF(X*) + Ei min((,.))) to (X*))
(primal feasibility) o ox(x) S / for all i E (1,2,... min(n,
(dual feasibility) μ 2 0 entrywise.
(complementary slackness) p. (o:(X*) - }) = 0 for all i e 1.22,...,,((((( m)).
At given point X, it is not immediately clear whether the KKT conditions satisfied. To address this challenge, work with equivalent characterization based on the KKT score function (6). We show the direction used in our analysis here and defer the rest of the proof to Appendix B.
Proposition 2. X E X is KKT point of (5) if and only if ||X S and S(X*) 0.
Proof. Suppose |AX* llop < and
S(X') VFF( )1rr + (AX*, VF(X")
- -XX" =
10

---

Published in Transactions on Machine Learning Research (03/2026)
Then XX is subgradient of I1e FX""),, since for all Y X,
| 2 |Y |X (AX*, Y) = 1- Irr + (AX", Y + VF(X"))
Let UEVT be a singular value decomposition of -VF((**). By Fact 2,
= Uggn ( + W, where U W 0, WV 0, and WWlpp S
Then setting i to be the singular values of VF((") in nonincreasing order shows that X satisfies the KKT conditions:
(stationarity) since U T W and WV 0, XX* has singular value decomposition
[n)
ur
Xx-r) diag (1,...,,,,) 1, (W)) (W)..... min((,,)-+((WW
Y1
mm-r/
where rank(VF((*)). In addition, || 1 implies that (X*) ox(X) 1 with corresponding singular vectors ui...., u, and Vi...,, Vr. But by construction, VF(X*) has singular values and singular vectors U1 Uh and Vi.-.., Vm- Thus
min(n.m)
VF(X')+
y.v.V
-M.(-u)v
Σ p.v.V. =0
(primal leasibility) ||AX* 1lap $ by assumption.
(dual feasibility) μ 2 0 entrywise by the nonnegativity of singular values.
(complementary slackness) for E 21, min(n,, m)), if 14, 0, then the condition holds. Other- wise, p4; o. (VF((*)) > 0 implies o (AX') so the condition holds.
7.2 Convergence rate of Lion-K with deterministic gradient
Our analysis in this section is an extension of the discrete-time analysis in Chen el al. (2024), which is turn inspired by the Lyapunov function for contimuous-tin time LION-K dynamics (cf. Appendix A).
Proposition 3. Under Assumptions 1, 2, and 5, let 0 s B1 B2 < and X > 0, and suppose X,, M,, and M, are updated using (3). Let
7
= mA(1 - 31 (1 B2)
31 - -32)
b, = (B2 B1 B1) X(1 - -B1) (1 32))
a +1
(12)
H. F(X,) F [xX,)))) (K0) + (AX,,
r, = VK((H+1) AX.11, - VK (AX++1))
4 = (VI(M).)) - VVC(M..1) M Meti).
11

---

Published in Transactions on Machine Learning Research (03/2026)
Then for all T O,
T-1
Ho
HT
L
Σ u(arTi+b.iA
" -
(13)
T
2T
0
Proof. By smoothness, we have
+ L
- -(X,) S
xx)+
|X411 - X
(14)
By convexity, have
(AX++t)) - -CX)XX) S X -X,)
(15)
X(M++1)- - K((M) & MoHI MM).
Finally, we have
(X+1,,M - M,) = (M, - X,) + (X441, - 1M)+
(16)
Combining (14), (15), and (16) gives
H4+1 H4 S (VxH4, X - XX)) + VMHH,MM41 Mi) + |x Xxel
(17)
where
(LL))
e.M. and VMH4 := - VK(()++1) cxX+11.
X
Recalling (3), we have
1 A2
X X, n, and M M, B2 - 31
M Me+1))
where 6, = VK((H++) - XX Substituting into (17),
He+1 Hi S (VxHi, 8,) + 1- 32
(VNTHee M.+1 - M ) + | - Xell
82 31
2
= (6c, as VA (AX++1) - ((a: + b,)31 - b, + -aa: - -(a; + +b) b133)
betted a ( (KVK((M))) - G CX4,1, Mc+1 Mc+1)
L
-x,1-
C
2
=-m (B. VK (XX(1X++1) + b. M(+1))
(18)
+ bem (VK(M)+)) - AX M.11 - -M+1) + 2 || - |X++1-X,
Xeel?
=-ampl - bere (6t, -MAMMM---M-I----
M--I) - (VIC(Met)) - AX MX411,M4111 Mc+1))
L | - Xxel2
+ 2
-alo' + b(De)) + m L ||0.1
where the second line uses a (at + by 31 b.3g and a, a, and the fourth line uses
M M,11 -3) + VF(X)).
Rearranging (18), summing T iterations, and dividing both sides by T gives the result.
7
12

---

Published in Transactions on Machine Learning Research (03/2026)
Proposition 3 establishes bounds for general LION-K optimizers in the discrete-time setting, and we will use this result to bound the convergence rate of the KKT score function (6).
To avoid tedium in the analysis, we assume that Xo is initialized so that |XXollop S 1. Note that by Proposition 1, this implies ||AX I for all 2 0. We remark that practical implementations of MEON, the weight decay parameter À is known, so Xo can always be chosen to satisfy ||AXollop S 1.
We state several helper lemmas and defer their proofs to Appendix B.
Lemma 1. Let K, X + foo} be a convEz, closed, and proper pair of conjugate functions with subgradients VK and VK* Then for all Y
(VK(X) - 2)20 (VK(X) Y,X - VX) 2 0.
(19)
(20)
Lemma Let (((X)= = |X|| for norm |I-II on X. Then (VI((X), X) I((X).
Lemma 3. In the setting of Proposition 8, let K(X)) | VK(X) = msgn(X), (X), |AXo S 1, ne = and Ck == /min(n,,) m). Then for all t > 0,
V.((X)) + M
2n + B1 - -32)
+ 10188-11V..F0 + Molle-
1- $2
Proposition 4. In the setting of Lemma 3, for all T' 0,
Ho Hr T Σ s(x))
1
+ + 281C% (X)) MollE
4n + B1 - -32)
(21)
1/
(1- B2)T
1-32
Proof. Using the notation of Proposition we have
S(X,) (msgn (msgn(VVF(X.) V)(X,)) AX, V=((,))
msgn MM] XX,, M, msggn (M,) AX,, VF(X,)+ +M,
msgg (FF(X))) msgg((M)) M,
msgn VFF(X,) - msgni M,,,, VF(X.) M.
msgn M.) - XX, M, Insgn (M) - AX,, VI((X.) + M.
-(-mggn -msgn (F((X)) Is M.), VF(X.) + M,
< P + msgn V/ XX, ,X, VF(X,) +.))
< T +
Issgn VF((x)) + IF VF(X.) + M,
2n
+ -
3z)
< T + 2Cx
+ +B11221 |VF(XX) + Molle)
1- 82
+ Cn [1( + B1
(2)
+ 2.21,55-1C VF(X,) MollF
82
where the first line uses Fact 2 and Lemma 2, the fifth line uses
(nsgu(VF(X.)) + msgn (M,), M, = mssgn VF(()))) M. + M !.-
v Iv
M,
M 11
|Iusggu (VF(()))
0.
13

---

Published in Transactions on Machine Learning Research (03/2026)
the seventh line uses
{0} (z E X (Z. Y) = [
if | op < 1 | llop
(Y)
the eighth line uses Mauchy-Schwaace and the ninth line uses Lemma 3. It follows that
(T-1+ Cr C /FF(x)) Molle) 4n 31 -32
s(X)) -
1- 32
T-1
(a,T, + 6,4, + 35 (VF(X.) + Molle) + 4nC L(1 + 31 - (2)
$ T
Σ
1 32
1=
Ho
Hr
n
2
Ingg(MM+) - XX
n/
2T
10
231 Cc
(V) + MolFe 4ncR 31 -82)
(1- B2)T
1- 32
Ho Hr
+2nC CK (V + Molle
4nJC L(1 + 31 32)
n/
(1 B2)T
1-
where the third line uses Proposition 3 and the fourth line uses
32 1
32
t=
1=0
Our convergence rates in the deterministic gradient setting follow directly from the previous results.
Theorem 3. Under Assumptions 1, 2, and 5, let 0 S B1 < B2 < 1, x
1, 0
and
|AXoollo 1 and suppose X,, M,, and M, are updated using (3) with determénistic gradients and that VK has bounded norm. Then
1
VK((M - M, VAX, =
T
VT
VK(M) - V.(M), M, -M.) 0
T
VT
Moreover, when X,, Mr, and M, are updated using (2) with deterministic gradients,
T Σ s(x)
= s(x))
T
VT
Proof. In the setting of Proposition 3, note that 11 = -
and both VK(M) and AX having bounded
VT
norm implics that the right-hand side of (13) is O (+). The claim follows after dividing both sides by n and realizing Fú A 0 by Lemma 1 and a, b, > 0.
Now in the setting of Proposition 4, we have that the fight-hand side of (21) is upon realizing
(1) VT
The claim follows
S(X,) IVF(X,)),, + (AX,, VF(X.)) |VF(X)) - ||Ax VFF(X)) 2 0.
(22)
14

---

Published in Transactions on Machine Learning Research (03/2026)
7.3 Convergence rate of Lion-K with stochastic gradient
We now show results analogous to the ones in Section 7.2 when using (3) with stochastic gradients. The following lemmas will be useful for bounding the noise arising from stochastic gradients.
Lemma 4. Let K(X) |X| for norm II-II on X. Then for all X E X, (VK((())) =0. 0.
Lemma 5. Let X, Y be X-valued random variables satisfying Var(Y) < 00, and let (((X) = ||||| for a norm |I-Il on X. Then there exists a constant Cxc such that
[ Y, VXXX + + VarVar
For MUON where (((X)
|| and VK(XX msgn(X) X), we can let Cx /min(n, m).
Lemma 6. Let X, Y be X-valued random variables satisfying
Var(X) so Var(Y) oo², and - s R.
Then E||X- YI1 S 20 + R.
Lemma 7. Let G, X, and Y be X-valued random variables satisfying E[G X Y. Then
E | + (AX, Y) s B||| Gll,, + (AX, G)).
The following result is a stochastic analog of Theorem 3.
Theorem 4. In the setting of Theorem 3 and under Assumption 3, let K. be norm on X, and suppose instead that X., M,, and M, updated using (3) with stochastic gradicnts. Then
T
=0 =0 (( (K(M)) - AX, (XX()X)))]=0
1 VT
T
E
+
Vbatch
T
(VK(M. V.CMM,, M.) =0 =o)
σ Whatthh
+
E
VT
v
t=1
Moreover, when X1, Mc, and M, are updated using (2) with stochastic gradients,
E(S(X.)) = 0 (1/
VT
/Tbbatch
Proof. Let o; VK(M.+1) Ax Using the notation of Proposition 3, we have
Het1 Ho S nh (5e, at VXX - ((a: + 1,)81 - b, + (a. (a, + b,)31 + b.32)
+ ben (V -XX,+1; M,41 Mc+1) ) + 2 |X,+1- -x11
= =-m (5.. (M+1 VXX( + b, - -M+11) + m (6., VF((,,) G.) -
(23)
+ ben (VK(M.+)) AX M.+1 M(+1) + |x -x,1
= =-m(ar, + 1.11) + nE || + (6,, VF(X,) - G.).
2
It remains to bound Ei(8,, VF(X,) G.)]. Recalling (3),
A X.:
6, 1 VK(M
+ nd
15

---

Published in Transactions on Machine Learning Research (03/2026)
( 1 mTd Ted
VK((HH)) I+mA x,, VF(X.)) - -G.) A
Bi VF(X.)
G.)l E
1
[VVK(M VF(X,) -G.) 1+ + Blix, VF((,) G.))
E
1 + n
1
E - --)))))))) VF((x,) G:)
+ ned
Cxo
ned) Abatch
where the last line uses Lemma with c- B,M,, Y G., t- -(1 Bi); and some constant Cx. Substituting into (23),
(a,ri + b44,) + L (1+ A) mbatch
7ee kxo
B - -H,] < E
|
(24)
2
Taking The 17, rearranging (24), summing over T ilerations, and dividing both sides by n/T yields
Ho
Hr
T T
2 Cxa
Bia, +b
SE
(M43) AX.+1
(25)
nT
2T
F
(1 + n7) Mbatch
To show the result for MUON, we adapt the proof of Proposition 4. We have
E|| + (AX,, Gt)]
< [[ + E
||msgn G.) + XX
G, -M
< [B[ + 2C 20x(( G2 G.-1lpel + BilEll G + Mc-Ille)
20
< <E[T,-]] 2Ck
+ Cz
Mbatch
t-1
Ck Σ
g GR- + 2,3,, OOx Molle]
A=1
2a
< [[[T-1]+ + +2Ck
+ 227CxL
Tbatch
Cx
20
+ 1- B2
+ +211CxLL
- 23, -5 CxBlCC + Moll
/Thatchh
4Cx(1 + +
=E] +
+ nCkl. + 281 CC Molle),
1 132
/Tbattch
16

---

Published in Transactions on Machine Learning Research (03/2026)
where the fourth and seventh lines use Lemma 6. Now, as before,
T
T
E[S(X,)
T Σ E()V(()))l + (AX, VF(X))) - 2
Ell Gell. + (AX, G.)
T
T
1=1
4Cx (1 + 31 - -
2 M
+ n
1 32
Tbatch
[Bla,T, 6, + 23 23 Mollrl)
40x(1 + -1 32)
+ #,xx
1 1-82 832
Mbatch
Ho Hr
Oxa
23 OK + Mollr]
E
+ +2nnRII+ (1+ Abbuch
+
nTT
(1 (1-Ba) B22)T
4Ck + $1
-22)
+
1 82
VMbatch
where the first line uses Lemma and the fifth line uses (25).
7.4 Convergence of Muon to the set of KKT points
In this section, use LaSalle's invariance principle (LaSalle, 1960). Proposition 2, and Theorem show that MUON converges to the set of KKT points of (5).
Definition 4 (w-limit set). Let (X, de stochastic process. A set M is called the w-limit set of the process if it is the minimal closed set salisfying
Pr lim d(X,, M) =0) =1
In other words, M is the smallest closed set such that the trajectories approach M a.s. as t - co. This is equivalent to being the support of the union of all limit measures of (X, J
Lemma 8. Let be a nonnegative random variable such that E[X] = 0. Then
a.8.
Lemma 9 (LaSalle's invariance principle for stochastic dynamical systems). Let {Xihee be stochastic process contained a bounded set a.s., and suppose there exist a nonnegative function V and a nonnegative, lower semicontinuous function such that
((V(X++2) Fil V(X,) S -aph(X+te) a.s.,
where (Fe)een is the natural filtration of (XI)teN l E N, and the nonnegative sequences {atlteN and frehee satisfy
Σ a. x and Σ n < 0c.
1=
t=0
Let M. be the w-limit set of (XI)EeN Then M. contained in the set (X E X ((X) = 0}.
Theorem 5. Under Assumptions 1, 2, and 4, let 0 S 31 < 32 1, X 0, ne S 1, and Ei Th oc, and suppose X,, M,, and M, are updated using (2). Then X, converges to
B X X
|AX| < 1} a.s.
Proof. If X, enters within finite amount time, then it remains there thenceforth by Proposition Now suppose X, does not enter within any finite amount of time. Let
v(X) = max [IXIlop - =.0). h(X) = max - o, = nc,, and 7t = 0.
-
17

---

Published in Transactions on Machine Learning Research (03/2026)
We verify that X2 is bounded a.s., V is nonnegative, h is nonnegative and lower semicontimuous, Σi= ar 00, Et= Y < 00, and
E[V(X++1) X.]-v(Xx =E |x X, - - -
-) -o-h -xph(X;) + To a.s.
by Proposition 1. Thus X, converges {x X ((X) 0} a.s. by Lemma 9. Theorem 6. In the setting of Theorem 5, let |AXollop
1, Ei=o Me < co, and
Ut
00.
Ithatch
1=0
Then X, converges to the set of KKT points of (5) a.s.
Proof. Let
n
=
F
nd(1 31) + (1 -33 ( (AX,M)),
0²
4(X, M, n)
M br
(Ax, M) ()),M),
O+ me and i = 20:2 + 1,
Pbatch
where Ck == /inn(n m). Letting Z, (X,, M,, M., mt) and using the notation of Proposition with
K(()= = ||,,, and VK((X)= msgn(X (ongn(X)nn we verify that Z, is bounded a.s., H. is nonnegative, h is nonnegative and lower semicontimuous, E a2 00, E 7t < 0o, and
E(H(Z(+1) Z,] -()) car b1d)) +7 (z,1(Z(+1) + 7 a.s.
by (24) and arT, + b, 2 r, (Z,41)) Thus by Lemma 9, l(z)) 0 a.s. as + oc, i.c.
(AX,, M, M,
lim 1+0
M2
= a.s.
tr
It follows that, with probability 1,
lim S(X.)
= lim
() (F((X) + M. - MM. + (F(X.) -M.-.. -M.)
M....)) 1r + |M] + (AX,, (F(x,) + -
11000
t0000
< lim
(x(,)) + M,
VF(x,) + M. + +IIAXrllon |F((X)) +M lim 2 / | + M,
lim
tr
[-1
lim
4nc-10RL + Σ
-k- 1
Mh-1 + 2/1 -CC VVF(X) + MolF
++000
1-2
+ 2(1 B1) G
VF + - -2) Σ
3/
gt-k-2 2-2
|G VF(X))lu
km0
0,
where the fourth line uses (31), the fitth line vanishes because of the asymptotically deterministic gradient, and the sixth line uses the Silverman Toeplitz theorem to show that the sums vanish. By Proposition 2, we conclude that X, converges to the set of KKT points of (5) a.s.
Remark 1. The KKT conditions for (5) are necessary for optimality, and under the additional assumption that F is convez, the KKT conditions also become sufficient for optimality. In this case, Theorem 6 shows that the algorithm converges to the set of optimal solutions of (5).
18

---

Published in Transactions on Machine Learning Research (03/2026)
1 < oc condition in Theorem 6 can be achieved by using deterministic
Remark 2. The Ei=o
Rbatch it
gradients or sufficiently increasing batch sizes, e.g. [Rbatckle = (tt) when O)
Remark 3. Theorem 6 does not guarantee converyence to single point; compare with SIGNSGD and Frank- Wolfe, which can fail to converge even in the smooth conter setting (Karimireddy et al., 2019; Bolte et al., 2024).
8 Experiments
In this section, we present several empirical studies to validate our theoretical results. Our experiments focus on verifying the implicit constraints enforeed by MUON, demonstrating its convergent behavior, and com- paring its performance to standard adaptive optimizers such ADAMW across various lasks, architectures, and choices of K.
8.1 Toy example
To clearly illustrate the behavior of MUON, we consider a simplified two-dimensional matrix optimization problem. Figure shows trajectories of singular values under two distinct constraint strengths (A = 1.5 and
À 5.0) with 31 = 32 = 0.95 and = 0.001. We initialize X as a random matrix with singular values (1, The singular values quickly move into and remain within their respective constraint regions, clearly demonstrating the enforcement of spectral norm constraints.
Singular Value Trajectory
Loss Curve f(x)
||||| =
3.5
3.0
0.5
Opt.
Init,
Region 1/A = 0.20
0.6
Region 1/A 0.67
0.4
155
À
0.1
Muon à = 1.5
0.0
0.6
1.0
1000
2000
3000
4000
5200
1000
2000
3000
4000
5000
01
Iteration
Iteration
Figure Trajectories of MUON for the matrix optimization problem minxeR2xe f(X), where f(X) |AX- - BI + || A, e g2x2 and E Rso, evaluated for two different values of X: 1.5 (blue) and 5.0 (green). The colored boxes illustrate the constraint sets induced by |X| S f: the blue box corresponds A 1.5, and the green box corresponds X 5.0. The red dot indicates the optimal solution.
8.2 Constraint verification
We examine the constraint enforcement of MUON using a toy example (Figure 3). In the upper panel, we set A 1.25 and initialize two trajectories within the feasible region (shaded green): one trajectory starts at X diag (0.01, 0.75) (rod) and the other X diag (0.7, 0.7) (bluc). Both trajectorics converge the feasible optimum, although the objective function exhibits nonmonotonic spike near iteration 2000. Despite this, the constructed Lyapunov function H from Proposition 3 decreases monotonically, verifying the theoretical convergence guarantecs. In the lower panel, we increase X to 4 and initialize trajectories outside the feasible region at X = diag (0.1, 0.95) (red curve) and X diag (0.7, 0.9) (blue curve). Here, since the optimal solution is infeasible, the trajectories converge to a feasible projection. Although the objective function plateaus around iteration 500, the Lyapunov function (7) continues to decrease monotonically.
We further validate the implicit constraint enforcement of MUON on standard benchmarks. Figure 4 demon- strates the rapid enforcement of singular value constraints in ResNet-18 trained CIFAR-10. Singular
19

---

Published in Transactions on Machine Learning Research (03/2026)
Trajectory
fix)
[||M]
(MI)
10
6
Opt.
Init. X dlag(0.01 01. 0.75)
Init. X diagto 7.0.)
|0
1/A
Muon from X
Muon from
Singular Value Trajectory
104
maxi 1/A.0)
|011
05
0.
140
182 Iteration
20R
00
280
2200 lteration
2000
1000
3000
120
206
2009
1007
0%
Iteration
Opt
Init. X = diag(o.1. 0.95)
Init. x 7.099
||0
1/A
Muon from x
Muon from X
Figure 3: MUON with À 1.25, initialized within the feasible region (upper panel), and À 4, initialized outside the leasible region (lower panel). The green region denotes the roist region. Both cases illustrate convergence and the monotonic decrease of the Lyapunov function H.
values initially outside the constraint set quickly enter within approximately 400 training steps and remain reliably bounded thereafter.
Iteration
Iteration 50
Iteration 100
iteration 200
Iteration 400
Iteration 2000
30
4.40
20
Hrequecy
20
20
15
2.
10
10
0.5 Singular Values
0.25
0.50
0.75
1.00
0.2
0.4
0.6
0.8
0.2 Singular Values
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Singulaa Values
Singulaar Values
Singulr Vallue
Singular Values
Figure 4: Histograms singular values of the weight matrices from each module of ResNet-18 trained on CIFAR-10 with the MUON optimizer (A= = 2.0). The constraint |W S (indicated by red vertical lines) rapidly enforced, with singular values initially outside the constraint region quickly moving inside within approximately 400 training steps. Once inside, singular values remain consistently bounded by the constraint throughout the remainder of training.
In Figure 5, we extend this verification to larger-scale tasks and architectures, including ImageNet classi- fication and language modeling using ResNet-50, ViT-B/16, Qwen-100M, and LLaMA-300M models. The results consistently confirm that the singular values remain bounded within the thcoretical upper limit (t), indicated by horizontal dashed lines, under different regularization strengths (A = 2.0 and 4.0).
8.3 Implicit spectral regularization in large models
We investigate the implicit spectral regularization induced by the MUON optimizer in comparison to ADAMW. Figure 6 displays the singular value distributions of converged weights for the query (Wq), key (Wk), and value (Wv) matrioes the LLaMA 0.5B model trained by both optimizers. We observe that MUON consistently produces regularized singular value distributions, reflecting the effect of its implicit spectral constraint. In contrast, singular values from ADAMW do not exhibit any spectral regularization.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Qwen 100M
Liama 300M
VIT
ResNet
2-200
1.00
2=
1.0
1.20
0.500
1/4
0.500
0.250
28
0,50
0.00
2/8 0.250
0.80
|
0.02
15
0.70
0.70
0.30
0.50
0.50
0.42
0.00
1/4
0.25
12A
100
licrations
200
200
402
502
Iterations
750
Iterations
4000
802
500
1200
250
400
Iterations
600
200
1050
Figure Verification of the implicit constraint enforced by the MUON optimizer with decoupled weight decay on ImageNet and language modeling tasks, across architectures including ResNet-50, ViT-B/16, Qwen- 100M, and LLMMA-300M. The red and green curves correspond to the choices À 2.0 and À 4.0, respec- tively. The horizontal dashed lines indicate the theoretical upper bounds of the implicit box constraints.
Distribution of singular values of Wa Distribution of singular values of WK
Distribution of singular values of Wy
Muon
A
Muon
Adam
Muon
AdamW
1803
Figure 6: Singular value distributions of converged weights for the query matrix (Wq), key matrix (Wk). and value matrix (Wv) matrices trained with MUON and ADAMW optimizers the LLaMA 0.5B model. From left to right, each subfigure compares singular value distributions obtained by MUON and ADAMW, respectively.
8.4 Generalizations via different convex functions
We explore the flexibility provided by the LION-K framework by varying the convex map K. Figure 7 shows results from applying LION-K with alternative convex maps on the previously defined matrix optimization problem. For instance, selecting the thresholding function
min(n.m)
((X)
Σ
max(lo - e,
induces noft-thresholdig penalty on singular values exceeding threshold e, illustrating how different convex maps yield distinct implicit constraints and penalty structures.
9 Conclusion
In this paper, showed that MUON is an instance of the LION-K optimizer when equipped with the nuclear norm and extended the LION-K analysis of Chen et al. (2024) to accommodate matrix-valued updates in both deterministic and stochastic gradient settings. We tailored our analysis to MUON with decoupled weight decay, demonstrating that it converges to the set of KKT points of a spectral-norms nom-ooostrnnnnnnnieinneee optimization problem. Beyond theoretical guarantocs, we developed a framework for the generalization of MUON and empirically validated our results. Overall, we present MUON as theoretically grounded optimizer for deep learning, with promising directions for future work.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Singular Va Trajectory
(x) + +
|
Opt
x(X) Σ max(
00
[P]] 1.,,
[|l] - =100g
min F(X) + + s.t. [|Aol(X)]lo> 5 1.
03
M/:
Cip Muon 12
3003 336
1000
1308
300
150
CON
Iteration
Iteration
Singular Trajectory
||01|
Opt.
Init.
К(() = (X01l
Hott
10g
F(X)
Muon As 5.0
Muon Ao 1.5
1/2g
Singular ue Trajectory
||lll
Dot
Int.
K(X) [x,(X) p- min T(X) st. [|Aol())lle S1.
[|]]]
104,
[|lle 100
550
Muon Au 1.5
122
10
10:00
2208
07
Iteration
Iteration
Figure 7: Behavior of the LION-K optimizer under different choices of K. Using the same objective function in Figure 2, varying the choice of IC induces distinct implicit constraints and penalty structures the objective function. The colored boxes illustrate the constraint sets induced by K. For example, choosing the thresholding function K(X)= Ei=l values o (X) exceeding threshold Here, ||o, (X) denotes the Schatten p-norm of X.
mmin(a,m) maxx((r;(X)) -e, 0) enforces a soft-thresholding penalty on singular
Limitations. While our theoretical and empirical findings provide substantial insights, several limitations suggest directions for future research. First, although our analysis focuses on key convex maps, such as the muclear norm and clipping functions, extending the LION-K framework to broader class of convex maps may reveal additional implicit regularization behaviors tailored to specific tasks. Second, extending our results to practical training conditions (e.g. general learning rates, nonsmooth objectives) warrants further investigation. Finally, scaling empirical ovaluations to larger models and more diverse tasks would help further validate and refine the practical applicability and robustness the MUON optimizer and its LION-K goneralizations.
References
Kang An, Yuxing Lu, Rui Pan, Shiqian Ma, Donald Goldfarb, and Tong Zhang. Asgo: Adaptive structured gradient optimization. CoRR, abs/2503.20762, 2025.
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett., 3163 2003.
Jeremy Bernstein and Laker Newhouse. Old optimizer, norm: An anthology. CoRR, abs 220922000 2024.
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. SIGNSGD: compressed optimisation for non-convex problems. Tn Proceedings of the 35th International Conference
22

---

Published in Transactions on Machine Learning Research (03/2026)
on Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 559-568, 2018.
Jérôme Bolte, Cyrille W. Combettes, and Edouard Pauwels. The iterates of the frank-wolfe algorithm may not converge. Math. Oper. Res., (0(4):5655225)) 2024.
Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. Found. Comput. Math., (6))7717-772, 2009.
Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves a constrained optimization: As Iyapunov predicts. In The Twelfth International Conference on Learning Representations, ICLR 2024, 2024.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, and Quoe V. Lc. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, 2023.
Alexandre Défossez, Léon Bottou, Francis R. Bach, and Nicolas Usunier. A simple convergence proof of adamn and adagrad. Trans. Mach. Learn. Res., 2022, 2022.
John C. Duchi, Flad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res., 12:2121 2159, 2011.
Vincet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35th International Conference on Mochine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1837-1845. PMLR, 2018.
Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 427 435, 2013.
Chi Jin, Pranceth Netrapalli, and Michael I. Jordan. Accelerated gradient descent escapes saddle points faster than gradient. descent. Conference On Learning Theory, COLT 2018, volume of Proceedings of Machine Learning Research, pp. 1042 1085, PMLR, 2018.
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024.
Sai Pranceth Karimireddy, Quentin Rebjock, Scbastian U. Stich, and Martin Jaggi. Error feodback fixes signsgd and other gradient compression schemes. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Reseorch, pp. 3252-3261. PMLR, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region oplimization. CoRR, abs/2503. 22//2/533112533 12645, 2025.
Walid Krichene, Alexandre M. Bayen, and Peter L. Bartlett. Accelerated mirror descent in continuous and discrete time. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pp. 2845 2853, 2015.
Joseph P. LaSalle. Some extensions of liapunov's second method. IRE Transactions on Circuit Theory, 7 (4):520 527, 1960.
Yann LeCun. Generalization and network design strategics. Connectionism in perspective, 19(18):143-15 155, 1989.
23

---

Published in Transactions on Machine Learning Research (03/2026)
Jiaxiang Li and Mingyi Hong. A note on the convergence of muon and further. CoRR, abs/2502.02900. 2025.
Xilin Li. Black box lie group preconditioners for SGD. CoRR, abs/2211.04422 2022.
Kaizhao Liang, Bo Lin, Lizhang Chen, and Qiang Liu. Memory-efficiett LLM training with online sub- space descent. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, 2024.
Bo Liu, Lemeng Wu, Lizhang Chen, Kaizhao Liang, Jiaxu Zhu, Chen Liang, Raghuraman Krishnamoorthi, and Qiang Liu. Communication efficient distributed training with distributed lion. In Advances in Neural Information Processing Systems 38: Annual Conference on Newral Information Processing Systems 2024, NewrIPS 2024, 2024a.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training, The Twelfth International Conference on Learning Representations, ICLR 2024, 2024b.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. ColR, abs/ abs/2552.16688,, 2025.
Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Chris J. Maddison, Daniel Paulin, Yee whye Tch, Brendan O'Donoghue, and Arnaud Doucet. Hamiltonian descent methods. CoRR, abs/1809.05042. 2018.
Yurii Evgen't 'evich Nesterov. A method for solving the convex programming problem with convergence rate ((1/x²). Dokl. Akad. Nauk SSSR, 269:543- 547, 1983.
Bowen Peng, Jeffrey Quesnelle, and Diederik P. Kingma. Demo: Decoupled momentum optimization. CoRR, abs/2411.19880, 2024.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norn-constraineed Imos. In Forly-second Internalional Con- ference Machine Learning, 1CML 2025, 2025.
Omead Pooladzandi and Xilin Li. Curvature-informed SGD via general purpose lic-group preconditioners. CoRR, abs/ Abs/2442.04553, 2024.
R. Tyrrell Rockafellar. Conver Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970.
Maria-Eleni Sfyraki and Jun-Kun Wang. Lions and muons: Optimization via stochastic frank-wolfe. CoRR, htss/2506.44192. 2025.
Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of muon. CoRR, abs/2505. 237377, 2025.
Daniel W. Shevitz and Brad Paden. Lyapunov stability theory of nonsmooth systems. IEFE Trans. Aulom. Control, 191001110- 1994.
Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of SIGNSGD under weaker assumptions. In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 23077-33099, PMLR, 2023.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Iya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference Machine Learning, ICML 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 1139 1147, 2013.
Nikhil Vyas, Depen Morwani, Rosic Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: improving and stabilizing shampoo using adam for language modeling. The Thirteenth International Conference on Learning Representations, ICLR 2025, 2025.
G. A. Watson. Characterization of the subdifferential of some matrix norms. Linear Algebra and its Appli- cations, 170:33 45, 1992.
Shuo Xie and Zhiyuan Li. Implicit bias of adamw: form constrained optimization. In Forty-first Inter- national Conference on Machine Learning, ICML, 2024, 2024.
Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Trans. Pattern Anal. Mach. Intell., 46(12):9508 9520, 2024.
Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. MARS: unleashing the power of variance reduction for training large models. In Forty-second International Conference on Machine Learning, ICML 2025, 2025.
A Continuous-time analysis of Lion-K.
For completeness, we give straightforward extension of the analysis of Chen et al. (2024) to establish the convergence of the LION-I ODE (9) for matrices.
Theorem 7. Under Assumptions I and 5, let F, K,, and K* be continuously differentiable and
X, VK(M, - c(yM, + +oVF(X))) -X,,
(26)
M, = -VF((X) - M,,
where a, 7,,, and ey S 1. Define
H(X, = ((F(X) F²) (IX) + ((O)) + 1 1- (XX (M, AX)).
(27)
1+ e
Then for all 1, H(X,, M.) 2 and के H X M.) 0, i.e. H Lyapunov function for (26).
Proof. For simplicity, we drop the index t. By assumption, F(X) 0, by the Fenchel Fenchel-Young inequality, (AX) + K(M) (M, AX) 0, and by definition,
X x*x (( sup ((X Y) XC(Y)) KC()) 2 (AX, 0) - (C + (C(O) 0.
YeX
Combining those incqualities shows that H(X,M) 2 0.
Let M
= M (..M+VVF((X By Lemma 1, we have
0 M VXX) VK(M) -Xx
Ea (FFF(( (1 - Ey M+ VA (AX), VI(M XX)
(28)
02 M M. VK((M - VK(M))
(FOVF(X) + +67,, + (VK(M) - AX) (VX(() - -Xx)
25

---

Published in Transactions on Machine Learning Research (03/2026)
By straightforward computation,
d
dt ([(X = (VxH(X M), X) (VMH(X,M)
(VF(( +XVI + 1-67 (AVIC* AM), VK(M) AX
1tel
1-67 (VK(() XX, -VV[(x) - M
1+
X+y
-1 -11- (1-e7) M (AX( VK(MM XX)
I+
+ 1-
(caVF(x)
+ + (VC(M) AX) - (VIC(M) - 1X))
c(1+e))
where the last line uses (28).
We recover (9) by setting a = = in (26). Although an important result, Theorem cannot be directly applied to MCON due to the nondifferentiability of the muclear norm.
Deferred proofs
Proposition 2. X E X is KKT point of (5) if and only if ||XX* op S and S(X*) = 0.
Proof. Suppose the KKT conditions for the original problem (5) are satisfied, i.e. there exist, E R2o and subgradient G of Il-llop at X', where |XX* llop I 1, such that
VF(X*) + P 0 and a(IIX* 1) 1)=
|XX* I is satisfied by primal fcasibility. If p 0, then VF((*) = 0, which implics S(X*) = 0. Otherwise, we have |I by complementary slackness. Let the multiplicity a (X+) be 4, with correspouding singular vectors U₁ and V1. Using Watson (1992)'s characterization of the subgradients of Il: have G U,HVI, where H gixt and Tr(H) 1. Thus ||G||, 1, and
S(X*) = |VF lee + (AX*, VF(X*)) = G (AX" G) = μ |G - / ||X* llop = 0,
where the third equality uses Lemma. 2.
Lemma 1. Let K, x RU (oo} be convex, closed, and proper pair of conjugate functions with subgradients VIC and VK* Then for all X, Y.Y. E
(VK(X) - VX (VK(X) - XXX-
(19)
(20)
Proof. By definition of subgradients, we have
K(Y) K(X) 2 (VX(X). Y-X X)
K(X) - KC(Y) 2 (V Y).
Summing the incqualitics gives 0 > (VI(Y) VK(X), - -Y), which shows (19). (20) follows by setting + (Y) in (19) and using Y E (K())
Lemma 2. Let K(X)= |X| for norm ||:||| on X. Then (VK((), X) K(X).
26

---

Published in Transactions on Machine Learning Research (03/2026)
Proof. By definition of subgradient and properties of norms,
0 <(()) 2 (X) + (VX(X)) x)= =X()) (VX()), 2K(X) ((X) 2 XX( + (VX((X), 2X -X) XX() (V)(X).).
Combining the inequalities shows that (VK(X)) X) (X).
Lemma 3. In the setting of Proposition 3, let (X) = | VK(X) = msgn(X), ||Xo I 1, 7)c = and Ck = Vmin(n, m). Then for all > 0,
VF(X))+ M -
LmCxL + B1 -32)
+ 31 + |VF((o)) + Molla
1- -B
Proof. By Proposition 1, we have |AXellp S 1 for t 2 0. Furthermore, for all > 0,
[F((X)) VF(X--I)ll= S L| X,-1Ilp 1/- Imsgg(M.) - 1x111 ||
(29)
S 5n-1CxL misgn(M.) - >x.|1 S 22
where the first line smoothness. Recalling (3),
VF))) Mellp |VF((, + 32M, (1 F))
(F((,) VF(X,-1) + &V M,-1)
*(FF((*) VF(X. + (5(( Mo)
(30)
3 - |VF(Xx) - V 1)) + (FX()))
(V + Molle
k=1
$ 2C L
8/2 k nh 7 3 | VF(X.) + MollF
k=1
where the third line iterates and expands the first two lines, the fourth line uses the triangle incquality, and the fifth line uses (29). Thus
VF(X.) M,
[VF(X) + B,M (1-))=-XXXXXXX
V VF + + Ma-1)llp
< {(V((X)) VF( + 31 [F((x-)) M.-IllF
(31)
22n-1OLL 31 20xL Σ
8 k 7 + 5/ |VF(Xx) Mollp
k=
[-1
= 22n-1CKL+23,00
3 Σ
8h-k-1
M7-
1 B18 V + MoolF
k=1
where the third line uses the triangle inequality and the fourth line uses (29) and (30). The result follows upon setting The and using E 3
gt-k-1
< 20=0
3/
I-
Lemma 4. Let (C(X) = | for norm ||-|| on X. Then for all X X, (VX(((() = 0.
Proof. By definition of subgradient and Lemma 2,
K(Y) + Y X) (VK(X),
27

---

Published in Transactions on Machine Learning Research (03/2026)
for all Y E X, which implics that
(VK(X), Y)
E [VX(()) = sup
K(Y) S
Y
by definition of the dual norm. We conclude that IC (VX(((( 0 by Fact 1.
Lemma 5. Let X, Y be X- Kavaleed rondom variables satisfying Var(Y) < co, and let K(X) |X|| for norm |l-|l on X. Then there erists constant Cx. such that
E| -Y, VKXX ++X)) CKvVarr()
Proof. By the Fenchel Young inequality and Lemma
Y Y. VK(( +)) s <(E * - Y).
By the equivalence of norms
vector spaces, there exists constant Ck such that
((()== & Cx |X Taking expectations,
E ( - Y, VK(X XYM) [KC - Y) < CcE BY - Y
S cCy E
| X11i
Cx CK Var(Y),
where the second line uses Jensen's 's inequality.
Lemma 6. Let X, Y be X-valued random variables satisfying
Var(X) S 0 Var(Y) So?, and
S R.
Then E| X Y < 20 + R.
Proof. We have
E
X 1 S B - X
+
[X
Y 11 + Y - E
E X
E
R
B | - E
s 2a R.
where the first line uses the triangle inequality and the second line uses Jensen's inequality. Lemma 7. Let G, X, and Y be X-valued random variables satisfying E X YY Then
E || + (AX, Y) < ElI + (AX, G)).
Proof. Trivial by Jensen's.
Lemma 8. Let be a nonnegative random variable such that E[X] = 0. Then X = 0 a.s.
Proof. By the layer cake representation,
/ Pr(X t)dt,
-= -[]
Pr(X 1) 0 for (Lebesgue) almost every t >> 0. Since Pr(X f) is right-continuous function of t, have that Pr(X > t) = 0 for all > 0. The conclusion follows from
Pr(X > 0) lim Pr(X > t) = 0.
ry0
28

---

Published in Transactions on Machine Learning Research (03/2026)
Lemma 9 (LaSalle's invariance principle for stochastic dynamical systems). Let {Xi)eeN be stochastic process conlained in bounded set a.8., and suppose there exist nonnegalive function V and a nonnegalive, lower semicontinuous function h such that
([V(X++1) Fil - V(X.) S -ath(Xe+e) T a.s.,
where (Fe) is the natural filtalion of (XI)IEN, l E N, and the nonnegative sequences {arLieN and (re)IeN satisfy
Σ
a
and En
1=0
t=0
Let M. be the w-limit set of (Xthee. Then M is contained in the set (X X h(X) = 0}.
Proof. Define the auxiliary function
V. = V(X.) - ge.
where 9r =
8=0
Then, we have
Diti Fi] - -[E[((X+++) |
- - - 5 -aih(Xe-1)) S a.s.
(32)
By (32), the nonnegativity of V, and limy-so 9: 0o, it follows that (Vihee is supermartingale with sUPteR E [(x.)=] < co. By Doob's supermartingale convergence theorem, conclude that (M. and hence (V(X) converges almost surely to random variable with finite expectation.
Now, taking expectations of (32) and summing from = 0 to oc, we obtain
Elarh(X(4e) a4 [((())] cc.
Since Exe a4 = oc, this implies lim in (X,( 0. By Fatou's lemma, we have
0 < E
lim inf h (X,)) 6-000
< inf B[4(X.) = 0
20000
and hence
lim inf 1( = a.s. 11000
and the (almost sure) boundedness of X.. the w-limit set M of {Xih must satisfy (X) = 0 for X E M, i.e.
by Lemma 8. By the lower semicontinuity and nonnogativity
M
X E X
h = 0}.
29


Reviews:
[{"review_id": "gSMhrahLxN", "reviewer": "Reviewer_6Xmk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the connection between two optimizers, Muon and Lion-$\\mathcal{K}$. In particular, Lion-$\\mathcal{K}$ uses a convex function $\\mathcal{K}$ to perform preconditioning by mapping the momentum $\\widetilde{M}_t$ to $\\nabla \\mathcal{K}(\\widetilde{M}_t)$. When $\\mathcal{K}$ is chosen to be the spectral norm, this preconditioning mapping $\\nabla \\mathcal{K}$ reduces exactly to matrix sign. Therefore, Muon can be viewed as Lion-$\\mathcal{K}$ with function $\\mathcal{K}$ being the spectral norm. \n\nIn addition, this paper shows that a variant of Muon with decoupled weight decaying can find an KKT solution of the spectral norm constrained optimization problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper clearly explains the relationship between Muon and Lion-K. The established convergence results consist of two phases: Phase 1, convergence to feasibility, and Phase 2, convergence to stationarity. These convergence results exhibit a pattern similar to \"Lion Secretly Solves Constrained Optimization\".", "audience_yn": "Yes", "audience_explanation": "This work treats Muon as a special instance of Lion-$\\mathcal{K}$ and analyzes it using a theoretical framework for Muon (\"Lion Secretly Solves Constrained Optimization\"), yielding a practical version of Muon with convergence guarantees. Furthermore, it demonstrates that Muon with weight decay implicitly enforces a spectral-norm constraint, which is of interest to researchers studying Muon.", "requested_changes": "Major Comments:\n1) Page 7. I do not understand why $\\widetilde{M}$ is said to be obtained using Nesterov momentum. It seems that $\\widetilde{M}$ and $M$ are updated using the same scheme. As I understand it, Polyak momentum (PM) and Nesterov momentum (NM) differ in the point at which the gradient is evaluated: PM takes the gradient at the current iterate $X_t$, while NM takes the gradient at a lookahead point $X_t + \\eta M_t$. Therefore, from my perspective, $\\widetilde{M}$ is also obtained using Polyak momentum, but with a different momentum parameter.\n\n2) Page 9. The feasibility guarantee is provided in Proposition 1. Later, $\\{\\eta_t\\}$ is chosen to depend on the maximum number of iterations $T$ (see Theorems 3 and 4). Can you show that the algorithm can find a nearly feasible point with this choice of $\\eta_t$.\n\n3) Page 10. Global convergence is established as a consequence of Lemma 9 (LaSalle's invariance principle). Is feasibility only guaranteed almost surely in the limit? It seems to me that feasibility converges deterministically at a certain rate, according to Proposition 1.\n\n4) Page 14. It seems the momentum parameters $\\beta_1$ and $\\beta_2$ are selected to be constant. This differs from the analysis of SGD with momentum for nonconvex optimization; for example, see \"Momentum Improves Normalized SGD\". Please comment on this.\n\n\nMinor Comments:\n1) Pages 3-4. Please add a reference to Proposition 2 for the proof of the statement that 'satisfying the KKT conditions is equivalent to the score function being zero and the solution being feasible'.\n\n2) Page 17. Please add a reference to the LaSalle's invariance principle.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u3PLQpyInc", "reviewer": "Reviewer_bGMr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a theoretical interpretation of the Muon optimizer by showing that it is a special case of the Lion-K framework with the nuclear norm as the underlying convex potential. Under this view, Muon with decoupled weight decay implicitly solves a spectral-norm-constrained optimization problem. The authors establish convergence rates and almost sure convergence to KKT points using Lyapunov analysis, and show that the framework naturally generalizes Muon to a broader class of spectral optimizers.\n\nStrengths\n- The paper is well-structured and technically sound, with clear connections to prior optimization theory.\n\n\n- The paper shows that Muon with decoupled weight decay does not merely penalize large singular values but instead enforces a hard spectral norm constraint, as evidenced by the conjugate-form objective, infinite cost outside the feasible set, exponential contraction into the constraint ball, and convergence to KKT points rather than unconstrained stationary points.\n\n\nWeaknesses\n- The overall approach is largely derivative of the Lion-K framework, and the conceptual novelty beyond specializing and extending existing theory may be limited.\n\n- Some technical sections are theory-dense, which may reduce accessibility for readers not already familiar with Lion-K or Lyapunov-based analyses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claims are supported by formal convergence proofs within the Lion-K framework, including explicit convergence rates and almost sure convergence in both deterministic and stochastic settings. The assumptions are standard and clearly stated, and the theoretical results are consistent with the provided empirical illustrations.", "audience_yn": "Yes", "audience_explanation": "The paper will be of interest to researchers working on optimization theory and implicit regularization in deep learning, particularly those studying adaptive and momentum-based methods. Its theoretical interpretation of the Muon optimizer and its connection to constrained optimization provides insights relevant to the TMLR audience.", "requested_changes": "- Please remove the proofs from the main paper to make the readability of theoretical results better.\n\n- Please discuss the connection between your work and the methods/results presented in https://arxiv.org/pdf/2503.12645.\n\n- Can you clarify the novelty of your stochastic convergence results?\nIn particular, is there an analogous result in the Lion-K paper similar to the deterministic case, or does your work establish genuinely new guarantees in the stochastic setting?", "broader_impact_concerns": "This work is mainly theoretical and does not concern major ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CJdSY9DLiy", "reviewer": "Reviewer_vroB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a theoretical convergence analysis of the Muon optimizer, which has recently shown promising empirical performance. The authors approach this by first framing Muon as a special case of the Lion-K family of optimizers, with K corresponding to the trace norm of the matrix parameter. This framing allows them to leverage the theoretical framework developed in (Chen et al. 2024) to analyze the convergence of Muon and to show that Muon optimizes the objective function under implicit constraints on the singular values of the parameter. \n\nTo support their theory, the paper presents empirical results on toy matrix optimization problems as well as image classification with different architectures, to verify that the solutions found by Muon satisfy the implicit singular-value constraints. They also further illustrate the difference in the singular value distribution when using Muon compared to AdamW.\n\nWhile the reduction of Muon to a Lion-K instance is clean and well presented, my main concern is that the theoretical results appear to be largely a special case of the existing results in Chen et al. (2024), rather than an extension as claimed. I expand on this concern below.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper’s central claim is that it extends the convergence analysis of Lion-K optimizers (in Chen et al (2024)) to Muon, where the challenge is extending the results to handle the non-differentiable trace norm. However, Section 4 of Chen et al. (2024) already provides a discrete-time convergence analysis for Lion-K optimizers that applies to non-differentiable K (theorem 4.1 for the deterministic case and the stochastic analysis in the appendix), which shows the convergence rate to the domain of the conjugate function of K* and also the rate of convergence to the stationary points of the corresponding implicit constrained optimization. Based on my reading, even the proof steps in the main body seem to closely follow the same steps in the earlier work. As a result, currently it is unclear to me in what precise sense the analysis constitutes an extension, rather than a direct application of existing theory to a particular choice of K.\n\nIt is possible that I am missing a subtle point where the analysis in Chen et al. (2024) does not apply to Muon, or where the present paper provides additional insight that is not already implied by plugging in the specific K in the earlier results. I find it hard to parse this distinction in the current presentation. If the theory indeed requires additional arguments or addresses a gap in the prior analysis, it is helpful to have this explicitly discussed and clarified. Otherwise, the claims and framing of the results should be adjusted to reflect the insights provided by the paper for the Muon optimizer.", "audience_yn": "Yes", "audience_explanation": "The Muon optimizer has received recent attention, and a principled theoretical understanding of its behavior or insights into its performance is definitely of interest to the TMLR audience", "requested_changes": "Given that the main focus of the paper is on the theoretical analysis, I suggest reframing the organization of the paper to better reflect and discuss which parts of the analysis in Chen et al (2024) are specifically extended/changed to handle the case of Muon (Please see the discussion on the claims).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)