FedMuon: Federated Learning with Bias-corrected LMO-based Optimization
Authors: Yuki Takezawa
Anastasia Koloskova
Xiaowen Jiang
Sebastian U Stich
Venue: ICLR 2026 Poster (2026)
Keywords: fderated learnig
muon
linear minimizatio oracle
Primary area: optimization

Abstract: Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than the existing adaptive optimization methods, such as Adam.
In this paper, we study how Muon can be utilized in federated learning.
We first show that straightforwardly using Muon as the local optimizer of FedAvg does not work since the LMO is a biased operator.
We then propose FedMuon, which can mitigate this issue and can converge to the stationary point.
We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FedMuon can converge for any number of Newton-Schulz iterations, while it can converge faster as we solve the LMO more accurately.
Through experiments, we demonstrated that FedMuon can outperform the state-of-the-art federated learning methods.

Published as a conference paper at ICLR 2026
FEDMUON: FEDERATED LEARNING WITH BIAS- CORRECTED LMO-BASEL OPTIMIZATION
Yuki Takezawa Kyoto University, OIST
Anastasia Koloskova University of Zurich
Xiaowen Jiang CISPA, Saarland University
Sebastian U. Stich CISPA
ABSTRACT
Recently, new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization methods, such as Adam. In this paper, we study how Muon can be utilized in federated learning. We first show that straightforwardly using Muon as the local optimizer of FedAvg does not converge to the stationary point since the LMO is biased operator. We then propose FEDMUON that can mitigate this issue. We also analyze how solving the LMO approximately affects the convergence rate and find that, surprisingly, FEDMUON can converge for any number of Newton-Schulz iterations, while it converge faster as solve the LMO more accurately. Through experiments, we demonstrated that FEDMUON can outperform the state-of-the-ae federated learning methods.
INTRODUCTION
Federated learning, which can train neural networks in parallel across many clients, has been attracting much attention (Kairouz et al.) [021 McMMhan et al., (2077: Karimireddy et al., (2020). In federated learning, each client has its own training datasets and updates its parameters using local optimizer, such as SGD. The central server collects the parameters from the clients and aggregates them. Since clients do not need share their local training datasets with others, federated learning inherently preserves data privacy.
For training neural networks efficiently, using an appropriate stepsize is one of the most critical factors. the stepsize is too large, the training collapses, whereas the stepsize too small, the training requires huge number of iterations. To adjust the stepsize the fly during the training, using adaptive optimization methods, such as AdaGrad (Duchi et al., 2011). Adam (Kingma & Ba 2017), Shampoo (Gupta et al. 2018), and other methods (Loshchilov Hutter 2019 Vyas al. have long been regarded as the de facto standard for training neural networks.
2025
Recently, Muon (Liu al. 2025a) has emerged a promising alternative, attracting significant attention. Many papers evaluated the performance of Muon and demonstrated that Muon can train neural networks faster and achieve higher accuracy than the existing optimization methods, such as AdamW W (Liu et al., 2025a Semenov et al. 2025). Roughly speaking, Muon projects the momentum in the Momentum SGD onto the space of orthogonal matrices. Muon closely related various optimization methods: it can be interpreted as a simplified version of Shampoo (Gupta et al. 2018). in which a certain momentum accumulation is disabled (L.iu al.) 2025a). and as an instance of optimizers with linear minimization oracle (LMO) under specific norm (Pethick et al.) 2025).
Kovalev 2025) also showed that Muon a special instance of the trust-region optimization method.
To use Muon for the large-scale training, developing the distributed version of Muon is important. However, Muon requires us to solve the LMO every iteration, which makes difficult to straight- forwardly Muon in distributed environment. [hn et al. (2025) proposed method to solve the LMO in distributed manner, although their method does not support multiple local steps and incurs huge communication cost. Thérien et al. (2025) proposed Mul.ooo, which extends Muon

---

Published as a conference paper at ICLR 2026
by allowing clients to update the parameters multiple times by Muon as in Local SGD (Stich) 2019 Woodworth et al. 2020). Although| Thérien et al. (2025) demonstrated that MuLoCo performss well when all clients share the same training dataset, their method is limited to homogeneous settings and lacks theoretical guarantees. As we show in Section 3 Mul LoCo fails to converge when clients have different datasets, which is fundamental characteristic of federated learning.
In this paper, we study the federated learning methods with the LMO and propose FEDMUON. (i) First, we show that straightforwardly using Muon as the local optimizer in FedAvg failed to converge to the stationary point since the LMO is a biased operator. We formally analyze the lower bound of this straightforward method, showing that it does not converge to the stationary point. especially in the heterogeneous setting. (ii) We then propose FEDMUON, which can mitigate the bias caused by the I.O and can provably converge to the stationary point. (iii) Furthermore, we derive novel analysis and reveal how the inexact LMO affects the convergence behavior of MUDMUON. Since solving the LMO exactly is computationally expensive, we solve the LMO approximately by running the Newton-Schulz iteration (Schulz) 1933) several times in practice. There were many papers that analyzed the convergence behavior of Muon (Riabinin et al., 2025; Liu et al., 2025a Shen et al. 2025), while most of them assumed that the LMO is solved exactly and ignored the effect caused by the inexact LMO. We analyze the impact of inexact solutions to the LMO on the convergence rate. We discover that for any number of Newton-Schull iterations, FEDMUON can converge the stationary point and can converge faster by up to a factor proportional to the square root of the dimension of the parameters as we solve the LMO more accurately. We experimentally demonstrated the effectiveness of FEDMUON, showing that FEDMUON can achieve higher accuracy than the state-of-the-ar adaptive federated learning optimization methods.
Our contributions are summarized as follows:
We show that directly plugging Muon into FedAvg as the local optimizer does not converge the stationary point since the LMO is a biased operator.
We propose FEDMUON, which mitigates the above issue by the bias correction mechanism and can converge to the stationary point.
We analyze the convergence rate of FEDMUON with the inexact LMO. Then, we show that for any number of Newton-Schull iterations, FEDMUON can converge, revealing how the Newton-Schulz iteration affects the convergence rate.
Through the experiments, demonstrated that FEDMUON outperform the laate-of-the-art federated learning methods.
Notation: We use || denote an arbitrary norm, and its dual norm is denoted by || 1l.... When we refer to a specific norm, we explicitly use the notation || - llp- - |I., |l ll and || |muese to denote the Schatten p-norm, Frobenius norm, spectral norm, and trace norm, respectively. We denote [n] (1.2......) for any n E N.
2 PRELIMINARY
In this section, we briefly introduce federated learning and Muon. The detailed discussion about the related works is deferred Appendix
Federated Learning: among n clients:
We consider the following problem where the loss functions are distributed
1 TI _ -
X0 {(()
min XEX
((X)
where X' is the parameter space (e.g., Rdd or Rddi)d)), X is the model parameter, DD the training that client holds, and f. X + H. is the loss function of client i.
The most fundamental algorithm for federated learning is Federated Averaging (FedAvg) (McMahan et al. 2017). In FedAvg, each client updates the parameter by using its own loss function, and then the central server aggregates the parameters sent from the clients. The update rule of FedAvg is described in Appendix C The original FedAvg uses SGD the local optimizer, while, in the non-distributed learning. it is important to use adaptive optimization methods for stable and fast training. Many papers proposed federated learning methods that use sophisticated optimizers,
2

---

Published as a conference paper at ICLR 2026
such as Momentum SGD Lin et al. 2021), Adam (Reddi et al., 2021). and the Newton method
(Elgabli et al.] 2022). Reddi et al. [2021) proposed a general framework and analyzed the convergence
rate with various optimizers.
Recently, optimizers with linear minimization oracle (LMO) have been attracting lot of attention (Liu et al. 2025a Pethick al. 2025 Riabinin LMO is defined as follows:
Optimizer with Linear Minimization Oracle:
et al. 2025
Imo X; = arg min Y); YED
where D is the convex set and (X, = Eig Niy Originally, the LMO has been used solve the convex constrained problems in the Frank- Wolfe algorithm (Frank & Wolfe 1956 Jaggi 2013). Recently.Jordan Jordan et al. (2024) proposed Muon, which uses LMO for training neural networks, which an unconstrained optimization problem. They showed that Muon can train neural networks faster than AdamW (Loshchllov & Hutter 2019) and Shampoo GGupta et al. 2018 Shi ct al. 2023), which
the most commonly used optimizers these days.
Specifically, the optimizers with LMO choose the unit ball as the constraint set D, measured in any chosen norm | ||. With slight abuse of notation, use Imo(-) to represent Imo(; D) with D (Y E X ||||| S 11, ie..
Imo (X) YE
arg min (X,Y).
Then, the update rules are given by:
((++1) (1 a) +OVF(XX" (7)) (r+1) (() tlmo
By varying the norm, we can recover different popular optimizers. Specifically, if parameter space is a vector when we choose the Euclidean norm and max norm, we can recover Normalized SGD with momentum (Cutkosky & Mehta 2020) and Sign SGD with momentum (Sun et al., 2023). respectively. Then, if the parameter space is R4 xd2 and we use the spectral norm for the LMO, we can obtain Muon (Jordan et al. 2024)). Note that the parameter space needs to the space of d₁ d2 matrices for Muon. Each layer is taken into account separately. For instance, the parameter of the convolutional layer is out nut_channel in channel x w matrix. When we use Muon, we consider dj out_ _ttttcechellllee and d2 = in_ channel h X The remaining scalar and vector parameters in the neural network are trained by other optimization methods, such as SGD or Adam.
For the remainder of the paper, we will not take separate layers into account and represent all the model parameters as a single matrix for simplicity of presentation. We refer to Riabinin et al. (2025) for explanation of how to take into account every layer separately in the analysis of Muon.
3 LOCALMUON DOES NOT ALWAYS CONVERGE
First, we provide a lower bound showing that straightforwardi using the optimizer with the LMO as the local optimizer in FedAvg does not always converge. For simplicity, consider the setting where all clients participate in every round and perform exactly one local update. Straightforwardly applying the optimizer with the LMO to FedAvg yields the following update rules:
M (++) = (1 (1-a) a VVE (r), E(())
(1)
x (++1) = ( + nlmo (M
(2)
(r+1)
(r+1)
(3)
n
We refer to the above algorithm as LOCAL (see Appendix for LOCALMUON with multiple local steps and partial participation). However, the above straightforward method fails reach stationary point due to the bias introduced by the LMO, and the optimization process stagnates.
Specifically, the LMO is biased, since in general we have
Imo (M(+)t) 4 Imo
M. ((
3

---

Published as a conference paper at ICLR 2026
The momentum M, is the estimation of the gradient V (X(), while the quantity of Ex i=1 Imo M,) biased and does not align with the gradient (.) This intuitively shows why LOCALMUON cannot converge to the stationary point, especially when clients have different loss functions. The following theorem formalizes this failure, with the proof deferred to Appendix [
Theorem 1. For simplicity, we consider the initialization M; = O. There exist convex functions
{f))i= such that for any Z rounds, the output of LOCALMMON (Eqs. [1 to 3 is the same as the initial parameter and does not converge to the optimal solution and satisfie the following:
(V) 2 (2()))
where 22 - | and X arg min f(X).
Note that LOCALMUON is simplified version of MuLoCo (Thérien et al. 2025), where the mo- mentum at the central server disabled. We formally analyze only LOCALMUON, while Theorem [ shows that the parameter stays at the initial parameters and does not converge to the stationary point. This indicates that MuLoCo also suffers from the same issue, as adding the momentum at the central server does not prevent the parameter from remaining at its initial value.
4 FEDMUON
In the previous section, we showed that due to the bias caused by the LMO, LOCALMUON does always converge. In this section, in Algorithm| we propose FEDMUON, which mitigates this issue and provably converges to the stationary point.
Instead of applying the LMO to the momentum alone, we apply the LMO to the bias corrected version of the momentum (line 8) in Algorithm [ Similarly to SCAFFOLD (Karimireddy et al. 2020) we introduce control variates c( and C to estimate the directions of the local client gradients V f (r)) and the global gradient V f(x(() respectively. Given that the local momentum
parameters M (1) estimate local gradients V f X (r.k),, the corrected update, M (((rA++1) -(("
C(r) a good estimation of the full gradient V 7f( (r.k),, mitigating the issue local bias. When we remove the LMO and set 1, FEDMUON recovers vanilla SCAFFOLD (Karimireddy et al., 2020). important to note that there are several papers that apply the momentum to SCAFFOLD
(Cheng et al. 2024 Karimireddy et al. 202I). however all of them incorporate momentum at the central server, differing from our proposed FEDMUNN.
Algorithm FEDMUON
1: Input: total number of clients number of sampled clients S, and the number of local steps K. 1} do (ar the server)
2: for E {0, 1,
3: sample S clients S, [m].
4: send (r) and (1r) to the sampled clients.
5: for E S, do (at the clients)
6: x (r.00 <- (+) and (7:0) +- M (r(K-.K)
7:
for k= 0, K do
M (r.+1)) -- (1- (1-a)M ( (r.k) + aVFi(X (r,)),
(1.)), c(rrk)).
8:
x (nd+1) - xx (ni mlmo (M. (r.+1)) ((r) + c(r)).
9:
10:
end for
c. (r.KK)
11:
send x and c(r+2) to the central server.
12:
13:
end for (end clients, back to the server)
for i [n] \ S, do
(!r+1] + (()) and M (r.K) + (r-Kk
15:
end for
((++1) +- (tr)
N Eies. 10511--91
(c(ott)-,"4
18: ( 4- 8-8 (r) E x(n:k)
19: end for
a
4

---

Published as a conference paper at ICLR 2026
5
CONVERGENCE ANALYSIS
5.1 ASSUMPTIONS
We first summarize the assumptions that we use our theoretical results. As common in the prior literature analyzing optimizers with LMO (Pethick et al. 2025 Riabinin et al.) (025). we use the following smoothness assumption. Note that the norm here the same as the one used in the LMO.
Assumption 1. There exists L 2 0 so that it holds for any X, X
[i() V X YII.
Since we consider non aun-uucidean norms, we measure gradient differences in the dual norm, while parameter differences are measured in the primal norm (cf. Nesterov 2018 Xie Li (022)) However, as shown in Remark [ any two norms are equivalent in finite dimensions, and thus the class of functions satisfying Assumption I and the conventional smoothness assumptions (formulated for Euclidean norms) is the same (see Remark (
Remark (Coonway 2019, Theorem 3.1)). If is fimtte-dimmessoonall vector space over F, then for any two norms all X E X.
Ip and || lll. there exist e, C such that cl X11 S | S II
Remark 2. If it holds that [V)(() Vi(() S CLIJX YIl for any X. Y E X, then X Ss satisfies Assumption it holds that XX - Y for any X, Y where e = supxex X
Ji satisfies Assumption 1 where C = SupXEX
[X] V)I(Y)II
X
For the analysis of FEDMUON, we often use the trace norm and Frobenius norm. The following inequality holds between the Frobenius norm and the trace norm.
Example 1. For any X E Rdxx xd2, it holds that | S || Jrace v (ii (idddddd) d2) X [|..
For the stochastic gradient noise, we use the following assumption, which is quite common in the optimization literature, e.g., (Bubeck 2015).
Assumption 2. The stochastic gradient is unbiased, i.e., [X;F)(X;E)] VA(()) for any X E X. Then, there exists so that holds for any X
Eg.DD. (1FolI/FF((X
(((()) 5
5.2 CONVERGENCE RESULT
We provide the convergence rate of FEDMUON in Theorem 2 For simplicity, we present the results for the special case 5 n. where all clients participate during the training. The general case with arbitrary is provided in Lemma 11 in Appendix E The proof is deferred to Appendix
Theorem Consider Algorithm
define (r.k)
E X ((rk) Note that (r-11)
X (r.K), Suppose that S and Assumptions
and 2 hold, C.OO M.000) and (())
1 SE ( there exists and a so that it satisfies
R-IK-1
TTroo nRK
1 +
T.roāt {INDO (RVK]
Tro
RK ΣΣ
so
R
1
O²K
O²K²
$1
+ōo
+
+
R
LraRin
Lrok?
where TO = (() - f-Pp = supxex
|X|| [
à
=
pa, ōo = poo, and of =
1 DEL,EIM (0,0) (
Discussion: Unlike LOCAL Theorem shows that FEDMUON can mitigate the issue that the LMO is biased operator and converge the stationary point. The dominant term is O Loq
RRK
5

---

Published as a conference paper at ICLR 2026
Algorithm 2 Newton-Schulz iteration
1: Input: matrix G and hyperparameters b, e IR.
2: G G
G
3: f {0,1,
1 do
4: G(++) t aG + b J (od"+o((" +d(G To
5: end for
6: Retern (T
which is almost the same as the terms appearing in the rate of FedAvg and SCAFFOLD (Kariiiredyy ct al. 2020), and the convergence rate is improved as the number of clients n increases. The only p. while this is because Theorem 2 analyzes the dual norm of the gradient. For instance, when the norm is the Frobenius norm, the dual norm is also the Frobenius norm and p 1. The last three terms arise from the initial error du, which diminish faster than the other terms as the number of rounds It increases.
difference is that the convergence rate of FEDMUON depends
We consider the as Muon (Liu et al., 2025a). Since the dual of the spectral norm is the trace norm, we have | I S [ rrace. Consequently, FEDMUON can converge faster than SCAFFOLD in certain cases. For instance, if the stochastic noise is sufficiently small, FEDMUON converges as:
where the parameter space is X' Te xd2 and the spectral norm is used,
R-1K-1 Σ
( Too
(r(
(r.k))
oo
RK
trace
and SCAFFOLD converges as follows (see Theorem 3 in (Karimireddy et al. 2020
R-1K-1
(0((12) (TOLF)) R
EΣ K=D
v so
RK
where Lp refers the smoothness of with respect the Frobenius norm. Thus, when (.P,E( 1I, (U, (X)U) x Lr, i.e., when the Hessians have a few dominant singular values caatte--quivllentty when they are approximately low-rank, then FEDMUON can converge faster than SCAFFOLD. More precisely, the terms on the right-hand side the same, and the only difference is the choice of the norm. We stress that Theorem 2 does not claim FEDMUON always converges faster, but does suggest that in certain cases FEDMUON can outperform. This helps explain the strong empirical performance of Muon and FEDMUON.
6 FEDMUON WITH INEXACT LMO
In the previous section, we considered the general case with an arbitrary norm and exact LMO. Here, focus the spectral norm, in Muon (Liu et al., 2025a), and analyze FEDMUON when the iteration. Then, thanks to the special
LMO is only approximately solved via the Newton-Schne property of spectral norm and Newton-Schut iteration, we reveal that FEDMUON can converge to the stationary point regardless of how accurately we solve the LMO.
With the spectral norm, the LMO takes the following form:
Imo
(X)
(XYY) YE(Y ERdixd2 |
arg min
Let the singular value decomposition of X be UEV. Then the LMO output is -VV, but computing this exactly is computationally expensive. To address this, [iu et al. (2025a) proposed approximating the LMO via fixed number of Newton-Schulz iterations (Schulz) (c.3) 5). The update rule of the newton-Schull iteration is described in Algorithm 2 Since the procedure involves only matrix multiplications, it can be efficiently executed GPU. In the following, analyze the convergence FEDMUON when the LMO solved approximately using Newton-Schull iterations and characterize how inexactness impacts convergence.
Under the same assumption as in Theorem 2 we provide the convergence rate when we run Newton- Schulz iteration 7' times solve the LMO approximately and show how T affects convergence. For

---

Published as a conference paper at ICLR 2026
simplicity, we set S and a b = - and for the Newton-Schulz iteration,
following the hyperparameter setting mentioned in Amsel et al. (2025). For the general case of arbitrary S we refer to Lemma 14 in Appendix []]
Theorem 3. Consider Algorithm 7 with the spectral norm and suppose that the LMO in line 8 is
solved approximately using Algorithm 2 with a b - and e 3. We define X (r
1 D x(r,)), Note that X ((+1)= x (r,K), Suppose that n = S and Assumptions
i-
1 and hold.
C.Oº = M, (0.0) and ((O)= == EE= c
Then, for any number of Newton-Schulz iteration T 2 0,
there exists 17 and a so that it satisfies
R-1K-1
so
(
((Tmmz²
[Indd2
7
(
LIDē ) RVK
Lra R
E
+
+
RK
nRK
km
1
ō²K K LrgRn
)
R2
5/7
+ōo
R
LroR?
where ro f( - fp
== po, ōo = pou. and o
+ E E (0.0) - (xx(X >|? Then, p is defined as follows:
log 1 -1-1 k5)..5
= 1+
log
1. == min J.rkk
8g,,rrk
(>0),
V Ey
where {8 are non-zero singular values of M. ((1++1) c C()
Remark 3. When T = 0, p 2. As T increase, p monotonically decreases to for any K > 0.
Remark 4. Recall that || 1lp is the Schatten p-norm. For any 1 S p S q. we have ||Alll S ||
Then, A becomes
[A] rrace and ||||r r when p = 1 and p = 2, respectively.
Discussion: Surprisingly, the above theorem shows that FEDMUON converges the stationary point, regardless of how many times we run the Newton-Schulz iteration. The only difference between the case when we solve the LMO exactly, i.e., Theorem 2 and the case when we solve the LMO is that Theorem 2 establishes the convergence in the trace norm of (I))) while Theorem 3 establishes the convergence in the Schatten p-norm fxx]l Ilp- We recover the convergence rate of Theorem 2 by setting T - oo, and therefore have p + 1. Since we have || I || when S p S 9. Theorem 3 implies that FEDMUON can converge faster when we increase the number of Newton-
approximately. i.e., Theorem
3
the gradient [V I
iterations T. More specifically, since it holds that ||A|| I /mmin(d,, d2 ||2, solving the LMO accurately can improve the convergence rate by up to factor of /mmin m/min(di,dd d2 our experiments, we will demonstrate that FEDMUON can train neural networks even if T = 0, while FEDMUON can achieve higher accuracy as T increases (see Scction 7.2). These observations are consistent with Theorem 3
The quantity of (1 - k)'.5" in the definition of p measures how fast the Newton-Schulz iteration converges. If we consider the worst case, could be arbitrarily close to zero, and thus a large T would be required to sufficiently decrease p. However, the main implication of Theorem i]s that increasing T leads to an improved convergence rate. Indeed, our experiments show that even increasing T from 0 to I dramatically improves accuracy (see Section 7.2).
Comparison with Existing Analysis with Inexact LMO: There are many papers that analyzed the convergence rate of Muon, while most of them assumed that the LMO is exactly solved (Pethick) ct al. 2025 Riabinin et al., (2025] Shen et al. 2023). The only study analyzing the rate with an inexact LMO is Refael et al. (2025). However, they also assumed that we run Newton-Schull iterations certain number of times (see Lemma 3.3 and Remark 3.6 in (Refael et al.] (025)). Compared with these prior analyses, our novel analysis provides a stronger claim that FEDMUON can converge to the stationary point for any number of the Newton-Schul iterations T' > O. Furthermore, it is first observed by Theorem 3 that the different norms of the gradient are bounded depending on T.
When ||
is the spectral norm, its dual norm is the trace norm.
7

---

Published as a conference paper at ICLR 2026
FashionMNIST
CIFAR-10
LocalMuon
FedMuon
FedAvg
Accuracy
0.8
Accuracy
FedAvg (Adam)
0.6-
SCAFFOLD
0.7
SCAFFOLD (Adam)
0.4
0.6
500
1000 Round
1500
2000
1000
2000
3000
Round
Homogeneous Case (3 10.0)
FashionMNIST
CIFAR-10
0.9
LocalMuon
FedMuon
FedAvg
Accuracy
0.8
Accuracy
FedAvg (Adam)
0.6
SCAFFOLD
0.7
SCAFFOLD (Adam)
0.4
0.5
0.2
500
1000
1500
2000
1000
2000
3000
Round
Round
(b) Heterogenenus Case (3 0.1)
Figure I: Training curves various methods. For all settings, FEDMUON can achieve higher test accuracy than other methods.
Proof Sketch: In the following, provide an intuition for why FEDMUON can converge for any 2 0. we solve LMO exactly, we have
GG, Im0 (G)) ||
traces
|lmomaon
G)) sp
The first equality holds from the definition of the dual norm (see Lemma Tb. and the second inequality holds since the solution of the LMO satisfies the constraint. Then, the output of the Sewton-Schulz iteration satisfies the following (see Lemma 12):
G Irace < -G,-G(T) < G|G
|
S 1. sp
(5)
The above inequality indicates that even if we run the Newton-Schulz iteration only few times to solve the LMO approximately. the output of the Newton- -Shoulz iteration is proper direction to minimize the loss function, and FEDMUON can converge to the stationary point. For instance, when 7' = 0, the output of Newton-Schulz iteration is T
C which corresponds to the normalized
gradient, and it natural that FEDMUON we run the Newton-Schul iteration T times, the output of the Newton-Schulz iteration comes close the exact solution of LMO and remains a proper direction minimize the loss function. Thanks to this property, FEDMUON can converge to the stationary point for any number of Newton-Schulz iterations T.
converge to the stationary point when T 0. Then, if
EXPERIMENT
FEDERATED LEARNING TASKS
Setup: We used FashionMNIST (iao al. 2017 and CIFAR-10 (Krizhevsky 2009) training datasets, and used LeNet (Lecun et al.] 1998) for Fashion MNIST and ResNet-18 (He et al. (016) for
CIFAR-10. Following the prior paper (Hsieh al. 2020). used Group Normalization (Wu & Hel
8

---

Published as a conference paper at ICLR 2026
10.0
a= 0.1
0.90
0.90
FedMuon (7= 0)
FedMuon (7=1)
Accuracy
Accuracy
0.85
FedMuon (T 2)
0.85
FedMuon (T= 4)
0.80
FedMuon (T=
25
50
75
100
25
50
75
100
Epoch
Epoch
Figure 2: Training curves of FEDMUON with various number of Newton-Schulz iterations. We used FashionM- NIST and LeNet.
2018) instead of Batch Normalization (loffe & Szegedy 2015) for 18.. We set the number of clients n to and sampled = 8 clients every round. We set the number of local steps K to 5 and set the number of epochs to 100 and 200 for MNSSI and CIFAR-10, respectively. Following the prior paper (Hsu et al. 2019), we distributed the training dataset to clients by using Dirichlet distributions with hyperparameter B. As / approaches zero, each clients come to have different training dataset. We tuned the stepsize by grid search. See Appendix [ for details. The experiments repeated with two different seed values, and reported the average.
Comparison Methods: We compared the following methods: (1) FedAvg (MeMahan et al. 2017):
We used Momentum SGD as the optimizer. (2) FedAvg (Adam): We used Adam as the optimizer of FedAvg. (3) SCAFFOLD (Karimireddy et al. 2020): ; We used Momentum SGD the optimizer. (4) SCAFFOLD (Adam): We used Adam as the optimizer of SCAFFOLD, (5) FEDMUON: Our
proposed method. Following the suggestion of Liu et (202a)), we changed the scale of the stepsize per layer, depending on the dimension.
We show the results in Fig. 1 The results indicate that FEDMUON can perform the best for all settings. By comparing FEDMUON with FedAvg (Adam) and SCAFFOLD (Adam), FEDMUON achieved the highest accuracy, which can demonstrate that Muon is also beneficial in the federated learning setting. By comparing FEDMUON and LOCALMUON, LOCAL performed well the homogeneous setting, but did not match the performance of FEDMUON in the heterogeneous setting. This observation is consistent with the discussion in Section 3 where we show that LOCALMUON does not converge the stationary point in the heterogeneous setting. These observations were consistent with Theorem
Results:
7.2 EFFECT OF INEXACT LMO
Next, we evaluate how the number of Newton-Schulz iterations affects the performance. Figure shows the training curves of FEDMUON with different T. In the homogeneous setting, the highest accuracy was achieved when T 4, and the heterogeneous setting, the highest accuracy was achieved when T = 2. Thus, performance. Notably, FEDMUON already worked with = 0, but increasing T from significant improvement in accuracy. These observations were consistent with Theorem which shows that FEDMUON can converge for any T and converge faster as increases.
observe that solving the LMO accurately can improve the
led to a
CONCLUSION
In this paper, we study the federated learning methods with the LMO and propose FEDMUON. We first propose directly plugging the optimization methods with the LMO into FedAvg, which we referred to as LOCALMMON, and show that LOCALMUON cannot converge to the stationary point since the LMO is a biased operator. We then propose FEDMUON to solve this issue and show that FEDMUON can converge to the stationary point. We analyze the convergence rate of FEDMUON and reveal how the approximate solution of the LMO affects the convergence behavior. Notably, we show that FEDMUON can converge for any number of Newton-Schulz iterations, and FEDMUON converge faster as we solve the LMO more accurately. Throughout the experiments, we demonstrated the effectiveness of MUDN and verified our theoretical discovery.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This paper was supported by JSPS KAKENHI Grant Number 23KJ1336.
REFERENCES
Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Langford. Dion: Distributed orthonormalized updates. In arXiv, 2025.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication- efficient SGD via gradient quantization and encoding. In Advances in Neural Information Process- ing Systems, 2017.
Shun-ichi Amari. Natural gradient works efficiently in learning. In Neural Compulation, 1998.
Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. matrix sign methods and their application to the muon algorithm. In arXiv, 2025.
The polar express: Optimal
Sébastien Bubeck. Convex optimization: Algorithms and complexity. In Foundations and Trends in Machine Learning, 2015.
Wenlin Chen, Samuel Horváth, and Peter Richtárik. Optimal client sampling for federated learning. In Transactions on Machine Learning Research, 2022.
Ziheng Cheng, Xinmeng Huang. Pengfei Wu, and Kun Yuan. Momentum benefits non-iid federated learning simply and provably. International Conference on Learning Representations, 2024.
John B. Conway. course in functional analysis. Springer. 2019.
Ashok Cutkosky and Harsh Mehta. Momentum improves normalized SGD. International Conference on Machine Leurning, 2020.
George E. Dahl, Frank Schneider, Zachary Nado, Naman Agarwal, Chandramouli Shama Sastry, Philipp Hennig, Sourabh Medapati, Runa Eschenhagen, Priya Kasimbeg, Daniel Suo, Juhan Bae, Justin Gilmer, Abel L. Peirson, Bilal Khan, Rohan Anil, Mike Rabbat, Shankar Krishnan, Daniel Snider, Ehsan Amid, Kongtao Chen, Chris Maddison, Rakshith Vasudev, Michal Badura, Ankush Garg, and Peter Mattson. Benchmarking neural network training algorithms. In arXiv, 2025.
Aaron Defazio, Xingyu Alice Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. In Advances in Neural Information Processing Systems, 2024.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Journal of Machine Learning Research, 2011.
Anis Elgabli, Chaouki Ben Issaid, Amrit Singh Bedi, Ketan Rajawat, Mehdi Bennis, and Vaneet Aggarwal. FedNew: A communication-effeee efffcccttn and privacy-preserving Newton-type method for federated learning. In International Conference on Machine Learning, 2022.
M Frank and P Wolfe. An algorithm for quadratic programming. In Naval Research Logistics Quarterly, 1956.
Yuan Gao, Rustem Islamov, and Sebastian Stich. EControl: compression and error control. In International Conference on Learning Representations, 2024.
Fast distributed optimization with
Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating newton-schulz iteration for orthogonalization via chebyshev-type polynomials. In arXiv, 2025.
Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. In Advances in Neural Information Processing Systems, 2021.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimiza- tion. In International Conference on Machine Learning, 2018
10

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/ / CVF Computer Vision and Pattern Recognition Conference, 2016.
Yutong He, Xinmeng Huang, and Kun Yuan. Unbiased compression saves communication in distributed optimization: When and how much? In Advances in Neural Information Processing Systems, 2023.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-llI) data quagmire of decentralized machine learning. In International Conference on Machine Learning. 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. In arXiv, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Satoki Ishikawa and Ryo Karakida. On the parameterization second-order optimization effective towards the infinite width. In International Conference on Learning Representations, 2024.
Rustem Islamov, Mher Safaryan, and Dan Alistarh. AsGrad: A sharp unified analysis of asynchronous- SGD algorithms. In International Conference on Artificial Intelligence and Statisties, 2024.
Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In International Conference on Machine Learning, 2013.
Xiaowen Jiang, Anton Rodomanov, and Sebastian U. Stich. Stabilized proximal-point methods for federated optimization. In Advances in Neural Information Processing Systems, 2024a.
Xiaowen Jiang, Anton Rodomanov, and Sebastian Stich. Federated optimization with doubly regularized drift correction. In International Conference on Machine Leurning, 2024b.
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse, and Jeremy Bemstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL. https
//kel Sellleer Ter g thub. ic wtt.hh/....ppp...///so http///sssss/sm/////o/oo
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi, Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song. Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. In Foundations and Trends in Machine Learning, 2021.
Sai Praneeth Karimireddy, Quentin Rebjock, Sehastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International conference machine learning. 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, 2020.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Breaking the centralized barrier for cross-device federated learning. In Advances in Neural Information Processing Systems, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2026
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference Machine Learning. 2020.
Anastasia Koloskova, Sebastian U Stich, and Martin Jaggi. Sharper convergence guarantees for asynchronous SGD for distributed and federated learning. In Advances in Neural Information Processing Systems, 2022.
Dmitry Kovalev.
Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. In arXiv, 2025.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical Report, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. IEEE, 1998.
Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data, International Conference on Machine Learning, 2021.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for Ilm training. In arviv, 2025a.
Liming Liu, Zhenghao Xu, Zixuan Zhang, Hao Kang, Zichong Li, Chen Liang, Weizhu Chen, and Tuo Zhao. Cosmos: hybrid adaptive optimizer for memory-effiiine training of llms. In arXiv, 2025b.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. Swan: SGD with normalization and whitening enables stateless Ilm training. arXiv, 2025.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication--Efiice Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics, 2017.
Konstantin Mishchenko, Francis Bach, Mathieu Even, and Blake Woodworth. Asynchronous SGD beats minibatch SGD under arbitrary delays. In Advances in Neural Information Processing Systems, 2022.
Angelia Nedié, Alex Olshevsky, and Wei Shi. Achieving geometric convergence for distributed optimization over time-varying graphs. In SIAM Journal on Optimization, 2017.
Yurii Nesterov. Lectures convex optimization. In Springer, 2018.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. In International Conference on Machine Learning, 2025.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný. Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021.
Yehonathan Refael, Guy Smorodinsky, Tom Tirer, and Ofir Lindenbaum. Sumo: Subspace-aware noomemt-othhogonaaizziiion for accelerating memory-efficient 1lm training. In arXiv, 2025.
Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik. Gluon: Making muon & scion great again! (bridging theory and practice of LMO-Laseee optimizers for LLMs). arXiv, 2025.
12

---

Published as a conference paper at ICLR 2026
Anton Rodomanov, Xiaowen Jiang, and Sebastian U Stich. Universality of adagrad stepsizes for stochastic optimization: Inexact oracle, acceleration and variance reduction. In Advances in Neural Information Processing Systems, 2024.
Günther Schulz. Iterative berechung der reziproken matrix. In Zeitschrift für Angewandte Mathematik und Mechanik, 1933.
Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining. In arXiv, 2025.
Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of muon. In arXiv, 2025.
Hao-Jun Michael Shi, Tsung-Hsicnt Lee, Shintaro Iwasaki, Jose Gallego-Posadee Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data- data-parallel pytorch imple- mentation of the distributed shampoo optimizer for training neural networks at-scale. In arXiv, 2023.
Sebastian U. Stich. Local SGD converges fast and communicates little. arXiv, 2019.
Sebastian U Stich, Jean- Sran-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. In Advances in neural information processing systems, 2018.
Tao Sun, Qingsong Wang, Dongsheng Li, and Bao Wang. Momentum ensures convergence of SIGNSGD under weaker assumptions. In International Conference on Machine Learning, 2023,
Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking:
Momentum acceleration for decentralized deep learning on heterogeneous data. Transactions on Machine Learning Research, 2023.
Hanlin Tang, Shaoduo Gan, Ce Zhang, Tong Zhang, and Liu. Communication compression for decentralized training. In Advances in Neural Information Processing Systems, 2018 8a.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. d: Decentralized training over decentralized data. In International Conference on Machine Learning, 2018b.
Benjamin Thérien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky. Muloco: Muon a practical inner optimizer for diloco. In arXiv, 2025.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. PowerSGD: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems, 2019.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. SOAP: Improving and stabilizing shampoo using adam for language modeling. In International Conference on Learning Representations, 2025.
Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. In Journal of Machine Learning Research, 2020.
Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local SGD better than minibatch SGD? In International Conference on Machine Learning, 2020.
Yuxin Wu and Kaiming He. Group normalization. In arXiv, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for benchmark- ing machine learning algorithms. In arXiv, 2017.
Shuo Xie and Zhiyuan Li. Implicit bias of adamw: foc-norm constrained optimization. In Interna- tional Conference on Machine Learning, 2024.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar, Adaptive methods for nonconvex optimization. In Advances in Neural Information Processing Systems, 2018.
13

---

Published as a conference paper at ICLR 2026
Jianyi Zhang, Ang Li, Minxue Tang, Jingwei Sun, Xiang Chen, Fan Zhang, Changyou Chen, Yiran Chen, and Hai Li. Fed-CBS: A neterogeneity-awaee aware client sampling mechanism for federated learning via mass-imbalance reduction. In International Conference on Machine Learning, 2023.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha C. Dvorek,, Xenophon Pa- pademetris, and James S. Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In Advances in Neural Information Processing Systems, 2020.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used LLM for proofreading, and it did not contribute to the content of the paper itself.
B RELATED WORK
Federated Learning:
The simplest algorithm for federated learning is FedAvg (McMahan et al.
2017 Stich 2019). The main challenge of federated learning is reducing communication between
the central server and clients. Various techniques such as client sampling (Gu et al. 2021 Chen
al. 2022 Zhang et 2023), multiple local steps (Woodworth et al.) 2020 Koloskova et al. 2020
Jiang et al. 2024a b). and communication compression (Alistarh et al. 2017] Stich et al.,
2018
Karimireddy et al. 2019 Vogels al. 2019 He et al., 2023] Gao et al. (024g have been studied to reduce the communication costs. However, FedAvg still requires huge amount of communication
when clients have different training datasets. Many papers proposed federated learning methods that are robust to data heterogeneity (Karimireddy et al. [020 Jiang et al. 2024a b). The seminal work is SCAFFOLD (Karimireddy et al.) (020), which can converge regardless of data heterogeneity.
Besides these methods, asynchronous methods (Koloskova al. 2022 Mishchenko et al. 2022
Islamov et al. 2024) and decentralized methods (Nedić et al. [2017 Tang et al. 2018b a Koloskoval al. 2020 Takezawa et al. 2023) have been widely studied further improve the efficiency.
Adaptive Optimization Methods:
Using adaptive optimization methods is standard for training
neural networks efficiently (Amari 1998 Ward et al. [2200] Duchi et al. 201 Kingma & Bal 2017;
Loshchilov & Hutter 2019 Zaheer et al. 2018 Zhuang al. 2020 Defazio et al. [024 Rodomanov
ct al.
2024). Over the last decade, Adam (Kingma Ba] 20 7) and Adam W (Loshchilov & Hutter
2019) are the most widely used, but recently, Shampoo (Gupta et al. 2018 won the External Tuning
Task of AlgoPerf (Dahl et al. (025) and is attracting considerable attention (Shi et al. 2023 Vyas
al. 2025 [shikawa & Karakida (2024). Muon (Liu et al.) 2025a) can be regarded as the simplified version of Shampoo, and many papers have demonstrated that Muon can train neural networks faster than Adam, AdamW, and Shampoo (Liu et al. 2025a Pethick et al. 2025 Amsel et al.) [2025] Liu Ma et al. 2025 Amsel et al. 2025 Grishina et al. (2025). Using Muon in distributed
ct al. 2025b) environments is one of the popular topics (Thérien et al. 2025 Ahn et 2025) Specifically, Ahn et al. (2025) proposed a method to solve the LMO in a distributed way, and Thérien et al. (2025)
proposed MuLoCo, which extends Muon by allowing clients to perform several steps before averaging the parameters as in LOCALMUON. However, since they consider settings where all clients have the same dataset, their objective differs from ours. As we explained Section 3 because the LMO is biased operator, bias correction mechanisms used in FEDMUON are necessary in federated learning, in which clients have different training datasets.
15

---

Published as a conference paper at ICLR 2026
C PSEUDO CODE
Algorithm 3 FedAvg (McMahan et al. 2017)
1: Input: the total number of clients n, the number of sampled clients S, and the number of local
steps K.
2: for E {0, (0,1,-.. [ do (at the server)
3: sample S clients S, C [n].
4: for i S, do (at the clients)
(!r.0)) t (x())
5:
6: for k = 0,1, - do
X ((nkt1) & x(1,k)
V/F (n,), c(rrk)).
7:
8: end for
9:
end for (end clients, back to the server)
10: X ((+11) t n-8 ( + Eies, x kr:
end for
Algorithm 4 LocalMuon
1: Input: the total number of clients n, the number of sampled clients S, and the number of local steps K.
2: for E 10.,,.... R- 1} do (at the server)
3: sample S clients S, [m].
4: for i E S, do (at the clients)
x-(r0)) (x) and M. (r,0) f- M, M. (r-.k)
5:
6:
for 0,1, K do
+ aVVi(XX (11) ; c(h1)) x ((nkt1) t (r.ik nlmo (M.Att))
(m++)) +- (1-11) M (r.k)
7:
8:
9:
end for
10: C. M(nnk)
11:
end for
12:
for i [n] \. S do
13: (((r.) + (T)k)
14: end for (end clients, back to the server)
X 2-8X(1 Eies. x(1.R)
16: end for
16

---

Published as a conference paper at ICLR 2026
D PROOF OF THEOREM 1
Proof. We consider the setting where n = 2, d = 1, and the norm the Euclidean norm. In this case,
have
Imo(x)
[o]|
Then, consider the where Si and ʃ2 defined as follows:
x²
fi(x)
2
(x
aa)
f2(x)
2
When M, and ( = a 41
we have
M aa
4
M
3ca
4
Imo(M, ( + lmo(M, (3) 0,
where we use (0, 1].
Thus, the parameter does change, i.e., (1) (0)
(0) For the next round, have
M - 4 (a a(1 a)),
M (2) 3a
(a + a(1 a))
Then, since it holds the following as in the first round:
Imo(M( + Imo - =0
The parameter does change. Due the above discussion, the parameter does not change for any r. Now, we have
a²
(6)
16"
Then, using E VX
5a
we obtain the desired result.
16
17

---

Published as a conference paper at ICLR 2026
E PROOF OF THEOREM 2
E. NOTATION
In this section, we use the following notation.
(rr.)) n- x
1
Σ x(nn))
n
n
iES.
G = MM - c + +
(8)
((r..t1)_ = Imo (G.nktt))
(9)
E.2 USEFUL LEMMA
Lemma 1. For any X E X, we have
(x Imo x)) - X||
Proof. From the definition of Imo(-), we have
(X, Ilmo X))
min Y
(X, Y)
max
(-X,Y)
YE(VEX([M)IS))
-X
Lemma 2. For any 0, R 2 and (0, 11, we have
R
1
k(1
a) kr <
k
r=0
Proof. We have
k (1 a)
Σ 1(1 hr <
Then, using (1 - a)x < aak Itak obtain the desired result.
Lemma 3. For any A, B X. we have
(A, B) S B||
Proof. We have
A
(A, B) = |||||
B < B|]
||]
Lemma 4. Suppose that Assumption holds. Then, it holds that for any X, E
f(X)= < f((Y) + (V) Y + X Y
(10)
18

---

Published as a conference paper at ICLR 2026
Proof. Using the Fundamental Theorem of Calculus, we have
= S. (V + +1xx- Y)), X Y) dt
=1(8+ (V - -x)+ (Vf(Y + + - -)) - -V)(), - X dt
1=0
+X(++ + (VX)) - xx)+ 1" IVf(Y + +t(x - -Y)) - - -
+ -x) + L'
Lt | - Y1 dt
1=0
= + + Y - x) + 2
| -
where we use Lemma B and Assumption
[ for the first and second inequalities, respectively.
E.3 MAIN PROOF
Lemma 5, Suppose that both r = r' and l: 2 W hold, or > r' holds. Then, we have
x (r())) - x (r) < n
n
((r K kk k')
Proof. From the update rule of X (m:)) and (('(k) we have
X (mn))_
S X
(n))
i.S,
= xx()
7
(nk*)
71
iES, k==1
k
xx0) 1
D ((n
" 3
D D("
n iSS, k"
n
r" iES,.t ====1
k' D(ri
X
x60
2
n
iES,/ k==1
Thus, we have
|x((rd)) - x(1'100
17
D (r.k")
7
D(
D
- ieS,, ="=k4+1
Σ
T1
n 1=rrr+1iEs
n
VES, k==1
< S1S
((r r' k
n
where we use ||
k for any r and k.
Lemma 6. Suppose that both r' and ls 2 K' hold, or > r' holds. Then, we have
|x|" [x((1>)) x #1 < (r -r-r' r' + 2)Kn.
Proof. We have
(x((
x((1))
< x - x (1:01
+ x(( -
+ _ x(
x -x((-001
+ ((())) x(0000
|xx! _ x(
S tyk K²)
xx( - 0l
Using Lemma 5 we obtain the desired result.
19

---

Published as a conference paper at ICLR 2026
Lemma 7. Suppose that Assumptions 1 Land 2 hold. Then, when 2 1, we have
7 n - (x(r))))
V ( + +2LK
s 2 17²
2 T/
| -C6031 -
+ 24 E
((nnt))) - -
M
+ 2
L 5/5 5
np²
2
IESr
When r = 0, we have
S/i (((())))))
(S)2 2
(x(x(()
S {Ef(X((0)))
+ +2LK
11 1/²
n2
211 F. Σ +
| M -,
||
1
S
n² + 2
S
poo7)
n iES
2
n
11
Proof. We have
F,
x((rh)+ + ! Σ D (r,))
Ef
n iES,
$ 51(x(1()))
+ ? Eri ((((())))) D!!rki) Br,
Lnp
Σ
D D
2n
iES, ()((()))) G(rrtt)) D.irkt11))
iES,
S ((((()
+ E
7 B
G G((k)))
D( +
LSn7²
n iES,
iES.
2n
S {((x(n))
+ 7 Em
Vf(x((())
+ 7 1|,
B Σ
D ((rrk+1))
LSn7²
n
2n
EES.
EE
where we use Lemma and third inequalities, and G, and D; are defined in Appendix E.I] Using Lemma] I and the triangle inequality, we have
4
|D | (n*+t)] S 1, and the Cauchy-Schwarz inequality in the first second,
Ti - |(( s | + /
| (01000))-((1 Gf -G(()))))))))((()
Then, it holds
Ex ( (((((())) D V [
+ 21 E Σ
-
- -cf
LSS1?
n E.
2n
Ta
When 2 1, we have
Ta = V (n))) M(nAt1) + (.t -c
(xx(x))) V1(( ||(X |
- -c,+ || + - ||
< L ((ra))
1> ||| + / / -c + | c0
LSK1
| - cn,l + ( c.:¹)
n
where we use Lemma 5 in the last inequality.
20

---

Published as a conference paper at ICLR 2026
When O, we have
T2 = V f (xx(0 - M, (0,+1) : - c
(f - (0((11000)) |1| + - +
M -
S T. x x + | -c:0 + +1|00,00 -0, ||
LSK
Vf - - + -| c,
- -c,00
n
Then, using the following inequality:
1 L n
E V - -
E
- pot.
obtain the desired result.
Lemma 8. Suppose that Assumptions and 2 hold, c. = M. and ((0) = 1
ni - c.0))
E VF C- -I-- - -M,0000/ < Spo
Sa
1 -
+ apv Ko² + 6LKnn
SE
Proof.
Let cr(r 1) be the number of times that client has been sampled by round r. We have
cr 1) K-1
M = -11aa)alr1)) 1)K (opalr11xxM00) (0.00 + a
Σ Σ (1- (1-
k==0
To simplify the notation, we denote ri(r') by the number of rounds that client is sampled for the r' time. Using this notation, we have
M (1.0)
c((r1))K 1)K
(1 (1-o) ()-r)alr-1) M. 0.00
+a Σ
(1 -oa))(--)---VVP(( (r - (
(1 - (1-a)a a)i(r11
(XF
0.0) (.00)) (f((x((000)
ci(r-1)K
+a Σ
(1-
(FF(
(r (T (T(T
- ( (r ( (1
a(r-1)K
+(1 +1 - a) 1)l 1)
(x (0,0)) +a Σ
(I-a)stt-K--d (
("
Using a(1 - am = a1 a0m m-1, we have
T= (Xx(X( (1((x(((0660))))--11
1)K-1
(V - - - +
(1 - (1-0))(-1))--
+ (1 (r) (VS((x((..)) ((
21

---

Published as a conference paper at ICLR 2026
Thus, we have
E M, (n:0)
- ||.
V
a) ()((r-1) po
Q(1-11kK 1)
(
(1 (I-a)a(t-)R-d ( ( ( -
+ aE
Σ
+ E
V (((xx((all0))))))))0)000
(X
-
1k--1)K-
( ( - 1. ( - (1-
Σ
+
E(1 - -a)aa))a--- a) | - (1100))
Using Assumption T have
M (( ( (
E
EE1 1))((r1)) pa tapy Ko²
T
c((-11K-1
L.
Σ
(1 (1-a) a1-
x (
+ LB - a) 11( x((000) x((r:11).0)
T5 K)), 1))
+ LE
x (1i(01600)1
(r-,KKII)
T4
The quantity of ca(r- - 1) is the number of rounds in which client i is sampled, which follows the binomial distribution.
T S pa
(1 a)
Kd
(5)
c'
s
r r-1
11
n
d'
S d'
(1-
S)
l-dd
(r-1)
pa
(1
a)
d
d==0
Sa r-1
po
1
n
(1( -rs (1x1)
Ta < EE
Σ
(1 (1-a)a(-1)k-oo
(r:
c((r-1)K-1
ri (p - 1)K - d" + -
+11 - (pe(r- - 1)K d" +
= n
(1- (1-))""
K
K
The quantity of Ts is the number of rounds from the time cline i was sampled to the next sampling, which follows a geometric distribution with expectation 5. Thus, we have
K S
To <
22

---

Published as a conference paper at ICLR 2026
Using Lemma 6 have
Ta - E
x (0.0) - - ( -E x - -x(r111)000
ns
(r)
where we use Lemma
5 in the last inequality. The quantity of r1) is the round in which client i is sampled for the first time, which follows a geometric distribution. Thus, we have
S Kn.
Using Lemma 6 have
Ta = (B 1))
+ 2)).
Since the quantity of rr(c((r-1))) (((c((r-))) 1)) is the rounds which client is sampled for the last time, we have
E (r 1- 1011--((((r-- 1))) = (r ---1 1) (1 1
5
(1
S r S 2n
+ Σ r'
v
S
y' =
Thus, it holds that
Ta S 4Kmn
S
By combining the above inequalities, obtain the desired result.
Lemma 9, Suppose that Assumptions 7 and holds. When 2 holds that
M (r.++1
c. s 2a
( pv Ko² Ko² +9KL
S
Sa
E ES
poo
1
n2
11
X F and oe = Σm E (IV - c. 12
where p = supxex
Then, when r = O, we have
(0,k+1) - -c:00 s (
S (5) poo-
E
M
pv Ko² IK1
Proof. We have
M(rk+1) a0) M (r.0)
M
+ Σ
a
)k
(F((X(())) (rok', c((())))
==
Since have clr) =M. (1:000 we have
E
Mr. (r.++) - c((1) = oF
Σ
a
(n)))) - 11.1000
IES.
iES. K==
23

---

Published as a conference paper at ICLR 2026
When 2 1, we have
k
E Σ SES,
- -c/70/
s a Σ
Σ (1 a)*
(VRi(x! ( : - (ra
iES,
1==0
+ aB Σ
Σ (1 A-
(V (
iES,
K'=0
+aB Σ
Σ (1 a) -
M
- )
1ES,
k=
< o Σ
Σ (1 (1-a)²= w)^ (VR(( -
(rock
iES,
K-=0
k
+ aB Σ
Σ (1 a) A (V
iES,
K==0
+ E Σ
[ 1KK -1)- - M,
(
SE
The first term is bounded from above as follows:
E
Σ
-a)a
(VB(( - -V (rack
k'=0
<
E
(1 - a)bc
(VF.(x (rkk) ;(; A (X( (n
k'=0
Σ (1- (1-
(F)(( (()))) V 112
K==0
$ PV
where we used Jensen's inequality in the first inequality. The second term is bounded as follows:
- - (
Σ (1 * kk
(1 a) k K' (x(( - v (-1
T. (1 a) * K |x(((r00)) x(--1,X-11
===0
3LKn
where we use Lemma 6 in the last inequality. Then, using Lemma we obtain the desired result when 2 1.
24

---

Published as a conference paper at ICLR 2026
When O, we have
k Σ
M -
(VE) :e/ -MM
aE
SESo
k'=0
k
aE
Σ a)
)
(VF((X 00.4 : -
iESo
A (1 a)
+ o
Σ
- (0.0))
M==0
&
o
Σ k'=0
(1 a) Kー
(V[i(x -M00.0
iES,
Vf((X(00) c:00|==
< aSpv Ko² + LKS +
SpE
where we use Assumptions andd n, we obtain the desired result.
2 and c.
(r,0) M( in the last inequality. Dividing both sides by
M,
Lemma 10. Suppose that Assumptions we have
and 2 hold, c, := M. and ((0) = 1/2 E (0)
E VJ(X ()((((- -c
Ps
aao2 S
4nLa as
6LKen
1 Sa
+
+ s
+ pa
Proof. Let e, 1) be the number of times that client has been sampled by round r. We denote ri(r") by the number of rounds that client i is sampled for the r'tth time. Using this notation, we have
(rr)
1 n
M (rt
Then, we have
c((-1) K-1
M. (r-1,K-))
-1- (1-a)a((-1)) M +a Σ
(0.0)
Σ (1
(F of KK
(r
r==1
k==
-11-a)a(r-1) (0.0)
(M.000) (((()(())))
(r(r-)) 1) K
Σ (1-
(X (T' (r' g' v(((x!
-
+
K=O
Ti
(r-1) 1) K
+ (1 -1-aa)tt (11-)) +
Σ
(1 (10)))((--)) (x
(((((()))))) ()))
T2
We can rewrite Ti and Tz as follows:
lies, (1 -
(VF.(X (F' (P ((((((111))
Ti
r/=0 K
25

---

Published as a conference paper at ICLR 2026
(r(r1))
To = (1 - 1)K
+ Σ (1 (((
1(6-1)K-1
(
D.''t1 - (1
Σ (1 -1-a) (1-a)a(r-1)--
+o
+ (1- +11-a)a(111) 1)k
(ffi(x
Thus, we have
- (](X(111,--11)
< B((-a))((-D))
1
V (0.0), - Vfi ()(()(00))
Ta
1
Less, (1 (()))((--))) * (VE(X! (((1)))))) V)((X)
+ aaB
i=1 r'=0k=l
T4
Vi ( - vx((X (+_,, |||
E
T5
(11-1)K-1
12
V (r:(1 11.0 - - (X ( *
E
(1 (1-a) ()-)))((-1)---
71
To
E[I a) a))
V
Tr
Ta E(1 1)) pa
The quantity of cr - 1) is the number of rounds in which client i is sampled, which follows the binomial distribution.
5
S
Ta S pa
a)
Rdd
1- n1
1
11
S pa
r-1
a)
mm
S
d'
(1-m)
S
r-
1
n
n
c²=0
Sa
r-1
po
1
n
K
(1 - - (VF((X!/M)) cl (((XX!(1)
T4 E
k= 1 iES
p
g2
n V
5(1 (1 - )+
26

---

Published as a conference paper at ICLR 2026
(n M (x Vii 1
T5 E
LE
X ( X x(--1.8-1)
X (((rl(a(r-1))) 1)) X x (1((0)))))))000)) 1)).0) LE x ((()(et(--1))))) - x((-1,K-1)
$ LE
Lmn( 1)+ LSn
B(((1 1.5
(((((r-- 1))) K + K 1).
n Since the quantity of r:(ca(r - 1)) is the rounds in which client is sampled for the last time, we have
-0-11 1) (1 (1- (1- +
(1 (1- 72
r' 2n
= (r -
- S
y==0
Thus, we have
Ts S 2LKn
Cx(m- [)K-1
To S LE
Σ
(1-a) (1-r))00-1))---
x (olll
x (r:11 13,e' KT x1)
d==1
(1(r-1) K-2
(1 a (o) -
x (((1e")))) - ( |
LE
(ir-1)
+ LE
Σ
a1 (10))(((1))-0)) x x((r(c").
(1(r1) K-2
+LB
Σ
(1 a) (1--))-(--)
d'
(1--1)-1 1)-1
+ Lny
(1- (c 1) (rale" +1) - -ri(c") + +2)
(r-11)-11
S L1 + LnE
Σ
(1 (1-a)
(ri(e" +1) rs(c") +).
a
The quantity r:(c" + +1 r,(c") follows the geometric distribution, which has the expectation of 5 Using Lemma 2 we obtain
4nLn 3nLKn
To S
as
S
Ta I LB(1 a)((611) 1)K
x (0.0)
x
S LL)) + 2), (
where we used Lemma 6 in the last inequality. The quantity of (1( is the round in which client i is sampled for the first time, which follows a geometric distribution. Thus, we have
Ta S 3nLKn
S
Lemma 11. Suppose that Assumptions there exists and ar such that we have
1 and 2 hold, c.0) (0.00 and ((0) = Σ C(O)
K-1
((( so
((
(Lmpio) SRK
+ ((5)
11 Lropo
+ ( R S
Lro (m)
()))
poo 71 R S
+
RK
RVK
K=
(Po22S LraRin
+ poo (()) n
PO²K22 $
+
Lr
27

---

Published as a conference paper at ICLR 2026
Proof. Combining Lemmas 7 and 10
when 1, it holds
Ef (f((((
- Ef( (r.k))
S
V
(r.k))
a022
_L1 +
5
+ 2pm
33LK + 4apT
VKo²
S
Sa
r-
+ 4pn
00
n
When 0, we have
7 Ef (0x++)) (((x)()+ - (x(
[ |||
n
S
S
+ 5LKi2 + 20pp
v
KKo2 + +pm
n
Summing up the above two inequalities, we obtain
R-1K-1 (n)
||_ s ((5)
n2
To RK
S
1x72
8Ln
n
-33KKK (7)
E
+
2p
V
+
+
RK
S
5
T= K=0
4app Ka² + Apro
R-1
Σ
Sa
r-1
Ap0o
R
n
R
3
12
RK
10
+ +2
(S)
a0?
S
+
8Ln
5
n2 + +3LK
(5))
8p00
71
+ +apV Ko² +
Ra
S
Then, using the following hyperparameters
aru
1 ro
17 min
SLRK
K
V
33LR)
2 5 V
8L
2Lro
21
a min
1,
RLP2o22
RK?
5
we obtain the desired result.
28

---

Published as a conference paper at ICLR 2026
F PROOF OF THEOREM 3
Lemma 12. Let G and GG be the input and output of Algorithm
with a 8. b -4 and
3 For any number of iterations T, we have
where p is defined as follows:
log 1 (1 K) x2)2.
p = 1+
log
min
0),
EE sf
and is the non-zero singular value of G.
Proof. Let the singular value decomposition of be UEV. Then, the output -G can be written
as follows:
G(T) -UY(VVV
where (TT) is defined as follows:
x == Φ (6
(o(....
Σj
G
Thimes
Since ((x) > x, we have
(T)
En
|G|
Using the above inequality, we have
(G, -(T)) = USV, (UVV
Σ Eie (1- 11-11-219) 21!
When 15 b 8 -
5 and 3. we have
3
3 r
1
0 0 o(x) (1 x)²
1) S (1 2.
Thus, it holds that
1.5T
1.3
(T) ii
1
Σj
Ex
1-
GG||F
E,
23
F
for any 15p p S 2. Using the above inequality and the definition of and x we get
(G. GGTG S
Σ
P D
Lemma 13. Suppose that Assumptions and 2 hold. Then, when r 2 1, we have
S
S
Ef(x (m++1) < (E((
v ( + 2LK
2 5
TH V (
-
I
2m
n
E
M
(r+11)
c(T))
+
2
L
KES.
29

---

Published as a conference paper at ICLR 2026
where p is defined as follows:
p p= 1+ log [1 - (1 - k;)=
log
k:= min Jarrk
84,,,,
(>0).
VEy yr 87 ,1r,.
and {s are non-zero singular values of M.r.et - -/,1) +C
When r = O, we have
n
()) 2
Ef ((((10+)))) S (x(()))))))
+ 2LK
17²
20 +
E Σ
M, (0k+t
0.00
I. (
5
+ 2
19² +
poon.
n 1650
prace
11
Proof. We have
/ (x(xe(((()1)11)))
-Eras
(x(r,))
+ Σ (m/))
-
iES,
$ (((((())))
? En. n
Vf(x(())) (1.)) (((itt))
Lnd
B Σ
D 1
2
2n
iES
S (((((n))
+ 2 Er
(f(((())) G. ((rktl)) D(rikt1))
_ Σ
G
G(nk+1))
LSS
+ 2n
1ES,
iES,
S {(((())))
+ D
Ey.
Vf((((()))
((rk+1))
+
E
D(nktt))
LS
n
trace
n
2n
iES,
ES
where we use Lemma
1sp S 1, and the Lemma 3 in the first, second, and third inequalities. Using Lemma 12] the definition of p, and the triangle inequality, we have
4
| [D (r(+1)
G (+++))
T
S VS(X
(ndi))
+ +|V
G c(N++1)
$ (((()()))
+ [
G. (1++1))
where we use the fact that 2 1 and ||Al|p < Ilale for any A. Then, it holds
E E (nk+1)) < {(x(())) - 5
+ 2n E Σ
[V
- -G (f.k+1))
LSn
n
n E..
trace
2n
Ta
When 1, we have
V ( (((K+t))) + (006)_c'0
T2
Irace
Vf(x(11)) (1) Vf(x(1 VF(X(1-1,X-1) trace
V ((-K)))) - -c0o
| + | (rdt1)
c.r
crace
trace
< L (rik) ( [ - c0
M (n++1) - -c!"-
+
Irace
trace
LSK
|
- c(7)|| crace + ( _ c(7))
trace
30

---

Published as a conference paper at ICLR 2026
where we use Lemma
5 in the last inequality.
When r O, we have
T2 f(x((0))))) M, (0.tt))
+ +.,²0 -C(0)
trace
- V Vf(x (0.6)) - (0()00
+ + - c(
+
M,
c.00
crace
nacc
mrcce
< L x
(O.R) - x || + [
- c
lnce +
/ M
-c,² I
LSKT
V (x,0) c
|
( -
Irace
Then, using the following inequality:
V (( - c.oo
E V 1, S poo.
E
-
we obtain the desired result.
Lemma 14. Suppose that Assumptions there exists 17 and a such that we have
landd
hold, C. M (0.0) and C(0)
1 E.= C.
E
Vf(
)
so
Tmrapp2 SRK
1 + ((S)
Tropa
Tro
(m)
3/4
+
pao
RK
RVK
R
R S
0==
( LroRan
p²o²k2
2
$
((5)
n
+po0
+ poo
Lro
Then, p is defined as follows:
log 1 (1 k=)2.7
p = 1
log
min Jarkk
8j,b,r.k
(>0), 0)₂
Ey y 8 dir,
where {s are non-zero singular values of M. M. (r, - -(1))
Proof. Even if we solve the LMO approximately, the statements of Lemmas 9 and 10 hold. Thus, combining Lemmas 10 and 13 and tuning the hyperparameters as in Lemma obtain the desired result.
31

---

Published as a conference paper at ICLR 2026
G HYPERPARAMETER TUNING STRATEGY
In our experiments, the hyperparameters were tuned individually for each combination of method, dataset, and random seed.
Table 1: Hyperparameter tuning strategy for each method.
FedAvg FedAvg (Adam)
Stepsize
Grid search over (0.1,0.01, 0.001
Stepsize
Grid search over 1.110..01,.0000
SCAFFOLD SCAFFOLD (Adam)
Stepsize
Grid search over 00.1,0.01,0.000
Stepsize
Grid search over {0.1, 0.001]
Stepsize of Muon
Grid search over {0.001, 0.0001}
LocalMuon
Stepsize of Momentum SGD
Grid search over [0.1,
Stepsize of Muon
Grid search over 0..001,0.0001
FedMuon
Stepsize of Momentum SGD
Grid search over (0.1,
In the following tables, we list the hyperparameters tuned by the grid search. The reported hyper- parameters correspond to the values selected from two independent trials with different random seeds.
Table 2: Hyperparameters tuned for FashionMNIST.
B 10.0
B 0.1
FedAvg FedAvg (Adam)
[0.1,
0.11
(0.1,0.1}
(0.01,0.01)
SCAFFOLD SCAFFOLD (Adam)
(..1,0.1)
(0.1,0.1}
{0.001,
[0.001,
LocalMuon
0..)
(00000.
FedMuon
(.0))
(0.0001 (00001,0.00),000000..000
Table 3: Hyperparameters tuned CIFAR-10.
β 10.0
B 0.1
FedAvg
{0.1,
(0.1,0.1}
FedAvg (Adam)
10.001,0.001
(0.01,0.001)
SCAFFOLD
0.11,
{0.1,0.1}
SCAFFOLD (Adam)
(0.001,0.001]
[0.001,0.001]
LocalMuon
(000001,0001),000000.00000
((0.01,0001,(00000..000)
FedMuon
(0.00110001,
[0.00110001)
32


Reviews:
[{"review_id": "eZHUSiewy0", "reviewer": "Reviewer_E1uB", "summary": "This paper proposes FedMuon, an algorithm improving usage of Muon in federated learning. They propose Scaffold-like modification in Muon's momentum term before LMO and observe improvement in stability. They also show some theoretical analysis in spectrum norm.", "questions": "What weight decay term was used in the experiments? This factor has a significant impact on the results, and certain related techniques (e.g., [1]) can help stabilize training.\n\n[1] FedNAR: Federated Optimization with Normalized Annealing Regularization", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The writing is clear and easy to follow.", "weaknesses": "I'm not convinced by the paper's solidity and novelty.\n1. The analysis in Section 3 regarding Muon’s failure to converge in FL seems rather trivial. Similar reasoning could apply to any momentum-based optimization algorithm. This claim should be supported by experimental evidence. In fact, based on the only experiments on LocalMuon shown in Figure 1, I don’t observe significant convergence issues. While there are some oscillations, I believe standard techniques such as learning rate or hyperparameter tuning could easily address them.\n\n2. The modification is just including Scaffold-like updates in Muon, which seems quite limited.\n\n3. The theoretical analysis is standard by combing Muon and Scaffold and classical SGD together. \n\n4. It’s already 2025, yet the experiments are limited to FashionMNIST and CIFAR-10 under very restricted settings, with only two baselines (FedAvg and SCAFFOLD). This level of evaluation would have been insufficient even several years ago. Moreover, the final accuracy shows almost no improvement over the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4R8VeefRJl", "reviewer": "Reviewer_B6xb", "summary": "This paper focuses on improving federated learning optimization by addressing the bias issues when applying the Muon optimizer, which uses Linear Minimization Oracle (LMO), in the FedAvg framework. The authors show that directly using Muon within FedAvg, referred to as LocalMuon, fails to converge due to the bias introduced by the LMO. To overcome this, they introduce FEDMUON, an optimization algorithm that incorporates a bias-correction mechanism inspired by SCAFFOLD, ensuring convergence. The theoretical analysis includes the convergence rate under inexact LMO computations, handled via Newton-Schulz iterations. Experiments on two benchmark datasets, FashionMNIST and CIFAR-10, demonstrate that FEDMUON outperforms several existing methods, including FedAvg, FedAvg (Adam), SCAFFOLD, and SCAFFOLD (Adam).", "questions": "$\\cdot$ Can FEDMUON be effective on larger models, such as transformers, and not just small-scale datasets and simple models?\n\n$\\cdot$ Besides SCAFFOLD from 2020, can the authors compare FEDMUON with more advanced federated learning algorithms？\n\n$\\cdot$ Could the authors report the communication or time costs of FEDMUON and baseline methods to demonstrate its efficiency advantages?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "$\\cdot$ The paper is well-motivated. Muon can outperform traditional adaptive optimizers like Adam in centralized settings, but its direct application to federated learning, specifically in FedAvg, fails to converge due to bias introduced by the LMO.\n\n$\\cdot$ The use of Newton-Schulz iterations to approximate the Linear Minimization Oracle (LMO) in FEDMUON is a key innovation, as it enables efficient bias correction while maintaining strong convergence guarantees even with inexact LMO solutions.\n\n$\\cdot$ The experimental results in the paper are highly consistent with the theoretical analysis, supporting the convergence guarantees of FEDMUON derived in the theory.", "weaknesses": "$\\cdot$ It seems that the authors just incorporate the existing Muon to SCAFFOLD framework, which is just incremental. The authors should clarify their original contribution.\n\n$\\cdot$ The experiments are conducted on small datasets and simple models. Since optimizers like Adam perform better with larger models, comparisons on more complex models like Transformers would provide a better evaluation of FEDMUON's scalability and effectiveness.\n\n$\\cdot$ It would be better to report the time cost or communication cost of FEDMUON and baseline methods to empirically demonstrate that FEDMUON's advantages hold in terms of efficiency as well as performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ArDoK4bPey", "reviewer": "Reviewer_paxH", "summary": "This paper proposes FedMuon, a federated optimization algorithm that extends the recently proposed Muon optimizer—an LMO (Linear Minimization Oracle)-based method—into the federated learning setting. The authors identify that directly applying Muon in FedAvg (termed LocalMuon) fails to converge due to the bias of the LMO operator. To mitigate this, FedMuon introduces a bias-correction mechanism inspired by SCAFFOLD, along with a theoretical convergence analysis that also accounts for inexact LMO computations via the Newton–Schulz iteration. The authors provide both theoretical results and empirical validation on benchmark datasets (FashionMNIST, CIFAR-10) demonstrating FedMuon’s improved convergence and accuracy over existing methods such as FedAvg, FedAdam, and SCAFFOLD.", "questions": "1. Can the communication cost be further reduced?\n\n2.Is the method still effective for training and fine-tuning large models?\n\n3.Can the code be open-sourced?\n\n4.The algorithm’s line 8 is written differently from the MUON optimizer — why is it designed this way?\n\n5.Is the number of local iterations too small（K=5） ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.Novel and technically solid contribution.\nThe paper identifies a subtle but important issue (bias of the LMO in federated settings) and provides a theoretically justified correction method. The proofs (Theorem 1–3) are rigorous and clearly stated.\n\n2.Theoretical generality.\nThe analysis handles both exact and inexact LMOs, showing convergence under any number of Newton–Schulz iterations—a strong and non-trivial result rarely seen in this line of work.\n\n3.Experimental validation.\nThe experiments convincingly show FedMuon outperforming strong baselines under both homogeneous and heterogeneous data distributions. The empirical trends match the theoretical predictions (faster convergence as LMO accuracy increases).\n\n4.Bridging two active research areas.\nThe paper effectively connects recent LMO-based optimization (Muon) with federated learning theory, which is timely and relevant for distributed LLM optimization", "weaknesses": "1.Limited experimental diversity.\nThe experiments are conducted only on small-scale benchmarks (FashionMNIST and CIFAR-10). Demonstrating results on larger or more realistic FL setups (e.g., cross-device or NLP tasks) would strengthen the empirical claims. ViT model or roberta-base model，imagenet,CIFAR100 data.\n\n2.The proposed method simply combines the MUON and SCAFFOLD algorithms, and its communication cost is about twice that of the original approach.\n\n3.The experimental results do not show a significant advantage.\n\n4.The theoretical analysis of the paper is solid, but there is still room for improvement in the writing and experimental sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bb30YWjWmx", "reviewer": "Reviewer_YvBN", "summary": "This paper studies how to adapt the recently proposed Muon optimizer, an LMO-based method, to the federated learning setting. The authors show that directly applying Muon as a local optimizer fails to converge due to the inherent bias of the LMO operator. To address this, they propose FedMuon, which introduces bias correction via control variates similar to SCAFFOLD. The paper provides a detailed convergence analysis under both exact and approximate LMOs, showing that FedMuon converges for any number of Newton–Schulz iterations, and converges faster when the LMO is solved more accurately. Experiments on FashionMNIST and CIFAR-10 demonstrate superior performance in both homogeneous and heterogeneous settings.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides the first formal analysis of bias introduced by LMO in the federated setting and rigorously proves that straightforwardly applying Muon fails to converge.\n2. The author establish the convergence under both exact and inexact LMOs, and clarifying how the number of Newton–Schulz iterations affects the rate.\n3. The experiments align well with theoretical findings.", "weaknesses": "1. The algorithm section (Section 4) only offers an intuitive, SCAFFOLD-like rationale for bias mitigation, but lacks formal theoretical support. It would be helpful to include a short proof sketch or lemma explicitly showing which terms or steps contribute to bias reduction.\n2. Appendix D includes an example to prove why LocalMuon does not converge, but it would be more convincing to show that FedMuon can indeed converge under the same problem setting.\n3. While illustrating the bias issue with a divergence example is interesting, similar bias phenomena commonly appear in other contexts such as bilevel optimization. A typical and simple approach in those settings is to communicate the momentum across clients in LocalMuon. However, the paper lacks discussion on why the authors chose not to adopt such straightforward momentum communication and instead opted for a control-variate-based correction mechanism.\n4. There is no discussion of heterogeneous settings in the theoretical section. Given that the experiments include heterogeneous data distributions, the theoretical analysis should also address how FedMuon behaves under such non-IID conditions.\n5. The empirical validation is confined to two small-scale vision datasets (FashionMNIST, CIFAR-10). The empirical validation should include large-scale or non-vision tasks to better demonstrate how FedMuon scales to realistic federated learning workloads or LLM fine-tuning scenarios.\n6. Figure 2 provides an example of how the number of Newton–Schulz iterations $T$ affects performance. However, prior works such as [1] and [2] commonly fix $T=5$, and [3] shows that increasing beyond 5 yields no performance gain but adds unnecessary wall-clock time. Thus, the absence of $T\\geq5$ in Figure 2 makes the empirical validation less convincing.\n\n[1] Jordan et al., Muon: An optimizer for hidden layers in neural networks, 2024.\n\n[2] Refael et al., Sumo: Subspace-aware moment-orthogonalization for accelerating memory-efficient llm training, 2025.\n\n[3] Semenov et al., Benchmarking Optimizers for Large Language Model Pretraining, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)