MuonBP: Faster Muon via Block-Periodic Orthogonalization
Authors: Ahmed Khaled
Kaan Ozkara
Tao Yu
Mingyi Hong
Youngsuk Park
Venue: ICLR 2026 Poster (2026)
Keywords: muon
orthogonalizaton
Primary area: optimization

Abstract: Gradient orthogonalization is a simple strategy that shows great utility in speeding up gradient descent. The Muon optimizer (Keller et al., 2024) combines gradient orthogonalization with first-order momentum and achieves significant improvement in data efficiency over Adam/AdamW (Loshchilov & Hutter, 2019) for language model training. However, when using model parallelism, gradient orthogonalization introduces additional overhead compared to coordinate-wise optimizers (such as AdamW) due to additional gather and scatter operations on gradient matrix shards from different devices. This additional communication can amount to a throughput hit of 5%-10% compared to Adam/AdamW. To remedy this, we propose Muon with Block-Periodic Orthogonalization (MuonBP), which applies orthogonalization independently to matrix shards on each device and periodically performs full orthogonalization to maintain training stability at scale. We show how to adjust the learning rate from the baseline to MuonBP and give convergence guarantees for this algorithm. Crucially, our theory dictates that we use two stepsizes: one for the blockwise orthogonalization steps, and one for the full orthogonalization steps. Our method is simple, requires minimal hyperparameter adjustments, and achieves competitive iteration complexity compared with baseline Muon while providing per-iteration throughput comparable to coordinate-wise methods such as AdamW. When training an 8B model with eight-way tensor parallelism and ZeRO optimizer state sharding, MuonBP achieves 8% throughput increase compared to Muon with no degradation in performance.

Published as a conference paper at ICLR 2026
MUONBP: FASTER MUON VIA THOGONALIZATION
OR-
Ahmed Khaled
Kaan Ozkara AWS
Tao Yu
Princeton University
AWS
ahmed. kha 1 eddprinceeoon edu
kaanozka@amazzon
tacu@amazon.com
Mingyi Hong
Youngsuk Park
University of Minnesota, Twin Cities and AWS
AWS
mhong@umn.
pyoungsueamazon.com
ABSTRACT
Gradient orthogonalization is a simple strategy that shows great utility in speeding up gradient descent. The Muon optimizer (Jordan al., 2024b) combines gra- dient orthogonalization with first-order momentum and achieves significant im- provement in data efficiency over Adam/Adam (Loshchilov Hutter, 2019a) for language model training. However, when using model parallelism, gradient orthogonalization introduces additional overhead compared to coordinate- wise optimizers (such as AdamW) due to additional gather and scatter operations on gradient matrix shards from different devices. This additional communication can amount to a throughput hit of 5%-10% compared to Adam/AdamW. To rem- edy this, we propose Muon with Block-Periodic Orthogonalization (MuonBP), which applies orthogonalization independently to matrix shards each device and periodically performs full orthogonalization to maintain training stability at scale. We show how to adjust the learning and give convergence guarantees for this algorithm. Crucially, our theory dic- tates that we use two stepsizes: one for the blockwise orthogonalization steps, and one for the full orthogonalization steps. Our method is simple, requires minimal hyperparameter adjustments, and achieves competitive iteration complexity com- pared with baseline Muon while providing per-iteration throughput comparable to coordinate-wise methods such as AdamW. When training an 8B model with eight- way tensor parallelism and ZeRO optimizer state sharding, MuonBP achieves 8% throughput increase compared to Muon with no degradation in performance.
from the baseline to MuonBP
INTRODUCTION
First order optimization methods have been the staple in the success of deep learning in the last decade. In particular, Adam (Kingma Ba, 2015; Loshchilov Hutter, 2019a) has become the de facto standard across both industry and academia. Despite numerous attempts to improve upon Adam's performance, has remained unchallenged as the optimizer of choice for training large- scale neural networks. But this wall might be starting to crack. A recent newcomer, Muon (Jordan et al., 2024b), consistently outperforms Adam on various LLM training tasks ranging from small scale benchmarks to larger LLM training setting with up to 1T model parameters Kimi-AI et al. (2025). Muon is more data efficient than Adam, requiring fewer tokens to reach the same validation loss (Liu al., 2025). It also enjoys a higher critical batch size, which allows for further use of parallelism (Essential AI et al., 2025) to accelerate training. Both of these aspects are critical in large-scale LLM pretraining, where even marginal efficiency gains can translate into substantial computational and financial savings.
Muon orthogonalizes the update matrix for each layer before using in a descent step, and can be seen as form of steepest descent (Bernstein, 2025) or as a Non-Euclidean Trust Region
Work done AWS.

---

Published as a conference paper at ICLR 2026
method (Kovalev, 2025). A key disadvantage of Muon, compared Adam, is that orthogonal- ization is not a coordinate- wise operation. Rather, it requires gathering the gradient matrix from different devices whenever model parallelism is used. This introduces additional throughput over- head compared to Adam (Essential AI, 2025). Although Muon is more token efficient, it is strictly slower than Adam on per-iteration basis under model parallelism.
The goal of this work is to bridge this throughput gap while preserving the data efficiency of Muon. To this end, we propose Muon with Block-Periodic orthogonalization (MuonBP, Algorithm 1). MuonBP block-orthogonalizes the matrix shards on each device independently and periodically gathers the shards for a full orthogonalization. any additional communication, recovering the communication efficiency of Adam. However, or- thogonalizing shards only not enough for a competitive performance. We observe this block-only variant (BlockMuon, (Boreiko et al., 2025)) suffers from potentially worse convergence guaran- tee and fails as the models scale up. Hence, we introduce periodic global orthogonalization steps. Combined, MuonBP recovers the performance of Muon with a drastic reduction in communication overhead. Our main contributions are as follows.
the off-period iterations. MuonBP does not require
We propose MuonBP, variant of Muon with local orthogonalization interleaved with periodic full orthogonalization. In the per-oo iterations, MuonBP treats each tensor parallel shard in- dependently and orthogonalizes it separately. In the on- on-period iterations we gather the tensors and do a full orthogonalization. Our experiments with a period of 5 indicate that we recover the performance of Muon with 5x reduction in the optimizer step communication volume.
We provide theoretical analysis of the algorithm (Theorem 2) that shows (a) the blocking period P smoothly interpolates between the convergence rate of Muon and BlockMuon, that (b) we should use two different learning rates in the blocking vs full iterations, and finally (c) gives us guidance on how to scale the learning rate when using block orthogonalization.
Empirically, we show that MuonBP converges faster than the baseline (non-blocking) Muon algo- rithm (Jordan et al., 2024b). Dion (Ahn et al., 2025), AdamW (Kingma & Ba, 2015; Loshchilov & Hutter, 2019b), and BlockMuon (Boreiko et al., 2025) in practical pretraining tasks in terms of the wall-clock time. We observe that our method recovers the original Muon's 's performance with a up to 8% increase in throughput under layerwise sharding and tensor parallelism.
We briefly outline the rest of this paper. In Section 2, we provide necessary background for a steepest descent view of Muon, which will be useful for other sections. We discuss related work and compare our work to few others who examined orthogonalized updates in large scale distributed settings. In Section 3, discuss
algorithm with convergence analysis, our goal is to analyze the effect of periodicity in the behaviour of our algorithm. Finally, in Section 4, we examine our algorithm in billion-scale training settings and compare other baselines in terms of accuracy and throughput.
2 BACKGROUND AND RELATED WORK
Training modern large-scale machine learning models has historically relied on Adam/Adam (Kingma & Ba, 2015; Loshchilov & Hutter, 2019a), but this might be start- ing change. Recent progress the AlgoPerf benchmark (Kasimbeg et al., 2025) and Moddeed-NnnGGPT speedrunning (Jordan et al., 2024a) show that alternative second-ordder-isspieed optimizers Shampoo (Gupta al., 2018) and Muon (Jordan al., 2024b) can be competitive or better at scale. In this section we first give a review of the algorithmic framework we use to understand it. Then, we consider the relative cost of Muon compared Adam from a systems perspective. As we are particularly interested in communication efficiency, we also review various tools from the literature on communication- efficcient optimization.
2.1 ALGORITHMIC FRAMEWORKS FOR OPTIMIZER ANALYSIS
The framework of steepest descent under non-Fuclidean norms allows us to study different opti- mizers in a unified and princpled manner (Bernstein & Newhouse, 2024b). This framework is very useful in analyzing Muon as it (a) clarifies what Muon is optimizing for, and (b) gives common template to compare Muon and its variants to coordinate-wise methods like Adam. Steepest de- scent posits that at each step of optimization, if is the current model, we choose the next model as x. + dar where dr minimizes f(x) + (VI(z), x Ar) + 2 |r| where f is the loss function
2

---

Published as a conference paper at ICLR 2026
and (z) is its gradient. The choice of norm ||-I| yields different optimizers. Choosing the Eu- clidean norm gives us gradient descent, and choosing the Eo norm gives us scaled sign descent, arg mree (f(x) + (f(z)+
When expo- nentially moving averaging is turned off in Adam, it reduces to unscaled sign descent (Bernstein & Newhouse, 2024b) and there is some evidence that Adam's 's superior performance explained by this connection (Kunstner et al., 2023). The steepest descent view results in the additional scaling by [V
(V)(z), Az) + 21
sig
| in the numerator, which means we have different parameter update norm every iteration (x (z))l)) and don't 't have the same connection Adam.
We can instead explicitly control the parameter update norm by using the Non-Euclidean Trust Region (NTR) formulation. This is the formulation used by Kovalev (2025): iterate x; NTR minimizes the first-order model of over a norm ball {A :|| A I| 1/A}, which yields the steepest- descent direction in that norm. Concretely, Ar arg min | (f(z) + (Vf(r), + Ar)).
For |1 this recovers (unscaled) sign descent. The NTR formulation also allows for elegant theo- retical analysis, including incorporating algorithmic techniques such momentum (Kovalev, 2025). For these reasons, we will adopt the NTR framework as our algorithmic template in Section 3.
Changing the norm used in either steepest descent or NTR from II-Ilxo to any other norm opens up large design space of optimization algorithms. For example, we may use different norms for different parameters in neural network depending on whether they are vectors matrices. For matrix parameters X Rmxx, using the operator norm | suP=ER" ||x=ll gives
Muon.
||=|||
arg min AX| II op < (f(X) + ( - - Orth(V((X)),,
1
(1)
where Orth(U)) = (UUT)-À and i denotes the Moore-Penross pseudoinverse. If we Newton- Schulz iterations (Algorithm 2) to approximately compute the orthogonalization and apply the max- imization to a running momentum buffer instead of the gradient directly, we obtain Muon (Jordan et al., 2024b). Bernstein & Newhouse (2024a) argue for using layer-dependent norms depending on the expected norm for the inputs and outputs of each layer. In practice, the choice of norm is also mo- tivated by empirical performance (Jordan et al., 2024b). If we instead use the l + lz- fz-induced norm, G.nn), [G,,a This was used for the first layer in Scion (Pethick al., 2025) and for every layer save the last in SCALE (Glentis et al., 2025). Glentis et al. (2025) show that this using column normalization with momentum on the last layer allows for training transform- competitive with Adam and Muon for up to IB parameters scale.
we obtain column normalization. That is, given a gradient matrix G [G., G.
we set AX - G
G..
G...
2.2 A SYSTEMS PERSPECTIVE ON OPTIMIZER COSTS
The choice of norm dictates the operation to be done at every step and its structure (e.g. coordinate- wise matrix wise). This, in turn, determines both the computational cost of the update and whether distributed execution requires cross-device collectives.
Computational costs. The computational cost of running an optimizer step is just the number of floating point operations (FLOPs) we need to do per step. For methods that only perform coordinate- wise operations (such Adam), this cost just scales with the number of parameters in the network. For methods like Muon that have to perform more sophisticated operations, this is higher. Con- cretely, for a parameter matrix of size m n, the per-step cost of (stochastic) gradient descent with momentum is just 2mn floating point operations (FLOPs) and 4mn FLOPs for Adam. In compar- ison, orthogonalization is more expensive. Using K Newton-Schun Appendix), the total is 2mn + 2K (2nm² + m²) FLOPs assuming without loss of generality that m S (Jordan et al., 2024b). Some approaches to reducing the computational cost of orthogo- nalization include tuning a, b, in Algorithm 2 to reduce the number of steps needed (Jordan et al., 2024b) or using adaptive per-step a, b, c (Amsel et al., 2025).
iterations (Algorithm 2 in the
In some large-scale pretraining regimes, the computational cost of running the optimizer steps might be small relative to the forward and backward passes in backpropagation. A common rule of thumb is fwd+bwd computation x 6NT FLOPs for a dense network with N params and input size of T tokens. For larger batch sizes, this becomes more dominant as the optimizer step is independent of the input size.
3

---

Published as a conference paper at ICLR 2026
Communication costs. Modern neural networks are trained with combination of data and model parallelism. Data Parallelism (DP) replicates model parameters, gradients, and optimizer states across the communication network but passes different data batches to each DP group. The gradi- ents are synchronized across the different devices before applying the optimizer step. While this replicates the optimizer step computation across different DP groups, it adds no additional com- munication cost. In contrast, model parallelism typically will shard some or all of these tensors. Tensor Parallelism (Shoeybi et al., 2019) (TP) shards the model parameters for both storage and computation; This sharding is done along one or more dimensions (e.g. row, column) of each ten- sor. Pipeline Parallelism (Huang et al., 2019) (PP) also shards model parameters for both storage and computation, but does so by dividing the layers among different PP groups. The Zero Redun- dancy Optimizer (Rajbhandari et al., 2020) (ZeRO), Fully Sharded Data Parallelism (Zhao et al., 2023) (FSDP), and FSDP2 (Liang et al., 2024) shard model parameters either by layer or on the first dimension, but do that for the purpose of saving memory. Before doing the forward/backwwar computation involving certain layer. ZeRO/FSDP2 undo the sharding they apply first. In practice, we often apply a combination of parallelism strategies for maximum compute utilization, with e.g- TP applied between different devices the same node and ZeRO/FSDP/FSDP2 applied between different nodes. Table 4 in the supplementary summarizes what the different parallelism strategies shard.
Communication cost of Muon. There are several strategies for parallelizing Muon and they deter- mine the communication costs involved (Essential AI, 2025). If we use TP or FSDP2, we have to do an additional all all-gather across the TP/FSDP2 groups to gather optimizer states (momentum buffers). A naive all-gather would force us to orthogonalize the same matrix in parallel which is redundant. A better alternative is to use two all-to-all communications to redistribute different layer tensors. This suffers from two issues: (a) we still have do two additional collective operations, and (b) if the number of matrices to be orthogonalized is larger than the number of GPUs, some GPUs would sit idle. If we use ZeRO, then the fact that the optimizer states, parameters, and gradients are already sharded layerwise helps greatly: we do not need to do all- aal-gather across the distributed optimizer groups and can apply orthogonalization layerwise in parallel. In this case, the only extra commu- nication cost we suffer from comes from all- all-gathering across the TP groups For 8B parameter Llama-style transformer, this gives a throughput reduction of 8%-10%
This additional communication burden has motivated the development of Dion (Ahn et al., 2025) and, concurrently to our work, Boreiko et al. (2025) introduce a variant of BlockMuon (Algorithm with P x) We compare against BlockMuon in detail in the next section and in the experiments. Dion (Ahn et al., 2025) maintains a low-rank approximation of the momentum matrix and distributes the orthogonalization process. For large enough batch sizes. Dion's computational cost is perfectly divided by the number of devices and its communication cost scales with the smaller rank. In Section C we study the computational cost of Dion in more detail and compare it to our proposed algorithm MuonBP.
Tools for communication efficiency. Under data parallelism, the gradient synchronization step (which happens regardless of which optimizer we use) can itself be expensive, particularly in highly distributed or federated settings (Kairouz et al., 2019). Researchers have developed techniques like gradient quantization (Alistarh et al., 2017; Horváth et al., 2019), intermittent communica- tion (Konečný et al., 2016; Stich, 2019; Douillard et al., 2023), and low-rank compression (Vogels et al., 2019) to reduce this cost. MuLoCo (Thérien et al., 2025) applies both gradient quantization and intermittent communication to reduce the gradient synchronization cost under data parallelism, while Dion (Ahn et al., 2025) optionally allows for reducing gradient synchronization cost via low- rank compression as well. In this work, we use the same technique of intermittent communication to reduce the communication costs arising from model parallelism. Our algorithm can be combined with any of the aforementioned techniques for data parallelism as well.
Many of the same computational and communication constraints discussed above also apply to other gradient preconditioning algorithms, c.g. Shampoo (Gupta et al., 2018), K-FAC (Martens & Grosse, 2015), and ASGO/One- -ideed Shampoo (An al., 2025; Xie al., 2025). Distributed Shampoo (Shi et al., 2023) uses blocking, intermittent preconditioner updates, and anyer-wise sharding similar to ZeRO- 1/FSDP to reduce the amount of communication.
4

---

Published as a conference paper at ICLR 2026
3 ALGORITHMS AND CONVERGENCE
Our starting point is the observation that column- or row- normalization can be viewed as or- thogonalization applied row-wise normalization and column-wise normalization would be orthogonalizing submatrices of dimensions p q each where p - m and q S n. This has two benefits,
submatrix of size x or 1 n. An intermediate method between
We reduce the amount of floating point operations per Newton-Schulz step from 2(2nm + m²) 2(2pq² + q²)) x mn 2(2mnq + m floating point operations (assuming without loss of generality that p < Q). For example, the MLP layers in Llama 405B (Grattafiori et al., 2024) have m, n E {53248, 16384). Here, orthogonalizing submatrices with 8-way TP gives speedup of R 2.36 for the up- up-projection and Rs 9.06x for the down-projection per Newton-Schen Schullz step relative to full orthogonalization.
we use blocks corresponding to the model parallelism used, we can entirely eliminate orthogo- nalization's communication overhead under any regime. We discuss this in more detail below.
How blocks align with model-paralle shards. We divide each parameter, gradient, and opti- mizer state tensor into blocks and define each of these blocks to be exactly the tensor shard that resides device under the chosen parallelaal layout. This makes the communication pattern explicit and ensures that a "block" step never requires cross-device traffic.
Tensor Parallelism (TP). In Megatron-style (Shoeybi et al., 2019) column (alum--parallll linear layers, weight W m is split by columns across TP ranks, so each rank holds W (m//c) and produces a local gradient shard (l1 E (m/ee) A block is GO)); bookk-orhogonnllzztion acts on m x (n/c) matrices and needs no gather/scatter. In row-parallel layers, W is split by rows across r ranks, so each shard is ((m/r) x n) and the block is G E @R(m/r)xn For hybrid 2D TP (row x column), the global W is partitioned into an r x grid of rectangular shards ((m/r) x (n/e)). TP is often applied not just to the linear layer but also to the attention weights well, and the same discussion applies.
FSDP2 (dim-O sharding). When parameters are sharded only for memory (layer/dim-O), each rank holds contiguous slice along the first dimension. During the optimizer step, block denotes this local slice; thus block-orthogon ortooooogoooniizattionn again requires no parameter all-gather. The same definition applies under TP+FSDP: the block is the intersection of the TP and FSDP partitions, shard.
i.e., single
m
In order to develop algorithms that minimize communication, we want to do block-wise operations much as possible and keep "global" operations minimum. To this end, we analyze the variant of Muon that only does blockwise operations in Section 3.1. Our analysis shows that in the worst case, the convergence of this variant might be much worse than full Muon. To remedy this, we develop and analyze our block-periode variant in Section 3.2.
3.1 BLOCK ORTHOGONALIIZTTION
BlockMuon (Algorithm 1 with P = oo) applies orthogonalization to these blocks, in parallel, on different devices (Boreiko et al., 2025). This removes the need for any added communication and reduces the computational cost of orthogonalization. To better understand the convergence of Block- Muon, we analyze the algorithm under the assumptions of smoothness, bounded stochastic gradient variance, and norm equivalence characterized by p. We state our assumptions more clearly below.
Assumption (Smoothness). We assume that f Rm + R is L-smooth with respect to a norm I--I. That is, let II- be the corresponding dual norm, then for all X, m R we assume [V[(X) - v Ill. < L|XX- - -Y||-
Assumption 2 (Bounded Variance). Suppose that the stochastic gradients G(X) are (a) unbiased,
E; [)] fX)), and (b) have bounded variance Eg [| {(x)] d²
Assumption (Norm Equivalence). The norm ||-I] satisfies ||X]| S ||p for some p > 0.
As mentioned before, we will use the Non-Euclidean Trust Region (NTR) template (provided next) to analyze both algorithms.
M, = 1Mr- + Ge,
X
arg min (M, X X(), Xxll
(NTR)
5

---

Published as a conference paper at ICLR 2026
where G2 is stochastic gradient with expectation f(()). This framework was adopted for the convergence analysis of Muon by Kovalev (2025) and the next theorem is a slight modification of Theorem 2 in their work. The proof is also similar to (Li & Hong. 2025, Theorem 2.1).
Theorem 1. Suppose that the function f satisfies Assumptions 1 to 3 and that f is lower bounded by Sa.. Then for any dj > O and μa E [0. 1] the iterates generated by equation NTR satisfy
f(Xo) - f- + 3VI(f(Xo) -fa)
P
E T-1 (1
1/T
T
1- P
2(1 - -p)po
Lap
1
LM
+ T
+ 1-p
+ po
+
1+
Theorem 1 applies to Muon, since under ||-11 = ll-llops eq. (NTR) reduces to orthogonalizing mo-
mentum. The next lemma shows that Block- can also be studied in the same framework.
Lemma 1 (Dual of the Block-Spectral Norm). Let X E Rmxx be partitioned into r x blocks. Define the block-spectral norm as (X) sisis 15/5e || Its dual norm is B (X) || is the nuclear norm.
EngliX,, llop............. where
BlockMuon is just eq. (NTR) with ||||| = B()). To compare between the convergence of Muon and BlockMuon, we consider the simplified setting when o = O and apply Theorem 1. Minimizing
Equation (2) over 77 and μ1 yields 7op,,* (-- and = 0 and the convergence guarantee Thop
[XV/(x>)llop+++ / (2 where Lop the smoothness constant of f with respect the
operator norm. Similarly, the best guarantee for BlockMuon is achieved by 1hllock,* / f(Xo)-f-
62'm
and is Bd (VI(X))) S / where x' = arg min, B" (V((((X)) and LB is the smooth- ness constant of f in the block norm B(.). To compare the two guarantees for BlockMuon and Muon, we use the facts that I1-1lop. S (-) and LB I rcLop (proved in Section A.1) to get
T
| (Xx I <v V /2L011(Xo)- S vrry (f(oo) f-). Thus, under the same operator norm T
metric, BlockMuon's 's best point X has a gradient dual norm that is at most a /rc factor worse than Muon's best point X, in the worst case; when LB Top (e.g., curvature well captured by blocks), the two bounds match up to constants. Note that in the former case, we would have LB &J (rp) and Nop. Yblookk- = =V Lp = vrc. Whereas, in the ideal scenario when Nop.. z +b the optimal Lop
learning rate would be the same for both algorithms. Thus the optimal ratio of the learning rate of Block-Muon and Muon is between 1 and 1/ \/v
The picture we see is thus clear: BlockMuon is faster on a per-step basis, as we do not need to perform any additional communication over coordinate-wise methods, but this comes at the cost of a worse convergence guarantee (by a factor of /rc in the worst case). It seems straightforward then that we should minimize wall-clock time by choosing block sizes and e that balance this tradeoff. While this is theoretically plausible, in practice the block sizes are naturally a function of network topology (i.e. FSDP TP degrees) and changing them would add more latency and require redistributing tensors to and from their original layouts.
3.2 BLOCK- PERIODI ORTHOCONALIZATTIO
We instead offer another alternative tuning block sizes that (a) has a simple implementation, and (b) gives us clear tunable knob that smoothly interpolates between BlockMuon and Muon. Given period P, Muon with Periodeii orthogonalization instead uses BlockMuon for P-1 steps and then uses full orthogonalization for one step. If P = 1 we get Muon, while if P + Muon. Using P in between both extremes allows us to balance out the tradeoff between iteration complexity and per-step communication cost. We state the algorithm in full below Algorithm 1. Note that we use two stepsizes, Thull and T/block. depending on whether we communicate during that step or not. We will later show this gives better convergence rate than just using stepsize.
get Block-
The next theorem studies the convergence of this algorithm and allows us to make the above intuition rigorous.

---

Published as a conference paper at ICLR 2026
Theorem (Convergence of MuonBP). Suppose that f satisfies Assumption with respect both the operator norm lop with constant Lop and the slock-spectral norm B(-) with constant LB, and that Assumption 2 holds. Assume is lower bounded by fa and let Do f(Xo) f.. Fix a period 2 momentum E [0, 1), and two stepsizes TJfull >> and 7/block > 0. Define M/max max Tblock). and
Thall + (P-oa(
n
Q LopPRull
LP
max Lop- Tblock v LB}:
2P
2P
2/
(Top Thull max (bboock /rc, Manl
LB Tblock max (alll Muluck (P-
1-P
P
P
Then for any horizon T divisible by the iterates of Algorithm satisfy
min.-o....T T E [IV
Ao 97
1(1 1(1-4)o Tmnax
Guv A (I-p)07 + Q+R R 20
(3)
To simplify the comparison we consider the noiseless case where = and the optimal momentum parameter then l 0. To minimize Equation (3), define the harmonic-average smoothness
LBP by Lip Lop + P-1 p-- Lg' The optimal stepsizes are then Thll
V 2A0 LEP and
Lup
Thlock In V 20d T LBP and the convergence rate is minkeT l (XX))llop.+
2dmLnr There-
fore, the convergence of BlockMuon, Muon, and MuonBP is proportional to LL LLB.VLoO Lop. and
/ LBP. respectively. is easy to see that Lop < LBP < LB and thus the convergence rate of MuonBP is in between Muon and BlockMuon. The period P acts tunable knob that lets slide between the two extremes and this is directly reflected the convergence rates we obtain. Observe two stepsizes Mhall and Thlock depending we are applying full orthogonalization or block-wise orthogonalization. On the contrary, if we were to force using single stepsize for steps 7): = 7/. the optimal choice becomes 17* T LBra
that get this rate, it is crucial that
whether
2A.
with LBP2 Lue LB, yielding convergence Since LBP the weighted harmonic mean and LBP2 is the weighted arithmetic mean of the same constants, we have LBP < LBP2 with strict inequality unless Lop LB. so tying the stepsizes generally yields worse convergence. Observe that, in our previous comparison, the optimal ratio between 7block and Mall is between and 1/vrc.
proportional to TBr2 rather than Lap-
Algorithm 1: MuonBP
M (m) +- 0 for all devices
for ( + 0 to
do for each device m do in parallel
3.3579
3.3635
3.3678
3.3659
3.46
Get local shard ((m) of the full gradient Gt
Block Muon Degres
3.3649
3.3832
M((") <- HMM (m) + G((")
3.42
3.3731
3.4005
3,4046
3.4127
if't mod p= then
33400
Gather (m)) mm form full Me
UL + Orthogonalize-via-NSt
16
3,3850
3.4561
3.4489
3.4794
X(+1 + Maabi
3.36
10
else
2.0
4.0
Block Period
8.0
16.0
11
for each device m do in parallel (((m)
Figure 1: Validation loss as a function of orthogonalization period for different TP de- grees (280M model).
Orthogonalize-via-NS// (M
x x(m) (+1
x(((m
Mbsoock U. (m)
14 return XT
AdamW W learning rate transfer. Liu et al. (2025) introduce learning rate scaling rule that allows reusing the AdamW learning rate for Muon by matching the root-mean square norm the updates to be the same Ad To ensure that the updates have RMS B, they scale the update matrices

---

Published as a conference paper at ICLR 2026
by 8- v 1max(mm m, n. where X are the update matrix dimensions. Following our theorem above, which shows using different learning rates for the blocking and non- non-blocking matrices is ideal, we also adopt this rule and scale the updates by the dimensions of the smaller matrix on block steps and the dimensions of the full matrix on non-blocking steps.
Communication cost of MuonBP. optimizer-stan all-gather/scatter. On a full step, it temporarily gathers shards to materialize M, (or Go) per tensor, performs global orthogonalization, and scatters back.
On block step, MuonBP orthogonolizes the local shard without
Choice of period. Ideally P minimizes the wall-clock time to reach accuracy 6, balancing the iterations (e, P) needed and the expected wall-clock time per iteration Twall (P). This tradeoff depends on network speed and tensor size, so in practice we test P on short runs. We found P = 5 balanced this well empirically.
4 EXPERIMENTS
We conduct experiments in two main settings both of which are Llama-style language model pre- training setups. Firstly, we use setting with FSDP2 and TP where we study the effect of varying blocking degree and orthogonalization period on convergence under extensive hyperparameter tun- ing. Then, we benchmark our method with small 160M model setup from (Ahn al., 2025); and compare MuonBP to Adam W. Muon (with full all all-gather at every step), BlockMuon, and Dion. FSDP2 shards optimizer states in Oth dimension to different workers, resulting in increased com- munication for Muon. In the second setting we use ZeRO layer-wise (Rajbhandari et al., 2020) optimizer state sharding and TP. Here, we primarily compare MuonBP (Algorithm 1), BlockMuon (Algorithm I with P = oo), and baseline Muon (with full all-gather every step), under billion scale model sizes and longer tokens. Both experiment groups are meant to showcase the accuracy and throughput improvements brought about by algorithm in realistic pretraining settings.
4.1 TRAINING WITH DIM-0 DATA SHARDING
Experimental setting and hyperparameters. We augment the Modded-NanoGPP codebase (Jor- dan et al., 2024a) with SimpleFSDP (Zhang et al., 2024) and TP (Shoeybi et al., 2019) via the DTensor API integrated into PyTorch 2.0 (Liang, 2023). We use the FineWeb dataset (Penedo et al., 2024) for the experiments in this section.
Figure/ shows the effect of varying both the TP degree and the period of orthogonalization on the final validation loss achieved. We use the modernized GPT--style architecture of Modded- NanoGPT (Jordan et al., 2024a) for this experiment. We use 12 layers, attention heads, and a model dimension of 768. We use the smaller model size (280M) in order to run extensive grid search. Following the codebase, we use separate learning rates for Adam (applied to 1D parameters and the input embedding) and Muon, and do not the RMS norm matching trick of Section 4.2. We tune the Adam/Muon learning rates over the grid (0.0001, 0.001, 0.01, 0.1, 0.5, 1, 2, 8) * base where base 0.012 for Adam and base 0.08 for Muon. We that decreasing the block period directly decreases the loss for all the degrees we consider, with the effect most pronounced at the highest degrees.
We use the Dion codebase (Ahn et al., 2025) for the second comparison and train a 160M pa- rameter model with a batch size of 1024, sequence length 1024, model dimension 768, 12 layers and 12 attention heads per attention layer. We use the WSD schedule with no warmup and a 20% cooldown. The learning rate is 0.02 for all methods (with AdamW rms norm matching) except for AdamW, where we found by grid search that 0.008 performed better. We use TP degree of 2 and FSDP degree of 4, and use Lion as the scalar optimizer in line with the codebase. The throughputs for all the methods were similar at this scale, although they were significantly lower compared to throughputs on Megatron-LM with layerwise sharding. We believe more experiments are needed to compare against Dion, particularly to integrate it into widely used open source frameworks such as Megatron-Lee M.. We also plot the loss curves in Figure 11 in Section B. Section C also gives a brief comparison of the cost of rumning MuonBP Dion from theoretical perspective. We note that only conducted the experimental comparison between Dion and MuonBP on a small scale, and that more work is needed to compare in this setting on larger scale. We leave this future work.
8

---

Published as a conference paper at ICLR 2026
Muon
BlockMuon
MuonBP
Dion
Adam
Min Validation Loss
3.36
3.36
3.34
3.37
3.62
Min Training Loss
3.02
2.97
2.94
2.95
3.21
Throughput (TFLOP//GPU)
50.90
51.77
51.40
45.64
52.80
Table 1:
Traininggvalidationn losses and throughput on 160M model trained with TP=2 and FSDP=4.
4.2 TRAINING WITH LAYERWISE SHARDING
Experimental setting and hyperparameters. We built upon the Distributed Muon implementation of (Liu et al., 2025) in the Megatron-LM framework (Shocybi et al., 2019) (which corresponds to ZeRO-2 optimizer state and gradient layerwise sharding) and modified it to support block-wise ten- parallel orthogonalization with periodic full orthogonalization. Note that and ZeRO interchangeably depending on the framework, as their sharding strategies are equiva- lent up minor implementation details. We used Llama-style model architecture (Touvron al., 2023a;b) with RoPE (Su et al., 2024), SwiGLU activation (Shazeer, 2020), and mixed-precisiee precision training (bf16 computations with fp32 master weights). We use the Llama tokenizer (Grattafiori et al., 2024) on the Open Web Text dataset (Gokaslan et al., 2019) for experiments at the 0.9-1.2B scale and the Fin data (Penedo al., 2024) for experiments the scale. For experi- ments in this section, we used nodes that have 8xA 100 GPUs with 40GB of RAM each.
the terms FSDP
We train models in the following scales and settings: 960M and 1.26B, 1.26B with extended training (3x Chinchilla tokens), and 8B parameters with large (1.2 10-3) and small (0.6 x 10- learning rates. The models below 8B in scale 128 sequences and each run takes place on a single node with 2 DP groups and TP nodes per group. The 8B model uses batch size of 256 sequences with 4 DP groups distributed across nodes and 8 TP nodes per group. As discussed in Section 3.2, AdamW RMS norm matching for learning rate scaling (Liu et al., 2025). All of the architectural details are provided in Table 5 in the supplementary material and more details on our choices of hyper- parameters, learning rate, and learning rate scheduling are found the appendix. We also present experiments on scheduling the pe- riod parameter in Appendix D. We do the two learning rate runs at 8B scale to show that with the larger base learning rate, even after adjusting for blocking with the RMS norm matching, BlockMuon becomes unstable.
Mi.3n Beoolitoon
8000
batch size of
7000
8000
3000
matring mrpe
Figure 2: Parameter norm vs iteration of competing meth- ods.
Muan
Muon
BlockMuon MuonBP
BlockMuon
MuongP
40
40
15% fasto!
aaldation pol
validation ppl
7% lower DGA
Inoer
30
30
25
20
15
40000
80000
120000
40000
80000 runtime (seconds)
120000
runtime (seconds)
(a) Validation ppl. vs. runtime
(b) Validation ppl. runtime for large Ir
Figure 3: 8B model validation perplexities. Comparison of Muon, BlockMuon, and MuonBP across wall-clock time. For target validation perplexity our method 10 - 13% faster in terms of the wall-clock time to reach it, and for given time point before the learning rate decay our method results in - 7% lower perplexity compared to the baseline.

---

Published as a conference paper at ICLR 2026
Table 2: Validation and training perplexity (lower is better). Columns show models; each model has validation and training sub-columns. Best perplexities within each model size are in bold.
960M
1.2B
1.2B*
8B
8B
Method
Val
Train
Val
Train
Val
Train
Val
Train
Val
Train
Muon
15.33
13.44
14.13
12.83
12.62
10.88
12.90
11.74
13.40
12.39
BlockMuon
20.29
18.08
16.28
14.86
13.29
11.51
13.68
12.62
24.68
23.17
MuonBP
15.12
13.21
13.78
12.44
12.45
10.71
12.77
11.59
12.97
11.93
Adam 22.51
20.16
15.03
13.25
14.47
13.48
2 Three-times data with large learning rate (small learning rate for Adam). "Large learning rate.
Results. Resulting perplexities are summarized in Table 2. The loss curves for all models are deferred to Section B. Table 2 shows that BlockMuon performs worse in both training and validation loss across all model scales considered. This still holds true for relatively long ( 3x Chinchilla) training. the parameter norms grow lot more for the fully blocked version of Muon compared to either baseline or blocking with intermittent orthogonalization. Note that this happens despite the fact that we use Adam W RMS norm matching scaled with the dimensions of the sliced blocks (as outlined in Section Section 3). We observe that we have to use smaller learning rates to keep BlockMuon stable compared to Muon and MuonBP; This is potentially a symptom of the instability we observe when using Blon. We do not observe instability when using smaller learning rates (Figure 10), but then baseline Muon, BlockMuon, and MuonBP all lead to the same suboptimal performance. Adam consistently underperformed across all scales, failing converge at the larger learning rates (hence its absence some columns). However, the performance gap between Adam and Muon also narrowed substantially with scale: the relative perplexity improvement decreased from 31.9% at 960M (22.51 vs 15.33) to 10.9% at 8B (14.47 vs 12.90), indicating that Adam's disadvantage diminishes at larger scales. This underscores the importance of MuonBP, at larger scales it is then critical to minimize throughput losses as much as possible-even eeeeeeeeeenn a 7% improvement in throughput would be highly significant. In Figure 3, we plot the validation ppl vs wall-clock time. We characterize our method's performance with respect to two related metric: firstly, given target ppl value our method reaches considerably faster in wall-clock time; secondly given a runtime budget our method results in lower validation ppl (we give exemplary points in Figure 3). These two views indicate the usefulness of MuonBP in practical scenarios.
Interestingly. overall, our method outperforms Muon despite doing less num- ber of full orthogonalization, we believe this may be due to a regularization ef- fect due to intermittency, we leave the analysis of this behavior as future work.
Table 3: Average through- put (TFLOP/S/GPU) for each method and model.
Throughput. We report throughput numbers in Table 3. We ob- serve similar throughput across methods in smaller scale experi- ments layer-wise sharding results in minimal all-gathers for the Muon. However, as the model scale increases the effect of all- gathers makes its presence felt. Consequently, in 8B model setting we observe a ~ 8% increase in throughput for our method com- pared to the Muon without any degradation in performance. This translates to hundreds of thousands of dollars saved in training costs in today's 's large-scale pretrain- ing runs.
Method
960M
1.2B
8B
Muon
112.97
118.29
105.09
BlockMuon
115.43
120.14
114.75
MuonBP
113.54
119.79
113.37
Adam
117.21
120.20
117.30
CONCLUSION
We have introduced a new algorithm, MuonBP, and analyzed its convergence properties. MuonBP shows promising performance in training models up to the 8B parameter scale compared to Muon, BlockMuon, and Adam. There are many questions still left: for example, while we empirically demonstrate promising initial results for scheduling the period P over the duration of training in Appendix D, how we might adaptively tune it based on observed properties remains an open question. Exploring the use of block orthogonalizztion with expert parallelism is also important topic we leave to future work.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Section and the Appendix provide all details necessary to reproduce the theoretical results presented in this paper. Our code-base built upon publicly available frameworks (Megatron- LM (Shoeybi et al., 2019) and Modded NanoGPT (Jordan et al., 2024a)). Section 4 and the Appendix describe the experimental settings and hyperparameters in detail. To further sup- port reproducibility, we will release our implementation and training scripts upon publication at gi github. com/ r /munnbp.
REFERENCES
Kwangjun Ahn, Byron Xu, Natalie Abreu, and John Langford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv22504005295, 2025. URL
05295v2.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong. Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 1895-4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 626.:///2222...mmmmmmin https: http:////aaaaaahhhaocc
URL
org/ 2023. emnlp-ma n. 2 298/.
Dan Alistarh, Demjan Grubic, Jerry Li. Ryota Tomioka, and Milan Vojnovic. QSGD: communication-effiieee SGD via gradient quantization and encoding. In Isabelle Guyon, Ul- rike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1709-1720, 2017. URL https: http://poooceeeeeeee/eeeeeececce ats.raaet.....t.tttaaaaat.....aa
www.didinnnnnnnnnnn.ooemom.m..ee .neurpp....... cc/ pttppppppe//1//////////
hash/
Noah Amsel, David Persson, Christopher Musco, and Robert M. Gower. The polar express: Optimal matrix sign methods and their application the Muon algorithm. arXiv preprint 0rXiv:2505.16992 2025. URL http: / arxiv. org/ abs 2505 16932v2.
Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, and Tong Zhang. ASGO: Adaptive structured gradient optimization. arXiv preprint arXiv:2503. 2025. URL http: //////aaa //aaaiiv. org/ wwwpor////////o/eeeee../eeeeeeee httr/bbs/25500 22////2553..2222222222 20762v2
Jeremy Bernstein. Deriving Muon, 2025. URL
seriving-muon.
Jeremy Bernstein and Laker Newhouse, Modular duality deep learning. arXiv preprint arXiv:2410.21265, 2024a. http:////aarx http:///aaaii..../////////////// 0.21265v2.
Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: an anthology. arXiv preprint, abs/2409.20325, 2024b. URL
org/ abs /20092409900 .23255.
Valentyn Boreiko, Zhiqi Bu, and Sheng Zha. Towards understanding of orthogonalization in muon. In High--dimensional Learning Dynamics 2025, 2025. URL
Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. DiLoCo: Distributed low- communication training of language models. arXiv preprint, abs/2311.08 2023. URL
httts::/arxivv/ org.
awwp////////////////ooo///sssa// alss 2tts/2211.000.... 08105.
Essential AI. Layer sharding for large-scale training with Muon. Essential Al Blog, May 2025.
URL /attps://wett essent atl///ogl..////loooo//oooooooo infra.
Essential AI, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma. Anil Thomas, Ashish Tanwer, Darsh Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik
11

---

Published as a conference paper at ICLR 2026
Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, and Ashish Vaswani. Practical efficiency of Muon for pretraining. arXiv preprint DrXiv:2505.02222 2025.
Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. A minimalist optimizer design for
LLM pretraining. arXiv preprint arXiv:2506. 16659, 2025. URL http:://axx...orrg/////// abs/
2506. 1669vv1.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. OpenWebText corpus. http: //Skylion007 github. io/ www.o/peeeeeeeeetorrpr.ooo 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXi::2407.2178. 2024.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor opti- mization. In International Conference on Machine Learning, pp. 1842-1848 1850. PMLR, 2018.
Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232-76264, 2024.
Samuel Horváth, Dmitry Kovalev, Konstantin Mishchenko, Sebastian Stich, and Peter Richtárik. Stochastic distributed learning with gradient quantization and variance reduction. arXiv preprint, abs/1904.051 05000400511,,... 15, 2019. URL https: /aaaiiiiiivv./...//aaaaaaaaaa org/ abs 1904. 05115.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA, 2019.
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, Cembear.bsky.socii. Boza Vlado, You Ji- acheng, Franz Cesista, Braden Koszarsky, and Grad62304977. modded-nanogpt: Speedrun- ning the nanogpt baseline, 2024a. URL https:/ modded-nanogpt..
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024b. URL https: / //kellerjordaa sarrdan.ithhun github. / posts /muon/.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi. Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Ad- vances and open problems in federated learning. arXiv preprint, abs/1912.04977.. 2019. URL https: arxiv.... org/ abss/ / 1912. 04977.
Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sas- try, Mark Saroufim, Boyuan Feng. Less Wright, Edward Z Yang. Zachary Nado, et al. Ac- celerating neural network training: An analysis of the algoperf competition. arXiv preprint 2501: 2025.
Kimi-AI, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding. Mengnan Dong, An- gang Du, Chenzhuang Du, Dikang Du Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu,
12

---

Published as a conference paper at ICLR 2026
Zhenxing Hu, Weixiao Huang, Zhiqi Huang, Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tian- wei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma. Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang. Lidong Shi, Shengyuan Shi, Feifan Song, Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung. Heyi Tang, Jiawen Tao, Qifeng Teng, Chensi Wang. Dinglu Wang, Feng Wang. Haiming Wang. Jianzhou Wang. Jiaxing Wang, Jinhong Wang. Shengje Wang. Shuyi Wang, Yao Wang. Yejie Wang, Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang. Zhexu Wang, Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jin- jing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang. Zhen Yang, Zhilin Yang, Zonghan Yang. Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang. Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang. Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi K2: Open agentic intelligence. arXiv preprint Xrv:::2507.2000 20534, 2025. URL https: / wwaaaaiivvoog../.aaaaeeea abs / /207 20534.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arXiv. org/ abs / /11221111111 12..2.66 6980.
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated Learning: Strategies for Improving Communication Efficiency. In NIPS Private Multi-Party Machine Learning Workshop, 2016.
Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. arXiv preprint arXivv2503. 2025.
Frederik Kunstner, Jacques Chen, Jonathan Wilder Lavington, and Mark Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//oppn/evieieeee net forum? (atmmmddda55x0@@aa5500
Jiaxiang Li and Mingyi Hong. A note on the convergence of muon. arXiv preprint WwXiv:2502.02900 2025.
Wanchao Liang. Pytorch dtensor rfc. GitHub Issue, 2023. URL https://github.. com/
pytorch/ http:pyyoocchshsc
188888 GitHub Issue #88838.
Wanehao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang. Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. TorchTitan: One-stop pytorch native solution for production ready I1m pre-training. arXiv preprint ar arXiv:2410.06: 06.00,0551110000 2024.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for LLM training. arXiv preprint 169882, 2025. URL http://arxiv org/aba / 0202. 6982.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. Stthstt:: httpas://pppreeeeee opw.revieeeeeeeeeeeeeeeeee.eeeee net. forum? wimmmk16r1igg""""
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019b. URL https: wwwpreveeneeeeeeeeeeeeeee net /formmiid-
13

---

Published as a conference paper at ICLR 2026
James Martens and Roger Grosse. Optimizing neural networks with kronecker- factoree approximate curvature. In International conference on machine learning, pp. 2408-2411 PMLR, 2015.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data 2024.
at scale. Advances Neural Information Processing Systems, 1000088111099,,,00000 2024.
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constrained LMOs. arXiv preprint arXiv:2502.07522, 2025, URL http:// /twpvvaxivvv//.////ii//.../...ia org/ http/ab//250000 07529.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
ISBN 9781728199986.
Noam Shazeer. GLU variants improve transformer, 2020. URL htwph::t////te https::/aaxxx...om abs/
2002.052000
Hao-Jun Michael Shi, Tsung-Hsien Lee, Shintaro Iwasaki, Jose Gallego-Posada, Zhijing Li, Kaushik Rangadurai, Dheevatsa Mudigere, and Michael Rabbat. A distributed data- mata-parallel py- torch implementation of the distributed shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06997 2023. URI. http:/tpt//// wwwvv...avvvvv/////v///../aa/vv. org/abs /2309. 0 644 97v1.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billio parameter language models using model par- allelism. arXiv preprint Stti::/9090005.5 2019.
Sebastian U. Stich. Local SGD converges fast and communicates little. 7th International Con- ference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Open- Review.net, 2019. URL httpps::/ppenreeiieest net/ wtt.oooommmddmddmmmaa
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Benjamin Thérien, Xiaolong Huang, Irina Rish, and Eugene Belilovsky, MuLoCo: Muon
practical inner optimizer for DiLoCo. arXiv preprint 25xi::2505.23725 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint, abs/2302. 2023a. URL atttp:://arkie org/ abs/ 2302.13975
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev. Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 02/p2/077009888 2023b. URL https: /tapps::/arkke /at.aaa.vvvv....a/aaaa.. abs/ 2207 09288.
Thijs Vogels, Sai Pranceth Karimireddy, and Martin Jaggi. PowerSGD: Practical low- rank gradient compression for distributed optimization. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc. E. Fox, and R. Garnett (eds.), Advances Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: https://pprceeedingg. wttp:ocweeddiin.....ne..eeeeee.e neurips,..... cc cc/paper. httpp/ppeeaaeeeeeeeeeeeeeeeeeeee d9fbed9da256e 64 4c1 fa 444b44443444445 wttp////paape/e/1111/////// -dd..
14

---

Published as a conference paper at ICLR 2026
Shuo Xie, Tianhao Wang, Sashank Reddi, Sanjiv Kumar, and Zhiyuan Li. Structured preconditioners in udaptive optimization: a unified analysis. arXiv preprint arXiv:2503.10537, 2025. URL http: / www.v.org/abs//5000000 10537v1.
Ruisi Zhang, Tianyu Liu, Will Feng, Andrew Gu, Sanket Purandare, Wanchao Liang, and Francisco Massa. SimplefSDP: Simpler fully sharded data parallel with torch.compile. arXiv preprint
arXiv:2411.00224. 2024. URL https: / /awpaaiivvvv.o....aaaaaaaa.aa..a abs/2411 00284.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: Experiences scaling fully sharded data parallel. arXiv preprint 20X1v URL http:/ //axivv.. org/ abs/ 2304 11277v2.
15

---

Published as a conference paper at ICLR 2026
Appendix
A MAIN PROOFS
A.1 NORM EQUIVALENCES
Lemma 2 (Dual of the block-spectral norm). Let X E Rm be partitioned into x blocks Xij E my X (not necessarily square). Define
B(X)
max 1sish, 15isc
| Kpp
With the Frobenius inner product GX = tr(X G)= Eis ir tr(X)) Giz), one hus
sup B(G)S
GX
ΣΣ
Xij 1l+
=1
Moreover, if Xij = Uig is an SVD, then
Xij # 0, Xij = 0,
Zi 0,
feasible with B(Z*) S and attains the supremum:
(x,7*) =E11
|| Xij ||.
Consequently the dual norm of B(-) is (Y) = Eig Yig Il..
Proof. For any feasible G with B(G) < 1, Cauchy-Schwartz gives us
(Xyy. Giy) S Xi ||+ Giy 1lop Xzy ||~
Summing over blocks,
(X (Xiy, (X.,, Gid) S Σ Xij |l.
1.j
i.
Taking the supremum over feasible G yields
(X. S Σ X1j 1|~
sup B
We now show the above upper bound is achieved by Z+. Let Xi₃ = Uig Vij an SVD and define 2 blockwise by Z = U2 V.J if X * 0 and Zij = 0 otherwise. Then || Zij 1lop= 1 when X,, * and O when X,, = 0, so B B(Z*)= < 1. Moreover,
(X Σry) tr EU (Un)))) tr(E., =| Xis llx
Summing over blocks gives (X, z*) En, Xij Il.. which matches the upper bound, hence z is optimal and the stated supremum value holds.
Lemma 3. The norm equivalence constants in Assumption 3 for the operator norm and block- spectral norm are both equal to one. That is,
Pop = and
Pblock = 1.
Proof. The operator norm || (the largest singular value) is always less than or equal to the Frobenius norm |X]| (the root-sum- squuurureuu of all singular values), hence Pop 1. The block-spectral norm is a maximum of block operator norms, B(X) = max;,5 X flop S
max,, ||X S E.s |X = ||X|| which implies Pblock = as well.
16

---

Published as a conference paper at ICLR 2026
Lemma 4. (Relations of norms) Suppose that B(-) is the block-norm corresponds partitioning a matrix of size m. x nt into r x blocks of size my x nh each. Then the following relations hold
B S | lop S VreB(G),
|Gl|op, & (B) I Vrel[G
Moreover, if function f is Lop -mooth with respect to the operator norn and Lg- Lg-smooth with respect the norm, have
Lop S LB (rc) Lop-
block matrix with blocks Gij mm and recall B(G) maxx,llGG, llop- We first prove that B(G) S ||G Let R2 0, mm select the i-th block of rows and Cj E 0, 1 select the j-th block of columns, so Gij R,GC,. Because RG and Cj are partial isometries, || = ||C,
Proof. Write G
By submultiplicatiiity.
||G llop = |C, S Rallop |Gp]|opllllpl
||
Taking the maximum over (i.j) yields B(G) - ||G||ap
Now we prove the upper bound on ||G|lop- Let E Imm v E R" be unit vectors and partition them as u [1 Upl, v = [Vl Vel with u, R and By Rs, Then
T Gv
u Gijuy
1=
i=1 j=1
Σ Σ
|| Gisllopllu5l
B(G)
|uellz
Σ 11ub12
B(G)
Σ
Iulll
1es ll2
Vrc
where we used Cauchy-Schwwarzapppeed the vectors blockwise l2 norms. Taking the supremum over unit u, v gives || I vrcB(G).
For the dual norm bounds, observe that norms then their duals satisfy
|| lla and || lb satisfy all -11 s 111-1b- s -111-11a
3 l1: s ||- |1ž < || I|:-
Applying this with || = B(-), |I. 1ls = =1-lllpp a = 1. vrc yields
| (G) 5
Finally, for the smoothness bounds, we have for any X + Y
()(Y)llop.
(VF V(())
X Yloop
|X - Y llpop
B (Vf vf(())
Vrc Vf(()
B(X-V
B(X -Y)
VV((X) V ((()))oop.
< (rc)
XX Yllop
Taking the supremum over X, Y such that X Y immediately yields Toop & ToB (rop
17

---

Published as a conference paper at ICLR 2026
A.2 CONVERGENCE PROOFS
Lemma (Descent Lemma). Suppose that Assumptions and hold for f: RRxx + R. Then the iterations of the algorithm without regularizer satisfy:
22llV/((x)) (1 + 3 Lnp²
Proof. By smoothness,
L
f(X++1) & f(XA) (VF(A)) -x)) +
2 = 111
Orth(Ma)) Tnp²
5((x) (nthMM)
-(xx) (V/(X)) - (1-1) Orth(MM)) 7711 14) (Ms, Orth(Ma)) Laf
Li7²
$ ((())) + /IV) - - -- -
- (1 MM)MMM +
2
f(x)) - - -1I+))M+l1 7 () LN²
S f(x)) + 20l Vf((K) -1 -1 11)MMlll-- - /l/V/(X>)ll. + Lnp
2
Lemma 6. Suppose that f satisfies Assumption 1 in some norm ||-I| and is lower bounded by f
then
[V 22L(5()) f.). f2)
Proof. Define
X+ = arg min [(X) [(x) (V((),, - _ Y -x| = =X -X-\),,(
where Z satisfies || /|| s 1 and (Vf(().2) [Vf()) ||.. By smoothness,
s(X+) S f(X) (V/(X), X+ x) + マ | x11
= =((x) - >| + LT²
2 | |V
= f(X) (1- - |f(()]
Plugging 1) = & gives
f. S (((4) F(x)- [VX(X)I
2L
Therefore,
IVS(1 s 2L(((X) fa).
Lemma 7. Let J satisfy Assumptions to 3 in some norm ||=I] with smoothness constant L, stochastic gradient variance d?, and Ez-norm ratio p. Let M, be defined as
M, = pM, + +G.
X,41 Xx 17, Zr.
where |2-||n for all k. Then,
LAp
1 P 1 μ
(X))) - - M1) I 1d 2 1 2T. +
+ po
1- μ
18

---

Published as a conference paper at ICLR 2026
Proof. Let Ma = (1 - --) We have,
Vf(Xa) - M3 = V)((x)) -
(1-
= V/((x) -
= -V/(Xx) - GR - 1M-1 + UGx
=VS(Xx) G. + (V
MG - (V
=V/(Xx) -Gy + (V/+ - + + - - - Let Eg vx(xx)) Mr S v -G2. and Rs vf((x) (X(xx1)).. Then the above
(1 (VS((()) - -Gx) + +x))+ (V[((XX))) -
is,
Es = -E 1 + - 1)) + +RR
k-1
f Eu + Σ nd [(1 - p))k-3 + HRRR-1l-
Now observe that
|| |R = - 54Ix - X X = LYN I S LA.
Therefore
E [+I.] S L [oll.] +
Σ 12((1 p)S + pRa-
1|| + E [||
+E
Σ 11 1)
[| + LA Σ ا + pE
Σ 1e (1 Sk
LAP
k-1
$ n [oll. +
+ p
Σ 11 1k)
1=
We have
27
k-1 Σ
R- R-1
E
jet (1 - S
= E
ve (1
122 ( Sk-i)
1= ==
k-1
Σ
11
21
2 [|S5-111]]
1=
A-1 Σ
50p - 11)2
25
1=0
02 n12
1- n12
@ 11)2
(1 (1 + 11)
22 11)
(5)
1+
19

---

Published as a conference paper at ICLR 2026
Using eq. (5) back in eq. (4) get
LAp
1 p pa V
Ex |.] S E [1E
(6)
1+
For the first term,
I| Boll.] = E (X))) - (1 - Moll.
E (X)) - (1 G) I.]
E [I[(1 (V Go) + f(Xo)l.
(1 MD (X)) Goll]] μ (X
By Lemma 6 we have / <
- and by Assumptions 2 and 3 we have E [[ - -Gol.] S pa. Plugging this back gives
E [I s p)pa + PV 2L
where Do =(Xo) fa. Using the last equation in Equation (6) yields
E Ex S (1 202ppr n ++ V2LA. + 1- 4
LAP
1 pa Vov 1 μ
Proof of Theorem 1. By Lemmas 5 and 7 we have
- (1 -1- -11-1) + Tn?²
E [f (+)) < [f(x))) - 13 - +
2
S E [f(x)] - ([
2np M (1 - 242 pa + 27 ++1 1 22L0
+ 2L F
1 +
Lnp
2mpo
1- F
1 1 2
Rearranging,
E [[IVS(XX)IJ] S 1)
E [S(x))
E ([(X++1]] + +2 - 11) + 2 22L00
Typa
1
Tnp
pa
1-H 1
++ μ
Summing up both sides as k- = 0, 1, ,,....IT T - 1 and telescoping
T-1
E 1IV
(((X)) - -E [[(T]]]] 2(1 11)² par Σ 77
nº 2pv
2LA
k=0
k=0
LupTl
1 1- Ln
poT
+
1
1
2
Ao + 2(1 - -ppp +
2 2LAn
LmpT
1 Lyj
1- + poT
1-
1 μ 2
Dividing both sides by T and lower bounding the average on the left hand side by the minimum yields the theorem's 's statement.
Proof of Theorem 2. Let Assumption hold for both norms with constants Pop and Pblock respec- tively and let PBP = P + P-1 Pulock- We will later show that Pops Pblock, and PBP are all bounded by 1. Let < T 1. If is divisible by P, then by Lemma 5 we have
Pop
E [f(X < E [f(x))] - ThanE [/ f (XX))lopp))
Lop 2
+ 2m [I|v/(X)) (1 -))
20

---

Published as a conference paper at ICLR 2026
To apply Lemma 7 with the operator norm, need to ensure that the updates 2, X,-X- X--1
"
always satisfy || lup?77 I for all - k. where 7 is the stepsize used on the 7-th iteration. Observe that on all such that 7%P 0, this is trivially true with A = Mull- On steps where is not divisible by P. we have || I Vrr I vrc (see Section A.I), since on those steps 1/r Vblock we therefore have 12-llop"/ S 7/hlock /rc. Therefore in all cases, Zr llpp% S max (Thlookk Mhan) and we can apply Lemma get
[IV])(()) - (1 M4)aaloo.. S 1² -1)= Pop + + F+1 V2loop
Top/" 1- μ
max(M Tblock v Vrc) Pupo
μ
We can then use this to bound the third term on the right hand side of Equation (7) to get E [f(X++1) & E [f(xa) - Man (II ((xx)]lopp.) 2M/aal
Papo
Papo
2mman 1+1 v 2Lopp Ao 2 V/rcH 21PPpor
1- Lop'2 (8)
p
n
2
Alternatively, if is not divisible by P. then similar to the above we first use Lemma and the fact that B" (Vx())) 2 [Xx)\=)ll to get
E if(X+++)l & E [f(x+)] Vblock (X + 27block (BB - (1 1))
LBnBlock
2
(9)
We now apply Lemma 7 to the block norm. Note that if r is not divisible by P. B(2+)
(Z Molock S Tolock. r is divisible by P. then B(Z,) ) = B(Z,) Meul I 1Z I M/lll- Therefore by Lemma we have
LB max (block. Mull )
E (V)(()) - (1 p)Ma S p1 - 2)² Pblock + 4 k+1 V2L +
1-
+ Polock V
1 1+
(10)
Using Equation (10) in Equation (9) we obtain
E [](x++1)] S E [S(Xx)) T/olock E [IIV (Xx)(Xx)llo.. LiThlook
+ 21/bock 10 (1 112 Phiacted + k+1 2I + LB max (Mulock Maal)
+ Polock
1
1 (11)
Let Sp = {k < (mod P) = 0} and SB {k: T & (mod P) * 0}. By rearranging the one-step descent inequalities and using Equations (8) and (11) we sum over O.....T
. 1
(Lkesp Mhlll LAESS Tblock (I
E [f(xx)] - [f(x++1)))
k=0
(12)
+
(Errork" full
+
(Error black
KESp
kESn
where
2o oop Mhlll max Thinck rrc Mhal)
Error 21/ 11) o 2maup k+1 / Lo Ao
Lop
+ 2nhallPpo
2 max(T Minl)
Error block
21pstock
11)² Pblocko + 2 k+1 2Ln
1 p
1 11
+ 37blockPblocke
I+
21

---

Published as a conference paper at ICLR 2026
The first term the right hand side of eq. (12) is a telescoping bounded by Au =f(X)) - f=.
For the error terms dependent on 1a" we can form a simple upper bound by summing over all & and using the larger constants (LB, Pblock):
Momentum- dependent error
Σ
2naau (1 11)2Popo Popo + 2mamp R+1 v /2 Ao
RESp
Σ
21pbockk (1 22) Polocko + Mhllock +1
#1 2LBA0
kESp
T-1
1c 122 max((NulPPp:: )ookk k+1 max( Lop. 7block v LB) 2LL
k=0
(1 11) (ax(PiiitPpp: PhilcckPhlnckk )o + maxx (mall / Lop: Tblock v LB LB)
For the other terms, sum them proportionally. Observe that |Sp T/P p. and |SBl 1)/P, as we assume that T is divisible by P. Therefore,
T(P
Lophhull
oo max( rrc Nand + 2nlll
Constant error
Σ
+
2
1 1-
1 1+ 11
KESp
LBBllook +
2LB max (axxT/al7/otcck Phoock) + 2/bloockPblockk
+
Σ
2
1 ++
kεšp
Th Lop
LETitook (P- (P- 1)
T
2P
2P
+ 2/1
Lop max (Mtull max(T Bolock v rC, phll))
La Mbull phinck (P 1)
P
WiallPPp
ChlockPhlocke (P 1 11
+
+ 4
P
P
Observe that,
Tiul LEES Molock
[II [IV 2
T
Mulll
Tblock (P 1P-1) P
min [[Xx
P
E [IV
Observe that for the operator norm and the block spectrum norm, have Pop and Pblock by Lemma 3. Using this and combining all the error parts we get
6μV Do
TT min E |II [(Xx)lool) S An + 4(1 pm)
max(ITllly V Lops Tolock v LB)
Mhnn Lop
LBBhlock (P 1)
1
T
+ 20
V
-7
2P
2P
1 1+
2p
Lop max(Tblock vTc, Mfull)
LiBhllokk max(HHull, Thlookk (P (P-1) 111
+
11
P
P
Dividing both sides by 7T yields
Au
4(1 udo Mmax
6pv/o p
max / Dops Malock LBB
min E [IV
keT
T
T
1j
Lop niun
LB Potook (P 1)
2P
2P
24
Lop Mall max Molock / /rc, Mull
LB nolock max Mall Molock (P - 1))
1 μ
1 1
P
20
V
11 μ
22

---

Published as a conference paper at ICLR 2026
Approach
Sharded (P/G/O)
How sharded
Params unsharded in F/B?
No parallelism
None
No
Tensor Parallelism
P+G+O
(any dim)
No
FSDP2 (dim-0)
P+G+O
dim-O
Yes
ZeRO-1
0
layer
Yes
ZeRO-2
G+O
layer
Yes
ESDP/Z=RO-3
P+G+O
layer
Yes
PP
P+G+O
layer
No
Table 4: "Sharded" lists which of parameters (P), gradients (G), and optimizer states (O) are parti- tioned across ranks. F/B forward/backward, TP=Tensor Parallelism. PP=Pipeline Parallelism.
where
M/ull P-1 7j =
T/block:
Tmax
(axll Mhull Tblookk
P
B ADDITIONAL ALGORITHMS, RESULTS, AND EXPERIMENTAL DETAILS
summary of different parallelism strategies is given in Table 4. The orthogonalization via Newton Schulz iterations procedure stated as Algorithm 2.
Algorithm 2: Orthogonalize-vinNSGG.. K, 10 10-
2, b 6 1.5, 0.5; X f G +E);
for t < to K do
4 XXT:
B bA + cA?: X + aX + BX;
return X;
All the architectural hyperparameters and learning rates used given in Table 5 below. We used the learning rates from (Liu et al., 2025) as a starting point for each run. We found at a smaller seale that the learning rates recommended therein could be increased by factor of ss 3 with no harm convergence for the baseline (Muon) algorithm, and we scaled the other learning rates accordingly. Nevertheless, we do two experiments with smaller learning rates. We use GQA (Ainslie et al., 2023), RoPE (Su et al., 2024), bfl6 miixed-precisio also apply gradient clipping with value 1.0 to the parameters optimized by AdamW (mainly ID parameters and the input embedding). We use cosine decay with no warmup for the 960M and 1.2B experiments and the Satemnuu--taaD---at to 4.2 x 10-5 for the 8B model.
training, and a weight decay value of 0.1. We
(WSD) schedule (Hägele al., 2024) with linear decay
Table 5: Section 4.2 experiments hyperparameters (sequence length = 8K, Batch size= sequences for 960M/1.21 .2B models and 256 sequences 8B models).
Model
Layers
Heads
Query Groups
Hidden Size
(DP, TP)
LR (x10-
Tokens (B)
Steps
960M
12
16
4
1536
(2, 4)
3.503
9,503
9063
14
1792
(2.4)
3.291
14.143
1.28 (3x long, larger Ir)
14
16
4
1792
(2
3,291
42.143
40191
1.2B (3x long. smaller It)
14
16
4
1792
(2, 4)
0.86
42.143
40191
(smaller Ir)
32
32
4096
(4.8)
9.99
4764
8B (lurger
32
32
8
4096
(4.8)
9.99
4764
We report the training curves for the 960M model in Figure 4, for the 1.2B model in Figure 5, for the 1.2B model trained to 3x Chinchilla with smaller learning rate in Figure 7, with larger learning
23

---

100000 200000 300000 400000 500000 Published as a conference paper at ICLR 2026
10
Stucktloan Thur of
6
Buctiluce Mocee
6x1d
Bleckkee Mic dt" Aciam
10/ne 100
valccaiion loaa
1-10
1x10
12000
4000
12000
iniking dags
Iox stpp.
numtin (seconds)
(a) Training loss vs. steps
(b) Validation loss vs. steps
(c) Validation loss vs. runtime
Figure 4: 960M model. Comparison of baseline, block, and periodic orthogonal block methods across training steps and wall-clock time.
6
5+ 16
Meun Me
Mfoc Mic at
1=
4-10
3=10
3-10"
3x10
baringg s0go
Dua11g staps
nartins Decorah)
(a) Training loss vs. steps
(b) Validation loss vs. steps
(c) Validation loss vs. runtime
Figure 5: 1.2B model. Comparison of baseline, block, and periodic orthogonal block methods across training steps and wall-clock time.
Figure 6, and for the 8B model in Figure 10 (smaller learning rate) and Figure 9 (larger learning rate). Our main observation here is that even after doing RMS norm adjustment (i.e. update is scaled by /max(A, B) where A and B are the dimensions of the update matrix, which scale inversely with blocking), BlockMuon can become unstable with larger learning rates. On the contrary, MuonBP does not.
A reason why this might be the case is that BlockMuon almost always causes the parameter norms grow larger over time compared to Muon or MuonBP, can be seen in Table 6. This holds even when we use small learning rates and learning rate adjustment.
Miar Mico dt
6x10
Minc dD?
10
valdudee loza
1-10
40:10
4=10
3x
3x
3x100
10000
20000
30000
40002
50300
20000
30000
40000
(a) Training loss vs. steps
(b) Validation loss vs. steps
(c) Validation loss vs. runtime
Figure 6: 1.2B model (larger Ir), trained to 3x Chinchilla. Comparison of baseline, block, and periodic orthogonal block methods across training steps and wall-clock time.
24

---

Published as a conference paper at ICLR 2026
Maze
Milnn Beoddeen Aater
Manon Beoddeec Adtrm
10
6-110
Ex10
10
1-10
6=70"
4x10
3x
3-110
3x10
10000
20000
30000
40002
10200
20003
50000
40000
100000 (secord)))
200000
500000
miiiing
trakiin dtrps
(a) Training loss steps
(b) Validation loss steps
(c) Validation loss vs. runtime
Figure 7: 1.2B model (smaller Ir), trained to 3x Chinchilla. Comparison of baseline, block, and periodic orthogonal block methods across training steps and wall-clock time.
7000
BiokMMun
2000
BlociMucc
Much
1000
Boockker
5000
Adar
5000
Micrit
7000
2000
8000
5000
2000
2000
1000
1000
3000
2000
4000
5000 braking stogs
8000
10000
12000
20305
Iraniing staps
Figure& 8: Comparison of parameter norms using Muon, BlockMuon, and MuonBP over training on 960M model (left), 1.2B model 3x- Sx-Chinchilla with smaller Ir (center), and 8B model (right).
Muson Miod
Muoe
Min Baroddeus
10'
Boochhlue
1x
10
4x10
3x10
Suiniing ptoci
narlars Datcrsh))
(a) Training loss vs. steps
(b) Validation loss vs. steps
(c) Validation loss vs. runtime
Figure 9: 8B model (larger Ir). Comparison of baseline, block, and periodic orthogonal block meth- ods across training steps and wall-clock time.
Mici Bhot
Mua Blckhhue
Mizz
3+110
3-10
1000
3000
3000
4003
5902
1000
2005
25000
50000 73000 100000 125003 nartara (aconab)
paatiing atapa
2ainingg aiace
(a) Training loss steps
(b) Validation loss steps
(c) Validation loss vs. runtime
Figure 10: 8B model (smaller Ir). Comparison of baseline, block, and periodic orthogonal block methods across training steps and wall-clock time.
25

---

Published as a conference paper at ICLR 2026
10
SlockMoo
10
16
Nacrop
Macndo
Soating Bax
Dion
validation lasa
Chan
10
XxX07
4x307
100
30P
Braiiing stape
training staps
runtime (sccondu
(a) Training loss vs. steps
(b) Validation vs. steps
(c) Validation vs. runtime
Figure 11: 160M model training with 2-way FSDP2 and w-y TP. Comparison of Baseline, Block- Muon, MuonBP, and Dion across training steps and wall-clock time.
Table 6: Validation/Trainiia perplexity (lower is better) and average parameter Best perplex- ities within each model size are in bold.
Model
Method
Val PPL
Train PPL
Param Norm
Muon
15.34
13.43
2680
960M
BlockMuon
20.28
18.09
5702
MuonBP
15.11
13.21
2648
Adam
22.51
20.16
1433
Muon
14.12
12.83
4237
1.2B
BlockMuon
16.29
14.86
7225
MuonBP
13.78
12.44
4195
Muon
12.62
10.88
2681
1.2B (3x, large Ir)
BlockMuon
13.29
11.51
5521
MuonBP
12.45
10.71
2868
Muon
13.27
11.40
1602
1.2B (3x, small
BlockMuon
13.23
11.29
3242
MuonBP
13.30
11.39
1571
Adam
15.03
13.25
1448
Muon
12.90
11.74
4369
8B
BlockMuon
13.68
12.62
6680
MuonBP
12.77
11.59
4471
Adam
14.47
13.48
4104
Muon
13.40
12.39
6841
8B (large Ir)
BlockMuon
24.68
23.17
11496
MuonBP
12.97
11.93
7063
26

---

Published as a conference paper at ICLR 2026
C COMPARISON AGAINST DION
Memory usage. For a weight matrix E RRXX, DION maintains the usual momentum buffer M Rmxn together with right subspace basis V E Raxr (of rank r). yielding persistent state memory usage + nr). Some transient memory is allocated to "skinny" factor matrices (c.g., U Rmxr W [ and small r X r solves, totaling O(mr + nr + for transient memory usage. MUONBP keeps only the momentum state per shard (globally O(mr)), i.e., no persistent low-rank bases. However, on the periodic "full" step temporarily materializes the full tensor to orthogonalize globally before scattering back, which introduces an (mm) transient buffer on that step: the intermediate "block" steps operate locally on shards and require only shard-sized temporaries.
r²)
Compute per iteration. Unsharded DION performs an amortized low-rank power-iteration and orthonormalization whose leading cost scales O mr² + mn) per update. In contrast, MUONBP uses
(NS) orthogonalization (Algorithm 2). On blocking steps, MuonBP runs NS local p q shard with cost proportional that block (per NS iteration), while every P steps it executes one global NS on the full matrix (cubic the larger dimension). Averaged over a period, the per-iteration cost is P-1 times the block cost plus of full NS. This comes out to
P P (Mq² (²) + [ + n²²) nn))-
Communication per iteration. With model parallelism, DION communicates only low-rank ob- jects each step, (mr) bits along the axis that forms broadcasts U and O(nr) bits along the axis that forms W. plus O(r²) micro-collectives for orthonormalization; critically, nothing scales with mn. Under data parallel, Dion can synchronize its low-rank optimizer state and gradients, reduc- ing DP 1/O from (mn) to O((m + n)r). For MUONBP, block steps incur no optimizer-specific model-parallel traffic beyond the baseline training collectives, while the periodic full step performs a gather/orthhogonaizzessattee of the entire tensor (layout dependent), so the average optimizer com- munication volume ((mn/ P). We can see that m/ P or n/ P act as the counterpart of Dion't 's rank parameter in MuonBP. For both algorithms, we need to tune one parameter (the rank vs the communication period P).
D ADAPTIVE AND SCHEDULED PERIOD P
While MuonBP fixed period P for periodic orthogonalization in all of main experiments, we also explored adaptively scheduling P to further reduce communication overhead without sacri- ficing convergence quality. We hypothesize that as training progresses, we can afford less frequent global orthogonalization.
To test this, we conducted an ablation study using the 960M parameter model architecture described in Section 4.2. We trained the model for 2.37 billion tokens over 3000 iterations with a batch size of 96 sequences and tensor parallelism degree 8. We compared four different configurations:
(a) Muon (Baseline): Full orthogonalization at every step (P= 1).
(b) Mlon Fully blocked orthogonalization with no global synchronization (P oo).
(c) MuonBP (Scheduled P): The period P is increased linearly from 2 to 20 over the course
of training.
(d) MuonBP (Fixed Expected P): The period is fixed at P = 11, which is approximately the expected value of the linear schedule.
Our experimental results, shown in Figure 12 and Table 7, indicate that MuonBP with the scheduled P (2 + 20) achieves a final validation perplexity 18.01. This outperforms both the baseline Muon, which achieves 18.28, and the fully blocked BlockMuon, which degrades to 28.60 28.60 validation perplexity. Moreover, MuonBP with a fixed expected period of P = 11 achieved a slightly higher validation perplexity of 18.11 compared to the scheduled variant. Notably, the scheduled MuonBP reached the baseline Muon's final perplexity around step 2800 (~~ 6% faster wall-clock time compared baseline Muon). These results demonstrate that scheduling P can offer promising
27

---

Published as a conference paper at ICLR 2026
avenue for accelerating convergence, and believe further exploration of adaptive or scheduled periods is an interesting direction for future work.
x 100
Muon (P-1)
BlockMuon (P=
MuonBP (P: 2-20)
5 x 10
MuonBP (P=11)
validation loss
4 x 100
3 x 10"
500
1000
1500 step
2000
2500
3000
Figure 12: Validation loss across training steps for different scheduling regimes for the period P.
Table 7:
Final Validation Perplexity for Period Scheduling Experiments.
Method
Validation Perplexity
Muon (Baseline, P= 1)
18.28
BlockMuon (P oc)
28.60
MuonBP (Scheduled : - 20)
18.01
MuonBP (Fixed p 11)
18.11
28


Reviews:
[{"review_id": "w4PySPzDlk", "reviewer": "Reviewer_nphQ", "summary": "This paper addresses a key performance bottleneck in the Muon optimizer, which suffers from reduced throughput in model-parallel training due to the communication overhead of its gradient orthogonalization step. The authors propose Muon with Block-Periodic Orthogonalization (MuonBP), a variant that eliminates this overhead on most steps by performing orthogonalization independently on the local gradient shards on each device. To maintain the stability and convergence of standard Muon, a full, cross-device gradient orthogonalization is performed only periodically. The authors provide theoretical analysis justifying this periodic approach and the use of two distinct learning rates for block and full steps. Experiments on an 8B parameter model demonstrate that MuonBP matches Muon's performance while increasing throughput by 8%, effectively closing the throughput gap with coordinate-wise optimizers like AdamW.", "questions": "1. I am curious about the key phenomenon in the experiments that explains why MuonBP is faster (in terms of wall-clock time, as seen in Figure 3) than standard Muon. Intuitively, MuonBP performs more local updates where the parameters are exposed to less global information (only the local shard gradient), which suggests it should not converge faster per step than standard Muon, which sees the full gradient. Could the authors elaborate on this surprising result?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a solid theoretical analysis of the proposed MuonBP algorithm. The convergence guarantees (Theorem 2) insightfully connect the block-periodic approach to the two extremes (BlockMuon and standard Muon) and justify the novel use of two distinct learning rates, which adds rigor to the method.\n\n2. The experimental validation is large-scale, convincing, and directly addresses the practical problem of throughput. By testing on models up to 8B parameters with 8-way tensor parallelism, the authors convincingly demonstrate that MuonBP achieves its goal of improving wall-clock throughput (by ~8%) without sacrificing the final model performance.", "weaknesses": "1. The core idea of using local updates to reduce communication is a well-studied direction in communication-efficient distributed training. While applying this idea to the Muon optimizer and achieving impressive experimental results is valuable, the concept itself is not entirely novel. A more fundamental solution to this problem would involve improving the all-gather operation itself to decouple the Newton-Schulz iteration from the communication, but this is admittedly a very difficult problem. Therefore, the paper's overall contribution is still commendable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqpcFQW9P5", "reviewer": "Reviewer_rAmz", "summary": "The paper introduces MuonBP, a variant of Muon that replaces per-step global gradient orthogonalization with block-local orthogonalization aligned to model-parallel shards, and performs a periodic full orthogonalization every P steps. This preserves Muon’s data efficiency while removing most cross-device communication on off-period steps. The authors (i) formalize MuonBP and its communication model, (ii) prove a convergence rate that interpolates between Muon and fully blocked Muon and shows two distinct step sizes (for block vs full steps) are theoretically preferable, and (iii) demonstrate up to 8% throughput gains at 8B scale with comparable or slightly better perplexity vs Muon, while BlockMuon is unstable or worse at scale.", "questions": "Is there any rule of thumb regarding how to choose a good value of P?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The Originality, quality and signifance are good because of the following details:\n1. Clear motivation with concrete bottleneck: Muon’s additional all-gathers cost 5–10% throughput vs AdamW under model parallelism, which MuonBP directly targets.\n2. Simple, practical algorithmic knob: Period P trades fewer collectives for slighly weaker per-iteration conditioning and it aligns blocks to TP/FSDP shards to eliminate extra communications on off-period steps.\n3. Compelling empirical evidence: At 8B, 8% throughput gain vs Muon with no loss in perplexity. BlockMuon underperforms/unstable at higher LRs, validating the need for periodic global steps.\n4. Wall-clock wins: For fixed perplexity targets, MuonBP reaches them 10–13% faster, and for fixed time budgets it achieves 5–7% lower perplexity.\n\nThe clarity is great because its presentation is easy to follow. The figures are intuitive.", "weaknesses": "1. Limited exploration of P scheduling. The paper largely uses a fixed P (often 5). It will be better to add ablations with adaptive P and report wall-clock to target perplexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqiZIkSVEr", "reviewer": "Reviewer_aDQM", "summary": "This paper proposes MuonBP, a communication-efficient variant of the Muon optimizer that combines block-wise orthogonalization with periodic full orthogonalization. It is a good attempt to scale Muon to modern training setup, e.g., model parallelism. The authors provide theoretical convergence guarantees, showing that MuonBP interpolates between BlockMuon (fully local) and Muon (fully global), delivering 8% throughput increase compared to Muon with no degradation in performance.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Clear motivation and practical importance.** The work directly addresses a key limitation of Muon, its poor scalability under model parallelism, by reducing communication overhead. It is a big issue for Muon as I know.\n* **Theoretical grounding** The authors rigorously gives a bound for the propose method.\n* **Realistic pretraining setting evaluation**. Experiments on multiple setups (FSDP2 + TP, ZeRO + TP, and scales from 160M to 8B parameters) demonstrate that MuonBP recovers Muon’s data efficiency with significantly better wall-clock performance.\n* **Clear and good writing**: the paper is clear, well-motivated, and easy-to-follow.", "weaknesses": "Overall, it is a good paper that solves a real problem. I only have a few questions: \n* **Exploration beyond pretraining**. All experiments focus on pretraining, I am curious whether the author can envision the performance on SFT or even RL work.\n* **How to adaptively tune the P**: The Author mentions that we might adaptively tune it based on observed properties. I wonder whether the author has some insight on how to monitor, especially in the practical training setting with multi-paralelesim", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TwUZBEJiKr", "reviewer": "Reviewer_Wrom", "summary": "This paper proposed MuonBP, which basically trying to solve BlockMuon's loss divergence issue by adding some global synchronization via extra communication and adjust learning rate accordingly. the end results shows MuonBP loss curve getting lower than BlockMuon.", "questions": "The whole paper uses fsdp and zero interchangeably, which makes it quite confusing. What zero stage is indeed used? zero 1,2,3. (please describe this more important and delete some unnecessary details like row-wise split, which is the by-default setting)\n\nAlso sometime the experiment says using Megatron? how could megatron-lm used together with simpleFSDP? they don't have the synchronization implemented and tested between TP and this DP.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.adding some global sync on top of BlockMuon to make loss curve converge better is a contribution.\n\n2.PPL analysis is detailed and reasonable.", "weaknesses": "1. Research novelty is minor, \n\n2. Comparing loss(PPL) with running time is misleading. By looking at figure 3, it is hard to see that baseline Muon trains less iteration thus loss curve is higher than proposed MuonBP. If the author want to show the iteration time, they can create a table on this. In model training, the most important thing is model quality, which means after training same amount of Iterations (Not same amount of time), which model converges to the lowest PPL or loss values. The second priority is iteration time. \n\n3. The whole paper does not even include how many iterations each training is in their experiments, which is quite concerning. \n\n4. The authors lack of basic distributed training knowledge, for example L147-148, \" If we use TP or FSDP2, we have to\ndo an additional all-gather across the TP/FSDP2 groups to gather the model parameters\", which is Wrong. TP never gather model parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)