Achieving low-bit Muon through subspace preservation and grid quantization
Authors: Huaijin Wu
Bingrui Li
Yebin Yang
Yi Tu
Zhanpeng Zhou
Jianfei Chen
Junchi Yan
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
memory-efficient
quantization
low-bit
Muon optimizer
Primary area: foundation or frontier models, including LLMs
TL;DR: We present 4-bit-Muon-GRASP, a method for compressing the Muon optimizer using subspace preservation and grid quantization to enhance memory efficiency.

Abstract: Training Large Language Models (LLMs) faces severe memory constraints due to the increasing size of model parameters and optimizer states. The Muon optimizer, which is based on matrix orthogonalization, has recently demonstrated significant potential and offers considerable memory advantages over AdamW by utilizing only the first moment. However, how to apply memory-reduction techniques to further compress the optimizer states of Muon remains underexplored. Directly applying existing methods may encounter significant difficulties due to the orthogonalization process. In this work, we investigate the low-bit compression of Muon and systematically analyze the quantization error exacerbated by orthogonalization. We identify that the error primarily originates from the top singular subspace and the outlier patterns of moment matrix appearing across both dimensions. To address this, we propose 4-bit-Muon-GRASP (GRid And Subspace Preserving), which compresses the Muon optimizer states to 4 bits using grid quantization, while preserving the top singular subspace with minimal overhead. We evaluate 4-bit-Muon-GRASP through pre-training on LLaMA-130M, 350M, and 1.1B architectures and fine-tuning on 7B models for various reasoning tasks. Extensive experiment results show that our 4-bit-Muon-GRASP achieves accuracy comparable to full-precision counterparts while reducing training memory consumption by up to 28\%. The source code is publicly available at ~\url{https://github.com/wuhuaijin/lowbit-Muon}.

Published as a conference paper at ICLR 2026
ACHIEVING LOW-BIT MUON THROUGH SUBSPACE PRESERVATION AND GRID QUANTIZATION
Huaijin Wu Bingrui Li Yebin Yang', Yi Tu', Zhanpeng Zhou', Jianfei Chen," Junchi Yan School of Computer Science & School of Artificial Intelligence, Shanghai Jiao Tong University TTiinghua university
{whj1201, RO. 00 yyb, zzpi0 22,,,,, yanjunchi Jesjuu. setjjtu.edu.co edu.cn tuyi beyi.sjtuul -sttu22238gmaa1111 com (1br22, (lanfeicc@tsingeee iebb. cn
ABSTRACT
Training Large Language Models (LLMs) faces severe memory constraints due to the increasing size of model parameters and optimizer states. The Muon op- timizer, which is based matrix orthogonalization. has recently demonstrated sigmificant potential and offers considerable memory advantages over Adam W by utilizing only the first moment. However, how to apply memory-reduction techniques to further compress the optimizer states of Muon remains underex- plored. Directly applying existing methods may encounter significant difficulties due to the orthogonalization process. In this work, we investigate the low-bit compression of Muon and systematically analyze the quantization error exacer- bated by orthogonalization. We identify that the error primarily originates from the top singular subspace and the outlier patterns of moment matrix appearing across both dimensions. To address this, we propose 4-bit- Sobit-MuonGRRAs And Subspace Preserving), which compresses the Muon optimizer states to bits using grid quantization, while preserving the top singular subspace with mini- mal overhead. We evaluate H-bit-Muon-GRASS through pre-training on LLaMA- 130M, 350M, and 1.IB architectures and fine-tuning reasoning tasks. Extensive experiment results show that our SeattMMon-GRAAA achieves accuracy comparable to full-preciss precision counterparts while reducing train- ing memory consumption by up to 28%. The source code is publicly available at https: / // thub. cwm/wouiooooooooo aonhaaauaai j in/ owb t tMMun..
(GRid
7B models for various
INTRODUCTION
Large Language Models (LLMs) have shown impressive performance across multiple do- mains (Yang al., 2026), including language translation and math reasoning. The growing size of deep learning models has led to significant challenges in terms of memory consumption and com- putational efficiency (Yuan al., 2025; Wu et al., 2025), particularly during training (Chowdhery et al., 2023; Rajbhandari et al., 2021). For instance, pre-training a 5B model from scratch using Adam W with only one sequence of length 1024 exceeds the memory capacity of an NVIDIA A A000, with the full-precision (fp32) optimizer state alone surpassing 40GB, due to the need for buffer of 2x the model size to track both the first and second moments.
Existing efforts including GPU sharding (Rajbhandari et al., 2020: Zhao et al., 2023) and CPU offloading (Ren al., 2021) have focused on reducing the memory consumption of optimizer states at the hardware level. On the other hand, there are two main approaches in the algorithmic domain: factorization uses low-rank approximation to optimize states (Shazeer & Stern, 2018; Zhao et al., 2024), while quantizing the optimizer to lower precision is particularly attractive due to its simplicity and broad applicability. Existing works have successfully compressed the optimizer state to low-bir bit (8- bit, primarily focusing Adam W and SGD (Wang al., 2024; Dettmers et al., 2021; Li
"Correspondence author. equal contribution. This work was partly supported by Scientific Research Inno- vation Capability Support Project for Young Faculty (U40) of the Ministry of Education of China (SRICSPYF. 2Y2055019.

---

Published as a conference paper at ICLR 2026
et al., 2023). As result, optimizing the memory usage of the optimizer allows the saved memory to be reallocated for a larger model or an increased batch size.
Recently, Jordan et al. (2024) introduced the Muon optimizer, which incorporates orthonormalized update rules and has demonstrated substantial advantages. Large-scale studies (Liu et al., 2025) re- port that Muon nearly doubles the efficiency of AdamW, and Muon has been successfully deployed in foundational models such as Kimi-K2 (Team et al., 2025). From the perspective of memory- efficient training, Muon can reduce optimizer state memory usage by approximately 50% compared to the widely-used AdamW in LLM training, as it only requires storage of the first moment. There- fore, further compressing the Muon optimizer state holds significant potential.
However, the application of memory techniques to Muon remains an open question. Di- rectly applying low-bit compression techniques, such those used in AdamW (Dettmers et al., 2021; Li et al., 2023), may encounter great challenges due to the orthonormalization process. In this paper, we explore the low-bit compression of Muon optimizer, problem that, to the best of our knowledge, has not been attempted before. Our contributions are as follows:
1) We conduct a systematic analysis of the 4-bit compression error in Muon and find that the Newton- Schulz iteration exacerbates the quantization error, primarily due to the top singular subspace. In light of this, we divide the moment matrix into two parts: the top singular subspace and the residual singular subspace, and compress them separately.
2) We propose 4-bit-Muon- Dbit-Muon-GRAAP (GRid And Subspace Preserving) with two key techniques: sub- space preservation and grid quantization. Specifically, we suggest using relatively mild compres- sion (8-bit) to preserve the top singular subspace, with the memory overhead being negligible, while compressing the residual singular subspace to 4-bit. Morcover, given that the outlier pattern of moments appears across both dimensions, we introduce grid quantization to provide more accurate bounds via normalizing both row and column directions.
3) We evaluate our Bitt-Muon-GRASP through both pre-training and fine-tuning. Specifically, pre-train on LLaMA-130M, 350M, and IB architectures with up to 31.5B tokens and fine-tune on 7B models for either general or specific reasoning tasks. The performance of our compressed optimizers is assessed through training curves and downstream tasks. Across all tasks, our 4-bit optimizers achieve accuracy comparable to their full cull-precision counterparts, while reducing total training memory consumption by up to 28%. To the best of our knowledge, it is the most memory- efficient optimizer among all low-bit optimization methods.
2
PRELIMINARIES
2.1
MUON OPTIMIZER
The Muon optimizer (Jordan et al., 2024) is a recently proposed novel optimization method, specif- ically designed for neural network weights representable matrices. Unlike traditional optimizers, Muon introduces a key innovation by integrating orthogonalization into the moment update process. At iteration t, given the current weight W E Raxm learning rate ne. momentum Jl, and objective Lr, the update rule for the Muon optimizer can be formulated as follows:
M, ,M, + (W,
(1)
O, Newton-Schulz, (M,, T)
(2)
W. W 0,
(3)
where M, represents the moment buffer at step f, initialized as a zero matrix, p is the degree of the Newton-Schula (NS) iteration polynomial, and T is the number of iteration steps. The NS iteration aims to approximately orthogonalize the update matrix, which equivalent to replacing the update with UVT, where UΣV M, is the singular value decomposition (SVD) of M,- The NS iteration approximation avoids the high computational cost of SVD while still achieving the orthogonalization of the moment matrix, leading to isomorphic parameter update.
For the NS iteration, we set Xo
| Following Jordan et al. (2024); Team et al. (2025), where both p and 7 are set 5, we denoted this setting as official choice. then the update X from X;-1
M.
2

---

Published as a conference paper at ICLR 2026
at each step & is as follows:
X - axa (X (Xx-1X - + (cx--t((x-- X
(4)
where a, b, and are the coefficients. ensure proper convergence, we tune the coefficients so that the polynomial f(x) oz bx + cr has fixed point near 1. In this paper, we follow the official design, using a 3. 3.4455,
4.7750, and c 2.0315 for LLM training.
Remark. A recent work (Liu al., 2025) has extended Muon from classic models to LLM-scale training by incorporating weight decay and carefully adjusting the per-parne per-parameter update scale. As goal is also for LLM, we adopt these techniques and follow their setting in the training of LLMs.
2.2 QUANTIZATION AND DEQUANTIZATION
Quantizing the optimizer states to lower precision is an effective method to compress optimizer states for memory savings. Specifically, the optimizer states M, are compressed M using quantizer at step & and then decompressed with dequantizer for use at step +
Quantization. Quantization is the process of converting full-preeision tensors into low-precision formats. Let X TR' represent a full-precision tensor, and let QUANT, be a b-bit quantizer that reduces X discrete value chosen from a set of 2 possible values. The quantization process involves two operations: normalization (-)- and mapping M(+), which are applied sequentially and element-wise. Specifically, for each element H; E X, the quantized value is given by
/i QUANT, = M e (a.))
Taking signed values as an example, the normalization operator transforms elements of X into the range |-1, 1] according to the following formula:
maxisisp [zy]
The scaling factors involved in normalization referred to quantization scales, and they stored along with the quantized tensor for dequantization, The normalization range determines the granularity of quantization, with common granularities including per-tensor, per-token, per-channel, group-wise, and block-wise.
The mapping operator M for x E R in a b-bit quantizer is defined follows:
M(x) arg min J -R())
jeT.
where R is the quantization mapping function. The set T, = {0, 1, 1,......."" represents the discrete set of possible values, and the mapping is an element-wise function that maps each element of T, into the normalized range [-1, 1]. Different quantization mappings can be employed, such as linear mapping and dynamic exponent mapping.
Dequantization. The dequantizer, denoted as DEQUANT, performs the inverse operation of the quantizer, recovering the approximate original value. This process defined as follows:
a DEQUANT(g)) - R(q.)
(8)
METHODOLOGY
In this section, we first present the challenge of quantizing the Muon optimizer states bits and analyze the associated quantization error. We then describe the design of our proposed 4-bit-Muon- GRASP, highlighting two key techniques: top singular subspace preservation and grid quantization.
CHALLENGES
A straightforward way to implement 4-bit Muon is to directly apply group quantization to the mo- ment matrix M, as Dettmers et al. (2021); Li et al. (2023). In this paper, we refer to this naive method as 4-bit-Muon-base.
3

---

Published as a conference paper at ICLR 2026
distrtbution
cistribution
#trezion vilue ditrivationn
singular value distribttin
0934
50
Stgglar Nalve
20
denitt
NE-0.07
NE-1.73
20
NS(raa))
EeSL
NSi
N51quant
quasst
NStquanti
0.03
0.80
1300
vella
vilue
Indrx
(b)
(d)
Figure I: Visualization of momentum in transformerlayers. 0.tttners...ttt...PP.. proj in a LLaMA model. (a) The distribution of matrix (real) and their 4-bit compressions (quant). (b) Distribution of the matrix after NS iteration (NS(real)) and its 4-bit compressions after NS iterations (NS(quant)). (c) and (d): Distribution of singular values of the matrices (a) and (b), displayed on a Iog scale.
To assess the quantization emrors of matrices, first define the relative error between matrix A and matrix B as follows:
RE(A, B) = - --BBl
(9)
|
NS iterations amplify quantization error. Unlike optimizers such as SGDM (Qian, 1999), Adam (Kingma Ba, 2014), and AdamW (Loshchilov & Hutter. 2017), which compute updates in an element- -wie fashion, the moment matrix in Muon undergoes an orthogonalization step, achieved through Newton-Schulz (NS) iterations. This process may introduce significant quantization errors. To assess whether this process amplifies the quantization error, visualize the real matrices and their 4-bit compressions. Fig. I(a) and (b) demonstrate that the distributions of M, and its 4-bit compressions are nearly identical (RE=0.07), while significant difference emerges between their distributions after NS iterations (RE=1.78). In other words, the disturbances and errors introduced by quantization become much
pronounced after matrix orthogonalization.
Error Not Caused by Insufficient Iterations. To study how NS iterations amplify quantization error, we investigate how quantization errors change with the arguments in NS iterations, including the degree of the polynomial and the number of steps. Fig. 2 shows that as the number iterations and/or the degree of the polynomial increases, although the accuracy of the NS iteration improves, the quantization error even becomes higher. On the other hand, we ana- lyze the singular values of the moment matrix. From Fig. I(c) and (d), it is evident that the quantization operation consistently increases the singular values before and after NS iterations. Instead, this suggests that the quantized matrix requires fewer NS itera- tions to converge. Together, the evidences indicate that quantization errors are not due to insufficient iterations.
NS degree 3
Quentiration Eirer Enver RE quaan))
1.5
NS degree
without NS
official choice
0.2
0.066
3
NS stop
Figure 2: Comparison of quantization error with different arguments in NS iteration.
Top singular subspace suffers large quantization error. To further investigate the source of error amplification in the NS iteration, we analyze the moment matrix by dividing it into the top singular space and the residual singular space, Formally, suppose UEVT =Mrr the SVD of the moment matrix M E Rm=x Then, we define:
Table 1: Quantization error before and after NS iterations. The results represent the average values obtained across all parameters during the first 100 training iterations of the 1.IB
LLaMA model.
Mop = M₁ = V.
M,es = M M-MMp (10)
RE (M
RE (Mes)
where Up m VA mR are top-k singular vectors
64
0.08 + 3.31
0.09 - 0.47
and Σn E kx contains top-k singular values. We choose
128
0.08
- 2.42
0.09
0.63
different ranks & to construct Map and M and compress
256
0.08
+ 1.76
0.09 - 0.35
them to b-t, resulting in Map and M and compute the quantization emrrr before and after NS iterations.
512
0.08
+ 1.26
0.09 0.42
4

---

Published as a conference paper at ICLR 2026
waass woog
2018
X|: S.:9]
X
Figure 3: Left: Outlier patterns of moment tensor. Right: Illustration of grid quantization.
As shown in Tab. 1, find that the quantization errors of the top singular spaces and residual singular spaces are comparable before NS iteration. However, after NS iteration, the difference between the two becomes significant. Specifically, when setting the model, NS iteration amplifies the quantization eπror of Mop by 40x, while increasing it only 5 for the residual matrix Mees This indicates that NS iteration primarily amplifies quantization errors in the top singular subspaces, suggesting that quantization methods can be designed separately for different subspaces.
64 for all parameter tensors in
3.2 SUBSPACE PRESERVING
Given that the top singular subspace incurs significant quantization error in Muon, we propose using relatively mild compression to better preserve the information contained in the top singular subspace. Additionally, the NS iteration amplifies all singular values, leading to the magnification of even originally small values to extent that cannot be overlooked. As result, relying solely on the top singular subspace fails to capture all the information in the moment matrix, leading to accuracy loss. To address this, we retain the residual subspace of the matrix and apply 4-bit quantization to it.
Regarding how to obtain the top singular space of the moment matrix, using Eq. 10 via SVD is computationally expensive. In this work, employ numerical iterative approximation method known as Power Iteration to get the top singular vectors, which has also been utilized in Vogels et al. (2019); Ahn et al. (2025). Following them, warm-start the power iteration using the results from the previous optimizer, so that a single iteration sufficient to accurately capture the top singular subspace of M, Specifically, at each step, first compute the moment M, ,M,-1 + (W,,W),) and then use Power Iteration to compute the top singular vectors P,, R, with rank k:
P.R. RO Mop where P. E Rink R. E R
(11)
should be noticed that we perform column normalization on R obtain Q.. and use Q to perform single step of power iteration. Since M, and M,-1 exhibit certain degree of similarity, can benefit from reusing R, the starting point.
We then get the residual singular subspace of M as described in Eq. 10 and apply 4-bit quantization it. To preserve more information of the top singular subspace, we utilize relatively mild com- pression (8-bit) to Pt, Rg, We store 8-bit P,, R₁, and the 4-bit M in the optimizer buffer. Since m & m n, the memory overhead of storing P,, R, relatively small.
the rank *
3.3 GRID QUANTIZATION
Fig. (a) shows moment tensors in Muon, and observe that the outlier pattern appears across both dimensions. As a result, neither per-channel group nor per-token group quantization can fully capture such outliers. To achieve higher precision, we propose grid quantization, which performs normalization in both row and column directions to obtain a more precise bound for each entry.
Specifically, for matrix X, we divide into several blocks of size s (s is the group size), and the element within the block are denoted as {ai,5 /r < i. < r2, c1 I j 02). Then, the row and column quantization scales of this block are formulated as follows:
scale,, = max Tijo
scale max Ti.j-
(12)
oiiiSe2
5

---

Published as a conference paper at ICLR 2026
Algorithm 4bit-Muon-GRASP
Require:
Weight W, objective L, learning rate 1), momentum 14, weight decay A, rank k, quantizer
QUANT and dequantizer DEQUANT.
1: repeat
2: if O then
3:
Randomly initialize right factor Qo E GR Mo 0 R
4: else
5:
Me DEQUANT(M Tesa-1) E IRAX
6:
Pt-1 < DEQUANT(P (-1) Rankk RA-1 < DEQUANT(R Roxk
7:
M,-1 < -Mesaa- + P, P R E ER
8:
Q₂ < ColunnNorralize(RA, E Baxk
9:
end if
10:
M, + pM,-1+ + (L E R
Pt, R2 < Powerlter MM, Qe) E Bmnk Ronk
get top singular vectors get residual subspace of M
12:
M < M, P,R, E ERRx
13:
M f- QUANT, (Messs)
Ruxn
B> 4-bit quantization
14:
P7 F QUANT (P.-1) ma
R QUANT
Ro*k
> 8-bit quantization
15:
O2 f Orthogonalize M,) E R
using newton-schut iterations
16:
W. + W.-1 1 (O; XW,-1) E RMxx
17:
until convergence criteria met
1: function POWERITER(B, Q)
R Baxk
single power iteration (from Q)
2:
P - BQ E IR
3: P - Orthogonalize(P) ER
using QR decomposition
4:
R B-P Baxk
5:
return P.R
P is orthonormal
6:
end function
Then the normalization of grid quantization Fiyy can be formulated as:
Fi.j
Ngria (24.5) min(scale,, scale,,)
(13)
Grid quantization utilizes information a more fine-grained manner and provides element-wise unique quantization scales, effectively addressing the outliers that appear across both dimensions. Although grid quantization requires storing twice the number of quantization scales compared to group-wise quantization, the resulting memory overhead is negligible.
3.4 OVERALL ALGORITHM
By employing the techniques of top subspace preserving and grid quan- tization, we achieve a lower quan- tization error for the Muon opti- mizer (NE=1.78 + NE=0.14) The comparison between NS(real) and NS(quant) is presented in Fig. 4, where the rank of the top subspace is set to 1/16 of the original matrix rank. Algo. I outlines the overall procedure of 4-bit-Muon-GBB ASP. including the detailed steps of Power Iteration.
singuler
Siespuler Valuo
10
decsitt
NE-0.14
10
NStrwal) NS(quant)
NStreal) NSiquan1)
-0.02
0.00
0.02
value
Index
(a)
(b)
Figure 4: The comparision NS(real) and NS(quant).
EXPERIMENTS
We evaluate cill-mmon-baar base and Aebi--muon-GRAAA on both pre-training and fine-tuning of LLMs.
We compare our 4-bit muon optimizers with their full-precision counterparts, namely fp32-muon.

---

Published as a conference paper at ICLR 2026
Model Size 130M
Model Size 350M
Model Size 1.18
fp82 Muon
fp32 muon
fp.2.muon
fp32 Muon
8-bit-Muon
bit Mutn
2.
8-bit-Muon
--bil-Muuu-basee
4 4-bit-Muun-base
2.8
4 M-bit-Muou-base
bit
2.0
2.5
Tokens (a)
Tokens
Tokens
(b)
(c)
Figure 5: Validation loss comparison of pretraining with different optimizers Slimpajama.
Table Evaluation of models pre-trained with three optimizers, across downstream tasks for differ- ent model sizes. The results are the average of multiple random seeds.
Model
Optimizer
HellaSwag
ARC-c
ARC-e
boolQ
OBQA
PIQA
SeiQ
Avg
fp32-Muon
28.4
21.8
34.3
62.1
25.8
58.3
62.2
41.8
Sbit-Muon
28.4
21.6
34.1
62.2
26.9
58.5
61.7
41.6
130M
4bit- -aso-bc
28.2
20.8
34.0
62.2
27.8
58.8
59.6
41.9
4bit- Abi--Muon-GRASP
28.7
21.6
34.2
62.1
28.0
58.1
60.4
41.9
fp32-Muon
32.4
23.5
38.3
59.4
28.8
62.0
68.0
44.6
Sbit-Muon
32.5
22.3
38.1
61.3
28.4
61.8
66.5
44.5
350M
4bit- Muon-base
22.4
37.7
61.9
26.2
61.8
64.4
43.7
4bit Www.-MuoMMaMM...
32.4
23.0
38.5
61.2
28.2
61.4
66.6
44.5
fp32-Muon
40.6
25.4
42.8
60.4
30.2
66.5
69.5
48.0
Sbit-Muon
40.5
25.2
42.5
60.4
30.2
66.8
71.4
48.2
1.1B
4bit Miil-Muun-bass
39.8
24.2
41,5
61.0
30.4
66.6
69.7
47.6
4bit- Obil-Muon-GRASP
40.4
24.8
42.3
60.5
30.6
67.4
71.3
48.2
Here we further include 8-bit-Muon as a baseline, which is implemented through group quantization. All experiments
NVIDIA A100 GPUs.
Following Liu et al. (2025), we employ the Muon optimizer for updating matrix parameters, while RMSNorm, the LM head, and embedding parameters remain optimized using AdamW. Except for the ablation study, the rank of the top singular subspace is set to 1/16 of the original, and we use the widely adopted INT4 and INT8 format for simplicity and efficiency. We implement the quantization- related code using OpenAl Triton kernel to enhance efficiency and achieve real memory reduction. Both the group size and grid size of quantization are 128.
4.1 PRETRAINING
Datasets, architecture and hyperparameters. Following Zhang et al. (2024b), utilize Slimpa- jama (Soboleva et al., 2023) as the pre-training dataset and adopt LLaMA-based architecture with RMSNorm (Zhang & Sennrich, 2019) and SwiGLU activations (Shazeer, 2020). We evaluate three model sizes: 130M, 350M, and I.IB, and all experiments are conducted with BF16 mixed- -reccsson training enhance training efficiency. For each model size, we first tune the learning rate for the fp32-Muon from the set {2e - 3, le 3, 6e 4, 3e 4}, selecting the best learning rate based on the validation perplexity, respectively. We then apply the same learning rate to the b-bitMMun-- base and 4-bit-Muon-GGee GRASP models to ensure fair comparison. See Appendix A for more detailed hyperparameter settings.
Training Curve. Fig. 5 illustrates the training curves of different optimizers across three model sizes. The results show that even the 4-bit-Muon-bare optimizer does not significantly affect training convergence, with accuracy loss of 1%. Our proposed Sait-Muon-GRASS reduces the training gap with fp32-Muon to less than 0.2%, and on the I.IB model, 4-bit-Muon-GRAse even achieves no loss in training accuracy.
Accuracy of 4-bit Optimizers. line with Xiong et al. (2025), to assess whether our memory efficient b-t optimizers could maintain accuracy downstream tasks, evaluate zero-shot per- formance using the Im-evaluation-hhmne (Gao et al., 2021) codebase on standard benchmarks, including HellaSwag (Zellers et al., 2019), ARC (Yadav al., 2019), BoolQ (Clark et al., 2019),

---

Published as a conference paper at ICLR 2026
Table Statistics of different optimizers: the step time (s), total memory usage (GB), and validation perplexity (4) after training for 10K steps.
130M
350M
1.1B
time
mem
PPL.
time
mem
PPI.
time
mem
PPI.
[p322MMun
31.8
1.76
19.21
45.8
4.04
15.57
61.3
13.22
12.48
8-bit-Muon
31.9
1.47
19.29
45.7
3.35
15.65
61.5
13.22
12.46
--bit-Muon-base
31.9
1.42
19.73
45.8
3.23
15.96
61.5
10.54
12.76
1-bit-Muon-GRASH
32.0
1.43
19.35
46.0
3.26
15.67
61.9
10.14
12.48
Memory Usage Different Optimizers
80
Ip32-AdamW
70
Ip32-Muon
8bit-Muon
60
4bit-AdamW
Memory
50
dbit-Muon-base
4bit-Muon-GRASP
40
30-
0-
350M
1.18
58
Model Size
Figure 6: Total training memory usage across different optimizers and model sizes, evaluated with one sequence of length 1024 on single device.
OpenbookQA (Mihaylov et al., 2018), SciQ (Johannes Welbl, 2017). PIQA (Bisk al., 2020), Winogrande (Sakaguchi et al., 2021). The corresponding results are shown Tab. 2. For the 350MB model, 4-bit-Muon-brat and 4-bit-Muon-GRAA. 44.5, respectively, while the baseline performs at 43.7. A similar trend is observed with the 130M and 1.1B models. These results show that 8-bit-Muon and 4-bit-Munn-GR..A could match or exceed fp32-Muon performance across all tasks, while 4-bit-Muon-base shows slight accuracy loss.
achieve average accuracy of 44.6 and
Memory and Computing Efficiency. The training statistics are presented in Tab. 3, where the 4-bt--Munn-base shows degradation in training accuracy, whereas our 4-bit-MunnGGAAPP demon- strates negligible accuracy loss, particularly with the 1.1B model size. Moreover, we find that the training time overhead introduced by quantization is minimal and a detailed breakdown of the cost of power iteration and other optimzier logics are shown in Appendix B.2. Fig. 6 provides a further total memory usage comparison with the pell-preeision Adam W. 4-bit AdamW, and 8-bit Muon op- timizers, with model sizes scaled up to 3B and 5B. We observe that the 4-bit Muon optimizer is the most memory-efficient among all, achieving memory reductions of up to 48% and 28% compared to fp32-AdamW W and Muun, respectively. However, for smaller model sizes, the memory reduc- tion is not substantial, and the memory savings plateau as the bitwidth decreases. This is because we report total memory consumption, encompassing data, activations, weight gradients, and memory fragments, rather than isolating the optimizer's memory consumption alone. We also compare our low-bit compression optimizer with cpu-offloading optimizer, see Appendix B. for details.
4.2 FINE-TUNING
Experimental Setup. We select two pretrained models: the general model Qwen2.5-7B (Yang et al., 2025a) and the domain-specifie model (wwn22557B-manl (Yang et al., 2024a), to evaluate the effects of different optimizers on both general capabilities and advanced mathematical reasoning.
Our implementation builds upon the verl framework (Sheng et al., 2025). Since Muon requires the full gradient matrix to calculate the updates, and PyTorch Fully Sharded Data Parallel is not directly applicable to Muon, we refer to the public implementation of distributed Muon (hor, 2025; Ahn al., 2025). We further implement the distributed 4-bit-Muon-B and distributed 4-bit-Muon-
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of three optimizers applied to the SFT of the Qwen2.5-7B and Qwen2.5-7B-
Math pretrained models.
Benchmark (Metric)
# Shots
Origin
SFT(()32)
SFT(4bit-base)
SFT(4bi--RRSP))
Pretrained model: Qwen2.5-7B
MML.U(EM) HumanEval (Pass@1)
0-shot(CoT)
71.9
72.0
72.1
72.0
0-shot
56.1
76.2
76.6
76.7
MBPP (Pass I)
0-shot
64.4
71.4
70.3
70,9
GSM8K (EM)
5-shot
80.3
85,4
84.8
85.2
Pretrained model: Qwen2.5-7B
MATH (Pass
0-shot
65.4
70.5
69.8
70.8
Minerva Math (Pass@1)
0-shot
12.1
27.9
Olympiad Bench (Pass
0-shot
27.3
36.0
35.9
35.4
Average
53.9
62.6
62.5
62.8
Model Size 350M
Model Size 350M
Mode] Size 350M
2.82
2.82
3.00
(aseelinefp32)
2.80
2.81
rack-1/2
baseline((p322)
2:00
2.81
2.50
2.50
only subspace (rank-1/2)
only subspacc (rank-1/16)
subspace (rank-1/8)
2.70
rank 1/4
2.79
rack-178
2.78
rank-1/16
2.78
boseline((p$2)
2.85
2.77
rack-1/32
2.77
group quant
2.80
roc3-1/64
grid quant
2.74
30.0
2.7%
7.5
10.0
Tokens (a)
Tokenss (B)
Tokons (B)
(b)
(c)
Figure 7: Ablation studies on pretraining: (a) Selection of different top singular space ranks, (b)
Comparison of group and grid quantization, (c) Preservation of only the top singular space.
GRASP, where the quantization and dequantization are performed in partitioned shape, and the subspace preservation of momentum carried out globally. See Appendix for training details.
Performance on Downstream Tasks. For general model, we use eeal--hrneess (Gao et al., 2021) evaluate on benchmarks including MMLU (Hendrycks al., 2020), HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and GSM8K (Cobbe et al., 2021). For high-level mathematical reasoning model, we evaluate on established benchmarks including Math (Hendrycks et al., 2021), Minerva Math (Lewkowycz et al., 2022), and Olympiad Bench (olympiad problems 2024, 2024). The evaluation metric and results are shown in Tab. 4, where we compare the performance of fine- tuned models with different optimizers. The results demonstrate that our 4-bit-Muon optimizers will not destroy the capabilities of pretrained models, and the --bit-MuoonGRAAA achieves performance comparable that of the 32-bit Muon across all tasks.
4.3 ABLATION STUDIES
How does the rank of the top singular space affect convergence, memory, and computing ef- ficiency? We preserve the top singular space with different ranks, ranging from 1/64 to 1/2, and perform pre-training on LLaMA-350M. Fig. (a) illustrates that the gap of the training curve be- tween our method and fp32 baseline widens as the rank decreases. Notably, when the rank of the top singular space is set to half of the rank of the matrix M, the training loss shows no difference from the baseline. The impact of different ranks on memory usage and time is discussed in Appendix B.4.
What happens if we preserve only the top singular space while discarding the residual singular space? Fig. (c) shows the results when only the full-precision top singular space is preserved, with the rank ranging from 1/2 to 1/16. The results demonstrate that discarding the residual singular space leads significant degradation in training accuracy. Even with 1/2 rank approximation, the training accuracy loss exceeds 2%. This underscores the importance of the residual singular space and suggests that the orthogonalization of Muon prevents straightforward low-rank approximation.
Comparison between grid quantization and group quantization. We compare the effects of grid quantization and group quantization training performance. To provide clearer comparison,

---

Published as a conference paper at ICLR 2026
directly compress the moment matrix without preserving the top singular space. As shown in Fig. (b), grid quantization reduces the accuracy loss of group quantization by half. See Appendix B.4 for more ablation studies.
Comparison of different number of Power Itera- tion steps. To evaluate the approximation accuracy of Power Iteration with different step, Fig. 8 report during training, with 1-step, 2-step,
Approximation error of Power Iterations
stop Poweriter
the relative and 3-step power iterations, respectively. The rel- ative enror here defined as RE(U,, V .PRR).
2 step Powerlter
Relative Errorr
0.08
3 step Poweriter
where U V represents the accurate top singu- lar space, and PR the approximation obtained through power iteration. The results demonstrate that the error gap between the 1-step and multi-step iterations is minimal. In fact, the approximation er- ror of the 1-step power method already reaches as low as 0.01, indicating that single iteration is suffi- cient for the algorithm to accurately identify the top singular vector.
0.02
0.00
7.5
Training Steps (10*2)
Figure 8: Approximation error during train- ing.
RELATED WORKS
Quantization-base memory efficient optimizers. Dettmers et al. (2021) propose block-wise dynamic quantization, allowing first-order optimizers to operate with 8-bit states, while Li et al. (2023) further compresses the Adam/AdamW optimizer states to bits by applying finer-grained quantization and removing points from the second moment. Moreover, Wang al. (2024) introduces 4-bit second-order optimizers and exemplifies by 4-bit Shampoo.
Other memory efficient techniques. Several works have explored approximating gradient statistics with sublinear memory cost relative to the number of parameters. For instance, Adafactor (Shazeer & Ster,, 2018) uses the outer product of two vectors approximate Adam's second moment. SM3 (Anil et al., 2019) approximates the second moment in Adam using the statistics of its covers. LoRA (Hu al., 2022) freezes the pretrained weights and tunes only the newly initialized low-rank parameters. Additionally, some approaches focus on reducing the memory consumption of activa- tions, such as peteraatinncoppresse training and gradient checkpointing, which can be integrated with our optimizer to achieve further memory savings.
CONCLUSION AND OUTLOOK
method for compressing the Muon optimizer to improve memory efficiency. By dividing the moment matrix into two parts and applying grid quantization, we are able to reduce memory usage by up to 28% while maintaining performance comparable full-precision optimizers.
In this paper, we introduce 4-bit-Muon-GRAte
The optimal quantization settings are likely dependent the task, datasets, and training details, but the exploration in this paper relatively limited common LLM training scenarios. Due to resource limitations,
Limitations and Future Works.
evaluation is currently limited to pretraining 1.IB models. We identify several open problems for 4-bit-Muon-GR. GRASP, which include: 1) pro- viding strategies guidelines for automatic rank selection; 2) further enhancing memory efficiency by employing activation reduction methods; and 3) optimizing the efficiency and communication of low-bit optimizer algorithms in distributed scenarios.
10

---

Published as a conference paper at ICLR 2026
DECLARATION OF AI USE
We used Gemini/ChatGPT to assist in writing:
1) Correcting grammar, improving clarity, and refining the flow of sentences.
The LLMs do not contribute to research ideation, methodology, experimental design, data analysis,
interpretation of results, the creation of substantive academic content references. We carefully review and approve all suggestions from the models, and we take full responsibility for the final manuscript.

---

Published as a conference paper at ICLR 2026
REFERENCES
Various approaches to parallelizing muon, 2025. URL htt.::/ss//ninin-oorr.....iiii /o/
Kwangjun Ahn, Byron Xu, Natalic Abreu, and John Langford. Dion: Distributed orthonormalized
updates, arXiv preprint arX1v:2504.05295, 2025.
Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimiza- tion. Advances in Neural Information Processing Systems, 32, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language 2021.
models. arXiv preprint
Yatao Bian, Huaijin Wu, and Junchi Yan. Deep learning for affinity prediction and interface predic- tion in molecular interactions. Deep Learning in Drug Design, pp. 183-296, 2026.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pp. 7432-7439, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.033: 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240); 1-113, 2023.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. /4168, 2021.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint 1rXiv:2110.02861, 2021.
Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8 9, 2021.
Ziao Guo, Yang Li, Chang Liu, Wenli Ouyang, and Junchi Yan. Acm-milp: Adaptive constraint milp instance generation. In
modification via grouping and selection for Forty-first International Conference on Machine Learning, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint vrXiv:2009.03300 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Matt Gardner Johannes Welbl, Nelson E. Liu. Crowdsourcing multiple choice science questions. arXiv: 17Xiv:1707.06209911 2017.
12

---

Published as a conference paper at ICLR 2026
Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordann. github. to/posts/muon, 6, 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 111.,9941..6980,, 2014.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training arXiv preprint arXiv:2411.15124 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Soke et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36: 1136-1115,,,, 2023.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Yang Li, Lvda Chen, Haonan Wang, Runzhong Wang, and Junchi Yan. Generation as search operator for test-time scaling of diffusion-based combinatorial optimization. In The Thirty-ninth Anmual Conference Neural Information Processing Systems, 2025a.
Yang Li, Jiale Ma, Yebin Yang, Qitian Wu, Hongyuan Zha, and Junchi Yan. Generative modeling reinvents supervised learning: Label repurposing with predictive consistency learning. In Forty- second International Conference on Machine Learning, 2025b.
Jingyuan Liu, Jianlin Su, Xingeheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin,
Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for Ilm training. arXiv preprint 209.:22202.16882. 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hong- sheng Li. Not all experts equal: Efficient expert pruning and skipping for mixture-of-experts large language models. In Lun-We Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6159-6172, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 18.18653/v acl- URL https: http//aclaatttt http///aaaaaaaggoyooaaao.. org/2024. acl-long 334/.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Mathematical olympiad problems 2024. International mathematical olympiad, 2024. URL https: / //www. 1 of f ficiall .rg.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1):145-151, 1999.
Chongli Qin and Jost Tobias Springenberg. Supervised fine tuning curated data is reinforcement learning (and can be improved). arXiv preprint arXiv:2507.1288 2025.
Xiaohan Qin, Xiaoxing Wang, Ning Liao, Cancheng Zhang, Xiangdong Zhang, Mingquan Feng, Jingzhi Wang, and Junchi Yan. sstoken: Self modulatdd and semantic-ane aware token selection for Ilm fine-tuning, arXiv preprint arXiv:25100 18250, 2025.
13

---

Published as a conference paper at ICLR 2026
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Perfor- mance Computing, Networking. Storage and Analysis, pp. 1-16. IEEE, 2020.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley. Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the interna- tional conference for high performance computing, networking, storage and analysis, pp. 1-14, 2021.
Jie Ren, Samyam Rajbhandari,, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Min- jia Zhang, Dong Li, and Yuxiong He. {Zero-offload): Democratizing (billion-scale) model training. In 202/ USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Comea,, Eric Dellinger, Kristof Denolf, et Microscaling data formats for deep learning. arXiv preprint arXiv22310.10577 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 106, 2021.
Noam Shazeer. Glu variants improve transformer. arXiv preprini arXiv:2002.052022 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers. Jacob R Steeves, Joel Hes- tness, and Nolan Dey. SlimPajama: A 627B token cleaned and dedu- plicated version of RedPajama. sl slimpa jamaajaaaaaaa 2023. URL https:
/ttps://www.cr http://wwwceebbreee net/blog///// and
-aceaandddddddddduud------- dedupplletted-veesion--- redpajama, http///gggggggceeeeee//// co/ /
627B.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization. arXiv preprint 1.Xiv:2501.17116 2025.
Runzhong Wang, Ziao Guo, Shaofei Jiang, Xiaokang Yang, and Junchi Yan. Deep learning of partial graph matching via differentiable top-k. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6272-6281, 2023.
Sike Wang, Pan Zhou, Jia Li, and Hua Huang. 4-t shampoo for memory--fficient network training. Advances in Neural Information Processing Systems, 37: 16699772222, 2024.
Kaiyue Wen, Zhiyuan Li, Jason Wang. David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-dest decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.051192, 2024.
Huaijin Wu, Wei Liu, Yatao Bian, Jiaxiang Wu, Nianzu Yang, and Junchi Yan. Ebmdock: Neural protein-protein probabilistic protein- protein docking via a differentiable energy model. In The Twelfth Interna- fional Conference on Learning Representations, 2024a.
14

---

Published as a conference paper at ICLR 2026
Huaijin Wu, Lianqiang Li, Hantao Huang, Tu Yi, Jihang Zhang, Minghui Yu, and Junchi Yan. Hshare: Fast IIm decoding by hierarchical key-value sharing. In The Thirteenth International Conference on Learning Representations, 2025.
Huanjin Wu, Xinyu Ye, and Junchi Yan. Qvae-mole: The quantum vae with spherical latent variable learning for 3-d molecule generation. Advances in Neural Information Processing Systems, 37:
02745-22771, 2024b.
Hao Xiong, Yebin Yang, Huaijin Wu, Xiaoqiu Zhong, Yehui Tang, Zhuo Xia, Xiaoxing Wang, and Junchi Yan. Reinvent the operation not the architecture: guantum-inspired high-order product for compatible and improved Iims training. In Proceedings of the s1 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 3356-3365, 2025.
Vikas Yadav, Steven Bethard, and Mihai Surdeanu. Quick and (not so) dirty: Unsupervised selection of justification sentences for multi-hop question answering. arXiv preprint arXiv: 1911.91176,, 2019.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement arXiv preprini arXiv:2409. 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen2.5 technical report. arXiv preprint vrXiv:2505.09388 2025a.
Nianzu Yang, Huaijin Wu, Kaipeng Zeng, Yang Li, Siyuan Bao, and Junchi Yan. Molecule genera- tion for drug design: a graph learning perspective. Fundamental Research, 2024b.
Nianzu Yang, Songlin Jiang, Jian Ma, Huaijin Wu, Shuangjia Zheng, Wengong Jin, and Junchi Yan. Repurposing alphafold3-like protein folding models for antibody sequence and structure co-design. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b.
Yebin Yang, Huaijin Wu, Fu Guo, Lin Yao, Xiaohan Qin, Jingzhi Wang, Debing Zhang, and Junchi Yan. Jtok: On token embedding as another axis of scaling law via joint token self-modulation. arXiv preprint 260.:20000000000, 2026.
Jingyang Yuan, Huazuo Gao, Damai Dai. Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xic, Yuxing Wei, Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. In Proceedings the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 13078-23099 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? arXiv preprint arXiv:1905007830 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in neural infor- mation processing systems, 32, 2019.
Changwen Zhang, Wenli Ouyang, Hao Yuan, Liming Gong, Yong Sun, Ziao Guo, Zhichen Dong, and Junchi Yan. Towards imitation learning to branch for mip: A hybrid reinforcement learn- ing based sample augmentation approach. In The twelfth international conference on learning representations, 2024a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang. and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.022. 1.03855. 2024b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. / 1277, 2023.
15

---

Published as a conference paper at ICLR 2026
Xinhao Zheng, Yang Li. Cunxin Fan, Huaijin Wu, Xinhao Song, and Junchi Yan. Learning plaintext- ciphertext cryptographic problems via anf-based sat instance representation. Advances in Neural 2024.
Information Processing Systems,
16

---

Published as a conference paper at ICLR 2026
A
DETAILS OF PRE-TRAINIA SETTING
Table Hyperparameters for LLaMA model pretraining.
Parameters
130M
350M
1.1B
Ir-schedule
WSD (Wen al., 2024)
WSD
WSD
max, min
(le-3, le-4)
(6e-4, 6e-5)
(6e-4, 6e-5)
warmup-ratio
0.1
Training
decay-ratio
0.99
0,99
0.99
AdamW-9
(0.95, 0.9)
(0.95, 0.9)
(0.95, 0.9)
weight-decay
0.1
grad_clip
1.0
1.0
1.0
hidden dim.
768
1024
2048
#layers
12
22
22
heads
12
16
32
Model
#kv heads
4
context-length
1024
1024
1024
FFN size
1024
2560
5632
tokenzier
LLaMA-2
LLaMA-2
LLaMA-2
#steps(B)
10K
20K
30K
Data
#Tokens(B)
10.5
21.0
31.4
Batch size
1024
1024
1024
Pre-training. We provide details of the LLaMA architecture and the hyperparameters used for pre- training. Tab. 5 presents the key hyperparameters for LLaMA models across different sizes. In all experiments, we use WSD (warmup-stable-dee warmed up for the first 10% of the training steps and then decays to 10% of the initial value during the final 1% of the training steps.
learning rate schedule. The learning rate is
Fine-tuning. For Qwen2.5-7B, we fine-tune the model using the open-source tulu-3-sft --llt---fttmiiture dataset (Lambert al., 2024). Following Liu al. (2025), the dataset is packed with sequence length of 8k tokens, and the learning rate follows a cosine decay schedule, starting from 5 10-5 and gradually decaying to 2 x 10-6 For (wwen2.-7B-MMan et al., 2024) for fine-tuning, which consists of approximately 860,000 mathematical problems paired with their corresponding solutions. In accordance with Qin Springenberg (2025), we randomly sample 100K instances from the dataset for training, and the dataset is packed with sequence length of 2k tokens.
adopt the NuminaMath CoT dataset (Li
B MORE EXPERIMENT RESULTS
B. COMPARISON WITH CPU-OFFLOADING OPTIMIZERS
Table 6: The optimizer updating time(s) of different methods.
Model Size
130M
350M
1.IB
3B
5B
Muon
0.18
0.25
0.39
0.55
0.75
fp32-Muon-CPU --bittMuon-GRASSS (rank 1/4)
0.73
0.99
3.89
11.32
17.93
0.55
0.91
1.82
2.85
4.21
Suit-Muon-GRAAe (rank 1/16)
0.41
0.61
0.96
1.21
1.93
4- 4-bit-Muon- MuuonGGRASP (rank 1/64)
0.39
0.54
0.76
0.88
1.43
Tab. and Tab. shows the optimizer updating times (s) and optimizer memory usage (GB) of dif- ferent methods, which demonstrate that the optimizer step time for CPU-offloaded Muon increases significantly, especially as the model size grows. For a 5B model, the optimizer time becomes 20 times longer, introducing substantial overhead. In contrast, our proposed 4-bit compression method, with a rank selection of 1/16, only doubles the time, while simultaneously reducing mem- ory usage approximately 1/7 of the original. This highlights that, compared to CPU-offloading methods, low-bit compression offers a more favorable balance between memory usage and time consumption.
17

---

Published as a conference paper at ICLR 2026
Table 7: The optimizer memory usage (GB) of different methods.
Model Size
130M
350M
1.1B
3B
5B
fp32-Muon
0.41
0.94
3.88
11.81
19.97
fp32-Muon-CCP
0
0
0
0
4-bit-Muon-GRAA (rank 1/4)
0.11
0.25
1.06
3.03
5.42
4-bit-Muon-GRASP (rank 1/16)
0.06
0.19
0.81
2.07
3.79
4-bit-Muon-GFB (rank 1/64)
0.05
0.16
0.74
1.92
2.86
B.2 DETAILED BREAKDOWN OPTIMIZER TIME USAGE
Table 8: Time (ms) of different optimizer logic.
Model size (rank) 130M(1/4) 130M(1/16) 130M(1/64) 350M(1/4) 350M(1/16)
350M(1/64)
1.1B(I4) 1.IB(1/16)
1.1B(1/44)
Power iteration
46
410
129
58
1010
242
NS iteration
98
98
98
224
224
224
378
378
378
peram update
12
12
22
22
22
Here, we provide a detailed breakdown of the cost of power iteration versus the rest of the optimizer logic and the results are shown in Tab 8. We find that when the rank is selected as 1/16, the time required for Power lteration is less than that of the NS iteration. However, when the rank is larger (1/4), the QR decomposition in Power Iteration becomes relatively time-consuming and exceeds the time required for the NS iteration. This pattern holds across different model sizes. In the overall training steps, the forward and backward processes account for the majority of the time, with the increase in optimizer time (0.2-0.6 seconds) considered a minimal overhead.
B.3 MORE ABLATION RESULTS OF DIFFERENT RANKS.
Table 9: The optimizer memory usage (GB) of different ranks.
Model Size
130M
350M
I.IB
3B
5B
4bit-Muun-GRASP (rank 1/4)
0.11
0.25
1.06
3.03
5.42
1-bit-Muon-GRASP (rank 1/8)
0.08
0.23
0.92
2.57
4.62
--bttMMunnGRRAPP (rank 1/16)
0.06
0.19
0.81
2.07
3.79
4 bit-Muun-GGRAP (rank 1/32)
0.05
0.17
0.78
1.99
3.32
4 -itt-Muon-GRASP (rank 1/64)
0.05
0.16
0.74
1.92
2.86
Here, we also provide the memory usage and time of different rank chosen, and the results are shown Tab. 9 and Tab. 10. To highlight the contrast more clearly, here we focus only on the optimizer itself. These results reveal a trade-off: as the rank increases, model performance improves, but this comes at the cost of increased time consumption and memory usage. Consequently, in practical training, we propose selecting the highest possible rank that remains within the acceptable limits of memory and time usage to achieve a good performance. Furthermore, we have included additional visualizations of the singular value distributions in Appendix B.5, which reveal that the distributions exhibit similar patterns across different model sizes. Therefore, the rank selected for smaller models also be transferred larger models.
B.4 DATA FORMATS AND LEARNING RATES
Here, we compare the impact of FP4 and INT4 data formats on quantization. The implementation of INT4 is straightforward: after dividing by the normalization scale, we apply the round() function. For FP4 representation, we adopt the E2MI format as defined in prior studies (Rouhani et al., 2023), which includes the following values:
(-6, 4, 3, 22, 1.5, 0.5, 0.5, 0,5,0,,05,,111,,,,22
(14)
18

---

Published as a conference paper at ICLR 2026
Table 10: The optimizer updating time(s) of different ranks.
Model Size
130M
350M
1.1B
3B
5B
Bit-MuunnGRRASS (rank 1/4)
0.55
0.91
1.82
2.85
4.21
A-bit-Muon-GRAAA (rank 1/8)
0.46
0.72
1.30
1.94
2.72
bit-Muuo-GGRASP (rank 1/16)
0.41
0.61
0.96
1.21
1.93
4- -bit-MMuon-GRASP (rank 1/32)
0.40
0.56
0.83
1.04
1.60
4-it-Muon-GRASP (rank 1/64)
0.39
0.54
0.76
0.88
1.43
INT4 vs FP4 (E2M1)
Model Size 1.1B
4.0
4.0
int4-1.1B
Ir6e-4
fp4-1.1B
Irle-3
3.6
3.6
int4-350M
Ir2e-3
3.4
fp4-350M
Loss
3.4
Ir3e-4
3.2
3.0
3.0
2.6
2.6
5
10
0
10
Tokens (B)
Tokens (B)
Figure 9: Left: Comparision of INT4 and FP4 data format. Right: Different learning rate.
cingular value distribution
singular distribution
singular distribation
10a
Singslar Valun
Singular Valoe
10-2
Singular Value
10
10
10
10
10
roal
quant
10-9
real
quant
10-1
quant
real
200
400 Index
600
200
600
1000
500
1000
1500
2000
Indea
Indox
singular distribution
singular distribution
singular distribution
100
10a
10-1
Singula Valun
10-1
Singular Value
30
Singular Value
10-2
10-3
10"3
NS(rral)
NStroal)
10-5
NStreal)
NS(quani)
10-4
NS(quanl)
NS(quant)
400 Index
600
1000 1500
2000
Index
Index
(a) 130M
(b) 350M
(c) 1.IB
Figure 10: Visualization of momentum in transformerloeer...aam.. proj in different size of LLaMA model.
Following Wang et al. (2025), we implement a look-up table for FP4 quantization within Triton kernel. Quantization functions typically involve element-wise operations large datasets, which can be parallelized to leverage the highly parallel computing power GPUs. Fig. 9 (left) shows that the training curves for both data formats nearly identical, but the look-up process in FP4 introduces a slight efficiency overhead.
We also compare the effect of different learning rates the convergence of training with 4-bit- optimizer. We select learning rates from the set (3c-4, 6c-4, le-3, 3e-3). Figure (right) shows that Abit-Muon-GRAAS converges to similar levels across different learning rates.
19

---

Published as a conference paper at ICLR 2026
B.5 SINGULAR VALUE DISTRIBUTION OF THE MOMENT MATRICES FOR MODELS OF DIFFERENT SIZES.
Here present single representative visualization illustrating the singular value distribution of the moment matrices across different model scales, see Fig 10.
C BROADER DISCUSSION OF LOW-BIT-MUON BEYOND LLMS
Beyond L.M pretraining, low-bit optimizers can benefit a wide range memory-constrained learn- ing settings (Qin et al., 2025; Lu al., 2024). In particular, Muon has already demonstrated strong empirical performance beyond language modeling, including in computer vision and other large- scale training regimes, highlighting the generality of optimizer-level improvements. combinato- rial optimization, where neural solvers often operate on large graphs and long trajectories (Wang et al., 2023; Guo et al., 2024; Zhang et al., 2024a; et al., 2025a; Zheng et al., 2024), reducing optimizer-state precision directly lowers the training memory footprint and enables scaling to larger problem instances. Similar benefits arise in bioinformatics (Wu et al., 2024a; Bian et al., 2026), where tigh-dimensional biological data and limited hardware budgets make memory efficiency practical bottleneck.
Similarly, in generative modeling-includdngg diffusion models, autoregressive transformers (Li et al., 2025b;b), and graph-base basph-based molecule generators terining (Yang et al., 2024b; Wu et al,, 2024b; Yang et al., 2025b) is frequently dominated not only by model parameters but also by opti- mizer states. Low-bit optimizers can therefore substantially reduce the training memory footprint, facilitating scaling higher resolutions, longer generation horizons, or larger latent spaces. This is particularly beneficial for scientific generative tasks such as molecular and materials design, where models must process complex graph or 3D structural representations. More broadly, low-bit opti- mizers provide an orthogonal direction to existing efficiency techniques such as parameter-efficiee training and model quantization, targeting the oveen-overlooked memory cost of optimizer states and enabling scalable learning across wide range of domains beyond LLMs.
20


Reviews:
[{"review_id": "0T4MPYYgAM", "reviewer": "Reviewer_H32P", "summary": "The authors propose 4-bit-Muon-GRASP, a novel approach for reducing storage requirements for Muon’s momentum state. This is important for practitioners training in memory-constrained scenarios. The authors find that directly quantizing Muon’s momentum to 4 bits leads to large errors relative to an FP32 baseline after Newton Shulz (NS) orthogonalization is applied. The authors show that these errors are mainly caused by quantizing the top singular space. Therefore, they break the quantization problem into 2 parts to reduce errors caused by NS: (1) they quantize the top singular subspace to 8 bits, and (2) they quantize the remainder to 2 bits. In their experiments spanning language model pre-training and finetuning, the authors achieve this naturally through power iteration. In their experiments, the authors show that 4-bit-Muon-GRASP nearly matched the convergence of the FP32 baseline.", "questions": "- Why are the singular values all below 1 for the NS(real) in Figure 1 (d)? Have you swapped the labels by accident? If not, the plot reads like the quantized NS is closer to UV^T (singular values 1) than real NS.\n- I’m not sure exactly what rank refers to in Table 1.  I assume you are referring to the number of dimensions in the top singular subspace, but the writing does not make this clear.\n- In a distributed optimization setting, [1] finds that Muon trajectories can be quantized to 2 bits. It could be interesting to discuss how this relates to your work.\n- What does official choice mean in Figure 2?\n\n\n**Nit-picks/suggestions:**\n- Figure 2:  The y-axis could show “Quantization Error RE(A,B)” without needing a title\n- line 244 missing citation\n- equation 8, tensors should be in mxk and nxk\n\n[1][MuLoCo: Muon is a practical inner optimizer for DiLoCo]", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to understand and follows a coherent story. \n- The experiments provide a good evaluation of the proposed method and are well-executed. \n- The authors provide the timing and memory complexity of their method.\n- The approach is intuitive.", "weaknesses": "- Due to grid quantization’s reliance on both the column and row scales, the dequantization step is not compatible with optimizer state sharding(zero-1/fsdp) out of the box, despite the combination of these methods being relevant for low-memory training. \n- While 8bit-Muon is included in Figure 6 (memory usage), I miss a comparison to it in most other performance-related experiments. It seems like 8-bit Muon is a much simpler approach than 4-bit-Muon-GRASP, so it would be beneficial for practitioners to understand how it performs.\n- I miss a description of how hyperparameters were chosen? Were the optimal values an interior point of all the values searched? This is crucial to have sound results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gHjZ6GwPIM", "reviewer": "Reviewer_TV6R", "summary": "The authors propose a variant of Muon, where the optimization state M is quantized into a low-bit version. By observing that the singular vectors with large singular values will have large quantization errors after NS, the authors propose to use 8-bit quantization for the low rank matrix associated with the singular vectors with large singular values and to use 4-bit quantization for the residual part. To further decrease quantization error, the authors propose to use grid quantization. The experimental results show that the proposed low-bit Muon achieves similar performance to the 32-bit version, while costing less memory.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors identify that after NS iteration, the relative error of the matrix associated with the singular vectors with large singular values will be amplified a lot. \n\n2. Based on the observation, the authors propose an algorithm that uses different precision for different classes of singular vectors.\n\n3. To further decrease the quantization error, the authors propose grid quantization method that scales the matrix according to the maximum absolute value in each row and each column.\n\n4. The experimental results show a similar performance between the proposed algorithm and the 32-bit Muon.", "weaknesses": "1. Why do the authors use a single iteration of the power method? Is it enough for the algorithm to identify the top singular vector? Will it affect the performance?\n\n2. The overall time seems to be more than the 32-bit version. How does the time compare to the CPU-offload muon, which costs 0 memory on GPU but costs more time than the 32-bit version, especially for large models and large k (the rank of Q).\n\n3. The selection of rank will affect the performance even for 350M model. How to choose the rank for even a large model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bFP9Q8wwQX", "reviewer": "Reviewer_r7tv", "summary": "The paper proposes an approach to quantizer momentum state in Muon optimizer to 4-bit. The approach involves preserving the singular values of momentum in 8-bit while the residual components are quantized to 4-bit. The idea of subspace preserving mixed precision quantization is intuitive and the grid based quantization scheme seem to work better than group based quantization.\nTo my knowledge, this is the first work to perform 4-bit quantization of Muon optimizer so there aren't any related baselines to compare with. The evaluations in this work are thorough and the ablations are useful to motivate the approach.", "questions": "1. Do the authors have any insight on the performance of quantized 4-bit Muon v/s 4-bit AdamW optimizer? \n2. Is muon optimizer more difficult to quantize than adamw optimizer?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach seems to work compared to baseline approach of 4-bit Muon quantization without subspace preservation.", "weaknesses": "1. Need to show results on multiple seeds.\n2. Code is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "60vbJ7lMUo", "reviewer": "Reviewer_n2Ba", "summary": "This paper addresses the challenge of reducing the memory footprint of the Muon optimizer, which is already a memory-efficient alternative to AdamW as it only stores the first moment. The authors investigate applying 4-bit quantization to Muon's optimizer states, a non-trivial task as they demonstrate that Muon's core orthogonalization step (the Newton-Schulz iteration) significantly amplifies quantization errors.\n\nThe paper identifies that this error amplification originates primarily from the top singular subspace of the moment matrix. To solve this, the authors propose 4-bit-Muon-GRASP (GRid And Subspace Preserving).\n\nExperiments on LLaMA pre-training (up to 1.1B) and fine-tuning of 7B models show that 4-bit-Muon-GRASP achieves accuracy comparable to its full-precision counterpart, while reducing total training memory consumption by up to 28% compared to fp32-Muon.", "questions": "1.  The main experiments use a rank of 1/16 for the top subspace. The ablation in Fig. 7a shows that rank matters, and 1/16 is a clear compromise between 1/8 and 1/32. How sensitive is this hyperparameter? Does it need to be re-tuned for different model sizes or architectures, or is 1/16 generally robust?\n2.  The ablation in Fig. 7c shows that discarding the residual singular space leads to poor performance. This is a key finding. Does this imply that the primary insight is that Muon (due to its orthogonalization) *cannot* be approximated by a simple low-rank matrix, and *must* retain the full-rank information, even if the residual part is heavily quantized?\n3.  Regarding the computational overhead, how does the cost of the single Power Iteration step compare to the cost of the 5-step NS iteration? A more granular profiling would help in understanding the trade-offs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper identifies the Newton-Schulz (NS) iteration as an amplifier for quantization error, and successfully pinpoints the source of the error amplification to the top singular subspace.\n* The proposed 4-bit-Muon-GRASP method is a logical and novel response to the identified problem. It directly treats the \"sensitive\" top subspace with higher precision (8-bit) while aggressively compressing the \"stable\" residual (4-bit).\n* The introduction of grid quantization to handle outliers in both row and column dimensions is a simple but effective technique that outperforms standard group quantization.\n* The paper demonstrates that 4-bit-Muon-GRASP achieves its goal, matching the performance of the full-precision fp32-Muon optimizer on both pre-training (130M, 350M, 1.1B models) and challenging fine-tuning tasks (7B models).", "weaknesses": "1.  **Limited Pre-training Scale:** The pre-training experiments are limited to models up to 1.1B parameters. While fine-tuning is performed on 7B models, the most critical need for optimizer memory savings is during large-scale pre-training. The memory gains are also less substantial at smaller scales. Figure 6 compellingly shows projected memory savings for 3B and 5B models, but lacks the corresponding training loss curves to prove that GRASP holds up at that scale. The authors acknowledge this limitation.\n2.  **Baselines for 4-bit Muon:** The main low-bit baseline, \"4-bit-Muon-base\", is a naive implementation. The paper's argument would be stronger if it discussed why other, more sophisticated 4-bit quantization techniques would *also* fail. Is the NS-iteration-based amplification a fundamental blocker for *all* previous quantization methods?\n3.  **Computational Overhead of Subspace Preservation:** The method adds a Power Iteration step at every iteration to compute the top singular vectors. While Table 3 shows the total step time overhead is \"minimal\", a more detailed breakdown of the cost of this step versus the rest of the optimizer logic (including the NS iterations) would be beneficial.\n4.  **Clarity of Algorithm 1:** Algorithm 1 is slightly difficult to parse. For example, the `PowerIter` function also uses `Orthogonalize(P)` (line 3) which is implemented via QR decomposition, but the main text refers to this as \"column normalization\", which is a bit ambiguous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)