A Convergence Analysis of Adaptive Optimizers under Floating-point Quantization
Authors: Xuan Tang
Jichu Li
Difan Zou
Venue: ICLR 2026 Poster (2026)
Keywords: Convergence Theory
Non-convex Optimization
Adaptive Optimization
Quantization
Low-bit Training
Primary area: optimization
TL;DR: We deliver the first rigorous convergence analysis for Adam and Muon when considering FP quantization for the entire training process.

Abstract: The rapid scaling of large language models (LLMs) has made low-precision training essential for reducing memory, improving efficiency, and enabling larger models and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware-aware quantization, leaving open the question of why low-precision training remains effective. We introduce the first theoretical framework for analyzing the convergence of adaptive optimizers, including Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quantization errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis further reveals that Adam is highly sensitive to weights and second-moment quantization due to its reliance on $\beta_2 \to 1$, while Muon requires weaker error control and is thus potentially more robust. These results narrow the gap between empirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.

Published as a conference paper at ICLR 2026
A CONVERGENCE ANALYSIS OF ADAPTIVE OPTIMIZ- ERS UNDER FLOATING-POINT QUANTIZATION
Xuan Tang"
Jichu Li" School of Statistics
School Computing and Data Science The University of Hong Kong angnt connect. hku.hk
Renmin University of China Lijichu52@@maal com
Difan Zou
School of Computing and Data Science & Institute of Data Science The University of Hong Kong dzoužhku. hk
ABSTRACT
The rapid scaling of large language models (LLMs) has made low-precision train- ing essential for reducing memory, improving efficiency, and enabling larger mod- els and datasets. Existing convergence theories for adaptive optimizers, however, assume all components are exact and neglect hardware- aware quantization, leav- ing open the question of why low-precision training remains effective. We in- troduce the first theoretical framework for analyzing the convergence of adap- tive optimizers, including Adam and Muon, under floating poiit quantization of gradients, weights, and optimizer states (e.g.. moment estimates). Within this framework, we derive convergence rates on smooth non-convex objectives under standard stochastic gradient assumptions, explicitly characterizing how quanti- zation errors from different components affect convergence. We show that both algorithms retain rates close to their full-precision counterparts provided mantissa length scales only logarithmically with the number of iterations. Our analysis fur- ther reveals that Adam is highly sensitive to weights and second moomntt quanti- control
zation due to its reliance on B2 + 1, while Muon requires weaker and is thus potentially more robust. These results narrow the gap between em- pirical success and theoretical understanding of low-precision training methods. Numerical experiments on synthetic and real-world data corroborate our theory.
INTRODUCTION
The rapid scaling of large language models (LLMs) has made low- low-precision training indispensable for modern deep learning. By reducing memory usage and improving computational efficiency, low-precision formats such as bfloat16 (BF16) and FP8 enable training with larger models and datasets contemporary hardware accelerators (Peng al. 2023 Fishman et al., 2025). The introduction of FP8 in Nvidia's s Hopper GPU architecture (NVIDIA | Micikevicius et al. 2022) further cements its role practical datatype for the next generation of LLM training. In practice, numerous frameworks now leverage mixed- or low-precision formats to quantize gradients, weights, and optimizer states (Liu al. 2024 2025). showing that aggressively quantized training can scale to trillion-token workloads without loss of accuracy.
Despite its empirical success, rigorous theoretical understanding of quantization, particularly for adaptive optimizers like Adam (Kingmal 2014) with decoupled weight decay (Loshchilov & Hutter] 2019) and Muon (Jordan et al. (2024). which are widely used in practice, remain largely under- developed. Existing theoretical work on the non-convex optimization analysis under quantization has primarily focused on Stochastic Gradient Descent with quantized gradients (QSGD) (Alistarh) Jiang al. 2017). For example, Jiang& & Agrawal (2018) established (1/T1/44 convergence under un-
"Equal contribution.

---

Published as a conference paper at ICLR 2026
biased quantization, while error-feedback mechanisms (Karimireddy et al., 2019) were later intro- duced to handle biased quantization with the same guarantees. Extensions to QSGD and Quantized SGDM with error feedback have been analyzed in various settings (Tang et al.
2019 Zheng et al.
2019 Koloskova al. 2020), again achieving O (1/TT1/4 rates. More recent efforts target Quan-
lized Adam (Chen et al. 2021 Modoranu et al. 2024, Ozkara et al., 2025). [Chnn et al. (2021)
proved convergence of Adam with quantized gradients and weights under error feedback achieves O(1/T//1), but the method requires storing error terms for every parameter, which is memory- intensive and impractical for modern low-precision LLM training. |Moorrau et al. (2024) reduced
this cost by compressing error feedback with unbiased compression, proving O11 (1/ conver- gence for Adam with quantized gradients. Ozkara et al. (2025) further explored stochastic rounding (SR) as a mechanism for mitigating numerical errors in low-precision training, providing analyses of implicit regularization and convergence of Adam under SR; however, their analysis omits optimizer state quantization or practical floating-point formats, which are increasingly central to low-bit LLM optimization (Dettmers et al.) 2021; Xi et al. 2025 Fishman et al. (025). This leaves critical gap: practical low-bit -it training cructally involves the quantization of optimizer states (c.g., momentum and second-moment estimates), component these analyses omit. Furthermore, these studies often rely on assumptions like unbiased quantization or error-feedback mechanisms that are not consistent with modern large-scale LLM training. Consequently, the community lacks theoretical framework to explain the robust convergence observed when adaptive optimizers are quantized in all components during LLM training.
This paper. The objective of this work is to develop the convergence analysis of adaptive opti- mization algorithms with more practical quantization configuration. In particular, we develop the first analytical framework for quantized adaptive optimizers under floating- point quantization. More importantly, following the practical configuration (Liu et al. 2024), our framework explicitly mod- els the quantization of all key components: gradients, weights, momentum, and second moments. We then establish convergence guarantees for both Adam and Muon optimizers, expressing the sults as a function of the quantization errors in these components. This clearly reveals how each type of error individually affects convergence. Crucially, rather than relying on unbiased quanti- zation assumptions or storing per-parameter error feedback, we require only relative error control, which aligns with the behavior of standard floating ppintt formats (FP32 - BF16 or FP8; Section Figure 5
10 see also (Kuzmin et al. 2022
We then summarize the main contributions of this work as follows:
We introduce a rigorous analytical framework for adaptive optimizers under hardware- low- precision training. explicitly modeling the quantization of weights, gradients, and optimizer states (Section 3) Unlike prior works that rely on unbiased quantization assumptions or crror-feedback mechanisms, which are impractical in large-scale LLM training, adopt a relative error model (Assumption 3.I) that faithfully captures the behavior of floating- point quantization. This facili- tates formal and rigorous convergence analysis for quantized adaptive optimization algorithms that closely align with rorl-world implementations.
We provide the first convergence guarantees for quantized Adam (Theorem 4.5) and Muon (The- orem 4.6) on smooth non-convex objectives under the relative eгror quantization model (Assump- tion 3.1. which closely reflects the behavior of floating-poin ppintt quantization. Our analysis shows that both methods attain the same convergence rates as their full-precision counterparts (Défossez) et al. 2022 Shen et al.) 2025), provided the mantissa length increases only logarithmically with the number of iterations, which is consistent with practical hardware precision.
Our analysis in Theorems 4.5 and 4.6 precisely characterizes how quantization errors in different components impact convergence. Notably, we show that Adam is particularly sensitive to quan- tization of weights and second moments due to their dependence on B2, which is typically set close to for convergence in practice and theory (Figure 77 (Zhang et al., 2022). This aligns with empirical observations from from/Peng al. (2023): Yu et al. (2024), where weights and second moments require slightly higher precision than gradients or the momentum. Our experiments (Figures 3 4 9 12 corroborate this, demonstrating graceful degradation with reduced preci- sion and near full full-precision performance at moderate mantissa lengths. In contrast, Theorem 4.6 reveals that Muon is more tolerant to quantization, requiring weaker relative error conditions (e.g., 1 /T1// versus 1 1/72 for Adam). This robustness stems from the SVD-based sign operator in
2

---

Published as a conference paper at ICLR 2026
Muon, which avoids the amplification of quantization errors by the inverse square root of histori- cal gradient variances. This theoretical insight also explains empirical findings in Liu et al. (2025) that Muon exhibits superior robustness to low-precision training compared to Adam (Figure B
Overall, our results narrow the gap between the empirical success of quantized adaptive training and its theoretical understanding, providing foundation for analyzing and designing future low- precision optimization algorithms.
Notations. Scalars are denoted by lowercase letters (s,...), vectors by bold lowercase (x, ..,..... and matrices by bold upperease ..). The i-th entry of x is Vi. and the (i, entry of X is Xij. The lz norm of x is ||x||2 = vE, xi, the Frobenius norm of X is | = v E X², and the nuclear norm of is ||X|| = E, (X), where (X) denotes the i-th singular value. For For real sequences {ar} and {br]. we write a, = (bb)) if there exist constants C, .N > 0 such that a, S Cb, for all t 2 N; a = [((be) if b = Oaa); a, = ((b)) if both 0 = Ob))) and (2(b:); and we use (-)) and f2(-) to suppress logarithmic factors. The quantization operator is 2(-). with d0 denoting the quantized version of x.
d N let [d] {1,2. ..... (1.,.... d}.
2 RELATED WORK
Adaptive Optimization. Adaptive optimizers are key part of deep learning because they can automatically respond to changes in the data. The progression of modern adaptive optimizers began with Adagrad (Duchi al. 2011), which scales learning rates based on the accumulated sum of past
squared gradients. Despite extensive convergence analysis (Zou et al., 2019 Chen et al.] 2018 Shi
al. 2020 Li & Orabona 2019 Faw et al., 2022). its aggressive learning rate decay often leads to premature stalling. RMSProp (Hinton et al. 2012) addressed this issue by using an exponentially decaying average of squared gradients instead, a method whose convergence has also been well- studied (Zaheer et al.] 2018 De et al. 2018 Shi et al. 2020 Li et al. 2025). Adam (Kingmal 2014) then synthesized these ideas by incorporating momentum, effectively combining the adaptive learn- ing rates of RMSProp with first-moment estimates. Its widespread success has motivated a vast body of theoretical work analyzing its convergence and implicit bias generalization under various settings
(Reddi et al.
2018
Défossez et al. 2022
Zou et al.) [001 Chen et al. 2018 Zhang et al. 2022
Wang et al. 2023 Zhang 2024 Zou et al. 2023 Cattaneo et al.) 2024). More recently, the Muon optimizer (Jor- dan et al.]
Wang et al.
2022; Guo et al. [2021
Hong & Lin 2023 Li et al. 2023
al. [025
2024) was proposed, which leverages a matrix- based perspective for optimization, with its convergence guarantees established by concurrent works (Shen al. 2025 Saao et al.) (025).
While convergence guarantees for these methods have been established in high-precision settings, their behavior under the low-precision quantization common in modern large model training is not well understood, gap that this paper aims to address.
Low-bit Training. As the field of deep learning continues to advance rapidly, the scale of mod- els, particularly Large Language Models (LLMs), has grown exponentially. Low precision train- ing (Wang et al. 2018; Wortsman et al.) 2023 Liu et al,, 2023 Xi et al.; 2024 Liu et al. 2024) has become prominent technique in modern deep learning, offering reductions both computa- tional costs and memory requirements. Mixed-precision
training typically performs forward and backward passes in low-precision formats like FP16 (Micikevicius et al. 2017) the more stable, wider-range BF16 (Kalamkar et al., 2019), while maintaining master weights and optimizer states in FP32. The advent of hardware like NVIDIA's Hopper GPU architecture (NVIDIA 2022) has made 8-bit floating-poii point (FP8) training practical reality for further efficiency gains (Micikevicius et al.)
2022 Peng et al. 2023 Xi et al. 2005 Fishman et al. (025). ). Even more aggressive approaches now extend to 4-bit (FP4) training (Wang et al.] 2025 Zhou et al. (025). Especially in adaptive op- timization, the optimizer states can consume as much memory as the model parameters themselves. This has motivated a class of methods that specifically compresses these states, decompressing them to higher precision just-in-timg for the weight update to save memory (Dettmers et al. 2021 Peng et al. 2023 Li et al. 2024 Fishman et al.) 2025: Xi et al. 2025). Despite the empirical success of these techniques, a comprehensive theory explaining their convergence behavior remains absent. Our work addresses this gap by establishing analytical framework that formally incorporates
3

---

Published as a conference paper at ICLR 2026
quantization errors from all parts of a realistic low-bit training pipeline, from gradients and weights to the crucial optimizer states themselves.
Quantization Convergence. Most convergence guarantees for optimizers assume ideal, high- precision arithmetic, failing to account for the quantization effects inherent in modern large-scale training. Much of the existing theoretical work in this area has therefore focused on the convergence of Quantized Stochastic Gradient Descent (SGD). Early analyses established convergence rates for
SGD with quantized gradients, often relying on the strong assumption of an unbiased quantizer (Alistarh et al. 2017 Jiang & Agrawal 2018 Wen et al., 2017). To handle more practical, biased quantization schemes, subsequent work introduced error-feedback mechanisms to compensate for the quantization bias and still guarantee convergence (Karimireddy et al. 2019 Zheng et al.) 2019)
Tang et al. 2019 Koloskova et al., 2020). Complementing these efforts on gradient compression, another line of research has analyzed the convergence of SGD when the model weights themselves are also quantized (Markov et al., 2023). Beyond SGD, analyzing quantized adaptive optimizers is a more recent challenge. Early work in this direction includes Hou et al. (2019), which studies Adam with 31 = O, analyzing joint quantization of both gradients and weights in convex settings. Other studies have applied error-feedback ensure the convergence of Adam under quantized weights and gradients in non-convex settings (Chen et al. 2021] Modoranu et al., 2024 Robert et al.) 2025). However, these existing analyses for adaptive methods rely heavily on error- feedback mechanisms, which are often impractical in state-of-the-ar LLM training pipelines (Xi et al. 2025 | Fishman et al.)
2025). Complementary work on stochastic rounding (SR) (Ozkara al. 2025) studies a different quantization regime: SR is approximately unbiased but introduces variance, and their Adam anal- ysis quantizes only the final weight update while assuming full-precision gradients and optimizer states (with 31 0). Such additive-noise formulation and simplification avoid the recursive and interaction interaction-heavy quantization error propagation that arises in adaptive optimization under re- alistic moating-point rounding. In contrast, work addresses this critical gap by providing the first convergence framework for adaptive optimizers under a realistic floating-pois error model that covers all components of the training process, without resorting to mredbfee or unbiasedness assumptions.
3 PRELIMINARIES AND PROBLEM SETUP
3.1 PRELIMINARIES
We begin by formalizing the quantization operator and its error properties. Our focus is on floating- point quantization, which is widely adopted in practice. Compared to integer quantization, floating-
point formats achieve strictly smaller reconstruction errors due to their exponent scaling (Kuzmin
et al. 2022). This explains why most large-scale scale low-precision training frameworks rely on floating-
point representations, including recent FP8 and mixed-preeiste systems (Peng et al., 2023 Liu
et al. 2024 Fishman al. 2025).
Floating- point quantization. Let Q + R be a scalar quantization operator applied elemen- twise to vectors and matrices. We illustrate Q through the common case of quantizing from single precision (fp32) to brain floating-poin point (bf16). The fp32 format uses 1 sign bit, exponent bits, and 23 mantissa bits (total 32 bits) (IEEE 2019), while bf16 keeps the same sign and exponent layout but truncates the mantissa to 7 bits Wang & Kanwar 2019). Thus, FP32 can be written as
Tpp32 = (-1)$ x 22-22 x (1.Mo.22),
where S is the sign bit, E the exponent, and Mo.22 the mantissa bits. Quantization discards the low-order 16 mantissa bits M7.22, possibly with rounding or truncation. The BF16 number becomes
Pans = (-1)$ E (1.Mos + C-72
where C (0, 1) is carry bit from rounding. Dequantization pads the truncated mantissa with zeros to recover an fp32 value. Figure I visualizes this process.
Relative error. The above construction implies that the quantization error satisfies
1b616 Zipa2 = |C 2 0..7.22 2- 2 < 26-277 z g/urpa2),
4

---

Published as a conference paper at ICLR 2026
Exponent
Maantsssa
Mer
hass
poto
p117
(M10
(Mia
Ive
Rounding Caryy
BF16
E0
M1
N2
NS
144
M4
198
M7
MIo
M12
Mis
M14
141
(M16
Mt7 jitte j (M20 (M21 (M22)
FP82
E0
E0
E7
M8
M1
N2
M4
108
Figure I: Floating point quantization from fp32 to bf16. Only the mantissa truncated, while sign and exponent remain unchanged.
where (22 and M is the mantissa length of the target format (here M = 7 for bf16). More generally, assume the absence of underflow and overflow, so that the sign and exponent remain unchanged after quantization. This prevents large quantization errors and guarantees convergence of the quantization process. In practice, this assumption well justified: low-precision LLM training commonly employs engineering techniques such as per-tensor per-channel scaling, which ensure that post-ouantizn post-quantization values remain within the representable range (Peng et. 2023] Fishman al.) 2025). Under this condition, the relative quantization error decays exponentially with the number of mantissa bits-a property that is intrinsic to floating-point representations. This observation motivates the following assumption.
Assumption 3.1 (Quantization Error). Let Q R + elementwise. Then, for any x IR, the quantization error is relatively bounded:
R be scalar quantization operator applied
l - x S g/
where q = O(2-M), and M is the mantissa length of the target floating-point format.
3.2
PROBLEM SETUP
with low-precision training under analytical quantization framework shown in Figure Formally, the goal to minimize the loss:
We study stochastic optimization (3.1
min WERmXn
F(W) = (e E)
(3.1)
where W denotes the model parameters, a random variable representing the data, and f(W; the sample loss. We denote F* infw F F(W) -oo as the optimal objective value.
Low-precision training framework. During training, both computation and communication are constrained by memory and bandwidth. Modern practice therefore quantizes weights, gradients, and optimizer states into lower-precision formats (e.g., BF16, FP8) to accelerate training (Peng et al., 2023 Liu et al., 2024 Fishman et al. (025). We model this process with the analytical framework shown in Figure
The key steps are:
The master maintains full-precision weights W, but transmits their quantized version W to workers.
2. Workers perform forward and backward passes with W compute gradients V VWW? E), quan- tize them, and send quantized gradients back.
The master dequantizes gradients, updates quantized optimizer states (e.g., momentum, second moment), and applies the optimizer update. Updated states are re-quantized for storage.
The first two steps can be illustrated by Algorithm I while the third step depends on the choice of optimizer (e.g., Adam in Algorithm 2 or Muon Algorithm 3). The dashed arrows in Figure highlight the quantization operations applied to weights, gradients, and optimizer states within the proposed framework.
Relative errors.
We denote the relative errors q of different components after applying Q Qw (weights),
[G
(gradients),
QM (first moment),
(second moment).
5

---

Published as a conference paper at ICLR 2026
Master Weight
Forward
Output
Gradient
Input
Optimizer States
Backward
Loss
Figure 2: An analytical low-precision training framework
Algorithm Analytical Adaptive Method Quantization Training Framework
1: Input 1: Algorithm A (Adam, Muon] and its parameters set θ E (2,1,22.)) c), {8)), initial weights Wo learning rate schedule (ne). ]. batch size B, quantization operator Q
2: for 0, ................................ T [--1 -100 do
3: Sample batch He uniformly
D B workers
4: G. Σe i= W Er,))
Master receives B quantized gradients D Update by Adam or Muon
5: Wo+1 A(W, G2 O,t)
6: end for
Algorithm Adam W., G,, 0,t)
Algorithm Muon Ww G2, O,t)
1: [8,, 32,,8,,c++ + +
1: {B} + O
2: M, B:M G. ift else Mo
M. -BM (1 B)G, if > else
Go
Mg = Go
3: V 1 G ift 0 else V, = G
(U,, V.).=.SSD
4:
return W, - VV..MM.VV...
4: return W. - -VVVVV
Each error term arises from applying a floating-poo point quantization operator Q that satisfies Assump- tion 3.1. Formally, for any quantized quantity X, (e.g.. W, G., M,, V,) iteration t, its relative quantization error is defined as the smallest constant 2 0 such that
|x Xclos s ax [Xlol
Vt ......777 -
In particular, have
Uw = inf {q 0: [W?), W. < 4/ (Wiee W)..
1G = inf {( 2 0: G [. < G. VI).
4M == inf b) ( 0: [M)], [M]y] S 9 [M, [M].] Vt..
ęV == inf p ( 2 0: [V9), - [V.]el S [[[..]lall.v)
This framework is more general than most prior theoretical analyses, which typically consider quantization of only subset of components (c.g., gradients).
Optimizers. We focus two adaptive optimizers: Adam (Kingma 2014) and Muon (Jordan cct al. 2024). Algorithm outlines the general quantized training loop, while Algorithms 2 and detail the specific update rules for Adam"" and Muon, respectively. Note that the quantization oper- ator Q can represent any floating-por toiiigg-point quantization (e.g., fp32 + bf16 or fp8) satisfying Assump- tion 3
In the following sections, we will analyze the convergence of quantized Adam and Muon under this framework with relative quantization errors (qw, 4G, 1MM q.
!Orr Al Algorithm slightly differs from the standard Adam, but will not affect the proof. We provide detailed discussion Appendix A.

---

Published as a conference paper at ICLR 2026
4
MAIN RESULTS
We now present our main theoretical results on the convergence of quantized Adam and Muon under the analytical framework in Section B We begin by stating the assumptions required for our analysis, followed by the convergence theorems for each optimizer.
Assumption 4.1 (Unbiased Stochastic Gradient). The stochastic gradient V ((W;; ) is an unbiased estimator of the true gradient V (F(W). i.e., EIV VF(W).
Assumption 4.2 (Stochastic Gradient Bounds). The stochastic gradient V VfWW; satisfies the fol- lowing bounds depending on the algorithm:
Adam: The stochastic gradient is fo uniformly almost surely bounded, i.e., there exists a constant R Vc (where > 0 is the stability constant used to simplify the final bounds) such that
[Vf(W: floo = max
| [V/(W:)) S R Ve, a.s..
1.j
Muon: The stochastic gradient has bounded variance, i.c., there exists a constant σ > 0 such that
E ((W:)
VF(W) S a².
The objective function mR + R is L-smooth, i.e., for any X, Y mm have
Assumption 4.3 (Smoothness).
VF(X) VF(Y) & L Y |
Assumptions 4.1
4.2
are standard in the analysis of smooth non-convex stochastic optimiza- 2018 Chen al. 2019 Zou et al.) 2019 Défossez al. 2022 Chen al. 2022
and 4.3
tion (Zaheer et al.
Zhang et al. 2022
Wang et al., 2023). They are usually employed to control the stochastic gradient noise and the local geometry the objective function.
We remark that more general (Lo. L1 }-smoothness condition (Zhang et al. 2020), which has
been adopted in recent analyses of Adam (Li et al., [2023] Wang et al., 2024] Hong Lin
2024 allows the smoothness constant depend the gradient norm: [VF(X) Lo L1 [FF(()))) - Y| While this condition can better capture practical deep learning scenarios, since focus is characterizing how quantization errors influence the convergence behavior of Adam and Muon, we adopt the standard L-smoothness assumption for simplicity. Ex-
VF(Y) S
tending our results to (Lo, L smoothhess remains interesting direction for future work.
Finally, we assume the optimization begins from a controlled initialization:
Assumption 4.4 (Bounded Initialization). The initial parameter matrix Wo and its gradient are bounded in Frobenius norm, i.e., |Woll - D, |/VF(Wo))lz I G, for some constants D, 0.
Bounding the initialization ensures that quantization errors remain controlled and their propagation through the optimization iterations can be rigorously analyzed, which is crucial for establishing convergence guarantees under low-precision training.
4.1 THEORETICAL RESULTS OF ADAM
We first present the convergence result of Adam under FP quantization in the following theorem.
Theorem 4.5 (Convergence of Quantized Adam). Suppose Assumptions 3.1
4.1 4.4 hold. Let
d mn be the number of trainable parameters, consider the Quantized Adam algorithm defined in
run for T iterations with 7), (1 3,)4 where Ω4 = / VE B3. Suppose 87 + qMx)
32 qv). (1 9m) < B2(1 qvv), and 2 31) < T, then for iteration index chosen
randomly from (0, ......,. T 1} with P(T j) (1 3/ -J), have:
Fu F. +
Q(T) 2qw Tn- (1 3 mL + qG) RG)
[[IVF(-+)1B3) s 4(1 + (G)
T/
Ve(1 B2)
(1+
pal1-qv)
4(1 + ga)d
(qaR3 Lqw R2 D) + T
C
In 1
e(L 32(1 (2(1-vv)) qv.))
((1 + R2)
-T In -vv1))
Ve(1- - B2)
7

---

Published as a conference paper at ICLR 2026
Gradient Frobenius-noot sis-oom
8000
600
1M M
M8
Gradlent Frobenlus-none
500
M16
6000
M32
4000
26-
200
2000
1000 sids wio skis 18s0
100
2000
4000
6000
6000
10000
9900
9920
9940
9960
9960
10000
Iteration
Iteration
Figure Rosenbrock: Adam gradient norms under different mantissa precisions M (left: full 10,000 iterations; right: last 100 iterations). Larger mantissa bit-lengths yield smaller converged gradient norms. Together with Figure [ this shows that higher precision reduces quantization error and improves convergence, consistent with Theorem 4.5
where C' is a constant depending on the problem hyperparameters, and Q(T) is a function with respect to T, gv, QG: and gM, which approaches zero when gv, -M 0 (please refer to Eq A.43 for their detailed formula).
Moreover, by setting = @(I/VT), 1 - 32 = ((1TT), 2a O(1/T) qи = O(IT)) qv = O(L/T²), = (L1/T²), then Q(T)/T = 1/2) (refer to Eq. A.44 for calculation details) and
B]] = =66 (T-1/4).
Theorem 4.5 provides the first convergence guarantee for Adam under practical floating-pon point quantization model (Peng et al.) 2023; Liu et al. 20244 Fishman et al.) 2025), in contrast to prior works that assume unbiased quantization
error-feedbact mechanisms (Jian & Agrawal 2018 Chen et al. 2021 Modoranu et al. 2024h. The most similar prior theoretical work is Ozkara et al. (2025), whose analysis also builds on Détossez al. (2022;; however, they consider only quantiza- tion of the final weight update, assuming full-precision gradients and optimizer states with 31 = O, and thus do not capture the recursive error propagation arising from fully quantized Adam under Our analysis demonstrates that by setting the hyperparame- ters as (1/v/) and 1 - B2 = O1/TT), and ensuring the relative quantization errors sat- isfy qG. QM O(ITT) and qw. ( Quantized Adam achieves convergence rate of O/441 which successfully matches the established one for its full-precision counterpart in
realistic Doating-poin rounding.
smooth non-convex optimization (Guo al. 2021 Détossez al. 2022 Wang al. 2023 Hong
Lin 2024).
Our theorem further reveals nuanced sensitivity different types of quantization error. The quired precision for the second moment (gv) is stricter than for the first moment (gM). This sensitiv- ity arises because accumulated errors in the second-moment estimate V, are non-linearly amplified by the update step's inverse square root. This theoretical finding provides a rigorous explanation for the empirical observation that the second moment often require higher precision than the first mo-
ment in low-bit training setups (Peng et al., [023] Yu et al. 2024 Fishman et al.) 2025). Similarly,
the stricter precision requirement for weights (qw = O(1 (T)) is necessary to control error aceu- mulation over the entire training trajectory. Our analysis must account for the potential growth of weight magnitudes throughout training, which acts as an amplification factor for the relative quanti- zation error. To guarantee convergence under this worst-case scenario of unbounded weight growth, the proof requires qw to decay rapidly to counteract this amplification. However, this strict condition is consequence of the proof's generality. In practice, where weight norms often remain bounded, this error amplification is less severe, and the precision requirement for qw could relaxed to (1/TT).
8

---

Published as a conference paper at ICLR 2026
500
Gradient Trobenius-nom
8000
M8
Gradlent Frobenlusnnom
mW My
Mmy M
400
M16
6000
M32
300
4000
200
4.00
2000
100
165 2930 skis 1030
2000
4000
6000
6000
10000
9900
9920
9940
9960
9960
10000
Iteration
Iteration
Figure Rosenbrock: Muon gradient norms under different mantissa precisions M (left: full 10,000 iterations; right: last 100 iterations). Larger mantissa bit-lengths yield smaller converged gradient norms. Together with Figure 6 this shows that higher precision reduces quantization error and improves convergence, consistent with Theorem 4.6
4.2 THEORETICAL RESULTS MUON
Then, we present the convergence result of quantized Muon in the following theorem.
Theorem 4.6 (Convergence of Quantized Muon). Suppose Assumptions [3.]] 4.1 hold. Con- sider the Quantized Muon algorithm in and 3 run for 7' iterations with 1): 7/, (( gM) 1, then
F B(((W,) - F( + 1-3 T(1 B)
2
v
B 6o vT 1 VB
E
+
T
r/
Lap + C2 (9G +
9MB
qw + taTIn gw qw Tw
(1 + Tn)
2
1 B( QM
where C2 absolute constant, r = minfm, m, n}. Moreover, suppose F(Wo) - F A for constant
0, set = 9(T )2),222))))) 17 = ((T 3/4), and = 1, qa = qw = qM = O(T 1/2), then
=
T
Theorem 4.6 establishes the convergence of Quantized Muon under relative quantization errors (qw, qa, (M) for weights, gradients, and momentum, respectively- practical setting for low- precision training (Peng al. 2023 Liu et al.) 2024 Fishman al. 2025). in contrast to prior works that assume unbiased quantization or feror-feedback mechanisms (Jiang & Agrawal 2018 Chen al. 2021, Modoranu et al. 2024). As sanity check, when qw QG QM
result recovers the exact convergence rate (///T1/44 of (/hhe et al. (2025) up to constant factors. More importantly, as long as the mantissa length the floating-poimt format scales logarithmically with T, i.e., aloog T), the quantization errors decay as qp = da = gw = OT With appropriate choices of 7) and (as in Theorem 4.6). the full-precision convergence rate (TT-1/4) is preserved.
Finally, we highlight a sharp contrast with quantized Adam. Theorem 4.6 requires only relative emrrr on the order of = 1/2), whereas Theorem 4.5 demands stricter conditions, at least q=00 (T-T)) and in some cases 9 = O(T-3). This theoretical distinction explains why Muon adapts more efficiently low-precision settings than Adam, corroborating the empirical observations of
Liu et al. (2025).
Experiments.
We evaluate our theory on synthetic, image, and LLM benchmarks.
Synthetic setup. For the synthetic benchmark, we adopt the classical Rosenbrock function (Rosen- brock 1960). Let W E Rmxx and define F(W) = ΣØ= (100)] - W W where W, denotes the j-th column of W and 1m the m-diimensional all-ones vector. We set 50, 100, and run T 10,000 iterations with learning rate 77 5 10-1 Mantissa

---

Published as a conference paper at ICLR 2026
bit-lengths are selected from .M = 4, 32, 52
quantize gradients, weights, and optimizer states. For Adam, we use 31 = 0.9. 82 = 0.999, and = 10 ; for Muon, we set = 0.9 and employ n, 10 power iterations, following Jordan et al. (2024).
CIFAR-10 setup. We train a 4-layer fully connected network (FC(512) ReLU - FC ReLU FC(64) ReLU FC(10)] on CIFAR-10 using Adam and Muon. Additional implemen- tation details are provided in Appendix C
nanoGPT setup. We train nanoGPT on OpenWebText (~ 26M parameters, 4 layers, 4 heads, embedding 384, batch size 128, block size 512). Both AdamW and Muon are tested under varying mantissa lengths M, with Muon applied to 2D parameters in transformer blocks and Adam W applied to all ID parameters (embedding, Im head, layernorm).
Empirical Validation of Theory.
Across all benchmarks, our results empirically validate Theo- rems 4.5 and 4.6 We observe a direct link between quantization error and convergence. As shown across the Rosenbrock, CIFAR-I0, and nanoGPT experiments (Figures 3 low mantissa lengths (M) lead to significant convergence degradation. This degradation correlates directly with high relative quantization errors (detailed in Appendix Figures 6 TO which stall the optimization. Conversely, moderate M values yield sufficiently small errors, enabling con- vergence nearly identical to the full-precision baseline. Furthermore, our experiment in Figure 7 explicitly confirms our analysis of Adam, showing that optimizer sensitivity to quantization error increases significantly as 32 + 1. The language modeling results in Figure suggest that Muon is more robust than W under now-precision training, consistent with Theorems 4.5 and 4.6 We provide full experimental details and results in Appendix
12 13), very
CONCLUSION AND LIMITATIONS
We introduced the first theoretical framework for analyzing adaptive optimizers under realistic floating-point quantization, jointly modeling the quantization of gradients, parameters, and opti- mizer states. Unlike prior work, our analysis does not rely on unbiased quantization or error feed- back assumpsntt that are impractical in modern large-scale low-precision training. Within this framework, we derived the first convergence guarantees for Adam and Muon, with rates expressed explicitly terms of component- wise quantization errors. Our results highlight that Adam is highly sensitive to parameter and second-moment quantization due to its reliance on B2 + whereas Muon requires weaker error control and is therefore more robust. These findings explain empirical observations in large-scale LLM training and narrow the gap between practice and theory.
Limitations and Future Directions. Several challenges remain. First, our analysis focuses on smooth unconstrained non-convex objectives. leaving open extensions to broader settings, including (Lo, Li -smooth functions (Zhang et al., 2020), non-smooth convex objectives (Mishchenko De- et al. 2025 and structured scenarios studied in recent works (Shen et al.) 2025). Second, our theoreti- cal guarantees assume increasing-be regime, M 2(log T'), to control cumulative quantization error. In practice, bit-width is typically fixed (e.g.. FP8 or BF16), which means convergence is guaranteed only neighborhood of stationary point; understanding why moderate fixed preci- sion suffices empirically remains an open question. Third, we focus primarily on fully quantized Adam/Muon and have not yet extended the framework to other popular optimizers benchmarked in LLM training (Vlassis et al., 2025 Semenov et al., 2025 Wen et al. 2025). Finally, our anal- ysis models quantized states under exact arithmetic and does not account for practical considera- tions such as co-pprccision operations (e.g.. FP8 matrix multiplications) or communication conmunication-effcient efficient distributed training, which are critical for large-scale training. Incorporating these aspects would provide a more complete theoretical account of large-scale low-precision optimization.
Tazio 2023 Detazio et al.) 2024). constrained or composite problems (Kovalev)
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and area chairs for their helpful comments. We acknowledge the support from NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A 1515111100122 50444444,, and the central fund from HKU IDS.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We have carefully reviewed the ICLR Code of Ethics and affirm that our work does not raise any significant ethical concerns. Our research is purely theoretical and experimental within the scope of optimization and quantization. It does not involve human subjects, personally identifiable or sensitive data, or applications that may pose harm. All experiments are conducted on synthetically generated datasets and standard benchmark datasets (e.g.. CIFAR-10) and are intended solely to validate the theoretical analysis. We believe our methodology and contributions adhere to principles of fairness, transparency, and research integrity.
REPRODUCIBILITY STATEMENT
We have made significant efforts ensure the reproducibility of our work. All theoretical results are fully detailed, with complete proofs provided in Appendix(A]ane Appendix B The experimental setup, including training protocols, hyperparameters, and evaluation details, is comprehensively documented in Appendix Experiments are conducted on both synthetically generated datasets and the CIFAR-10 benchmark dataset; for synthetic datasets, precise generation procedures included to eliminate ambiguity. Together, these details allow independent researchers to reproduce both the theoretical and experimental results that support the main conclusions of the paper.
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communication-efficii sgd via gradient quantization and encod-
ing.
Advances in neural information processing systems, 30, 2017. URL
https: http::///ccceeeeeg..cc. neur wttpppppppppppppttpptppptppp..ep files/ wttpe////p//p/////////E/. (23302288eeecccc1351455222222250
Matias D Cattaneo, Jason Matthew Klusowski, and Boris Shigida. On the implicit bias of adam. In International Conference on Machine Learning, pp. 5862-5906. PMLR, 2024.
Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized adam with error feedback. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-26, 2021.
Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. Towards practical adam: Non-convexity, convergence theory, and mini-batch acceleration. Journal of Machine Learning Research, 23 (229):1-47) 2022.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of class of Adam-type algorithms for non-convex optimization. arXiv preprint
2018
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. In International Conference Learning Representa- tions, 2019. URL /tttps://cpenreviii
net www.:enetfooriieeee forum? ii-iiidl--xx-- meeeHHxxxx300000 x309tm
Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for RMSProp and Adam in non-convex optimization and an empirical comparison to Nesterov acceleration. arXiv preprint arXiv: 1807 2018.
Aaron Defazio, Xingyu Alice Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. In The Thirty-eigtth Annual Conference Neu- Information Processing Systems, 2024. URL htl.https:// www.oppeevvieeeeee www.wweww..... net forumrrrmmiio
OXeNkkENuI
Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL
net htt.:eeeeeeeeoooo///////r///prp
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization, arXiv preprint arXiv:21 10.02861, 2021.

---

Published as a conference paper at ICLR 2026
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic pptimization. Journal of Machine Learning Research, 12(7), 2011.
Matthew Faw, Isidoros Tziotis, Constantine Caramanis, Aryan Mokhtari, Sanjay Shakkottai, and Rachel Ward. The power adaptivity in sgd: Self-tuning step sizes with unbounded gradients and affine variance. In Po-Ling Loh and Maxim Raginsky (eds.), Proceedings of Thirty Fifth Conference on Learning Theory, Proceedings of Machine Learning Research. PMLR, 2022.
Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling FP8 training to trillion- token LLMs. In The Thirteenth International Conference on Learning Representations, 2025.
URL
net
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus. //Skylion007 wtwp:taw....gggeeeeeeeeeetoooooo
http:
2019.
Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. A novel convergence analysis for algorithms of the Adam family and beyond. arXiv preprint arXiv:2104. 14840, 2021.
G Hinton, N Srivastava, and K Swersky. Lecture 6d-a separate, adaptive learning rate for each connection. Slides of Lecture Neural Networks for Machine Learning. 1:1-31, 2012.
Yusu Hong and Junhong Lin. High probability convergence of Adam under unbounded gradients and affine variance noise. arXiv preprint arXiv:2311.02000 2023.
Yusu Hong and Junhong Lin. On convergence of adam for stochastic optimization under relaxed assumptions. Advances in Neural Information Processing Systems, 37: 17:10077-10877, 2024.
Lu Hou, Ruiliang Zhang, and James Kwok. Analysis of quantized models. In International Conference on Learning Representations, 2019.
IEFE. leee standard for floating-point arithmetic. IEEE Std 754-2019 (Revision of IEEE 754-2008), pp. 1-84, 2019. doi: 10.1 1100///EEESTDD220197766662
Peng Jiang and Gagan Agrawal. A linear speedup analysis of distributed deep learning with sparse and quantized communication. Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL sos 7d1 0aw.5660001111111188664448804130
S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-
neur neur wwtpppppppppppppeeeepppppppp.ppp
fil filess/pperr/2111
0tt.L///22.772200101022222211110 Paper .ddf
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https:
//kellerjordan. hwt.oorraanng.ghu.bbo... github. /.sthubb.io/ppstt/mmmo...
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das. Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jamnalamadaka, Jianyu Huang, Hector Yuen, et al. A study of b 6 for deep learning training. arXiv preprint arXiv: /905. 10.9905...... 12322, 2019.
Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In International conference on machine learning, pp. 3252-3261. PMLR, 2019.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Anastasia Koloskova, Tao Lin, Sehastian U Stich, and Martin Jaggi. Decentralized deep learning with arbitrary communication compression. In International Conference on Learning Represen- tations, 2020. URL /ttsttttps//ooeeeeeeiee ner.forummueeemeeoeeooooo Fid SkkkkkkkgkKK
Dmitry Kovalev. Understanding gradient orthogonalization for deep learning via non-euclidean trust-region optimization. arXiv preprint urXiv:2503.12645 2025.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

Published as a conference paper at ICLR 2026
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. Advances in Neural Information Processing Sys- tems, 35:14651-144622 2022.
Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36, 2024.
Haochuan Li, Alexander Rakhlin, and Ali Jadbabaie. Convergence of adam under relaxed assump- tions. Advances in Neural Information Processing Systems, 36:52166-522 166112996,6,, 2023.
Huan Li, Yiming Dong, and Zhouchen Lin. On the O(vd/T!/4) convergence rate of RMSProp and its momentum extension measured by 4 norm. Journal of Machine Learning Research, 26(131): 1-25, 2025. URL/ttt: org papers wwweeess/v6//// v26 22223325223322
0523 html
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. In Al Stats, 2019.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for Ilm training. arXiv preprint arXiv:2502.16982, 2025.
Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, and Kwang-Ting Cheng. Llm-fp4: 4-bit floating-point quantized transformers. arXiv preprint arXiv:2310.16836, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer-
ence Learning Representations, 2019, URL
Bkg6RiCqY7
Ilia Markov, Adrian Vladu, Qi Guo, and Dan Alistarh. Quantized distributed training of large models with convergence guarantees. In International Conference on Machine Learning, pp. 24020- 24044. PMLR, 2023.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 1710v:1700 2017.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisen- thwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint 2rX1v:2209.05433 2022.
Konstantin Mishchenko and Aaron Defazio. Prodigy: An expeditiously adaptive parameter-free leamner arXiv preprint arXiv:2306.06 202,3
lonut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtić, Thomas Robert, Peter Richtárik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space over- head and provable convergence. Advances Neural Information Processing Systems, 37:1-43, 2024.
NVIDIA. Nvidia h100 tensor core gpu architecture, 2022. URL https://resoure
nvi vvidia..
Kaan Ozkara, Tao Yu, and Youngsuk Park. Stochastic rounding for LLM training: Theory and practice. In The 28th International Conference Artificial Intelligence and Statistics, 2025.
URL /tttps://openrev wwwpenrwwwwwwwweeeeeeeeeee
www.:/nneeffummmomoomm http:/ffrmmiidd3333 33Nt
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang. Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310. 18313, 2023.
13

---

Published as a conference paper at ICLR 2026
Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and Volkan Cevher. Training deep learning models with norm-constraime LMOs. In Forty-second International Conference on Machine Learning, 2025. URL forum? d=20cm2120y2
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In Proc. of the International Conference Learning Representations (ICLR), 2018.
Thomas Robert, Mher Safaryan, Jonut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive opti- mization from low-dimensional gradient statistics. arXiv preprint arXiv:2410.16100. 2025.
HoHo Rosenbrock. An automatic method for finding the greatest or least value of a function. The computer journal, 3(3): 175-1444, 1960.
Naoki Sato, Hiroki Naganuma, and Hideaki liduka. Convergence bound and critical batch size of muon optimizer, 2025.
/wtpvvaaavvvv//aa///a//ea./aaaia wttp://orgggbb//////eeeeee 015988
Andrei Semenov, Matteo Pagliardini, and Martin Jaggi. Benchmarking optimizers for large language model pretraining, arXiv preprint arXiv:2509.01 01440, 2025.
Wei Shen, Ruichuan Huang. Minhui Huang, Cong Shen, and Jiawei Zhang. On the convergence analysis of muon. arXiv preprini arXiv:2505.23743 2025.
Naichen Shi, Dawei Li, Mingyi Hong, and Ruoyu Sun. RMSProp converges with proper hyper- parameter. In International Conference on Learning Representations, 2020.
Hanlin Tang, Xiangru Lian, Shuang Qiu, Lei Yuan, Ce Zhang, Tong Zhang, and Liu. Deepsqueeze: Decentralization meets coope-oomeestted compression. arXiv preprint 1eXXiv1907..0 2019.
Georgios Vlassis, Saleh Ashkboos, Alexandra Volkova, Torsten Hoefler, and Dan Alistarh. Beyond outliers: study of optimizers under quantization. arXiv preprint arXiv:2509.23555 2025.
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma. Tie-Yan Liu, and Wei Chen. Provable adaptivity in Adam. arXiv preprint arXiv:2208.099: 09900, 2022.
Bohan Wang, Jingwen Fu, Huishuai Zhang, Nanning Zheng, and Wei Chen. Closing the gap between the upper bound and lower bound of adam's iteration complexity. Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL http:////ppppeeeee.eeoo net forrmm? Id=yDvb3m1cm oggA
Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Ruoyu Sun, Zhi-Ming Ma, Tie-Yan Liu, Zhi-Quan Luo, and Wei Chen. Provable adaptivity of adam under non-uniform smoothness. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2960-2969, 2024.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Ye Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31, 2018.
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang. Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization. In Forty- second International Conference on Machine Learning, 2025.
Shibo Wang and Pankaj Kanwar.
Bfloat16: The secret to high perfor-
mance cloud tpus.
Google
Cloud Blog, 2019. URL https:
//cloud.goog//oooooo..oooooooool google..collecooooooooo. htt.l////pooouutssaiaeeeeeeeeeee bfloat16- secret to high www.ghhherffrraaccccoo on can-cloud tpus 2025-09-21.
learning/
Accessed:
Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fantastic pretraining optimizers and where to find them. arXiv preprint Stti:250900004:4 2025.
Wei Wen, Cong Xu, Feng Yan, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Temgrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems, 30, 2017.
14

---

Published as a conference paper at ICLR 2026
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-preen preciiiionn training for large-scale vision-language models. Advances in Neural Information Processing Systems, 36:110211110999 2023.
Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu. Jetfire: Effi- cient and accurate transformer pretraining with int8 data flow and per-block quantization. arXiv preprint arXiv:2403. 12422, 2024.
Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, and Song Han. COAT: Compressing optimizer states and activations for memory-efficiett FP8 training. In The Thirteenth International Conference on Learning Representations, 2025. BRLLhttps:// /tttssht:/// /ppprrevee..........
Tao Yu, Gaurav Gupta, Karthick Gopalswamy, Amith R Mamidala, Hao Zhou, Jeffrey Huynh, Youngsuk Park, Ron Diamant, Anoop Deoras, and Luke Huan. Collage: Light-weight low- precision strategy for LLM training. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4lst International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 37459-57479. PMLR, 21-27 Jul 2024. URL https:// proceedings mlm /ooeeeddinng.rrrr.rrss/////////2 wtt.pppees////u2e222222
htmml
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive meth- ods for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of adam on separable data, Advances in Neural Information Processing Systems, 77:23988-24022 2024.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations, 2020. URL
net http://rrweeeeeeeeeooooooololao. Bttp:/urunnnn///gp/////ee/
Qi Zhang. Yi Zhou, and Shaofeng Zou. Convergence guarantees for rmsprop and adam in generalized-smootk non-convex optimization with affine noise variance, 2025. URL https:
Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. Advances in Neural Information Processing Systems, 2022.
Shuai Zheng, Ziyue Huang, and James Kwok. Communication-effiienn distributed blockwise mo- mentum sgd with error-feedback. Advances in Neural Information Processing Systems, 32, 2019.
Jiecheng Zhou, Ding Tang, Rong Fu, Boni Hu, Haoran Xu, Yi Wang, Zhilin Pei, Zhongling Su, Liang Liu, Xingcheng Zhang, and Weiming Zhang. Towards efficient pre-training: Exploring fp4 precision in large language models. arXiv preprint arXiv:2502.1 1458, 2025.
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https::/p//// htttps://openzeview. net forum?id 1JYYN14qqTH
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A sufficient condition for conver- gences Adam and RMSProp. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11127-11135, 2019.
15

---

Published as a conference paper at ICLR 2026
USE OF LARGE LANGUAGE MODELS
Polishing writing. We used multiple large language models (LLMs) to polish the presentation of the paper, focusing on grammar, fluency, and readability while preserving the technical meaning of the content. For each passage, we generated outputs from several LLMs and selected the best version based on clarity and accuracy. The LLMs served only as editorial assistants, and all suggested outputs were carefully checked and revised by the authors. The prompt used for polishing is as follows:
I am preparing a paper for ICLR in Optimization. Please help me polish the following seettence/aaaagraph/secttooo to make more logical, precise, clear, and accurate, while preserving the technical meaning and mathematical correctness. Focus on improving sentence structure, clarity, flow, and readability, and enhance logical coherence between statements. Highlight any ambiquities or imprecise statements and suggest more rigorous alternatives. seenterce/paaaggaahh/eeetoo
Assisting 1TEX code. We also used github copilot/cursor as typing assistant to conveniently type WTEX code for mathematical formulas and derivations. All generated code was manually checked, corrected, and integrated by the authors.
A PROOF OF THEOREM 4.5
A. PRELIMINARIES
We consider optimization problem in d-dimensional space (let d = mn be the number of train- able parameters), where coordinates are indexed by E [d] = {1.2..... d]. Our algorithm generates sequence of vectors (Ur)eN, with the i-th component of denoted by Wr The objective is to find a critical point of global function F : Rd I within stochastic framework, where we have access to sequence of i.i.d. sample functions (fi) (c.g.. the loss on a data minibatch). For any differentiable function h Rd + R. denote its gradient by Vh and its i-th component by V,h. Finally, we use a small constant > O for numerical stability and let E, [-] denote the conditional ex- pectation given the history of samples fi,..., fi-1- We use vec(.) to vectorize matrix and mat(-) for the inverse operation.
Recall the dynamic system of theoretical Quantized Adam. In the proof, denote we vec( (W.), g: = vec( G.) and the dimension = m n. For an iteration E N we define:
me,s =1mm,- 1,1 I1. = 31 E1-1 (Vif(((-11) &g,),
Vt.i
32 1, + Fr
+ + (V.f(w,-1))
5z,))²
(A.1)
miii
W7,s W1-1s T7 Vetul.
with the step size given by
1 1 35
T) 1 1--1-1111 B1)
(A.2)
32
Here, 6r,i, 1--1, and 0,-1,5 represent the quantization errors for the gradient, first mo- ment, and second moment, respectively. Especially, Vrfí(w,-1) 1 - V.J (i
8,, E f((((nim))) 1 D. V.fi(we-1i1aa). And we have E [.f.((::-)) V.F(W,,))
Our convergence analysis for Quantized Adam is predicated on a specific, analytically convenient formulation of the algorithm. This section serves to rigorously justify theoretical framework
16

---

Published as a conference paper at ICLR 2026
by establishing two foundational equivalences. First, demonstrate that our representation of the Adam update is equivalent to the standard formulation. Following the methodology of Défossez et al. (2022). we absorb the scaling factor into the learning rate, which simplifies the recursive structure of the momentum term. Second, and more critically for our work, we prove that the theoretical analysis of quantizing these weighted-sum states is directly applicable to the practical scenario of quantizing the standard weighted-average states.
Equivalence with Standard Adam Our formulation in (A.I) utilizes a weighted sum for the moments, which differs slightly from the standard weighted-average approach in the original Adam algorithm (Kingmal 2014). The standard first moment, often expressed as my,i = (1
(1) 1 "9... is simply a scaled version of our definition, i.e., Mygi = (1 -1- 31,, This constant scaling factor can be directly absorbed into the learning rate.
ik.i.
Furthermore, the standard Adam algorithm includes bias correction terms to counteract the zero- initialization of moments. These corrections equivalent to using time-dependent step size of the form:
1- B1 A]t, = 17 VT-
- B
(A.3)
39
For analytical tractability, our analysis adopts the simplified step size ne from A.2 which omits the bias correction for the first moment (m,,). This simplification is motivated by several practical and theoretical considerations. First, it ensures that our step size n/ is monotonic with respect which is advantageous for the convergence proof. Second, for typical hyperparameter values (e.g., 31 = 0.9, B2 0.999), the omitted term 1/(1 - (1) converges its limit of I much more rapidly than the retained term /1 - 33.. Finally, removing this term effectively implements a learning rate warm-up, a common and beneficial practice, while retaining the correction for Ut,i prevents an undesirably large initial step size that could lead to training instability.
Equivalence of Quantization Schemes. A subtle but crucial aspect of setup is the object of quantization. Our theoretical framework analyzes the quantization of weighted-sum moments (m,, Vz), while practical implementation would quantize the standard weighted-average moments (m; V.). We now prove that these two approaches are, in fact, analytically equivalent.
To establish this rigorously, first abstract the core dynamic behavior into general mathematical lemma. We will show that two discrete-time systems, representing the weighted-sum and weighted- average accumulation methods under relative error perturbations, are analytically indistinguishable. We will then apply this result to the specific case of Quantized Adam.
Lemma A.1 (Equivalence of Perturbed Dynamical Systems). Consider two scalar sequences {ax h2z0 and {c*) evolving according to the following dynamics for k 2 1, with initial con- ditions ag co = 0:
as B(ax + dx-1) + bb
(A.4)
C = 3(ck-1 + cx-1) + (1 B)bb
(A.5)
where 3 E (0, 1) is decay factor, {bx} J is an external input sequence, and {dx), {ex) are perturba- tion sequences. These perturbations are bounded by a relative error model with factor q E [0, 1):
[dg- S q a-111 and
|ek-1| 1| S q/ck-1/-
(A.6)
Then, the sequence {cx} and the scaled sequence {at} 2 (1 - 9)4 are analytically equivalent. Specifically, they follow identical recurrence relations where their respective perturbation terms satisfy identical relative error bounds with respect to their own states.
Proof. To prove the equivalence, we derive the recurrence relation for the scaled sequence (af) } and compare its structure and error properties to that of {cx).
Step 1:
Derive the recurrence for {a Starting from the definition d'. = (1 - B3)az and substi- tuting the dynamics from A.4):
d = (1 - 3) [3(ak-1 + d+-1) + bk]
3(1 3)ax + B(1 - B)dx 1 + (1 B)bb.
17

---

Published as a conference paper at ICLR 2026
Now, replace ax-1 with d/-1/(1 B):
d/ = 3 - 33 '/-1
1- + -11(11)db-11 B)bb
= 3ak=1 + 3(1 - 3)dx-1 1 + 3)bb
Step 2: Compare recurrence structures. Let place the recurrence relations for {ct} and { side-by-side:
Cj 3ck-1 + - 5)-g
d/3 3d B(1 3)dk 1 + (1 3)by.
Both sequences share the identical structure: X = 1- + Perturbation; + (1 - 8)bg- The only difference lies in the form of their respective perturbation terms.
Step 3: Compare relative bounds of the perturbation terms. The equivalence hinges whether these different perturbation terms satisfy the same relative error property with respect to their own system's s state.
For system {c the perturbation term - Using the bound from (A.6):
Perturbation, [Be_ 1| B ex Bglex-1]-
For system {agJ, the effective perturbation term is 3(1 - 100-33)pp 3)dx-1- Using the bound from (.6)) and the scaling relationship ak-1 /
l
|Perturbation, = 18(1 (3(1-3)d4-- 3)dx-11 3(1- S (0100)=-l)((110llll000000 3(1-8)9
3glak
Conclusion of Proof. Both systems, {ca} and the scaled {ak adhere to the same mathematical dynamics. Their evolution is governed by identical recurrence structure, and their respective perturbation terms are bounded by the exact same relative factor 3q with respect to their own pre- vious state. Therefore, from analytical standpoint, the two systems indistinguishable. Any conclusion regarding the long-term behavior (e.g.. convergence, stability) of {cx} under its perturba- tion model will apply directly to {ak) (and thus proportionally to {ax) under its own perturbation model.
With Lemma A. establisshd we can now apply this general result to our specific case of Quantized Adam. The weighted-sum moment m, from our analysis corresponds to the abstract sequence {ax}. while the standard weighted-average moment m, corresponds to {c The gradient term g: corresponds to (ba), and the relative quantization errors in both schemes are modeled by the perturbations {dx} and {e*} with the bound factor q.
Lemma A.I thus formally proves that analyzing the quantization of our weighted-sum moment me analytically equivalent to analyzing the quantization of the standard -eighted-averrge moment me This rigorously justifies our proof strategy and ensures that our theoretical findings are directly relevant to practical implementations of Quantized Adam. A parallel argument holds for the second moments v, and v, with decay factor 82.
A.2 DETAILED PROOF
To systematically analyze the effects of quantization, we begin by isolating the different sources of error. We introduce auxiliary moment estimates, m/,s and v',,, which defined to incorporate the quantization error from the stochastic gradient, òr,i, but are themselves assumed to be stored with perfect precision. Their dynamics are given by:
m/, Bim + (V:f((w:-1) +
(A.7)
o/,5 82 (V.f(W.:-1) 52,))
Throughout the following proof we note E,-1 [+] the conditional expectation with respect to fi...., ft-1- In particular, W;-1, Vt-1 is deterministic knowing ft--...fe-1. With slightly abuse of notation in the detailed proof, We introduce
G. VF(w.-1) and g. Vs((we-)).
(A.8)
18

---

Published as a conference paper at ICLR 2026
We introduce the update u, R well the update without momentum U, R
Ma,i and
Ut It,i + E2,5
(A.9)
Ve+
Vt
vet 1//
For any &o E N with k < t, we define Ue E R by
i 1/ + Ee-
Σ
( + dg,)²
(A.10)
j=t-k+1
i.e. the contribution from the k last gradients are replaced by their expected value for know values of f ft-k-1- Using the smoothness of F, we have
Ti |
F(w:) S F(w--1) - ni u,
The overall motivation of our proof is to find lower bound for n.G u,
gff u = Σ 9r,
mas / Ue
(A.11)
iE|d
We can rewrite G.T u, as:
m
m4,
My,i Gr.i
Gr. Ve
Mi,i
Σ Gr.
Σ iE|d]
Gr
(A.12)
E + Vt.
+
Ut
Vc
VC+
sE[d]
iEId
VE(d]
Here, Term A represents the error component arising from the quantization of the momentum aceu- mulators (m,, Vz), while Term B represents the behavior of the update driven by an ideal accumula- tor.
Now we can bound term A. We split A into two parts as before:
| S
g My,5
m
+ Σ Gr.
m/,
m/.
(A.13)
Ve
ve,2
Vc
+
Ve+
1//1
KE(d)
sE|d
A
Ag
The first term, A, which arises from the quantization noise on the first moment m, is bounded using Lemma A.3 (with q qMr) and Lemma A.5
m mhee
A
R
UE
sc(d]
( (09-*((( *(1+(((1)** -1) da.ol
< R
S gas - dR Cg (A.14)
Et. 82 (1= - - (Wk-1) 62,)²
iE[d]
Vr(++r') where C, (1+gM)(1-)))) and r'
1 1a(1 -v
For the second term, A2, we use the bound on the gradient ||G|| < R and apply Lemma A.2 which requires a case analysis.
|A2|
Σ Rlmi,sl kC(d]
1
Ve vt,
Ve+ v/.s
19

---

Published as a conference paper at ICLR 2026
S Σ Rlm max {
1 Ve+ + L.B.i
1
(A.15)
Ve+uls t/ Ve+uis VC+ UB.
kE(d]
Let Ik. (i + ōg.s. We analyze the two terms inside the max for single coordinate i.
Case (Deviation from Lower Bound): Following the approximation in the provided sketch, we have
U/.1 - LBss
|m/,al
= =Imb,
Ve+ LBr,
Vetvlsl v/ss
VE+ L ct 2/ 1/, E LB + ve+
mm/ai
,/. - LBr,
Vc+ LBei
Ve 1/ (Vee ee+ + LBu, + ve+
Im/
V/yi- L.
Vc + LBu,
c+ C+1 V'e
t k=0 88 -I9,.. + -k.il
Rk= Ek=0 k - -(32(1 - gv) (
[gk,s + 8k,1)
/C Ek= (32(1 -Qv)) qv)))( (a))-(((( (gk,i + )2.
6 2i 32 "an,i + +0x,))²
(A.16)
Eizo 31 -kil k.il
The first fraction is bounded by Lemma A.4
V H8 (f The
Ve+>i ho (1/211 -oy a)
second fraction is a ratio of weighted sums the maximum ratio of its coefficients:
2
(0) (32 qv Jac-kk
This ratio is bounded by
+I 8 a.
3/ (32(1 qv))*
(1- (1 =1 -1- (1--v)1 gv)"
max ke 6)
3/
max kE(O.....
Combining these bounds, the term for Case is bounded by
V1-B5/(B2() (Bv(1- (1 (1 gv))))
Case II (Deviation from Upper Bound): Similarly, we bound the second term:
/mf UB - th,
Vettks 1/,
Ve+UBLs
Vet u/.s + UB
Applying Lemma A.4 and the maximum ratio:
S V1-3//2 (1-
Comparing the bounds from the cases, the bound from Case I is larger since 1 1- (1 -1--qv)²² > 1- (1 + gr) and the denominator term VI- 1i2/1(B2(11 qv)) is smaller than / 33/
Therefore, taking the maximum and summing over d dimensions:
R (1 qv)*
dR(1 - (1-qv)"
A S
(A.17)
B7
KE[M
Fa(1-qv) g
32(1 (2(1-vv)
Combining the bounds for |A1 and AAgl, we can get the bound for term A:
|A| < |Ail + |Aal < gas dR- C. + dR(1 - (1 gv)")
Q(1)
(A.18)
32 (211gv gv
Now we move on to bound Term B. Let us now focus bounding Term B from A. By expanding the definition of the first moment estimate m we can decompose Term B into two parts, which we will call Term C and Term D:
Σ G mhes KE(W
36 It-ki + Se-ki
Eg
G
(A.19)
Ve Ve+ils
V o/
k=0
20

---

Published as a conference paper at ICLR 2026
kt- + &o 1-1 8 (9t, - ge-kke) gt-kii + 8r k.
g k,i
Ve vet + r/s
ve+ 1/4.2
EE 0=
veld 0=
(A.20)
The magnitude of Term D, which captures the error from gradient drift, is bounded by Lemma A.8
D 1 3 4(1 + R
(1
qc) R
81 A: vk
+
1
U
1.
32
k=
(A.21)
For Term C, we establish a lower bound on its expectation in Lemma A.9
1)
1-1
PLas
2(1 + Rc)
31 k
B
ΣΣ
vk+ + 1E
[||
2
VI- - 31
322
vE(d] k=n
kE|d] k=0
(-1
3
Mi-k-
(A.22)
where Mi-
R² Low Rl 1il Ve
Injecting A.22
A.21
and A.20 into A.12
We get the
final lower bound for g.f un:
01,6+1,1 ))
1-1
Σ g
mii
N 2
B
-Q10
Q(t)
Σ
3 M
VE+ U2.s
HEId]
HE(d] 0
k=0
oi L2
1-1
3(1 + Q
3 #
B1
| 112
B/VK k
Σ
vk= - 1|
4(1 + R
VI- 31
32
k=1
(A.23)
Now lets look back at:
n2L
F(w:)= S F(w.-1)= n ur +
|ue 1.
inject
A.23) into it:
R
n E
E [w,) S 1E [F((((
77c
Σ
u [2] +t)(t) + md Σ
3 Mi
VE+ 1-
kE(d K=
4(1 + 9G) R R v1 B1
12
Σ
1
Σ
vkk
+
3m.(1 + 2.)R R
{1 k
Vkc 1
V1-Bi
Br
(A.24)
We have for any h: E N, I < f,, and any coordinate E [d], vc 81,41,5 S (1 ( E 3z.
/
Introducing Ωt- = V D 3y we have
1-1
1/ L +
[[F(w:)] < R (F(,
B+E [1]
E
[| [2 + 71)(())) + MdF
st Me
2(1 + 9G)
R.
2
k==
L2
V1-B B1
Σ
Vk
31 + 9R
31
Vk U1
4(1 + (G)
-1-
32
(A.25)
21

---

Published as a conference paper at ICLR 2026
Now summing over all iterations t [T] for T N", and d7 is non-decreasing, well the fact that F is bounded below by Fx, we get
L T
T
1 2(1 + qG)
B
E
S F(wo) - F. +
E
((t)
+
d
E
pf
M
R
2
k=U
t=1
t=1
t=
k=
Eq
M
17 1?
T t-1
1-1
3 (1 + R R
T 1-1
B1 h
+
VI
81
E
+
Vk +
IE
|
4(1 + 9G)
R
v1 - B1
B2
t=1
t=1
(A.26)
First lets bound Term A. We have = (1 - 24m Thus, we can simplify the A term from (A26 also using the usual change of index
k, to get
T
A=
1
EE
||9
12
2(1 + 9G) R.
1=1
j=1
n/1 31)
T
|1|
2(1 + R
j=1
t=
n
8T T- )B |||
2(1 + R
j=1
(1 ST -j+ }B VF
2(1 R
T-1
1
Σ - 01- -) E [ [F
(A.27)
2(1 + R
1=0
To establish our convergence guarantee, analyze the expected gradient norm at randomly lected iteration т, drawn from the set [0...., T 77-1, 1}. The selection is not uniform but is instead weighted to properly account for the influence of the momentum term over the iterations. The prob- ability of selecting a specific iteration t is defined as:
E (0, .0,,............ T
T-1}, 1}, P(T=1) x
84 1-
(A.28)
We can notice that
T-1 (1- (1-31-1)-73 =T 8 1- ST
11
> T
(A.29)
1 - B1
1=U
Introducing
31
T= 1 Bi
(A.30)
we then have
n
Ā >
E
WW
(A.31)
2(1 R
Next looking at B, we apply Lemma A.13
B B' In
1
c(1
((1 + (q)) R2)
32(1 -111)/
-11n(2211- -vx1))
(A.32)
22

---

Published as a conference paper at ICLR 2026
with
dnf
B'
(A.33)
2(1 31 (1 qm ))
81 (1(1+
12(1 (x()-
Then looking at Õ and introducing the change of index j = t 1
1
T
1-31
31
E
u
4(1 QG R
t=1 t=1
1 1/
T
T Σ
4(1 + QG R v1 - 3
E
Σ
j=1
[= K=1ーj
1/T L2
T
T-1
itk
V1 3
E 11
Σ
Σ
4(1 + qG
R
j=1
t=1
12
v
31
E
||
Σ B+vk(++1)
4(1 + qG R
j=1
1
31
E
(A.34)
(1 + qG)
R
(1-31)²
using Lemma A.12
Finally, using Lemma A.13
we get
((1 + 9G R)2
c² In [+
c(1 - B2 - -vvii - -T1n(8((--v))
(A.35)
with
drp. 13
C'=
(A.36)
(1 + +M))
(1+tar)
32(1-9v)
introducing the same change of index j = t
/ for D, we get
D 3mr 1 + R
ΣΣ
VB+
[IU,
v1 - 31
32
1=1j=1
3mm (1 + R
E 1|1,
VI+1-
VI 81
32
j=1
t=j
61T (1 + R
T
(A.37)
v1 B1
(1 - 31/32)31
j=l
using Lemma A.ll Finally, using Lemma A.10
get
6nr (1 + R
1 1 1/2,1
D VI B1 - 3//22
In
-Tin(32)
e
6dnr + R
((1 + 9c))1)2)
In
1 1
T Tln(322
(A.38)
-1 31
<(1 - (2)
Then rewrite the quantization error term Eq:
au dR C, + dR(( (1 gvv
Eq 1T Σ
V1 32-1
89
23

---

Published as a conference paper at ICLR 2026
dR
dR(1 9v)
Σ QM dR- C₂+ +
Σ
19
B2(l-ov)
12(1 [vV)
T
dR
dR
= T/
T
9M dR- Cq +
(1 Qu)y)
84
B2(1-
72 (v-
tm1
TdR
dRR
1 1- 9V (1 (1 - am
T- 9M TM.. dR- Cg +
8f
84
qv
(o(l-ov)
(A.39)
Finally, we bound Term M using Lemma A.14 dT
1 Lgw RT
M s
(IG
Lqw R2 (ollz)
+
(A.40)
ve(1 B1)
2 2v - 81) V
Now putting A.3II. A.32
(A.35).
A.38
A.39 and
A.40
together into (A.26) and noting that
aT S
we get:
vI V1-32
Fo F.
E.
1+ ((1 + 96) RG2
22 1q
In
-ll((22)
n/T
T
t(1 - 32)
((1 + gc) R)²
- -T'n(32(1 - (v1)) (11()((1-11))
+ Q(T)
+ T In
+
<(1 - B2(1 qvv))
T
+ 2(1 + qa)
(gaR + LQw (Ow.R%Iloold)) + (1 B1 n (1 + 9G) R222
T /e((1= 32)
Tve(l 822)
(1(1+MM)
(A.41)
with
E 12d((1 + R)2
(1- 3//22 1/1/3/2/11111 --8
d1 + 9c) RG) 31)2
2dnj² 1 B1)
H-
+
(1 B1 + 9M ()(1 11(1+
[11 32)
(1-B1(1- 9M (1
81 11(1+028)
32(1- qv ) 32)
82(1- qv)
Q(T)- 2(1 + qa) dd B.TTT
(1+ (1-1(11110) - -3/33/2 + 2(1 + 9g) -3:12
r'r)
VI 1-33 32
V(1
32(1 3a(1-av) )1 -32)
19
2(1 qG) (c) (1 - B1)
1 (1-9V qv (1 (1 q/)²²)
((1
3- wr) (1 - 82))
07
Qv
(1 + 9M)
where r'
B2 qv)
(A.42)
For clarity in the main theorem statement, we can present a slightly looser but more accessible version of this bound. By noting that for a sufficiently large T', we have T > T/2, and
In (1 (1+ ((1 + 9G) R)) - TIn(82) < In
c(1 - 32(1 - qv)) (1n(3((11111)) - (Bx(((11))))
9c)
c(1 - (2))
can state the simplified bound presented in Theorem 4.5
E [[|FF(w m|]]]] < 4(1 + q6) R Fo F.
C
((1 + qG) R)2
(1 qv)))
In
1 +
nT
T
c(1 B2 qv
24

---

Published as a conference paper at ICLR 2026
Q(T)
4(1 + qa)d
2(1 (11-Bidd3 + qa)) RRT
ve(l - 82) (no R2 Law R&D)
T
VC(1 - v
88(1+900 )2
B2(1-9v)
with
24d((1 + Q2) (2)
2dn + gc) 31)²
C
(1 11-31/32)3/2 VI 32
(1 -1- 11 +M)))(1 31(++
82(1-gv) ) 32)
4dn 1 B,)
(1-B1(1 + (nr))(1 - (((1+ (1 32)
32(1-ov)
4(1 + gc)gmddd3 - (1- -31IT
(M) -r/33/2 + 4(1 + ga)dR2 BI)T
(r +r)
Q(T)
VI B2
V F2(1- )(1 - 82)
19
4(1 9G) (1 B1)
1 (1-9V qv (1 (1 - av)?))
(1
32 8v )) 322
87
where r' 89 [1 + qM)²
B2 Qv)
(A.43)
Theory 4.5 states that under specific schedule for the hyperparameters and gradual reduction in quantization error, Quantized Adam achieves the same convergence rate as its full-precision coun- terpart. We prove this by performing detailed asymptotic analysis of each term in the main bound from Theorem 4.5 as the total number of iterations T + 00.
However, to perform precise asymptotic analysis and derive the tightest possible convergence rate from our framework, we will now analyze the order of each component from the more detailed bound A.41
E (VV)) < 2(1 + 4G) R Fo F. 1)T
Term! 1
12d (1 + qG) R) /1 31
((1 (11 9c) Rc)
In
(32)
T(1 /1/32)31 3/2 /1 B3
c(1 132)
((1 + R)²
In
1+ <(1 32(1 -qv))
Tln(3(11 - -qr))
T
Term
Q(T)
2(1 + gc)
T / (22) (goR3 + Lqw R2 Hellvec(Wo)le
T
Term 4
Term 5
(1- (1- d LQw (1 + R272
T TVE(L 82) P2
Term 6
Our proof strategy is to analyze the asymptotic order of each term under the following scaling assumptions.
Scaling Assumptions.
We adopt the scaling assumptions provided in the Theorem:
Quantization Error Schedules: The quantization errors are annealed over time such that 9G = O(T-1), qM O(T-1), gw OTT-2), and qv O(T-2).
25

---

Published as a conference paper at ICLR 2026
Adam Hyperparameters: The learning rate and second- moment decay set 1) ((T-1/2) and - B2 = ((1/T), while 31 is treated as a constant.
Asymptotic Analysis of Bound Terms. We now analyze the order magnitude for each of the six terms.
Term (Initial Condition Term):
R--FF. This term is given by Ti = 2(1 + 96 R
We analyze the
n7
components of its denominator. The effective number of iterations is T T
1- ( The
8
learning rate scales 7 @(T-1/2). The denominator thus scales n/T
O(T-1/2))((T)
(3)) Since all other quantities constants and qG O, the entire term scales as:
Ti = - T1/2 = ((T-1/3)
Term (First Logarithmic Term): This term is:
31
In 1+ ((1 + R)) c(1 - (22) T (I(n(3))
T3 T(1 1 B/ 32
The leading fraction's order is determined by its denominator, TV1- - B2. With 7 = (T) and 1- 82 ((1/T). we have -I- 32 (/T=) Thus, the fraction scales ((7-7-177) ( The term the parenthesis scales as In(1 + O(T) @(1) = @ T). The overall
order is:
T- = ( 0(In))
(T-1/2) . (
O(n T (wT7
Term 3 (Second I Logarithmic Term): This term is Ta = (In(...) - Tl((..)). First, we determine the asymptotic order of E, which is defined as:
dnL(1 ga) RR( 31)
2 31) 8, [1(1+am)] ([1 B2) (1- 31 gMl ))
E (1 (1- (1 (1
3,(1+ya 72(1-10 qv)
B2)
72(1-gv)
For the first part of E, the numerator scales as = (TT3 and the denominator dominated by (1- 32) = (TT This part is 0(T-1/2)/0(T--) = @(T1/2). For the second part, the numerator scales as 1/2 = O(T-1) and the denominator is dominated by (1 - 82) = 3/T33/2). This part is ((T-1)/0(T (In T), so the entire term scales as:
O(T2/2). Thus, E O(T1/2). The logarithmic part scales as
111/2 T
In
T3 = θ OlmT) =
O(n
θ
/T
Term 4 (Moment Quantization Error): We rewrite this term as:
2(1 + QG dRR (1) Q
T4
(A.44)
T TVT 32
where Q = 9M (1+ 3/33
((4+))
+
1
1-gy (1 - (1 - (1-qv)77 )T)
87
T
o7
2
4v)
22
Our goal is to show that T4 = 1/2).
First, the pre-factor has an asymptotic order of:
2(1 + 96 (RR 31)
= 12)
TV1 32
T-1/
T
26

---

Published as a conference paper at ICLR 2026
The core of the analysis thus lies in determining the order of Q. We rewrite Q by combining its second and third components:
(r + r-)
(1 1 qv (1 - (1 (1-(1-1)) 111--11)1)) |
Q = qM (1 + (1 - r/)3/2
T qv
(2(1-
The first part of Q is clearly O(qM) O(T-
The common factor in the second part, A
converges to constant as + 0c, so is O(). The analysis therefore simplifies to finding the order of the bracketed term.
Let x = qv = ((TT)-) 3). We perform a Taylor expansion on (1 - ===":
(1 (1-z)²== - T(T - 1) x² 03)
2
This allows us to analyze the term inside the bracket:
1
x (1 (1 --)))) = 1
1
1
Tx T(T - x² x O(T
T
T
2
1
T T(T - 1) O(T
1
(1 x)
T
1
T
T(T - 1)
Tx + O(T
T
2
T(T + 1)
1 1
O(T
21
T+1
x²)
2
Substituting back x = gv = O the bracketed term has an order of:
OTT qv) - T -(T
Therefore, the entire second component of Q is O(T-1) O(T-1). Combining both components of Q, we find its overall order:
= Q O(T + O(T
Finally, we compute the order of Term by combining the fre-factor and Q:
T4 (( O(T 1/23)
Term (Initial W/G Quantization Error): This term is:
2(1 + qa)
T3 = Tevel1 32)
(gaR² + LQw (e)))l)))
The leading fraction scales as TvT ह jeT 172 = (2T2 The parenthesis scales with its
e
dominant term 9G = -( The total order is:
TB = (T3 Ogga + qw) = - OTT-1/3)
Term 6 (Weight Growth Quantization Error): This term is T (1-31)d2 w (1 +go 12222
1
(2)((1822) 1- (1(1-ov)
First, we analyze
70 As T + xc, the denominator converges to the constant 1 - 87. P20 [
so its contribution is 0(). The term's order is determined by the scaling of its other components:
n 1(2-1/2), qw -2) T ((T), and (1 -32) ((T-1). The total order
To = 0 7. qw: T
= T 1/2 1 1 T2
112-)
T (1- 32)
T.T -1
27

---

Published as a conference paper at ICLR 2026
Conclusion. By comparing the asymptotic orders of all terms, identify those that converge to zero at the slowest rate, as they will dominate the overall convergence bound. The orders are:
Term 1, 1,4,5,6 6: O(7-1/2) or (TT-1/2).
Term 2, ((T In)
The dominant terms the second and third, which of order ((T- T). These terms form the bottleneck that determines the overall convergence rate. Thus. under the specified parameter schedule, the expected squared gradient norm converges to zero the following rate:
In T
[VF θ
VT
This matches the known convergence rate for full-precision Adam.
Furthermore, we derive the convergence rate for the expected gradient norm, E [ from the rate of its squared value. We use Jensen's 's inequality, which states that for convex function and random variable X, ([[[ < E[o(X)).
Let the random variable be X //VF(w. ww )|1 and the convex function be @(x) x² Applying Jensen's 's inequality yields:
E [ E [I
By taking the square root of both sides, we obtain a bound on the expected norm:
VF (w,) (l))) < / (EII/F((w,))
Substituting our previously derived convergence rate:
E B[[VF((>)]ld]] S
VT
= (VT-1/2)
= = (T-1/1)
Thus, the expected gradient converges rate of (TT-//4). This finalizes the proof of the theorem.
A.3 PROOF OF LEMMA A.2
Lemma A.2 (The value range of Ve,s and upper bound of Ve-u, Vetol,
). Let LB
3 at 8 (1--(v (V and UB, = Ek ( (Veff((W-)
We have:
8 k (1 - + 8x.))² u. S
3 -- (1+ ( Sx.1)
k=0
k=0
(A.45)
max
VE+ U2
V o/
V LB
V6 1/, 4/4
VC+ UB.
(A.46)
Proof. The proof consists of two parts.
Part 1: Bounding V4,
28

---

Published as a conference paper at ICLR 2026
The update rule for the second moment estimate is u2,; 1, (V. + The quantization noise assumed to be a relative error, bounded by 10c-1,l I qv 111-1il- This implies that (1 qv ), I 1,-1, 1,- (1 + )
+
Applying this to the update rule, we can establish the lower bound by recursively unrolling the inequality:
Ut 2 82(1 - Qv)0 (V or,,)
2 B2 - Qv) [ )2) + (Vift-1(we-2) + 86-1,) + (Vif((we-1) + )2,)2
8/ (V. 2 Σ
(A.47)
k=
Similarly, we can establish the upper bound:
V1,i S 32(1 + 1v) ( (Viff((w-)))) + 81,4)
8k.1)
(A.48)
k=U
This completes the proof of the first statement in the lemma.
Part 2: Bounding the difference of the inverse square roots
Let oy, be the idealized second moment estimate, updated without the quantization noise 0. Its explicit form is:
t/, =
32 k (Wef(( + 6p.a)
(A.49)
From Part 1, we know that uh is in the interval [LB Br,), where LB and Br,, are the bounds established.
Now, we compare v/, with these bounds. Since 0 < 83 < 1 and we assume 0 < ạv < 1, we have 32(1 - qv) < B2 < 32 qv). This implics term-by-term inequality. leading to:
LB < 1/. U.,
(A.50)
Consider the function f(v) = 1//C- y for y 2 0. This function monotonically decreasing and convex. The value Ut,; lies in the interval [LBt.;, UB,,,. and v/,; is point within this interval. The maximum absolute difference (f(v,,)) f(v(,)) must occur when Vt,; is at one of the endpoints of the interval. Therefore, we can bound the difference as:
{
1
VC+ {
1
S max
VC+ De Ve+ole th
Ve+ uhes
Ve+UBB UB
Ve+ole th
(A.51)
Since LBi, < the < U and the function decreasing, we have 1/ 1/ 1/ c < 1/ 1/ + LB,,. We can therefore remove the absolute value signs:
<
max
VC+ Ut,s
Ve+ LBu.
V V Ve+
Vc+UB.
(A.52)
This completes the proof of the second statement.
29

---

Published as a conference paper at ICLR 2026
A.4 PROOF OF LEMMA A.3
Lemma A.3 (Bound on Discrete Error). Given two discrete-time systems defined for / 2 1:
System A: dr k(a;1 c1-1) + de
.
System B: be = kb,-1 d2
where the perturbation term ot is bounded by |cs| S glael for all t, and the constants q satisfy 0 < k < 1 and q < k.
Under zero initial conditions, where ag bo 0, the absolute error between the states of the two systems is bounded by:
1-1
la0 - bel - + (6)(+-9))---- - |d,1
(A.53)
Proof. First, define the error ea Ur by. Subtracting the two system equations yields the error recurrence relation:
6 ket-1 + ke--
(A.54)
The explicit solution to this recurrence is = e + E)=0 k0==0t=-e Jej. Under the zero initial condition ag = bo = O, this simplifies to:
Joz
(A.55)
1=0
Taking the absolute value and applying the given condition
|cyl s Elazl, we have:
t-1
leel S Σ
1e S
k ka
(A.56)
j=0
5=0
Since ag = 0, the sum starts from j 1. The state layl can be bounded from its own recurrence
lael S k(1 + 4)) + (de), which for ao 0 unrolls to:
layl S Σ (k(11+))) 9))++(dd
(A.57)
Substituting the bound for |را| into the inequality for leel gives a double summation:
leal S
E(<(1 (k(1 + (1)
(A.58)
By swapping the order of summation and evaluating the inner geometric series, obtain the final
result:
|a:- bel < Σ
[(k(1 ((((1+)))+++ g)) x²- |d,l
(A.59)
1=1
L
A.5 PROOF OF LEMMA A.4
Lemma A.4 (Finite Geometric Series Ratio Bounded by Infinite Sum). Let (sw)k-n be a sequence of scalars for any finite f E N. Let the weights be terms of two geometric series, A& = at and B₁ b4, where a, b (0, 1) are the base ratios.
the condition a² < b holds, then the ratio of the weighted sum is bounded by a constant derived from the corresponding infinite series:
Σ1 o a |g9al
<
(A.60)
a²/b
Ei=o bt
30

---

Published as a conference paper at ICLR 2026
Proof. Let the numerator be N; Σ k= a² Igx] and the denominator be Dr -i bb
We rewrite the numerator as:
t
(Vbtl (9ul)
N₂
(A.61)
vib
k=0
Applying the Cauchy-Sehwarz inequality to these finite sums, we get:
Σ (Vbllau)) )
N?
2k
bA gk
D
(A.62)
The first term is a finite geometric series. Since the condition a² < b implies that the ratio r = a²²/b /b positive and less than all terms in the series are positive. Therefore, the finite sum is always less than or equal to the sum of the infinite series:
k
Σ
1
(A.63)
a-2//6
k=
k==
Substituting this upper bound back into the inequality for N2, we have:
N? E 1 - i- /2/bb)
D2
(A.64)
Taking the square root of both sides gives:
N: 1
a2/b Dr
(A.65)
Finally, dividing by Dj yields the desired result for any finite t:
No Ei 1apl
1
(A.66)
D: Ek 64
1 1- d2/b
A.6 PROOF OF LEMMA A.5
Lemma A.5 (Bound on the Quantized Momentum Error Ratio). Let (gh) be a sequence of scalars. Let the weights be As = (41 + 4* 1) and Bß (32(1 - @v)) If the condition (8 + 22r)² < B22(1 q holds, then the ratio of the weighted sum is bounded by:
Eiso A& 19kl S UM (1+ qM (M) (1 r/33/2
(I+++r
V Ei= Bigh
where r' (2((
32(1-qv]
Proof. Following the proof of Lemma A.4,
we apply the Caucch-Schward inequality to get:
A3
Σ Au
Aulgal
B
(A.67)
Bk
K=0
A=U
k==
31

---

Published as a conference paper at ICLR 2026
This implies that the ratio is bounded by the square root of the first term the right-hand side. We now focus on bounding the term
A >x B
First, we express the ratio B2 as:
1 (3f + qm 11)
BB
(32(1 qv))*
B2 (6(1-vv)) )
87
((1 + gM)* 1)2.
(A.68)
To bound the term ((1 + qM) - 1)² we first establish an inequality for (1+ gar)** - 1 using the
Mean Value Theorem. Let (() x.. For qM by the Mean Value Theorem, there exists c E (1, 1, + qar) such that:
f(1 + qM) f(1
-1 = (() = (1+9M)** gas (kch-1).
(A.69)
(1 + qM)
Since e < 1 + 9M, and for k 2 1, we have e-1 11 < (1 + qM) )=1)=111110010 This leads to the inequality:
(M) -1 k qM (1 + 9Ms)
(A.70)
Squaring both sides of (A.70) gives:
((1 + )M)* - -1)2 < 1.2 12 (1 + 9M) ) 2((--))
(1 + qar) (1 + 9M1 921)²²
(A.71)
and using the definition r' (1+
Substituting this back into A.68
P2(1= we get:
A
BJ
<
1MM) )z (1 + qm)
B₁
32
gv)
(1 9M )2
2
(89(1 + 9M1
(1 + qM )2 k2
B2(1 1 qv)
2
9M
(rr))
(A.72)
(1 + )M
Now we sum this term. The condition r' < ensures the convergence of the infinite series. We first derive the closed-form expression for Σ k=
2 for |a| < 1. We start with the geometric series:
(A.73)
0=
Differentiating with respect to x and multiplying by x gives:
Eke
ka*
x dr
- x
=
(1 x)2²
(A.74)
1=
Differentiating one more time and multiplying by x yields:
Σ 12 dx ( (1- x)² x)² =
1(1 - x)² x<2(1 - (-1)
x(1 + x)
(A.75)
(1 x)d
(1- x)3
Using this result with rr, we can bound the sum Σi
A3 By
by extending it to an infinite series:
A2
Bk
0=
k=0
8
2
1:2 (r*g)*
<
+ qxs )²
k=0
32

---

Published as a conference paper at ICLR 2026
4M
2
(r'))
(1 + qM) )2
k=0
4
r(1+r)
(A.76)
(1 + qM)) )2 (1-r"3
Finally, taking the square root of (A.76) and substituting it back into the result from the Cauchy- Schwarz inequality (A.67) gives the desired bound:
Ex=o AR load
2 BR
RM r((+r')
(r((1 +r²)
VE BLok
V
(1+ QM)² (1 (1-r')s
QM
(1 QM qM) 1'33/22
(A.77)
A.7 PROOF OF LEMMA A.6
Lemma A.6 (Bound on the Quantized Gradient Estimator). Let the stochastic gradient be bounded in infinity norm almost surely by (w; S R /ee for any parameters w. Let the gradient quantization operator satisfy the relative error model [Q(z) - 2| S galz| for any scalar 2. The
quantized gradient estimator g: is defined component-wise for i [d] as:
Øt,1 B
1
V
(A.78)
where we use f.)( as shorthand for Q(V)()) and [-]s to denote the i-th component. Then, the infinity norm of the estimator is bounded almost surely:
g &o S (1 + (R vτ).
(A.79)
For notational simplicity in subsequent proofs, we will use the slightly looser bound || S (1 + R. R.
Proof. We first bound the infinity norm of single quantized gradient vector v f(-). For any component i E [d], we have:
v SW Vt/ (
// (
S Vif(w ? 71.)
v.f(we 7(,)
s V.f(w 2 9c.))
+ 96 Vif(
= + (G)
V V.
(A.80)
Since this holds for any component, it also holds for the component with the maximum absolute value. Therefore, by taking the maximum over E [d]. we can bound the infinity norm:
| f(w (i < (1 gc)ll mi 1o,)
< (1 + gG) R vt)-
(A.81)
Finally, we apply the triangle inequality to the full estimator gi, which is the average over B such vectors:
B
g
loc =
A Q f(w
B
j=1
1
<
| f(2 2
B
j=1
1
$
(1 + (1+ ( Vc)
B
= (1 + q Ve).
(A.82) n
This concludes the proof.
33

---

Published as a conference paper at ICLR 2026
A.8 PROOF OF LEMMA A.7
Lemma A.7 (Bound on the Expected Gradient Error with Biased Quantization). Under the assump- tions that the infinity norm of stochastic gradient is up bounded (Assumption 4.22, the objective F is L-smooth (Assumption 4.3) and the quantization relative error model holds (Assumption 3.1 the magnitude of the conditional expectation of the total error term òt,5 bounded by:
E [8t qa R Lqw |[w-11l2-
Proof. We start from the decomposition of the conditional expectation of òz,s, which we derived previously:
E [6,1 = E, f(w ( V. + By Vif(( V.f(wo-1:7)
B,
(v.F(w(L)) v.F(w,-))
(A.83)
Term I: Gratient Quantization Bias
Term II: Weight Quantization Bias
Using the triangle inequality, we can bound the magnitude as:
| [ I [Temm I| Term III.
(A.84)
We bound each term separately.
Bounding Term This term the expected bias from the (potentially biased) gradient quantization. We first apply Jensen's 's inequality for absolute values, i.c., [ [||
Term = E,, -
s By
(A.85)
By the relative error model for gradient quantization (Assumption 3.1 with factor q0):
Term < E, |gc V.f(w
R.
(A.86)
Bounding Term II: This term represents the bias from weight quantization. Using the L-smoothness
of F (Assumption 4.3) and the relative error for weights (Assumption (311
Term III = V.F((2L) - v.F(w:-1) [[VF(w2,)) VF (F(w--1)ll & Lqw l|
(A.87)
Combining the Bounds: Summing the bounds for Term I and Term II, we arrive at the final result:
E [6,,]] S Term I| + Term II| R Lqw l/wcll
(A.88)
A.9 PROOF OF LEMMA A.8 (BOUND TERM D)
Lemma A.8 (Bound on Term D). The term D, which captures the eтror from gradient drift as defined in A.20 is bounded by:
DD s 1.2 B1
(1 (c)
t-1
B1 4
Bi
Σ
Vk:
U
4(1 + 9G) R
vI - 31 Bz
(A.89)
Proof. We start with the definition of Term D:
3% (G., G.- kb-k. + Sc-
D Σ>
Ve+ 1/2
k=U
34

---

Published as a conference paper at ICLR 2026
To tackle this, employ the weighted Young's inequality, which states that for any 0,
A x² xy - 2
2A y
(A.90)
We apply this inequality to each product within the summation for Term D, setting
1 k.s + 61-
VI 3
r = Lr.s Go-k,ll y
and
Ve+ii v/s
2(1 + 46 Rvk+1
This application gives us an initial bound on the magnitude D:
-I
(1 q0) Ro) + (ii + 8,-
|D] S ΣE 36
Rv (9 Ge-k,)²
31 C 1/,65
iE
(A.91)
To simplify this expression further, we must establish bounds for two of its key components.
First, we can find a lower bound for the denominator term. For any coordinate E [d]. the recursive definition of v//s implies that e 1/,55 e + 35 fraction as:
1/ 2 ( u/ This allows us bound the
(ge ge-k.i k,i k,)
1 3/
U2
(A.92)
+ U1
Second, we bound the squared gradient difference using the L-smoothness of the objective function F.
2
|||, - S | W:-1 W. w--k-1 T??
11
k L m
(A.93)
The final step above follows from Jensen's inequality and the fact that the step size schedule is non-decreasing.
With these two intermediate results, A.92 and A.93) we can return to our main inequality (A91 Substituting these bounds yields:
|D|
B
k:
u 11
dG
R. V
+
U x
4(1 96
Rv
3133
L2 B1
Σ
1-1
k
+
(1 g3)
R
[-1
h
Vk +
U.-11
4(1 + R R
v1 3
B2
km
k
Finally, by rearranging the order of summation in the first term, arrive at our desired bound:
1/-3V1-- 31 S S 4(1 + R
1-1
+ (1+ (G+
B1 k
Σ
B
Σ
vk
+
U,
V1 31
B2
k=1
k=0
(A.94)
A.10 PROOF OF LEMMA A.9 (LOWER BOUND ON TERM C)
Lemma A.9 (Lower Bound on the Expectation of Term C). The expectation of term C, defined in lower-bounded by:
A.20),
1)
GL
2(1 + R
31 k V +
E [C
E
ΣΣ iE|d k=
IE
U x
2
+
01,6+1,1
VI 3r
Br
KE(d
K=
35

---

Published as a conference paper at ICLR 2026
3
Mi-k
(A.95)
k=0
where Mi-k = LQw We-k-illz
Vi
Proof. We study the main term of the summation in i.e. for i [d] and k <
Gr-kit 9e-k,s + 8c- Ve+
+ 8 E V V.F(W(---1) V Vife-x x (W:
E
(A.96)
VE Uz,
We will further drop indices in the rest of the proof, noting g = Gi-kii. 9 = -e-k,s 8 = Ō1-k,5.0 t,, and v/,3- Finally, let us note
Σ j=t-k
82 (03s + )g,s)2
and
y² = 1 [52].
(A.97)
In particular we have δ - p² - s². With our notations, rewrite (A.96
9 +
g 9 +
1
g
Ve+u
= E
VE+ +
+ 0)
Ve+u Ve+t
G + 5
8²
= E
B--k-1
+ +9(g+0)
VC+u
Veteyct VC+
G
GE, -
E
+ E
+E
G(g + 8)
VE +
Ve
VE UVE + 80 V +
F
G
ya R R2
Law
2 E
(A.98)
and the bound for /FFl))lo. We denote 20 + Low Bl
The inequality uses Lemma
A.7 and
Ve
Mi-k-
Then we focus on E:
E S 15 8)
+ [ + 8)1
Ve + vie
(e + v) +
due to the fact that vC+u+ Vc+ max max VC+ Vc+ and 1 s21 S r2 +2. 2
Applying Young's inequality to with
A VI - Bive+ō
1g + 8/72
Ve+ + Ve+ + Cove+e
we obtain
g2
1
(g+ B1 (e + 3/332 (e v)
K S
+
-1-
Given that + + ,2 and taking the conditional expectation, we can simplify as
g2
1
2
5)2
E E-
E
(A.99)
4
VT-
3r Vc
+
E v
Now turning to p. we use Young's inequality with
V- Bive +
[
[s(g + 6) +
==
2r2
Ve+
36

---

Published as a conference paper at ICLR 2026
obtain
g
1
x² (
2
+
(A.100)
4vc
VI- 31 ve+ (e+0)²
Given that + v 2 s², and E
pr
= 1, we obtain after taking the conditional expectation.
G
(g (g+8)² 6
E [p]
(A.101)
4VE+7
31 VE +
Notice that in (A. 100), we possibly divide by zero. It suffice to notice that if 7 = 0 then 8² = a.s. so that p and (OT) still verified. Summing (A.99) and (A.1OT).
we get
g²
2
(g + 6)² c c v
k-1 -
E
[|]]] S
+
E
(A.102)
2V++0
-I-
31 Ve
Given that S V + 8 by definition of ö, and that r S Vk + 1(1 + go) R.) reintroducing the indices we had dropped
2(1 + R
vk + LE (gi-k.i + k(
E [|]] S
(A.103)
2/c+ U1,6+1,3
VI-
c t/
Taking the complete expectation and using that by definition e + 2 e+ 1/ k, 2 (/ we get
1 E E F
2(1 R
(gi-k,t + d c + o
V 1E
(A.104)
2
V U2 1,2
v1 3.33
Injecting
A. 104
into A.98) gives
gr-ki 8.
F E
-i-
V Ve,i
EE k=
g
E
E [[]] Mi-k)
(A.105)
KE(d] k=
G ki
GZ-Ri
2(1 + (G) Rvk+
(gi- k,t 5r k,) c + 2/ 2//
E
E
+
+ Me-
V + Ut.
vI
3,,35
0=0
Gias
2(1 + R VI- Bi
B1 *
B
Vk: 1E [|U
d
B Me-k.
2
LVE+ 0
Br
ield
0=
iE k=0
k=0
(A.106)
This is the desired lower bound for [C].
A.11 PROOF OF LEMMA A.10
Lemma A.10 (Lemma A.2 in Défossez et al. (2022)). Assume we have 0 < 31 < 32 S and
sequence of real numbers nnn). We define for all N
b, Σ
and
Cn
Σ
5r 0g.
1=1
Then for any > O, we have the following incquality:
Σ E bj (1 31)(1 31/B2)
1
In (1 bn ) mll(32))
(A.107)
e
37

---

Published as a conference paper at ICLR 2026
Proof. First, we use Jensen's inequality on cj. noting that Ei=18
1-81 I-81
S 1-31
to get:
2
oj =
a1
S
Σ
Σ
01 of
5
3 1 oj
31
Dividing by c+ byby and using the fact that for 1 we obtain:
S j.by 2 3/2
b which implies c+b; 2 3/ - (c+br),
Sj
Σ
5
j-1
a
(A.108)
+ bj
31
bj
31
32
6 +
bi
i=1
Now, we sum over j E [n] and swap the order of summation:
0
B1
5-1
31
32
E E+ 1j
1
31
E bj j=l
B2
1
n of
$ (1 - 31) 31
(A.109)
c c bt
where the last step uses the sum of geometric series, since 31/B3 < 1.
The next step is to bound the final sum. Let's denote IΙ = o7. The sum is Ei ztb. where by Exm B/- k zg. Note that by -4 32b-11 (with by 0). Using the inequality _ S In(y) - (y))
for 0- < < y, we have:
21 < In(e + b1) - In(e + bi - xi)
+ bi
In(e + bi) In(ee + (B2b-11)
c bl 6 + b1-1
b1-1
In
In
6 + Bzby-1
Summing from to n, the first term forms telescoping series equal to In(e + bn) In(e) In(1 + bn.c).. For the second term, since f32 S 1 and b- 2 O, we have etDabi-1 implies In cthb_
cth 2 82, which
(et S In(32). Thus, summing over gives:
n of bt S in (1+
bm) ) ) n In(2))
Σ
(A.110)
This inequality is useful result in itself, corresponding to the special case where of is replaced by a3 (or equivalently 3 + 0 and a, is replaced by dz).
Finally, substituting the bound from A.110 into (A.109) yields the desired result.
A.12 PROOF OF LEMMA A.II
Lemma A.11 (Lemma A.3 in Défossez et al. (2022). ). For any scalar p (0, and any integer K E N, the following bound holds for the finite geometric sum:
K K-1
2
/K +15
(A.111)
01-
Proof. Let the sum be denoted by Sx
Ek=0 /² /k + 1. We analyze the term (1 - pk:
K-1
[-p)Sk=
Σ
Love
1=1
38

---

Published as a conference paper at ICLR 2026
K-1
=I+
Σ
(vk++ vk) VK.
k=1
By the concavity of the square root function,
vk+l - vk S 1
This implies:
2V
K
p²
(1- S
Σ
S +
2vt dl.
2V
The integral is standard Gaussian integral form which evaluates to
/
Using the inequality
2v 2/- Im(p)
-In(p) P. we have:
π
2
p)K S 11
S
2
v1
Dividing by (1 p) yields the desired result.
A.13 PROOF OF LEMMA A.
Lemma A.12 (Lemma A.4 Défossez et al. (0022 ) For any scalar p E (0, and any integer K N, the following bound holds:
k
4p
p
V S (1 (1-p)5/2
(A.112)
k=
Proof. Let the sum be denoted by SK = Ek=o /v(((++)) We proceed by analyzing (1 Pk
K-1
- p)k
Σ
/ AVk -
Σ
(2vk),
k=1
where the inequality holds because vk(k 1) -11 2vk. Re-indexing the sum gives:
K-1
K-2
(1 - P) < 2p
Σ k V/ = 2p Σ 0²
Vivi+ 11
km
1=
Applying the result from Lemma
A. to the final sum, we get:
4p
- p)= s 2p (1 (1- (1 p)3/2
Dividing both sides by (1 p) completes the proof.
A. PROOF OF LEMMA
Lemma A.13 (Upper bound of ΣT E |||u
[l]) Under the condition that 31 + qur) < B2(1
). ), the expected sum squared updates over iterations bounded by:
d
E E
(1 Bi(1 + gl)) (1+tar)) x
32(1-903)
((1 + (11+ R)2
-Tin(3(1--q1) - -qv1)
In 1
(A.113)
c(1 32(1 -qv1)
39

---

Published as a conference paper at ICLR 2026
Proof. The proof proceeds by first expanding the term of interest, applying bounds the moment estimates derived from their recurrence relations, and then leveraging Lemma A.10 to bound the resulting sum.
separating the sum over the dimension d, and taking We begin by expanding the definition of | the expectation:
r Σ
T
m
T
m
R
=
R
IE
(A.114)
+
FEId
sEId
For each coordinate i, we bound the numerator m from above and the denominator + Dr from below. By unrolling the recurrence for my,s and applying the triangle incquality along with the relative error model, we get an upper bound on its magnitude:
ma,al S
0k.s-
(A.115)
k=1
For the denominator, Lemma A.2 provides lower bound for Uz
B/ (1 -
qv) ( - 02,0)2
(A.116)
k=1
Substituting these into the sum gives the inequality:
(Ei 55 (1 + qas) )t 1
27
Σ E
E
(A.117)
c+ Ei= $ st-ha b (1 qv
ic(d
t=1
where for brevity we denote sk. V.Ja(W + 5p.
The inner sum over t for each coordinate in (A.117) perfectly matches the form required by Lemma A.10 To apply it, we make the following substitutions into the lemma's notation:
Let the sequence (aj)jeN- be an 1.v.il-
Let the effective decay factors be 8 (1 + qM) and B% B2 qv). The lemma's condition 8 < 3g is satisfied by our assumption.
With these substitutions, the numerator term becomes (8)) of and the sum in the
denominator becomes (B,)* of yields:
by Applying Lemma A.10 to the sum over t for fixed i
In
I+ br) - (ln(3)))
E
<
(A.118)
by
(1 1133)
1=01
The final step is to find an upper bound for br. By definition:
bT
TB)) a (v)
Ik.i
(A.119)
From LemmalA have uniform bound on the quantized gradient estimator, 10c,l (1+qg)R Therefore:
T
by <
(B2(1 qv)) T-)))=+((()) ()++))((++)t))) R)
k=1
=(11 qa) R)) Σ
(82(1 -v
40

---

Published as a conference paper at ICLR 2026
1
$ ((1 Rc)
(A.120)
1- 1-B2(11 qv)
Substituting the bound for by back into (A.118). and re-inserting the definitions Bi and By we obtain the bound for single coordinate As this bound is identical for all d coordinates, multiply by d to get the final result stated (A.113). This completes the proof.
A.15 PROOF OF LEMMA A. 14 BOUND ON TERM M)
Lemma A.14 (Bound on Term M). The term M, representing the accumulated quantization bias from (A.66). is bounded by:
mdl
URT
M
ve(1- - 31) (galf + Lqw R|] wolle)
(A.121)
2//(1 -31)
where U
# 2a(1-
Proof. First, establish uniform bound on the update norm ||ue||2 using Lemma A.4
m
||ue||2
+ Ut.s
id
k= 3
(1 + qM) V. 0R
EE 32 (1 ) (V 0k,1)2)
d
A U
(A.122)
(1++gr) ) Pa(1-av]
Now, let's recall the definition of M:
where =, R0 + Law R
[2][::-k--o w:
M = yrd
M Mo
Vc
R==0
We can split M into two components: a constant part Mconst and a weight-dependent part Mweights
nrdq
t-1
Meonst
ΣΣ
st
Ve
dLW
T
Mweights
Ve
Bounding Moonst is straightforward. The inner sum is a geometric series bounded by I- so the double summation is bounded by 1-61
RPd
Moonst Ve(1 B1)
(A.123)
The main challenge is to bound Mweights. To do this, we first need bound on the expected weight norm E [|[[w//l]]] From the update rule w, = w - uy. we can unroll the recursion:
Wo Applying the triangle inequality and taking the expectation, we get:
u
ww,llll < ||
112 +
E
Σ m 1ullz
41

---

Published as a conference paper at ICLR 2026
Using the uniform bound ||u/| 12 $ U, we have:
E [y]ll]
S |woll 12
m
woll2 + Ujur.
1=1
Now we substitute this bound back into the expression for Mweights- We first swap the order of summation. Let = t - h - 1. For fixed j E (O...., T - 1), the term E [1 [1w [|wl2l] appears when = - This is valid for from j + 1 to to
ΣΣ stEll k
E w, 1|2
Σ
87-5-1
T-1-1 Σ
Σ /=0
E [1] 51|2]
3T
m=0 0
T
<
E [|| [|]]
1
B1
==
Next, we substitute the linear bound:
1
E
[12]
1
Σ
(llwolla +j. U m))
1 31
1=0
T-1
Tl|wol12 + UIT
Σ 1=
- 31
The sum of the first T - integers is (T-1)T
2 <
T 2.
This gives:
T2
[T/lolll + Un
1- B1
2
Finally, we assemble the complete bound for Mweights:
I "rrLgw R
Ve(l -e(1-3,) (Tllollz T
M Mweights
2 Unr
Combining Moonst with Mweights, we get the final bound for M.
TT
(RG LQw Rolol/)olll). + dR URB R
M 5 Ve(1 - 31)
2/e(1 - -31)
(A.124)
B PROOF OF THEOREM 4.6
B. PRELIMINARIES
The momentum of the quantized Muon (Algorithm 1 3 is defined as
1 n
M, - + +1 3) (8) Σ (9 E..), M, = 2
F Co.)).
(B.1)
B
We define the following auxiliary variables for analysis:
C. Cf + -1 (B-)))F(..).
Co = VF(Wo)
(B.2)
B
= (1- Σ
X, =
1
Σ
Vf(Wo: Eo.i)
(B.3)
B
1=1
42

---

Published as a conference paper at ICLR 2026
B v ((W,,)).))
B
+
Yo-
f:
(B.4)
B 1=1
i=1
1 Z. =3 (1
V f(WP.
Zo
1
V
2 f(W 0 Eo.)).
(B.5)
B
We also define the following relative quantization emrors qG, I,, qM according Assumption 3.1 and Lemma B.2 i.e., for any 1 E {0, 1, 1,..,, , }} and i E (1,2,..., B}.
| J( &.,) Va0--//(W:. V.(W:: dn WW(W,; (L,))l (VF((W,) - VF 9: VF(W.)
[ - V 4G
V f(W? 8 &,,)lee,
[V9F(W9)) - VF(W))) 4G VF(W?)llF
W W. S gw W.llF,
| Mill - qM
M
(B.6)
B.2 PROOF OF THEOREM 4.6
Proof. Set V)- = 1). denote r min (m, n), and according to the T-smoothness of F()), we have
((F(W,) F(W++1))
(W,)) W. W _ |W, Wo-1lli L
-E/ 1) U.V.) 2 [.)
I. "
>EE/(EF( 1) U.VT)
-R V. (VF(W.) M. U.VT) L
-
M VF(() M |U.V [p]
2 n
L ""²r
/I/ M.Il- 7VF - M.
L nr
- 2 - VF
- M.1e]] 2
(B.7)
The second inequality is due to |U. 12 = Ir(V.U, <r= = min(m, The third inequality is due to M, U.S. and Sauchy-Schwart inequality. The last inequality we used the fact that ||| < A|| F for any A gmxn
Summing Eq. B.7 over 0, 1, T 1, we get
E(F(Wo) F()
2 B(IVF((W) M,lee] nLr
(B.8)
Tn
Next, we focus on term El VF(W)) - M, With auxiliary variables defined in Eq. B.2 B.5 we have
E|| B[]VF((W)) |
VF(W.) C.Il++1|e C. X,Ile + xX, Y.Ie+ IY, Zelle + IZ, M,
43

---

Published as a conference paper at ICLR 2026
By Lemmas B33
B.4 B.5
B.6 and B.7 we have
BIIVF((W () M.
<B <BII) C.ll= + |C - X,I + + - Y.Ilo +|Y, Z,IlF + +Z, -
BLaVT
30
B 30
+ +
+ V 1+ + B VB
+ +3 + G) + 33c + qw (1 + DL+
1- B
VB
qw + (w(1+9G) TTvrI + 1 3(1 + 9M)
9M
VB
a
+
V
1 1- 0 + + + 1+ vB
gw(1+ + (1+ (11gg)110 qw) (1+ +
(L
Summing over t = 0,1, T 1, we get
BII -Mllel]
1=0
BLMVF
30
1 B 30
T(1 - B)VB +
+3 + G)
TggTTy qw (G+ DL+
1-3
1+ B VB
g
0
1- B
Qw (1 + (G++GGTTN/L+++ + 1 B(1 + 9M)
+ G G 4a + G)+
VB
V
1+ vB
qw) +1+ Tc)
(B.9)
Substitute
B.9 into B.8). with Assumption 3.1
we have
B[| VI
((F - F(
Lnp
2/F T-1 T
Σ
VFW)) - M
n/
2
t=
B(F(Wo) F(WT)]
Lap
2,Lrr
60
1 - B 60 /r
nT
1-B
T(1 - T(1-3)vB
V 11+ 3 vB
61GVF(e + G) +
+ 2/w (1 + qq) DLVF + 2/w (1 + Ipr
2/MB//
1- B
1 1- 3(1 + qM) /B + V
+G+ +
1+B VB
dw (1 + DL + (1 + (w) + Tc)T
ELF(Wo) - F[WT)]
Lip 28L
6r
8 Vr
nT
1-3 -B
T(1 - B)VB
1+ 3 VB
9M
θ 9G + qw + qnI'n + qwTn +
1+ 1 + 9G + qw + +TT)
1 3(1 gM)
Let F(Wo) Fr A, where A > is constant. By setting B 1, 1 3 1/2-)
" @(1 - ///22 have Tn- (TT) Then we have
E(F(Wo) F[WT)] Lap 23
6T
1 B bo 0 71/4
T/
1 1
B
BVBB
-
1 3 VB
44

---

Published as a conference paper at ICLR 2026
Moreover, with condition ((1+9MM) 1, suppose 1-3 C3T C3 0 is constant. Choose ạM Car T-12, where Cм Cg. Cas > 0s constant, then have
6(1 + 9M) (1 C3T T (Cs - -
1.
Thus, by setting qG = ((T- qw O(T 1/2 qm = O(T we have
9M
QG qw qw + 9GT qw T + - 3(1 + qm)
1 v 81 q0 + gw + Tn)
-0 1/2 + T 1/2 14 - T 1/2 (1 + T- 1/4
+ + 1/2 + +T1/4))
=O(T-1/4),
and B(1 + gM) < 1. Combining the above results, with the fact that ||A]]- 2 || for any matrix A, we complete the proof.
where we used the fact
9M
= O(T-1/2), ((1++()(1)))))
T- 0 qqx))
B.3 PROOF OF LEMMA B.
Lemma B.1 (Bound of | F and [[VF())] for Muon). Suppose Assumptions 4.3 and 4.4 hold. The iterates of Muon satisfy that for any 2 0,
|Wlll s D + tiVT,
Proof of Lemma B.1
According the update of Muon, have
W. W
W - n IE
W.-1lle 7| U.V.
|W-1ll + WW VV
W
+
<D + lnyr.
The third inequality is because U, and V. are orthogonal matrices, and the last inequality is due to Assumption 4.4]
IVF(W.)lF
VF(
(VF VFF(W))
SG +
L W W
SG +
LnVr
=G + tT/LL.
The first inequality is due to the triangle inequality, the second inequality due to Assumption 4.3 and the last inequality is due the update of Muon.
45

---

Published as a conference paper at ICLR 2026
B.4 PROOF OF LEMMA B.2
Lemma B.2. Suppose Assumption 3.1 holds. For any matrix X E Ruxx and its quantized version
X we have
|X x S q
F.
Proof of Lemma B.2 According Assumption 3.1
have
x
F
x
X,yl
?
Taking the square root both sides, complete the proof.
B.5 PROOF OF LEMMA B.3
Lemma B.3. Suppose Assumptions and 4.4 hold. For any / 2 0, we have
BLAVT 1 1-3
E VF C.Ils]
Proof of Lemma B.3
This proof is a standard technique for bounding the bias term of momentum.
We have
((W C.Ile]
F[IVF((W, )) - (3C,11 + - 3VVF(W)))ll)
-F]3] - C--IlF
<
r-)) - (
--)
<F(8)VF(W(-1) C.-IlF++ +B.1 W.-1 W]
-F]3/VF(W C.-ilF++ + BLn) U,- U.-IVT
VF - C.-IllF + [LIVT]]
<B'1)F((Wo)) Colle + Σ LLaV
1=1
BLAVT 1-3
B.6 PROOF OF LEMMA B.4
Lemma B.4, Suppose Assumptions 4.1 and 4.2 hold. For any f 2 0, we have
E|||C X, ge VB ++*11 1
Proof of Lemma B.4 Expanding C, and X, by their definitions in B.2 and B.3 we have
C. + (1- (1
WFF(W))
B
X, - + (1- 3) Σ gt-h
V Vf(We:
B
46

---

Published as a conference paper at ICLR 2026
Thus, have
B||[C X.1lr]
<BIII& [o)]la] + BB(( ()] Σ
(VF(W))
1
V (k..)ll)))
B
k=
B
<3'E1IICo - -xol=]+ +
B[(1 - 8) Σ
3 (VP
v/(Ww::..))11
k=1
B
=B' - Xol1=]+ +
BI(1 - 8)² Σ 32(1-k) k)
VF(W - VV
B
1=1
32(t-k) k)) o2
S/'E1l| Co - Xolle] +
(1- 332
R=1 B
Sigt σ +
3 VB $ 1+3VB
The second inequality is due to Jensen's 's inequality, the first equality is due to the independence of Sk. for different k or and the third inequality is due to Assumptions 4.1 and 4.2
B.7 PROOF OF LEMMA B.S
Lemma B.5, Suppose Assumptions 4.1 and 3.1 hold. For any 2 0, we have
E|| - Y & (a + G tTL)
Proof of Lemma B.5
By the definition of X, and Y, in B.3 and B.4, we have
E E Yells]
B :lI. - ((W::)l.))
< Y--ille + (1 - 3)
B
1
<B 1-1 Y.-1 +
8)
B[gg|| (VW::
B 1=1
<B|3 X, Y + B) [[VF(W)))))
X +1- (o + G LL))
X. Yoll= + (1 ( + (ILF
km0
(1
The second inequality is due Assumption Lemma B.2 B.2 and Definition The third inequality
is due to Assumption 4.2 The fourth incquality is due to Lemma B.I
B.8 PROOF OF LEMMA B.6
Lemma B.6. Suppose Assumptions
4.1
4.2
4.3 and
3.1
hold. For any 2 0, we have
2x
1
B
20
E| - -zlle]
<3
24c(a + G) 24LLn/rLL
VB
1+
3
VB
47

---

Published as a conference paper at ICLR 2026
qw (1 qG) 9G) + qw gc)tn/TLL
1- o
Z Fl 5 VB +
+ G yr + + qw(1 + DL+
V 1 VB
(1 + qw qw)(w))))) 4a VTLFL
Proof of Lemma B.6
By the definition of Y and Z, in B.4 and (.5). have
B
1
+ +1 3) Y,
3
B
B
Z, = (1 3)
g
:
B
t=1
Thus, by the triangle inequality, we have
Y - P.
<B(3' Yo Zoll + B|
gt-k
7 ((((W:: V9 (W.( (x.))ll
B
1
<B|3'1 Yo Zolla]+ (1 31-111111
gt-k
4 V/W
B
k=1
A
B
-1- [|
Vf(Ww:E..)
B
k=
(1 ||||||
g k. (VF(WA) - -V(W)
#=1
H
1 B
(1- B|
ge (VF(W))
B
B
(1 - [|
VS(W
(99(
(B.
B
1=1
Next, we bound each term in (0.10) one by one.
Bound on Y'EillY Zollel By the definitions of Y, and Zu in (B4) and B.5). we have
Yo
V Q @(Wo::
B
1
Zo
f(W Eo.))
Thus, have
B' Zo|r]]
48

---

Published as a conference paper at ICLR 2026
B
6 (Wo
Σ
9 f (W): : [l
B i=1
B i=1
V f(Wo: Co.) V o(Woo
B
Vo.Wo::o.)
VF(Wo)ll]++
- VF(W
8BBl1
VF(W3) VSWW [o,)ll+++
B
g
Eo.)) v : BlIVf( Eo.i)
+
+ 3e
B
VB
vB
B
8
B 1=1
<88
1
Σ
aG(a + +) Br
20
8
DL 3
Σ
qa o qw DL G)
B
VB
B
20
=3g(2aG + GG + qw (1 + 9G)
(B.11)
VB
The first inequality is due to the triangle inequality. The second inequality we used Definition B.6 for the first and last terms, Assumption 4.1 4.2 and Jensen's inequality for the second and fourth
terms, and Assumption 4.3 and Definition B.6 for the third term. The third inequality is due to
Assumption 4.2 4.4 and Definition B.6
Bound on A.
B
A
g
(Wa::E..) Vf(We:G.))lle
B
5(1 - 3) Σ
g gt-h
B
k=1
gL-k 1
<(1 - 3)
B
k=1
B
5(1 3) Σ 8t-h
+ G
1=1
=(1 - (a + (G+D))
(B.12)
The first inequality is due Definition B.6 and the triangle inequality. The second inequality is due to Assumption 4.2 The third inequality is due to Lemma B.I
Bound on C.
Similar to Lemma B.4 we have
B
C -(11 -
Σ k
Wf( k: Ex.))
VF(W
B
49

---

Published as a conference paper at ICLR 2026
B
<(1 8)
[[1
gr-
(WWa; - VF(WA))1
B i=1
k=1
B
(2(t- k) 1
-(1 8)
E
[ Ex.) - VF(W
B
B
1
<(1 8)
(2(t-k) k)
B
821 0²
9)
1 B2
B
8 0
(B.13)
1+ B
VB
Bound on H.
H T - B)
8r (VF(WA)) - +FW
<(1 3)
8 - VF(W
<(1 - 3) Σ
LE W W
<(1 - B)
LqW El| [
<(1 8)
L4W (D+ (n)r)
<(1 8* gqw T(D + tmyr).
(B.14)
The first inequality is due to the triangle inequality. The second inequality is due to Assumption The third inequality is due to Definition B.6 The fourth inequality is due to Lemma B.
Bound on I. Similar to Lemma B.4 we have
1 - B[l ge
1
V (x.
B
B
5(1 B)
[1|
gr-k. (VF(W)) 9
1
B
B
=(1 8)
Σ 82(t- k)
1
B|IVf(W? (
E
VF(W?)
B²
i=1
B
1
S(1 B)
(2(t-R) k)
x2
B2
k=
i=1
B21 02
=(1 8)
32 B
50

---

Published as a conference paper at ICLR 2026
1 - B
(B.15)
1 B B vB
Bound on J.
Σ B 1 (V
-=
5(1 8)
8
B
k=
<(1 8)
B
aala + E| V
B
k=1
8 1
5(1 - 8)
aa + LE w Walls] + VF(a))l)
B
k=1
84-k 1
<(1 - 8)
8
aala + LQw
B
k=1
1=1
1
<(1 3)
8
(a + Lqw (D (n G
B
5(1 8 (a + G qw qw DL (1 qw qw) (W)))))
(B.16)
The first inequality is due to Definition B.6 and the triangle inequality. The second inequality is due Assumption 4.2 The third inequality due Assumption 4.3 and the triangle inequality. The fourth inequality is due to Definition B.6 The fifth inequality is due to Lemma B.T
Bound on E| Ze|la]. B.10), we have
Substituting
B.1
B.12
B.13
B.14
B.15 and (B.6 into
2x
1 3
2x
BiJl,, Z.llr)
+
+
2gol G) 2aattn//LL
VB
vB
qw + (G) 9G) DL + (1 - -3") gg)
20
3
20
5
+
+ 2g0 + G) + 296t1V/LL+
VB
1 3 VB
qw + DL) qw (1 (c)Ln/LL
Bound on [[] [l.
By the definition of Z, in B.5
we have
E Z [z]]
B
<B13 Zollo]
+ [1(1 B)1
9
B
<8' Zollel + [1 3)
Σ
gt-
1
B
k=1
E[(1 B)11 Σ
g1-k
- VF(W
B
k=1
B( - Σ
) +
k=1
51

---

Published as a conference paper at ICLR 2026
B
E ZollF] + (1 -
8
1G
:
B
k=1
t=1
1-8 B
(1 3)qw L
E W
(1- 3) "IIV
V 1++ 3 VB
B
1 Zol Zolla]
+ (1 3)
gi-
qa(a + qw LW BIIVE
B
k=1
i=1
1- B
+ (1 g) L
[| + (1- 3) Σ 8 V>*E||VF(W<>)ll
V 1+ 3 VB
2 Zolle]
+ (1 96 qw L(D + G b+)
1-8 σ
+ (1 8 qw L + (G+ IIFFL
V 1+ VB
58 + qw DL + G) + VB +
(1 39)a(( q L +
1-8
+ + - 8 Mw LW - (L tWVFL)
V 1+B VB
S 0 + V 1+B VB
B
0
+ G +G) DL DL+
VB
(1 (1) + qw)(1 + go)L
1-3 3 of
+ go GG) qw DG) + (1 + qw) (c)tIVVLL VB 1+3 VB
The first and second inequalities are due to the triangle inequality. The third inequality used Definition B.6 for the second term, Jensen's inequality, Assumptions [4.] 4.2 for the third term, Assumption 433 and Definition B.6 for the fourth term. The fourth inequality used triangle
inequality, Assumptions 4.2 4.3 Definition B.6 The fifth inequality due to Lemma B.1
B.9 PROOF OF LEMMA B.7
Lemma B.7. Suppose Assumptions
4.1 4.2 4.3 and 3.1 hold. For any t 2 0. if 3(1 + gas) < 1,
have
E Z M. 1
9M
a
+ V 1+
1 B
GG+ + G) +
3(1 + qar)
VB
VB
qw (1 + + (1+ (1 (1+ (T)
Proof of Lemma
B.7
By the definitions of Z, and M, in B.5 and B. we have
E Mclee
E Zi M
1z + - M
< Z - M,-1ll
qm M
<B|3(1 gMr) [ - - + qm Z, [|
52

---

Published as a conference paper at ICLR 2026
1-1
SQM 3
Σ (1
k=
9M3
1 1-8 3 σ
+ colo + GG+
1 3(1 qM)
VB
V
(1+
3)
vB
qw (G) + (1 + qw +1+ tG)ttttLL)
(B.17)
The second inequality is due to the triangle inequality. The third inequality due to Definition B.6 The fourth inequality is due the triangle inequality. The fifth inequality is due to Zo M2. The last inequality we used Lemma B.6
C ADDITIONAL EXPERIMENTS AND DETAILS
C.I IMITATING QUANTIZATION AND DEQUANTIZATION
We emulate floating-poin quantization and dequantization by reducing the mantissa length from its original precision (52 bits for float and 23 bits for float to M bits, while keeping the exponent and sign bits unchanged. This design choice is motivated by the fact that practical scaling techniques can effectively prevent overflow and underflow (Peng et al. 2023). After truncating the mantissa, we apply stochastic rounding to the nearest two representable values, and then dequantize the result back to standard f float32 or float64.
C.2 S SYNTHETIC EXPERIMENTS
We conduct synthetic experiments on the Rosenbrock function, defined as
F(W)=
Σ jml
(100/ j+1 - W + +11 - W
where W- 1 W 21. Wal mm is the weight matrix. The global minimum is at 1 ml with F(W") = O. We set m 50, d = 100, and initialize Wo N (1mxn: 0I12I). For Muon, we apply the default hyperparameters in the Newton-Schut
W
1 1m
iteration to compute the zeroth power / orthogonalization of G (Jordan et al. 2024), using double precision.
Figure 3 shows the gradient norms of Adam with different quantization errors on the Rosenbrock function. Figure 4 shows the gradient norms of Muon with different quantization errors on the Rosenbrock function.
Figure shows the function values of Adam with different quantization errors on the Rosenbrock function. Figure 6 shows the function values of Muon with different quantization errors the Rosenbrock function. The relative quantization error is defined [X-Q(X)]e measuring the aver- |||||
age quantization error of X, where ((-) is the quantization operator.
Figure shows the effect of quantizing the second moment in Adam different mantissa lengths M, with all other components kept in FP32. As 332 - 1, the optimizer exhibits larger converged gradient norms and becomes more sensitive to quantization errors induced by reduced M. This phenomenon aligns with our theoretical analysis in Theorem which highlights the amplification of guantization errors by the inverse square root of historical gradient variances in Adam when 83 is close to 1.
C.3 CIFAR-10 10 EXPERIMENTS
We conduct real-data experiments on the CIFAR-10 dataset (Krizhevsky et al. 2009) using a 4-layer fully connected network (FCN). The architecture is follows: input layer with 3072 neurons (corresponding to 32 x 32 x 3 images), followed by three hidden layers with 512, 256, and 64 neurons, respectively, and output layer with 10 neurons for classification. ReLU activations
53

---

Published as a conference paper at ICLR 2026
0.025
0.020
144
M8
0,020
Quantizatio W
0.015
Quantizatio Error,
M52
0.015
0.000
0.000
2000
4000
8000
8000
10000
2000
4000
8000
0000
10000
Iteration
Iteration
0.020
0.020
Quantzation Error.
0.015
0.015
on
0.010
0.010
0.000
0.000
lteration
Iteration
Figure 5: Rosenbrock: Adam relative quantization error of different mantissa bits (M)) Weights еπror (top left), Gradient error (top right), First moment error (bottom left), Second moment eπтor (bottom right). These results show that the more mantissa bits, the smaller the relative quantization error. Combining with Figure 3]
we can see that the more mantissa bits, the smaller quantization error, the better convergence performance (Theorem
X4
R2b
Quarratte Eroe
0000
6430
Ineator
Itraston
Figure 6: Rosenbrock: Muon relative quantization error of different mantissa bits (M). Weights eтror (left), Gradient eиror (middle), Momentum error (right). These results show that the more mantissa bits, the smaller the relative quantization error. Combining with Figure 4 we can see that the more mantissa bits, the smaller quantization error, the better convergence performance (Theo- rem 4.6).
used for all hidden layers, and the network is trained with the cross-entropy loss for 100 epochs. We evaluate both Adam and Muon under varying quantization precisions.
For Adam, we use mantissa bit-lengths M E (1,2,3, 10, 23). batch size B 256, learning rate ?) = 1.5 x 10-4 31 0.95, 82 0.999, = 10-8, and weight decay 0. For Muon, vector parameters updated using Adam, while matrix parameters ization step. We choose mantissa bit-lengths M 32,3,7 7, 10, 23). batch size B = 512, learning rate 1 0.001, 3 0.99, weight decay 0.1, and 5 Newton-Schull iterations, following the iteration hyperparameters in Jordan et al. (2024). The auxiliary Adam optimizer in Muon uses learning rate 10 and weight decay 0.05.
updated with Muon's orthogonal-
77 2 x 10 3 0.9. 3g 0.999.
Figure shows the gradient norms of Adam with different quantization errors on CIFAR-10. Fig- ure shows the gradient norms of Muon with different quantization errors CIFAR-10 10. Figure 10
54

---

Published as a conference paper at ICLR 2026
shows the quantization enrors of Adam with different precision CIFAR-10. Figure 11 shows the quantization errors of Muon with different precision on CIFAR-10. The relative quantization error is defined as [|x=Q(X]ll= measuring the average quantization error of X, where Q(-) is the quantization operator. ||
C.4 NANOGPT EXPERIMENTS
We evaluate the impact of quantization on training the nanoGPT model on the Open Wex dataset (Gokaslan et al. 2019). The model has 26.4M parameters, with weight tying between the embedding and the output layer (lm_head). Its architecture includes 4 transformer layers, each with 4 attention heads, embedding dimension 384, without dropout, and no bias terms. The dataset contains ~ 655.4M tokens, and we use block size (context length) of 512. Training is performed with a batch size of 32 and gradient accumulation of 4, resulting in an effective batch size of 128. Models are trained for up to 10,000 iterations.
Optimizer and Training Settings. We experiment with both Adam W and Muon optimizers, as summarized below:
AdamW: learning rate 3 10-4, weight decay 0.1, 81 0.9, 32 0.95, 10- gradient clipping norm 1.0. Learning rate decay is disabled.
Muon: 2D parameters in transformer blocks (~ 7M) are updated with Muon's 's orthogonalization- based step NNwtonnSchhll iteration), while all remaining parameters (~ 19M, including embed- dings, layer norms, and output layer) are updated with AdamW, Muon hyperparameters are: learn- ing 10-2, B 0.95 (with Nesterov momentum), Newton-Schulz steps 5, e 1 10-7 for NS iteration. Auxiliary Adam learning rate 6 10-3, 81 = 0.9, B2 0.95, 10-5, weight decay 0.01, gradient clipping norm 1.0.
Quantization. Following the procedure in Section C we apply mantissa truncation to weights, gradients, and optimizer states. We vary the mantissa length M 2, 10, 23}, keeping exponent and sign bits in full precision.
Results. Figures 12 and 13 show training and validation loss dynamics for nanoGPT under differ- quantization precisions. We observe that:
Lower mantissa lengths (e.g., M 2) induce slightly slower convergence and higher final training loss, consistent with the observed gradient norm amplification in Theorem 4.5 and Theorem 4.6
Muon exhibits greater robustness low-precision quantization compared Adam W, achieving lower training and validation loss at M = 2. This aligns with our theoretical findings that Muon's 's quantization error amplification is less sensitive than Adam's.
As the mantissa length increases, both AdamW and Muon converge almost identical training and validation loss, indicating that higher precision mitigates quantivation-induced degradation.
Overall, these results on nanoGPT extend the findings from synthetic (Rosenbrock) and CIFAR- 10 experiments to real large-scale language modeling setting, highlighting the interplay between quantization precision, optimizer dynamics, and convergence stability.
55

---

Published as a conference paper at ICLR 2026
800
Gradlent Foberius-nom
0.98
0.999
6000
0.9999
9.99999
4000
2000
Iteration
Iteration
(a): Mantissa length M = 1.
Gradient CradennFootehiic Probenius-nomm 1-00---mm
$0.00
6: 999
500
Frobentus
0.9999
0.99999
4000
0: 0 999999
300
2000
200
2000
4000
600C
8000
10000
4900
4010
9000
9000
10000
Iteration
Iteration
(b): Mantissa length M 2.
Grodient Frobenius-noi 1-momm
8000
0.99
150
200
6000
0.9999
250
0. 9.99999
200
9.999999
2000
2000
40CD
6000
5000
10000
5900
9920
0p00
9060
9000
10000
Iteration
Iteration
(c): Mantissa length M 4.
Grodlent Frobenius ribentus-narm -harm
0.99
0.999
8000
9.9999
0.99999
4000
2000
3900
9p8o
(d): Mantissa length M = 7.
Gradient Frbbanlus-narr naanu
0000
8: 0.999
8000
A; 0.9999
4000
A 9.99999
20-
10
Iteration
Iteration
(e): Mantissa length M 10.
Figure 7: Rosenbrock: Effect of quantizing second moment Adam to different mantissa lengths
M. with all other components kept in FP32. As B2 + 1. the optimizer exhibits larger converged gradient norms and becomes more sensitive to quantization errors induced by reduced M.
56

---

Published as a conference paper at ICLR 2026
0.35
M1
Gradient Frobenius-norm
M
0.30
MWW M
M2
M3
0.25
M7
0.20
M10
0.15
M23
0.10
0.0002
0.05
0.0000
90
95
100
0.00
20
40
60
80
100
Epochs
Figure 8: CIFAR-10: Adam gradient under different mantissa precisions M. Larger mantissa bit-lengths lead to smaller converged gradient norms. Together with Figure I0 this demonstrates that higher precision reduces quantization error and improves convergence, consistent with Theo- rem
14
Gradient Frobenius-norm norm
M2
12
M3
MW
10
M7
M10
8
M23
6
4
2
0
0
20
40
60
80
100
Epochs
Figure 9: CIFAR-10: Muon gradient norms under different mantissa precisions M. Larger mantissa bit-lengths lead to smaller converged gradient norms. Together with Figure [I] this demonstrates that higher precision reduces quantization error and improves convergence, consistent with Theo- rem 4.6
57

---

Published as a conference paper at ICLR 2026
Relative
Relative
0.25
M2
M3
0.15
0.10
M23
0.05
0.05
0.00
2500
5000
7500
10000
12500
15000
17500
20030
0
2500
5000
7500
10000
12500
15000
17500
20000
Relative Error m
Relative Error-v
0.175
0.25
0.330
0.100
0.075
0.05
0.025
0.00
0.000
2500
5000
7500
10000
12300
15000
37500
20000
2500
5000
7500
10000
12500
15000
17500
20000
Figure 10: CIFAR-10: Adam relative quantization error of different mantissa bits (M), Weights error (top left), Gradient error (top right), First moment error (bottom left), Second moment error (bottom right). These results show that the more mantissa bits, the smaller the relative quantization eπror Combining with Figure 8 we can see that the more mantissa bits, the smaller quantization error, the better convergence performance (Theorem 4.5)
Error
Relative Error
Error
0.08
0.87
M7
0.06
0.05
M10
0.05
M23
0.94
0.04
0.02
0.07
2000
4000
6000
8000
10000
2000
4000
10030
4000
6000
10000
Relative Error
Relative Error
0.14
0.12
0.12
0:10
0.08
0.06
0.04
0.02
0.00
0.00
2000
4000
6000
8000
10000
2000
4000
5000
8000
10000
Figure 11: CIFAR-10: Muon with auxiliary Adam relative quantization error of different mantissa bits (M). Weights error (top left), Gradient eитоr (top middle), Momentum eтror (top right), Auxil- iary Adam first moment error (bottom left), Auxiliary Adam second moment error (bottom middle). These results show that the more mantissa bits, the smaller the relative quantization error. Combin- ing with Figure 9 we can see that the more mantissa bits, the smaller quantization error, the better convergence performance (Theorem 4.6)
58

---

Published as a conference paper at ICLR 2026
10
M10
M23
M23
6-
2000
4000
6000
8000
10000
2000
4000
6000
8000
30000
Iteration
Figure 12: Training loss of nanoGPT on OpenWebText with varying mantissa lengths M. Lower M slightly increases the training loss due to amplified quantization error.
11
M1AdammN
M2-AdamiW
M1 Muon
Muon
10
8-
Valication
7-
1.2
KID.-AdmiN
M2 192-Mammm W
MIO Musn
M23 Muon
10
Valication
6-
2000
4000
5000
8000
10000
2000
4000
6000
8000
30000
Ieeraion
Iteration
under varying mantissa precisions M. Higher precision reduces quantization and improves validation performance, particularly at low M. Notably, Muon exhibits greater robustness to low-precision quantization compared to AdamW, suggesting potential advantage for low-precision training of large language models.
Figure 13: Validation loss of nan on
59


Reviews:
[{"review_id": "f2GDeJ7v3T", "reviewer": "Reviewer_GWqn", "summary": "This paper presents the first theoretical framework analyzing the convergence of adaptive optimizers like Adam and Muon under floating-point quantization of gradients, weights, and optimizer states. It shows that both can maintain near full-precision convergence rates if mantissa precision scales logarithmically with iterations, with Muon proving more robust to quantization errors than Adam.", "questions": "See weakness", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "complete quantization error analysis under certain settings, both for Adam and Muon", "weaknesses": "Basically experiments are limited, theory is not that informative either in proving practicality. The novelty and contribution is limited.\n\n- line 402 \"the second moment (qV) is stricter than for the first moment (qM)\" -> there are well known fact existing works e.g., https://arxiv.org/abs/2405.03637 and https://arxiv.org/abs/2405.03637\n- many missing connection with stochastic rounding work where it give unbiased estimation, but brining higher variance. In modern low bit training like 4 bits, SR is widely adopted. for 8 bit, MX and NV leads to minimal quantization errors in computing gradient. Moreover  thm3 https://arxiv.org/pdf/2502.20566 has some simpler Adam analysis under quantization error of q_V . better compare.\n- experiments are very limited, not covering practical scenarios like LLM, missing of which does not affect any practicality\n- more through theoretical analysis is needed, otherwise, it is just naive extension. For example, for β_12(1 + qM)2 <β2(1−qV),the effect of qM and qV is real? any toy example to test? \n- for \"ensuring the relative quantization errors satisfy qG,qM = O(1/T)\", need more explanation as, although theretical understandable, qG is not controllable as a function of T in practice, also dependency on W, W_Q. \n- which quantization is most important between weight, gradient, M1, M2? are these supported from theory and aligned with empirical studies?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKlOX2LV4E", "reviewer": "Reviewer_adJ6", "summary": "In summary, the paper fills an important gap: it builds on existing convergence results for adaptive optimizers but in a new setting --- floating-point errors. The authors establish new converge guarantees for Muon and Adam, showing that both methods retain rates close to their full-precision counterparts. The key assumption --- the boundedness of the relative quantization error is quite realistic and achievable in low-precision-aware architectural designs. To the best of my knowledge, no previous paper has offered similar full-precision-vs-quantization convergence rates for adaptive methods.", "questions": "Se the **Weaknesses** part", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Rigorous analysis and theoretical insights that align with recent practice.** Providing clear statements (Th. 4.5 and Th. 4.6), the work explains why Muon tolerates quantization better than Adam --- mostly due to an important assumption of $\\beta_2\\to1$ in the Adam analysis. This theoretical insight matches practitioners’ observations [1], narrowing the theory–empirical gap.\n\n**Empiritical validation confirms theory.** Experiments on the Rosenbrock function and small  fully connected models confirm theory. For instance, Figs. 3–4 show that increasing mantissa bits reduces final gradient norms, consistent with the $\\Omega(\\log T)$-bit requirement \n\n**Framework.** The idea of analyzing jointly quantized gradients, weights, and optimizer states, whereas prior work focused mainly on gradient-only quantization sounds promising for future research.\n\n[1] \"Beyond Outliers: A Study of Optimizers Under Quantization\", 2025", "weaknesses": "**Missing research on convergence of matrix-based optimizers, leaving a room for improvement.**  Unlike Kovalev et al. [2] who handles constrained/composite and star-convex settings, or Shen et al. [3], who exploits Hessian structure in several assumptions, the presented theory is only for unconstrained smooth non-convex functions. Also discussions with the results on constrained / unconstrained LMO optimization [4] --- resulting in the Scion optimizer --- would benefit the theoretical flavor of the work. Please, offer any ideas on how to extend your findings to assumptions in mentioned works.\nAdditionally, can you provide an idea of how to extend your results to another, promising setting regarding --- non-smooth, convex functions? As it is demonstrated to be a setup which explains LLMs fairly well [5,6].\n\n**Mantissa growth requirement.** For Muon, to retain the rate of the full-precision training, you assume the logarithmic grows of mantissa $M = \\Omega(\\log{T})$. However, in practice, the bit-width is typically fixed (e.g., 8-bit). This means convergence is to a neighborhood in fixed precision. The paper does show empirically that moderate precision suffices, but the theory only covers the increasing-bit regime. So there is a mismatch that can be left for future research. If this issue is not solved, I recommend to live it as a limitation. Otherwise, it would be nice to offer some discussion regarding this topic in the paper.\n\n**Missing research on optimizers trained in low precision formats.** Naturally, the state-of-the-art LLM training is held in the mixed-precision format --- when optimizer states, softmax, normalization layers are in float32 and other parameters are in bfloat16. Recent work [1] studies optimizer behavior in quantization-aware training paradigms, running models in precisions of up to 4 bits. A notable takeaway --- Shampoo consistently yields the lowest accuracy drop. I believe this can be helpful because studying convergence (in the low-precision setup) of other matrix-based optimizers that emerge as a steepest descent under the spectral norm can be a direct consequence of your research. Moreover, two concurrent works [7,8] have benchmarked a zoo of optimizers at scale, showing that matrix-whitening methods are highly performant.\nNaturally, extending your theoretical findings to other \"matrix\" optimizers would be very useful for the community. Can you give a couple of comments regarding how it is possible to extend you framework to optimizers validated in the works above?\n\n[1] \"Beyond Outliers: A Study of Optimizers Under Quantization\", 2025\n\n[2] \"Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization\", 2025\n\n[3] \"On the Convergence Analysis of Muon\", 2025\n\n[4] \"Training Deep Learning Models with Norm-Constrained LMOs\", 2025\n\n[5] \"The Road Less Scheduled\", 2024\n\n[6] \"Prodigy: An Expeditiously Adaptive Parameter-Free Learner\", 2023\n\n[7] \"Benchmarking Optimizers for Large Language Model Pretraining\", 2025\n\n[8] \"Fantastic Pretraining Optimizers and Where to Find Them\", 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3tIZxNqCnK", "reviewer": "Reviewer_d8Q8", "summary": "The paper presents a theoretical framework analyzing the convergence of adaptive optimizers (Adam and Muon) under quantization of gradients, weights, and optimizer states, extending prior work that studied only partial components. It quantifies how these errors influence convergence and when performance remains close to full precision. Adam is shown to be more sensitive to quantization, while Muon is more robust. Results are mainly supported by synthetic experiments.", "questions": "- Around assumption 3.1, isn’t the definition of relative error and quantization error (via mantissa length) essentially the definition of machine epsilon for the underlying floating-point format? If not, what distinguishes it?\n\n- [Ozkara et al., 2025] derives a bound involving error term $T\\sqrt{\\log(1/\\beta_2)}$, while this paper presents one with $-T\\log(\\beta_2)$. what causes this discrepancy, and could the authors provide a direct comparison? The difference implies distinct requirements for the quantization-error schedule.\n\n- In real training, the actual gradients come with end-to-end training , a compound effect of quantization across multiple layers, how does that correlate or concludes to the quantization error or machine epsilon $q_W$, they are different. \nIn real mixed-precision training, gradients are perturbed by end-to-end quantization and matmul across multiple layers, leading to compounded effects. How does this aggregate behavior appear in the end to only the relative error (machine epsilon) $q_G$ term from single quantization function?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper extends prior work with a rigorous convergence analysis of adaptive optimizers under quantization of gradients, weights, and momentum terms.\n\n- It proposes a quantization schedule that aligns the behavior of quantized optimizers with their full-precision counterparts, offering insights into the sensitivity of different components to quantization error.\n\n- The inclusion of the Muon optimizer broadens the analysis and enhances the paper’s practical relevance.", "weaknesses": "The paper omits key references and makes inaccurate claims about prior work. For instance, \n\n- [Hou et al. 2019] analyzed not only SGD but also adaptive optimizers such as Adam under weight and gradient quantization (without the first-order momentum term, i.e., $\\beta_1=0$). As shown by [D´efossez et al. 2022], omitting momentum only introduces a multiplicative slowdown term, which should be acknowledged unless the new quantization error model changes this relationship.\n\n- Another closely related but uncited work is [Ozkara et al., 2025], which studies the convergence rate of Adam under weight quantization (again without first-order momentum) and analyzes the effect of stochastic rounding, an increasingly important direction. It remains unclear whether the proposed framework can naturally incorporate stochastic rounding into analysis.\n\nSome theoretical setups are impractical. For example, \n- assuming $\\beta_2 \\to 1$ is unrealistic, in practice a fixed $\\beta_2 < 1$ is used, when the term $T \\log(\\beta_2)$ would diverge. Besides, [Ozkara et al., 2025] already emphasizes the reliance of convergence on $\\beta_2 \\to 1$. \n\n- even accepting the assumption, the proposed quantization error schedule that makes the error term vanish is infeasible in practice, as precision cannot be controlled dynamically without effectively reverting to full-precision training. In reality, quantization error is bounded by the machine epsilon of the floating-point format unless altering the format during training\n\nThe experimental validation is limited and basic. There are no studies examining the influence of (\\beta_2 \\to 1) or verifying how the proposed quantization error schedule aligns with theoretical predictions.\n\nLu Hou et al., Analysis of quantized models. In International Conference on Learning Representations, 2019.\n\nOzkara et al., Stochastic rounding for LLM training: Theory and practice. In The 28th International Conference on Artificial Intelligence and Statistics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2WbpRnNurA", "reviewer": "Reviewer_kJLe", "summary": "This paper presents a theoretical analysis of the convergence properties of Adam-type optimization algorithms. The authors propose a unified framework to analyze convergence rates and provide theoretical guarantees for both convex and non-convex settings. The work includes detailed proofs in the appendix and experiments on synthetic datasets and CIFAR-10 to validate the theoretical claims. The paper also discusses practical implications for hyperparameter selection and algorithm design.", "questions": "Q1: Could you provide more detailed analysis of the practical trade-offs between precision (bit-length) and computational efficiency in Quantized Adam? Specifically, how does the convergence rate $O(T^{-1/4})$ translate to practical training time and memory usage?\n\nQ2: The empirical evaluation appears limited to a single dataset (Rosenbrock). Could you expand the experiments to include more diverse optimization problems and model architectures to better validate the theoretical results?\n\nQ3: How does Quantized Adam compare to other quantization techniques for optimization algorithms (e.g., error-feedback mechanisms, stochastic rounding) in terms of convergence rate and practical performance? A comparative study would strengthen the paper's contribution.\n\nQ4: Could you elaborate on how the theoretical convergence rate $O(T^{-1/4})$ translates to practical training performance (e.g., final model accuracy) for different quantization levels? This would help bridge the gap between theory and practice.\n\nQ6: The paper mentions \"we use a small constant $\\epsilon>0$ for numerical stability\" but doesn't discuss the impact of  $\\epsilon$ on convergence. Could you analyze how the choice of $\\epsilon$ affects the convergence rate and practical performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper provides the first convergence guarantee for Adam under a practical floating-point quantization model, addressing a significant gap in the literature.\n\nS2: The theoretical analysis is rigorous and well-structured, with careful handling of quantization errors and their impact on convergence.\n\nS3: The paper establishes concrete hyperparameter settings that ensure convergence at the same rate as full-precision Adam, making the theoretical results practically applicable.\n\nS4: The empirical results (Figure 4) provide strong validation of the theoretical findings, showing that increased precision (larger mantissa bit-lengths) leads to smaller converged gradient norms.\n\nS5: The paper carefully justifies the theoretical framework by establishing two foundational equivalences, demonstrating that the analysis of quantizing weighted-sum states is directly applicable to practical quantization scenarios.", "weaknesses": "W1: The paper does not sufficiently discuss the practical implications of the theoretical results for real-world applications, particularly regarding the trade-off between precision (bit-length) and computational efficiency.\n\nW2: The convergence analysis assumes certain conditions that might be restrictive in practice, but the paper doesn't fully explore how these conditions affect real-world implementation.\n\nW3: The empirical evaluation appears limited to a single dataset (Rosenbrock) in Figure 4, which may not be sufficient to generalize the findings across different optimization problems and model architectures.\n\nW4: The paper doesn't provide a comprehensive comparison with other quantization techniques for optimization algorithms, making it difficult to assess how Quantized Adam compares to alternative approaches.\n\nW5: The connection between the theoretical convergence rate and practical training performance (e.g., final model accuracy) is not explicitly established, which would strengthen the practical relevance of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d7bXCpuqr6", "reviewer": "Reviewer_qhqY", "summary": "This paper provides the theoretical convergence results for Adam and Muon, under floating-point quantization of gradients, weights, and optimizer states (e.g., the EMA moment estimation in Adam). They show that both Adam and Muon can derive convergence rates of $O(1/T^{1/4})$ on smooth non-convex objectives, even under the quantization.", "questions": "- Literatures such as [1] and [2] usually consider the constant compressed coefficient. Could the convergence results be extended to a broader range of compressed coefficients, such as any constant within $(0,1)$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(a). To my best knowledge, this seems to be the first theoretical convergence result for Adam under the quantization of two EMA moment estimations, and for Muon.  \n\n(b). The results somehow reflect the effects of quantization on the convergence rate. In addition, the results seem to provide the insight that Adam is sensitive to weights and second-moment quantization, while Muon is potentially more robust, as Muon allows for a weaker quantization error control than Adam.", "weaknesses": "My major concerns lie in the theoretical part.\n\n- The term $\\tilde{Q}(T)$ in the convergence bound of Theorem 4.5 is not very clear. Although the authors provide a detailed expression in Eq. A.43, it's still very complicated and lacks of detailed discussion with regard to the dependency on $T$. The dependency on $T$ is crucial since it is the dominating order in the convergence rate. I suggest providing a detailed calculation on the order of $\\tilde{Q}(T)$, particularly when $\\eta,\\beta_2$ and terms like $q_G,q_W$ are set as in Line 372.\n\n- Assumptions 3.1 requires the compressed coefficient to be $2^{-M}$, where $M$ is the mantissa length of the target floating-point format. It seems that $M$ is an important parameter to quantify the accuracy of quantization, which, however, does not appear in the convergence bound. \n\n- The convergence results are heavily relied on sufficiently small $q_G,q_W,q_M,q_V$ (relative quantization errors), which assembles the case of non-quantization. Based on the existing convergence results for non-quantization of Adam and Muon, it seems that the results in this paper are not very novel. In addition, it lacks very clear definitions of these terms in the main body of the paper. \n\n- The convergence results require relative quantization errors to be $O(1/T)$ or $O(1/T^2)$ order. However, under the quantization error made in Assumption 3.1, is it possible to achieve a sufficiently small relative quantization error, such as $O(1/T)$ order?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)