MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
Authors: Yu Zhang
Hui-Ling Zhen
Mingxuan Yuan
Bei Yu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM training
FP8
tensor scaling
Primary area: generative models

Abstract: Training large language models with FP8 formats offers significant efficiency gains. However, the reduced numerical precision of FP8 poses challenges for stable and accurate training. Current frameworks preserve training performance using mixed-granularity quantization, i.e., applying per-group quantization for activations and per-tensor/block quantization for weights. While effective, per-group quantization requires scaling along the inner dimension of matrix multiplication, introducing additional dequantization overhead. Moreover, these frameworks often rely on just-in-time scaling to dynamically adjust scaling factors based on the current data distribution. However, this online quantization is inefficient for FP8 training, as it involves multiple memory reads and writes that negate the performance benefits of FP8. To overcome these limitations, we propose MOSS, a novel FP8 training framework that ensures both efficiency and numerical stability. MOSS introduces two key innovations: (1) a two-level microscaling strategy for quantizing sensitive activations, which balances precision and dequantization cost by combining a high-precision global scale with compact, power-of-two local scales; and (2) automatic scaling for weights in linear layers, which eliminates the need for costly max-reduction operations by predicting and adjusting scaling factors during training. Leveraging these techniques, MOSS enables efficient FP8 training of a 7B parameter model, achieving performance comparable to the BF16 baseline while achieving up to 34\% higher training throughput.

Published as a conference paper at ICLR 2026
MOSS:
Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling
Yu Zhang'. Hui-Ling Zhen? Mingxuan Yuan?, Bei Yu!
1 Chinese University of Hong Kong 2 Huawei Noah's Ark Lab
Abstract
Training large language models with FP8 formats offers significant effi- ciency gains. However, the reduced numerical precision of FP8 poses chal- lenges for stable and accurate training. Current frameworks preserve train- ing performance using mixed-grant graeularanllarity quantization, i.e., applying per- group quantization for activations and per-tensor/ /lock quantization for weights. While effective, per-group quantization requires scaling along the inner dimension matrix multiplication, introducing additional dequan- tization overhead. Moreover, these frameworks often rely on just-in-ting scaling to dynamically adjust scaling factors based the current data dis- tribution. However, this online quantization is inefficient for FPS training, as it involves multiple memory reads and writes that negate the performance benefits of FP8. To overcome these limitations, we propose MOSS, novel FP8 training framework that ensures both efficiency and numerical stabil- ity. MOSS introduces two key innovations: (1) two-level microscaling strategy for quantizing sensitive activations, which balances precision and dequantization cost by combining a high-precision global scale with com- pact, power-of-two local scales; and (2) automatic scaling for weights in lin- car layers, which climinates the need for costly max-reduction operations by prodicting and adjusting scaling factors during training. Leveraging these techniques, MOSS enables efficient FP8 training of 7B parameter model, achieving performance comparable to the BF16 bascline while achieving up 34% higher training throughput.
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, including reasoning, language understanding, and generation (Achiam et al., 2023; Grattafiori et al., 2024; Liu et al., 2024; Adler et al., 2024). However, training these models, which often comprise billions of parameters, incurs substantial computational costs. By leveraging tensor quantization in deep neural networks, low- wwwprrcisson training has emerged as a promising solution to accelerate computation and reduce memory requirements. Among existing methods, BF16 training is currently the most widely adopted low ww-prrccisin ap- proach, supported by major large-scale training frameworks such as DeepSpeod (Rasley et al., 2020) and Mcgatron-LM (Shocybi et al., 2019).
With recent advancements in hardware support for FP8 data formats, FP8-based low- precision training (Micikevicius et al., 2022) has emerged the next-gencration training technique. FP8 computation offers significant advantages over BF16 training, including up to 2x speod-up, 50% reduced memory footprint, and 50% lower communication overhead in thcory. However, the limited range and resolution of FP8 introduce substantial challenges for LLM training stability and convergence. To achieve practical benefits, Transformer Engine (Nvidia, 2025) first applics FP8 for general matrix multiplication (GEMM) while maintaining master weights and gradients in higher precision (FP16/FP32) to preserve train- ing stability. FP8-LM (Peng et al., 2023) extends this by further quantizing gradients and first-order momentum to FP8, achieving additional memory savings. Due FP8's sensitiv- ity to activation outliers, recent studies have proposed fine-grained quantization strategies to enhance training stability. Notably, COAT (Xi et al., 2024) and DeepSeek-V3 (Liu et al., 2024) adopt per-group activation quantization to preserve numerical stability, while employing per-tensor caasoorrblod /block weight quantization minimize memory overhead.

---

Published as a conference paper at ICLR 2026
CO Ruentime (mes
Standard FP32 Format
Twe-lovel MXF Pd Formrtt
bi
bala
23 bita
4 bib
bib
by 10
52
0.72
Sign
Exponent
Martisaa
Signn
Exponent
Nantosa
COAT DeepSeck Ours
Figure 1: Quantized Figure 2: Depiction of MXFP8 data format with level-1 FP32 scale GEMM Runtime Com- factor and level-2 E8M0 scale factor, with group size k ~ 10K and parison on H800.
h = 32 over which these two scaling factors are shared.
However, despite the high fidelity, existing per-group FP8 quantization schemes apply scal- ing factors along the inner dimension (K) of GEMM exccuted on Tensor Cores, which incurs additional dequantization overhead on CUDA Corcs. As illustrated in Figure 1, the per-group quantized COAT (Xi et al., 2024) significantly underperforms compared to the per-tensor approach used in Transformer Engine (Nvidia, 2025), due to significant de- quantization and partial sum overhead within the main computation loop. Additionally, dynamically computing per-tensor scaling factors for weights introduces memory access and max-reduction overhead during training, diminishing the benefits of FP8 training.
In this work, we present MOSS, a framework for efficient and accurate FP8 training that addresses the aforementioned limitations. For sensitive activations, MOSS introduces progressive two-level quantization strategy: global per-tensor scale in FP32 precision, combined with local microscaling using power-of-two factors. This design enhances numeri- cal fidelity while reducing dequantization overhead in the GEMM kernel. For weights well- suited to per-tensor quantization, MOSS exploits the bounded update property of Adam-like optimizers (Kingma& & Ba, 2014), where weight updates are inherently limited by the learn- ing rate. Building on this, MOSS proposes an automatic scaling mechanism that predicts scaling factor evolution based on optimizer hyperparameters thereby eliminating runtime max-reduction and its associated cost. Torether, these innovalions make MOSS a more elficient quantization framework for FP8 training, better suited to Tensor Cores while pre- serving training stability and accuracy.
We evaluate MOSS by training large language models, including OLMo-7B (Groeneveld et al., 2024) and LLaMA-2-7B (Grattafiori et al., 2024), under GPU settings, specifically, using only 8 Hopper GPUs for full-scale pretraining. Notably, although Hopper GPUs do not natively support microscaled data formats, we enable this functionality by implementing custom GEMM kernels using Triton OpenAl (2025). Experimental results demonstrate that MOSS achieves lossless accuracy on both pretraining and fine-tuning tasks, matching the performance of BF16 basclines. In terms of efficiency, MOSS delivers 1.34x end-to-end training spcodup on OLMo-7B compared to BF16 and outperforms the state-of- the-art FP8 training framework, COAT, by 12.3%. significantly reducing the training cost of large-scale models. To further validate the efficiency of MOSS's FP8 GEMM operations, conduct an ablation study comparing its runtime against other FP8 GEMM kernels, including DccpGEMM from DeepSeck- V3 (Liu et al., 2024), which confirms the superior efficiency of MOSS's GEMM kernel design.
2 Preliminaries
2.1 FPS and Microscaling Data Format
In FP8, each tensor X is mapped to its pocci-prniiioon counterpart Qx via Qx = [X/s|, where s is the FP32 scaling factor, enforcing all values to fall within the dynamic range of the FP8 format. This often necessitates the use of the less precise E5M2 format for certain tensors, such as gradients and activations, to accommodate wider ranges Nvidia (2025). The microscaling (MX) data format, initially proposcd in (Darvish Rouhani et al., 2023), has been standardized by the Open Compute Project (Bita et al., 2023) with support from Microsoft, Intel, NVIDIA, and others. The primary difference between standard FP8 and MXFP8 quantization lies in the granularity of scaling. In particular, MXFP8 assigns dis- tinct scaling factor to each block of 32 consecutive values, enabling all values to be quantized using the higher-precision E4M3 format. A second key difference is in the representation of scaling factors: FP8 stores them in full-precision FP32, whereas MXFP8 uses an 8-bit power-of-two format (FSM0), significantly reducing storage and computational overhead.
2

---

Published as a conference paper at ICLR 2026
4x
tero
Far
Qv
Qy
Qx
Q
Com
OUDACONE
(b)
Figure 3: FP8 Quantized GEMM on GPUs, (a) Per-group FP8 GEMM in COAT suffers from significant dequantization overhead in the main loop. (b) In contrast, MOSS achieves faster matrix multiplication by confining the main loop Tensor Core operations, with all dequantization deferred to the epilogue. Leveraging the fast MXFPS GEMM module and a two-level microscaling quantization strategy, MOSS significantly reduces dequantization overhead and improves kernel efficiency.
2.2 Weight Update Rule
Optimizers play crucial role in deep learning by updating model weights during train- ing. The most common gradient-basse optimizer is Adam/AdamW (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)), which uses first-order m and sccond-order momentum to achicve better convergence. Define VWL(W,) as gi, the update rule for I-th layer at time step t using AdamW can be formulated as:
m
m, = 3,m,- + (1 - B1) m, 1
B8
Vi = 32 + (g)) Vo 35
Vi
(1)
m:
W W. -
VV+ AW;
where gi is the gradient, m2 is the first-order momentum, V, is the second-order momentum, 31 and B3 are iyperparameters of AdamW, 1) is the learning rale, weight decay, and e (e.g,, le 8) is used to prevent NaN or Inf. A key advantage of Adan-like optimizers that the magnitude of weight updates, A, 17. m invariant. to diagonal rescaling of the gradients. Specifically, scaling the gradients g: by a factor 8 results in m, scaling by s and v; by s², which cancels out in the update rule: (s m²)/v3² v, m. VV.. This scale-invariance contributes to the robustness of Adam-like optimizers in FP8 training, especially when handling FP8 gradients with high dynamic range.
3 Methods
3.1 Two-Level MicroScaling
Overview. Per-group activation quantization in current FP8 training frameworks (Xi al., 2024) introduces critical drawback: the insertion per-group scaling factors along the inner dimension K of GEMM operations incurs significant dequantization over- head. This overhead is executed on slower CUDA Cores rather than high-throughput Ten- Cores (sce GEMM kernel illustration in Figure 3a). As result, the doquantization steps within the main loop of FP8 GEMM become major efficiency bottleneck. Similar dequantizattinninnduee
CUDA Core overhoad has also been observed in inference systems such as QServe (Lin et al., 2024). Specifically, on modern data center GPUs such as the NVIDIA H100, the peak throughput of FP32 CUDA Cores is only 1.6% of that of FP8 Tensor Cores. As result, dequantizing even single partial sum a per-group quan- tized FP8 GEMM can cost the equivalent of nearly 60 Tensor Core multiply-acuumulate (MAC) operations. This imbalance causes the GEMM main loop to be dominated by slow dequantization on CUDA Cores, effectively nullifying the performance advantages of FP8 training-
To strike a balance between quantization accuracy and GEMM efficiency, we propose a two- level microscaling approach that effectively approximates full-precision values with minimal overhead. As illustrated in Figure 2, MOSS hierarchically partitions global block of size
3

---

Published as a conference paper at ICLR 2026
4y into multiple equal-sized subpartitions of size l2. A global scaling factor is applied uniformly across all partitions, while local sub-scale factors ss; are quantized for efficient storage and computation and applied to each corresponding i-th subpartition. In the first stage, we compute floating-poing scale factors at fine gramllarity For the i-th partition with size 22 8; is computed as
(2) where 8j is the scaling factor of subpartition i, X, is the i-th group vector and A FP8 is max the maximum representable value in the chosen FP8 format. For common FP8 encod- E5M2 ings (Micikevicius et al., 2023); A E4M3 = 448 and A max = 57344. In the second stage, we max
= max (aa))) max:
FPS where i E /0 -1]
further quantize the per-group scale factors s; by separating them into ESMO microscaling components ss; and Moating-point global component s,
s = ==ax(((s))).
ss, [ ESMO = 2 [log((
(3)
where [.] Elss denotes rounding to the target data format E8M0, and 2 (=///) denotes the smallest power of two that is greater than or equal to s:/s. Dividing by helps shift the values toward the fractional range, making them more favorable for E&M0 quantization rounding and thereby improving quantization accuracy. In this way, the level-1 global scal- ing factor is encoded in full-precision FP32 for high accuracy, while the level-2 microscaling factors 88; are stored using an efficient 8-bit exponent-only format (E8MO), making them inexpensive to store and compute with. The dequantization operation that maps a quan- tized tensor back to FP32 can be expressed as: DQx, = Qx, 8 88;- This two-level microscaling approach shifts the costlier FP32 scaling to a coarser granularity while in- troducing lightweight FSMO scalers at finer granularity, effectively balancing numerical accuracy with hardware efficiency.
Following the MX format specification (Bita et al., 2023), we set the level-2 group size to k2 = 32. To support latencyy access within the inner loop of Tensor Core MMAs, the level-2 scaling factors ss, are stored in a contiguous memory layout aligned with the ex- pected access patterns. This organization ensures efficient memory utilization and prevents stalls during the GEMM main loop. For model weights, we apply per-tensor quantiza- lion with FP32 scaling factors to fully exploit Tensor Core performance. The quantized matrix multiplication the FP8-quant ized activation, scaled by E8M0-encoded level-2 factors 882; and Q is the FP8-quantized weight tensor.
Figure 3b is performed as: Q, Que x (Qz 882), where Qz is
We illustrate our quantized GEMM kernel design in Figure 3b. To enable MXFP8 GEMM on Tensor Cores, we assign an artificial level-2 scaling factor with value 1 (encoded in E8M0) for Qw, ensuring that GEMM operations between quantized activations and weights can be executed efficiently at fine gramularity, balancing throughput and numerical precision. The resulting FP32 partial sum Qg is then dequantized in CUDA Cores using the FP32 per-tensor scale factor sw associated with the weight tensor and the level-1 FP32 global scale factor 8₂ of activations (sce CUDA Core in Figure 3b). Compared to traditional per- group quantized GEMMs, this two-level quantization schome maintains high fidelity while significantly reducing dequantization overhead in the GEMM main loop.
Quantization Precision Analysis. In the numerical precision analysis for quantization meth- ods, we use Signal to Noise Ratio (SNR), which mcasures the ratio of the original signal's power to the noise power introduced by quantization. The quantization SNR is calculated as,
P Paignal Puoise
E [| DQ x - X
SNR:
10 log(
(4)
where || denotes the root mean square (RMS) norm, defined as |X| - VE ag. A higher SNR indicates better quantization fidelity, as reflects lower noise relative to the original signal. In the context of language model quantization, various schemes are used to balance precision and efficiency. Beyond our proposed two-level microscaling in MOSS, common approaches include per-tensor quantization and per-group quantization. In per- tensor quantization, a single scaling factor is shared across the entire tensor, resulting in minimal overhead but limited adaptability to local value variations. Tn contrast, per-group quantization partitions the tensor into sub-groups along channels, each with its own scaling factor, improving accuracy at the cost of increased metadata and computational overhead.
4

---

Published as a conference paper at ICLR 2026
We then present theoretical analysis of the quantization precision for per-tensor and per- group approaches, as well as our proposed MOSS framework.
Theorem 1. The quantization SNR of per-tensor, per-group quantization and two-level mi- croscaling in MOSS follows: SNR perttttono < perpprrgroup < SNRMOSS.
Proof 1. In the statistical analysis, assume without loss of generality that the elements of X are zero-mean. Under this assumption, the signal power is given by Paigual = dx where dx denotes the standard deviation of X. First, given that moating-poine quantization belongs to uniform quantization, the per-tensor quantization error is uniformly distributed in the range -// s/2], where s is the scale factor. Due to the round-to-nearest behavior, the error has zero mcan and RMS s² /12, which represents the noise power. Thus, the SNR for per-tensor quantization given by:
120 X = 10 loog( max X )
120 (A FP8 max
(5) Next, consider per-group quantization with N groups, each with group size of 128. Each group g is quantized independently using its own scaling factor sg. Since the quantization error within each group is assumed to be uniformly distributed, averaging the noise across
SNRtensor = 10
22
all groups yields Pnoise = 12N Eg 22 Hence, the SNR for per-group quantization is given by:
12No ) 10 10 12No ( (PP))) Eg max )22)
SNRgroup = 10 log(
(6)
Eg %
This formulation shows that finer granularity (larger N) can yield a higher SNR by better adapting to local signal variations. Through comparing Equation (5) and Equation (6), we observe that SpPperggroup -ereettttn since E, max([X, ( /) max( indicating that per-group quantization offers higher quantization fidelity than per-tensor quantization due to its finer scaling adaptation. Now, consider the two-level microscaling quantization scheme used in MOSS. In this design, the effective scaling factor for a subgroup is given by SMX = Sglobal 38,, where Sglobal is the global per-tensor scale, and ss; is the second-level microscale for each group, defined in Equation (3). Assuming the input tensor is divided into Ng micro groups, each of size 32, the resulting quantization noise power becomes P Pooise I2N, Eg 2 "loball ss.. Thus, the corresponding SNR for MOSS's two-level microscaling is:
12N O global 5s?
~ 10 12N, 3 (A 2 max
SNRMOSS = 10
Σ.
52
Σ, max [2
where Ng- 4N with micro group size 32. Since each subgroup in MOSS contains fewer values, its maximum magnitude is upper bounded by that of the per-group quantiza- tion, i.c., max([Xa) I mmxx([X,l) Conscquently, by averaging over all subgroups, we have Σ/ max([x,))))
EN max((x, (x,)² )² which implics SNRMOSS > SNRper-group There- N
N₂
fore, combining with the SNR formulations for each scheme, we arrive at: SNRper-tensor < SNRper-group < SNRMoSS- This inequality shows that the two-level microscaling in MOSS achieves lower quantization error than both per-tensor and per-group quantization. This formulation demonstrates that MOSS retains high quantization fidelity by combining coarse- grained global scaling with lightweight power-of-twe microscales, effectively balancing accu- racy and hardware efficiency.
3.2 Automatic Scaling
Various scaling techniques have been proposed to adapt tensors to the limited dynamic range of the FP8 format (Nvidia, 2025; Blake et al., 2023). However, these methods often introduce training overhead due to additional memory accesses and max-reduction operations. For example, just-in-time scaling requires reading all FP32 values from High Bandwidth Memory (HBM) to compute the maximum absolute value for quantization. The resulting FP8 values are then written back to HBM, only to be read again during subsequent MMAs, incurring significant latency and bandwidth costs.
5

---

Published as a conference paper at ICLR 2026
1e-4
2
Autoratic scale
Table Time usage to calculate per-tensor scal- ing factors for parameters. Automatic scaling in MOSS incurs negligible and constant runtime overhead compared to just-in-tine scaling.
Just time scale
Scale
Tonsor Size
Just-in-time Scaling Automatic Sealing
0
1000
2000 3000
11008 x 16354
0.54ms
0.02tms
Traning Steps
11008 x 8192
0.32ms
0.02ms
Figure 4: Automatic scaling trend un-
4036 12288
0.17ms
0.02tms
1086 x 1096
0.08ms
0.02mns
der interval = 500.
To address this inefficiency, we propose a novel automatic scaling approach that estimates the evolution of scaling factors for parameters throughout training. LLMs are typically trained using Adam its variants (Kingma & Ba, 2014; Loshchilov & Hutter, 2017), which maintain additional copies of model weights, gradients, and first- and second-order momen- tums for parameter updates. A key advantage of these optimizers is their ability to produce stable updates, with changes to model parameters bounded by the step size 17, regardless of gradient volatility. By exploiting this bounded-update matic scaling scheme that can predict and adjust scaling factors ahead of time, eliminating the need for repeated memory reads and writes, and thus avoiding runtime overhead.
property, we propose novel auto-
Thcorom 2. During training, the effective update at time stamp has two upper bounds:
1- 17 <
V if 1 v1 3
A
(8)
17.
otherwise,
where of and 82 denote 31 and B2 raised to the power of t, respectively.
Proof 2. Based on the weight update rule in Equation (1), the effective parameter update
at timestep is given by A. = 11 m./VVr, assuming 0. This expression reveals that A is governed by three key components:
1 1 3 VI 32
A = Vv
To analyze the bound of Atl, we first examine the term |m:/Vvel. During LLM training, both the first-order moment m, and the second-order moment mg and Vo = As training progresses, m, becomes the exponential moving average (EMA) of the gradients, i.e., m, = Elg]EMA. and v, becomes the EMA of the squared gradients, ie., V, E[g"]] JMA. Applying Jensen's inequality (Abramovich al., 2004) to the convex function f(x) x² yields the relation (m.)² S Vi, which in turn implies that |m:/ vv 1.
initialized to zero, i.e.,
Next, consider the term (1 3//11-11// in Equation (8). The first case of Equation (8) corresponds to an extreme scenario of gradient sparsity: when the gradient is zero at all previous timesteps and nonzero only at the current step, which is extremely unusual in practice. Under this worst-case condition, the effective update is bounded by |Ael S π (1-31)/ 34. less sparse scenarios where gradients appear across multiple steps, the EMA terms accumulate over time, leading to smaller normalized updates. As a result, the effective update |Ae] generally bounded by r7.
In more typical cases, the effective update |Ae is strictly smaller than the step size n). For example, when - 3 = /1- /1-B3 13 and given that / I 1, it follows directly that Adl < 17. In practice, during LLM training, it is common to have 1 3 < 11- 3 because LLMs typically adopt smaller values for 32 (e.g., 82 = 0.95) (Grattafiori et al., 2024; Groeneveld et al., 2024). This choice keeps the second-order moment estimate more responsive to recent gradients, helping to prevent both exploding updates and excessively slow learning. Overall, the effective parameter update at each timestep is approximately bounded by the step size 7, i.e., |A, S n- This behavior can be interpreted as implicitly defining trust region around the current parameter values, within which the optimizer assumes the gradient estimate remains informative and reliable.

---

Published as a conference paper at ICLR 2026
10
BF16
Table 2: OLMo-7B pretraining performance on down-
Training Loss
8
COAT
stream tasks. For system performance, we report re- sults of training throughput. For model performance,
MOSS
6
we present the Perplexity on WikiTexxt-033, C4, and Pile.
Throughput
Model Performance (PPL)
20
1000
2000
3000
4000
Models
(tokens/s)
Wiki
C4
Pile
Training Step
BFI6
33,805
39.59
30.59
25,18
Figure curve.
OLMo-7B pretraining
COAT
40.416(
40.62
30.89
26.05
MOSS
45,374
40.96
30.63
25.08
Since the step size is typically known in advance, we can infer the potential evolu- tion of scaling factors throughout training. At initialization, the scaling factor is deter- mined by the maximum absolute value of the model parameters. Based on the update bound in Equation (8), the maximum absolute value of the weights at timestep t satisfies: max( S. (aax(/Wo] +n-t. This relation provides a predictable upper bound on param- eter magnitude over time. Consequently, the scaling factor for model weights at timestep t can be determined as
max max((Wol (.)) +
n.t 80 + APPS
(10)
APP8
where 8, denotes the scaling factor for the weight block at limestep and so is the initial scaling factor, determined via & max-reduction operation at initialization. This formulation eliminates the need for dynamic max-reduction at every step, while still ensuring that the scaled weights remain within the representable range of FP8. As shown in Table 1, our proposed automatic scaling introduces negligible overhead, operating in constant time across all tensor sizes. In contrast, just-in-time scaling incurs significant computational costs due to its max reduction process, especially for large tensors.
To maintain training stability and numerical accuracy, MOSS performs dynamic re-scaling at fixed intervals. Specifically, within each interval, scaling factors updated using the rule defined in Equation (10), incurring negligible memory access overhead. At the end of each interval, MOSS rescales the current weight tensors and updates the associated per-tensor scaling factors accordingly. The effectiveness of this approach is illustrated in Figure 4, where interval length of 500 is used. We observe that the scaling trajectory produced by automatic scaling consistently lies above that of just-in-time scaling, helping to ensure that values remain well within the representable FP8 range and reducing the risk of overflows or underflows. Notably, the two scale curves remain relatively close throughout training, indicating that MOSS maintains a high degree of precision even with infrequent updates. The choice of interval length presents a trade-off between scale estimation precision and computational efficiency. While the above theoretical analysis assumes the use of the Adam optimizer, the same principles apply to AdamW. Additional discussion can be found in the Appendix. This periodic scaling strategy enables MOSS to perform automatic parameter quantization with minimal overhead, offering a practical alternative to more expensive just- in-time or delayed scaling methods.
4 Experiments
4.1 Experimental Setup
Setups. We train the language models using the AdamW optimizer (Loshchilov & Hutter, 2017), an extension of Adam (Kingma & Ba, 2014) with decoupled weight decay. Following common practice, we use decay rates B1 = 0.9, 82 = 0.95, and a weight decay of 0.1. The learning rate, beginning from 2e-4, follows cosine decay schedule, with the final learning rate set to 10% of the peak value. Models are trained on a total of 22B tokens with a global batch size of 256, which is sufficient to preserve the Gaussian distribution of weight updates. The input sequence length is fixed at 2048 tokens, and warm-up phase of 2,000 iterations is applied. Additional model configurations and training details are provided the Appendix. Training is conducted on 8 NVIDIA H800 GPUs, which do not natively support the mieroscaling data format available on the newer Blackwell GPU series. To enable MXFP8 computation on standard GPUs such as NVIDIA Hopper, we implement custom matrix multiplication kernels using Triton (OpenAT, 2025). We use the
7

---

Published as a conference paper at ICLR 2026
Training Loss
1.0
BF16 MOSS
Table 3: LLaMA-2-7B fine-tuning performance and evaluation accuracy on downstream lasks. System performance is assessed with model throughput, while
0.5
model performance is evaluated on Mathematics, GSM8K, NumGLE using accuracy.
Throughput (samples/s)
Model Performanne (Aoe%)
5000
6000
7000
Models
Mathematics
GSMSK NumCLUE
Training Step
BF
108.2
52.3
65.2
58.7
Figure 6: LLaMA-2-7B fine-tuning.
MOSS
241.8( 443.8%)
52.8
64.7
59.4
proposed two-level microscaling method to quantize activations and per-tensor quantization with automatic scaling for model weights in linear layers.
Baselines and Datasets. We compare MOSS against BF16 training and the COAT base- line (Xi et al., 2024) to evaluate its effectiveness in both LLM pretraining and fine-tuning tasks. Transformer Engine (Nvidia, 2025) also adopts MXFP8 for quantization; however, its microscaling strategy is supported only on NVIDIA Blackwell GPUs, which limits its broader applicability. For LLM pretraining, we train the OTMo-7B model on the Dolma corpus (Soldaini et. al., 2024) and report. perplexity on standard benchmarks including WikiTexi-103 (Merily al., 2016), C4 (Raffel et al., 2020), and The Pile (Gao et al,, 2020). For LLM fine-Luning, we focus on mathematicaa reasoning lasks by fine-tuning a LLaMA-2-7B model (Grattafiori el al., 2024) on the MAmmoTH datasel (Yue et al., 2023). We then evaluate the instruction-tuned model on three downstream benchnarks: Mathe- matics (Davies et al., 2021), GSMSK (Cobbe et al., 2021), and NumGLUE (Mishra al,, 2022).
4.2 LLM Pre-Training
We begin by comparing the performance of OLMo-7B models (Grocneveld al., 2024) pre- trained using FP8 mixed prccision with those trained using BF16. The pre-training loss over training steps is shown in Figure 5. All models use identical training configurations and hyperparameters as specified in the official report. As illustrated in Figure 5, the loss curves of BF16, COAT, and our proposed method, MOSS, closely align, exhibiting nearly identical trends. These results clearly demonstrate that the proposed FP8 mixed-precision scheme can match the training performance of the widely adopted higher-precision BF16 baseline.
We further evaluate the pre-trained models diverse set of downstream tasks, with results summarized Table 2. The proposed MOSS-traine models achieve zero-shot performance comparable to their BF16 counterparts, demonstrating that MOSS preserves both model accuracy and in-context, learning capabilities at a level on par with high-precision training. In terms of efficiency, Table 2 also reports training throughput, where MOSS delivers 34% speedup over BF16 and a 12% improvement over COAT, underscoring its superior performance. To assess convergence behavior, we extend pre-training of the OLMo-7B model with MOSS for an additional 30,000 steps (Lotaling 35,000 steps), presented in the Appendix, to validate the robustness and scalability of the MOSS framework.
4.3 LLM Fine- Fine-Tuming
We further evaluate the proposed FP8 low-prccision training scheme in the context of in- struction tuning for LLMs. To ensure fair comparison, adopt the same finc-tuning sctup as ToolBench (Guo ct al., 2024), using the open-source LLaMA-2-77 (Grattafiori et al., 2024) as the base model. Experiments are conducted on the math-focused MAmmoTTH dataset (Yue et al., 2023), finc-tuned for 5 epochs. During fine-tuning, MOSS matches the performance of the BF16 baseline, with loss curves that either closely aligned or exhibit. slightly improved stability (Figure 6). Downstream task accuracies are reported Table 3. These results further demonstrate the numerical precision and robustness of MOSS, con- firming that the proposed FP8 quantization scheme generalizes well beyond pre-training and remains effective in fine-tuning settings.
To further evaluate the scalability of MOSS on larger models, we fine-tune Qwen-3-14BB 14BB and Qwen-3-32B under both BF16 and proposed MOSS framework, and report their downstream reasoning performance. Experiments are conducted on curated dataset of MegaScience Fan et. al. (2025) and NuminaMaath-CoT Li et. al. (2024). All metries are averaged over three
8

---

Published as a conference paper at ICLR 2026
Table 4: Downstream evaluation accuracy (%) of Qwen-3-14B and Qwen-3-32B fine-tuned under BF16 and MOSS. All results are averaged over three independent runs.
Model
Precision
MATH1500
GPQA-Diamend
C-Eval
AIME24
MMLU-Redux
Qwen-3-14B
BF16 MOSS
96.8
62.1
86.2
79.1
88.6
96.7
61.5
86.7
79.4
88.5
Qwen-3-32B
BF16
98.3
64.2
88.2
79.3
89.6
MOSS
98.1
87.8
78.8
independent fino-tuning runs on five representative reasoning and knowledge benchmarks. Across both model sizes, MOSS matches the BF16 baseline on every task, demonstrating negligible performance loss despite operating at reduced precision. Notably, MOSS maintains strong stability on long-horizon reasoning tasks (e.g., MATH500, ATME24), where quantization methods often struggle with scale drift or cumulative error. These results indicate that MOSS scales effectively 30B- class models while preserving competitive downstream performance.
statistical reliability. Table 4 presents the accuracy
4.4 Memory and Communication Gains
We conduct detailed profiling of peak activation memory usage and inter-GPU communi- cation volume during LLaMA-2-7B fine-tuning on an 8xH200 cluster (3.2 TB/s aggregate NVLink bandwidth). All experiments use per-GPU batch size of 4, sequence length of 4096, and FSDP with ZeRO-2. Communication statistics are collected using the NCCL profiler and averaged over 1,000 training steps.
The measurements in Table 5 highlight the substantial memory-efficinncy benefits intro- duced by MOSS. Even on bandwidth-rich H200 GPUs, MOSS delivers 1.8x reduction in peak activation memory, confirming that the improvement arises from its FP8 microscaling algorithm rather than from hardware-level memory bandwidth limitations. We additionally evaluate compute- communication overlap using the PyTorch Profiler. MOSS achieves an overlap ratio exceeding 83%, indicating that it preserves strong scalability under high-speed NVLink interconnects and effectively reduces inter-GPU communicatin overhead during distributed training.
Table 5: Memory footprint and communication efficiency across BF16, COAT, and MOSS.
Memory
Communccation
Models
Peak Activation
AllReduxe Volume (GB/step)
AllReduce Latency
Overlap Ratio
Saving
(GR)
(ms)
(%)
BF16
42.3
3.84
1.00x
24.8
71.3
COAT
28.6
3.12
1.48x
18.6
78.5
MOSS
23.5
2.74
1.80x
16.2
83.4
4.5
Ablation Study
4.5.1
GEMM Performance
To evaluate the efficiency of the GEMM kernel MOSS, we conduct an abla- tion study comparing our implementation with several state-of-the-art FP8 quan- tization schemes, including Transformer Engine (Nvidia,, 2025). DeepGEMM from DeepSeek (hao et al., 2025), and
Table 6: Runtime of quantized FP8 GEMM on NVIDIA H800 GPUs.
Runtine(uss) DoepSoekk
M
N
K
TE
COAT
MOSS
2018
7168
4096
0.26
0.32
0.2
0.25
11008
0.63
0.86
4096
2045
7168
0.37
2.27
0.23
0.45
COAT (Xi et al., 2024). This evaluation
1096
1096
8192
0.52
2.63
0.39
0.60
4096
4096
12288
0.75
3.91
0.62
0.72
focuses on both raw GEMM performance
5120
5120
102.00
1.16
5.57
0.65
1.04
and the practical trade-offs in quantization and hardware deployment.
8192
8102
8192
2.16
AvE
0.84
3.73
0.54
0.77
As shown in Table 6, MOSS achieves GEMM performance that is comparable to TE. Im- portantly, MOSS significantly outperforms COAT GEMM throughput, highlighting the efficiency of our low-overhead quantization strategy. In comparison to DeepGEMM, our GEMM kernel runtime is relatively higher. This is expected, as DeepSeek leverages the In- creasing Accumulation Precision technique to reduce dequantization overhead and employs
9

---

Published as a conference paper at ICLR 2026
extensive hardware-specific optimizations targeting NVIDIA's Hopper GPUs. In contrast, our implementation is not tailored to Hopper GPUs, which lack the native Tensor Core support for MX data format. Despite this, MOSS remains competitive in performance across widely available hardware and maintains robust gencralizability without relying on hardwaro-specific enhancements.
4.5.2 Quantization Fidelity
To evaluate the fidelity of different quantization stralegies, extracted activation tensors from LLaMA-2-7B during the actual fine-tuning process. Specifically, we sampled tensors from three critical layers attention outputs, FFN intermediate activations, and LayerNort inputs every 100 training steps over a total of 10, 10,000 steps. For each sample, we computed the signal-to-noise ratio (SNR) under three quantization schemes: 1) per-tensor quantiza- tion; 2) per-group quantization (group size 128); 3) two-level microscaling (MOSS, micro group size = 32). The results, segmented into early (step < 2K) and late (step > 8K) training stages, are summarized in Table 7.
Table 7: SNR (dB) of activation tensors across different layers and quantization strategies during LLaMA-2-7B fino-tuning.
Per- Tensor
Per-Group
MOSS (Two-Level)
Layer
Early
Late
Farly
Tate
Early
Tate
Attention Output
28.4
26.7
34.8
33.2
37.6
36.1
FFN Intermediate
25.9
24.1
32.3
30.7
35.9
35.3
LayerNorm Input
31.2
29.5
36.7
35.1
39.4
38.0
Geometric Mean
28.3
26.6
34.5
32.9
37.5
36.0
The experimental results confirm our theoretical analysis in Section 3.1. Specifically, MOSS consistently achieves superior quantization fidelity, improving SNR by 3.0 3.4 dB over per- group quantization and 9.2 9.4 dB over per-tensor quantization. More over, this SNR advantage is maintained across all layer types and throughout the training process, demon- strating the robustness of our two-level microscaling approach. These findings provide strong empirical support for the effectiveness of MOSS and validate the conclusions discussed in Section 3.1.
5 Related Works
5.1 Low-Precision Training
Low- Low-precision training refers to using low-precision bits, such FP16/BF16, in the training process, as opposed using FP32 frel-precision throughout the training process. It is one of the most prominent directions in modern deep learning, as it offers reductions in com- putation costs, memory footprint and communication overhead (Micikevicius et al., 2017). Most existing training frameworks, e.g., Megatron-LLM, LLM (Shoeybi et al., 2019) and Deep- Speed (Rasley et al., 2020). adopt BF16 LLM training by default. With the release of Nvidia Hopper GPUs, FP8 training (Micikevicius et al., 2022) becomes the next-generation low-precision technique. Transformer Engine (Nvidia, 2025) from NVIDIA is the first frame- work designed for FP8 linear layers while leaving all other operations in full precision. FP&-LM (Peng et al., 2023) extends FP8 quantization to gradients and first-order optimizer states, further improving the training throughput. However, they fail to reduce the memory footprint of activa- tions, not fully exploiting the potential of memory and communication benefits of FP8 in fully shardod distributed training scenarios. COAT (Xi et al., 2024) and DeepSeck-V? (Liu et al., 2024) both extend the per-group FP8 quantization to activations, achieving mixed granularity quantization. However, the fine-grained quantization strategy incurs additional computation cost in the dequantization process.
training, however, it only supports FP8 compute for
5.2 Scaling Techniques
Quantization transforms high-precision values within given range into lower- val- of different range by tensor scaling. Scaling factor determination during training
10

---

Published as a conference paper at ICLR 2026
typically follows one of two strategies: Delayed scaling estimates the current scaling fac- using historical record of maximum absolute values from previous iterations. This approach assumes statistical consistency across iterations, making it vulnerable to outliers that violate this assumption and potentially destabilize training (Fishman et al., 2024). In contrast, just-in-time scaling dynamically adjusts scaling factors based the current data distribution, typically using on-the-fly max-roduction. While more adaptive, this method troduces significant computational overhcad overhead-partieulleeeee particularly in FP8 training training-whent where multiple memory accessos can offset the performance gains of low-precision computation. Beyond these non-automatic scaling approaches, Unit Scaling (Blake et al,, 2023) recently intro- duced an automatic scaling method that maintains unit variance weights and activations by applying static scaling constants throughout the network. However, Unit Scaling re- quires manual modification of the computational graph to insert unit-scaled operations and preserves certain "critical matmuls" (e.g., attention output projection and FFN down pro- jection) BF16 for stability. In contrast, MOSS offers a generic and automated scaling mechanism that integrates seamlessly into modern LLMss achieving full FP8 precision with- out requiring precision fallbacks or manual graph modifications.
Conclusion
In this work, present MOSS, novel framework for efficient and accurate FP8 training of large language models. To address the significant dequantization overhead on slower CUDA cores within the GEMM main loop, we introduce a two-level microscaling quanti- zation scheme that minimizes overhead while preserving high numerical precision at fine granularity. Then, by leveraging the observation that parameter updates bounded by the learning rate, we propose an automatic scaling mechanism that eliminates the runtime cost of online scaling. Through extensive experiments on a 7B parameter model, MOSS achieves training accuracy on par with BF16 baselines while delivering 34% improve- ment in throughput. Furthermore, the framework generalizes effectively to downstream fine-tuning tasks, demonstrating versatility beyond pre-training. Overall, MOSS offers practical and hardware- hardware-friendly solution for scalable FPS training. Future work could fur- ther explore combining our proposed MOSS with other low- low-precision compression methods to reduce computation overhead in non- con-linear layers.
11

---

Published as conference paper at ICLR 2026
References
Shoshana Abramovich, Graham Jameson, and Gord Sinnamon. Refining jensen's inequality. Bulletin mathématique de la Société des Sciences Mathématiques de Roumanie, pp. 3 5-14, 2004.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brun- dyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprinl arXiv:2406. 11704, 2024.
Darvish Rouhani Bita, Garegrat Nitin, Savell Tom, et al. Ocp microscaling formats (mx) specification, 2023. URL httpp://wwww.oppoooe/ wwtp:oopeccompte.org.. wwtp:ttt....oooooeeeeeeeeeee /tt////oouum//////oooooo haais///ccccmicccaaiinniee
formats- vermats-mx-v----aee mx-v-----speeffina
Charlie Blake, Douglas Orr, and Carlo Luschi. Unit scaling: Out-of-the-box low-precision
training. In International Conference on Machine Learning, pp. 2548 2576. PMLR, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. With shared microexponents, little shifting goes long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-13, 2023.
Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, Daniel Zheng, Nenad Tomašev, Richard Tanburn, Peter Battaglia, Charles Blundell, András Juhász, et al. Advancing mathematics by guiding human intuition with ai. Nature, 600(7887):70 74, 2021.
Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812, 2025.
Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms. arXiv preprint arXiv:2409.12517, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint #xXiv:210100002 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Ka- dian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaa 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafford, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. et al. Olmo: Accelcrating the science of language models. arXiv preprint axXiv:2402.00888, 2024.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking tool learning of large language models, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
12

---

Published as conference paper at ICLR 2026
Yujun Lin, Haotian Tang, Shang Yang. Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. arviv preprint arXiv:2405.04522 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint urXiv:2412.19433) 2024.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011 2017.
Stephen Mcrity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
Paulius Micikevicins, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.037400, 2017.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint K5iv:2209005433, 2022.
Paulius Micikevicius, Stuart Oberman, Pradeep Dubey, Marius Cornea, Andres Rodriguez, lan Bratt, Richard Grisenthwaite, Norm Jouppi, Chiachen Chou, Amber Huffman, et al. Ocp 8-bit floating point specification (ofp8). Open Compute Project. Rev 1.0. 2023, 2023.
Swaroop Mishra, Arindam Mitra, Ncoraj Varshney, Bhavdcep Sachdova, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks. arXiv preprint arXiv:2204.06660, 2022.
Nvidia. Fp8 training with transformer engine, 2025.
OpenAI. Triton, 2025. URL https:/ttrioonllangg.o hwwp://aangggggggaaaaaaem htt///mainniiie org/mmain/indexxxtmll
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310.13313, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1-67, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505 3506, 2020.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arxiv: 19119999005,, 2019.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 2xXx:2402.00055. 2024.
Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Koutzer, Jianfei Chen, and Song Han. Coat: Compressing optimizer states and activation for memory-efficiett fp8 training. arXiv preprint arXiv:2410.19313, 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0565) 2023.
Chenggang Zhao, Liang Zhao, Jiashi Li, and Zhean Xu. Deepgemm: clean and efficient fp8 gemm kernels with fine-grained scaling. 2025.
13

---

Published as conference paper at ICLR 2026
A Model Configuration
For all experiments, we adopt the default hyperparameters the official training recipe. Table presents the details of model configurations.
Table 8: Model sizes, architectures, and training hyperparameters.
Model
Paramns
Dimension
#heads
leyers
Learning
Weight decay
Sequence length
Batch size
OLMo
7B
4096
32
32
3e-1
0.1
2018
2
LLMMA
7B
4096
4096
2
B MOSS Pretraining Performance
We evaluate the training performance of MOSS by training it for 36,000 sleps on approx- imately 144B tokens. Due to resource constraints, we do not perform full pre-training for the BF16 and COAT baselines. As shown in Figure 7, the training curve demonstrates the stability of our proposed MOSS framework.
10
MOSS
Training Loss
8
4
2
10000
20000
30000
Training Step
Figure 7: Loss Curve of MOSS FP8 Training.
C Discussion of Bound in AdamW Optimizer
Compared Adam optimizer (Kingma Ba, 2014), AdamW (Loshchilov & Hutter, 2017) decouples the L2 regularization from the loss function and directly inject the weight decay, ie., W,,,, into the weight update. In this way, AdamW enforces stable shrinkage and tends to converge to smaller absolute values, particularly in high-magnitude regimes. Based on the update rule Equation (1), we have
M.
m.
W = -1-nA) VV < W.
(11)
VV +
Therefore, the maximum absolute value of the weights at timestep remains bounded by max([W,) < max( Wol) + n.tt, allowing the scaling factor update rule in Equation (10) to be directly applied to AdamW-optimized training.
D Ablation Study on different intervals
To understand how the update interval affects the trade-off between computational over- head and model accuracy, we conduct an ablation study on I.LaMA---7B fine-tuning with range of interval values. Table 9 reports the corresponding scaling overhead, effective throughput, and downstream accuracy NumGLUE.
Our findings reveal a clear pattern. Extremely frequent updates-sude as the just-in- time configuration that recomputes scaling every step-introduce substantial overhead (3.8 ms/step) without improving throughput. In contrast, moderate update intervals in the range of 100 500 steps reduce scaling overhead to negligible level (<0.03 ms/step),
14

---

Published as conference paper at ICLR 2026
Table 9: tuning.
Impact of scaling interval overhead, throughput, and accuracy during fine-
Scaling Overhead (ma/step)
Effective Throughput
Accuracy (NumGIUUB)
Method
Interval
BF16 JIT
1.xx
58.7%
1.xx
59.3%
MOSS
100
0.03
1.043x
59.4%
MOSS
500 (default)
0.02
1.044
59.4%
MOSS
2000
0.01
1.044x
58.1%
yielding the highest effective throughput (1.043-1.044x) while matching or slightly ex- ceeding BF16 accuracy. When the interval becomes too large (e.g., 2000), we observe mild drop in accuracy due to insufficient tracking of scale drift, although the throughput remains comparable.
Overall, these results demonstrate that MOSS highly robust to the choice of update interval across broad regime. The default value of 500 steps strikes an optimal balance, minimizing scaling overhead while preserving accuracy and maximizing training efficiency.
E End-to-End Throughput Improvement of Automatic Scaling
To quantify the end-to-end throughput benefits of different weight quantization strategies, compared three scaling methods during LLaMA-2-7B fine-tuning on 8x GPUs:
1. Just-in-time (JIT) scaling: Performs on-the-fly max-reduction at every forward pass.
2. Delayed scaling: Uses historical maximum values with moving window.
3. Aulomatic scaling (MOSS): Predicts scaling factors based on the learning rale schedule (our method).
Table 10: End-to-end throughput of different weight scaling strategics during LLaMA-2. 7B fine-tuning on 8xH800 GPUs
Scaling Overhead
Total Step Time (ms)
Throughput (token/s)
Method
Speedup
(ms)
JIT scaling
3.8
68.5
38,642
1.0x
Delayed scaling
1.2
65.9
40,182
1.04x
MOSS (Automatic)
0.2
63.1
41,998
1.087x
Table summarizes the end-to-end throughput during fne-tuning (batch size = 4, se- quence length = 4096), averaged over 1,000 steps. As shown in Table 10, our automatic scaling method achieves 8.7% end-to-end throughput improvement over the JIT base- line. This improvement primarily arises from eliminating the costly memory access pat- tern in JIT scaling, which requires reading the entire FP32 weight tensor from HBM for max-reduction. In contrast, MOSS incurs only negligible, constant-time overhead while maintaining accuracy.
Furthermore, to demonstrate the scalability and accuracy, we evaluated the downstream task accuracy of our automatic scaling method against just-in- must-in-timme scaling under iden- tical fine-tuning settings (LLaMA-2-7B on the MAmmoTH dataset). The results are summarized in Table 11.
Table 11: Downstream task accuracy comparison between JIT scaling and MOSS (Auto- matic). Differences are reported relative to JIT scaling,
Scaling Method
Mathematics
GSM8K
NumGLUE
JI'T Scaling
52.6%
64.9%
59.2%
MOSS (Automatic)
52.8%
64.7%
59.1%
Difference
+0.2%
-0.2%
+0.2%
The observed differences in accuracy are within the statistical noise margin (+0.3% across three independent seeds), confirming that the theoretical upper bound in Equation (10)
15

---

Published as conference paper at ICLR 2026
BF16 (Baseline)
38
MOSS
36
(dR()B)
34
Per-aroup
32
Goal: Higher Throughput Higher SNR (Points towards the topiright optimal)
30
28
Per-tensor
32500 35000 37500 40000 42500 45000 47500 Throughput (tokens/second)
Figure 8: Quantization Technique Trade-offs: Throughput vs. Precision.
and the dynamic re-scaling mechanism provide sufficient numerical precision for practi- cal training. Consequently, the 8.7% end-to-end throughput improvement achieved by automatic scaling is obtained without any degradation in model quality.
F
Visualization of Quantization Methods
To give an overview about the trade-off between quantization fidelty and throughput, visualize the performance landscape of different quantization strategies on 2D plane (as shown in Figure 8), where the X-axis represents throughput Cookens/secoue and the Y-axis represents quantization fidelity (SNR. in dB). This visualization integrates our ablation study results and clearly illustrates that MOSS lies on the desirable Pareto frontier, effectively avoiding the core limitations exhibited by prior approaches.
G Limitations
While MOSS demonstrates promising results in FP8 training efficiency, our work has several limitations. First, our focus on GEMM operations leaves room for improvement in non-linear layers (e-g., activations, LayerNorm). where low-precision overheads persist. Then, the current implementation optimizes for CUDA cores, and its performance gains may not generalize fully to other architectures, i.e., TPUs or AMD GPUs, without further adaptation. Finally, although MOSS achieves parity with BF16 on 7B model, its efficacy at larger scales (e.g., 100B+ parameters) remains to be validated, particularly where gradient dynamics may differ.
H Broader impact
Our proposed algorithm, MOSS, can improve efficiency and reduce the energy consump- lion of training neural networks, which helps reduce the carbon footprint caused by deep learning. By reducing computational costs, MOSS could lower barriers train- ing and fine-tuning large models, benefiting resource-constrained researchers. Moreover, widespread adoption of FP8 training might necessitate hardware/software co-design, in- centivizing vendors to prioritize FP8 support and standardization.
16


Reviews:
[{"review_id": "YlSTDZlwpu", "reviewer": "Reviewer_CWko", "summary": "This paper introduces MOSS, a framework for efficient FP8 training of LLMs. Its core contributions are a two-level microscaling strategy for activations to reduce dequantization overhead, and a novel automatic scaling mechanism for weights that exploits the bounded update property of Adam to eliminate runtime max-reduction costs.", "questions": "1. Could you zoom in Figure 5 to demonstrate alignment with the baseline? \n2. Why were no experiments conducted on newer LLMs like Qwen? Will you validate MOSS on these models, or are there technical barriers?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Automatic weight scaling (Adam’s bounded update) avoids real-time max-reduction, outperforming TE’s delayed scaling, which is novel and efficient.\n2. Proofs (SNR, bounded updates) validate designs; experiments with clear metrics compare MOSS to BF16/COAT.\n3. Well-structured framework with visualizations and detailed experimental setups for reproducibility.\n4. Custom kernels enable MXFP8 on non-native hardware.", "weaknesses": "1. Limited originality of two-level microscaling: The strategy overlaps heavily with the MXFP standard (OCP’s microscaling format), which already defines tensor subblock partitioning and E8M0 local scale factors to optimize FP8’s dynamic range. The addition of a FP32 global scale is also used in NVFP format, limiting originality in this module.\n2. Experimental gaps: Figure 5 (OLMo-7B pretraining loss) obscures the BF16 baseline curve for steps > 2000, precluding direct verification of MOSS’s claimed loss alignment with BF16. For steps < 2000, the BF16 curve exhibits unexpectedly worse performance, which is unconvincing. Additionally, the training loss remains far from convergence, casting doubts on MOSS’s long-term stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JAyqtXsih2", "reviewer": "Reviewer_5kdF", "summary": "The paper introduces MOSS, an FP8 training framework for large language models that aims to keep Tensor Cores busy while maintaining accuracy. It has two core ideas: a two‑level microscaling scheme for activations (a coarse global scale plus tiny per‑32‑value power‑of‑two scales) that avoids dequantization work in the GEMM inner loop, and an automatic scaling method for weights that predicts scale changes from optimizer settings to remove costly per‑step max‑reductions and extra memory traffic. Implemented with custom Triton kernels on NVIDIA H800 GPUs, MOSS trains 7B‑parameter models. Overall, MOSS offers a practical, hardware‑friendly path to fast and stable FP8 training.", "questions": "- Fig. 4 fixes the interval at 500; how do accuracy/acceleration change with different intervals or learning‑rate schedules?\n\n- Can you report peak activation memory and inter‑GPU bandwidth for BF16, COAT, and MOSS to substantiate the Intro’s memory/communication claims?\n\n- Table 3 shows 241.8 vs 168.2 samples/s but lists “+47.5%”; by calculation it’s ~44%. Is this a typo or based on a different baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Kernel‑aware two‑level microscaling keeps the GEMM inner loop on Tensor Cores and shifts dequantization to the epilogue; the mechanism is clearly illustrated.\n\n- Solid empirical parity with BF16 at 7B alongside better throughput.\n\n- The writing and figures are clear and the limitations section is candid about scope", "weaknesses": "- The paper focuses on throughput but does not report memory/communication gains,\n\n- MOSS’s GEMM is slower than DeepGEMM on several shapes (Table 4)\n\n- Longer runs in Appendix B report only MOSS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CyxUq0pgqk", "reviewer": "Reviewer_9Gkb", "summary": "This paper presents MOSS, a new FP8 training framework for large language models that aims to make low-precision training both faster and more stable. The key idea is a two-level microscaling quantization that uses a global FP32 scale together with small local 8-bit power-of-two scales, greatly reducing dequantization overhead without hurting accuracy. In addition, MOSS introduces an automatic scaling method that predicts how scaling factors evolve during training using properties of Adam-style optimizers, avoiding the costly max-reduction steps typical in FP8 systems. Experiments on OLMo-7B and LLaMA-2-7B show that MOSS matches BF16 accuracy while improving training throughput by about 34–47/%, making it a practical and hardware-friendly approach for efficient FP8 large-model training.", "questions": "1. The automatic scaling method assumes that parameter updates remain bounded by the learning rate. Have the authors observed any failure cases when using different optimizers (e.g., Lion or Adafactor) or under larger learning-rate schedules?\n2. While the SNR analysis (Theorem 1) shows clear theoretical benefits, the empirical gains in perplexity appear small. Could the authors comment on whether the higher SNR mainly improves training stability (e.g., fewer overflows) rather than end-task accuracy?\n3. MOSS focuses on GEMM layers; however, many practical LLM bottlenecks come from nonlinear layers and communication. How difficult would it be to extend microscaling or automatic scaling to these components?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation and practical relevance for FP8 LLM training.\n2. Elegant two-level microscaling design balancing accuracy and efficiency.\n3. Simple yet effective automatic scaling removing runtime overhead.\n4. Strong empirical results: BF16-level accuracy, 34–47% faster.\n5. Works on standard GPUs without special hardware support.", "weaknesses": "1. Evaluation is limited to mid-sized models (up to 7B parameters); scalability to larger settings (e.g., 30B–32B models) is not demonstrated.\n2. The paper mainly reports throughput improvements, but does not deeply analyze memory, communication, or energy efficiency, which are also key for FP8 training.\n3. While results are strong on core GEMM operations, extensions to other components (e.g., LayerNorm, activation functions, or optimizer states) remain unexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lBtFtSIE0k", "reviewer": "Reviewer_XXJt", "summary": "The paper presents MOSS, a framework for efficient and accurate FP8 training of large language models. It addresses the computational overhead and instability issues in existing FP8 methods, which often use mixed-granularity quantization and costly just-in-time scaling. MOSS introduces a two-level microscaling strategy that combines global FP32 scaling with local scaling, and an automatic scaling mechanism based on optimizer properties. Experiments show that MOSS matches BF16 accuracy while improving training throughput compared with other SOTA methods.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper pinpointly identify the quantization overhead related to on-the-fly quantization.\n\n2. The proposed two-level microscaling method effectively balances numerical precision and hardware efficiency.\n\n3. The proposed optimizer-based automatic scaling mechanism is novel and effective\n\n4. Extensive experiments on pretraining and SFT demonstrates promising performance.", "weaknesses": "1. The evaluation focuses mainly on 7B-parameter models, leaving scalability to larger models (e.g., 14B, 30B+) untested. Considering the huge cost of pretraining such a huge model, demonstrating the effectiveness on SFT should be enough.\n\n2. The authors should provide more explanation about Figure 1 (a). Is there any comparison between the BF16 GEMM baseline, and its a little bit strange about why COAT's FP8 GEMM is 5x slower than the TE baseline or DeepSeek baseline.\n\n3. (minor) There's a missing citation of QServe, where it also observe the quantization overhead introduced on Ampere architecture (as illustrated in Figure 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FBgRDKrDUD", "reviewer": "Reviewer_ZzQN", "summary": "This paper introduces MOSS, a specialized strategy designed to accelerate FP8 training for Large Language Models (LLMs). Traditional training frameworks often face a trade-off between quantization accuracy and computational overhead, dictated by the granularity of quantization. MOSS innovatively proposes methods to better balance these two factors within the FP8 training framework: First, for sensitive activation tensors, it employs a two-step quantization strategy using coarse-grained FP32 quantization scale factors combined with fine-grained MXFP quantization factors, addressing the high computational overhead associated with the dequantization phase in traditional per-group quantization. Second, for weight tensors, it adopts a strategy of real-time prediction of learning rates during training, thereby avoiding the computationally expensive real-time calculation of scaling factors. Experiments demonstrate that their strategy achieves a throughput improvement of approximately 35% over a BF16 baseline method while maintaining the training performance of a 7B model.", "questions": "1.  Have you surveyed other efficient FP8 GeMM implementations besides COAT and DeepSeek? What do you consider to be the advantages of your method compared to highly efficient kernels like DeepGeMM, which rely primarily on system-level code optimizations?\n2.  Do you apply any special processing to gradient quantization, similar to the specific handling done for weights or activations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1.  **Important Problem, Insightful Perspective:** The trade-off between quantization accuracy and computational overhead is a central focus in the field of quantized training. The paper's focus on FP8 training is one of the most important and active problems in this area, and the proposed solution for balancing these factors is highly instructive.\n2.  **Sufficient Novelty:** The two-step quantization framework using coarse-grained FP32 scale factors combined with fine-grained MXFP scale factors effectively addresses the significant computational overhead associated with the dequantization phase within traditional per-group frameworks on CUDA Cores. This approach cleverly leverages high-performance Tensor Cores. The framework for automatic weight quantization is novel; by integrating with the learning rate within the optimizer, it directly bypasses the step of calculating scaling factors in real-time, offering potential for acceleration.\n3.  **Relatively Complete Theoretical Analysis:** The theoretical analysis is convincing, whether discussing the motivation for the two-step quantization or the mathematical proof regarding the combination of autoscaling and learning rate. These analyses also incorporate practical considerations. For example, periodically recalculating the true scaling factor during autoscaling maintains mathematical correctness while adhering to the practical requirements of quantized training.", "weaknesses": "The paper has few shortcomings elsewhere, but the ablation experiments regarding the critical aspect of \"computational efficiency\" are insufficient. For example:\n\n1.  **End-to-End Throughput Impact of Automatic Weight Scaling:** How much throughput improvement does automatic scaling on weights *actually* provide in an end-to-end scenario? The paper only shows improvements for GeMM alone (Table 1). While automatic scaling is certainly better than in-time scaling for GeMM separately, in-time scaling might not cause significant throughput reduction end-to-end. If the accuracy degradation caused by potentially less precise weight quantization under automatic scaling outweighs the throughput benefits end-to-end, the approach would be counterproductive. The authors are encouraged to provide analysis on this aspect.\n2.  **Two-Stage Scaling Strategy for Activations:** Although the authors theoretically prove that MOSS can yield higher Signal-to-Noise Ratio (SNR), experimental validation is lacking. Ideally, a training curve like Figure 5 would be helpful, though I fully understand computational resource constraints. Alternatively, extracting activation tensors during the actual training process and providing quantized SNR results for per-group, per-tensor, and MOSS would better support the conclusions in Section 3.1.\n3.  **Overall Performance Visualization:** A single, comprehensive experimental result chart showing that MOSS's two-stage activation quantization strategy achieves higher accuracy than per-tensor methods and higher throughput than per-group methods (combining existing ablation study results) would be ideal. This would clearly demonstrate the excellent trade-off achieved and make it easier for readers to grasp.\n\nIn summary, while the paper provides detailed theoretical analysis for its two main methods, it lacks corresponding experimental validation. Even a slightly more detailed ablation study with concrete results would significantly strengthen the conclusions. This is somewhat regrettable. A more thorough ablation study would incline me more towards accepting this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)