Metis: Training LLMs with FP4 Quantization
Authors: Hengjie Cao
Mengyi Chen
Yifeng Yang
Fang Dong
Ruijun Huang
Jixian Zhou
Anrui Chen
Mingzhi Dong
Yujiang Wang
Jinlong Hou
Yuan Cheng
FAN WU
Fan Yang
Tun Lu
Ning Gu
Li Shang
Venue: ICLR 2026 Poster (2026)
Keywords: FP4
Full Quantization Training
LLM
Primary area: foundation or frontier models, including LLMs

Abstract: This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLMs). These spectra are dominated by a small fraction of large singular values, inducing wide numerical ranges that cause quantization bias and severe spectral distortion, ultimately degrading training performance. This work presents \emph{Metis}, a spectral-domain quantization framework that partitions anisotropic spectra into narrower sub-distributions for independent quantization, thereby reducing errors and preserving spectral structure. To minimize overhead, 
Metis leverages two key properties of the dominant spectral subspace: preservation via sparsely random sampling and preservation via random projection, reducing decomposition cost to a negligible level. On LLaMA-3 8B trained with 100B tokens, Metis enables robust W4A4G4 training with FP4 quantization of weights, activations, and gradients, yielding only a 0.4\% training loss gap and a 0.1\% degradation in downstream accuracy relative to BF16. Beyond matching BF16 fidelity, Metis also surpasses Nvidia’s FP4 recipe, consistently achieving lower loss and higher downstream accuracy while incurring significantly lower computational overhead. 
The code implementation for Metis is available at: \url{https://github.com/sii-research/Metis}.

METIS: TRAINING LLMs WITH FP4 QUANTIZATION
Hengjie Cao'
Mengyi Chen
Yifeng Yang!
Ruijun Huang'
Fang Dong' Jinlong Hou Li Shang
Jixian Zhout
Anrui Chen
Mingzhi Dong
Yujiang Wang". 4.5
Yuan Cheng
Fan Wu
Fan Yang' Tun Lu Ning Gu
Fudan University
University of Bath Shanghai Innovation Institute
University of Oxford
5 Oxford Suzhou Centre for Advanced Research
Huawei
ABSTRACT
This work identifies anisotropy in the singular value speetra of parameters, ac- tivations, and gradients the fundamental barrier to low-bit training of large language models (LLMs). These spectra are dominated by a small fraction of large singular values, inducing wide numerical ranges that cause quantization bias and severe spectral distortion, ultimately degrading training performance. This work presents Metis, a spectral-domain quantization framework that parti- tions anisotropic spectra into narrower sub-disse sub-distributions for independent quanti- zation, thereby reducing errors and preserving spectral structure. To minimize overhead, Metis leverages two key properties of the dominant spectral subspace: preservation via sparsely random sampling and preservation via random projec- tion, reducing decomposition cost to a negligible level. On LLaMA-3 8B trained with 100B tokens, Metis enables robust W4A4G4 training with FP4 quantiza- tion of weights, activations, and gradients, yielding only 0.4% training loss gap and a 0.1% degradation in downstream accuracy relative to BF16. Beyond matching BF16 fidelity, Metis also surpasses Nvidia's FP4 recipe, consistently achieving lower loss and higher downstream accuracy while incurring significantly lower computational overhead. The code implementation for Metis is available at: /tttps:/ggithnn com ccw./iirreeerrrreeeeeeee Metis
INTRODUCTION
Training large language models (LLMs) with low-bit quantization of parameters, activations, and gradients offers substantial gains in efficiency, cost, and scalability. In recent years, progress has advanced from FP32 to BF16 and, more recently, FP8 training (Micikevicius et al. 2022 Peng et al. 2023: Perez et al. 2023). Looking ahead, Nvidia's 's Blackwell technical report shows that the NVFP4 format reduces memory consumption by 1.8x and accelerates General Matrix Multiplications (GeMM) by 7x compared to FP8, underscoring the efficiency potential of FP4 training (Avvree al. Devleker & Ghodsian). However, pushing the training frontier further down FP4 is not straightforward continuation: FP4 imposes exponentially tighter constraints on precision and dynamic range, which conflict with the inherently wide distributions of parameters, activations. and gradients.
This study investigates the origins of these wide distributions and analyzes their impact on training stability and effectiveness under FP4 quantization. The key findings are outlined below and visualized in Fig.
Anisotropy is universal in modern LLMs. In weight, activation, and gradient matrices, small fraction of singular values dominate, yielding highly imbalanced spectrum. This phenomenon is consistently observed across model architectures and parameter scales up to 671B.
Anisotropy induces wide numerical distributions. Wide distributions of weights, activations, and gradients arise from the superposition singular components: larger components contribute to the value region, whereas smaller ones concentrate near zero. This spread originates from variability in singular values, which projects aligned components into entries of corresponding magnitudes.
"Co-first authors.

---

Quantization bias induces spectral distortion. Commonly used block-level low-bit quantization introduces bias that disproportionately favors large values, thereby reducing the effective resolution for small values. Under anisotropy. this effect leads to severe distortion in spectral space: smaller singular components incur substantially larger value errors and direction perturbations than their larger counterparts.
10
(A)
109 (C)
10
Quasrized
Original
100-
(D)
1+64
1-512
10
1000
2000
3000
0000
10f
1000
2000
3000 4000
Spectrum Index
10
Spectrum Index
Proguency
im
dim
10
6.5
109
6.0
10:01
10
[Value]
[Value]
Spectrum index
Figure 1: Overview of anisotropy and its impact on quantization, illustrated using gradient matrix from LLaMA-3 8B. (A) Singular value spectrum exhibits strong anisotropy, with a few singular values dominating the spectrum. (B) The wide matrix distribution arises from the superposition of singular components: large components (e.g., i=0) drive the high-value region, while small components concentrate near zero. (C) Quantization bias disproportionately rounds many small values zero. (D-E) In spectral space, smaller singular components incur substantially larger relative quantization errors in singular values and more severe perturbations singular directions. Details corresponding to (A) in Section 2. (B) in Section 2.2 and (C-E) in Section 2.3
Inspired by these findings, we propose Metis, a quantization framework that preserves the spectral structure under FP4 formats and allows robust FP4 training. Metis operates the spectral domain, applying decomposition to weight, activation, and gradient matrices to disentangle dominant from long-tailed singular components, thereby allowing quantization over substantially narrower distribu- tions. A central challenge is the computational complexity of spectral decomposition. To address this, we leverage two key structural properties revealed in our empirical study:
(i) Subspace Preservation via Sparsely Random Sampling, where the dominant subspace estimated from a sparsely random sampled subset is reliably generalized to the whole batch;
(ii) Subspace Preservation via Random Projection, where the dominant subspace can be faithfully captured within a reduced hidden dimension via random projection.
Building these insights, Metis renders the decomposition cost negligible by employing sparse random sampling over sequences and random projections on the hidden dimension, each reducing complexity by approximately two orders of magnitude.
Metis enables robust W4A4G4 training by quantizing all GeMM matrices to 4-bit floating point, On 8B LLaMA-3 model (et al.) 2024) trained with 100B tokens from the DCLM dataset (Li et al. 2025), Metis narrows the gap to BF16 to only 0.4% increase in training loss and a 0. 1% degradation downstream accuracy. Compared
implementation of NVIDIA's FP4 recipe (Devleker & Ghodsian), Metis consistently achieves lower training loss and higher downstream accuracy while incurring significantly less computational overhead.
ANALYSIS
Anisotropy emerges as a key structural factor underlying the wide distributions observed in weights, activations, and gradients. This section analyzes these anisotropic matrices and examines how they misalign with low-bit quantization schemes. Unless otherwise specified, all experiments are conducted on the LLaMA-3 8B model trained on 100B tokens from the DCLM dataset.
2.1 ANISOTROPY: A UNIVERSAL PROPERTY OF MODERN LLMS
For a matrix M E m, we perform Singular Value Decomposition (SVD) to obtain singular min(m.n) left singular vectors {u:) Rm and right singular vectors (v:) R". such values {at)
2

---

that M min(m.n) -i=1 y. We assume singular values are sorted in descending order, i.e., 9 02 2 ... o, with min(m,n).
Anisotropy is characterized by a spectrum where few leading singular values dominate, yielding a highly imbalanced distribution across directions. Previous studies have reported anisotropy in activation matrices (Ethayarajh) 2019 Mu et al. 2017 Puccetti et 2022 Rudman Eickhoff] 2023 Yu et al. 2021), and our analysis further demonstrates that it is universal across weights, activations, and gradients. As shown in Figure (A), their singular value spectra exhibit pronounced anisotropy, with only 0.63%, 3.15%, and 2.91% of components (identified by the elbow point of maximum curvature) dominating the spectra of weights, activations, and gradients, respectively. We further validate this pattern on publicly released models, including the Qwen family (Bai et al. (023) and DeepSeek-R2 (Liu al., 2024); where weight matrices from 7B to 67 IB parameters consistently show fewer than 3% of singular values dominating the spectrum, confirming anisotropy universal property across architectures and scales.
(A) Singular spectrum of matrix
10:
J+0
Singoler
201-
1..6600.63%
10
155
2ow02.91%
1=2b6
2=2040
10'
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
300n
400n
Weight Spectrum lodex
Spectrum Index
Gradient Spectrum Index
(B) Numeric value distribution various rank-1 components
Prequency
10a
10
1=0
10
301-
101
2040
10-
10
10
10
10* 10
10-7
10
10
10
10
10
10-7
10-5
10-
[Woight]
[Activation
[Gradient]
Figure 2: Analysis of weight, activation, and gradient matrices (layer 32, FeedForward(FFN). (A) The singular value spectra exhibit strong anisotropy, with only 0.63%, 3.15%, and 2.91% % of components (identified by the elbow point of maximum curvature) dominating the spectrum. (B) Filled regions denote full-matrix distributions; dashed histograms showes selected rank-l components (u,o,v; for i 0, 16, 128, 1024). Dominant components (e.g., = 0) drive the high-value region, while smaller ones (e.g., i = 1024) contribute near zero. See A.1 for additional results.
2.2 ANISOTROPY INDUCES WIDE NUMERICAL DISTRIBUTIONS
The wide distributions of weights, activations, and gradients arise from the superposition of singular components (u,ayv;); ): dominant components (c.g., i 0) account for the large-value region, whereas smaller components (c.g.. 10244 concentrate near zero, as illustrated in Fig, (B).
Wide distributions arise from the skewed singular value spectrum. This spread originates from variability in singular values, which project aligned components into entries of corresponding magnitudes. In the SVD decomposition M = Σ; ".u.v," large singular values amplify aligned components into mign-magn entries, whereas small singular values suppress others toward near zero, thereby producing long-tailed distributions. After isolating the impact of singular values, Fig. 3 A) shows that the singular vectors all exhibit similar shapes with widths far narrower than that of the full matrix. Thus, the wide ranges are direct consequence of the skewed singular spectrum.
Residual singular components confirm the dominant subspace effect. To validate this mechanism, we analyze residual matrices, corresponding to the long-tail singular components, after subtracting the leading singular components amplified by large singular values. As shown in Fig. (B), removing the top 3% of components, which dominate the spectrum, yields residual distributions one two orders of magnitude narrower than those of the full matrix. This demonstrates that wide ranges arise primarily from the dominant subspace, while the residual remains quantization-fries frienddly..
3

---

(A) Numeric value distrbution of singulaar vectors
107
Proqueney
10a
1-0
100-
10a
1-256
101
10
101
1=2048
10"18
10"1
10
10-1
10
10
10-4
10
102
10-10
10
10
10-1
(B) Numeric value distribution of residual matrix
10"
Proquency
10.
Residnal
10x
10
Ovarall
101-
101
10
10-10
10-1
10-10
10-7
[Woight]
[Activation)
[Gradiont]
Figure 3: Analysis of weight, activation, and gradient matrices with hidden dimension 4096 (layer 32,
FFN). (A) Left singular vector distributions: all exhibit similar shapes with widths much smaller than that of the full matrix. (B) Yellow regions show the residuals after removing the top 128 components (3% x 4096 x 123, rounded to the nearest power of two), while the grey region represents the original matrix distribution. The residuals are
two orders of magnitude narrower than the full matrix, confirming that wide ranges originate from dominant components. More results in A.2
2.3 QUANTIZATION BIAS: SPECTRAL DISTORTION
In extreme low-bit regimes such as FP4, block-wise quantization a commonly-used approach. By restricting scaling to small blocks with fewer entries, the likelihood of encountering extreme values is reduced, resulting in narrower local distributions and finer ainer-griieed scaling that mitigates quantization error (Rouhani al.) 2023 Nvidia 2025). Moreover, fixed-size blocks align naturally with Tensor Core execution, enabling efficient parallelism (Rouhani et al. 2023
Nvidia 2025)
Formally, a b-bit quantizer Q, maps a matrix M E |R=X" to its quantized form M = (M).
Throughout the paper, we will use the bar symbol to denote the matrix after quantization for simplicity. The matrix is partitioned into fixed-size blocks (e.g.. size 16 in the NVFP4 format (Nvidia 2025 ) For each block E R NVFP4 selects a single scaling factor s FP8-E4MB set by the block's maximum magnitude and rounded up to the nearest representable FP8 value to ensure coverage. Each element then quantized as QNV (e:) = round & ) s8, where round(.) denotes rounding each scaled element to the nearest FP4-E2M1 representable value.
Quantization bias. broad distributions, where the range can span multiple orders of magnitude, as observed in weight, activation, and gradient matrices in Fig. 2 determining the scaling factor by the block maximum introduces bias. This bias disproportionately favors large values while suppressing the resolution available for small ones. As shown in Fig. (C). nearly half of the values are rounded entirely to zero after quantization, resulting in the destructive loss of the information they represent.
Spectral distortion. In anisotropic matrix with uneven singular value distribution, quantization bias causes severe distortion in spectral space, especially for small singular components. As shown in Fig. (D) (E), all components suffer relative errors in singular values and directional perturbations, with smaller ones exhibiting greater distortion in both magnitude and direction.
3 METIS
We propose Metis, an FP4 training framework designed to address the challenge of spectral anisotropy. Metis partitions the spectrum into narrower sub-distributions and applies quantization independently within each. This design yields two benefits: (i) shown in Fig. 3 each sub-distribution is significantly narrower than the full spectrum, reducing quantization error; and (ii) it prevents small singular components from being overwhelmed by large ones, thereby mitigating perturbations in singular values and preserving directional consistency within the subspace.
The central challenge, however, lies in the high cost of repeatedly performing spectral decomposition during training. Section 3.T shows that anisotropy induces two key properties of the dominant subspace, preservation via sparsely random sampling and random projection, making scalable

---

integrates this scalable decomposition into GeMM. the source of over 95% of the training workload in large language models (Vaswani et al. 2017 Wang et al. 2025). applying FP4 quantization to all GeMMs in both forward and backward passes. Section 3.3 analyzes the additional computational complexity introduced by Metis.
spectral decomposition feasible. Section 3.2
3.1 ENABLING PROPERTIES FOR SCALABLE SPECTRAL DECOMPOSITION
The prohibitive cost of spectral decomposition presents a major obstacle to deploying spectral-domat quantization at scale. Let X E R denote the activation tensor, where / = b s with b the batch size, s the sequence length, and m the hidden dimension. In practice, can reach millions while m is typically in the thousands. Performing full SVD on such matrices at every iteration incurs cost of Olm²), making direct application impractical.
However, anisotropy offers crucial opportunity: it causes small number of singular values & to dominate, with k < I, m, so spectral decomposition only needs to isolate the corresponding dominant subspace, yielding markedly narrower spectral sub-distributions. Building on this, our further investigation reveals two properties that enable dimension reduction along both the sequence and hidden axes: (i) the dominant subspace estimated from sparsely sampled subset generalizes reliably to the full batch, and (ii) the dominant subspace can be faithfully captured within a reduced hidden dimension via random projection.
Subspace Preservation via Sparsely Random Sampling. The sequence dimension is typi- cally orders of magnitude larger than the dom- inant subspace dimension abundance of samples ensures that the dominant subspace can be estimated stably even from
Dominant sabspace
similbrity
Intuitively, this
0.8
0.7
small random subset of samples. From theoret- ical perspective, the anisotropic structure guar- antees that random subsampling preserves the dominant subspace with high probability. Let = f X IR mm be the covariance. For random subset X RR xn with 4 << the X satisfies the
0.6
1%
10%
100%
Sample rolio
Figure 4: Subspace alignment between the domi- nant subspace of the full batch and that of randomly sampled subsets of sequences. Alignment quickly saturates as the sample ratio increases, with just 1 1 of sequences achieving nearly 0.9 alignment with the full-batch subspace. More results in B.1
sample covariance Σ
matrix Chernoff Tropp 2012 :
Pr [|= -
E1l2 I
E
2 1-- 8, whenever 4. = =0 n log m)
2
Combined with the Pavis-Kahan theorem DDvis & Kahan 1970p. | A
where Aş and AS Rmn* are the top-k eigenspaces of Σ and Σ, and A is the eigengap, this guarantees that the dominant subspace estimated from a small random subset faithfully recovers that of the full batch. Empirically, we measure the alignment of the dominant subspace between the full batch and randomly sampled subsets using the mean squared canonical correlation between their orthonormal bases. We observe rapid saturation as the sample ratio increases, where the sample ratio is defined as the fraction of tokens included in the sampled subset relative to the total number of tokens in the global batch. For instance, shown in Fig. sampling only 1% of sequences achieves nearly 0.9 overlap with the full-batch subspace, demonstrating the effectiveness of this subspace approximation.
Subspace Preservation via Random Projection. We further establish that the dominant spectral structure remains stable under random projections of the hidden dimension. Let Ω E (X++s) be a Gaussian test matrix with oversampling parameter s, and form the sketch Z = XS with thin QR factorization = HR. Randomized SVD theory (Halko et al. 2011) shows that, with high probability, the projection error II(I HH |X is controlled by the tail singular values of X. Furthermore, by Davis-Kahan, the subspace error for the top-k eigenspace satisfies ||
(HÁ,) (HA))T l|2 < [HH )x||2 where A E Rmmk are the top-k eigenspaces of H ΣH. This demonstrates that modest oversampling parameter and nontrivial spectral gap suffice to ensure that Gaussian projections preserve the dominant subspace with high fidelity, enabling accurate recovery without operating in the full ambient dimension m.
5

---

Scalable Spectral Decomposition. Leveraging these properties, adopt two-step procedure for efficient decomposition. First, sparse random sampling selects less than 1% of sequences to estimate the dominant subspace, which is then broadcast to the full batch. Second, apply randomized SVD to recover the top-k components. This reduces the complexity from ( to O (lynk)), while the additional cost remains asymptotically negligible compared with forward and backward GeMM.
Temporal Reuse Across Steps. We further observe strong temporal stability in the dominant spectral structure: the leading low-rank subspace of both activation and output-gradient matrices remains highly aligned across adjacent steps. Thus, after one spectral decomposition, we reuse the extracted low-rank factors for several subsequent steps. In practice, we refresh the decomposition every 8 steps. further reducing the amortized overhead.
3.2 SPECTRAL DECOMPOSITION
Each matrix in GeMM decomposed into a low-rank component and a residual: low-rank singular values are kept in high precision, while the associated singular vectors and residual are quantized to low bit. Metis handles parameters, activations, and gradients differently: for parameters, low-rank and residual parts are stored as separate trainable variables and updated independently; for activations and gradients, spectral decompositions are recomputed dynamically at each iteration.
Forward pass. Let W E REx" denote weight matrix and X' E Rxxxx an input activation tensor, where 6 is the batch size, s the sequence length, and m the hidden dimension. For the GeMM operation, X' is reshaped into matrix X I with / bs. The output is then computed as =x E R x Our goal is to quantize this GeMM computation under low-bi bit formats.
rank-k: approximation of W is given by W. = U V where U2 E m Ve E Rank and S₁ diag(o).......
E R The residual is WR w We. so that W W + Wn n V WR. Similarly, the activation matrix X is decomposed as: X = X, + XR A A XR, where A E R' B, Rm and A₃ R
Accordingly, the forward GeMM can be written as
Y -A + X + WR)
The quantization function 2, is then applied separately to matrices in Eq. other than S₁ and A Specifically, the forward computation of Y under b-bit quantization, denoted Y, is computed as
Y = (..( + 2x (. 2
V + XWRR
Backward pass. In backward propagation, we quantize the GeMM operations associated with derivative computations of matrices in Eq. T Formally, denote the derivatives of the loss function we need to compute the following derivatives ac 8x aU, öS, "V and ac Similarly, we first decompose D into oW
L with respect to (w.r.)) Y as D
Oc
E Lxx"
BY
for updating parameters: dc Oc dc
SVD low-rank approximation and the residual, which is D P& + DR, where Py E gi is computed as
Qg E x
Ty E & The derivative aL ax
OC
OL OY
OY OX = U. WH),
0X
=P.T.Q. V.S. U +P.T.Q. WT +DRVAS DR
We then compute
ac under 4-bit quantization as 0x
OC 0X = P.T.QQ. V.S. U. + P.T.Q WT + DR VR U. + DR WT
Following a similar pathway, we compute other derivatives under 4-bit quantization as follows:
ac
Wc =
+ + -
U. x P. V
U V
aUk
as
Øc
Oc
0V: -S
X
X X +
OW R

---

3.3 DISCUSSION ON TRAINING EFFICIENCY
We analyze Metis's 's additional computational overhead, arising from (i) the extra small-scale GeMMs introduced by the decomposition and (ii) the decomposition itself. To mitigate this cost, we further develop a customized optimized operator for low-rank spectral decomposition; detailed optimization strategies are described in B.2 Kernel-level overhead comparisons and end-to-end empirical results
are reported in B33
Forward pass. In the baseline, evaluating Y = XW requires (Imn) operations. Under Metis, the forward computation follows Eq. [ where the additional mixed products introduce Ollk + mank + Ink) overhead on top of the baseline cost. Moreover, the activation decomposition performed at each step adds (lymm)), where 41 < / by sparse random sampling.
Backward pass. In the baseline, gradients with respect to X and W require (lmm) operations. Under Metis, gradient computation follows Eq. 3 where mixed products contribute O(mm makk Ink). In addition, output gradient decomposition at each step adds (lgnk)).
Overall complexity. Combining forward and backward contributions, Metis introduces an additional cost of (lmk + mnk + Ink + lank + lgnk) per training step, which is asymptotically much smaller than the baseline (lmm),, making Metis tractable at the scale of modern LLMSs
4 EXPERIMENTS
Experimental results show that Metis under FP4 training maintains, and in some cases improves, both training loss and downstream accuracy relative to BF16 baselines, supporting FP4 as a viable setting for large-scale model training.
(C) Llama--30 308, B) Train Loss (8) GPT-2 1B) Train Loss (A) GPT-2( 330M) Train Loss
Models and Datasets. We conduct experiments on GPT-2 (130M and 1.1B) (Radford et al.)
BF16 buseline
Metis+NVEP4
3.5
2019) and LLaMA-3 (8B) (et al. 2022)) For pretraining,
NVFP4
the DCLM (Li et al. 2025)
3.0
50B
92 3B
950
97.50
1008
dataset and train each model on 100B tokens.
The raw data are segmented to split long doc-
uments and concatenate short ones, filtered to remove non- con-Unicode and non- non-English content, and further processed with Qwen3 to discard entries whose perplexity deviates by more than two standard deviations from the mean. For downstream evaluation, we consider three task
3-
00
20B
408
60B
80B
100B
BF16 beseline
Netis+NVEP1
3.5
NVFP4
3.0
90B
92,5B
95B
97.5B
1000
types: question answering, classification, and cloze prediction. For question answering, we
use ARC (Clark et al.] 2018). RACE (Lai et al.)
2017), and BoolQ (Clark et al. 2019); for classi- fication, use HellaSwag (Zellers al. 2019)
0B
20B
40B
60B
80B
100B
BF16 beseline
and PIQA (Bisk et al.) 2019;; and for cloze pre- diction, we use LAMBADA (OpenAI) (Kazemi
Metis Netis+NVEPP
2.6
NVFP4
ct al. 2023).
90B
92,5B
95B
97.5B
1008
FP4 Training. The efficacy of Metis under FP4 quantization is evaluated against both FP4 and BF16 baselines. All FP4 training in this work adopts W4A4G4 quantization, where weights, activations, and gradients are represented in the E2M1 NVFP4 format. Our main experiments with NVFP4 are conducted through simulation in BF1 Stochastic rounding (SR) is applied by default in all FP4 experiments, as it mitigates quantization bias, is orthogonal to other meth-
0B
20B
40B
60B
80B
100B
Token Processed
Figure 5: Training loss curves for (A) GPT-2
130M, (B) GPT-2 1.IB, and (C) LLaMA-3 8B. Direct NVFP4 incurs a loss gap of 3-4% relative to BF16 baseline, while Metis reduces the gap to
0.4% on LLaMA-3 and even slightly surpasses the BF16 baseline on GPT-2 models.
ods. The Metis rank is fixed at 1.5% for low-rank approximation both forward and backward C.2 shows that 1.5% is sufficient maintain performance.
passes, our sensitivity analysis in C.2
7

---

4.1 MAIN RESULTS
Training Loss. Fig. 5 shows the training loss curves of the BFI6 baseline, NVFP4, and NVEP4 + Metis, while Table reports the corresponding final test losses. NVFP4 exhibits clear gap relative to the BF16 baseline, particularly on LLaMA-3 8B, with test loss increasing by 3-4% across all models. In contrast, Metis narrows the gap to 0.4% on LLaMA-3 8B and even achieves slightly lower loss than the BF16 baseline on GPT-2 models. This expressivity enhancement of Metis over the BF16 baseline may stem from the separation of low-rank and residual branches of weight matrices during training, which reduces interference between feature subspaces and contributes to the observed performance gains. We further examine the spectrum the residual matrix to test whether anisotropy re-emerges. The results in C.3 show it does not: the singular value spectrum of the residual matrix remains flat, indicating that anisotropy is effectively addressed by the low-rank branch.
Performance Downstream Tasks. As shown in Table [ direct NVFP4 quantization results in an average drop of 1% relative to the BF16 baselines across all models. In contrast, Metis consistently outperforms these FP4 baselines, reducing the drop to 0.1% LLaMA-3 8B and even slightly surpassing the BF16 baseline on GPT-2 models. These findings are consistent with the loss results reported earlier and demonstrate Metis's ability not only preserve but, in some cases, to enhance model expressiveness under ultra-low-bit constraints.
Table 1: Downstream performance across different settings, reported with task specific metrics. Direct NVFP4 quantization leads to an average drop of 1% relative to the BF16 baseline, while Metis reduces the gap 0. % on LLaMA-3 and slightly surpasses BF16 on GPT-2 models.
Model
Method
Loss
ARC-C
ARC-E
BoolQ)
LAMBADA
PIQA
RACE
HellaSwag
Avg
GPT-2
BF16 FP4
3.23
24.3
32.1
60.4
31.3
62.1
47.3
32.5
41.4
(130M)
3,32
22.9
31.5
60.7
31.2
60.9
47.0
31.7
40.8
FP4-Metis
3,20
24.1
48.4
GPT-2
BF16
3.09
30.7
59.5
60.0
35.4
64.9
47.3
41.1
48.4
(I.1B)
FP4
3,22
29.8
57.6
59.8
35.5
63.6
46.8
39.5
FP4 Metis
3.01
30.1
60.0
59.5
36.2
64.1
48.5
41.9
48.6
LlaMa-3
BF16
2.44
32.4
60.4
59.2
37.6
70.5
47.0
50.9
51.1
FP4
2.53
31.3
58.5
58.2
36.5
69.8
46.9
49.4
50.0
FP4-Motis
2
32.7
39.6
59.8
37.0
70.9
46.5
50.7
51.0
4.2 ABLATION STUDY
We ablate the two key components of Metis: Spectral Decomposition and Sparse Random Sampling.
Spectral Decomposition. To assess the role of spectral decomposition weights, activations, and gradients, we replace it with direct FP4 quantization for each case. As shown in Table [ removing spectral dccomposition from gradients yields the largest degradation relative to Metis, with training loss increasing by 2.4% and downstream performance dropping by an average of 1.0%. In comparison, removing it from activations or weights results in smaller and similar degradations: training loss rises by about 0.5% and downstream performance decreases by average of 0.3%.
Sparse Random Sampling. We further evaluate Sparse Random Sampling by replacing it with full-batch spectral decomposition. As shown in Table 2 sparse random sampling reduces the decomposition cost by orders magnitude while introducing negligible performance impact.
Table 2: Performance 3 8B under different ablation settings of Metis. "w/o" denotes removal of the corresponding component. For FP4 training, spectral decomposition is most critical for gradients (removal yields the largest performance loss), followed by activations and weights Sparse random sampling performs on par with full-batch spectral decomposition.
Method
Loss
ARC-C
ARC-E
BoolQ
LAMBADA
PIQA
RACE
HellaSwag
Avg
Metis
2.45
32.4
59.2
50.9
$1.1
w/o Weight Decompusition
2.47
32.7
58.6
58.5
36.9
71.0
45.8
51.4
50.7
wlo Activation Decomposition
246
33.9
59.2
58.7
35.8
71.6
46.2
51.2
50,9
wlo Grudient Decomposition
2.51
30.6
60.3
59.1
36.2
68.5
46.8
49.7
50.1
wio Sparse Random Sampling
2.44
32.9
60.0
59,1
37.8
70.7
46.7
50.5
51.1
8

---

4.3 EXTENDED EVALUATION
Metis is designed as training-time method to address the compute and memory bottlenecks of high-precision LLM training such as BF16. We therefore focus primarily on FP4 pre-training results, and additionally evaluate lightweight inference-time variant under post-training quantization (PTQ) for completeness.
Pre-training Comparison with Nvidia's Recipe. We compare Metis with Nvidia's 's FP4 recipe (De vleker Ghodsian), which incorporates a random Hadamard transform alongside SR to mitigate the impact of outliers. Our implementation of this recipe is described in Appendix [..] As shown in Table 3 for GPT-2 models, Nvidia's 's recipe results 1-2% higher test loss and an average 0.5% drop in downstream performance relative the BF16 baselines, whereas Metis surpasses BF16 on both test loss and average downstream performance.
Table 3: Nvidia's recipe yields 1-2% higher test loss and 0.5% drop in downstream performance relative to BF16, whereas Metis surpasses BF16 on both metrics.
Model
Method
Loss
ARC-C
ARC-E
BoolQ
LAMBADA
PIQA
RACE
HellaSwag
Avg
GPT-2
BF16
3.23
24.3
32.1
60.4
31.3
62.1
47.3
32.5
41.4
(130M)
TP4 Nvidia's Recipe
3.27
23.7
31.2
59.4
30.5
62.0
47.6
31.9
40.9
GPT-2
BF16
3.09
30.7
59.5
60.0
35.4
64,9
47.3
41.1
48.4
(1.18)
HM4-Metis
30.1
59.5
64.1
41.9
48.6
Nvidias Recipe
3.15
28.6
59.8
99.3
36.7
63.2
48.0
40.2
47.9
The advantage of Metis lies in two aspects: (i) as shown in A.3 spectral decomposition yields much narrower residual distributions. whereas random Hadamard transform only smooths few outliers without reducing overall spread; and (ii) as shown in Fig. 6 Metis attains better alignment of singular directions and lower singular- value error than both direct NVFP4 and NVFP4 with Nvidia's recipe, thereby preserving the spectral structure critical for model performance.
30
1.00
Brror(()
20
NVFP4+NV Recipe
Coo-sim
0.75
NVPP4+NV
NVFP4+ Motis
NVFP4 Metis
10
0.00
1000 Spectrum index
500
1000
1500
2000
Spectrum index
(a)
(b)
Figure 6: Spectral preservation under different quantization strategies. (a) Alignment of left singular vectors measured by cosine similarity. (b) Relative error in singular values. Metis achieves the highest vector alignment and the lowest singular-value error relative to the BF16 baseline.
PTQ Comparison. We evaluate lightweight inference variant of Metis on the LLaMA-3 8B model and compare it with strong PTQ baselines including LQ-LoRA (Guo et al., 2024) and OmniQuant (Shao et al., 2023}. inference time, Metis applies spectral decomposition only to model parameters, eliminating activation-side spectral processing and adding only 3% FLOPs when using a 1.5% rank. As shown in Table 4 Metis achieves higher downstream performance than both baselines, with an average improvement of 0.6% over LQ-LoRA and 0.7% over OmniQuant, while remaining close to the BF16 baseline.
Table 4: Metis improves average downstream performance by 0.6% over LQ-LoRA and 0.7% OmniQuant, while requiring only 3% additional FLOPs and remaining close to the BF16 baseline.
Model
Method
Loss
ARC-C
ARC-E
BoolQ
LAMBADA
PIQA
RACE
HellaSwag
MMLU
Avg
LLaMA-3
BF16
2.44
32.4
60.4
59,2
37.6
70.5
47.0
50.9
30.7
48.6
(8B)
LQ
2.51
32.6
60.1
58.4
34.9
71.3
45.5
48.6
29.4
47.7
OmniQuant
2.47
30.9
59.4
58.1
36.8
69.3
46.3
48.7
31.0
47.6
Metis
37.3
48.3

---

5
RELATED WORKS
Block-wise microscaling alleviates FP4's range mismatch but still compresses small values into few bins, causing information loss. Existing methods fall into three categories:
Channel-wise Re-parameterization. Methods like SmoothQuant (Xiao et al. 2023) use calibrated per-channel scaling balance activation and weight magnitudes, folding scales offline. Outlier Suppression+ (Wei et al. 2023) adds per-channel shifts, while OmniQuant (Shao et al. 2023)
introduces learnable transforms and weight clipping. These diagonal methods struggle with residual extremes in low-hit settings.
Hadamard transformations. Orthogonal Hadamard rotations (Suresh al. 2017) redistribute
outliers across channels to relax per-block ranges. QuaRot (Ashkboos et al. (024) applies them
to hidden states and weights for end-to-end 4-bit inference, while QulP (Chee et al. (003) uses
randomized preprocessing for weight-only 4-bit PTQ. HALO (Ashkboos et al. a (025)) inserts rotations both forward and backward passes to stabilize low-precision training. NVIDIA's FP4 recipe (De vleker & Ghodsian) similarly combines random Hadamard transforms with stochastic rounding for W4A4G4 training. Despite these advances, all Hadamard- basedd methods incur notable overhead
from repeated rotations and mainly smooth outliers without narrowing overall distributions, leaving quantized tensors vulnerable to precision loss.
Outlier Separation. These methods divert extreme values into small high-precision branch. Outlier Clamping and Compensation (Wang et al. (025) clamps top-quantile activations and recovers clipped residuals via sparse GEMM, while SVDQuant (Li 2024) shifts activation mass into weights and absorbs outlier energy in a high-precision low-rank branch. Both reduce error but rely on auxiliary precision paths, deviating from full FP4.
These approaches fall short for end-to-end FP4 training: channel-wise methods are calibration- dependent, Hadamard rotations add complexity without ensuring uniform variance, and outlier separation requires high-precision branches. Instead, we use randomized SVD to disentangle dom- inant and long-tail tail singular vectors, transforming broad distributions into ceatiizationfrrinndj. narrow-range forms with minimal overhead, enabling fully FP4 training.
6 CONCLUSIONS
We identified anisotropy in the singular value spectra of parameters, activations, and gradients as fundamental obstacle to low-bit L.LM training and introduced Metis, spectral-domain quantization framework that mitigates this challenge by partitioning spectra into narrower sub-distributions and preserving structural fidelity with negligible overhead. On LLaMA-3 8B, Metis enables robust W4A4G4 training with less than 0.4% loss gap and under 0.1 % downstream degradation relative to BF16, while also surpassing Nvidia's ' FP4 recipe. These results establish Metis as a practical, high-fidelity approach for efficient FP4 training of large language models. Ongoing work explores scaling to larger models and hardware-native deployment, with the goal of broadening access to high-performance, energy-efficient L.Mss.
ACKNOWLEDGMENTS
This work supported by Fudan University and the Shanghai Innovation Institute. Partial support for this work was also provided to Yujiang Wang through the Basic Research Program of Jiangsu (BK20240414) and the Leadership Talent Program (Science and Education) of Suzhou Industrial Park (KJQ2024204).
10

---

REFERENCES
Eduardo Alvarez, Omri Almog, Eric Chung, Simon Layton, Dusan Stosic, Ronny Introducing nvfp4 for efficient and accurate URL sttp::///eveeooorr..vdidaaa..aa. combblog..
Krashinsky, and Kyle Aubrey. low-precision inference.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213-100222 2024.
Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L Castro, Torsten Hoefler, and Dan Alistarh. Halo: aeltamard-assister lower-precision optimization for Ilms. arXiv preprint TrXiv:2501.02625. 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
www.xii.org//boo.. 1911
11641
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36: 4396 4429, 2023.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019. URL
Ming-Wei
https: httpppss//xxxvv.orrg/o
905. 10044
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL httthtp:://// pectps://arxiv/ wttp://w./.bbb....oooo 05457
Chandler Davis and William Morton Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1-46, 1970.
Kirthi Devleker and Farshad Ghodsian. Nvfp4 trains with precision of 16-bit and speed and efficiency of 4-bit. URL https:// oper wwwppppwww......o../..... www.diia......cooo/oo/oooo ott.-:/ee--eoo-eoeeaeeeaaaaaaiia
ivvp4-trains-wiiet
A. Grattafiori et al. The Ilama herd of models. 2024.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings, arXiv preprint
2019.
Han Guo, Philip Greengard, Eric P. Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.1202. 2024.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2): 217-288, 2.
Mchran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. Lambada: Backward chaining for automated reasoning in natural language, 2023. URL https://arxiv
orq/ htgg/bs// 2212. 13894
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations, 2017. URL https: :/tarr///..i.../ http://wwww..////b/b///.//////// org/abs./////..eooooo 68.
11

---

Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-lm: In search of the next generation of training sets for language models, 2025. htthttts::/// //tari/aavvvvvv/ org/ abs 2406 11794
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:241 1.05007, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenth- waite. Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433 2022.
Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. arXiv preprint arxiv:1702.01411, 2017.
Nvidia.
Introducing nvfp4 for efficient and accurate low-precision inference nvidia
technical blog, 6 2025. URL
introducing
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan Xiong, Ziyue Yang, Bolin Ni, Jingeheng Hu, et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310. 18313, 2023.
Sergio P Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy- Kramer, Paul Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon. Training and inference of large language models using 8-bit floating point. arXiv preprint arXiv:2309. 002009972200 /7224, 2023.
Giovanni Puecetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. Outliers dimensions that disrupt transformers driven by frequency. arXiv preprint arXiv:2205.11380 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310. 10537, 2023.
William Rudman and Carsten Eickhoff. vrXiv:2305.19353 2023.
Stable anisotropic regularization. arXiv preprint
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.133333 13137, 2023.
Ananda Theertha Suresh, X Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International conference on machine learning, pp. 3329-3337, PMLR, 2017.
Joel A Tropp. User-friendly tail bounds for sums of random matrices. Foundations of computational mathematics, 12(4):389-453 2012.
12

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization. arXiv preprint arXiv:2501.17116. 2025.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint 23XX::2300400145, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099, PMLR, 2023.
Sangwon Yu, Jongyoon Song, Heeseung Kim, Seong-min Lee, Woo-Jong Ryu, and Sungroh Yoon. Rare tokens degenerate all tokens: Improving neural text generation adaptive gradient gating for rare token embeddings. arXiv preprint arXiv::1109.03127, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL /ttp:///////a//////oooooo///////
07830
13

---

APPENDIX
This section is organized to present empirical results on additional model structures and modules, which complement the Analysis and Methods sections, as well as experimental details and supple- mentary results that support the Experiments section.
A ANALYSIS
A.1
ANISOTROPY: A UNIVERSAL PROPERTY OF MODERN LLMS
The universality of anisotropy further supported by evidence from open-sourced LLM weight matrices.
SVD Spectra Deepest MLP weight
Owen2 5-78
Owend 328
0wen2.5-728
Deep R1 6710
2016--122
348-764
Siqudor coloe
17%
Sigudar volue Indox
Singiar value Index
Singular volue Indox
Siegula value Indox
Figure 7: Singular value spectra of the final FeedForward module in Qwen2.5-7B, Qwen3-32B, Qwen2.5-72B, and DeepSeck-R1-671B. The elbow fraction f k* /r. where k* is the index of maximum curvature, indicates that only small fraction of singular values dominates the spectrum (1.9%, 2.2%, 2.1%, 2.4%), demonstrating the universality of anisotropy modern LLMs.
A.1.1 LLAMA-3 8B
Fig. demonstrated that weight, activation, and gradient matrices exhibit highly anisotropic spectra, with only a few singular values carrying most of the energy. To further substantiate this observation, provide additional results for LLaMA-3 8B in Figures below. Specifically, we show the singular value spectra and matrix distributions of W& and W Wyfnl from the first layer, as well as from the last layer. Consistent with the main analysis, these spectra are strongly anisotropic: the leading singular values dominate, while the majority of smaller components contribute values concentrated near zero. These results confirm that spectral anisotropy is a persistent phenomenon across different layers and parameter types.
(A) Singular spectrum of matrix
Singular Values
101-
B.615
10a
9b5%=9.57%
j-0
10a
10-1
10-2
I=256
10-3
10-3
1000
2000
3000
400n
1000
2000
3000
4000
1000
2000
3000
4003
Weight. Spectrum Index
Activation Spectrum Index
Gradieat Spectrum Index
(B) Numeric value distribution of various rank-1 components
Frequency
10a
10a
10
1-0
10a
10a
10
1-256
101-
107
10-0
10-1
10
10-2
10t4
10-1
30-3
109
10-11
10
10-7
16
[Weight]
|Activation|
[Gradient]
Figure 8: Analysis of weight, activation, and gradient matrices (layer 1, Attention Key). Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
14

---

(A)
Singular spectrum of matrix
Singulor Values
101
10
10
10
1.8800.80%
10
1.76%
2=0
100.
1ow-6.07%%
1-256
10a
1+2040
10
4000
5000
10000
1000
2000
4000
Weight Spectrum Index
Activation Spectrum Index
Gradient Spectrum Index
(B) Numeric value distribution of various rank-1 components
107
Frequency
10
10
1-0
10*-
107
250
10-11
10
10
10
10
10
10:11
10"
10
10"
[Weight]
AAtivvttion]
[Crodient]
Figure 9: Analysis of weight, activation, and gradient matrices (layer 1, FFN dense2). Singular value spectra show strong anisotropy. with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
(A) Singular spoctrum matrix
Singulor Valtes
101-
1.3%
1bok-3.66%
10-1-
1Ga
2.1%%..17%
10-3-
1n322
1-256
10.3.
101
10
1-2048
1000
2000
3000
4000
1000
2000
3000
4000
1000
2000
3000
4000
Weight Spectrum Index
Activation Spoctrum
Gradient Spectrum
(B) Numeric value distribution various rank-1 components 10"
Prequency
1-32
10*
101
10
10
2-246
10
10
10
10 10
11
10
[Weight]
[Activation]
[Gradient]
Figure 10: Analysis of weight, activation, and gradient matrices (layer 32, Attention Key), Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
(A) Singular spectrum of matrix
Singular Values
101-
102
1loww99557
10-1
1ow-2.66%
J+0
10p-
1bow=0.05%
10
1=256
10-4
1000
2000
3000
4000
2500
5000
10000
1000
2000
3000
4000
Weight Spectrum Index
Activation Spectrum Index
Gradient Spectrum Index
(B) Numeric value distribution of various rank- components
10a
10"
Frequency
10
10a
10
10
258
10
10
10
10" 1
10'
10 10 30
10
10
[Weight]
[Activation]
[Gredient]
Figure 11: Analysis of weight, activation, and gradient matrices (layer 1, FFN dense2). Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
A.1.2 GPT-2 1.1B
Fig. demonstrated that weight, activation, and gradient matrices exhibit highly anisotropic spectra, with only a few singular values carrying most of the energy. To further substantiate this observation, we provide additional results for GPT-2 8B in Figures below.
15

---

(A) Singular spectrum of matrix
Singulor Values
10-
0.w-0.25%
10a
ilow-11777
10
0.9000.00%
I=0
10-1
107
10:4-
im64
1=012
400
600
1000
400
1000
400
600
1000
Weight Spectrrm Index
Activation Spectzum Index
Gradicnt Spectrum Index
(B) Numeric value distribution of various rank-1 components
10a
Prequency
10
101
10
10
10
10
10"2
10"
(Weight)
|Activation]
[Gradient]
Figure 12: Analysis of weight, activation, and gradient matrices (layer I, Attention Key). Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant
components drive the high-value region, while smaller ones contribute near zero.
(A) Singular spectrum of matrix
Singulor Values
lbow-2.00 03%
10
howw1,.88
I=C
10
2.9%=0.29%
10
1=812
i-64
400
1000
1000
400
600
1000
Weight Spectrum Index
Activation Spectrum Index
Gradient Spectrum Index
(B) Numeric value distribution of various rank-1 components
10"
Frequency
10
10'
I-
10a
10a
10r
101
10
10
10
10
10
10
10a
10
10
10
(Weight]
|Activation|
[Gradintt]
Figure 13: Analysis of weight, activation, and gradient matrices (layer 1, FFN dense2). Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
(A) Singular spectrum of matrix
Singulor Values
10
Ibow=2.39%
0.29-0
10-3
I=C
101-
10->
ie64
10-1
10
10-1
1=512
1000
400
1000
400
Weight Spectrum Index
Activation Spectrum Index
Gradient Spectrum Index
(B) Numeric value distribution of various rank-1 components
10a
102
10a
I-C
102
10a
101
10
10
10
10
10
10
[Wightt]
[Activation]
[Gradient]
Figure 14: Analysis of weight, activation, and gradient matrices (layer 32, Attention Key). Singular value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
A.2 ANISOTROPY INDUCES WIDE NUMERICAL DISTRIBUTIONS
This section provides additional empirical evidence from a broader range of models and modules, further supporting the analysis in the main text. The results demonstrate that the distributions of singular vectors remain narrow and largely scale-invariant, while the residual components are substantially compressed after removing the dominant singular values.
16

---

(A) Singular spectrum of matrix
Singulor Values
10
101
10a
bow
1bow=1.518
I=c
2.4%
10-2
1m64
10
1-012
10
400
1000
1000
4000
500
600
1000
Weight Spectrum Index
Activation Spoctrum Index
Gradient Spectrum Index
(B) Numeric value distribution of various rank-1 components
10
Frequency
10
10
I
10p-
10a
101
107
10
10
(Welght]
[Activation|
Gradient
Figure 15: Analysis of weight, activation, and gradient matrices (layer I, FFN dense2). Singular
value spectra show strong anisotropy, with a few values carrying most of the energy. Dominant components drive the high-value region, while smaller ones contribute near zero.
A.2.1 LLAMA-3 8B
We include further results for LLaMA-3 3 8B, specifically from layer and layer 32, as well as from the Key projection in the Attention module and the second dense layer in the Feed-For Ford-Fr Network (FFN).
(A) Numeric value distribution singular vectors
Frequency
10a
10:
10
101
10
2-2
[-2048
10P
109
10a
10-1
10->
(B) Numeric value distribution of residual matrix
Frequency
109
10
10'
10a
10'
Overall
10
10
10-1
10-3
10-6
10-4
10-2
10-11 10
10-7 10-5 [Gradient]
10-> 10-1
[Weight]
|Activation|
Figure 16: Analysis of weight, activation, and gradient matrices (layer 1, Attention Key), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
(A) Numeric value distribution of singular vectors
102
Prequency
10F
(w32
301
10
10
[-2048
[-256
10a
10
10
10
10
10
10
10:
10t
10
10
10:7
10
1 10
(B) Numeric value distribution of residual matrix
107
Frequency
10a
10a
Residual
10a
10'
10P-
Overail
101
10
10-
30
10-5
10:4
10
10-1
[Weight]
|Activation
[Gradient]]
Figure 17: Analysis of weight, activation, and gradient matrices (layer 1, FFN dense2), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
17

---

(A) Numeric value distribution of singular vectors
Prequency
10
C im0
1-32
101
2-18
10-10
10
10
10
10-18
10"2
10
10
10
10-11
10
10
10
10 10
(B) Numeric value distribution of residual matrix 10"
Frequency
10a
10a
10
Residual Overall
10a
101-
101
101
10
10
10
10 10
10
10" 10 10 10
10
10
10 10
[Welght]
|Activation]
[Gradlent]
Figure 18: Analysis of weight, activation, and gradient matrices (layer 32, Attention Key), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
(A) Numeric value distribution of singular vectors
10a
Frequency
10'
10a
10
101
1-2018
1-256
10a
10a
10a
10
10
10
10
10
10
10
10
10-5
10 10
(D) Numeric value distribution of residual matrix
10"
Frsquency
10a
Residual
10a
10a
10'
Overail
10
107
10a
11
10
(Weight]
|Activation]
[Gradient]
Figure 19: Analysis of weight, activation, and gradient matrices (layer 32, FFN dense2), showing that singular vectors are narrow and residuals 1 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
A.2.2 GPT-2 1.IB
We include further results for GPT-2 I.IB, specifically from layer and layer 32, as well as from the Key projection in the Attention module and the second dense layer in the feed-Forward Network (FFN).
Numeric value distribution singular vectors
10a
10k
Frequency
10a
O
101
10
10P
10
10"
10
10
10
10-11
10
10
10-4 10-1
(B) Numeric value distribution of residual matrix
10f
Frequency
10
102
Residual
Ovenall
201
101
101
m
10:4
10-2
10 6
10
10a
10 10 10-7
10-5 10 10-1
[Weight]
Ieciivationl
[Gradient]
Figure 20: Analysis of weight, activation, and gradient matrices (layer 1, Attention Key), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
18

---

(A) Numeric value distribution of singular vectors
Prequency
10a
10
102
101
101
5-2
10
10
10
10ma
100
10
10-1
10-2
10"
10-19
10
10
10
10
(B) Numeric value distribution of residual matrix 10"
Frequency
10
10P
10a
10a
Residual
10
Overall
101
10
10
10
10
10
10F
10
10
10
10"
10
10
[Welght]
|Activation|
[Gradlent]]
Figure 21: Analysis of weight, activation, and gradient matrices (layer 1, FFN dense2), showing that singular vectors are narrow and residuals 1 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
(A) Numeric value distribution of singular vectors
102
10
10a
Prequency
101
10
5-2
10.5
10-1
10T2
100
10
10
10
10
10-9
10
10
10
10-1
(D) Numeric value distribution of residual matrix
102
107
Frequency
10
10a
16
10a
Residual
Overail
101-
10
10
10a
10
10
10
10
(Welght]
|Activation
[Gradient]
Figure 22: Analysis of weight, activation, and gradient matrices (layer 32, Attention Key), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
(A) Numeric value distribution of singular vectors
10"
10a
10a
Frequency
102
10
5
10
10
10
10
10a
10
10
10
10a1
10-
10
10
10
10"
(B) Numeric value distribution of residual matrix 10"
Frequency
10
10a
102
10a
Residual
10
Overail
101-
10
10
10
10
10
10
10
10-10
10-7
10
10-1
10
10-7
10
10
10
[Weigh!]
|ccivationn]
[Gradient]
Figure 23: Analysis of weight activation, and gradient matrices (layer 32, FFN dense2), showing that singular vectors are narrow and residuals 1-2 orders of magnitude smaller, confirming wide ranges arise from dominant components amplified by large singular values.
A.3 NUMERICAL DISTRIBUTION COMPARISON OF HADAMARD AND METIS
We compare the effects of Hadamard transforms and Metis the element space. As shown in Fig, 24 the Hadamard transform redistributes only small fraction of outliers, modestly smoothing the tails but leaving the overall distribution wide and still misaligned with FP4's narrow representable range. In contrast, Metis applies spectral decomposition separate dominant singular directions and values;
19

---

the residual matrix after removing these components exhibits substantially narrower distribution, inherently more compatible with low-bit quantization.
Numerical Distribution Originel, Hedamard, and Residual Netrices
Orignall
Figure 24; Element-wise distributions under diffcrent preprocessing strategies: (i) original tensor, (ii) Hadamard transform, and (iii) Metis spectral decomposition. Hadamard smooths few outliers but leaves a wide spread, while Metis isolates dominant components and produces a narrower residual distribution well suited for FP4 quantization.
x2
we* 20t 1e
B METHOD
B.
SPARSE RANDOM SAMPLING SUBSPACE APPROXIMATION
This section provides results on additional modules, further supporting the findings in the Methods section and showing that the dominant subspace of large batch can be efficiently approximated using only subset of samples.
1.0
1.0
Dominent rabagare
Dominant sullegace
01.9
0.3
0.8
0.8
0.7
0.7
0.8
0.6
1%
10%
100%
15
10%
100%
Sample rallo
Sample ralio
(b)
Figure 25: Subspace alignment between the dominant subspace of the full batch and that of randomly sampled subsets of sequences. (a) The input activation of W& module of layer 32. (b) The input activation of FFN module of layer 32. Alignment quickly saturates as the sample ratio increases, with just 1 % of sequences achieving nearly 0.9 alignment with the full-batch subspace.
B.2 OPERATOR- LEVEL OPTIMIZATION OF SPECTRAL DECOMPOSITION.
To further reduce the practical overhead of repeated randomized SVD, we replace the generic torch. svd lowr ank path with specialized two-stage pipelined eig SVD implementation tai- lored to our training setting (CUDA, small target rank, and Riteer = 1). Instead of performing full SVD on the compressed matrix = Q A, we form the small Gram matrix G BBT E Roxq, compute its cigendecomposition G Ue diag(5²)U and then recover the singular vectors
= QUei
V = =BBu U. diag(S-1).
This converts the final factorization into a 9 x q eigensolve, significantly reducing the cost of the decomposition tail.
20

---

Table 5: Runtime breakdown of Metis quantization. Overhead is measured relative the NVFP4 backward baseline.
Operation
Time (ms)
Overhead vs. NVFP4
NVFP4 backward (baseline)
5.43
0%
BF16 backward
9.35
72.19%
Spectral step components
Spectral decomposition (random sampling)
0.732
13.48%
GeMM of US
0.587
10.81%
GeMM of USV
0.253
4.66%
Quantization of low-rank branch
0.359
6.61%
Metis quantization
1.931
35.57%
Amortized cost (1/8 spectral frequency)
Metis quantization (amortized)
0.241
4.45%
The implementation is organized into front graph and back graph. The front graph performs randomized range finding and basis construction, including Y = AR, one power iteration, and a Cholesky- -RR orthogonalization, and then builds and G. The back graph reconstructs U and V from the eigensystem. To reduce launch and synchronization overhead, both stages are captured with CUDA Graphs and reuse persistent device buffers. In addition, the small matrix copied to pinned host memory, and its cigendecomposition is executed through a dedicated CPU worker thread. This design moves the small eigensolve off the main GPU factorization path and exposes a pipelined interface for overlapping the CPU-side eigensolve with surrounding work when needed. We further cache the graph executor by shape, device, and dtype, avoiding repeated memory allocation and graph recapture across iterations. Together, these optimizations substantially reduce the wall-clock overhead of the spectral decomposition operator Metis.
B.3 RUNTIME EVALUATION
We evaluate the kernel-level wall-clock overhead of Metis NVIDIA B300 GPU. Compared to the NVFP4 baseline, Metis introduces an overall runtime overhead of approximately 10.12% This additional cost comes from two sources: (1) spectral decomposition- based quantization, which contributes 4.45% amortized overhead, and (2) the extra low-rank matrix multiplications introduced by the spectral branch, which contribute 5.67% overhead.
Setup and Overall Breakdown. To analyze the runtime overhead in a representative setting, we benchmark the backward pass of linear module with weight shape [2048, 2048] and input shape [256, 2048, 2048], which matches typical matrix sizes in practical LLM training. A single Metis quantization step incurs additional 1.93 ms over the NVFP4 backward baseline due to spectral decomposition, low-rank quantization, and the associated low-rank GeMMs.
Amortized Spectral Decomposition Cost. The decomposition overhead can be substantially reduced through temporal reuse. With the Sparsely Random Sampling strategy, the dominant low- rank spectral components
be reused across tradient-accumulan subspace estimated from only 1% of the data already achieves approximately 0.9 similarity to the full-data subspace (Fig. 4), we reuse interval of 8 steps without observable accuracy degradation. As a result, the deconposition--elltee overhead amortized from 1.93 ms per spectral update to 0.24 per training step, corresponding to only 4.45% overhead relative to NVFP4.
steps. Since the dominant
Low-Rank Branch Overhead and Kernel Fusion. Beyond the decomposition itself, Metis also introduces extra low-rank matrix multiplications. To reduce their runtime impact, we design kemel- fusion strategy that minimizes additional GMEM traffic from the low-rank branch. Specifically, we: (1) load the input tile X from GMEM into SMEM only once; (2) load the corresponding low-rank factor tiles together with the residual branch tiles into SMEM in the same stage; and (3) perform the fused computation entirely in SMEM, accumulate partial results locally, and write the final
21

---

Table 6: Overhead breakdown of the fused low-rank branch.
Overhead SMEM writes
Operation
Low-rank parameter reads
0.74%
0.62%
Computation
4.30%
Others
0.01 %
Overall overhead
5.67%
output back to GMEM only once. This avoids redundant global-memory reads and writes that would otherwise dominate the cost of the small low-rank branch.
After fusion, the extra low-rank matrix multiplications contribute only 5.67% total overhead. As shown in T most of this cost comes from computation itself, while memory traffic and other auxiliary costs remain minor.
Overall, these results show that the additional cost of Metis remains modest in practice. By combining sparse random sampling, temporal reuse of spectral components, and kernel fusion for the low-rank branch, Metis keeps the total runtime overhead to 10.12% while preserving the training benefits of spectral decomposition- quantization.
C EXPERIMENTS
C.I IMPLEMENTATION OF NVEP4 AND NVIDIA'S RECIPE
In our implementation, the NVFP4 format is simulated by explicitly casting tensors to FP4 before invoking low-precision operators and subsequently restoring them to higher precision when passing results subsequent high-precision operators. Concretely, the quantization function maps the scaled values of each block into the discrete representable set [00, 0.5, 1.5, 2, 4, 6]. This process ensures that the simulated tensor values adhere the constraints of the FP4 e2m1 format, thereby capturing the representational limitations of the hardware specification. By alternating between quantized (FP4) and restored (higher-predd (recisr-recision) states, the simulation reproduces the effective numerical behavior of NVFP4 operators while remaining compatible with standard high-precision computational routines.
Rationale The soundness of this simulation stems from the design of FP4 hardware multiply units, which typically retain extra exponent headroom when computing intermediate products. This architectural property guarantees that low-precision multiplication does not incur overflow, even though the operands are quantized. Consequently, performing multiplications in higher precision faithfully reflects the outcome of true FP4 multipliers, since additional rounding error overflow introduced in the product stage. Following the multiplication, accumulation is conducted in 6 6 (bf16) precision, which aligns with hardware practice and preserves numerical consistency with higher-precision accumulation. Taken together, these considerations indicate that the proposed simulation of NVFP4 matrix multiplications both practical and theoretically well-justified.
C.2 SENSITIVITY ANALYSIS OF RANK
Sensitivity analysis of spectral decomposition rank ranging from 1.5% to 12.5%. The curves for 1.5% and 12.5% closely match, indicating that rank of 1.5% sufficient maintain performance.
C.3 ISOTROPY OF THE RESIDUAL BRANCH
We inspect the singular spectrum of residual matrix from GPT-2 trained with Metis to examine whether anisotropy re-emerges, using a baseline-traiie trained matrix for comparison. As shown below, the baseline matrix exhibits strong anisotropy, with few singular values dominating the spectrum, whereas the residual shows a much flatter distribution, indicating that anisotropy is effectively addressed by the low-rank branch in Metis.
22

---

k=16 (%1.5)
5.0
k=32 (%3.1)
k=64 (%6.3)
k= 28 (%12.5)
4.5
Train Loss
3.2
4.0
3.0
3.5
98B
99B
100B
3.0
0B
20B
40B
60B
80B
100B
Token Processed
Figure 26: Training loss curves of GPT-2 1.1B using different ranks in spectral decomposition, showing that rank of 1.5% is sufficient maintain performance.
16
Baseline
Residual
14
12
Singular val
4-
2-
200
400
600
800
1000
1200
Index
(a)
Figure 27: Inspection of residual anisotropy in Metis compared with baseline-trained matrix (layer 16. FFN densel, GPT-2 I.) While the baseline matrix exhibits strong anisotropy, the residual spectrum shows much flatter distribution, indicating that anisotropy is effectively addressed by the low-rank branch Metis.
23


Reviews:
[{"review_id": "5xEn0ddXoU", "reviewer": "Reviewer_mj9Y", "summary": "The paper tackles the core problem of why end-to-end quantized (specifically FP4) training of LLMs underperforms. Quantization plus strongly anisotropic tensors (weights, activations, and gradients) leads to disproportionate distortion of the small singular directions. They propose Metis, which does a cheap, sampled spectral decomposition per step, splits tensors into a small low-rank branch and a quantization-friendly residual, and quantizes the resulting narrower parts separately. This gives an almost fully quantized setup (W4A4G4) and, on LLaMA-3 8B pretrained for 100B tokens, the method stays close to BF16 with modest benchmark drops. Experiments are quite strong (8B training, meaningful ablations), and the intuition is clear and well-argued.", "questions": "1. You mention that Metis adds only modest complexity due to low-rank factors and sampled decomposition, but there is no end-to-end timing. Can you provide actual wall-clock comparisons (forward, backward, and full step) against a plain NVFP4 training baseline on the same hardware/software stack? Given the available kernels.\n\n2. For the quantized GEMMs, how much of the extra time comes from (i) subspace estimation (sampling + projection), (ii) extra GEMM terms from the expanded $ (X_k + X_R)(W_k + W_R) $, and (iii) quantization/dequantization?\n\n3. Can you provide a comparison against strong recent PTQ/few-shot quantization methods (QTIP, Quip#, AQLM, OmniQuant) and to fully-quantized training lines such as rotation-based FP4 (“FP4 All the Way”, HALO-like)\n\n4. For inference, do we still have to run the spectral decomposition / special activation quantization, or can the model be exported to a simpler FP4/PTQ format? How much would the model lose if done so?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper correctly identifies spectral anisotropy (few large singular values + many small ones) as the real reason FP4 fails, and the “separate big directions, then quantize” story is both simple and compelling.\n\n2. Using sequence subsampling and randomized projection to estimate the dominant subspace is a sensible way to make SVD affordable; even if the theoretical bounds are loose, the justification is acceptable for practical use.\n\n3. 8B pretraining at 100B tokens, even if not at full budget, is a significant scale for a quantization paper, and much stronger than GPT-2-only demonstrations.\n\n4. Doing weights, activations, and gradients almost FP4 is a decent advance over weight-only or weight+activation.\n\n5. The ablation that removes spectral decomposition from W/A/G and shows that gradients matter the most is especially convincing and supports their main diagnostic.\n\n6. Modest but consistent benchmark results. The reported downstream gaps are small but believable given the setup.", "weaknesses": "1. There is no actual wall-clock or kernel-level overhead comparison, despite the fact that FP4 kernels (NVFP4-style) are already available in systems like QuTLASS and vLLM. Even an FP4 (not NVFP4) timing would have been possible, as it has similar FLOPS properties to NVFP4.\n\n\n2. At inference time, you still need the fancy activation-side spectral quantization. That limits how much speedup and how much deployment simplicity you actually gain, and it’s unclear how this scales.\n\n\n3. The experiments are single-seeded, and no CI is provided. Especially for smaller models (GPT-2 scale), results should be multi-seeded; the reported downstream gaps are within typical variance.\n\n\n4. Benchmarks are modest. Stronger discriminative tasks, such as MMLU, GSM8K, or other more challenging evaluations, would better reveal the gains from this method.\n\n\n5. No comparison to recent strong PTQ like QTIP, Quip#, AQLM, OmniQuant. \n\n6. No comparison to recent FQT methods, such as “FP4 All the Way” or rotation-based FP4 training (like “HALO” in NVFP4). So it’s hard to situate Metis among existing efforts.\n\n\n7. The paper says Hadamard incurs notable overhead and doesn’t really fix the distribution, but fast Hadamard is $ O(n \\log n) $, and HALO/HALO-like methods do notably reduce kurtosis to a Gaussian level (see HALO §A.4).\n\n\n8. The method is algebraically described but not presented as a concrete algorithm.\n\n\n9. It would have been more informative to include validation loss curves instead of or along the training loss curve.\n\n\n10. The decomposition bounds presented are quite loose; that’s acceptable as motivation, but it weakens the theory-backed framing.\n\n\n11. In Fig. 1, studying relative error per singular value is not the best target, since what matters is the matrix-level reconstruction. Although Fig. 3 partially fixes this.\n\n\n12. Fig. 4 “sample ratio” is underspecified. Using a percentage of sequences is a vague measure because it depends on sequence length; an $ l_k/k $ or $ l_k/m $ would be clearer.\n\n\n13. Looks close to LQ-LoRA + QAT but generalized. Conceptually, the method identifies the important subspace and quantizes the rest, similar to low-rank+quantization approaches like LQ-LoRA but with SVD approximation and STE-QAT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OiAy6WPbA8", "reviewer": "Reviewer_eipG", "summary": "The paper begins by studying the spectral space of weights, activations, and gradients. The findings show a clear \"anisotropy\" in the singular distribution, which causes wide numerical ranges, hence high low-bit quantization errors. METIS is then designed to 1) decompose the matrices into a low-rank component with large singular values, and a residual component with smaller singular values, 2) do so efficiently using sparse random sampling and random projection, and 3) quantize each component separately. The results show that NVFP4 METIS is able to closely recover BF16 baseline across GPT/Llama models, and outperform naive NVFP4 baseline.", "questions": "1. Could the authors provide real-world speedup number? I understand the rebuttal timeframe is too short for writing kernels, but from my point of view, this is critical. The authors can maybe take advantage of the kernels in [2].\n\n2. Several times in the paper it was mentioned that gradients are being quantized. I think this is ambiguous, as it could mean both activation gradients and weight gradients (I believe the authors mean the former). Can you please make this clear?\n\n3. The authors mention that l >> k (around line 222). What will happen if that's not the case? Note that this actually happens quite often, especially in research (with gradient accumulation).\n\n[2] https://github.com/IST-DASLab/qutlass \n\nIn general, my biggest concern regarding this paper is about the runtime analysis. If my concern is addressed, I will consider increasing my score, possibly significantly. But I don't think the paper is complete without it.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. I believe METIS is generally a very strong paper on the analysis side, and some of their findings are novel and interesting.\n\n2. Very clear presentation.\n\n3. Clean and clear ablation studies, including showing that the residual term in the weights does not \"develop\" large singular values.", "weaknesses": "1. The asymptotic runtime analysis, in the era of Blackwell GPUs, is not convincing at all, especially given all the memory-bound operations (e.g., multiple quantization calls, etc). On the other hand, the authors claim that NVIDIA's FP4 stack was not released at the time of submission, but cutlass already supported NVFP4 matrix multiplication in April 2025 [1]. I think speedup numbers, including layer-wise and end-to-end, are crucial for this paper.\n\n2. Some minor concerns exists, see questions.\n\n[1] https://github.com/NVIDIA/cutlass/releases/tag/v3.9.0", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvL8UVvVOa", "reviewer": "Reviewer_kaek", "summary": "This paper introduces Metis, a spectral-domain quantization framework designed to enable robust 4-bit (FP4) training for Large Language Models (LLMs). The authors identify anisotropy—the domination of parameter, activation, and gradient spectra by a few large singular values—as the fundamental barrier to low-bit training. Metis solves this by partitioning the anisotropic spectra into narrower sub-distributions, which are then quantized independently to reduce errors and preserve spectral structure. To make this computationally feasible, Metis uses sparse random sampling and random projection to reduce the cost of spectral decomposition.", "questions": "1. what is the meaning of   $\\bar{X} in Eq.(2) ,as well as other symbol with bar? This paper do not give definition for these symbols.\n2. Why NVFP4 training can achieve lower loss compared to BF16 training in GPT models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper provide comprehensive analysis to gradient, weight, activation based on SVD, reveaing that a few singular values dominating the spectrum.\n2. This paper demonstrate that absmax quantization favors larger values, and bring significantly bias to small values.\n3. The performance is impressive, achieving nearly lossless performance in NVFP4 training.", "weaknesses": "1. This paper focus on FP quantization, but the quantization illustrated in Lines 181 are INT quantization formulation.\n2. Two different \"X\" symbol in line 269 and line 270.\n3. The writing in section 3.2 is unclear, it should be emphasize which GeMM are execute in 4-bit and which GEMM are execute in BF16.\n4. Metis introduces too much additional operation, though the computational FLOPs is relatively small, the additional memory overhead or IO overhead are not negligible. This should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9knhfzgfXd", "reviewer": "Reviewer_vMTK", "summary": "This paper proposes Metis, a spectral-domain quantization framework for FP4 training of large language models. It identifies anisotropy in the singular value spectra of weights, activations, and gradients as a key barrier to low-bit training. Metis partitions these anisotropic spectra into narrower sub-distributions for independent quantization, greatly reducing distortion and preserving spectral structure. It introduces efficient decomposition techniques—sparse random sampling and random projection—to minimize overhead. Experiments on GPT-2 (130M, 1.1B) and LLaMA-3 (8B) show that Metis achieves only 0.4% higher loss and 0.1% accuracy degradation relative to BF16, outperforming NVIDIA’s FP4 recipe.", "questions": "Questions\n\nHow well does Metis scale to larger models (e.g., 70B or 100B)?\n\nWhat are the measured training throughput and memory savings compared with NVFP4 or FP8 baselines?\n\nWhat is the difference bitween the training method of Metis and BitNet-family (binary of tenary training)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Addresses a fundamental bottleneck (anisotropy) in ultra-low-bit LLM training.\n\nThe spectral-domain perspective is novel and well-motivated.\n\nTheoretical analysis and empirical evidence are solid and clearly explained.\n\nThe proposed techniques (sparse sampling, random projection) make FP4 training computationally feasible.\n\nDemonstrates strong results on multiple models, surpassing existing FP4 baselines.", "weaknesses": "Evaluation is limited to up to 8B models. Since anisotropy worsens with scale, validating on 70B+ models would better demonstrate scalability.\n\nEnd-to-end efficiency (e.g., wall-clock speedup, memory usage) is not quantitatively reported; the paper mainly provides theoretical overhead analysis.\n\nMore comparison with other recent FP4/low-bit training frameworks (e.g., HALO, SVDQuant) would strengthen the positioning.\n\nExperiments rely on FP4 simulation in BF16, not native FP4 hardware, which limits the practical verification of speed and stability claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)