Is Finer Better? The Limits of Microscaling Formats in Large Language Models
Authors: Andrea Fasoli
Monodeep Kar
Chi-Chun Liu
Swagath Venkataramani
Viji Srinivasan
Leland Chang
Naigang Wang
Venue: ICLR 2026 Poster (2026)
Keywords: microscaling
fine-grained
FP4
quantization
low-precision
llm
Primary area: foundation or frontier models, including LLMs
TL;DR: Naive microscaling formats hit their limits when block size is too small

Abstract: Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we reported the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.

IS FINER BETTER? THE LIMITS OF MICROSCALING FORMATS IN LARGE LANGUAGE MODELS
Andrea Fasoli
Monodeep Kar
Chi-Chun Liu
Swagath Venkataramani
Viji Srinivasan
Leland Chang
Naigang Wang
IBM Research, USA
Sandrea.faolle
monodeep... kaz, swagath. vennhearaaaaaaaaaaaaaaaaa @ibma com
{eliu, viji, lelanda, (wang)@us rim. com
ABSTRACT
Microscaling data formats leverage per-block tensor quantization enable ag- gressive model compression with limited loss in accuracy. Unlocking their poten- tial for efficient training and inference necessitates hardware-friendly implemen- tations that handle matrix multiplications in native format and adopt efficient aror-mitigation strategies. Herein, we report the emergence of a surprising be- havior associated with microscaling quantization, whereas the output of quan- tized model degrades as block size decreased below given threshold. This behavior clashes with the expectation that smaller block size should allow for a better representation the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error be- hind it. Experimentally, we analyze the distributions several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we pro- for-
pose the use of FP8 unsigned E5M3 (UESM3) as a novel hardware- for the scales in FP4 microscaling data types. We demonstrate that UESM3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.
INTRODUCTION
The unprecedented growth of large language models (LLMs) has brought dramatic improvements in natural language processing, but at the expense of escalating compute, memory, and energy de- mands (Kaplan et al. (2020) [Hoffmann et al. (2022) Samsi et al. (2023). )) With model sizes reaching hundreds billions of parameters and context windows extending to hundreds of thousands of to- kens, reducing numerical precision has emerged
cornerstone for enabling efficient training and inference (Gupta et al. (2015) Kuzmin et al. (2022) Xiao et al. (2023)). Hardware vendors have progressively shifted from FPI6 to FP8, and now FP4, in order increase throughput and en- ergy efficiency. Yet, pushing precision below 8 bits often leads to substantial accuracy degradation, particularly when both weights and activations are quantized (Dettmers & Zettlemoyer (2023 Ashk
boos et al. (2024) Li et al. (2025)).
Over the past few years, quantization techniques for LLMs have steadily evolved towards finer- grained control in order to balance accuracy and efficiency. Early approaches primarily employed tensor-wide quantization, assigning a single scale factor to an entire weight or activation tensor. However, this often introduced large quantization errors in regions with high dynamic range (Lin) et al. (2020), Dettmers et al.] (2022)). To address this limitation, per-channel and per-token quanti- zation were introduced, assigning independent scales to individual output channels tokens, and thereby achieving significant accuracy improvements at modest storage and compute costs (Dai et al.
2021 Yao et al. (2022)). More recently, per-group quantization has gained traction, wherein small

---

groups of elements (e.g., 128) share scaling factors, offering compromise between reduced quan- tization error and the increased overhead due to the proliferation of quantization parameters (Shen
et al. 2020) Frantar et al. (2022) Xiao et al. (2023)).
Microscaling formats (Rouhani et al. 2023al) push this trend towards even finer quantization reso- lutions: by means of efficient hardware implementations coupled with scaling factor quantization, these formats enable grouping elements in smaller blocks, while limiting data transfer overheads. The benefits of this strategy has motivated rapid industry adoption, with microscaling formats now supported in commercial Al accelerators (NVIDIA 2025a); AMD (2025). Despite these successes, the robustmess of microscaling, particularly sub-8 8-btt precision, remains an open question that still subject of investigations due to the recentness of its introduction.
In light of this trend from coarse to highly localized quantization, whereas aggressive quantization of progressively smaller blocks is being pursued, herein we examine the dynamics of quantization errors across a range of block sizes, uncover potential pitfall emerging in existing formats and exacerbated by smaller blocks, and propose an efficient hardware design solution to avoid it.
Our contributions are threefold:
Discovery and analysis of quantization anomaly microscaling formats, whereas de- creasing block size paradoxically increases quantization error.
A theoretical framework that decouples sources of quantization emors and explains their interaction with LLM tensor statistics, allowing us to pinpoint the origin the anomalous behavior to the quantization of the microscaling scales which hinders the representation of low magnitude blocks. The framework applicability extends beyond the case of FP4 elements and FP8 unsigned E4M3 scales.
A friendly- design proposals to support FP8 unsigned E5M3 scales effectively mitigate these issues at minimal hardware cost.
2 BACKGROUND
2.1 MICROSCALING FORMATS
Microscaling, as defined by the Open Compute Project (OCP, Rouhani et al. (2023a)), refers to group of numbers sharing a scale and represented as elements with lower precision. The original OCP specification requires elements in either FP4, FP6, INT8, or FP8 format, block size 32, and scales as E8MO, i.e., a biased Power-of-Two (PoT) scale, covering a very wide dynamic range, 2 2-1 to 2 with limited precision. Since its conception, multiple commercial vendors have added native hardware support for MXFP8 and MXFP4 formats (NVIDIA) (2025a); AMD (2025). ). In addition, NVIDIA GPUs with Compute Capability 10.0+ support NVFP4, which refers 16 FP4 elements sharing an FP8 unsigned E4M3 (UE4M3) scale (4 exponent bits, 3 mantissa bits). The selection of a floating point scale in lieu of PoT is motivated by accuracy considerations (Lee al. (2024): NVIDIA (2025b)), making NVFP4 a strong contender as quantization format of choice for microscaling FP4.
In a typical formulation, microscaling quantization is applied to weights and activations by parti- tioning each tensor in blocks of size N. For each block j with elements 2,(5) scaling factor s(i) derived as s(i) Quatte (xriaa/( /C). where xii) s=1....N
[x:""] Qseate defines the scales quan- tization, and C a constant. Each element is normalized by its block scale and mapped into the nearest discrete level g. Qetem (2.5)//(6) where Qeiem defines the elements quantization
mapping. Values can be reconstructed (i.e., dequantized) by rescaling: a,(5) s(() g.)
2.2 RECENT WORK ON MICROSCALING
A Power-of- scale simplifies the arithmetic complexity of hardware implementations, but sig- nificantly degrades accuracy when applied to FP4 elements (Rouhani et al. (0023))) Recent works have addressed this issue by proposing techniques that can largely be classified enhancing the
2

---

[8]
(b)
4.0
+g-ante-33300
gionite 3.3 88
+-llmm-3.11-i 1-8b
3.1-8b
-rostral-B77-iistruus
+llamm-3-7h
Uoro 2.7b
2.0
perplexity
block size
block size
Figure 1: (a) FP4 microscaling quantization using BF16 scales. (b) Impact of FP8 UE4M3 scales. Anomalous data points showcasing perplexity inversion have been circled.
element range for better outliers representation, or improving the scale precision. Lee et al. (2024)
asymmetric FP8 E5M2 scales mitigate the impact of outliers in 32-element blocks. Chen
et al. (2025) propose BitMoD, where the redundant zero representation in the elements is exploited
for additional quantization level. BlockDialect (Jang Tambe (2025) ) uses a 16-way codebook where the two largest elements in block can be set to 16 configurable pairs. Lo et al. (2024) present Nanoscale floating point, where both the scale format and the element format are adapted based on the distribution of the elements in block. Collectively, these approaches demonstrate broader trend toward hardware-efficient quantization, where fixea-format representations, block- level adaptability, and fine-grained floating-poit point encoding jointly push the limits of low-bit infer- ence while preserving model quality.
MICROSCALING WITH FINER GRANULARITY
3.1 EXPECTED ERROR DEPENDENCY BLOCK SIZE
We begin our investigation by monitoring the difference in perplexity (herein, perplexity gap) be- tween models quantized with FP4 microscaling formats and their 16-bit precision baseline, as function of block size. Fig. T shows the perplexity gap upon FP4 quantization when the scaling factors of each block are kept as BF16, thus not quantized. Notably, all models follow similar trend: block size is reduced from 256 8, the perplexity gap decreases monotonically. This the generally expected behavior: reducing block size allows for finer-grain representation of the quantized elements, hence lower quantization error.
To strengthen this intuition, consider for example the quantization of block i of N elements ()) (f) is derived from the block maximum Tmax (as per Sec, 2.1). the elements in this block are instead quantized into B separate blocks of size L = N/B, each sub-block is
single scale $o
......, B. Therefore, every element belonging to a sub-block where < $n is quantized using scale which, unlike s by definition does not exceed the maximum of the sub-block. The smaller scale would typically, although not strictly, enable better representation of the sub-block elements, resulting in lower error.
assigned scale sb I (3) for
The behavior showcased in Fig- a underpins the motivation drive per-block quantization to- wards ever smaller blocks. However, reduced block size comes at cost, associated with the in- creased number of quantization scales, which limits the gains memory storage and data transfer bandwidth provided by the compressed representation. Specifically, memory requirements for a for- mat with N 4-bit elements per block and 16-bit scales, are 1/2 + 2/N bytes and every halving of block size increases storage by factor of 4/(N + 4). Moreover, for scale format with M bits of mantissa (including the implied 1), the complexity of multiplication in a microscaling multiply- and-accumulate operation grows by M K where K is the width of the partial sum. Therefore,
3

---

[c)
5.
granite3.3--bb
heed granthe-3 3-4h
beeB grantte be Lame--77
bumid grantted
ba-18 gran
0.0
12
Iu lamma--77
bam18 lame-d-7a
MSE (per tentori
bloke
MSE (per
0.4
scales FP8 E4M3
scales BF16
00.
0.4
9/8 12 85
85
(per-tansor)
1e-2
(per-tansor)
le-2
Figure 2: (a) Per-block MSE of the first Query weight tensor, block size (bs) 8 16. (b) Per- tensor MSE vs standard deviation a of each weight tensor of granite-3.3-8b and llama-2-7b. Tensors quantized as FP4 using FP8 UF4M3 scales, with bs 8 or 16. (c) MSE vs using BF16 scales instead.
irrespective of the block size, the arithmetic complexity of handling 16-bit scale (BF16 or FP16) increases the hardware complexity, compared to 8-bit scale. Due to these limitations, 8-bit scales have been the de-facto standard since the introduction FP4 microscaling formats. Scales quanti- zation FP8 UE4M3, however, brings about unexpected behaviors hinting at fundamental limit to the block size reduction trend.
3.2 ANOMALOUS ERROR DEPENDENCY ON BLOCK SIZE
Figure bb) shows the relation between perplexity gap and microscaling block size using FP8 E4M3 scales. Going from large to small values of block size, a monotonic decrease in perplexity gap is initially observed, as in Fig. Ta), in line with the general expectation outlined in the previous Sec- tion. Surprisingly, Fig. [ b)) shows that a further decrease in block size can lead to an increased perplexity gap. We refer to this behavior as perplexity inversion, or simply inversion. Perplexity version is model dependent: gramite-3.3-2 shows clear upswing at block size 16; Ilama-3.1 3-bb and mixtral-8x7b ---ittrrttt present it at block size on the other hand, llama-2-77 has strictly mono- tonic dependence, with no inversion down to block size 8. Inversion (or lack thereof) in additional model architectures presented in Appendix A
Although perplexity evaluates performance at the model level, the appearance of perplexity inversion may suggest an increase the quantization error of each individual weight tensor. To verify this hypothesis, we quantize the same tensor twice with different block sizes, and compute the Mean Squared Error (MSE) in terms of the larger block enable direct block-to-block comparison. The expectation is that blocks quantized using finer granularity would predominantly show lower MSE. Fig. 2 a) shows the 2D density plots for one weight tensor of gramite-3.3 3.3-,, quantized with FP4 elements and FP8 UE4M3 scales using block size 8 and 16. Data points on the diagonal correspond to blocks having the same MSE both conditions. Strikingly, a large number of data points, about 25% in Fig. aa)). sit above the diagonal, indication that larger quantization error smaller block size is a frequent occurrence. As the magnitude of the weights never exceeds 1.0 in granite-3.3-8hh this behavior cannot be driven by large outliers truncation. The pattern seen in Fig, 2
a)) is consistent
across weight tensors and models, shown in Appendix B These results provide experimental validation the counterintuitive hypothesis that quantization error per-block can often increase as block size decreases.
To understand the root causes for the error increase, and why it differently expressed by different models, in Figs. (bbcc) we look at the MSE of each weight tensor, computed on per-tensor basis under a given quantization scheme, as a function of the standard deviation a of the same tensor, pre-quantization. In Fig. b). quantize with FP4 elements and FP8 UE4M3 scales all weights tensors of granite-3.3-m common between the two models. In Appendix C demonstrate that this behavior is maintained across a variety of models, from attention -aasntio----eddd LLM, to State State-Space Models (SSM), to hybrid SSM. Hence, range of weight distributions, potentially with different tails and outliers incidence,
and 1lama-2-7b. The plot shows a distinct dependence of MSE on σ. in
4

---

express the same dependence. We emphasize that relation between MSE and ø is not necessarily surprising: a larger a is representative of distribution with larger weight magnitudes, and given that the maximum block value determines the normalization factor, it is conceivable that non-maximum elements could incur higher error for large maximum. In Sec. 4.3 we explore this relation further.
Irrespective of general dependence, striking feature emerges from Fig. 2 2bb when comparing block size 8 to 16: the two curves overlap, with a crossover at a = 2 10 10-2. Under this em-
pirical threshold, the MSE of weights quantized with block size 8 is larger than their block size 16 counterpart. Granite-3.3-8b, which showed pronounced inversion in Fig. [b). is characterized by having most weights below this threshold. On the contrary, 22-ma-2-7b, which showed no inversion in Fig. [b), has large fraction of weights above = 2 10 1002 The crucial role played by the quantized scales in this phenomenon is confirmed in Fig. - which shows the same curves for non-quantized scales. A monotonic dependence of MSE vs a is still observed, but the error associ- ated with block size 16 quantization consistently larger than the error using block size 8. These observations provide first set of insights helping us understand the model-dependent perplexity inversion shown Fig. [b) and guiding towards a solution: models presenting narrow weight dis- tributions quantized using microscaling formats are prone to larger quantization errors when finer granularity is employed. This behavior is driven by the quantization of the scaling factors.
4
THEORETICAL FRAMEWORK
4.1 MSE OF IDEAL DISTRIBUTIONS
18)
(0)
[o]
Bame-277
vorrilaut,
11
carro 70
logition, 13
block size
qurrbcko. 8)
block size
block size
2
2
20
0 (ppertenzor)
/ppc-to (561)
0 01per-t2naa
Figure 3: (a) MSE a dependency for weights of 3 pre-trained models (dots) against Normal dis- tribution (black line). (b) Behavior of several ideal distributions. (c) Normal distribution compared theoretical results, and decomposition of contributions to the theoretical error.
To elucidate the mechanisms by which scales quantization drives up the error at smaller block sizes, investigate the phenomenon from first principles. As first step towards setting up theoretical framework, we establish what distribution is appropriate to model. To this extent, we recreate the MSE vs a plot using tensors randomly drawn from various ideal distributions. Tensors are quantized using FP4 elements and FP8 UE4M3 scales. Fig. 3(a demonstrates an excellent agreement between experimental data from real models and the curve associated with a Normal distribution with mean 0 and variable a. At the lowest end of the range of a, previously unexplored, for < 0.5 10-2 new feature emerges: as σ decreases, the MSE increases dramatically, before dropping again towards zero. To verify this behavior arises in pretrained models as well, we also plot the distribution of weights of mamba-codestral-75 in Fig. (() (yellow dots), which is especially narrow compared to other models. These data also agree with the results from the Normal distribution.
Fig. 3(b) shows the results across multiple ideal distributions. We sweep a by drawing elements with a given distribution and scaling them in magnitude using range of constants. The parameters determining the shape of each distribution were chosen arbitrarily, aiming at spanning a similar range of given the same range of scaling factors applied to the drawn tensors. The shape of each distribution (prior scaling) is presented in Appendix D showing a variety of tail behaviors. Fig. 3(b) highlights that all distributions show qualitatively similar trends, but some deviation from
5

---

the Normal distribution curve is also present. Hence, although Normal distribution appears to be good approximation of the behavior of pretrained model weights, significant deviations can results in some scattering around the main trend line. The pattern is consistent across block sizes, as shown in Appendix D Given these observations, we will use a Normal distribution to theoretically model the main trends of the MSE
4.2 MICROSCALING FP4 WITH NON-QUANTIZED SCALES
We begin by analyzing the case of microscaling quantization using non-quantized scaling factors. We examine this simplified scenario to introduce key concepts and methodologies that will serve as foundation to the more complex case of FP8 UF4M3 scales. We stress that although herein we are focusing on FP4 E2M1 quantization, the framework is readily adaptable to reproduce the behavior of other quantization formats, both floating point and integer. We present key details of this process below, and provide the full derivations in Appendix
Consider random variable X that follows a Normal distribution with mean μ 0 and standard deviation σ. We draw elements 2; from and define Zmax = max N. |z,|-- We introduce a second variable Y, representing scaled version of X, where the scaling factor depends Tmax X/s 6ax/mmxx The factor is the maximum representable value in the FP4 E2MI format.
The Probability Density Function (PDF) of Y conditioned on Fmax is:
Ø (ay)
fly (mmax)= 2 (6a)
- for y E [-6, 6]
with ø being the PDF of the standard Normal distribution (p = 0, a = 1), 4b its Cumulative Density Function (CDF), and Xmaxx
s/o.
To derive the MSE conditioned on Tmaxx we consider each FP4 E2M1 quantization level Øj with Voronoi boundaries [aj, by), and compute the conditional MSE of Y for the bin j:
MSEy,j (gj [Tmax)
(y 93)² f(y [maxx) dy
a
To obtain the MSE as used in Fig. bb) and Fig. ((a.,), we need to express MSEy,; with respect to the random variable Z 8. QrP4 ). which described the elements distribution after discretization Opp and dequantization. This error, still conditioned on Faax is:
0²
-
| (u - )30)² b(u) du
(ey(a)
MSEz, (g) (maax)
(3)
24(6aa)
N
vy((a)
with u = oz,, wy aja, and Uy = bja. In this equation, all terms in a contain the Tmax condition- ality. To remove the conditioning and obtain the total MSEz, we compute the expected value with respect to mmaxi
MSEz
E Exgan
Σ MSEZ,s (gi Imax)
MSEZ., (gj Imax)) Samas (x) dx
(4)
where Sz is the PDF of Tiaax derived the derivative of the CDF of N i.i.d. variables drawn from a half-Normal distribution:
2N
N-1
Famay (0)
2Φ
(5)
We combine eq. 3 and integrate numerically over Tmax while varying a, to obtain an MSE vs theoretically-derive from first principles, when scales are not quantized. As shown in Fig. TO (Appendix D.. the agreement with the experimental results from an ideal distribution, and hence with the experimental data from pretrained models, is extremely good (x2 ~ 2 10 9).
4.3 MICROSCALING FP4 WITH FP8 UE4M3 SCALES
We now introduce into the framework the quantization of the scales as FP8 UE4M3. Full derivation provided in Appendix This condition breaks several previous assumptions. First, the scales

---

discretization requires to compute all the contributions to the emror of each possible scale value, weighted by the respective probability mass. Second, is no longer the case that 2; = 2max has zero enror. Third, will have to separately account for the edge case where all the values within block are rounded to zero.
When 2; # Zmaa and + the the dequantized variable Z can be expressed as:
Σ pk (SE = - _ fs(s) ds MMSEZA,00) |
MSE
(6)
Here, the index i refers to elements of block, j to the quantization levels the elements, and k to the quantization levels of the scales. pk is the probability mass of scale quantization bin, ag and by its Voronoi boundaries, (s) the PDF of the random variable S associated to the seales. The error of quantized element MSEz, has similar expression eq. but conditioned on the scale 85 instead of Tmax-.
When æ; Tmaz and + 0, the directly as:
single element at given scale Sy be computed
Efr. #uox (z, 8k)
QFP4
8k
and incorporated in the computation of the MSE by summing over all possible & quantization levels of the non-zero scales:
MSE 7. Fmas
Σ I
Emr, Tmax (x,, 8k)) SSmma (x) dx
(8)
N
6a
The third contribution to the MSE relates to the lowest FP8 scale bin, associated to 8 = 0, for which eqs. 66 and g are ill-defined. In round-to-neare process this is the interval [0, Smim where smin is the lowest non-zero representable FP8 value (subnormal S-0000-001, of values of generates a scale within this bin, all values in the block are rounded to zero. This error is then:
)) If the maximum
MSE
P(s -E [X²
Tmax
Smin
E [x²
||X|
38min]
The total MSE is the sum of these 3 separate contributions:
MSEz MSEZ,a,ZZmax + MSF ZS MSF
(10)
and can be integrated numerically, in the non-quantized scenario.
Fig. 3(c) compares the estimates for MSE obtained from the theoretical framework (thick black line) against the experimental data derived from Normal distribution (red dots), under FP4 mi- croscaling quantization with FP8 UE4M3 scales. The curves are found to be in perfect agree- ment (x2 4 10-8) with the experimental data derived from Normal distribution (as per Sec. 4.1.. across the full interval of a, providing strong validation of the theoretical model. Fig. [T in Appendix F further supports this conclusion by demonstrating that the agreement with the ex- perimental data carries across block sizes and leads to the emergence of crossovers between curves. Fig. 13 in Appendix G demonstrates that the applicability of our theoretical formulation extends across data types, as the modeling of INT4 microscaling quantization also shows very close agree- ment (x2 1.3 10 6) with the experimental data. Similarly, modeling of microscaling FP4 with FP6 scales provides MSE estimates that are consistent with corresponding perplexity measurements (see Appendix HH. Hence, in the context of new data format exploration, this framework can play role in analyzing the impact on the quantization error of scaling down precision, to sub-4-bit element formats, sub-8-bit scales, and smaller block sizes.
Fig. c() also plots the three separate contributions to MSE along with the total MSE. We find that for sufficiently large a, the eгror for all elements I; * Tmax dominates the total MSE. The main dependence of MSE dependence of MSEz , when I; + Tmaxx the quantization error each FP4 bin at a given scale $g, 0, seen in eq. and 6 However, narrower distributions the error on the representation of
Ø, first highlighted in Fig. 2(b), therefore due the superlinear

---

the maximum value of each block (z, Zmax). which is zero if scales not quantized, assumes larger relative importance, and can even dominate the total error. As expected, the smaller the block size, the more the weight of this factor (see Fig. [2 Appendix [F. Finally, at the lowest end of the range of a, the error entirely dominated by the zero rounding of all block elements. The relative importance of these three separate factors provides direct explanation of the dependence of MSE on block size (additional commentary in Appendix F.4
ERROR MITIGATION STRATEGIES
As demonstrated in Sec. the width weight distribution determines the relative importance of different contributions to the total MSE. In particular, narrow distributions appear to be unfavorable, with a large MSE compared to the magnitude of the elements. This behavior would materialize a large relative error and be detrimental to model accuracy. Similar considerations equally apply to activation distributions. Several solutions can be proposed to tackle this problem.
5.1 PER-TENSOR SCALING
A recent implementation of FP4 microscaling on GPU (NVIDIA 2025al) applies a per-tensor scaling factor to weight and activations, prior to the FP4 microscaling quantization step with FP8 UE4M3 scales. Such scaling extends the range of narrowly distributed tensors T using the combined maximum representable value of FP4 E2M1 and FP8 UR4M3:
8T max(E2MI) max maa (UE4M3)
(11)
max(abs(T))
The matmul output is then scaled back replacing the denominator of sT with estimate of the full precision matmul output, to account for the scaling of both the weight and activation tensors. As shown in Figs. Figs.{((.c) (b,c) and Table well Appendix []), this strategy perplexity and accuracy across a variety of models. However, it also rises two potential concerns. First, per-tensor scaling is susceptible to outliers: single element of large magnitude hinders the effective scaling of the whole distribution. In this context, when exploring microscaling FP4 applied to attention, in an effort to limit degradation, Zhang al. (2025) have already empirically resorted to pre-matmul per-channel scaling. Second, while the scaling factors associated to weights can generally be precomputed, scaling activations necessitates to either perform an on-the-fly absmax operation, increasing the computational cost of running the model, or to use pre-calibrated estimates, which may not reflect the actual distribution of the tensor being quantized and introduce errors in the computation. For these reasons, it would be desirable to develop mitigate error while not resorting to the use a global scale.
be effective in improving
ceedware--fienddl strategy to
5.2
FP8 UE5M3
(a)
Scale processing
(b)
granite-3.3-B0
(c)
Ilame-S.1--
RiDii exporiont
Unusod
scolo,
sapo hert
sighbe
8.0
mawa 1p4
partal product
7.5
2-11-9
Partal
Penisi Sumn Mantaea
Exponent
perpleaity
7.0
pemplauty
7.5
molization
0.5
2.0
FRALIEMNS 4E4M3
Scale generation
5.5
scole RPg UESMg
scole UE
6.5
SF16
FRALIESMS
5,0
BF16 oefrrenc
block size
block size
Figure 4: (a) Details of UE5M3 hardware implementation. (b.c) Perplexity vs block size using microscaling FP4 with FP8 UESM3 scales.
8

---

Table 1: Accuracy under the proposed FP4 microscaling quantization schemes, at block size 8.
Notation: UE4M3-S: per-tensor scaling + UE4M3; Hsw HellaSwag; Wng = Winogrande.
Model
Format
Wiki L
PIQA
Hsw
Wag
GSMBK
MMLU 1
granite-3_3-88 3-bbb
BH16
4.72
80.41
61.49
72.35
62.47
60.55
55.98
48.82
UF4M3-S
5.39
78.84
58.86
71.27
44.88
55.23
UESM3 (ours)
5.04
79.98
60.26
73.01
56.17
57.51
1aama-311 1.taa-.1-80
BF16
6.24
79.87
60.05
73.48
50.49
63.28
57.72
56.18
UE4M3-S S
6.76
78.84
58.71
72.61
43.21
60.99
UESM3 (ours)
6.79
78.84
58.94
72.14
42.15
60.97
nemotronn nano mano-9h-y 9h-v2
BF16
8.08
80.30
58.22
73.24
79.61
73.86
UE4M3
8.92
79.32
57.67
70.56
72.71
71.12
UE4M3-S
8.42
79.54
57,62
72.53
78.92
72.03
(ours)
57.57
bamba-99b-2
BF16
6.21
80. 96
62.18
73.95
12.15
64.96
UE4M3
21.25
76.44
45.90
37.85
2.65
36.36
UE4M3-S
6.53
80.41
61.64
71.90
40.41
63.80
(ours)
6.53
61.51
As mentioned in Sec. 2.1 existing hardware formats utilize unsigned FP8 scales (UE4M3). This choice allows the major hardware vendors leverage support for existing 8-bit FP8 formats, but leaves one scale bit unused. As an alternative to per-tensor scaling, we propose to repurpose this bit as an exponent and extend the FP8 scale range. The new scale format, unsigned E5M3 (UE5M3) offers greatly increased dynamic range compared to UE4M3, while maintaining its precision. In the context of narrow distributions, this allows better representation of the scale associated with blocks containing exclusively small magnitude elements: the minimum non-zero representable absolute value drops from 2-9 for UF4M3 to 2-17 for UE5M3. Naturally, the extended range would also provide a better representation of large outliers in any given block. An alternative repurposing for the unused bit leading to an UE4M4 format explored Appendix J but is less hardware friendly and found be less robust than UE5M3.
UE5M3 scale requires hardware modifications in two key areas: (1) within each microscaling instruction, where scales are fused with elements, and (2) in the quantization operation, where acti- vation outputs are quantized per block to generate scales and elements for the subsequent operation.
Scale Processing: Figure illustrates the typical floating-point processing steps in Al processors. The UESM3 format retains the mantissa processing logic of the scale unchanged, while extending the exponent logic by one additional bit. Since mantissa processing primarily determines hardware complexity, this modification introduces only a negligible increase in hardware cost.
Scale Generation: AI processors commonly support both E4M3 and E5M2 formats for standard FP8 quantization, and may reuse the same hardware features for generating scales in MX-FP4 quan- tization. To support E4M3 and E5M2, existing hardware must include logic cast an 8-bit exponent (from FP32 value) to either 4-bit or 5-bit exponent. Similarly, it must be capable of rounding a 23-bit mantissa to either 3-bit or 2-bit precision. Consequently, UESM3 scale generation can be achieved with minimal hardware changes.
Hardware Design: demonstrate the feasibility of the UE5M3 format hardware level, the solu- tion was incorporated into the design of systolic array processing engine with microarchitecture similar to that described Agrawal et al. (202T). Additional details and overhead estimates are provided in Appendix K
Fig. b.c) and Table show that applying FP8 UE5M3 scale to FP4 weights and activations without per-tensor scaling achieves better comparable performance as using FP8 UE4M3 scales with per-tensor scaling (UE4M3-S). We emphasize we computed the per-tensor scaling dynamically for UE4M3-S. thus the results reflect the best accuracy that be achieved with this format. Fig. 16 and Table 3 in Appendix [ show the UE5M3 results hold across block sizes and for a variety of models, from attention-based L.MM, SSM, to hybrid SSM-attention models.

---

CONCLUSIONS
Microscaling formats are poised to become in the near future the format of choice for model quan- tization. Proper understanding of the sources of error associated with these formats critical to avoid potentially costly pitfalls at training and inference. In this paper. we analyzed the unexpected trends that quantization errors can present when quantizing narrow distributions with microscaling formats. We demonstrated that such anomalous behavior is direct consequence of the quantization of the scaling factors, and introduced a theoretical framework to identify the separate contributions driving this effect. Theoretical results are in remarkable agreement with experimental observations. On the basis this understanding, we have proposed hardware-friendly implementations, FP4 microscaling with FP8-UE5M3 scales, that effectively and efficiently mitigates these errors.
REPRODUCIBILITY STATEMENT
We share UESM3 implementation and our theoretical model httpp::/github.... com/1 clr201 oeeea//ooooeeeeeeeeoooaaaaaaaaaa Full derivation of the theoretical model is provided in Appendices and
REFERENCES
Ankur Agrawal, Sae Kyu Lee, Joel Silberman, Matthew Ziegler, Mingu Kang, Swagath Venkatara- mani, Nianzheng Cao, Bruce Fleischer, Michael Guillor, Matthew Cohen, et al. 9.1 7nm 4-core ai chip with 25.6 tflops hybrid fp8 training, 102.4 tops int4 inference and workload- throttling. In 2021 IEEE International Solid-Stad Circuits Conference (ISSCC), volume 64, pp. 144-146. IEEE, 2021.
AMD. Hip: Low precision floating point types. https: docs. amd. www.co/////p/..// cwm/projeeeeeeeoooooo ts/HIP/ en docs deve 1 wwtpeefeeenee/llooooooooo ow wt_.pptttpsspeee html 2025. Accessed: September 6, 2025.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian I. Croci, Bo Li, Pashmina Cameron, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-fre 4-bit inference in rotated 1lms. Advances in Neural Information Processing Systems, 37:100213-100241 2024.
Yuzong Chen, Ahmed F. AbouElhamayed, Xilai Dai, Yang Wang. Marta Andronic, George A. Con- stantinides, and Mohamed S. Abdelfattah. Bitmod: Bit-serial mixture-of-dataty. Ilm accelera- tion. 3rXiv:2411..1720
2025.
Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany. Vs-quant: Per-vector scaled quantization for accurate now-precision neural network inference. In A. Smola, A. Dimakis, and I. Stoica (eds.), Proceedings of Machine Learning and Systems, volume 3, pp. 873-884, 2021.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference Machine Learning, pp. 7750-7774, PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers scale. Advances in neural information processing systems, 35: 30318-30332 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint orXiv:2210. 17323, 2022.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. International conference on machine learning, pp. 1737-1746. PMLR, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv:2203. 15556, 2022.
10

---

Wonsuk Jang and Thierry Tambe. Blockdialect: Block- wise Ain-grained mixed format quantization for energy- ehergy-efficient Ilm inference. arXiv:2501.01144, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.0883 2020.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. Fp8 quantization: The power of the exponent. Advances Neural Information Processing Sys- tems, 3551466111466,, 2022.
Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, and Jungwook Choi. Amxfp4: Taming activation outliers with asymmetric microscaling floating-pon -oitin for 4-bit Ilm inference. arXiv:2411.09909 11 2024.
Zhen Li, Yupeng Su, Runming Yang, Congkai Xie, Zheng Wang, Zhongwei Xie, Ngai Wong, and Hongxia Yang. Quantization meets reasoning: Exploring Ilm low-bit quantization degradation for mathematical reasoning. 1xXiv:2501.0303: 2025.
Ye Lin, Yanyang Li, Tengbo Liu, Tong Xiao, Tongran Liu, and Jingbo Zhu. Towards fully 8-bit integer inference for the transformer model. arXiv preprint arXiv:2009. 08034, 2020.
Yun-Chen Lo, Gu-Yeon Yeon Wei, and David Brooksnknown. Nanoscaling floating-point (nxfp): Nanomantissa, adaptive microexponentss and code recycling for direct-cast compression of large language models. 1988 12 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv:1609.0740 0000//0090077...... 2016.
NVIDIA. cublas: D-block quantization. https::/doct: nvidia. http//udaaccuu wwwpo//caaccubaa/////ooo 4d bbocx-quantization. 2025a. Accessed: September 6, 2025.
NVIDIA. Fine-tuning gpt-oss for accuracy and performance with quantization aware training. ht y- http:///ww..gee...aaaaaooooooooa November 17, 2025.
for-accurac
2025b. Accessed:
Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Ritchie Zhao, Mathew Hall, Jasmine Klar, Eric Chung, Yuan Yu, Microsoft Author, Michael Schulte, Ralph Wittig, Amd Author, Ian Bratt, Nigel Stephens, Jelena Milanovic, John Brothers, Arm Author, Pradeep Dubey, Marius Comea,, Alexander Heinecke, Andres Rodriguez, Martin Langhammer, Intel Author, Summer Deng, Maxim Naumov, Meta Author, Paulius Micikevicius, Michael Siu, Nvidia Author, and Colin Verrilli. Ocp microscaling formats (mx) specification ocp microscaling formats (mx) specification version 1.0. Technical report, Open Compute Project OCP, 2023a.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Comea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Ven- mugil Elango, Maximilian Golub, Alexander Heinecke, Phil James- Dharmesh Jani, Gau- rav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Ro- driguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Mi- cikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Mi- croscaling data formats for deep learning. 1Xiv:2310011537, 10 2023b.
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Perfor- mance Extreme Computing Conference (HPEC), pp. 1-9. IEEE, 2023.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami. Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8815-8821, 2020.
11

---

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pp. 38087-38099. PMLR, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in neural information processing systems, 1127168-27183 2022.
Jintao Zhang, Jia Wei, Pengle Zhang, Xiaoming Xu, Haofeng Huang, Haoxu Wang, Kai Jiang, Jun Zhu, and Jianfei Chen. Sageattention3: Microscaling fp4 attention for inference and an exploration 8-bit training. arX1v:2505.//594 2025.
12

---

A PERPLEXITY GAP ACROSS VARIOUS MODELS AND LOW BLOCK SIZE
In our experiments to measure perplexity, we quantize both weights and activations of all linear layers except the last (model head), with the selected microscaling quantization format. Atten- tion matmul are not quantized. Perplexity is computed based on nex(-token prediction task on the Wikitext2 dataset (Merity et (2166), using the test split, with samples of sequence length 2048 tokens. We use perplexity as proxy metric for accuracy, as a starting point to highlight potential issues associated with the quantization process. We provide a comprehensive analysis of accuracy across multiple benchmarks in Sec. 5.2 and in Appendix !
(a)
(b)
1.8
- -enemmtron--aaie 90-12
bamba-9h-v2
1.8
-lama-22-7
--nomotoonnnno-Be
100.0
qwaan-3-88
perplexity gap
perplexity gap
10.0
0.8
0.8
1.0
0.4
0.2
0.0
64
96
128
160
192
224
256
32
64
block size
black size
Figure 5: (a) Perplexity gap vs. block size for FP4 with FP8-UE4M3 scales quantization. Plotted with logarithmic y-axis to account for the significant perplexity inversion observed in some models. (b) Even when perplexity inversion is not present down to block size (as llama-2-7b in Fig. [b)), it can still emerge at even smaller block sizes (2, 4).
13

---

B PER-BLOCK MSE: BLOCK SIZE 8 VS 16
right
Figure 6: Comparison of per-block MSE for block size 8 vs 16 microscaling quantization, using FP4 elements with FP8-UE4M3 scales. As shown by the prevalence of data points above the diagonal, emrrr using the smaller block size routinely exceed errors with larger block size, across different weight tensors and models. This pattern holds even for models not showing perplexity inversion at block size 8.
14

---

C MSE VS O OF MODELS AND BLOCK SIZES
(0)
D-Bb-bas
Bamo- 2.1 de lardaa oi
Heet Spen
3.4
block
block size
block size 32
25 20 e(peeteesoo))
i (per-tensorr
(per-temsor)
25
30
12-3
Figure 7: Per-tensor MSE standard deviation of each weight tensor of various LLMs, highlight- ing a common dependence across models. Very narrow weights may appear to be deviating from the predominant trend, but this behavior is also present in ideal distributions and captured by theoretical framework (see Fig. 3).
15

---

D MSE VS O ACROSS IDEAL DISTRIBUTIONS
Norma((0.1)
Laplace(0.0.7)
VonMises
131
10"
101
frequency
frequenci
327
frequency
107
10t
127
10a
0
36
magnitude Logistic(O,1)
magnitude
magnitude
Gumbel(0,0.88
10
101
423
frequency
frequency
320
822
10h
325
10
10a
15
10
20
12
magnitude
magnitude
Figure 8: Shape of the ideal distributions used compute the MSE in Fig. Bb), show- casing different tails and asymmetries. We stress that the selected parameters are arbitrary and were only chosen such that the corresponding for each distribution type would cover similar interval, upon application of the same vector of scaling factors.
16

---

block
block
54
1X per-tarerr
20C2
mome citettution
Argee beoud
grente3 3-4t--3--- aateoor
block
block
3renic3 3.40 fand lamad-Te
block
blocksizs
orpertennci
Figure 9: (left column) MSE vs comparison of experimental data from 3 models and a Normal distribution of mean 0 and variable a. Across multiple block sizes, the data point from the pre- trained models are in excellent agreement with their Normal counterpart. Notice the sharp increase towards very narrow distributions is observed at any block size. (right column) MSE vs a from experimental data drawn from different distributions.
17

---

E THEORETICAL FRAMEWORK: NON-QUANTIZED SCALES
In this appendix, we present the full derivation of the theoretical framework in the case microscaling quantization with scales quantized (i.e., infinite precision is assumed), and comparison of theoretically-derived MSE-a curves against experimental data from a Normal distribution (Fig. T0
As mentioned in Sec. 4.2 we model the drawing of N elements 2, from a random variable X with probability density P(X). P(X) is a Normal distribution of mean 0 and standard deviation a: X (N0, )²). We use to mimic weight distribution. We quantize these elements using symmetric block-wise FP4 E2M1 quantization scheme, with non-quantized scale defined as
Tmax
(12)
with Tmax = max used for element quantization. For example, m 6.0 for FP4 E2MI. The scale is used derive the scaled elements prior quantization, as
N-1 |or]]_ The factor m is the maximum representable value for the data format
m2;
(13)
Tmax
Accordingly, the scaled elements ys to be quantized are drawn from random variable Y with scaled and truncated gaussian distribution:
mX
Y
(14)
Tmnx
The distribution of is truncated to [-m, m| because xi S Imax Vi. Post-quantization, the de- quantized elements belong to the random variable Z:
Z Emax Q(Y)
Tmax
Q
mxX
Q
X
(15)
17
Tmax
with Q being the mapping function to the quantized levels defined by the selected element format (c.g., FP4 E2M1).
The first step towards deriving the MSE of Z is to compute the PDF of Y conditioned to maxx- As the MSE associated with z; = Tmax is zero, we will only derive an expression for the PDF associated with the - 1 elements satisfying x; # Fmax For z; mmax the PDF is in the form of two Dirac functions located at 4ax but as it does not contribute to the MSE, we will leave out of the derivation.
We define (v) as the distribution of Y prior normalization. We derive (y) by applying the change of variable formula for the transformation of continuous random variable, from X to Fmazy we get:
g(X). With g(x)=
me (g)
fr.nteman (9) xmax) = fx (x))
(v)
dy
Tmaxx!
Fmax m
= fx
m
1
Tmaxy!
2
Tmax
exp
V2To
20
m
Fmax
1
Fmmax
exp
m2
2
mo
(ya))
exp
V2T
2
= a p(ay)
for E [-m,
(16)
with
Tmax
(17)
ma
18

---

and @ the Probability Density Function (PDF) of the standard (1+ 0, 1) Normal distribution:
1
(
22x exp (-12/2)
(18)
To obtain fy (v), the PDF of we normalize fy (t) considering that the probability density is zero outside [-m, m|:
fy (yy Fmax) fr xmax) p((a))
fr (y) Tmaxx
(19)
I ad(ay)dy J o(u)du (4(ma) -
Here, the Cumulative Density Function (CDF) of the standard Normal distribution.
Next, we compute the conditional expected square error on Z, the distribution after dequantization. We focus on the error on Z, instead of QY)), for direct comparison with the experimental data in Fig. (bc) and Fig. Ba-c). This error is the sum of the errors associated to each quantization bin MSEY (gj Tmma)) Each quantization bin comprises one of the quantization levels gj (with j = 1, ,,..., NQ and No the number of quantization levels of the format used to quantize the elements) and is defined by the Voronoi boundaries [ag, by]. The bin error is:
N 1 1 [" "v --()()))² I (3) dy
MSEY (g) [maax]
N 1 ( (y qj (x)) play)
(20)
24
where the (N - 1)/N factor takes into account that only N elements x; contribute to the error.
The square error with respect to Z is related to the square error on Y via the scaling factors s Tmax m as:
2 X -- = Tmax (Y - Q()) (X 2)² [Xmax) ) (Y Q()))
(21)
Therefore, the error becomes:
MSEz,j (g) [maax MSEY
s2
(y )3)² ffy Fmaxx dy
(ao)² N (') (y 43)² a. b(aa))
24((ma) - dy
N a;
N (usta)
(u- - 0,ar² (lu) du
(22)
20(66)=
Tes(a)
We remove the conditioning on Tma by computing its expected value:
MSEz = B Birnas
Σ MSEZj (gj Xmaxx
MSEz, (Ø) Fmax) Szmaa (x) dx (23)
All that is left is to compute Samns (I)). As Fmnx is computed with an absmax operation, its distribu- tion is one of i.i.d. variables drawn from the half-Normal distribution θ:
|X|
HalfNormal(o)
(24)
The half-Normal CDF with 0, drawn from θ is the known formula:
0
F Faiaa (0)
2
(25)
19

---

and since |8 are i..d. the probability of drawing N variables with P([X Fmax) is the product of each individual probability:
P([X) S Xmax) = F(|xs] S x: Vi
N) [P((aa S x))
(26)
This is expressed by the product of N CDF:
(g)
0
N
F Famax (0)
(27)
The PDF is obtained by differentiating this CDF:
d
0
N-1
d
0
fzmax (0) =
F
( =
N
2Φ
2Φ
-1
d0
Emax
d0
0
du
2N
20
{(u)
u
du
de
N-1
1
2N
2Φ
0
6
2N
0
0
24
(28)
As the integral from 0 to do of Samaa is 1, this is already PDF, no additional normalization is needed. Using this formulation for Jaamas the MSE using non-quantized scales is:
MSEz %
MSEzJ (Øs Imax) Σ
2N σ
2Φ
θ
N-1
0
da
(29)
0
MSEz / MSEz (u) Szmma (1) du
(30)
MSEz = Ex E [x (18-2) -z)² X mmax]
(31)
MSEx (w) [X-- (X- 2)²² Xmax = ②
(32)
Finally, we compute this error by discretizing the domain of Zmax over a series of 0, computing all 0-dependent parameters (implicit in all a dependencies), and integrating numerically. The resulting MSEz can be compared directly with the per-tensor MSE obtained experimentally, from either pre- trained models or ideal distributions.
As shown in Fig. 10 the agreement between theoretical model and experimental data is very re- markable, across range of and block sizes bs.
20

---

bs-4
normal bs
normal bs=
theory bs=4
MSE (per tensor
theory
theory bs-16
theory bs-32
1.3
0.3
(per-tensor)
1e-2
Figure 10: Per-tensor MSE standard deviation curves comparing estimates from the theoretical framework with non-quantized scales against corresponding results obtained experimentally from elements drawn from a Normal distribution. The remarkable overlap validates the correctness of the theoretical framework.
21

---

F THEORETICAL FRAMEWORK: FP8 UE4M3 SCALES
In this appendix, we present the full derivation of the theoretical framework the case of microscal- ing quantization with FP8-UE4M3 scales. This derivation naturally builds the process described in Appendix B and summarized in Sec. where we derived a formulation in the case of non- quantized scales.
When scales are FP8, they are additionally quantized with a "casting" operation to FP8:
F'ax
s = QFFs
(33)
m
As before, m the maximum representable value of the quantized elements (m 6.0 for FP4 F2M1). Qrps is the operator that maps each high-precision scale value to its FP8 counterpart. Dif- ferent quantization schemes can be modeled by simply replacing the Q operator and summing over the & quantization levels associated with the new mapping.
When dealing with non-quantized scales, we only considered the x; + Zmax case, because the ele- ment 2; Tmax was represented exactly, and thus did not contribute to the error. This consideration resulted in an (N 1)/N scaling factor applied to the MSE (see eq. 3 When scales themselves quantized, must also account for the 2; max scenario, resulting in non-zero quantization error. This error is to be scaled by I/N instead of (N 1)/N, accounting for 1 clement out of N block elements being used for scale derivation in this condition.
In addition, we will need to separately account for the case where all elements within a block are rounded to zero. This occurs when Σmax Smin with Smin being the lowest non-zero representable scale value. This contribution is not included in the previous derivation, where the case = 0 was both ill-defined and contributed error due the continuous nature of the non-quantized scales.
ys
Overall, we will be modeling 3 separate sources error:
J3 * Fmax with FP8 scales +
3; Emax with FP8 scales
. FP8 scales = 0
F.1 A Tmax WITH SCALES 7
The case of D; + Tmax with s * O was covered in the non-quantized scales framework by remov- ing the conditioning over Tmax via integration. However, when introducing discretization of the scales, we will have to sum over each quantized scale level 8j, so it becomes more convenient to express all formulas in terms of this parameter. We start by deriving an expression for fs, (s), the distribution of the random variable Sx. representing the continuous scales prior quantization. This can be computed using the dependence of Sx on Imaxx with a change of variable the function fzmax (z), the mreiouuly-comppte distribution of Tmax- We start from:
dr
g(())
(8) m. = ds
m
(34)
m
and obtain the expression for the distribution:
dx
fsx (s) Sansn (m 5).
ds
femse (m s)
2N
m
N-1
m
m
2Φ
-
(35)
To get the probability mass p of single scale quantization bin around a value 5k, we can integrate within the bin's Voronoi boundaries au, bk. assuming round-to-nearest:
=x) 3=)== = 12
FP8 Pk
P(Q(s)
fsx (8) ds
(36)
22

---

To remove the conditioning 8k, instead of integrating the continuous distribution fa sum over the FP8 bins the product of p and MSEz,,($k), the latter being the MSE with respect to the random variable Z of the quantized/dequantzze MSE2, (8k) requires knowledge of the PDF of Y::
elements, conditioned to scale sg. Deriving
fr (k) = fr 8k) ok ((((()))))
(37)
__m ao(ag)dy
20(ma)) -
with on 8k/o. Notice that for 8y 0, the normalization factor denominator of fr 8k) is zero. Hence, we will have to treat the = 0 case separately.
The expression for the MSE per-bin is similar the non-quantized scale scenario, except for the dependence on k:
MSEZ,s (gj 8k) 8 b) (y 43) -f( 88) dy
(uston)
(u (aa o(u) du
(38)
24(6)))
N Jestau)
where the (N - 1)/N factor addresses the condition F Tmax-
Finally, we remove the conditioning on 8ą by summing over the scales quantization levels and the elements quantization levels j:
MSE Z ar Fmaa
Σ
pk
FP8
MSEE,
Σ 1
be fsx (6) ds Σ MSEZ,s (Øi sk)
(39)
F.2 = Tmax WITH SCALES +
When Tmax and scales are not quantized, the eπrо on T; is zero: the scale is derived directly from Tmax and one quantization level is aligned mmax exactly. However, this no longer the case upon scale quantization, as the scale Qres introduces quantization error.
We can compute the error conditional on Fmax directly, as it pertains a single element 2; Zmaxx in lieu of having to integrate over the distribution of y:
EIT,, #aaos (I, Tmax)
(Qolem
(40)
5k
where Qeiem is the element quantization mapping (e.g., FP4 E2M1). Notice this computation not defined for 8 O, the third source of error to be treated separately.
The corresponding MSE for the scale bin i (MSE Err z by 1/N, because the eтror ErT E only applies to a single element in block of size N.
is derived by simply scaling
To obtain the total MSE across all scales, we integrate MSE Rmm along with the probability mass FP8 of a scales Sk, and sum over all non-zero scales:
MSE
Σ
FP8 Pk
MSE 2, =xmaeik
mbb
Er Errz, (x. 81) Semox (x) dx
(41)
N
The interval [mag, mbx] comes from the boundaries (ag, bu] of the bin of sk. expressed terms of 8k-
23

---

F.3 SCALES
The third and last source of MSE eтrоr comes from the lowest FP8 scale bin which, assuming round-to-nearest process, goes from to Swin, where Smin is the lowest non-zero representable FP8 value (2-9 for IEEE FP8 E4M3, which includes subnormals). If the maximum N values of X generates a scale that falls within this bin, all values in the block are rounded down to zero. This increases the error for ultra-narrow distributions.
The eπrоr for zero quantized scale is:
MSE,=0= = P(s P)==0- - == =0]
(42)
where E[x² s= 0) is the expected error when the scale is zero, and P(s 0) is the probability of having zero scale. The probability of zero scale P(s = 0) is derived considering that once Fmax- Hence, I; do not conform to the original distribution of X with standard deviation a but to a distribution that is truncated in the 1-bmin mmin]. Therefore:
value mmx is drawn, all other satisfy the condition
interval
P(s
(Xmax <min) - (Fix) (bmin))
(43)
where (2) is the CDF of |X| that we already computed as eq. 25 The expected error with zero scale is computed using the known expression (which includes normalization factor at the denominator):
Musin x fx)x(()) dr
F[X² 8= 0] F[X² |X| < bmiin]
Drnin
(44)
Buii
[x(x) dx
buan
F.4 TOTAL ERROR
The total MSE when using quantized FP8 scales is the sum of the separate contributions we computed in the previous sections:
MSEz MSE ZEN MSE MSEZ,s0
(45)
The first two terms contain block size-dependent scaling factor: (N 1)/N or 1/N, respectively. Both these terms sum over all 8, * O, while 2. single scale bin computation. in the case of non-quantized scales, we compute MSEz by discretizing the domain of T'ax and integrating numerically.
Fig. I1 compares theoretical results (lines) against experimental data derive by quantizing elements from Normal distribution (dots). The agreement is found to be excellent across wide range of and various block sizes, capturing all features of the experimental curves, including the presence and location of crossover points, and steep MSE increase for ultra-narrow distributions.
Fig. 12 expands Fig. 3).. presenting the individual contribution of the 3 sources of errors for block size 4, 8, 16, and 32. The relative weight to the total error of the different contributions varies. At large block size, the error is largely dominated by MSEZ,z, As block size is decreased, MSEZ.,, becomes more prominent, materializing the crossover between curves of different block size. At small block size, MSE2,s=0 impacts wider interval of lower-end o and its magnitude increases. Hence, narrow distributions become more and more susceptible to this source of error as block size is decreased.
24

---

1e-5
3.0
nomaal bs=4
normal b
2.5
normal b5=16
normal bs=32
theory bs=4
theory b5=8
theory b5=16
theory b5=32
MSE
1.5
1.0
0.5
4
(per-tensor)
1e-2
Figure 11: MSE estimates from the theoretical framework for FP4 microscaling quantization with FP8 UE4M3 scales (lines) against corresponding results obtained experimentally from elements drawn from a Normal distribution (dots).
2.00
Theary, b5-4
Theary, bs=
1.5
Yeass
1.75
Krear
Frese
150
Sipe
Sryo
MSE [pen (ersee)))))
1.25
2.23
3.00
3.00
MSE
0.75
0.75
0.30
0.35
0.25
0.00
[per tensor)
(per tenstr)
1e-2
2.00 16-5
2.00
1
Theory, bs=16
Theory, ba=32
1.75
Xi Xeree
# Ares
Kees
Swe 0
Sway
MSE [perseensort
0.25
MSE [pen
3.00
2.00
0.50
6.50
0.08
0.08
[per-tansor)
(per-tensor]
1e
Figure 12: Separate contributions to quantization error for block sizes 8. 16. and 32.
25

---

G MICROSCALING INT4 QUANTIZATION WITH FP8 UE4M3 SCALES
To provide further validation of the flexibility of our framework in modeling custom quantization strategies, we applied simple modifications extract the MSE estimates under INT4 element per- block quantization (instead of FP4), with FP8 UE4M3 scales. This change only required an update the specified list of quantization levels and the simulation parameters that are dependent on the maximum of the element data format. For asymmetric INT4 quantization, which quantizes in range [-7.7]. the format maximum 7, instead of 6.0 FP4. The comparison against experimental data obtained by directly quantizing elements drawn from Normal distribution and computing the corresponding per-tensor MSE is shows Fig. 13 Similarly the FP4 case, theoretical results and experimental data are found in excellent agreement (x2 1.3 10 10- 5).
normal bs=4
normal bs=8
2.5
normal bs=16
normal bs=32
theory bs=4
2.0
theory bs=8
theory bs=16
theory bs=32
MSE
1.5
0.5
0.0
2
(per-tensor)
1e-2
Figure 13: Theoretical MSE estimates against experimental data from Normal distribution, under microscaling INT4 quantization with FP8 UE4M3 scales.
Perplexity evaluation on microscaling INT4 formats across various models (Fig. 14 shows that per- plexity inversion still observed, albeit in less pronounced manner compared microscaling FP4. Both UE4M3-S and UESM3 are effective in mitigating the inversion and improving performance, with UE5M3 achieving comparable better perplexity than UE4M3-S.
These results are consistent with the analysis ideal normal distributions (synthetic data) as well our theoretical framework (Fig. 13) for microscaling INT4: both show that narrow element dis- tributions quantized as INT4 do experience a qualitatively similar effect as FP4, with smaller block size producing larger error than larger block size narrow distributions. For block size vs. 8, the crossover does occur at lower standard deviation (= 1.5 10-2) than for FP4 (= 2 10-3), suggesting that considering
tensors within a model, and their spread of standard deviations, we should observe a less pronounced effect, possibly arising at lower block sizes than FP4, at which the increase in error is larger.
26

---

Granite-3.3-8b
Llama-3.1-8b
24
15
FP6
14
FP8 UESM3
20
FP8 UE4M3-S
13
18
BF16
12
perplexity
perplexity
11
14
10
12
10
8
8
32
64
96
128
180
192
224
256
32
96
128
160
192
224
256
block size
block size
Nemotron-nano-9b--2
Qwen3-8b
12.0
20
FP8 UE4M3
UE4M3
11.5
FP8 UESM3
FP8 UESM3
18
FPB UE4M3-S
UE4M3-S
11.0
BF16
BF16
16
perplexity
10.5
parplexity
10.0
9.5
12
9.0
8.0
0
32
64
96
128
160
192
224
296
32
64
96
128
160
192
224
256
block size
block size
Figure 14: Perplexity evaluation of microscaling INT4 with different scales: UE4M3, UE4M3-S, and UE5M3.
27

---

H MICROSCALING FP4 QUANTIZATION WITH FP6 SCALES
As further validation of our framework, we have measured the impact of microscaling FP4 with two custom FP6 formats. We stress that no standard format has been recommended for FP6 scales at this time, and while OCP (Rouhani et al. (2023a)) prescribes FP6 E3M2 and FP6 E2M3 for elements quantization MXFP6 formats, based on our findings herein reported we believe supporting the widest dynamic range is of utmost importance. Hence, we tested FP6 UESMI and FP6 UE4M2 scales (assuming repurposing of the unused sign bit). in combination with FP4 E2M1 elements, to quantize weights and activations of Ilama-3.1 1-bb (BF16 baseline perplexity 6.242). Consistently with UE4M3-S nomenclature, UESMI-S and UE4M2-S represent the use of per-tensor scale on both weights and activations.
Table 2: Perplexity (the lower the better) of Ilama-3.14 3.1-8b using microscaling FP4 elements vari- able block size and two FP6 formats for scales, UESMI and UE4M2, with or without per-tensor scaling. BF16 baseline is 6.242
Block size
UESMI
UESMI-S
UE4M2
UF4M2-8
7.217
7.240
795.660
6.598
7.265
7.280
343.621
6.726
7.346
19.611
6.872
16
7.414
7.424
11.398
7.038
7.547
64
7.715
7.696
9.607
7.385
7.955
7.952
10.955
7.660
256
8.328
8.307
9.017
8.019
Results are entirely consistent with our previous conclusions related to FP8 scales: the wider dy- namic range provided by UESMI achieves limited degradation in absence of per-tensor scale. In fact, applying such global scale has negligible effect on perplexity when using this format. How- ever, compared to UE5M3, this format suffers from the loss of 2 bits of precision. On the other hand, UE4M2 performs very poorly without per-tensor scale and we once again we observe a perplexity inversion behavior. This can be effectively mitigated by means a per-tensor scale (UE4M2-S).
With respect to the validation of our theoretical framework, we modeled these two microscaling formats based on FP4 E2MI elements and FP6 scales, either UESMI and UE4M2. MSE estimates reported in Fig. a5), across different block sizes. In the case of FP6 UE5MI scales, crossover observed between curves different block size, consistently with absence of perplexity inversion reported above. By contrast, MSE estimates for FP6 UE4M2 show similar features as FP8 E4M3 (as in Fig. 3 and Fig. TTp. but accentuated by the moderate decrease dynamic range and lower precision brought about by the smaller mantissa. Interestingly, for FP6 UE4M2, crossover occurs at larger a compared to FP8 E4M3 (for example, at R 3.8 10-2 going from block size 16 8, compared
2 10-2 reported in Sec. 3.2). which suggests even wider distributions. and hence more models, would experience larger eтror at smaller block sizes. Hence, mitigation becomes even more important as more aggressive quantization formats deployed.
28

---

(a) 12.5
(b) 1
bs-4
FP4 E2M1 elements
FP4 E2M1 elements
bsm8
FP6 UESM1 scales
FP6 UE4M2 scales
2.5
bs=16
2.0
2.0
1.5
1.5
1.0
1.5
bs-4
0.5
bs=8
bs=16
0.
(per.tensor)
1e-2
(per-tensor)
1e-2
Figure 15: Theoretical MSE estimates for microscaling FP4 quantization with FP6 scales for- mats (without per-tensor scaling)
29

---

I ACCURACY USING UE5M3 vs. UE4M3 WITH PER-TENSOR SCALING
m1amb12
Cambea St2-
(wer18-00
FW namoirnn-ann--0-2
162
140
03
70
blstsson
mammba-codesten
srituct Tedb-iistuu
U.mo-d.1
pupplntt
50
2lcck.000
Mook 920
Mook 922
Across various LLMs, UESM3 scales perplexity consistently on par with UE4M3 scales using per-tensor scaling, and shows large improvements with respect to UE4M3 scales with- out per-tensor scaling.
Figure 16:
Table 3: Accuracy under the proposed FP4 microscaling quantization schemes, block size 16.
Acronyms are the same as in Tab.
Model
Formal
Wiki 1
PIQA
Hsw
Wag 1
GSMBK
MMLU 1
granite-3.3-88
BP16
4.72
80.41
61.49
72.38
62.
60.55
6.45
78.50
56.98
70.71
30.17
50.67
UE4M3-S
5.51
77.
58.58
72.22
36.47
54.11
UE5M3 (ours)
5.15
79.71
60.08
71.11
51.25
56.12
Ilama-3.1-8b
BF16
6.24
79.87
60.05
73.48
50.19
63.28
57.96
56.37
UE4M3 S
6.95
78.29
58.73
72.69
39.27
59.16
(ours)
78.51
58.50
71.74
58.96
nemotron namo 9h.v2
BF16
8.08
80.30
58,22
73.24
79.61
73.56
UE4M3 S
8.50
79.60
$7.46
73.80
76.50
71.78
(ours)
57.02
73.17
harnha 9b-v2
BF16
6.21
80.96
62.18
73.95
42.15
64.96
UF4M3
13.95
78.13
51.72
65.59
10.39
44.85
UE4M3-S
6.64
79.
61.50
73.64
38.89
63. 11
(ours)
30

---

J ALTERNATIVE BIT REPURPOSING: FP8 UE4M4 SCALES
The unused bit of FP8 E4M3 can be alternatively repurposed to extend the mantissa, instead of the exponent. The resulting FP8 UE4M4 format for scales not only benefits from higher precision, but also a moderately extended dynamic range: the lowest representable subnormal element decreases from 2 to 2- Based on our findings, this is expected to lower the етrог associated scales quantization. Fig. [7] confirms that UE4M4 is indeed beneficial, but UESM3 remains the superior solution, more effective and robust across various block sizes. addition, it important to remark that, as mentioned in Section 3.1 the complexity of multiplication in hardware scales quadratically with the number of mantissa bits M,, once again favoring UE5M3 as the most hardware friendly option.
(a)
llama-3. 3.1--b
(b)
granite-3.3-B8
4.5
4.0
scale FPB UESMB
scale FP8 UE4M4
3.5
+e-scale FP8 UE4M4
perplexity gap
perplexity gap
3.0
2.5
2.0
1.5
0.5
0.0
48
16
24
block size
block
Figure 17: Perplexity gap for alternative scale format FP8 UE4M4.
31

---

K UESM3 HARDWARE DESIGN
We synthesized a systolic array processing engine (PE) with a microarchitecture similar to that described in Agrawal al. (2021). The engine has eight Single Instruction Multiple Data (SIMD) lanes, and each lane contains multiple multiplyannd-acuumlla MAC operations on multiple weights and imput terms, corresponding different precisions. Each SIMD lane supports BF16, FP8 (both E4M3 and E5M2), INT8, and microscaling FP4. Two versions of microscaling FP4 were synthesized: one with an E4M3 scale and the other with an ESM3 scale.
(MAC) engines, each performing
Both E4M3 and E5M3 incur the same multiplier cost for processing the sum of FP4 product terms and the product of the scale mantissas. E5M3 requires a 5-bit adder to compute the product scale exponent, compared to 4-bit adder for E4M3. The resulting product exponent is further sub- tracted from the 8-bit exponent the inter-PE partial sum; therefore, the width of the subsequent adders/datapath remains unchanged.
Logic synthesis was performed using a nm process node in a production-grade EDA flow. The area for the E5M3 scale is 0.5% larger than that for the E4M3 scale, which is negligible and does not affect the bounding box area for place-and-route or subsequent SoC floorplanning. Critical path timing increases by 4 picoseconds, which is negligible for setting the SoC frequency. The intuition behind this small area/timing impact is that the effect of the wider adder is diluted by the arithmetic pipelines for other precisions, as well as non-arithmetic logic such operand staging and the local register file for operand reuse.
32


Reviews:
[{"review_id": "YSG1UTHWdP", "reviewer": "Reviewer_qJkV", "summary": "The paper investigates an unexpected limitation of microscaling quantization—a fine-grained, per-block quantization method increasingly used for efficient training and inference of large language models (LLMs). While smaller block sizes are generally assumed to improve quantization accuracy, the authors discover a quantization anomaly whereby further reducing block size below a threshold increases model perplexity, a phenomenon they term perplexity inversion. They diagnose this anomaly through extensive experimentation across various LLMs and develop a robust theoretical framework that decomposes the Mean Squared Error (MSE) into three distinct contributions, revealing that the quantization of scaling factors and the effect of zero-rounding are the primary drivers of the inversion, especially for narrow tensor distributions. To address this, the paper proposes FP8 unsigned E5M3 scales, demonstrating that this hardware-friendly solution effectively mitigates perplexity inversion by offering an increased dynamic range without the need for global scaling operations.", "questions": "1. The paper effectively demonstrates perplexity inversion using FP4 elements and FP8 UE4M3 scales. It would be valuable to understand if this phenomenon is specific to this combination or broadly applicable. Have the authors observed similar inversion when using other common low-bit quantization formats (e.g., INT4, INT8, or different FP variants) with their corresponding scales also quantized? Clarifying whether the identified mechanism (interplay of narrow distributions and limited scale dynamic range) is a universal challenge or uniquely pronounced with the studied configuration would better define the scope and novelty of this important discovery.\n2. The benefits of UE5M3 are predominantly highlighted for inference. However, its role in the training phase, particularly within a Quantization-Aware Training (QAT) framework, needs further clarification. Could the authors elaborate on how the extended exponent range of UE5M3 impacts gradient calculations, numerical stability, or convergence during QAT compared to UE4M3? Understanding whether UE5M3 is primarily geared towards Post-Training Quantization (PTQ) or if it seamlessly integrates with QAT, potentially requiring specific modifications or offering distinct advantages, would provide a more complete view of its practical utility.\n3. While the UE5M3 format improves scale representation through a fixed hardware-defined design, it would be valuable to compare it with adaptive scaling methods such as VS-Quant (Dai et al., 2021) and GWQ (Yang et al., 2024), which optimize scales per vector or group. These methods share the goal of mitigating quantization errors via finer-grained scaling. An empirical or qualitative comparison discussing complexity, overhead, and accuracy trade-offs would help clarify UE5M3’s unique advantages and positioning within the current quantization landscape.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper makes a highly original and significant contribution by identifying \"perplexity inversion,\" a counter-intuitive phenomenon where smaller block sizes unexpectedly increase quantization error in microscaling for LLMs. This challenges common assumptions and reveals a critical pitfall for future low-bit quantization efforts.\n2. The FP8 UE5M3 solution stands out as a key strength due to its practical and well-reasoned approach to mitigating the identified perplexity inversion. By repurposing an unused bit to extend the exponent range, UE5M3 significantly increases dynamic range, enabling better representation of small-magnitude elements crucial for narrow tensor distributions. This design is not only hardware-friendly, requiring minimal modifications to existing infrastructure, but also achieves comparable or superior performance to more complex per-tensor scaling methods, effectively simplifying the quantization pipeline while preserving or enhancing model accuracy. Its foundation in the paper's theoretical insights ensures it is a targeted and robust solution to the core problem.", "weaknesses": "1. The paper effectively shows perplexity inversion with FP4 elements and FP8 UE4M3 scales. Do the authors observe similar inversion with other low-bit formats (e.g., INT4, INT8, other FP formats) and quantized scales? Clarifying if this mechanism is universally applicable or specific to the studied configuration would define the discovery's scope.\n2. The paper emphasizes the hardware-friendly nature of UE5M3, particularly for inference. However, the practical implications of integrating UE5M3 during the training phase, especially for quantization-aware training (QAT), are not fully elaborated. The discussion mainly focuses on FP8 UE4M3 for existing hardware. Clarifying how the extended exponent range of UE5M3 impacts gradient calculations, potential numerical stability issues during training, or if it primarily targets post-training quantization (PTQ) scenarios would be beneficial.\n3. While the proposed FP8-UE5M3 format effectively extends the dynamic range of fixed scales, the paper lacks a comparison with adaptive scaling methods such as VS-Quant (Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference) and GWQ (Gradient-Aware Weight Quantization for Large Language Models). These approaches also refine scale granularity through per-vector or group-wise learnable scaling. Without an empirical or qualitative comparison, it remains unclear whether UE5M3 offers distinct advantages over these adaptive strategies or if their benefits overlap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gLFfF1YBuI", "reviewer": "Reviewer_fiMv", "summary": "This paper investigates the unexpected behavior of microscaling FP4 quantization when FP8 scale quantization is used. The authors find that reducing block size does not always reduce error and in fact can worsen accuracy for narrow weight distributions. They develop a theoretical framework that decouples sources of quantization error and show excellent agreement between theory and empirical results across multiple models. Finally, they propose an FP8 UE5M3 scale format that mitigates the anomaly without requiring additional hardware cost, and demonstrate improved model accuracy compared to UE4M3 or per tensor scaling.", "questions": "Why does FP16 scaling not suffer from this anomaly if the cause is related to the deviation of the maximum weight in a block and scale resolution?\n\nCould you briefly comment on expected behavior for future lower precision scale formats such as FP4 or hybrid exponent mantissa configurations. For example, if microscaling continues to push toward fewer bits for scale, should we expect similar inversion behaviors and would your theoretical framework still apply.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I appreciate the solid empirical observation and thorough investigation of a subtle but important anomaly in microscaling quantization. \n\nThe paper formulates a clear theoretical framework that generalizes the understanding of error behavior and matches experiments well. \n\nThe analysis in figures such as Fig 2b and Fig 3c is especially compelling as it isolates the dependence on distribution width and scale quantization. \n\nThe proposed UE5M3 solution is simple, hardware friendly, and demonstrates practical effectiveness.", "weaknesses": "The anomaly is a surprising phenomenon for readers and it may help to offer a concise intuitive explanation earlier in the introduction, rather than waiting until later sections, so that readers understand the high level mechanism before diving into the detailed framework. For example, a short statement that quantization of scales interacts with narrow distributions and reduces effective representable range could improve clarity.\n\nIt would also be valuable to expand the discussion to other scale precisions. The paper focuses on FP8 scales versus FP16 and the new UE5M3 format. Discussion on whether the same anomaly is expected for future lower precision formats such as FP4 scales or mixed mantissa exponent configurations would help generalize the insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xtenp8KkNV", "reviewer": "Reviewer_bRVj", "summary": "This paper uncovers a surprise in LLM quantization: making quantization blocks too small (\"finer\") can paradoxically hurt performance. The authors trace this \"perplexity inversion\" anomaly to the standard FP8 UE4M3 format used for the per-block scales, which fails to accurately represent tensors with very small values (narrow distributions). They provide a rigorous theoretical model to prove this and propose a simple, hardware-friendly fix: FP8 UE5M3, a new scale format that uses a spare bit to add a 5th exponent bit. This new format solves the anomaly and achieves high accuracy without requiring expensive per-tensor scaling operations.", "questions": "How does this anomaly, and the UE5M3 fix, perform with the different, often-asymmetric distributions of activations?\n\nDid you investigate adding a mantissa bit (i.e., UE4M4) for precision instead of an exponent bit (UE5M3) for range?\n\nTable 1 shows that for several models (e.g., llama-3.1-8b, bamba-9b-v2), your UE5M3 proposal achieves nearly identical performance to the UE4M3-S (with scaling), not a clear gain. Given this, what is the primary motivation for a hardware change when a software mitigation performs comparably?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Identifies the counter-intuitive \"finer is worse\" quantization anomaly.\n\nDevelops a mathematical framework that perfectly explains the why behind the anomaly, which is a significant step beyond just observing it.", "weaknesses": "The theory is heavily based on weight distributions (modeled as Normal), with less focus on how the anomaly impacts different and often asymmetric activation distributions.\n\nThe claim of \"minimal\" hardware cost for UE5M3 is asserted but not analyzed in-depth (e.g., no area or latency estimates).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQYQy8ldp7", "reviewer": "Reviewer_UtsX", "summary": "This paper investigates the counterintuitive behavior in microscaling quantization where smaller block sizes can lead to higher quantization error. The authors provide both experimental evidence across several LLMs and a theoretical framework explaining the phenomenon, attributing it to scale quantization effects. They further propose a hardware-friendly fix using FP8-UE5M3 scale representation, demonstrating improved performance.", "questions": "How does the anomaly behave with integer quantization (INT4) in practical LLM inference, not only synthetic distributions?\n\nWould dynamic or learned scale clipping alleviate the same issue without new hardware?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The discovery of the “perplexity inversion” phenomenon is novel and well-motivated.\n\nThe theoretical modeling is rigorous and matches experimental data convincingly.\n\nThe proposed UE5M3 solution is simple, practical, and hardware-friendly.\n\nWriting and figures are clear; experiments cover multiple models and tasks.", "weaknesses": "The experiments mainly focus on inference; it would strengthen the paper to evaluate whether the same anomaly occurs during training.\n\nWhile hardware feasibility is discussed qualitatively, more quantitative data (e.g., area, latency, or energy cost of adding one exponent bit) would clarify the trade-offs.\n\nThe proposed format is only tested on sub-10B models. Given the claim of generality, evaluating larger-scale LLMs (e.g., 30B–70B) would enhance credibility.\n\nSome connections to existing FP8 and mixed-precision deployment standards (e.g., NVIDIA MXFP4, OCP spec) could be more explicitly compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)