Accumulator-Aware Post-Training Quantization for Large Language Models
Authors: Ian Colbert
Giuseppe Franco
Fabian Grob
Jinjie Zhang
Rayan Saab
Venue: Accepted by TMLR ()

Abstract: When quantizing weights and activations to increasingly narrower representations, the cost of additions begins to dominate that of multiplications in multiply-accumulate (MAC) units. Recent studies show that reducing addition costs via low-precision accumulation improves throughput, power, and area across inference platforms, albeit with an increased risk of overflow. Accumulator-aware quantization research has so far only considered the quantization-aware training (QAT) paradigm, in which models are fine-tuned or trained from scratch with quantization in the loop. As models and datasets continue to grow in size, QAT techniques become increasingly more expensive, which has motivated the recent surge in post-training quantization (PTQ) research. To bridge this gap, we introduce AXE—the first accumulator-aware quantization framework explicitly designed to endow overflow avoidance guarantees to PTQ algorithms. We present theoretical motivation for AXE and demonstrate its flexibility by implementing it on top of two existing algorithms: GPFQ and OPTQ. We design AXE to support multi-stage accumulation, opening the door to full datapath optimization for the first time. We evaluate AXE using recent language generation models; when quantizing Llama3 8B for a 16-bit multi-stage accumulation datapath, AXE maintains up to 98% of the FP16 perplexity, surpassing naïve bit width manipulation by up to 15%.

Published in Transactions on Machine Learning Research (12/2025)
Accumulator-Aware Post-Training Quantization for Large Language Models
Ian Colbert
ian. www.coobbetaad.cccomm com
AMD
Giuseppe Franco
@iuseppe.franccoaamddddmm
AMD
Fabian Grob
bubian.grobbtumm.d
TUM
Jinjie Zhang
ajinjii
Amazon
Rayan Saab University of Colifornia San Diego
rsaab wwccd. edu
Reviewed on OpenReview: https: http:///epereeeeeeeemmee net/ forum? pdl669679
Abstract
When quantizing weights and activations to increasingly narrower representations, the cost of additions begins to dominate that of multiplications in -ccumull- (MAC) units. Recent studies show that reducing addition costs via low-precision accumulation improves throughput, power, and area across inference platforms, albeit with an increased risk of overflow. Accumulator-awan quantization research has so far only considered the quantization-awaal training (QAT) paradigm, in which models are fino-tuned or trained from scratch with quantization in the loop. As models and datasets continue to grow in size, QAT techniques become increasingly more expensive, which has motivated the recent surge in post-training quantization (PTQ) rescarch. To bridge this gap, we introduce AXE-the first accumulator-awaa quantization framework explicitly designed to endow overflow avoidance guarantees to PTQ algorithms. We present theoretical motivation for AXE and demonstrate its flexibility by implementing on top two existing algorithms: GPFQ and OPTQ. We design AXE to support multi-stage accumulation, opening the door to full datapath optimization for the first time. We evaluate AXE using recent language generation models; when quantizing Llama3 8B for 16-bit multi-stage accumulation datapath, AXE maintains up to 98% of the FP16 perplexity, surpassing naïve bit width manipulation by up to 15%.
1 Introduction
Neural network quantization is reaching an inflection point. Existing techniques commonly reduce inference costs by restricting the precision weights and activations to exploit low-precision datapaths in hardware. Al- though substituting the standard full-precision floating- point operands with low prepiiiiioon integor counterparts can drastically reduce the cost. of multiplications, this only accounts for part of the core mulliply-accumulae (MAC) operation; the resulting products are often still accumulated at 32 bits.
Amdahl's Law (Amdhhl 1967 suggests that focusing solely on weights and activations yields diminishing returns. While narrower operand datatypes reduce multiplication costs substantially, they reduce addition
"ork done while at AMD
Work done while University California San Diego

---

Published in Transactions on Machine Learning Research (12/2025)
GPFQ GPFQ+EPinit GPFQ-AXE
Parplexity
Float
20-
16
18
20
22
24
26
28
60
70
80
90
100
Accumalator Bit Width
Relative BR Operations
Figure 1: We use GPFQ Lybband & Saab 2021 to quantize SmoILM2-135M AAlal et al. 2024 using naive
bit width manipulations (red stars) within the design space described in Section 5 We compare AXE (green circles) to EP-init (blue triangles) (Colbert et al., (024)) when targeting reduced accumulator bit width. We use Pareto frontiers to visualize the trade-off between WikiText2 (Merity et al., 2016) perplexity and either (left) accumulator bit width or (right) bit operations relative to W8A8 with 32-bit accumulation. Note that our bit operations cost model is highly correlated with relative power savings, as shown in Section 3.1
costs at a much slower rate. Indeed, recent studies have demonstrated that addition becomes the bottleneck as datatypes shrink, reporting significant benefits when the accumulator precision is restricted during inference. For example, Ni et al. 2020 show that, when constraining operands to 3-bit x 1-bit multipliers, the cost of 32-bit accumulation consumes nearly 75% of the total power of their MAC unit; they report up to 3x power savings when reducing to 8-bit accunulation. As few-bit integers increase in popularity (Ma et al., 2024 Liu et al. 2025 Zhang et al.) 2025b), we expect neural network quantization techniques will need awareness of the accumulator to intentionally address this emerging bottleneck.
Exploiting low-precision accumulation is non-trivial in practice due to three challenges: (1) the benelits of reducing accumulator precision like those of reducing weight and activation precisions vvary across platforms and workloads, and often depend on specific hardware and software support: (2) even with careful design, reducing accumulator precision exponentially increases the risk of overflow, potentially introducing arithmetic errors that significantly degrade model accuracy Ni et al. 2020: Colbert et al. (023); and (3) existing solutions do not scale to modern billion-paramete large language models (LLMs). We focus on the latter two challenges and propose a scalable solution with theoretical justification.
To eliminate the risk of overflow, Colbert et al. 2023 proposed an accumulator-aware quantization paradigm that infuses strict learning constraints informed by theoretical guarantees into quantization-awar training (QAT). The resulting scope of research has since been limited to this QAT setting, where models are trained from scratch fine-tuned from checkpoints with quantization in the loop (Colbert et al. 2024 Zhang et al, 2025a). With the high training costs of modern deep learning models, it is important to develop methods that are equally as effective in the post-training quantization (PTQ) setting, where pre-trained models are directly quantized and calibrated using relatively modest resources. However, controlling accumulation requirements in such scenario is non-trivial. To the best of our knowledge, there has been no formal study that explores accumulator-aww aware quantization in the PTQ setting.
Contributions.
We provide the first formalization of the accumulator-an -warre post-training quantization (PTQ) setting and propose AXF, as approximate solution with theoretical justification. AXE infuses overflow avoidance guarantees into layerwise PTQ algorithms that greedily correct quantization error, for example, GPFQ (Ly- brand & Saab 2021 and OPTQ (Frantar et al.] 2022). We present AXE as composition of functions designed to control the dot product ranges throughout error correction, and demonstrate its flexibility by presenting accumulator-aware variants of both GPFQ and OPTQ. Our open-source implementations are made available as part of the Brevitas quantization library v (Pappalardo al. 2025). We evaluate our
https://github.
2

---

Published in Transactions on Machine Learning Research (12/2025)
accumulator-aware variants across pre-trained language generation models and show significant improvements in the trade-off between accumulator bit width and model quality when compared to alternative methods, thereby enabling lower power consumption with better model quality as shown in Figure I Unlike prior accumulator-awann QAT methods, which assume a monolithic accumulator, we design AXE to support multi-stage accumulation, which opens the door to datapath optimization and enables large language models (LLMs) for the first time. Indoed, our results show that AXE scales extremely well to billio--arammete language models when targeting multi-stage accumulation. For example, when quantizing Llama3 8B for a 16-bit multi-stage accumulation datapath, AXE maintains up to 98% of the baseline FP16 perplexity.
2 Preliminaries
We first introduce our notation. We denote the Ky-dimensional input activations to layer I as x0) e KK, where (1) E KixD denotes a matrix of D such inputs. The weight matrix for layer with K2 input neurons and Cr outpul neurons is similarly denoted as ( E RCiKK; its quantized counterpart is Q E AixK, where we use Agax to denote the space of all m x n matrices whose elements are part of a fixed b-bit alphabet defined by the target quantization space. For example, the alphabet of signed b-bit integers is As := {k: : -2 +15 k < 2h-1 - 1, E Z), assuming a sign-magnitude representation, where Z is the space of all scalar integers. For layer 1, our notation yields Ct independent dot products of depth K2 for each of the D inputs. For clarity, and without loss of generality, we often assume C₁- = 1 when focusing on a single layer I so that can use w()) to denote the weight matrix for layer I. When dropping their superscript, x and denote generic inputs and weights in R K and t and q denote their quantized counterparts.
2.1 Post- Training Quantization
Standard quantization operators, referred to as quantizers, are commonly parameterized by zero-point 2 and scaling factor s, as shown in Eq. for weight tensor Our work focuses on uniform integer quantization, where z is integer value that ensures that zero is exactly represented in the quantized domain, and is a strictly positive scalar that corresponds to the resolution (or step size) of the quantizer. Scaled values are commonly rounded to the nearest integer, denoted by [-J, and elements that exceed the representation range of the quantized domain Ab, are clipped.
@(w)= = s. (cclip u
([
+ 2; min As, max Ao) -2)
(1)
Methods for tuning quantized models broadly fall into two paradigms: quantization aware training (QAT) and post-training quantization (PTQ). QAT mothods train or fine-tune a neural network with quantization in the loop, which often requires significant compute and sulficiently large datasets. Our work focuses on PTQ methods, which directly calibrate pre-trained models and rely on minimal data without end-to-end training. Many recent PTQ methods follow a common general structure, greedily casting and calibrating quantized models layer-by-layer or block-by- by-block while seeking to approximate the minimizer of the reconstruction error in Eq. 2 where q" is the optimal set of quantized weights and X is the quantized counterpart of X.
q" = arg min qe.4K
| w
X XTql3.
Recent LLM PTQ mothods often concentrate on weight-only quantization to solely minimize data storage and transfer costs [Lybrand & |aab 2021 Frantar et al. (002). This focus has been justified the ever-increasing weight volume of staate-of-the-art models has rendered many hyper-scale LLMs memory-bound (Zhang 2023 In this context, weight-only quantization algorithms preserve model quality and still improve end-to-cnd throughput just by reducing data transfer costs, even with FP16 compulations (Frantar el al. 2022 Tseng el al. 2024). However, with the progression of continuous batching in cloud-based based LLM serving (Yu et al. 2022) and the rise of ressurce-efficien sampling methods like speculative decoding (Leviathan et al. 2023), which exploit available compute when memory is the bottleneck, it is increasingly important reduce the cost of arithmetic operations, even for hyper-scale LLMs. In these cases, weight-activation quantization presents an opportunity to not only increase throughput from
et al.] 2022a| Biderman et al.
3

---

Published in Transactions on Machine Learning Research (12/2025)
reduced data traffic, but also to benefit from accelerated computation and decreased requirements for area and power. However, as further discussed in Section B even weight-activation quantization may start to yield diminishing returns as narrower datatypes are used.
2.2 Accumulator-Awarr Quantization
Let Pa denote the minimum accumulator bit width required to guarantee overflow avoidance for given dot product. Aside from universally fixing the accumulator at 32 bits (or any other arbitrary maximum width imposed by a processor), the most conservative method to calculate P considers the width of the dot product operands. Given that inputs ø E A& and weights q E A are quantized, P is given by Eq. where L signed (Ø) is 1 if ø is signed and 0 otherwisc.
p = logg (
+1) +1
33)
Note that p increases linearly with the bit widths of the operands and logarithmically with the depth of the dot product. Thus, for a fixed neural architecture, one could heuristically manipulate the weight and activation bit widths according to Eq. 3 to reduce p" However, the quantization design space ultimately limits the minimum attainable accumulator bit width, as well the maximum attainable accuracy for any target accumulator bit width (Colbert et al. 2023 2024
Colbert et al. 2024 show that one directly target the accumulator bit width independent dimension of the quantization design space while still theoretically guarantecing overflow avoidance. When accumulating a"q q into a signed P-bit. accumulator, and assuming that Σ; = 0, one need only constrain ||/]] such that:
2P 2
|glli S
(4)
2 -
Motivated by this result, accumulator-aware QAT methods avoid overflow by constraining the fn-nrr of weights during training to ultimately restrict the range of dot product outputs during inference. Although these approaches have yielded promising results, their scope is limited to the QAT setting (Colbert, al. 2023 2024). To the best of our knowledge, ours marks the first formal study of accumulator-awan PTQ, and the first solution to scale to modern LLMss
3 Motivation
The quantization research landscape is slanted towards low-precision operands (f.e., weights and activations). However, low-precision operands reduce multiplication costs significantly more than addition costs. Thus, we hypothesize that, via Amdahl's Law (Amdahl 1967), this skewed focus will yield diminishing returns.
Indeed, recent works have already demonstrated that high-precision additions can bottleneck throughput, power, and area. For example, multiple studies have reported a 2x throughput increase when reducing the accumulator width from 32 to bits on general-purpose platforms (Khudia al.] 2018b] de Bruin al. 2020; Xie et al. 2021). Furthermore, when constraining operands to 3-bit 1-bit multipliers, Ni el. al. 2020 show that the cost of 32-bit accumulation consumes nearly 75% of the total power of their scalar MAC unit, reporting up to 3x power savings and 5x area reduction when reducing to 8-bit accumulation. Here, we further substantiate our hypothesis that reducing operand width will yield diminishing returns.
3.1 Reducing Operand Width Yields Diminishing Returns
Low-precision accumulation is not widely supported in modern hardware, substantiale our hypothesis using an accumulator-ane
variant of the bit operations (BOps) cost model (Van Baalen et al.] 2020 Hawks ct al. 2021), presented in Eq. 5 as a hardware-agnostic proxy for power consumption. For a fixed dot product size K, our cost model scales quadratically with the product of the operand bit widths M and but linearly with the accumulator width P, and increased weight sparsity S only reduces the cost. of additions.
BOps K x (M N S )) P)
(5)
4

---

Published in Transactions on Machine Learning Research (12/2025)
100
Baleriue Aoarncee Bil Pige
Ralatize Aceeaoe Hips.
Base
80
0% of Max Stall
-P--
Relltiie Aowerne Bit
(7 Mfx Total)
P-12
P-16
P-16
Pa-32
Pa:2
(15
14
40
40
20
20
20
30
50
60
70
80
90
100
20
30
40
60
70
80
90
100
Effective Operand Witth
Relative Bit Operations (%% Maxi
Relative Br Op Operations 15% Maxi
Figure 2: Left: Using average bit flips as power proxy, the cost of additions (adds) begins to dominate that of multiplications (muls) as the effective operand width (vM x NN) is reduced below 4 bits in 128-element vector MAC with a fixed 32-bit accumulator. Center: When varying the vector size K, accumulator width P, and operand widths M and N, our cost model (circles) shows a strong correlation (black trendline) with average bit. flips. Right: The baseline multiplication cost model (crosses) is unable to account for the benefits of reduced accumulator precision (blue trendline).
To the best of our knowledge, van Baalen et al. (002)) made the first connection between BOps and power by correlating multiplication costs (K x M x N) with the power consumed when executing vision models on an NPU. We extend their cost model and analysis to include the impact of accumulator-awan
quantization on power consumption. To support our accumulator-aware variant, we used Yosys (Wolf 2016 to synthesize several integer vector MAC designs while varying K E {32, 64, 128]. M, E 43,4,55, 6, 7,8), and P (10, 12, 16, 32}. We then used the Arbolta simulator (Redding et al.) 2025 to count bit flip while passing discrete random Gaussian data through each synthesized design. For each P and N, we constrain the random weights according to Eq. which incidentally increases sparsity S (Colbert et al. 2023 2024
As shown in Figure 2 our cost model exhibits strong 96% correlation with all observed data while the baseline multiplication cost model is unable to account for the benefits of reducing accumulator width. Interestingly, we observe that addition costs begin to dominate multiplication costs when the effective operand width (v x N) falls below bits. Moreover, our BOps cost model is consistent with the data presented by Ni et al. 2020, whereby reducing the accumulator width from 32 to 8 bits in a scalar MAC unit with a 3 x multiplier resulted in 3x power savings our model predicts 3x exactly when assuming 25% sparsity (Le., S ==000 0.25). Thus, rescarchers continue to stabilize 4-bit weights and activations (Ashkboos et al. 2024 Liu et al. 2024 Zhang al. 2025b), suspect that neural network quantization will reach this inflection point in the near future, suggesting accumulator-awarr quantization will be in the critical path for optimization as the cost of additions begins to overtake that of multiplications.
3.2
Limiting the Risks of Low-Precision Accumulation
Reducing the cost of additions is commonly done by reducing the accumulator bit width, which exponentially increases the risk of overflow, often introducing numerical errors that degrade model accuracy (Ni et al, 2020 Colbert et al. 2023). Existing methods that prepare quantized models for low-precision accumulation often aim to either reduce the risk of overflow (Xie et al.] 2021 Li et al. 2022) or mitigate its impact on model accuracy (Ni et al. 2020). These empirical approaches rely on assumptions that limit their real-world applicability. First, empirical estimates of overflow rely on a priori knowledge of the input distribution, which is often impractical to assume and can even introduce vulnerabilities (Baier et al., 2019h. Second, overflow behavior can vary across platforms and programs, so designing methods mitigate the detrimental impact of one particular behavior (e.g., wraparound two's complement arithmetic) limits portability. Finally, empirical approaches are unable to support applications that require guaranteed correctness, such as encrypted inference (Lou & Jiang 2019 and are known to break down when overflows occur too frequently (Ni et al.
2020 Colbert et al. 2023 Thus, avoiding overflow improves reliability, portability, and model quality.
2Bit. flips are known
effective proxy power consumption in both compute (van Baalen al. 2022 and
memory (Bittman al. 2018
5

---

Published in Transactions on Machine Learning Research (12/2025)
only apply EP- init (Colbert et al. 2024) to the PTQ setting without modification. However, E.P-init has two shortcomings: (I) it relies on mounding-to-zers to ensure [Q(w;) S |aa] for all i, which is known to introduce catastrophic errors in PTQ (Nagel et al.) 2020): ; and (2) it is a channel-wise projection that is not amenable to error correction, as discussed Appendix D.2, In Section 5.1 we show that AXE better preserves model accuracy as the accumulator width is reduced, and yiclds a new Pareto frontier for power-efficient PTQ methods.
From the family of existing accumuultoo-aave aware QAT methods that avoid overflow,
4 AXE: A General Framework for Accumulator-Aware PTQ
In the standard PTQ sctting, one often assumes the quantizer paramcters are fixed (d.c., scaling factor s and zero point =) and that the individual weights can move freely (Lybrand & Saab, 2021: Frantar el al.j 2022p. Building from these assumptions, we formalize accumulator-aware PTQ with the objective function in Eq. 6 where the optimal quantized weights q" minimize local quantization error while also satisfying an accumulator-awan aware f--rr constraint, where Z is given, up to scaling, by Eq.
1 | w g12 s.t. 11/11 S
q = arg min 2EAR 2
(6)
In particular, the constraint ||al|1 S Z ensures, via Hölder's inequality (Hardy et al., 1952). that any inner product | ql remains appropriately bounded, as long as || is bounded. To approximately solve this accumulator-consttaanee reconstruction problem, we introduce AXE-a flexible accumulator-ave quantization framework that endows overflow avoidance guarantees to the family of layerwise PTQ algorithms that greedily assign bits femment-y--eemeed (e.g., GPFQ and OPTQ).
We present AXE as the following composition of functions:
$ = Q a V Var.nd IIx
(7)
which acts on the (possibly error-corrected) weights, shown in Algorithms [ and 2 AXE provides catumllatto-waaeness by first projecting its argument onto the l ball of radius X via II,, then greedily clipping the result to the range [0;-1, via V Va--1 and finally quantizing it to the alphabet A, as presented in Section 4.2 Here, 11 is a per-channel, or per-tile, penalty that discourages the underlying algorithm from opportunistically selecting quantized weights with high magniludes, and V is a strict per- element constraint that greedily limits the range of each selected quantized weight while error is iteratively corrected. The resulting set of quantized weights is then guaranteed to avoid overflow when accumulating its inner product with any X E A A into P-bit signed registers.
In its coarsest form, AXE applies these constraints per-channel so that each dot product in the network is guaranteed to independently avoid overflow. Furthermore, without violating our constraints, we design AXE to support multi-stage accumulation in the form of tiled dot products by applying our constraints in finer gramularities. Without loss of generality, we theoretically justify our solution using GPFQ, then provide accumulator-aware variants of GPFQ and OPTQ in Algorithms D and 2 respectively. We highlight that, to ensure || is bounded, our accumulator-awarr variants of GPFQ and OPTQ require quantizing activations.
4.1 Accumulator Constraints without Zero-Centering
Our goal with AXE is to provide a theoretical guarantee of overflow avoidance when accumulating the dot product of q by any & e A into a signed P-bit register. To this end, if q is a zero-centered vector such that Σ, 9- 0, then it is sufficient to constrain ||q|| to satisfy the upper bound given by Eq. 4 However, enforcing such zero-centering constraint on vector of integers is non-trivial in practice.
For any ã E A each element ã; lies within the closed interval [e, v] for all i = (1.... ,K}, and リ- =- ====21 -1. It follows that the maximizing vector, u = arg max,, x" q, and the minimizing vector, v = arg ming x" q, are
v. where qi 2 and
H. where q, 2 0
03-
(8)
Me where qi <
U, where di <<
6

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 1 Accumulator-Aware GPFQ. Our accumulator-awary GPFQ variant quan- tizes W to M bits given input activations X and their N-bit quantized counterparts X. Note that W,, V ER Q, AM X; E RD, and X, E AR. all interpreted as row vectors.
Require: W RRKC X E ERRDD X E E
1: Q - 0E AM
R
Quantized output Per-sample quantization error Initialize running sums
2: U 0E D Pxx
3: + ACR, b+ BER
4: À < deriveThreshold
Derive per-channel Lagrangian thresholds
5: for i i 1,.... K do
Wi W. (X,.
X
Adjust for quantization error Accumulator-awarg projection & clipping Quantize weight
||;111
7:
V, Vab (W;
8:
Q: f Q(V))
9:
< Qi ④ 1Q,20
Update positive range Update negative range
10:
b b- Qi 0 10,50
11:
U+ X. W. X,Q:
Update quantization crror
12: end for
13: return Q
Fundamentally, to avoid overflow when accumulating x² into a P-bit register, the result needs to fall within the register's representation range for any & E A Without loss of generality, we derive our algorithm assuming a sign-magnitude accumulator for clarity and concisencss. Thus, to safely use a signed P-bit accumulator without overflow, both the following inequalities need to be satisfied:
u 2=9=2'= 22-1--11 - 2
To avoid zero-centeri centecenn,,,ng, one could generalize the result derived by Colbert et al. 2024 such that the bound relies on variable center, e.g., Σ, 4: = c. However, this precludes the use of greedy sequential algorithms where would be just as difficult to enforce as zero-centering, i.e., = 0. Thus, rather than constraining the center, we greedily constrain the boundaries, as further discussed in Section 4.2
4.2 Accumulator-Aware GPFQ
At the 1-th layer, GPFQ greedily selects each element zi to minimize the squared distance wetw the rumning sum Σ/=1 X, and its analog Σj=1 wy X, such that
9:
arg min
Σ
x0
(10)
PEAM
where _ denotes samples for the i-th input neuron the 1-th layer assuming the first. / - layers are quantized, and Ay is M-bit fixed alphabet defined by the target quantization space. This simplifies to the following itcration rule, as derived by Lybrand & Saab 2021 where un
{X
2 + 20?
((9)
9.0 -
(11)
X 1
113
u, W w
(1)
x
1 X
(12)
Soft f-norm regularization penalty. By design, greedy sequential quantization algorithms (e.g., GPFQ and OPTQ) opportunistically alter weights to correct for as much error as possible each step, often yielding high-magnitude quantized weights. However, this unfavorable the accumulator-aware PTQ sctting as high-magnitude weights consume more of the allocated & n-normm budget (see Eq. 4 To address this, we penalize high-magnitude weights throughout error correction via the soft e penalty proposed by [Zhng et al.
7

---

Published in Transactions on Machine Learning Research (12/2025)
which yields the & projection given by Eq. 13 where (-) denotes the rectified linear unit (ReLU), and À 0 is arbitrary tuneable regularization parameter.
2023
x sign(w) -
(13)
Noticcably, this formulation is amenable to leverage EP-init (Colbert et al. 2024 which takes the same functional form. Thus, we determine À as the Lagrange multiplier derived from the optimal Euclidean projection of w onto the & ball of radius Z, given by the following convex optimization problem, where v" is the weight vector that minimizes the Euclidean projection of w onto the boundary of our constrained set before quantization.
min
min 2 11 w subject to
||0|| S Z
(14)
An efficient solution to this problem is derived by Duchi al. (2008) ) Define p as the number of non-zero elements in the optimal projection and μ as the result of sorting the magnitudes of w in descending order, where wyl for i.j E (L."..,K)) The optimal Lagrange multiplier X is
===
Σ U3 -Z
(15)
Thus, (x) yields the optimal Euclidean projection onto our & ball before quantization. As such, it is important to note that, because this projection is derived before applying error correction algorithms (f.e. GPFQ or OPTQ), it cannot guarantee overflow avoidance on its own. Thus, we need our subsequent strict constraint; however, we observe our penalty consistently improves model quality (see Appendix D.2
Greedy f--nr constraint. For clarity, and without loss of generality, we motivate our strict constraint using the case where represented with unsigned integers such that μ == and 2 -1 Note that this is common when following activation functions with non-negative dynamic ranges.
Let denote the sum of all negative elements in q, and let 3 denote the sum of all positive elements in q. From Eq. we can similarly derive the upper bounds on 3 and -a in the case of sign- magnitudg representations Indeed, u² s 2P-1-11 is guaranteed whenever Bv + ap S 2 - 1, which holds in the case of unsigned activations if
8 2P-1 -
(16)
2 -
To accumulate at P bits layer without overflow, we greedily control the dot product range via
a (x) clip (x; a (.60)
(1)
(17)
( A ai (00 = B B 3.
(18)
where o; denotes the sum of all negative elements in whose index is less than i and B is its positive counterpart, and 41)) and ( (defined in Eq. [9) are respectively the upper limits of a; and 3, The range greedily enforced on becomes the closed interval defined by Eq 18 By independently constraining these, our accumulator-awart variant avoids overflow without explicit zero-centering. To ensure rounding errors do
not compromise Eq. 16 we use
2P-1 2N 11 max (A)
- = BL))
(19)
where max (A) denotes the worst-case difference in magnitude caused by rounding. We note that, while our derivation considers the sign-magnitude representation for its symmetry, the separate consideration of A() and (!)) is useful for asymmetric representations (e.g., two's complement).
At each step i, this yields the function composition &; presented in Eq. 7 which can generally be extended to the family layerwise adaptive rounding algorithms that includes GPFQ, OPTQ, Qronos (Zhang et al., 2025bb, and others. Focusing on the former, we present the pseudo-code for our accumulator-awarc variants of GPFQ and OPTQ in Algorithms 1 and 2 respectively, where we define l.)(()) to denote the clipping function applied elementwise so that (V),b())); = Vajdy (1z). Note that. Algorithm in Appendix A
8

---

Published in Transactions on Machine Learning Research (12/2025)
4.3 Multi-Stage Stage Accumulator-Awan Quantization
Our accumulator-awart constraints can be generalized to target customized datapaths beyond user-specific accumulator bit. widths. To this end, we design AXE to support multi-staged accumulation as visualized in Figure 3 In such a scenario, our constraints are enforced on the quantized weights in tiles of size T so that each partial dol product can be concurrently compuled by an alomic MAC unit. Let Pr and Po denole the inner and outer accumulator bit widths, respectively. If a K-dimensional dot product is executed in tiles of size T, where cach tile constraincd to P/-bit accumulator, then the minimum Po required to guarantce overflow avoidance is
Po = Pr + log, (K) (og)
(20)
The benefits of multi-stage accumulation are well-established. Khudia et al. 2018b report a 2x throughput uplift on compute-bound workloads by accumulating at 16 bits 64-clement tiles instead of at 32 bits, albeit without any theoretical guarantees overflow avoidance. Currently, inference libraries such as FRGEMM (Khudia al. 2018a). XNNPACK (Dukahn & Barchard] (021); and Ryzen AT (AMD) 2024) typically disable this optimization if overflows are observed too often during testing. To our knowledge, AXE provides the first mechanism to simultancously quantize and constrain pro-trained model for low-precision multi-staged accumulation while guaranteeing overflow avoidance, safely enabling this optimization for the first time. As shown in Section 5 this generalization is critical in maintaining the quality of billion-pan parameter LLMs, which often have dot products containing more than ten thousand elements.
Acc
Adder
MAC Unit
(a)
(b)
Figure 3: We visualize abstractions of (a) an atomic MAC unit and (b) parallelized multi-staged accumulation.
5 Experiments
The core contribution of this work is enabling a user to propare a quantized model for low- www-prccision accumulation the PTQ setting via AXE. We note that the computational and memory overhead introduced by our accumulator-aware constraints is negligible practice. Thus, our primary comparison metric is preserving model quality in challenging accumulator-aware PTQ scenarios.
Models & Datasets. We conduct experiments GPT2 (Radford et al. 2019 OPT Zhang et al. 2022a SmolLM2 (Allal et al.) 2024). Pythia (Biderman et al. 2023), and Llama3 (Dubey et al.) 2024 models using Wiki'Text2 (Merity et al.) 2016)) for calibration. While perplexity serves as a standard and sensitive proxy for generative quality, we complement it with an analysis of zero-shot accuracy to demonstrate
generalization. When analyzing zero-shot accuracy, we use LightEval (Fourrier al. 2023 to evaluate reasoning tasks: ARC-challenge (Clark et al. 2018). HellaSwag (Zellers et al.J 2019 PIQA (Bisk al. 2020), and Winogrande (Sakaguchi et al., 2021). We note that LLM evaluation is increasingly complex, and leave additional deeper analyses (such as impact reasoning capabilities) for future work. Moreover, to demonstrate that our approach works well across neural architectures and tasks, we provide additional experiments with image classification models in Appendix
Quantization Design Space. We constrain our design space to uniform- precisson models such that every hidden layer has the same weight, activation, and accumulator bit width, respectively denoted as M, N, and P. We consider 3- to 8-bit integers for both weights and activations, unike Frantar et al. (2022) and Zhang et al. 2023), which focused weight-only quantization. Rather than evaluating each combination of M
9

---

Published in Transactions on Machine Learning Research (12/2025)
and N, we restrict ourselves to configurations where N 2 M to reduce the cost of experimentation as such configurations tend to dominate the Pareto frontiers (Colbert et al. 2024). We implement our methods using the open source Brevitas quantization library (Franco et al. 2025), and quantize all models using a single AMD MI210 G with 64 GB of memory. We include more implementation details in Appendix D
5.1 Pareto Analysis
We first consider the scenario in which QNNs are optimized for coaumulatoo-constraiini processors in the PTQ setting. As discussed in Section 2.2] one could heuristically manipulate M and N according Eq. 3 To the best of our knowledge, EP-init serves as the only alternative for accumulator-aware quantization in the PTQ setting. Therefore, we use EP-init and naïve bit width manipulation as our baselines.
In Figure 4 use Pareto frontiers to visually characterize the trade-off between accumulator bit width P and WikiText2 perplexity for both GPFQ and OPTQ, respectively, across a range of models. We assume a monolithic accumulator in these experiments (i.e. P P1 = Po). For each model and each PTQ algorithm, the Pareto frontier shows the best perplexity observed for a target accumulator bit width P when varying M and N within our design space, with the full-precision floating-point model accuracy provided for reference. Additionally, in Figure ! we visually characterize the trade-off between accumulator bit width P and relative bit operations when quantizing SmolLM2 with GPFQ. Again assuming monolithic accumulator, the left. Pareto frontier shows the lowest observed perplexity for each overflow avoidance method we reduce P while varying M and N within our design space with the perplexity of the full-precision floating-por point model provided for reference. The right. Pareto frontier similarly shows the lowest perplexity observed for a given amount of bit operations relative to W8A8 with 32-bit accumulation, which we demonstrate is a strong proxy for power consumption in Section 3 Thus, our results show that AXE establishes a Pareto-dominant frontier for both accumulator bit width and power consumption, which we further support in Appendix B using additional experiments with image classification models.
We provide a detailed breakdown of each Pareto frontier in Appendix D where we report the perplexity of each Pareto-dominant model, their weight and activation bit widths, and resulting unstructured weight sparsity. Overall, we observe trends that are consistent with [Cobbrr et al. (2024); the Pareto-optimal activation bit width N decreases as P is reduced, and sparsity conversely increases. This suggests that our accumulator-aware boundary constraints obey similar mechanics as the f--nrr constraints of QAT methods, as our theoretical justification predicts. Moreover, as in the QAT setting, the quantization design space ultimately limits the minimum accumulator bit width attainable via naive bit width manipulation.
Interestingly, Figure 1 shows that EP-iint breaks down on SmoILM2 when weights are quantized below 5 bits, likely because EP-init relies on rounding-to-zers which is known to introduce catastrophic quantization errors in PTQ settings (Nagel et al. 2020). We highlight that this breaking point is before the inflection point we observe in Section 3.1 where the cost of additions overtakes that of multiplications with 4-bit operands. Thus, naive bit width manipulation dominates EP-init in power efficiency, which is consistent with our theory.
5.2 Scaling Analysis
The 4-norm of an unconstrained weight vector inherently grows as its dimensionality increascs. This suggests that accumulator-aware quantization scales well to strictly deeper neural architectures since the constraints tighten with width rather than depth; experimental results on the ResNet family support this hypothesis (Colbert et al.) 2024). However, this also suggests that accumulator- poorly in neural network families that grow in width, as is the case in transformer architectures (Zhang et al., 2022a: Biderman et al. 2023). Thus, to scale our accumulator-aware PTQ framework to billion-parameter language models, we turn to our multi-stage accumulation variant of AXE, as introduced in Section 4.3 Here, one assumes the partial sums of a dot product are concurrently computed in fixed-length tiles of size T. Our goal in this setting is to minimize perplexity for a target inner accumulator bit width Pt that is assumed to be universal across all tiles. Hence, our accumulator width is constant even models grow wider.
quantization scales
3 AMM,, AMD Instinct, and combinations thereof are trademarks of Advanced Micro Devices, Inc.
10

---

Published in Transactions on Machine Learning Research (12/2025)
OPT-125M
GPT2-137M
Pythia-160M
10
GFFG+AXE GPEGUERINE GPPQ
GPFO-AXE E CPPQ
GPPG-EP-ini
CPPQ
14
26
1.
22
24
16
20
22
26
2.5
14
18
20
24
25
28
Accumalater Bin Witth
Arcumtlllte Whdth
Accummleter Bit Width
GPT2-137M
Pythia-
10a
OFTC+AXE OPTQUEMNiK
OPTO-AKE OPTQ+ENint OPTQ
OPTO-AXE
OPTQ-EP-ING
OPTO
10a
107-
301
16
1.6
22
26
24
30
Figure 4: We show that AXE (green circles) yields the best trade-off between accumulator bit width and WikiText2 perplexity for several language models, namely OPT-125M, GPT2 (137M), and Pythia-160M. Note that we also show this for SmolLM2-135MM in Figure [] We compare AXE with EP-init (blue triangles) and naive bit width manipulation (red stars) using either GPFQ (top) and OPTQ (bottom).
Rather than exploring the full quantization design space, focus 4-bit weights and 8-bit activations (W4A8) to maximize utility across platforms with reasonable number of experiments
prior studies have
established this configuration is generally useful (Dettmers & Zettlemoyer 2023 Li et al. 2024). We evaluate AXE on top of both GPFQ and OPTQ using tiles of 128 elements under 16- bit accumulator constraints (note that P7 = 20 when T = 128 for W4A8 via Eq. 3r. Prior work has also cstablished 128 to a generally useful tiling size: AVX-512 ISA supports T 32 elements (Khudia et al. 2018a), Ryzen AT NPUs support
T 64 elements (AM 2024), and many works allocate scaling factors groups of 128 elements (Lin et al.,
2023
Liu et al.
2024h. For these experiments, apply Hadamard-basee incoherence processing (Ashkboos
al.
2024
Tseng et al. 2024 to mitigate the impact of outliers when quantizing activations in LLMs.
We focus our scaling analysis on the Pythia model suite, which was specifically designed to facilitate such a study (Biderman et al. 2023).. From our results in Table 1 we observe that, as model size increases, the quality of the 16-bit constrained models approaches that of the 32-bit baselines -aaiiness-AAE -AXE preserves 99% of the relative perplexity for Pythia- for both GPFQ and OPTQ, compared to 74% and 59% for Pythia-70M, respectively. We similarly observe that the gap reduced between the 16-bit constrained models and their FP16 counterparts model size macreassswhhe performance for both GPFQ and OPTQ, compared to the respective 54% when quantizing Pythia-70M, impressive +42% incrcase in relative perplexity when scaling from 70M to 12B parameters. Under the scaling hesos, this suggests the narrowing accuracy gap is in part because model capacity is growing without tightening the constraints since T held constant even as K increases (Pythia- 12B is at most 10x wider than Pythia-70M).. In Appendix D.2] we provide an ablation study targeting monolithic 16-bit accumulator (i.e., Po 16). There, we show the gap conversely increases constant via tiled multi-stage accumulation is critical in LLM scaling.
quantizing Pythia-12B. AXE preserves 96% of the FP16
increases, confirming that keeping P
11

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: We report the WikiText2 perplexity results when evaluating AXE on Pythia models quantized
W4A8 for 32-bit or 16-bit accumulation tiles of 128 elements using either GPFQ or OPTQ with Hadamard baased incoherence processing, We use x28x 16b denote Pr 16 and T 128, from Eq. 20
70M
160M
410M
1.0B
1.4B
2.8B
6.9B
12B
Float16
41.1
23.7
14.1
11.7
10.5
9.2
8.3
7.7
128x32b
56.8
35.2
19.4
12.3
11.2
9.5
8.6
7.9
GPFQ
128x 16b
76.4
55.5
23.2
12.7
11.8
9.8
8.7
8.0
128x32b
50.6
32.7
22.4
12.4
11.4
9.5
8.5
7.9
OPTQ
128x 16b
85.6
75.5
34.5
13.1
12.4
9.9
8.6
8.0
Table 2: We report the WikiText2 perplexity when evaluating Llama3 models quantized to 16-bit accumulation in tiles of 128 elements with either OPTQ GPFQ. We also demonstrate that AXE is compatible with pre-processing algorithms like SmoothQuant and Hadamard-based incoherence processing (HIP). We compare AXE (in bold) with a bit width manipulation baseline (Base) and provide the reference FP16 results. We use 128 16b denote P1 16 and T 128 from Eq. similarly denoting P1 32 as 128 32b.
3B
8B
Float16
9.1
6.5
128x32b
W4A8
AXE
W4A8
AXE
W4A8
AXE
GPFQ
23.5
23.5
10.8
10.8
20.6
20.6
GuaQuSnoottQuant
15.0
15.0
10.3
10.3
7.6
7.6
GPFQ HIP
12.9
12.9
9.7
9.7
6.9
6.9
OPTQ
45.8
45.8
12.7
12.7
7.8
7.8
SPPO+SSmoothuaaa
14.6
14.6
10.3
10.3
7.5
7.5
OPTQ+HIP
12.8
12.8
9.7
9.7
6.9
6.9
128x16b
W4A4
AXE
W4A4
AXE
W4A4
AXE
CPFQ
inf
22.1
inf
14.8
inf
37.9
GPFQ Smoothhuunt GPFQ+HIP
inf
15.4
inf
10.5
inf
7.7
16.1
13.8
10.8
10.0
8.0
7.1
OPTQ
33.3
inf
14.1
inf
8.3
SmootSSmothQuam OPTQ+HIP
inf
15.0
inf
10.7
inf
7.6
15.6
14.0
10.6
10.0
7.8
7.1
5.3 Zero-Shot Analysis
We conclude our experiments by evaluating zero-shot reasoning on Llama3 instruction fine-tuned models. again focusing on constraining W4A8 models for 16-bit multi-stage accumulation. As discussed in Section 5.2 multi-stage accumulation critical to scale accumulator-awen (see Appendix D.2 for ablations). Therefore, as EP-init does not support multi-stage accumulation, the only existing alternative for accumulator-awarre PTQ bit width manipulation. Note that, via Eq. 3 W4A4 guarantees overflow avoidance for 16-bit accumulation in tiles of 128 elements. Therefore, we compare AXE to W4A4 as baseline when constraining W4A8 model to target 16-bit accumulation in tiles of 128 elements (denoted as 128 x 16b); note that this corresponds to T 128 and Pı 16 in Eq. 20 We compare against. 32-bit accumulation, which we similarly denote as 128 x 32b.
PTQ to increasingly large language models
Since our primary comparison metric is preserving model quality in challenging accumulator PTQ scenarios, we use ostablished PTQ methods that solve orthogonal problems with the intention to create high-quality reference basclines. To this end, we demonstrate compatibility with equalization methods such SmoothQuant, (Xiao et al. 2023) and rotation-based methods such Hadamard-based incoherence
12

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: We report the average accuracy on zero-shot reasoning tasks when evaluating Llama3 models quantized 16-bit accumulation in tiles of 128 elements with either OPTQ or GPFQ using Hadamard- incoherence processing (HIP). We report the bit width manipulation baselines alongside the AXE results (in bold) and provide the reference FP16 results.
1B
3B
8B
Float16
46.5
53.8
62.4
128x32b
W4A8
AXE
W4A8
AXE
W4A8
AXE
GPFQ+HIP
45.4
45.4
53.3
53.3
60.2
60.2
HII
45.0
45.0
52.1
52.1
59.7
59.7
128x16b
W4A4
AXE
W4A4
AXE
W4A4
AXE
GPFQ+1IIP
41.4
45.1
50.1
51.2
57.3
58.6
OPTQ+HIP
42.1
44.5
49.9
51.1
56.1
59.3
processing (Ashkboos et al. 2024 Tseng et al. 2024
We provide perplexity results in Table 2 along with
the FP16 reference perplexities.
AXE has the desired feature of being functionally equivalent to the base algorithm when the accumulator is large enough. As such, one should expect benefits to manifest most when targeting low-precision accumulators. Indeed, we observe that AXE improves performance the challenging 128 16b setting for all compositions of PTQ algorithms, with fadamardd-asse incoherence processing (HIP) establishing the strongest baseline. Therefore,
this composition of PTQ algorithms to evaluate zero-shot reasoning under accumulator constraints. We provide our results in Table 3 along with the FP16 reference accuracies.
Coupled with HIP, AXE enables low-precision accumulation for Llama3 with minimal degradation from the 32-bit baselines, preserving 98% of the relative 8B perplexity for both GPFQ and OPTQ. Furthermore, the gap between the 16-bit constrained models and their FP16 counterparts decreases as the model size increases; AXE preserves 92% of the relative perplexity for 8B compared to 86% and 84% for GPFQ and OTPQ, respectively. This result is consistent with the scaling hypothesis in Section 5.2 Finally, AXE preserves up to 95% of the FP16 zero-shot performance and 99% of the 32-bit baselines, which is up to +3% better than bit width manipulation, the only existing alternate solution that scales to billio-ppraamtter LLMs.
6 Conclusions
The cost of additions overtakes that of multiplications as operand precision is reducod, as shown in Section 3 As few-bit. integer representations are increasing in popularity, one expects further reducing weight and activation precision to yield diminishing returns. However, reducing the cost additions is non-trivial due the complexities of system design and risk of numerical errors. While prior work on accumulator-wwen quantization has been limited QAT, ours marks the first solution that extends accumulator-awaan PTQ setting and scales billinn-araameter LLMs. We demonstrate the flexibility of AXE by presenting and evaluating accumulator-awan variants of GPFQ and OPTQ. Furthermore, unlike prior accumulator-awarr quantization methods, which assume monolithic accumulator, we design AXE to support multi-stage accumulation for the first time. Our experiments demonstrate that AXE establishes a new state-of-the-art for accumulator-awwn PTQ, yielding a Pareto-dominant frontier the trade-off between power and accuracy.
the
Statement of Broader Impact
As researchers stabilize model quality with weights and activations at 4 bits or less (Ashkboos et al. 2024 Liu et al.] 2024 Ma al.) 20224 Liu et al.] 22225] Zhang et al.) (025b), our work shows that adding the accumulator as a new dimension the quantization design space can yield new Pareto frontier that balances power consumption and model quality during inference; reducing power consumption may reduce carbon
13

---

Published in Transactions on Machine Learning Research (12/2025)
emissions [Luccioni et al. 2023), and alleviate power and thermal bottlenecks in LLM serving (tojkooicc et al. 2025 J. In addition to these potential benefits, reduced accumulator precision presents new set of practical deployment risks (see Section 3.2p. Our work limits these risks by theoretically avoiding overflow and its impact on model quality. However, reducing accumulator precision, like that of weights and activations, invariably reduces the maximum attainable model quality (as evidenced by our Pareto analysis). We note that lossy compression methods (e.g., quantization, pruning, distillation) can shift model behavior in subtle ways. Thus, it is important for compressed models to be further evaluated before deployment.
References
Loubna Ben Allal, Anlon Loshkov, Elie Bakouch, Gabriel Martín Blázquez, Lewis Tunstall, Agustín Piqueres, Andres Marafioti, Cyril Zakka, Leandro von Werra, and Thomas Wolf. SmolLM2 with great data, comes great performance, 2024.
AMD. Ryzen AI column architocture and tiles. https://riallto.a1// ai/3 http://aaalao...33....aaa.aa...a Syyzenai_capailittieee 2024. [Accessed 06-30-2024).
Gene M Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18-20, 1967, spring joint computer conference, pp. 483-485, 1967.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated llms. arXio preprinl WrXiv.2404000455 2024.
Lucas Baier, Fabian Jöhren, and Stefan Seebacher. Challenges in the deployment and operation of machine learning in practice. In ECIS, volume 2019.
Stella Biderman, Hailey Schoclkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohamnad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397 2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp- 7432 7439, 2020.
Daniel Bittman, Matthew Gray, Justin Raizes, Sinjoni Mukhopadhyay, Matt Bryson, Peter Alvaro, Darrell DE Long, and Ethan L Miller. Designing data structures to minimize bit flips on NVM. In 2018 IEEE 7th Non- Volatile Memory Systems and Applications Symposium
(NN (NVMSA), pp. 85-90. IEFE, 2018.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. QulP: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Peter Clark, Isaac Cowhey Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint N.Xxv:8803.05/57 2018.
Tan Colbert, Alessandro Pappalardo, and Jakoba Petri-Koenig. A2Q: Accumulator-aware quantization with guaranteed overflow avoidance. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 16989-16998, 2023.
Ian Colbert, Alessandro Pappalardo, Jakoba Petri-Koenig, and Yaman Umuroglu. A2Q+: Improving Accumulator-awant weight quantization. In Forty-first International Conference on Machine Learning, 2024.
Barry de Bruin, Zoran Zivkovic, and Henk Corporaal. Quantization of deep neural networks for accumulator- constrained processors. Microprocessors and microsystems, 72:102872, 2020.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical
image database. In 2009 TEFE conference computer vision and pollern recognation, pp. 248 255. leee, 2009.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learnöng, pp. 7750-7774. PMLR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11999, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 24072407.217555 2024.
John Duchi, Shai Shalev-Shwarlz, Yoram Singer, and Tushar Chandra. Elficient projections onto the 1 1-ball for learning in high dimensions, In Proceedings of the 25th international conference on Machine learning, pp. 272-279, 2008.
Marat Dukahn and Frank Barchard. Faster quantized inference with XNNPACK. https://blog. httpss//bllogteesrffoorr tensorflow html, 2021. [Accessed 06-30-2024].
Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. LightEval: A lightweight framework for Ilm evaluation, 2023. URL
Giuseppe Franco, Alessandro Pappalardo, and Nicholas J Fraser. Xilinx/brevitas, 2025.
URL https:
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learning Representations, 2022.
Godfrey Harold Hardy, John Edensor Littlewood, and George Pólya. Inequalities. Cambridge university press, 1952.
Benjamin Hawks, Javier Duarte, Nicholas J Frascr, Alessandro Pappalardo, Nhan Tran, and Yaman Umuroglu. Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference. Frontiers in Artificial Intelligence, 4:676564, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
IST-DASTab. gplq. http:://gtttbb..mm/ssssaaaaaaa//
2022.
Daya Khudia, Protonu Basu, and Summer Deng. Open-sourcing FBGEMM for stato-of tho---rttth--ttttt server-side inference. httpg//wwgggaaanacc..c.ccoooooop 30-2024].
2018a. [Accessed 06-
Daya S Khudia, Prontonu Basu, and Summer Deng. Open-sourcing for state-of-the-art server-side inference, 2018b. URL
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274 19286. PMLR, 2023.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datascts: community library for natural language processing. arxie preprint orXiw:2109.0255 028.46 2021.
Haokun Li, Jing Liu, Liancheng Jia, Yun Liang, Yaowei Wang, and Mingkui Tan. Downscaling and overflow- aware model compression for efficient vision processors. In 2022 IEEE 42nd International Conference on Distributed Compuling Systems Workshops (ICDCSW), pp. 145-150 IEFF, 2022.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. arXiv preprint wrXim:2422.88155 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. AWQ: Activation-awan aware weight quantization for LLM compression and accelcration. arXiv preprint arXi::2906.009 2023.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. SpinQuant-IIn quantization with learned rotations. arXiv preprint arXi::2405. 16406, 2024.
Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. ParetoQ: Scaling laws in extremely low-bit Ilm quantization. arXiv preprinl Xiv:2250 02631, 2025.
Qian Lou and Lei Jiang. She: A fast and accurate deep neural network for encrypted data. Advonces in neural information processing systems, 32, 2019.
Alexandra Sasha Luccioni, Sylvain Viguier, and Anne-Laure Ligozat. Estimating the carbon footprint of bloom, 176b parameter language model. Journal of Machine Learning Research, 24(253):1 15, 2023.
Eric Lybrand and Rayan Saab. A greedy algorithm for quantizing neural networks. The Journal of Machine Learning Research, 22(1):7007-70AA 2021.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xuc, and Furu Wei. The cra of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprinl ar 2iv:2400.. 17764, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint Xr0: 07843, 2016.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for cost-training quantization. In International Conference on Machine Learning, pp. 7197-7206 PMLR, 2020.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. white paper on neural network quantization. arXiv preprint orXiw:2106.0822 2021.
Renkun Ni, Hong-min Chu, Oscar Castañeda, Ping-yeh Chiang, Christoph Studer, and Tom Goldstein. Wrapnet: Neural net inference with ultra-low-resolution arithmetic. urXiv preprind ar 2rXiv:2007.132/ 2020.
Alessandro Pappalardo, Giuseppe Franco, nickfraser, Ian Colbert, Fabian Grob, Timothy Costigan, Oscar Savolainen, Andrei Stoian, jinchen, alexredd99, Anton Gerdelan, vfdev, derpda, Yaman Umuroglu, Tim Paine, Kenneth Witham, Saad Khan, Pablo Monteagudo Lago, Omar Peracha, MichalMachura, Luis Montero, Javier Duarte, and Felix Jentzsch. Xilinx/brevitas: Release v0.12.0, May 2025. URL
https://doi.org. 10. wttp://or///...../eeeeoo// 15375011
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alexander Redding, lan Colbert, Yaman Umuroglu, and Jakoba Peeri-Koenig, Arbolta: A framework for efficient hardware-software co-design, 2025. URL
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--06, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiol Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp.
2018.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Jovan Stojkovic, Chaojic Zhang, fñigo Goiri, Esha Choukse, Haoran Qiu, Rodrigo Fonseca, Josep Torrellas, and Ricardo Rianchini. TAPAS: Thermal-and power-aware scheduling for 1lm inference in cloud platforms. arviv preprint arXic:2501.0600 2025.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulP#: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396, 2024.
Mart Van Baalen, Christos Louizos, Markus Nagel, Rana Ali Amjad, Ying Wang, Tijmen Blankevoort, and Max Welling. Bayesian bits: Unifying quantization and pruning. Advances in neural information processing systems, 33:5741-515 5752, 2020.
Mart van Baalen, Brian Kahne, Eric Mahurin, Andrey Kuzmin, Andrii Skliar, Markus Nagel, and Tijmen Blankevoort. Simulated quantization, real power savings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2757-2761, 2022.
Clifford Wolf. Yosys open synthesis suite. 2016.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Stato-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087 38099. PMLR, 2023.
Hongwei Xie, Yafei Song, Ling Cai, and Mingyang Li. Overflow aware quantization: Accelerating neural net- work inference by low-bit multiply-accummlaa operations. In Proceedings of the Twenty-Ninth International Conference on International Joint. Conferences on Artificial Intelligence, pp. 868 875, 2021.
Gyeong-In In Yu, Joo Seong Jeong, Geon- Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). pp. 521-538, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint XrXiv:1905.0 07830, 2019.
Chi Zhang, Xu Yang, Shuangming Yu, Runjiang Dou, and Liyuan Liu. ISQ: Intermediato-value slip quantization for accumulator-aware training, IEEE Signal Processing Lellers, 2025a.
Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural networks with provable guarantees. STAM Journal on Mathemalics of Data Science, 5(2):373 009, 2023.
Shihao Zhang, Haoyu Zhang, Ian Colbert, and Rayan Saab. Qronos: Correcting the past by shaping the future... in pest-training quantization. arXiv preprint XXXiv25505.16695 2025b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXív preprint ar Xiv: 01068, 2022a.
Xinyu Zhang, Ian Colbert, and Srinjoy Das. Learning low-precision structured subnetworks using joint layerwise channel pruming and uniformn quantization. Applied Sciences, 12(15):7829, 2022b.
17

---

Published in Transactions on Machine Learning Research (12/2025)
A Accumulator-Aware OPTQ
Algorithm Accumulator-Aware OPTQ. Our accumulator-avan OPTQ variant quan- tizes W to M bits given H Cholesky(22X X + +nd)=²) where 7j is small dampening factor to avoid numerical issucs. Following Frantar et al. 2022 we set nf be 1% of the average diagonal value. Note that W,, V; E RC and Q: E A ir, all interpreted as row vectors.
Require: W& E RXXC H E RR
1: Q + 0 AxxC
Quantized output Per-channel quantization errors Initialize running sums
2: E 0 DERC
3: AE ARR b+ BERE
4: A + deriveThrrehool(WW))
Derive per-channel Lagrangian thresholds
5: for i = I.,..,, K do
6:
V, - Vab olx
Accumulator-awant projection & clipping Quantize processed weight
7:
Q: - Q(Vi)
8:
E + -W; Qi)/H,,
Calculate quantization error
1.3
9:
WeK W E H L
Update weights
10:
at Qi 0 1Q.20
Update positive range
11:
b+ Q; @ 1Q,50
Update negative range
12: end for
13: return Q
B Additional Results on Computer Vision Models
To further evaluate the robustness and generalizability of our approach, we include additional results diverse set of computer vision models namely R8 (He et al.. 2016), MobileNet (Sandler et al., 2018), and VIT (Dosovitskiy et al. 2020) on the ImageNet dataset (Deng et al. 2009). We again perform an extensive Pareto analysis within the quantization design space described in Section 5.1 Once again, AXE outperforms EP-init and bit width manipulation across models using either GPFQ or OPTQ. These experiments further demonstrate that AXE maintains consistent performance across varying architectures and tasks, underscoring its applicability beyond the primary benchmarks presented in the main text. We provide details of the resulting Pareto frontiers for each model and algorithm in Tables and 9
C Memory-Efficient GPFQ
As discussed in Section 4.2 GPFQ approaches the standard quantization problem by traversing the neural network graph to sequentially quantize each element in cach layer while iteratively correcting for quantization error. The derived ileration rule is formalized by Eqs. 11 and quantized weight q, depends on the inner product
12 In this standard formulation, the i-th
(X
(
w (1)
where x ,.(" E E are samples for the i-th neuron of the inputs to layer 1, and ui E RD is the running error from quantizing the first weights. Thus, at layer I, GPFQ requires collecting and storing 2D samples for the K₁ input neurons, and updating the running quantization error for each sample for the C output neurons. This implies potential difficulty scaling to larger models and larger calibration sets as the memory requirements are O(D x (2K, + Ct)). Indeed, assuming 128 samples with a sequence length of 2048 at 32-bit precision, Pythia-6.9B [Biderman et al. 2023) requires a peak memory usage of roughly 30 GB at the first FEN layer excluding pre-trained weights. We set out to reduce this overhead.
We start with the observation that OPTQ is far more memory efficient. OPTQ uses the Hessian proxy 2X X which be efficiently computed one sample at time and stored K₁ K₁ square matrix,
18

---

Published in Transactions on Machine Learning Research (12/2025)
ResNet18
MobileNetV2
70
60
60
00
GPTO-AXE
GPFD+AXE
GPPO-AXE
- GPrO-ENmit
- GPIO-EP-in
GRPG-EP-iNI GPTO
GPPQ
GPPQ
12
14
26
18
24
26
28
30
12
14
16
18
24
26
28
20
12
14
16
18
20
24
26
20
30
Accumallter Bit Whlth
Accumt Bit
Accumaddoor Bit Witth
70
80
Top: A.CuUSS
OPTO-AXE
OPTO-AXE
OPTO-AKE OPTQ-EP-INE OPTO
OPTQ-EP-nIz
OPTO+ER-IN
OPTO
16
18
30
20
Figure 5: We show that AXE (green circles) yields the best trade-off between accumulator bit width and ImageNet accuracy for several image classification models, namely ResNet18, MobileNet V2, and ViT-B-32. We compare AXE with EP-init (blue triangles) and naïve bit width manipulation (red stars) using either GPFQ (top) and OPTQ (bottom).
O(Ki Ki) memory requirement that is 36x less than GPFQ for Pythia-6.9B. Thus, we reformulate GPFQ to use square matrices via mathematical manipulation of singular value decompositions.
Theorem C.1. Let H (XXT)* 1/2 and = XX", For pre-trained weights W Rxc, quantization alphabet A, and GPFQ function of the form of Algorithom 1 it follows that:
GPFQ(W x, 8, GPFQ(W, GH H.A)
(21)
Proof. According to the iteration steps in Algorithm L it suffices show that the argument of quantizer is unchanged after substituting X,, X2 with (GH-1), and H, respectively. Specifically, at the i-th iteration of GPFQ(W,GH GH-1, "HH"HH,A) HH,), we have
H, (GH )))
HJU,-1 |
V++ W.
(22)
H
where the quantization error is given by
UL1 Σ GH " -H,,,,,,,,
(23)
Let E R denote the vector with in the i-th coordinate and 0's elsewhere. It follows from H (XXT) 1/2 and G T that
H. = ||e -
e =,,- X =1
H.GH - "
e G ej e Xxx ej X,XX,
19

---

Published in Transactions on Machine Learning Research (12/2025)
and
= H() H
xx
X
Plugging above identities into equation 22 and equation
23
wC obtain
X x 11
+ W
(24)
with Ū.-1= = Ejn -xy- Since V; in equation 24 is identical with the i-th quantization argument in GPFQ(W, X, X, A), both algorithms derive the same quantized weights Q: = 2(V.).
At layer 1, this memory-efficient GPFQ formulation requires collecting and storing G, H, and U, which are each Ki x K₁ matrices, reducing to an O(Ki x Ki) memory requirement that is 12x less than the standard GPFQ formulation for Pythia-6.9B. We leverage this functionally equivalent formulation for our LLM evaluations in Section 5.2
D Experimental Details & Ablations
D.1 Hyperparameters & Quantization Schemes
Below, we provide a detailed description of the quantization schemes and the specific hyperparameters used in our experiments. As discussed in Section S we consider pre-trained autoregressive language generation models that are respectively made publicly available via the HuggingFace (Wolf et al. 2020) librarics. All models are quantized via the Brevitas (Franco al. 2025) quantization library using single AMD MI210 GPU with 64 GB of memory.
We leverage the unmodified implementations of the various LLMs discussed in Section provided by HuggingFace (Wolf et al. 2020), as well as their pre-trained floating-point checkpoints and datasets [Lhoest et al. 2021). We use drop-in replacements for all linear layers in the networks except the embedding layer or final prediction head, leaving them at full-precision floating-point. As is common practice (Frantar et al. 2022) we build our calibration set using 128 samples randomly selected from the WikiText2 dataset (Merity et al. 2016) without replacement using a fixed sequence length of 2048 tokens for all models except GPT2 (Radford ct al.
2019 which is restricted to maximum sequence length of 1024 by the library.
When quantixing weights with OPTQ or GPFQ, we do so in descending order according to the diagonal value of the Hessian proxy (2XX* by our notation in Section 2 (IST-DASaab) Lin ct al.) 2023 Chee et al.) 2024). For GPFQ, we find that the peak memory utilization of the algorithm in its standard form ultimately limits its evaluation on billin-parameetr LLMs. Thus, we introduce a functionality equivalent memory-efficient reformulation to enable the algorithm to scale to larger models (see Appendix C which we use in our experiments. When inverting // in both OPTQ and GPFQ, we use the standard dampening factor of 1% of the average of its diagonal. We use the AXE variants of GPFQ and OPTQ introduced in Section 4 When evaluating EP-init, we do so after applying the baseline OPTQ GPFQ algorithms.
Implementation Details.
2022
Quantization Scheme. We quantize activations asymmetrically, tuning & to the lowest 99-th percentile based on the calibration data. While AXE not reliant on symmetric weight quantization, we eliminate zero-points in all weight quantizers such that 2 = O, as common practice so as to avoid computational overhead of cross-Lerms (Nagel et al.] 2021 Zhang al.) 2022b).. Throughout our experiments, we adopt. full-precision floating-pott scaling factors defined as max (2 1), where max(w) is calculated per-channel for the weights and per-token for the activations quantized for b-bit quantization.
To quantize our models, we first load the pre-trained checkpoint and merge normalization layers when possible. When applying SmoothQuant (Xiao et al. 2023) or Hadamard- incoherence processing (Ashkboos et al.) 2024), we do so before calibrating the scaling factors and zero-poii pointo.ostt...... When applying SmoothQuant, we perform light grid search over its a parameter and find a 0.4 to generally perform the best on average for Llama3, we use this for all models. We then apply either GPFQ or OPTQ (with or without AXE).
20

---

Published in Transactions on Machine Learning Research (12/2025)
D.2 Ablation Studies
Impact of error correction and choice of rounding function. Previous reports had suspected EP-init is limited by its reliance on the cound-tozeero (RTZ) rounding function (Colbert et al.) 2023 2024h, which has been shown to be a poor choice (Nagel ct al. 2020). AXE removes this reliance and also enables greedy error correction. We design an ablation study to isolate the impact RTZ and error correction. We quantize OPT-125M (Zhang al. (022a) and Pythia-160M (Biderman et al. 2023) to 4-bit weights and 8-bit activations while targeting 20-bit accumulation since our Pareto front shows this configuration to be both reasonable and challenging. We evaluate AXE with round-to-zero (AXE-RTZ) and AXE with round-to-nearest (AXF-RTN). We report the results in Table 4 We interpret the gap between EP-init and AXE-RTZ as the benefit of error correction, and the gap between AXE-RTZ and AXE-RTN as the benefit of the selected rounding function. We observe that error correction has a greater impact than rounding function selection for GPFQ, but we observe the opposite for OPTQ. Finally, we evaluate AXE with our hard constraint only (AXE-HCO); that ,,,-,, from Eq. 17 isolate the impact of our soft constraint, which is not necessary for guaranteeing overflow avoidance. We interpret the gap between AXE-RTN and AXE-HCO the impact of our soft constraint, which consistently provides improved maintained performance.
Multi-stage vs. monolithic accumulation. In Section 5.2 we analyze how our accumulator constraints scale to increasingly large language models within the Pythia model suite (Biderman et al. 2023) There, we discuss our observation that, as model size increases, the quality of the accumulator-constrriine models approaches that the unconstrained baselines for both GPFQ and OPTQ. This suggests the narrowing gap in perplexity in part because model capacity growing without tightening the constraints. To verify this, perform an ablation study largeting monolithic 16-bit accumulator (f.e., Pi = Po 16). We quantize all Pythia models up to Pythia- using either OPTQ or GPFQ, and report the results in Table 5 Not only do we observe siguificant instability, also observe 7.4x regression in perplexity between Pythia-70k and Pythia-1B confirming that fixing Pi improves scaling as models grow wider.
Table 4: We evaluate round-to-neared (RTN) and round-to-zero (RTZ) to directly compare against EP-init. We also evaluate AXE with our hard constraint only (HCO) to isolate the impact of our soft constraint. All models are quantized to W4A8 while largeting a 20-bit monolitic accumulator (i.e., Po = 20).
Algorithm
Model
EP-init
AXE-RTZ
AXE-RTN AXE-HCO
OPT-125M
8828.3
165.2
31.9
31.9
GPFQ
Pythia-160M
2500.2
211.0
43.0
49.2
OPT-125M
998.6
539.3
37.1
70.0
OPTQ
Pythia-160M
4524.4
1798.7
84.9
194.8
Table 5: We evaluate AXE using Pythia models quantized to W4A8 when targeting monolithic 16-bit
accumulator (i.e., Po = 16). Note that this is in direct contrast with Table ! which targets multi-stage accumulation (i.e., Pt = 16).
Algorithm
70M
160M
410M
1B
GPFQ
4397
7135
10496
32601
OPTQ
2438
4439
9759
34387
E
Pareto Frontier Details
We provide the detailed Pareto frontiers visualized in Figures and 5 for GPFQ and OPTQ, For each model, we report the perplexity, quantization configuration, and unstructured weight sparsity.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Table GPFQ: We provide the test perplexity (PPL) and quantization configuration the Pareto-optimal models that form the frontiers visualized Figure Note that M and N respectively denole the weight and activation bit widths.
GPFQ
GPFQ+EP-init
GPFQ+AXE
Model
P
PPL
(M,N)
Sparsity
PPL
(M,N)
Sparsity
PPL
(M,N)
Sparsity
16
13152.0
(3.4)
71.7
79.8
(4,5)
24.3
57568.0
(3,3)
10816.0
(3,4)
71.7
27.8
(4,6)
22.7
18
1075.0
(3,4)
48,8
283.7
(4,5)
38.5
22.5
(5,6)
13.2
171.5
(3,5)
40.3
137.8
(4,6)
35.5
19.0
(5.6)
10.1
Smo1LM2-135.M
20
61.9
(3,6)
38.0
126.4
(6.6)
13.0
16.0
(5.7)
9.9
(Float: 14.4)
21
23.4
(4.6)
19.9
19.8
(6,6)
9.9
14.8
(6,7)
5.0
22
19.0
(5.6)
16.3
(6.7)
9.7
14.2
(6,8)
4.9
23
16.0
(5,7)
9.9
15.1
(7,7)
1.9
14.0
(7,8)
2.6
14.8
(6,7)
(7,8)
4.9
13.9
(8,8)
32
14.0
(8,8)
1.2
14.1
(8,8)
2.4
13.9
(8,8)
1.2
16
9148.8
(3.4)
76.5
249.8
(3,6)
55.6
7624.6
(3.4)
72.7
91.2
(4,6)
37.9
18
11007.2
(3.3)
58.3
7471.2
(3,5)
75.5
41.8
(4,6)
27.8
9567.6 6
(3,4)
34.5
1059.3
(5.6)
32.3
(4,7)
27.0
OPT-125M
20
874.4
(3,5)
50.5
86.1
(5.6)
29.8
29.3
(5,7)
15.7
(Float: 27.7)
21
101.0
(3,6)
46.4
42.4
(5.7)
28.1
28.6
(5,8)
15.6
22
40.5
(4.6)
26.3
30.4
(6,7)
16.0
28.1
(6,8)
9.6
23
31.8
(1,7)
25.9
29.5
(6,8)
15.9
27.9
(6,8)
8.6
24
29.0
(5,7)
28.2
(7.8)
9.5
27.8
(7,8)
5.4
32
27.8
(8,8)
3.8
27.8
(8,8)
5.3
27.8
(8,8)
3.8
16
3345.8
(3,3)
93.2
552.4
(3,6)
55.4
17
2705.3
(3.6)
75.1
310.1
(3,7)
52.8
18
3760.3
(3.3)
82.3
1100.5
(4,5)
52.9
134.3
(4,7)
34.9
2782.2
(3.4)
402.9
(4.6)
47.3
67.5
(4,7)
25.6
GPT2-137M
20
742.4
(3,5)
55.3
213.2
(4.7)
44.3
40.4
(4,8)
24.5
(Float: 29.9)
21
356.2
(3,6)
48,8
85.2
(5,7)
24.9
33.2
(5,8)
13.2
22
189.9
(4.6)
26.4
46.3
(5,8)
23.8
32.1
(6,8)
7.3
23
65.8
(4,7)
24.7
34.2
(6.8)
13.0
31.8
(6,8)
6.3
24
39.8
(1.8)
23.8
32.1
(7.8)
7.1
31.5
(7,8)
3.2
32
31.5
(8,8)
1.6
31.6
(8,8)
3.2
31.5
(8,8)
1.6
16
4501.1
(3,4)
76.8
386.0
(3.6)
53.2
17
3095.1
(3.5)
72.5
198.6
(3,6)
46.3
18
9887.1 1
(3.3)
49.4
1070.2
(4.5)
46.7
74.5
(4,6)
25.1
1946.8
(3.4)
391.7
(4.6)
42.9
46.2
(4.7)
24.4
PyOMia-160M
20
456.2
(3,5)
47.8
117.5
(5.6)
23.6
34.6
(5,7)
13.3
(Float: 26.7)
21
198.3
(3.6)
45.1
78.5
(5.7)
23.4
32.4
(5.8)
13.3
22
69.6
(4.6)
23.5
48.6
(5.7)
21.2
30.1
(6,8)
7.8
23
44.4
(4,7)
22.6
37.2
(6,8)
13.0
28.2
(6,8)
5.5
24
33.2
(5.7)
11.3
31.8
(7.8)
7.4
27.6
(7,8)
2.8
32
27.4
(8,8)
1.4
27.5
(8.8)
2.7
27.4
(8,8)
1.4
22

---

Published in Transactions on Machine Learning Research (12/2025)
Table OPTQ:
We provide the test perplexity (PPL) and quantization configuration of the Pareto-optimal models that form the frontiers visualized in Figure Note that M and N respectively denote the weight and activation bit widths.
OPTQ
OPTQ+EP-int
OPTQ+ AXE
Model
P
PPL
(M.N)
Sparsity
PPL
(M,
Sparsity
PPL
(M,N)
Sparsity
16
3333.8
(4,5)
62.2
225.0
(3,6)
52.8
17
1722.6
(4,5)
53.6
80.2
(3,6)
45.7
18
9942.5
(3.3)
54.5
409.8
(5,5)
36.1
41.3
(4,6)
26.6
19
8278.3
(3,4)
47.5
136.0
(5.6)
35,7
35.0
(5,6)
15.1
OPT-125M
20
281.1
(3,5)
45.5
46.9
(5.6)
26.8
31.3
(5,6)
14.2
(Float: 27.7)
21
60.4
(3,6)
44.7
(5,7)
26.8
29.0
(5,7)
14.2
22
35.7
(4,6)
25.8
30.3
(6,7)
15.6
28.5
(5,8)
14.2
23
31.5
(5.6)
14.6
29.7
(6,8)
15.6
28.0
(6,8)
8.6
24
29.2
(5,7)
14.6
28.1
(7,8)
9.5
27.8
(7,8)
5.4
32
27.8
(8,8)
2.2
27.8
(8,8)
5.6
27.8
(8,8)
2.2
16
2765.6
(4,4)
52.6
1513.6
(4,5)
34.0
17
2465.0
(4,4)
49.0
496.4
(3,6)
43.4
18
1140.7
(3.3)
59.3
2465.0
(1,4)
49.0
117.9
(1,6)
24.2
19
2782.2
(3,1)
43.9
1108.4
(5.6)
31.5
59.9
(1,7)
24.2
GPT2-133M
20
2149.8
(4,4)
26.0
361.7
(1,7)
43.6
45.5
(5,7)
13.1
(Flost: 29.9)
21
1153.8
(4,5)
24.7
73.1
(5.7)
21.7
37.3
(5,8)
13.2
22
176.9
(4.6)
24.0
42.7
(5,8)
24.5
33.1
(6,8)
12.2
23
50.1
(4,7)
23.2
33.5
(6,8)
13.4
32.1
(6,8)
6.2
24
37.4
(5,7)
12.2
32.0
(7,8)
7.3
31.8
(7,8)
3.1
32
31.8
(8,8)
1.6
31.7
(8,8)
3.3
31.7
(8,8)
1.6
16
6739.6
(4,6)
79.7
1521.2
(3,5)
41.7
17
5345.7
(4,5)
49.9
311.7
(4,5)
22.9
18
27098.1 1
(3,3)
40.5
1372.4
(4.5)
41.1
126.1
(4,6)
23.1
19
5644.0
(3,4)
40.3
641.2
(4.6)
41.0
61.4
(4,6)
21.3
Pythin 160M
20
948.4
(3,5)
40.1
132.9
(5.6)
23.4
43.5
(5,6)
10.9
(Float: 26.7)
21
151.3
(4,5)
21.4
108.5
(5.7)
23.5
32.8
(5,7)
10.9
22
61.4
(4.6)
21.3
74.1
(5.7)
22.0
30.0
(5,8)
10.9
23
43.3
(5,6)
10.9
40.4
(6,8)
13.0
28.0
(6,8)
5.5
24
32.8
(5,7)
10.9
32.1
(7,8)
7.5
27.4
(7,8)
2.7
32
27.2
(8,8)
1.4
27.6
(8,8)
2.9
27.2
(8,8)
1.4
23

---

Published in Transactions on Machine Learning Research (12/2025)
Table 8: GPFQ:
We provide the top-1 test accuracy and quantization configuration of the Pareto-optimal models that form the frontiers visualized in Figure 5 Note that M and N respectively denote the weight and activation bit widths.
GPFQ
GPFQ+EP-init
GPFQ+AXE
Model
P
Top-1
(M, N)
Sparsity
Top-1
(M,
Sparsity
Top-1
(M,N)
Sparsity
14
0.2
(3.3)
50.8
(3,3)
56.9
15
1.0
(3.3)
74.3
58.1
(3,4)
58.6
16
10.8
(3.3)
63.3
(4,4)
40.4
17
43.9
(4,4)
57.2
66.5
(4,4)
29.9
ResNet 18
18
59.6
(4,4)
43.5
67.9
(4,5)
30.7
(Float: 69.8)
19
54.5
(3,3)
47.4
62.0
(4,5)
44.1
68.9
(5,5)
16.7
20
62.0
(3,4)
49.5
67.3
(5.5)
27.6
69.3
(5,6)
16.8
21
66.5
(4,4)
29.8
67.9
(5,6)
27.8
69.5
(6,6)
8.7
22
67.9
(1,5)
30.5
69.1
(6,6)
15.7
69.7
(6,7)
8.7
32
69.8
(8,8)
2.3
69.7
(8,8)
4.4
69.8
(8,8)
2.3
14
5.2
(4,4)
17.6
15
35.1
(4,5)
16.7
16
0.1
(3,3)
33.8
0.3
(4.4)
26.5
51.4
(4,6)
16.6
17
0.9
(3,4)
30.2
18.2
(5.5)
16.7
61.7
(5,6)
9.6
MobileNeLV2
18
6.4
(3,5)
28.8
49.0
(5.6)
16.8
67.2
(5,7)
9.7
(Flott: 71.9)
19
37.6
(1,5)
15.2
60.9
(6.6)
10.1
68.3
(5,7)
7.9
20
54.1
(4,6)
14.9
66.7
(6,7)
10.3
69.9
(6,8)
5.9
64.3
(5,6)
8.0
68.2
(7,7)
6.5
70.6
(7,8)
4.2
22
68.3
(5,7)
7.9
69.7
(7,8)
6.4
70.7
(7,8)
2.3
32
71.0
(8,8)
1.4
70.8
(8,8)
71.0
(8,8)
1.4
16
0.1
(3.3)
89.0
64.1
(3,5)
46.9
17
0.4
(3,4)
67.5
69.1
(4,5)
31.3
18
0.2
(3,3)
51.2
6.4
(4,5)
47.5
72.9
(3,7)
46.0
19
35.7
(3,4)
47.1
41.7
(4,5)
43.4
75.1
(4,7)
30.0
VIT-B-32
20
64.9
(3,5)
45.0
66.6
(5,5)
27.8
75.3
(4,8)
29.2
(Float: 75.9)
70.0
(4,5)
28.7
(4.7)
75.7
(5,8)
18.8
22
73.2
(3.7)
43.6
72.7
(5.8)
27.3
75.8
(6,8)
12.3
23
75.2
(1.7)
27.4
(6.8)
17.2
75.8
(6,8)
9.3
24
75.6
(5.7)
16.5
75.6
(6.8)
15.1
75.9
(8,8)
6.3
32
75.9
(8,8)
5.4
75.9
(8,8)
5.2
75.9
(8,8)
5.4
24

---

Published in Transactions on Machine Learning Research (12/2025)
Table 9: OPTQ:
We provide the top-1 test accuracy and quantization configuration of the Pareto-optimal models that form the fronticrs visualized in Figure Note that M and N respectively denote the weight and activation bit widths.
OPTQ
OPTQ+EP-init
OPTQ+AXE
Model
P
Top-1
(M,
Sparsity
Top-1
(M,
Sparsity
Top-1
(M, N) Sparsity
14
(3.3)
51.9
(3,3)
58.9
15
1.5
(3.3)
71.6
57.7
(3,4)
59.9
16
(3,4)
72.0
62.5
(4,4)
39.1
17
34.4
(4,4)
55.6
66.4
(4,4)
32.0
ResNet 18
18
43.1
(4,4)
49.9
68.1
(4,5)
32.0
(Float: 69.8)
19
58.0
(3,3)
54.1
58.2
(5,5)
37.7
69.1
(5,5)
17.1
20
64.9
(3,4)
54.1
63.7
(5.5)
30.3
69.4
(5,6)
17.0
21
67.2
(4,4)
31.9
64.5
(6,6)
25.0
69.5
(6,6)
8.7
22
68.6
(1,5)
31.9
68.2
(6,6)
16.9
69.6
(6,7)
8.7
32
69.7
(8,8)
2.3
69.7
(8,8)
4.4
69.7
(8,8)
2.3
14
25.2
(4,4)
16.2
15
42.5
(4,5)
16.4
16
1.0
(3,3)
26.6
0.1
(4.4)
32.4
58.0
(5,5)
9.5
17
6.6
(3,1)
26.5
8.4
(5.5)
17.0
63.7
(5,5)
8.0
MobileNeLV2
18
36.6
(1,4)
28.8
19.4
(5.6)
17.2
67.8
(5,6)
7.9
(Flott: 71.9)
19
55.6
(1,5)
15.2
58.5
(6.6)
10.2
69.4
(5,7)
8.0
20
63.7
(5,5)
14.9
60.6
(6,7)
10.2
70.7
(6,7)
4.2
67.8
(5,6)
7.9
68.5
(7,7)
6.5
71.2
(7,7)
2.4
22
69.4
(5,7)
7.9
69.0
(7,8)
6.5
71.5
(7,8)
2.4
32
71.6
(8,8)
1.4
(8,8)
71.6
(8,8)
1.4
16
0.1
(3.3)
89.0
28.3
(3,6)
63.2
17
2.7
(3,5)
66.8
47.4
(3,6)
53.0
18
1.4
(3,3)
52.5
13.0
(3,6)
65.4
68.8
(3,7)
53.1
19
7.1
(3,4)
50.8
48.9
(4,6)
48.1
73.3
(3,7)
48.2
VIT-B-32
20
10.2
(3,5)
54.1
64.8
(4.7)
47.3
74.8
(4,7)
32.8
(Float: 75.9)
47.7
(3,6)
53.0
72.6
(5,7)
75.6
(4,8)
32.4
22
73.3
(3.7)
48.2
73.5
(5,8)
33.2
75.7
(6,8)
22.0
23
74.8
(1.7)
32.8
75.0
(6.8)
75.9
(6,8)
14.4
24
75.6
(4.8)
32.4
75.6
(6.8)
21.5
75.9
(8,8)
9.4
32
75.9
(8,8)
8.6
75.8
(8,8)
10.0
75.9
(8,8)
8.6
25


Reviews:
[{"review_id": "Lw8NNQKgwv", "reviewer": "Reviewer_i5PD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a post-training quantization (PTQ) framework that explicitly incorporates accumulator-aware constraints to prevent overflow, providing theoretical guarantees absent from prior work. \n\n\n### Strengths\n\n1. Solid theoretical foundation with explicit hardware constraints\n1. Scalability evaluation shows quality maintained for larger models.\n\n\n### Weaknesses\n\n1. Evaluation relies heavily on perplexity and limited tasks.\n1. Experiments cover a narrow set of models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The computational efficiency gains are theoretically proven and are numerically demonstrated for different architectures: Llama3 and SmolLM2.\n\nThe modularity of the method is demonstrated through implementation on top of two algorithms: GPFQ and OPTQ.", "audience_yn": "Yes", "audience_explanation": "By introducing accumulator-aware PTQ with theoretical guarantees, the paper addresses a foundational problem that connects algorithmic quantization design with hardware-level implications.", "requested_changes": "1. Explicitly state which aspects of AXE are original contributions versus adaptations of existing accumulator-aware or bounded-range quantization ideas.\n\n2. The paper should discuss the computational overhead of AXE during PTQ calibration.\n\n3. Add more benchmarks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V5AsK7LkuF", "reviewer": "Reviewer_cpM1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces **AXE**, the first framework for accumulator-aware post-training quantization (PTQ) of large language models (LLMs). Prior work in this area was limited to quantization-aware training (QAT), which requires costly retraining or fine-tuning. AXE provides a theoretically justified method that guarantees overflow avoidance during low-precision accumulation, making it feasible to optimize not just multipliers but also adders in MAC units.\n\nKey contributions include:\n\n* **Formalization of accumulator-aware PTQ** and derivation of constraints ensuring overflow avoidance.\n* **General framework (AXE)** applied to existing PTQ algorithms (GPFQ, OPTQ), shown to work with multi-stage accumulation.\n* **Experimental validation** on models from GPT-2 to LLaMA-3 (up to 8B parameters), showing AXE preserves up to **98% FP16 perplexity** with significant power/bit-operation savings.\n* **Scalability analysis**, demonstrating improved preservation of model quality as model size increases.\n\n**Strengths**:\n\n* Novelty: First systematic treatment of accumulator-aware PTQ.\n* Theoretical rigor: Overflow-avoidance constraints derived and integrated into greedy PTQ algorithms.\n* Strong empirical validation across multiple model families, scaling to billions of parameters.\n* Practical impact: Multi-stage accumulation enables real hardware benefits (power, throughput).\n\n**Weaknesses**:\n\n* Limited evaluation of real-world latency/power on actual hardware platforms (results rely mainly on proxy cost models such as bit-flip counts).\n* Focuses mostly on perplexity and reasoning benchmarks; does not deeply investigate downstream fine-tuned tasks.\n* Comparisons are mainly against EP-init and naïve bit-width manipulation; stronger baselines (e.g., very recent PTQ methods) could strengthen claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper supports its claims with both **theoretical analysis** (derivation of constraints and guarantees) and **extensive experiments** across a wide range of models and settings. The claims about Pareto dominance (better perplexity vs. accumulator width) are well-supported by multiple figures and tables. The extension to multi-stage accumulation is motivated theoretically and confirmed empirically. The main caveat is that the evaluation of power/latency benefits is indirect (via cost models), but the evidence is still clear and convincing for the quantization/ML research community.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience includes researchers in machine learning efficiency, model compression, and LLM deployment. This work directly addresses a **pressing bottleneck in LLM quantization**, the under-explored role of low-precision accumulation. By bridging QAT and PTQ in this domain, the findings are highly relevant for both theory-focused researchers and practitioners seeking scalable deployment of LLMs.", "requested_changes": "* **Critical**: Provide more discussion of **real-world hardware implications**. The paper currently relies on cost models and simulation (bit-flip counts) but does not benchmark actual inference throughput/latency/power. Even small-scale hardware experiments would significantly strengthen claims.\n* **Critical**: Expand comparison baselines. Including recent PTQ methods beyond EP-init and naïve bit-width manipulation would provide a fairer picture of where AXE stands in the current literature.\n* **Optional**: Explore additional downstream tasks (beyond perplexity and zero-shot reasoning) to evaluate robustness of AXE-quantized models.\n* **Optional**: Improve clarity by summarizing design trade-offs (weights/activations vs. accumulator precision) in a concise table for practitioners.", "broader_impact_concerns": "The broader impacts are mostly positive: more efficient quantization methods can reduce energy consumption and enable deployment of LLMs on resource-constrained devices, improving accessibility.\n\nPotential concerns include:\n\n* **Environmental impact framing**: While AXE reduces energy at inference, the work does not address training costs or lifecycle impacts.\n* **Misuse potential**: As with any efficiency method, lowering barriers to deploying LLMs may accelerate the proliferation of models in harmful applications.\n\nThe paper could benefit from a more explicit **Broader Impact Statement** addressing these points.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V20FK1iPay", "reviewer": "Reviewer_64Px", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces AXE, the first accumulator-aware framework for post-training quantization (PTQ) of large language models (LLMs). Unlike prior work that focused on accumulator-aware quantization-aware training (QAT), AXE provides theoretical guarantees against overflow in PTQ, and supports multi-stage accumulation. The authors implement AXE on top of existing PTQ algorithms (GPFQ and OPTQ), showing consistent improvements in power–accuracy trade-offs across a range of models from GPT-2 to Llama3.\n\nStrengths:\n1. Addresses a real hardware bottleneck as the field moves toward 4-bit and sub-4-bit inference.\n2. Strong theoretical grounding with clear hardware-aware constraints.\n3. Scalability to billion-parameter LLMs, which is non-trivial.\n4. Flexible design—AXE is algorithm-agnostic and demonstrated on two popular PTQ methods.\n\nWeaknesses:\n1. Heavy reliance on perplexity as the main evaluation metric; broader task-level evaluations could strengthen the empirical case.\n2. Coverage of models is somewhat narrow and outdated (mostly GPT, OPT, Llama); evaluation on additional families (e.g., Qwen, Seed, etc.) would strengthen the generality claim.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s central claim—that AXE enables safe, high-accuracy PTQ under low-precision accumulation—is well-supported by both theoretical analysis and comprehensive experiments.\n1. The Pareto frontier analyses convincingly demonstrate AXE’s improvements over baselines such as naïve bit width manipulation and EP-init. \n2. Results scale from small to large models and include both perplexity and zero-shot reasoning tasks, supporting the generality of the approach.", "audience_yn": "Yes", "audience_explanation": "This paper directly addresses a growing pain point: as quantization pushes into 4-bit and below, accumulation—not multiplication—becomes the performance and power bottleneck. Given the surge in LLM quantization research, this work bridges a critical gap between theory and practice.", "requested_changes": "1. Add evaluation on downstream tasks beyond perplexity (i.e., more benchmarks).\n2. Extend experiments to include additional model families (e.g., Qwen) to demonstrate broader applicability.", "broader_impact_concerns": "No ethical risks are apparent.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zZEI0vYGCr", "reviewer": "Reviewer_vzxV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces AXE, the first accumulator-aware post-training quantization (PTQ) framework that guarantees overflow avoidance.\n\nFirst Formalization of Accumulator-Aware PTQ: The paper provides the first formal study and definition of accumulator-aware PTQ, extending the concept beyond the previously limited quantization-aware training (QAT) setting.\n\nIntroduction of AXE for Overflow Avoidance: AXE is proposed as a flexible framework that infuses overflow avoidance guarantees into layerwise PTQ algorithms. It achieves this by controlling dot product ranges throughout the error correction process.\n\nTheoretical Justification and Flexibility: The paper offers theoretical motivation for AXE and demonstrates its adaptability by implementing accumulator-aware variants of existing PTQ algorithms, GPFQ and OPTQ.\n\nSupport for Multi-Stage Accumulation: Unlike prior accumulator-aware QAT methods that assume a monolithic accumulator, AXE is designed to support multi-stage accumulation (e.g., tiled dot products). This opens the door to full datapath optimization for large language models (LLMs).\n\nImproved Trade-off between Accumulator Bit Width and Model Quality: Experiments show that AXE significantly improves the trade-off between accumulator bit width and model quality compared to alternative methods like EP-init and naive bit width manipulation. This translates to lower power consumption with better model quality (Figure 1, Figure 4).\n\nScalability to Billion-Parameter LLMs: AXE demonstrates excellent scalability to billion-parameter LLMs. For instance, when quantizing Llama3 8B for a 16-bit multi-stage accumulation datapath, AXE maintains up to 98% of the FP16 perplexity, surpassing naive bit width manipulation by up to 15%. This is critical for wider transformer architectures.\n\nCompatibility with Pre-processing Algorithms: The framework is shown to be compatible with established pre-processing methods like SmoothQuant and Hadamard-based incoherence processing (Table 2).\n\nAnalysis of Accumulator Bit-width Impact: The paper provides evidence that addition costs can dominate multiplication costs as operand precision is reduced, highlighting the growing importance of accumulator-aware quantization (Figure 2, Section 3).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission presents a strong case for AXE, providing a good balance of theoretical justification, empirical evidence, and clear explanations. The claims are generally well-supported.\n\nClaim: AXE is the first accumulator-aware PTQ framework explicitly designed to endow overflow avoidance guarantees.\nEvidence: The paper states \"To the best of our knowledge, ours marks the first formal study of accumulator-aware PTQ, and the first solution to scale to modern LLMs\" (Page 4). It also explicitly mentions that prior accumulator-aware research was limited to QAT (Page 1, Abstract; Page 2).\n\nClaim: Prior accumulator-aware quantization was limited to QAT, and PTQ lacked a formal study/solution .\nEvidence: The abstract and introduction explicitly state this limitation (\"Accumulator-aware quantization research has so far only considered the quantization-aware training (QAT) paradigm...\", \"To the best of our knowledge, there has been no formal study that explores accumulator-aware quantization in the PTQ setting.\"). This is a claim about the state of prior research, and the paper implicitly supports it by not citing any PTQ accumulator-aware methods that achieve similar guarantees or scaling.", "audience_yn": "Yes", "audience_explanation": "Here's why, considering TMLR's scope:\n\nRelevance to Core ML Research (Quantization): Quantization is a fundamental and active area of research in machine learning, particularly for deploying models efficiently on hardware. TMLR's audience, which includes researchers and practitioners working on model optimization, hardware-aware ML, and efficient inference, would naturally be interested in advancements in this field.\n\nAddressing a Critical and Emerging Bottleneck (Accumulator-Awareness): The paper highlights that as operand bit widths shrink, the cost of additions in MAC units becomes a dominant bottleneck, potentially even more so than multiplications. This is a crucial insight that challenges traditional focus and points to a new frontier in quantization. Researchers striving for maximum efficiency would find this motivation highly relevant.\n\nFocus on Large Language Models (LLMs): LLMs are currently at the forefront of AI research and application. Their massive size makes efficient deployment (especially on edge devices or with limited resources) a paramount concern. A framework that specifically addresses accumulator-aware quantization and scales to billion-parameter LLMs (like Llama3 8B) is directly relevant to a large segment of TMLR's audience working with these models.\n\nTheoretical Guarantees (Overflow Avoidance): For many applications, particularly those requiring high reliability or safety, theoretical guarantees (like overflow avoidance) are extremely valuable. The paper's mathematical framework and explicit guarantees would appeal to researchers interested in robustness and correctness in ML systems.", "requested_changes": "Discussion on Computational Overhead of AXE:\nIssue: The paper focuses on the quality and hardware efficiency (power/area through bit operations and accumulator size). However, PTQ methods inherently have some computational cost during calibration. AXE introduces additional operations. Briefly discuss the computational overhead introduced by AXE's components during the PTQ calibration phase. Is it negligible compared to the base PTQ algorithms (GPFQ/OPTQ)? Are there any situations where it becomes significant? This would provide a more complete picture of the framework.\n\nBriefly Discuss Other Metrics Beyond Perplexity for LLMs:\nIssue: The evaluation primarily uses WikiText2 perplexity for model quality and zero-shot reasoning accuracy for Llama3. While perplexity is standard, LLM evaluation is increasingly complex. Acknowledge the broader landscape of LLM evaluation. The paper already mentions zero-shot reasoning tasks (Table 3 in Appendix B) which is good. A sentence stating that perplexity is a strong proxy for generative quality and a necessary first step, followed by the zero-shot task evaluation, would be sufficient. This shows awareness without adding extensive, potentially distracting, new results.", "broader_impact_concerns": "While AXE prevents overflow-induced catastrophic failure, it doesn't guarantee full fidelity to the FP16 model's behavior in other aspects (e.g., subtle shifts in bias, reasoning quality degradation before catastrophic failure). The technical \"guarantees\" for overflow might lead deployers to over-trust the model's overall performance in these extreme low-precision settings, potentially leading to unfair or incorrect outcomes that are difficult to debug in an edge environment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)