The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
Authors: Jiale Chen
Yalda Shabanzadeh
Elvir Crnčević
Torsten Hoefler
Dan Alistarh
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Quantization
Lattice Algorithm
Closest Vector Problem
Primary area: foundation or frontier models, including LLMs
TL;DR: The GPTQ algorithm is exactly Babai's nearest plane algorithm for the closest vector problem, giving a geometric view of LLM quantization.

Abstract: Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators.
While GPTQ emerged as one of the standard methods for one-shot post-training quantization at LLM scale, its inner workings are described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees.
In this work, we show that, when executed back-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's nearest plane algorithm for the classical closest vector problem (CVP) on a lattice defined by the Hessian matrix of the layer's inputs.
This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences:
first, the GPTQ error propagation step gains an intuitive geometric interpretation;
second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped.
Leveraging this bound, we design post-training quantization methods that avoid clipping, and outperform the original GPTQ.
In addition, we provide efficient GPU inference kernels for the resulting representation.
Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for billion-parameter models.
Source code is available at https://github.com/IST-DASLab/GPTQ-Babai.

Published as a conference paper at ICLR 2026
THE GEOMETRY OF LLM QUANTIZATION: GPTQ AS BABAI'S NEAREST PLANE ALGORITHM
Jiale Chen', Yalda Shabanzadeh', Elvir Crněević², Torsten Hoefler', Dan Alistarh!. 1.2 Institute of Science and Technology Austria (ISTA), 'Red Hat, Inc., *ETH Züiichh j jiale. chen@ www.eisstac....
ABSTRACT
Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more af- fordable accelerators. While GPTQ emerged as one of the standard methods for one-shot post-training quantization LLM scale, its inner workings described as a sequence of algebraic updates that obscure geometric meaning or worst-case guarantees. In this work, show that, when executed mack-to-front (from the last to first dimension) for a linear layer, GPTQ is mathematically identical to Babai's s nearest plane algorithm for the classical closest vector problem (CVP) lattice defined by the Hessian matrix of the layer's 's inputs. This equivalence is based on a sophisticated mathematical argument, and has two analytical consequences: first, the GPTQ error propagation step gains intuitive geometric interpretation; second, GPTQ inherits the error upper bound of Babai's algorithm under the assumption that no weights are clipped. Leveraging this bound, we design post- training quanti- zation methods that avoid clipping, and outperform the original GPTQ. In addition, we provide efficient GPU inference kernels for the resulting representation. Taken together, these results place GPTQ on a firm theoretical footing and open the door to importing decades of progress in lattice algorithms towards the design of future quantization algorithms for DO--AS/SAAA//-GG-aaaaaaa lattps:// Stt./IIST-DALAACCCC
models. Source code is available at
INTRODUCTION
Generative pre- pre-trained transformers (GPT) models contain hundreds of billions of parameters and have massive computational and memory costs (Luccioni et al.) 2024). Post-training quantization (PTQ) has emerged as a practical solution for reducing their footprint (Gholami et al. 2021). Among a growing family of methods, GPTQ (Frantar et al.] 2023) was the first to push one-shot quantization down to the 4-bit regime, while retaining near-baseline accuracies. GPTQ is still very popular nowadays and yields state-of-the-art results in some regimes (Kurtic et al. 2025).
Despite its empirical success, the GPTQ algorithm was only presented as sequence of greedily applied algebraic operations: the procedure picks one weight at a time, quantizes it via rounding or clipping, and then optimally updates the not-yee-quantized weights to correct for the remaining per-layer loss; it then continues with the next weight, and so on. This procedure leaves an obvious open question: why does local greedy rule work well globally? Current literature does not answer this question, leaving little guidance for principled extensions or failure case analysis.
Our contribution. This paper is the firs{ to provide geometric interpretation for GPTQ, which implies layer-wise global error bound. Our main theoretical results (Section[4) are (i) the GPTQ optimization problem, i.e., linear-layer quantization with the L2 objective on the output, equivalent to the closest vector problem (CVP) w.r.t. L2 distance; (ii) the GPTQ algorithm executed from the last first dimension is the same as Babai's 's nearest plane algorithm on the basis of the factorized Hessian matrix, without I.I. basis reduction, and this finding holds independently of whether large weights are clipped to the quantization grid (a process known as weight clipping); and (iii) the worst-case layer- wise error in the no-clipping setting is bound tightly by the trace of the diagonal matrix of the
The concurrent work of Birnik 2225 appeared on arXiv slightly later than our preprint (Chen al. 2025

---

Published as a conference paper at ICLR 2026
LDL decomposition of the Hessian matrix. In addition (Section tie theoretical findings to practical quantization by introducing new no-clipping methods of better accuracy than the original GPTQ, together with efficient GPU inference kernels for the resulting representation.
2 RELATED WORK
Second-order compression (pruning and quantization). The idea of using Hessian information
to guide parameter removal dates back to Optimal Brain Damage (LeCun et al. 1989) and Optimal
Brain Surgeon (OBS) (Hassibi et al. 1993). Optimal Brain Compression (OBC) (Frantar & Alistarh
2022) generalizes OBS to the post-training setting and unifies structured pruning and quantization (also called Optimal Brain Quantizer, OBQ) under single exact solver. GPTQ (Frantar al. 2023) inherits OBQ's s error propagation method but applies it in fixed order, so that the inverse Hessian can be shared and only needs to be computed once. GPTQ only has cubic computational complexity in the column/row dimension, making it suitable for LLMs. QulP (Chee et al. (023) proves an eπrоr guarantee for GPTQ and proposes the LDLQ method as an equivalent variant of GPTQ.
Lattices, CVP algorithms, and hardness. The closest vector problem (CVP) is NP-complete to approximate within any constant factor under polynomial-ting reductions (van Emde Boas 1981 Micciancio & Goldwasser, 2002 Dinur et al. (2003), motivating decades of approximation algorithms. Babai's nearest plane heuristic (Babai) 1986) delivers a solution in polynomial time and, when preceded by LLL basis reduction (Lenstra al. 1982). enjoys 20(m) approximation. BKZ basis reduction (Kannan 1987) further tightens the constant in an exponential-timg solver.
3 PRELIMINARIES AND NOTATIONS
We use Python-style indexing inside square brackets to select elements and sub-matrices from a tensor, c.g., [ selects the j-th row vector, [:,]] selects the j-th column vector, and [j ;, selects the sub-column consisting of rows after the j-th (inclusive) row in the j-th column, [:, JJ selects the column vectors indexed by set as sub-matrix, etc?)
3.1 LINEAR-LAVER QUANTIZATION PROBLEM
Problem. Let X [zi....,.. Fp] E Raxe be the sampled calibration input data of batch size and input dimension e with 2; E Re and 2 rank (X). Let W [.,,..., w, E Rexr be the linear layer weights of input dimension c and output dimension r with w, E R Let S [s....., s, Rexr be the non-zero quantization scales with s; E R6 Here consider general case that applies any grouping pattern: each weight element w:li] has its own scaling factor s.ji]. Assume S is statically computed using methods like AbsMax or MSE before any weight updates. Let 2 S Z be the quantization grid (representable integers). In the clipping setting, e.g., for INT4 format, Z, {-8...., -,,0, 1...., 7). In the no-clipping setting, Z Z, which allows any integer as the quantization results. Let Z = [z1...., 2r] E 2, exr be the (unknown) quantized integers with z; žf. Denote Q = [q qrl E geer as the dequantized weights with qi = diag (st 2; E R The goal is to minimize the L2 error on the layer output XW E Rnxr; |XQ XWI = E. | diag (8,) 2, Xw i.e, finding argmin X diag (s,)
| for all isisr.
OBQ algorithm. Let set Jj initialized to (1....,c} be the set of not-yet-quantized indices of wn, We denote Jj as J as a short-hand notation. For each weight vector w,, OBQ chooses
j + argminges (q. -wbil=w;[j))"
bijj
as the next dimension to quantize. OBQ quantizes the chosen element wr[j]] as q:bi] + 8t
ROUND w. Z,) via the ROUND 2,) function which rounds the inputs to the nearest values
For more details, please see http::/:///////..o//////////aa/a exi ng. html
2

---

Published as a conference paper at ICLR 2026
in 2, OBQ then optimally updates the subset of weights w.[l] via an error propagation step w; [ +- w.l' + Aw,[5"] for all j' E with
(X [s J] X
[ [. J]) [J (q: w;(r))
Aw; f
(2)
JJ) [ن,
OBQ continues iteration with J + J {i} until J is empty.
GPTQ algorithm. GPTQ reduces the computational complexity of OBQ by applying the OBQ quantization and error propagation steps in fixed dimensional order, e.g., from the first to last
dimension < 1 to c), instead of dynamically determined orders (Eq. [ The fixed order is
[3,.]] can be shared across wo for all i, without recomputation. Furthermore, the Hessian information for all can be precomputed at once using Cholesky or LDI. decomposition of the Hessian matrix X X.
independent of the output channel i, thus the Hessian information (X[:,)) [J [
Algorithm T iss the pseudocode of GPTQ. The algorithm identical to the original GPTQ paper (Fran- lar et al., 2023) except for missing the blocking mechanism that only affects the memory access pattern and computational speed, but not the numerical results. Additional notations are as follows.
Pe {0. 1 a permutation matrix that modifies the dimensional order of GPTQ quantization. The default order is front-to-back back (from the first to last dimension), i.c., P = I. X E R+ is small damping factor for computing the Hessian matrix, ensuring the matrix of full rank, typical L || Il Function LDL returns the lower triangular matrix in the IDI. decomposition. Symbols and denote element wise multiplication and division, respectively.
I choice is = Tooe -5 (x X) [ = Tone
Algorithm 1: GPTQ
Input: original weights W Reer per-coordinate scales S E R calibration activation
X E [RRxx permutation P E 1), damping ratio À >> 0, integer grid Z, C Z
Output: quantized weights Z E 24 dequantized weights Q E Rexr
1 H + PT (X x XI) P dampen and reorder Hessian
2 LDL (H-1) factorize (take the L matrix from the LDL decomposition) the inversed
Hessian as the shared coefficients for error propagation
3 W,
P- 'W p-1pp1 S reorder weights and scales 4 Q, z - W, 0 initialize dequantized and quantized weights
5 for j t to do
6
Ç W Wjj:\/S j.: :) element-wise divide current row by its scales
7
Z[j.:] <- ROUND (C, Z4. // quantize coefficients to the target grid
8
Q + - S j.:] // dequantize current row back to weight space
9
E+ Qbi W : quantization error for current row
10
W :,:] + Wij :,:] L[j f // propagate error to not-yet-quantized rows; broadcast
over columns
11 end
12
Z. Q PZ, PQ // undo reorder to restore original input order; return integers and dequantized weights
3.2 THE CLOSEST VECTOR PROBLEM (CVP)
Problem. Let B = bb....,,bJ Raxe be set of e basis vectors of dimension with by E R" and n 2 = rank (B). Let y E R an external target vector to approximate. Let E Z be the (unknown) integer vector representing the basis combinations of the lattice vector. The goal is to find the vector on the lattice defined by the basis B that is closest to the target vector y. ie., finding argminggz: Bz yl' A visualization of a two-dimensional CVP is shown in Figure (a).
Babai's nearest plane algorithm. Babai's algorithm iteratively projects the target vector onto the nearest hyperplane of an LLL-reduced lattice and rounds the corresponding coefficient. Figure (b) visualizes the basis reduction step and Figure T
(c-d) visualize the projection steps.
3

---

Published as a conference paper at ICLR 2026
Bahai's Algarition
Latrice
Turget Point
Returned Lattice
Projected
(a) Closest Vector Problcm
(b) Basis Reductton
(c) Projection Step
(d) Projection Stop 2
Rasis Direction
Rounding Boundaris
Schmidt
Hlyperplane
Rounding Boundary
Optimal Voranoi
(1) Round-to-Nearee
Robai
Rabai (Another Order)
Figure 1: Top row: (a) CVP in a two-dimensional lattice; (b) Basís reduction can find shorter, more orthogonal basis that can potentially improve the results; (c-d) The projection steps Babai's nearest plane algorithm. Bottom row: rounding boundaries of (e) optimal rounding or Voronoi cells; (f) round-to te-reesstt (RTN); (g) Babai's ' nearest plane algorithm without basis reduction; (h) Babai's ss algorithm without basis reduction under the reversely ordered basis.
Algorithm is the pseudocode of Babai's nearest plane algorithm to solve CVP. For better computa- tional efficiency, the pscudocode uses conceptually equivalent approach. Instead projecting the target vector onto the nearest hyperplane, it moves the target vector along the basis direction towards the hyperplane where the origin lies. The projection error is retained in the updated target vector is orthogonal to the hyperplane and will not affect subsequent projections. Additional notations are as follows. Function LLL returns the transformation matrix delta defaulting to 3. Function QR returns the orthogonal matrix in QR decomposition, which is the same as the normalized Prram-Schmidt orthogonalization process. (..-) denotes the vector dot product. Function ROUND is defined as in the GPTQ algorithm.
LLL reduction with the parameter
Algorithm 2: Babai's 's Nearest Plane
Input: lattice basis (column vectors) B E iRxxe target vector y E 1" Output: closest lattice vector's basis coefficients g
1 T + LLL (B) // unimodular transformation matrix from LLL basis reduction
2 A < BT // reduce the basis
3 Φ QR (A) // normalized Gram-Schmidt process (take the Q matrix from the QR
decomposition)
4 y' = <- y, 0 // initialize residual target and integer solution in reduced basis
5 for j + to 1 do
6
< ( [1:j..10 y²) ( A(: jl) // exact coefficient along the unnormalized Gram-Schmidt vector; ratio between the projections of residual and the reduced basis the Seha-- direction
7
zlj] + ROUND (c, Z) // round to the nearest plane
8
y y A i. j ] // update the residual
9 end
10 z + Tz // map integer solution back to the original basis and return
Babai's error bound. Figure D shows the rounding boundaries of the optimal (c), round-to- mond-tto-earest (RTN) (f), and Babai's algorithm without basis reduction (g-h). Compared to RTN, Babai's algorithm generates rectangular partitions and thus has a smaller worst-case error. The error bound has been proven in Babai (1986). Formally, let + ( . be the set of normalized Gram-Schmidt vectors of the LLL-reduced basis A a1..... acl. Let À [ā]... ... a.] denote the unnormalized Gram- Schm-S vectors with ā, = (Ø), a,) Φj. At iteration j. the algorithm replaces the exact coefficient Ç with the closest integer, so the deviation satisfies |< - x[i]| < 2. Hence, the error component along à has norm at most } || Because the À is orthogonal, these error components
4

---

Published as a conference paper at ICLR 2026
add in Euclidean norm, giving a bound on the residual (error) vector y': 11y'11 E 1|²,1
EE= (Ø), ay)² Babai's algorithm guarantees to return the center vector of the hyper-cuboid
Figure (g)) constructed by the unnormalized Gram-Schmidt vectors where the target is located. Equality attained when the target y lies the corner of the hyper-cuboid, so the bound is tight. Babai
(986) additionally proved a relative error bound for with Bz yll S 7 m
B y
The bound is 7 maxigise Sy a
Vc+I majss 0gy
4 THEORETICAL RESULTS
We first show that weight quantization is instance of the classical closest vector problem (CVP) in Section [.1 which allows us to work in a lattice defined by the Hessian. We then reinterpret OBQ's, equivalently GPTQ's, eπrоr propagation step nearest hyperplane projection in Section 4.2 establishing our main equivalence in Section 4.3 GPTQ, running back-to-front, coincides exactly with Babai's s nearest plane algorithm. This equivalence allows us to import Babai's guarantees to obtain a tight, layer- layer-wise error bound in the no-clipping setting in Section 4.4 Finally, we analyze how quantization order influences this bound Section 4.5
4.1 EQUIVALENCE BETWEEN L2 QUANTIZATION AND CVP
quantization problem with the L,2 objective argmin || diag (s:) z; Xw; and CVP with the L2 distance argminge2: B= yl1² share the same solution (z 2;) whenever the structural conditions B diag (s,) and y Xu; hold and the solution domain matches. To ensure the solution domain matches, we can either disable the clipping in the quantization setup (setting 2, = Z) or enable the clipping the CVP setup (making z E 29). Table is take-away dictionary showing the correspondence between the quantization and CVP concepts.
Table 1: Quantization-CVV dictionary for the output channel i.
Quantization symbol
CVP interpretation
Input activation X Raxe
Basis directions (columns generators)
Scale 8; E IR
Basis stretches
B(g) = X diag (s;) E Raxe Weight w; IRc
Lattice basis (columns are generators) Floating-point coordinates the unstretched basis Integer coordinates on the lattice basis
Integer weight representation E Z
Dequantized weight = diag (8;) 2; E RR
Dequantized coordinates on the unstretched basis External target vector to approximate
Target output activation V(4) Xw; E IR"
We can introduce a factor the Hessian matrix, X [X Xel with X can then be reformulated as X diag (8:) z; X w.
TTx
X. The loss
Theorem (Quantization and CVP) The CVPs using any possible factors the Hessian matrix X X are equivalent under an orthogonal transformation (rotation and reflection) of the lattice and external target vector.
Proof Let X' and X' be two possible factors of the Hessian matrix with X X TT The inner products (Xi, Xis) and (X²,+ X²,)) must be equal for all 1 S J1.32 I c. In other words, the lengths ||x;,,
|x/, ||. and the angles 2 (X X;,) L (x,, X1x). for all S j1. S.
According to Theorem any decomposition factor X' of the Hessian matrix X T X can be used instead of X without changing the geometric properties of the CVP and its associated quantization problem. This is useful for reducing the computational cost, e.g., we may use a square matrix X Rexx instead of the rectangular matrix X E Raxe.
5

---

Published as a conference paper at ICLR 2026
4.2 OBQ's GEOMETRIC INTERPRETATION
We first demonstrate the geometric interpretation of OBQ (GPTQ's slower predecessor) to facilitate our equivalence proof of GPTQ and Babai's algorithm Section 4.3
Auclltary Onhogonal Dinoctions
Basis Vector by
Basis Vector t.
Tinger Poist you 22
Nasea Hyperplane A/MPP Co Spaa((y) (5t
Hyperlize HCC K2)) b, Span (5+1)
Babat's Projected Point Prolecer - Eytk A6)8
Erro Vector 4g Projenciu oo - E/AA//))
Ermor Component Vector DGG Ag
(a) (3D) Babai's Projection
(b) [3D) Babai OBQ Equivalence
Ermor Component Vector 262 A.
Remaining Eroe Component Vector E,
Inverse Bois Vactor Ry, (R) b2)
lnvens Busi Vectur H2 B1g my 6.6 Onthogoral Projection
Spann (1,
Projected
Paijem(())) (b,) Poniero
Projected
Projected Ence Vecior Projecet (Ag) Ag M Peolecer(
Projecta Ensor Component Vastrr AC, Ponjecm (4)
Projecta Finor Component Vanor AC, Ponjeern (1.)
Angle
(Projann (b,). Projecent
(c) [2D) Nearest Hyperplane
(d) [2D] Orthogonal Projection Plane
Figure 2: Equivalence OBQ's error propagation and Babai's projection. (a) 3D plot showing the target being projected onto the nearest plane. (b) 3D plot showing how the projection error is propagated. (c) 2D plot showing the vectors on the nearest hyperplane (a-b). (d) 2D plot showing the vectors the orthogonal projection plane in (b).
Theorem (Error Propagation and Babai's Projection) Babai's ' nearest plane algorithm itera- tively projects the target vector onto the nearest hyperplane and rounds the coefficient. The OBQ error propagation step (Eq. 2 is exactly this projection on the original basis B X diag (s:) without basis reduction.
Proof Let B .bb.....b.. b.] ] be the basis with by being a basis vector. Let be the set unprojected indices with j1, f2 E J and ji * f2- Let y Ejer J Lybj be the current residual target where Sj E is real number be rounded integers. Let AHP = LSja b;2 Span (b, | j2)
be the nearest hyperplane that is orthogonal to the Crram-Schmidh vector bjz -j Projs, (bj.).
Figure 2 (a) is 3D plot showing the projection error vector Ay ProjNHP (3) y. We focus on analyzing the error propagation in the direction of basis b, induced by the projection bü and collapse the span of other basis vectors to a single dimension illustrated by the hyperline HL = [6, bjz + Span (bylj li * f2.. Figure 2 (b) is a 3D plot showing the decomposition of the error Ay Ejes bc the error component vectors in the basis directions. Figure (c) is 2D plot showing the vectors plane NHP. The number Çj will be updated to & + Ac; such that ProjNHP (y) Ejen (G AC2) bj- Next, let N B-T
basis
[n1...., ne.] be the inverse basis. Then, we have (n,j b,) and n, 1 by, + j'. We project all the vectors in Figure (b) onto the orthogonal projection plane OPP = Span {n,, j2 that is orthogonal the hyperline H., and continue the proof in the 2D geometry Figure (d). Denote the angle
(b)
AS, Projorp(Or
(n n2,,)) = ग - (Projopp (b,,). Projorp (b,,)). Then, AC Proiorr (b/z)
cos =
(n 12ja)
n2 (12 Roy2) For j j1.j2. ||Prooorp (b;) ||r,]|| = (Pro)orr
18
coe cos(= (0)
(b.,n,)
For j.j (irr22), (nz, n.,) (N N) [i-57] -(B BBB) 1i.5%.
coo -8)
cos
cos()-0)
Proiors (b)2) ||| mig (m.)
(mi .211)) ACjo
Combining the above equations, AG;, Protopp (b,)) [,, (122)
AGia

---

Published as a conference paper at ICLR 2026
(B B) [2] (B B) 522 proof.
A5j2- Finally, substituting (X diag (s:)) [ and ()
w. x.
completes the
Ausiliay Ortiugural Directios
Basis Vocter
Target Point yom M,5,
Notral Hepoplane (ipprline),AMPPPM (G,1%, Spun(0, (ittae)
Babai's Projected Point Projoare Ey(4 20)8
Finor Vector Agr Prolauu 3- y RyAGA
Emoo Component Vector Si,
Remaining Emor Component Vector Ryay Apa,-46
lnvens Bass Vector M, - (Bt, by) - Ri 63. Projected Poola, (bg)
Figure 3: Geometric interpretation of OBQ's quantization order. This 2D plot shows the target being projected onto the nearest plane.
Corollary (OBQ Dimension Selection) At each dimension selection step (Eq. D. OBQ selects the not-yel-quantized dimension j such that the nearest hyperplane of dimension is closest to the target residual vector.
Proof We use the same notations defined in Theorem Figure a 2D plot showing the distance (projection error or quantization error) between the target residual vector and nearest hyperplane NHP of the basis bj.- For better illustration, we collapse AHP into
single dimension. The distance || can be expressed as | Projn., (Av)
A Proing. (b/²)
AC
(b) m
For each w, OBQ independently selects
(a.Li- 2:2
[eJ] ([:JI) "[s] = argmin jEJ (n..my) = argmin |AC,]
(AC,)²
j = argmin jed (X(:J)
[|,|]] as the next dimen-
sion to quantize, which is exactly minimizing this distance.
4.3 GPTQ AND BABAT'S ALGORITHM
Originally, GPTQ [Algorithm] D runs from the first to the last dimension (j <- to c) while Babai's algorithm (Algorithm 2 runs from the last the first dimension (j + e to 1). This the only (superficial) difference between the two algorithms, as formalized below.
Theorem 4 (GPTQ and Babai) GPTQ and Babai's 's algorithm without basis reduction will have the same results if we align the dimensional order of these two algorithuss, e.g., running GPTQ from the last to the first dimension.
Proof We prove this theorem both geometrically and algebraically. We first present the geometric proof. Theorem 2 shows that each intermediate weight vector produced by OBQ, equivalently GPTQ. can be viewed as Babai's residual vector in the activation space. At step j (running from the last to the first dimension, + e to 1), GPTQ's 's error propagation update is exactly Babai's 's projection at step j. which projects the current residual of the target vector onto the hyperplane orthogonal the j-th Gram-Schmidt vector.
Alternatively, we present more rigorous algebraic proof. Section describes the exact quantization procedures using Babai's algorithm in more detail, with the pseudocode in Algorithm 4 Section C contains the equivalence proof, which we proceed in three steps. First, we rewrite GPTQ to track the cumulative quantization error and show that this form is algebraically equivalent to the standard implementation. Second, we run GPTQ in the back-to-from order and replace the lower triangular factor with upper triangular Third, we prove that the step-wic wise rounding decisions the back-to to-k-to-front GPTQ coincide with those of Babai's s algorithm.
that each update affects only the nut-yet-quantized coordinates.
Geometric interpretation of GPTQ. Theorem shows that, if we regard the activations as the lattice basis and transform the floating-point weight vector target vector in the activation space,
7

---

Published as a conference paper at ICLR 2026
GPTQ performs an orthogonal walk through a nested sequence of affine subspaces in pre-computed dimensional order.
Ineffectiveness of composing algorithms. A scemingly appealing idea is take the solution returned by any Babai iteration and then perform one further GPTQ-style error propagation step on the weights in the activation space, hoping push the approximation even closer to the optimum. However, as proven in Section C.4 such an extra update vanishes: the final results Z and Q remain unchanged. In other words, once Babai's projection has been executed, any subsequent GPTQ-style correction is algebraically redundant. This confirms that the equívalence in Theorem 4 is already tight; neither algorithm can be strengthened by composition.
4.4 GPTQ'S ERROR BOUND
Having established the correspondence between GPTQ and Babai's nearest plane algorithm, we can now import Babai's approximation guarantee to obtain an upper bound on the layer-wise quantization error in the no-clipping setting.
Theorem 5 (GPTQ Error Bound) Assume clipping (24 Z) and let T be the permutation matrix of the reversed GPTQ quantization order (equivalently P with the reversed column or- der). Let D be the diagonal matrix of the LDL decomposition of the permuted Hessian matrix T X XT. For every output channel i (1 S i S. r) produced by Babai's 's algorithm, or equiv- alently the GPTQ algorithm executed back-to-front, the (absolute) quantization error has tight (TT For the relative bound for y with X| diag (s:) 2, -xw;ll S 2/ X diag (s.) z Xwiil. we have 1 S S
upper bound: X diag (8:) zi -Xw,1l < (T- 8.) D
1 maxicise
v max t where d, - / D (T-18:) [i].
Proof The full proof of Theorem 5 is presented in Section D.I
If the scales 8; are small enough, we may assume the weights w; are nearly uniformly distributed within the hyper-cuboid constructed by Babai's 's orthogonalized basis vectors, the expected absolute error will be 3 of the worst-case bound. See Section D.2 for proof.
4.5 THE ROLE OF QUANTIZATION ORDER IN GPTQ
The quadratic form on the right-hand side of the absolute error bound in Theorem S is sensitive the pivot order of the LDL. decomposition of the Hessian matrix; this is the quantization order. Reordering the dimensions changes the entries of the diagonal matrix D before the scale 8; is "weighted" by them. A poor order may place large D entries against large s; entries and hence inflate the bound. For a batched quantization algorithm like GPTQ, the order should be independent of the output channel i. To develop a good heuristic order, reasonable approximation large quantization group sizes, is that the elements of &([i] are equal for all 1 < j S c. Then we can focus finding the optimal pivot order for the LDI. decomposition of the Hessian matrix X X to minimize tr (D).
make, especially for
Finding the optimal order is NP-hard (Rose et al. 1976). However, heuristics often effectively reduce the trace term in practice. Even with clipping, heuristics can reduce the error. GPTQ introduces the act-order, the descending order of the Hessian diagonal, i.e., the ascending order of the Hessian diagonal when applied to Babai's ' algorithm.
To improve upon act-order, we propose the min-pivot order, which essentially taking the minimum diagonal entry at each LDL (or Cholesky) decomposition step. This order can be calculated by Algorithm 3. which has cubic time complexity and does not increase the overall time complexity of quantization. This order also has a geometric interpretation, as the order of the Gram-Schmide orthogonalization process of the basis: always taking the shortest residual vector as the next one to orthogonalize, agreeing with Babai's 's relative error bound. Across our preliminary runs (Section D.3). min-pivor consistently reduces tr (D) relative to act-order, but the downstream accuracy gains are modest. We nevertheless report min- pivot as a principled choice, and view act-order as a cheap approximation that only considers the Hessian diagonal, which already captures most of the benefit when the Hessian matrix is well-conditioned.
8

---

Published as a conference paper at ICLR 2026
Algorithm 3: Min-Pivot
Input: Hessian H E Rexe
Output: order encoded as a permutation matrix T E {0, 1
1 + (I....,c} initialize the not-yet-pivoted indices
2 T + inittialle the output permutation matrix
3 for j < 1 do
4
I' < argmin y H ji choose next index with the smallest current diagonal
5
H + - - H H H , 5") updates remaining entries with rank-1 Schur
complement
6
T + 1 // record the index
7
J (i') // mark pivot as used
8 end
5 APPLICATIONS
The original GPTQ algorithm clips the overflowed integers at the rounding step, introducing large errors that violate the error bound in Theorem 5 of GPTQ that work in the no-clipping regime.
In this section, we explore error-guaranteed variants
We notice that enforcing no-clipping by simply increasing scales is counterproductive: larger scales enlarge the bound, and the resulting errors can exceed those of clipped scheme such as MSE. Hence, any practical no-clipping design must account for the weight distributions that are known to have heavy outliers (Li et al. 2025). We would still like to apply small scales, but use small bitwidths for the bulk of inliers while handling the overflowed outliers with more storage budget without clipping them. We therefore propose two overflow-toleran schemes.
Scale-adjusted SpQR (SSQR). SpQR (Dettmers et al. 2024) keeps small set of outliers in full precision, but it still leaves clipping place: weights are grouped, the outliers and a shared scale chosen per group before the GPTQ updates, and there is guarantee the updated inlier weights stay within the representable range. We design SSQR with a acale-adjustment mechanism to fix this issue. For simplicity, we discard SpQR's 's second- level quantization for the scales. For weight vector w; E R', we represent the quantized weight q: E IR' as diag (s;) z; + & where 2 E Z is the low-bitwidth integer weight vector, 8; E çe is the floating-point scale vector with each scale shared per group (only one number per group is actually stored), and & E RR the sparse floating- point outlier vector (stored in the compressed sparse row format, CSR) that captures all the overflowed weights after GPTQ's error propagation. The scale-adjustment mechanism tunes the scale 8, until the density of &; satisfies the specified rate. Because exhaustive trial-and-emt infeasible in large layers, the mechanism only proportionally changes 8; so that the search space reduces to dimension. With the observation that the outlier rate is negatively related to the scales in general, this can be done via binary search: initialize 8; using MSE, quantize w, with the specified format using GPTQ without clipping, calculate the density of &,, and adjust 8; and iterate. Section E. Algorithm is the pseudocode.
over per-group scales
nulfman--encode post-training quantization (HPTQ). To better align with the infinite, uncon- strained lattice in CVP, we design HPTQ, which represents both inliers and outliers in a unified, equal-spaced integer grid. The idea is to use Huffman encoding, which was also explored for net- work compression by Choi et al. (017)). We quantize the weight matrix W E Rexr as Q = sZ with a single scalar s E Rgz0 and integers search: initialize range proportional to the maximum weight, quantize to unclipped integers with GPTQ, measure the Huffman coding cost of Z, and adjust s until the encoded bits meet a target average bitwidth. This yields aneven-bitwidde representations that preserve accuracy while meeting compression budget. Section E. Algorithm ii is the pscudocode.
Zexr We select via an entropy-guided binary
Experiments compare round-to-neares (RTN), original GPTQ, HPTQ, and SSQR with 1-5% outliers. We also include Huffman- -encoded RTN (HRTN) as a baseline to HPTQ. which mirrors HPTQ but replaces GPTQ with RTN (pseudocode: Section | Algorithm 12). The quantization order is act-order for all methods. RTN, GPTQ, and SSQR use group size 128. RTN and GPTQ calculate the scales with the MSE method. Figure (a-b) shows that HPTQ sustains low perplexity on Qwen3-8B

---

Published as a conference paper at ICLR 2026
at reduced bitwidths and scales favorably across model sizes, with 3.125-bit emerging Pareto optimal in terms of perplexity vs compression. Further information can be found in Section E including the experimental setup (Section E.2), additional metrics such as benchmark results (Qwen
models: Section E,3
Llama models: Section E.4 and comparison with other methods (Section E.5
Different Methods on Qwen3-8B
HPTQ on Quen3-00661,7744 7/48//11
SSQR on Qwen3-8F (A6000 GPU)
80
80
TPOT TPOT Spoodup vs PyTorch BF16
2.5
RTN
Avcrage Rinwidt [14]
WilcText-2 Peplexity
Wiki7ext-2 Paplexity
4.125
2.0
HRRN
40
3.125
40
10
HPTQ
1.5
BF16
20
3.125
4.125 5.125
SSQR 3%
SSQR-1N %%
20
1.0
SSQR-5%
10
Intiar Birwith [hi]
Pureto Optimal
0.5
0.0
2.125
3,125
4,525
5.125
2
5
6
7
Average Bitwidth [bit]
Model Size [GBB]
Outlier Rate [%6]
(b)
(c)
Figure 4: (a) Comparison of quantization methods (RTN, GPTQ, HRTN, HPTQ, and SSQR with 1-5% outliers) on Qwen3-8B evaluated on WikiText-2. Perplexity is plotted against the average effective bitwidth per weight, with the BF16 baseline shown horizontal line. HPTQ has the best (lowest) perplexity. See Section F.3 for zero-shot evaluation results. (b) Scaling behavior of HPTQ across multiple model sizes (0.6B, 1.7B, 4B, 8B, 14B) and bitwidths (4. 3112,,,.1225 3.125, 2. 2.).... The x-axis denotes the effective model size after quantization, and the y-axis shows perplexity on WikiText-2. Each curve corresponds a fixed bitwidth, while points along a curve represent different model scales. Using our HPTQ method, 3. 1.t5 stands out as the Pareto optimal bitwidth (optimal perplexity vs compression trade-offs). (c) End-to-end inference speedups of our SSQR kernel vs the PyTorch BF16 matrix multiplication kernel on NVIDIA RTX A6000 GPU. We run the Qwen3-8B model across multiple outlier rates (0%~5%) and inlier bitwidths (4, 3, 2) and measure the TPOT (time per output token) metric. Our kernel achieves about 2x speedup end-to-end.
CUDA inference kernel. We implement inference kernel for SSQR in CUDA/C++, optimized for low-batch latency, handling both dense inliers and sparse outliers while targeting the Ampere platform. The kernel supports group quautiigd inlier weights in the 2-4-bit range with scales in 16 bits and support for unstructured sparsity, used to avoid weight clipping. Figure (c) visualizes the end-to-end speedup in the L.M decoding phase the PyTorch BF16 kernel. Our kernel achieves about 2 x speedup across different bitwidth and outlier rate settings when generating 128 new tokens batch size of 1. Technical details and layer-wise speedups are described in Section E.6
CLOSING REMARKS
Summary. We have shown that GPTQ, when executed back-to-front, is mathematically identical to Babai's 's nearest plane algorithm applied the lattice defined by a layer's Hessian without basis reduction. Based on this theory, we propose arror-gurrante practical methods and provide optimized CUDA kernels that deliver low-latency inferences. More broadly, the lattice perspective opens a two- way channel: decades of the closest vector problem (CVP) heuristics can refine practical quantizers, while the behavior of massive neural networks may, in turn, inspire new questions for lattice theory.
Future Work. Looking ahead, extending the analysis to clipped grids and exploring (scale-aware) basis reductions are the immediate next steps. However, we emphasize that the state-of-the-a 4-bit floating-poie formats (e.g., MXFP4 and NVFP4) are essentially no-clipping (Egiazarian et al. 2025 Chen et al., 2026 since they use very small quantization groups (32 and 16, respectively). the near-optimal choice of scale is AbsMax per-group. which leads to no weight being clipped. As such, a no-clipping analysis of these formats would directly apply to actual practice. We will also extend the lattice view beyond weight-only linear layers activation and K KV-cache quantization.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Throughout this work, we have strictly adhered the ICLR Code Ethics. All datasets utilized in experiments are publicly available and widely recognized within the scientific community. We ensure that these datasets do not contain any personally identifiable information or sensitive content. Our work does not involve human subjects, animals, any form of personal data collection. We have thoroughly considered potential dual-use concerns and do not foresee any harmful applications of our methods. There are conflicts of interest declare, and external sponsorship influenced the outcomes of this research. All experiments were conducted with integrity and transparency.
REPRODUCIBILITY STATEMENT
We committed to ensuring that work is transparent and reproducible. To facilitate this, clear explanations of any assumptions and a complete proof of the claims have been included in the main text and appendix. We also share the source code as part of the supplementary materials. The code is documented and includes instructions for setting up the environment, running the simulations, and reproducing the results presented in our paper. By making our resources openly available and providing detailed explanations, we aim to enable the research community to validate and build upon our findings.
ACKNOWLEDGMENTS
This research was supported by the Scientific Service Units (SSU) of ISTA through resources provided by Scientific Computing (SciComp). The ISTA team was supported by generous grants from Google and NVIDIA. The authors thank Vage Egiazarian for the discussions on this work.
REFERENCES
László Babai. On lovász' lattice reduction and the nearest lattice point problem. Combinatorica, 6
(1):1-13, March 1986. ISSN 1439-6912. doi: 10. 10.1077/B002579403 URL https:///inkk
pringer. www/artiiiioooo.. 1tteeeee....0..........
Johann Birnick. The lattice geometry of neural network quantization - short equivalence proof of gptq and babai's 's algorithm, 2025. URL (Rtttttss://axxii> wttp:/iwwwwwvvorrbbooareeeeeaee hbwpg//ab///50... 01077
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36. pp. 4396 4429. Curran Associates, Inc., 2023. URL /Rt.ttpss./pppoceeeeeee pss.
neur
/paper_ aper Foer-Confee Cler-Cofference.oo pdf
Jiale Chen, Yalda Shabanzadeh, Elvir Crrêvvćć, Torsten Hoefler, and Dan Alistarh. The geometry of Ilm quantization: Gptq as babai's 's nearest plane algorithm, 2025. URL
abs/ 2507 18553
Jiale Chen, Vage Egiazarian, Roberto L. Castro, Torsten Hoefler, and Dan Alistarh. Wush: Near- optimal adaptive transforms for Ilm quantization, 2026. URL
12. 00956
Yoojin Choi, Mostafa El- El-Khamy, and Jungwon Lee. Towards the limit of network quantization. In International Conference on Learning Representations, 2017. URL et/foorm?id- haaad-rruuuuugggg
Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantizer representation for noar-lossless LLM weight compression. In The Twelfth International Conference on Learning Representations, 2024. URL /tl/ittpp/.//peeeevveww..
http///pppeeeeieeee/ee/ee net forum? /forummid=0112
5ahSuy
11

---

Published as a conference paper at ICLR 2026
1. Dinur, G. Kindler, R. Raz, and S. Safra. Approximating cvp to within almost- cetthosst-plnoommill factors is np-hard. Combinatorica, 23(2):205 243, April 2003. ISSN 1439-6912. doi: 10.1007//00499-003 -0019-y. URL https:://iint http://///...n....iinnng.reeeeee springer.ie..... com/ wttmm//ariieeeee0.m 100.1///00111110000000000000000 019
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 12284-12303. PMLR, 21-27 Jul html
2024. URL https: settps::/proceediing
mlr pttpm//.ppeeee//e////aaaaaaaaaaa
Vage Egiazarian, Roberto L.. Castro, Denis Kuznedelev, Andrei Panferov, Eldar Kurtic, Shubhra Pandit, Alexandre Marques, Mark Kurtz, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Bridging the gap between promise and performance for microscaling fp4 quantization, 2025. URL https::/arrs /arxivv/rivvvv./aaa org/ /tt./////00 23202
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruming. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 4475-4488 Curran Associates, Inc., 2022. URL
cC cttpsssccppppppee paper ffre_filespppp
Conference Contrrre
pdf
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference Learning Representations, 2023. URL /ttpss: net/ forum wwd-
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference, 2021. URL https
7/ //axxi..rrg/ab////1000
Babak Hassibi, David G. Stork, and Gregory J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299 vol.1, 1993, doi: ieee org/ dow.ument/2300
10.1 10.1109/CCNN.11932298777 URL
572
Ravi Kannan. Minkowski's convex body theorem and integer programming, Mathematics of Operations Research, 12(3):415- 440, 1987. ISSN 0364765X, 15265471. /ttp://////ww istor. (tt.sttaabbl33899944 3689974
Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. "give me BF16 or give me death"? accuracy-perfon perfoormnee trade-offs in LLM quantization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 26872-26886, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 979-8-89176-2555 doi: 10. 11556661222255 acl- 1on..1304. 1304. URL .otg//025.acl-ueeee 1304/
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- 1989, URL https: http:///wweeeeeeebbbeeeee ppzooeediingg..nuriiss.coo httpl/appppfffllllaaeppppllpp http:/eeeeeeeaape/999////LLLL
$799bo2254481222755---88
Arjen Klaas Lenstra, Hendrik Willem Lenstra, and László Lovász. Factoring polynomials with rational coefficients. Mathematische Amnalen. 261(4):515 534, dec 1982. ISSN 4332-1807. doi:
10.1007/BF01457454, URL www.iiii..spiiin..... wttp::/www.....tt.eeeeeeeeeeeeee 1007/E0 01457454
Xinlin Li, Osama Hanna, Christina Fragouli, and Suhas Diggavi. ICQuant: Index coding enables low-bit bit LLM quantization. In Second Conference on Language Modeling, 2025. (tthts
//opnnrevi www.:nettuunte 6
12

---

Published as a conference paper at ICLR 2026
Sasha Luccioni, Yacine Jernite, and Emma Strubell. Power hungry processing: Watts driving the cost of ai deployment? Proceedings of the 2024 ACM Conference on Faimess, Accountability, and Transparency. F '24, pp. 85 85-99, New York, NY, USA, 2024, Association for Computing Machinery. ISBN 9798400704505. doi: 10. 14111/333311006665542222
URL http::tts:s/dlt cmn
.org/doi/10.
3658544
Daniele Micciancio and Shafi Goldwasser. Complexity of Lattice Problems: A Cryptographic Perspective, volume 671 of The Springer International Series in Engineering and Computer Science.
Springer, New York, NY, edition, 2002. ISBN 978-0-7923--668447 doi: 10. 00.10077978-1-4615--00 7-7. URt.tttss:: //w:n//nnn appinggrr com wwo. -
Donald J. Rose, Robert E. Tarjan, and George S. Lucker. Algorithmic aspects of vertex elimination on graphs. SIAM Journal on Computing, 5(2):266 283, 1976. doi: 10.1137/0205021. URL
https:
8 www.....mmmm....m.m...... wwwp//////oooooooooooo 1110////..1177725000
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulP#: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 48630 48656. PMLR, 21-27 Jul ml www.s..............ee htleess/22355tsee Sttps/////22//tstee222223 2htmmm2....llmm.....
2024a.
Albert Tseng, Qingyao Sun, David Hou, and Christopher De. Qtip: Quantization with trellises and incoherence processing. A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomezak, and C. Zhang (eds.), Advances Neural Information Processing Systems, volume 37, pp. 59597-59620. Curran Associates, Inc., 2024b. doi: 10.5220020079017-1994 URL
http:://w....a..........e..eaa/a 47 17b22e5e2ac//6666d200------ Cawfeee--ocfeeecceeco pdf
P. van Emde Boas. Another np-complete partition problem and the complexity of computing short vectors lattice. Technical Report 8104, University of Amsterdam, Department of Mathematics, Netherlands, 1981. URL https: :/ttpss:/sttt :/stafff//ff.. fnwi. uva. nl/p. wwnewdeboas/ /vectors/
mi 04c. html
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX TABLE OF CONTENTS
B Babai's Algorithm for Batched Quantization
15
Algebraic Equivalence Proof of GPTQ and Babal's Algorithm
C.I Step
16
C.2 Step
2
22
C33 Step
23
C.4 Proof of Inellectiveness of Additional GPTQ Relinement on Babai's s Algorithm
26
D Further Discussion on Quantization Error Bound
27
D.I Proof Absolute and Relative GPTQ Quantization Error Bounds
27
D.2 Expected Quantization Error over Uniform Hyper-Cuboid
27
D.3
Empirical Verification on Quantization Order and Error Bound
28
E
Further Applications and Experimental Results
29
E.1
Overflow Tolerant Quantization Algorithms
29
E.2
Experiment Setup
31
E.3
Accuracy Results for Qwen3 Models
32
Accuracy Results for LJama Model
36
Comparison with Other Quantization Methods
40
6.6
Technical Details and Perlormance of SSQR's CUDA Kernel
41
LLM Usage
43
14

---

Published as a conference paper at ICLR 2026
B BABAI'S ALGORITHM FOR BATCHED QUANTIZATION
Given the equivalence we have shown in Section 4.1 the quantization problem can be converted to CVP, allowing us apply Babaï's nearest plane algorithm in the context of quantization. A naive way is to compute ( X' diag (8;) and y( = Xw; and run Babai's algorithm independently for all 1 < i S r. However, this is computationally inefficient, as we will need compute the expensive (O (c') LLL basis reduction transformation T() for the basis (3) and the expensive (O (c")) QR decomposition of (+) (1) for r times. However, a few adjustments can made to simplify the computation and enable batched processing.
Disabling basis reduction. The LLL basis reduction is unfortunately scale-sensitive, generating different transformations (() for different scales 8; (unless all the 8; vectors are parallel), which prohibits the reuse QR decomposition results. Furthermore, LLL basis reduction incompatible with clipping, as the roundings are performed in another basis, and there is no easy way to do the clipping for the original basis.
Changing quantization order. Quantization order is a feature in GPTQ that controls the rounding and clipping order of the dimensions. This order influences the quantization error, as we discuss in Section 4.5 In the context of Babai's algorithm, this corresponds to the order of the basis in the Gram-Schmin orthogonalization and the hyperplane projections, as shown in Figure [ (g-h). To do so, we can replace the LLL basis reduction in Babai's algorithm with permutation by setting the transformation matrix T to a permutation matrix that is independent of i.
Theorem 6 (Babai's Quantization Order) If is permutation matrix that does not depend on i, the orthogonal matrix & can be reused without recomputing the QR decomposition for each
Proof The permutation matrix T E {0, 1]cxo has exactly one non-zero element in each row and column. Scaling the rows of can also be interpreted as scaling the columns of T, therefore its multiplication with diagonal matrix has property: diag (8:) T T diag (T)) Let A = XT, A(s) X diag (s:) T. Denote the QR decomposition of A as A = & with & being orthogonal matrix and R being an upper triangular matrix. Then, the QR decomposition of A(j) becomes A() =
X diag (s:) T = XT diag (T s;) = A diag (T-'s;) s;) = + (R diag (T (=)s:)). Therefore, the QR decompositions of (s) share the same orthogonal matrix + for all <isr.
As shown in Theorem 6 changing quantization order does not require repeated computation of the QR decomposition. Note that, we also need to permute the scale S accordingly T-
Selecting basis. Putting things together, we are interested in A XT and its QR decomposition &. Theorem allows us to choose any Hessian factor X while keeping the result intact. With- out loss of generality, we can choose a X' such that A is an upper triangular matrix and the QR decomposition becomes trivial: $ = I, which simplifies the computation. The upper triangu- lar matrix A can be directly computed from the Cholesky decomposition the permuted Hessian matrix A A T X XT.
Algorithm 4: Babai's 's Quantize
Input: W. S. S.X,T,AA, X,4
Output: Z, Q
1 H T (X X AI T
2 A + CHOLESKY H
3 W, + - T-1S
,YQQ 2 AW, W.O
5 forjecto do
7
(
8
Zfj. ROUND (c,
Applying all the above considerations, we con- struct Algorithm 4 for batched quantization using
9
Q... - i]. j
10
Y -
Babai's s algorithm.
11 end
12
Z,
Q TZ,TQ
15

---

Published as a conference paper at ICLR 2026
C ALGEBRAIC EQUIVALENCE PROOF OF GPTQ AND BABAI'S ALGORITHM
In this section, we prove Theorem that GPTQ (Algorithm | and Babai's algorithm (Algorithm
equivalent if the dimensional orders are opposite.
Because a permutation matrix acts only as re-ordering coordinates, we may apply the permutation once the beginning (to W, S, and X) and once at the end (to Z and Q) without affecting any intermediate arithmetic. Hence, all algebra performed inside the two algorithms can be analyzed on the permuted basis where the permutation matrix the identity. On that basis. the sole distinction between GPTQ and Babaï's 's algorithm lies in the direction of the iterations. Proving that GPTQ running back-to- tack-to-front (j < c to 1) reproduces Babai's updates in Babai's default iteration direction would complete the equivalence proof.
We follow a shree-step proof scheme.
Step 1. Proving that the original GPTQ algorithm (Algorithm 5 that uses relative quantiza-
tion error row vector & E Ixr is equivalent to a new algorithm (Algorithm 6 using the absolute quantization error matrix A Rex,
Step 2. Reversing the iteration in Algorithm 6 and writing the reversed-iteration algorithm as Algorithm[7]
Step 3. Proving that the reversed-iteratio algorithm Algorithm is equivalent Babai's algorithm Algorithm g
Algorithms 5 are intentionally written in the linear algebra form. Cj R is the standard basis vector whose elements are 0 except the j-th element being 1, which is used as the row or column selector of matrix. The superscripts in parentheses denote the versions the variables during the iterations. S E R are intermediate row vectors. Additionally, L is the LDL
decomposition of the Hessian inverse H = LD D where L is lower triangular matrix with all diagonal elements being 1, and D is non nen-negative diagonal matrix. Similarly, U is the "UDU" decomposition of the Hessian inverse H = UD D DÂUT where U is an upper triangular matrix with all diagonal elements being 1, and D is non-negative diagonal matrix.
Note: the symbols are overloaded in Algorithms to X and the variables using the same symbols may carry different values, even if the inputs to the algorithms are the same.
C.I
STEP
To distinguish the variables using the same symbol Algorithms and we use symbols without to denote the symbols in Algorithm and use the symbols with for Algorithm
Claim
w Sj 18ise,
(3)
and consequently,
z 200, 05i iis
and
Q0i = Q0), 0šjšc.
(5)
Proof Byy Induction
The following equalities are held by the design of Algorithms
and
e = (0) = (0))) = 10)
(6)
(ا) = o, w 1 S
(7)
(10 e, W0-1), 1ši5c.
(8)
Q0)=Q01D Q(-11 ej e 200 diag (ST es)) - ee ((QU-1)) 1 15ise.
(9)
16

---

Published as a conference paper at ICLR 2026
Algorithm 5: GPTQ Original (Front-to-Back)
Input: W,S.X,,
Output: Z,
1 H X XI
2 L < LDL (H-1)
3 W + W
4 Q - W00)
5 t to coo
6
w6) e W 1)
7
(6) + wit diag (ST ej)
Z6) Z(-1) tej ( RRUUND (50), 21) -, 26-1))
8
Q + Q(--)) +e; (e] z diag (S e,) - e. Q(0-11)
9
10
eb) < e, Q - (()
W + W Leje(1)
end
13 Z,Q ( Q(6)
Algorithm 6: GPTQ 2 (Front-to-Back)
Input: V.S.X, 2,
Output: Z,
H X X XI
2 + LDL (H-1)
3 ((0) + +W
4 Q Z(0) W00)
5 for j + 1
6
w6) + Wli-1)
7
(6) + wii diag (STe,)
8
Z + zú-1) to ( ROUND (5) - 2(i-1))
9
QUi QU--) +ej (e) z diag (S e,, e. QU--1))
10
AU) + Q(i - ( // new
11
W + ( --1 A // new
12 end
13
Z, Q.Q
- Z (0,00' (Q))
17

---

Published as a conference paper at ICLR 2026
Algorithm 7: GPTQ Type-2 (Back-to-Front)
Input: ,,,
Output: Z,Q
1 H X X + XI
2 U t UDU (H-1) // new
3 W((+1) + W
4 Q(++1), z(+11) +- W 0
5 for j do
6
witl e W(++1
7
(6) - wltl diag (S 0j)
8
Z + z(+1) tej ROUND (ç -1) -ej e] zú+l))
9
QUI + Q(++)= + (e. z diag (ST 0,) o Q(++1))
10
AU) - Q6i -((+1))
11
WG) +- W(c+1) U new
12 end
13 Z, Q ((1) ((1)
Algorithm Babai Quantize (Default Order)
Input: W, W,S. X, 2
Output: Z, Q
1 И X X IK
2 A + CHOLESKY (H)
3 (e+1) (c+1), ((++)) + AW, W.
for j t co do
w6) r(++)
5
9 e,Ae;
(6) - with diag (ST (ST,,)= c,)-
6
7
Z - z(+1) oj (ROUND (5()), o, 2(+1))
8
QUi + Q(+1), + (e) 700 diag (ST e,) o Q(i+1))
9
Yu) t y(+1)) AAccc Q6)
10 end
11
Z, Q.Q+ z() Q())))
18

---

Published as a conference paper at ICLR 2026
@ = Q(--1)
+ te (e] 2 diag (S (5e,, - -e) -0-)))) 1 isisa
(10)
et) =e Q() ) - wli, 15isc
(11)
A =Q W 1 isjsc.
(12)
W = w - Le isisc.
(13)
W ( L A 6
1sjsc.
(14)
Extend the definition of A
(
(Eq. 12 for j O,
A (j) Q - (0)
0 Ssjsc.
(15)
Then we have A =Q(0) W W w 0 that Eq.
14 can also be extended for
j 0,
w = w L 6 05isc.
(16)
(1) Eq. 3 holds for j 1:
Using Eqs.
w 0 ( = W (1))
(17)
(2) Assume Eq. 3 holds for all j. S j. <c
Because L is lower triangular matrix with all diagonal elements being 1, --1 is also lower triangular matrix with all diagonal elements being 1.
For S j k S.
c Le, = = L e =0
=o
(18)
For siSc.
e, Le, e, L L-te,
(19)
For <j<c
e L
Σ extt
Σ cRok )
kmj+2
eei+ L
e
e Le - -
Σ exer
k=j+2
(20)
=e LI e,+1
Σ efti
(Eq. 19
A=/+2
-e L e
0eg
(Eq. 18
k=ド+2
=0/+1 (I-I).
With Eq. 9 for 1 S S 1, < k < and k,, k.
e TQ =or (QU-1) + ej (e] z diag (ST (STez) e, (]
=a Q 111
e e, (e) z diag (s e,) e (/Q611)
(21)
c z (1) diag (S o,) of Q((i-11)
=e 11-11)
19

---

Published as a conference paper at ICLR 2026
Recursively applying Eq. 21 for 1 sj ksc
ek Q = to W 133)/31
f (TT
(22)
Similar to Eq.
22 with Eq.
1 < c, cc
g Q (k)
e TQu
(23)
e (0)
e w
With Eq. 23 for S sssclskse
ek
(
0 T (Q (0)
(Eq. 15
ek QQ) ek W (0)
(24)
e (6) 0 W 0 A
0 0
39 (1) e W (0) = =
A
For 1 1 kksscc
0, LAW
0 LIA (
e
ew
A (i)
#=1
of Leweee TAA AAA
Lew e A0)
Σ
Le A (i)
+
W=
Σ C Le A (6)
+
Oc
AAA (1)
(Eqs. 18
(25)
Σ ey Le A (K)
+
Σ
Dej A (k)
(Eq. 24
('=++
Lew e (8)
+
Σ
ey
e A (k)
(Eq.. 18
W==+1
e Le e (k)
L
e
(k)
W=1
LIA
(k)
ej LA
(k)
20

---

Published as a conference paper at ICLR 2026
For <
0 L- A (1-1)
-0 L - IA (/-1)
L-
Σ ekek
AU-11)
L
exe (1-11)
[ Σ o J A
AU-))
k=j+1
Σ ej exe, A6-1)
tej L e
Σ 0el a (1-1)
A=j+1
(Eqs. 18 24
Σ ej
L exee
A0-1)
Σ
0e (1-1)
+ (00-6,) -e] A,A00
k=
Σ
0
L- c.
T A
L
Σ
oj L
-
L A c. (3)
k=
N=
(Eqs. 19 24
Σ 9
L
exer
e A
k=l
=e L-1
Σ e
A 6)) A.
1=1
= 'IA (5)
AA
=e (L-1-1) 11) A
T
(26)
According to the assumption, for 1 5 S. <, have
e, ((-11) (k)) ((() e, (k
(27)
and
@(A) (())
(28)
21

---

Published as a conference paper at ICLR 2026
For k J..
e (k) = (6) w (k)
(Eq.
Q(4) 0% (
(Q(h) W
% (Q() (((11))
(Eqs. 27 28
-0 (Φ() (w - (1-11)
---
(Eq. 16
(29)
-
((00) - + +
(k-1)
-e (A(4) + - (6-11)
(Eq. 15)
=el A (K) + (0-1 1) ^ (1)
(Eq. 26
- 11- (A)
=e L (-))
(1)
(Eq. 25
((++1)
= W
(Eq
% +1
( + Lej. eli.)
(Eq. 13
(cx)(H)) ))
-/
W +
(Eq. 13
Le - L-IA (A6.))
w +
(Eq. 29
1.41
(1)) + L Σ e
L L L-1 (-12())) (1-)
(30)
(0
(L - [i))-'''10
A (1.))
(Eq. 20
(0 1-7 1 (.)) + A (1.)
0j.+1
(0 L A (.) + +0)
(Eq. 24
(W - A (..)
2.+1
(U-)
(Eq. [6]
(1++1)
(Eq. 8)
Eq. 3 holds for 1.
C.2
STEP 2
Algorithm 7 (back-to-front order) is generated by reversing the iteration direction of Algorithm 6 Besides changing the direction of the index j. also need to change the LDL decomposition to so- called "UDU" decomposition so that the error propagation is correctly applied to the not-yet-quantized weights in the front dimensions.
Justification
Let P be the anti-diagonal permutation matrix with P P P Let be the LDL decomposi- tion of the permuted Hessian inverse PH PH- D iT where i is lower triangular matrix with all diagonal elements being 1, and D is non-negative diagonal matrix.
22

---

Published as a conference paper at ICLR 2026
Since changing the iteration direction instead of applying the permutation, we permute the matrix L back, yielding U PŁP. Alternatively, U can be caleulated using the decomposition --1 = PLPPD PPi P UD DOUT where U is upper triangular matrix with all diagonal elements being 1, and D P P is non-negative diagonal matrix.
The decomposition to calculate U from H is what we call "UDU" decomposition, which can be considered variant of the LDL decomposition.
C.3
STEP
To distinguish the variables using the same symbol in Algorithms 7 and g use symbols with to denote the symbols in Algorithm 7 and use the symbols with for Algorithm g
We have the Cholesky decomposition of H: H ( (H-1)- (UDiDAUT)
(Du (Du+U-1) Du U-, so that A = D U-!
Claim
3 = Cjj isisc.
(31)
and consequenlly,
20) 70) 15J
(32)
and
Q0I = Q0, 15j
(33)
Proof Eq.31 by Induction
For Sssc. c,
501) ((+1)
Ae
(6+1)
D le T(++)
(34)
DJ [ijl
Dili,ile y (i+1)
=e_ D D
The following equalities held by the design of Algorithms 6 and 8
Q(o+1) = ((++1) = w(e++) W.
(35)
Y (o+1) AW D
D U-W..
(36)
:6) = = Twi+l) isisa.
(37)
Q = QU+1) +e (e] 20) diag (TTe,) - -e]
15jsc.
(38)
Q6) Q6+1) ej (e) (0) diag (Se,,) e((()t))
1sise
(39)
A (i) Q6) (++1) 15i5c.
(40)
いい = (e+1)) - U-1 A (1) 15isc.
(41)
-0))((+)) A y(++1) D 1
15s5c.
(42)
23

---

Published as a conference paper at ICLR 2026
Because U is upper triangular matrix with all diagonal elements being 1, U-! is also upper triangular matrix with all diagonal elements being 1.
For h j
0 Ue = =C =0.
(43)
U -. c,
(44)
For SiS
e Ue, e U ej
(45)
(1) Eq. holds for =
Using Eqs.
e D = De. D U-W = = e eT = w = W/etl] wl0)
(46)
(2) Assume Eq. holds for all j 2 j.. jx c.
With Eq.
38 for 1 SSj c, < and # k.
et Q = (Q++))
+ ej (e] z diag (S
(STe;) ez) [ ((()tt)))
=a (1+41)
e e, (e) z diag (STe) e;) ej ((Q++1))
(47)
=o (0+1)
+
c z diag (S 0,) c.j (Qu+t))
=aa =e Q0+1).
Recursively applying Eq. 47 for 1 S Ejcclske
(K)
fl flsk<jsc
e
QQ0
e Qlo+1)) e W(c+1)
(48)
Similar to Eq.
48 with Eq. 39
for 1 S sclskss
c Q (k)
if iflsjskse,
e Q
(49)
e Q (c+)) = = w
fflk<ssc
For S S
V -y++1) D U ee Q6)
(Eq. 42
-x(o+))_
Σ
U UU-eeeeT0 Q(N)
(Bq. 42
(50)
-D
Σ
D
U Q6)
U
U
(Eq. 36
-D. -ー1
W
Σ
e
Q6))
24

---

Published as a conference paper at ICLR 2026
For < j <c c, xهت) ; D ((++1)
(Eq. 34
- D u-
W-
Σ c
Q(++1)
k=j+1
(Eq.55)
c Σ
--
W-
exel
(li+1)
k=j+1
-e, UW
Σ ej
U-ee
U
T
((+1)
=0 W
ej eee
Σ 0j U ex
- e U e QU-11)
ej
U-JUeLe T
U
0e -ee] (u+1)
(Eqs. 43 45
=c -.J W Σ c
U U-OO c1OK
(u++1 + +0. (QQQ0+))
Σ e
U
toTr T
@(it1)+ + w
k=1
(Eq. 49
W U-1
Eeoo c20% (U-1)- w)
Σ
k=
W U-1
[Q(H+1] w)
_o W U-1 (QU+)) w) w))
(51)
Because e! (W- (Qlet U--(Qiett)) w)) = W =l))), Eq. 51 can be extended for j = c,
5 ej (w (W-U- U- VV--U-(0Ut1)-- (QU+I)-W)) ww)) isisc.
(52)
According to the assumption, for j S k S e, we have
Q (k) (k)
(53)
1(1-11)
W U-1 (Qúi) -w) w)) T
(Eq. 52)
T
[W(c+1) 1-1 (Q w((tt))
(Eq. 53)
[W(c+t1] - U- (
(Eq. 40
(54)
Wi.))
(Eq. 41)
(1--1)
(Eq. 37
Eq. 31 holds for j. 1.
25

---

Published as a conference paper at ICLR 2026
C.4 PROOF OF INEFFECTIVENESS OF ADDITIONAL GPTQ REFINEMENT ON BABAI'S S ALGORITHM
We may try apply further GPTQ updates in Babai's algorithm by changing Line 9 in Algorithm
to
y y + AUe, = y - (5) AUe,
(55)
However, as A Du U-1 the اا1-1 remains the same:
w11--1)
- D D (5)
(Eq. 34
= 8 Di
y + Du U Ue
D D
+ tej D Dúy Ueje()
D
+e]11,ee"
(56)
D D
- Di Y
(1111)
(Eq. 34
26

---

Published as a conference paper at ICLR 2026
D FURTHER DISCUSSION ON QUANTIZATION ERROR BOUND
D.1 PROOF OF ABSOLUTE AND RELATIVE GPTQ QUANTIZATION ERROR BOUNDS
We prove Theorem 5 as follows.
Denote the basis (3) = X' diag (s:), (()) = Xw; as in Section 4.1 so that the quantization problem
becomes the CVP minimizing (1) y0ll Applying permutation T gives the permuted basis
(()) = (a)) T = X' diag (8.) = XT diag (T- Write the unnormalized Gram-Schmidt
vectors of (i) (() [ (i)).......... and). Babai's guarantee therefore yields the tight bound
(1) y(a) || || (1) (T 2.) y0ol S E |
We may, without loss of generality, use Theorem
to rotate X so that A()) is upper triangu- |لتبقق| simplifies to [A Let D()) be the diagonal ma- trix of the LDL decomposition of A()A()) such that [(lli.]] = |A
lar. In that case, the norm
|
The summation Σj-1 || can then be expressed tr (D()). Let C be the lower trian- gular matrix in the LDL decomposition of T X XT CDC so that the LDL decom- position of A (1) (6) = diag (T-1s:) T X XT diag (T-)s:) (1) (()) has D()
diag (T s.) D diag ss) and (() diag TT st) diag (T (T-'s:) The absolute no- clipping error bound is therefore + Σÿ= وا || Ir (D)) (T-)s:) D (T-
For the relative no-clipping quantization error bound, we can plug in | || ( =
/ (D V(diag (T-ls:) s,) D diag
(s)s (s)) VDj.j (T-1s) s:) [i|l
dj into
Babai s relative error bound Section
D.2 EXPECTED QUANTIZATION ERROR OVER A UNIFORM HYPER-CUBON
We have shown that, when clipping is disabled, Babaï's nearest-plane (hence back-to-front GPTQ) ensures the tight worst-case bound
X diag (s,) z; Xw:li
< 4 j=1
Σ |a
Ā = a
ā
(57)
where áj are the unnormalized Gram- Cram-Schmidt vectors of the permuted lattice basis A.
Introduce the half-edge lengths
= |اررا
j= 1=1.....5
(58)
so that the Babai residual always lies in the axis-aligned hyper-cuboid 11; [-aj, a,] and Eq. rewritten as
57 is
(59)
Eworst
j=1
Uniform prior on the unknown weight vector. Assume now that the continuous, not-yct-quantized weight offset u X (w: diag(s; )z;) is uniformly distributed inside this hyper-cuboid, i.c., each coordinate u, ~ Uniform (-a,, a,) and the coordinates are independent. The squared error becomes the random variable
(60)
j=1
Lemma
For a sealar Uniform (-a, a) one has E[u²]
3
Proof
1
1
a
a
E
u 2d
(61)
2a.
2a
0
27

---

Published as a conference paper at ICLR 2026
Expected residual norm. Using Lemma] 7
E
aj.
(62)
Ratio to the worst-case bound. Comparing Eq. 62 with Eq.
59 gives
1 12
E
Eworst
E X diag (s,) 2,
Xw,12
|
(63)
Hence, under uniform prior on the weights inside Babai's orthogonal hyper-cuboid, the average layer-wise quantization error is exactly of the worst-case guarantee stated in Theorem 5
D.3 EMPIRICAL VERIFICATION ON QUANTIZATION ORDER AND ERROR BOUND
Changing the quantization order alters the diagonal matrix of the LDL decomposition of the permuted Hessian and therefore the no-clipping GPTO/Babai bound (see Section (..5) When per- group scales are approximately uniform, minimizing tr [D) is a good proxy for tightening this bound. To assess different orders (back-to-front, front-to- -onkk, random order, GPTQ's act-order, and our min-pivot order), we run the calibration dataset from Section E.2 through the full-precision Qwen3-8B model and compute per-layer Hessians and calculate the tr (D). For the random order, we average the results over 100 runs. Table reports tr (D) for the layers in transformer block 18; other blocks and models show similar patterns. In block 18, act-order already reduces tr (D) relative to the
baselines, especially in the Q-K-V and Gate-Up layers (s:35-00% lower). Our min-pivot heuristic consistently attains the smallest trace. practice, this tightens the theoretical layer-wise error bound and yields modest but consistent improvements. We can use act-order as a cheap option and reserve min-pivot for cases where tighter bound required.
front/front-to-batt
Table 2: ir (DD with different quantization orders of layers in Qwen3-8B block 18.
Order
Q-K-V
O
Gate-Up
Down
back-to-front
1.169e+08
1.824e+08
1.181e+08
1.323e+09
front-to-back
1.161c+08
1.841c+08
1.202c+08
1.320c+09
random (averaged)
1.168c408
1.856c+08
1.194c+08
1.322e+09
act-order
7.400e+07
1.786e+08
6.052e+07
1.222e+09
min-pivot
7.323c+07
1.772c+08
5.990c+07
1.221c+09
28

---

Published as a conference paper at ICLR 2026
E FURTHER APPLICATIONS AND EXPERIMENTAL RESULTS
E OVERFLOW-TOLERAN QUANTIZATION ALGORITHMS
Algorithms and T2 are the pseudocodes our proposed SSQR, HPTQ. and HRTN algorithms in Section Additional notations are as follows. P [0. 1] is the target outlier rate in SSQR. L ... Er] E Rex the sparse weight matrix in SSQR. e R>o the target average bitwidth in HPTQ and HRTN.
Algorithm SSQR
Input: W. P.P.P.. 2.P
Output: Z, Z,S,E,
1 SMSE compute the MSE scale using W and 2,
2 8min: 8max <- 0", , 2" // initialize the binary search boundary per output channel
3 (Smin + 9max) /2 the scale for scale
4 while s not converge do
5
S SMSE diag (s) // ourpptt-hhaneee-wiselly proportionally adjust the scale
6
7.E.Q t PSQRINNERR WW S, x, P, X, Z4) Algorithm 10
smin E] s(t]
if ||t illbo pe for i {I....,r)
7
Smin [t], mmax [d]
s max
otherwise
s (Smin + Smax) //2
9 end
Algorithm 10: SSQR Inner Procedure (GPTQ with overflowed elements in floating-poin poinig
Input: W, W,S.X. S. P.
Output: Z, T,E,C Q
1 H P
X
X + +I) P
2 LDL (H 1)
3 W, P-1 P-IW, P-
4 Q, -,0
5 for + coo
6
< Wbiis//
7
Zbj. + ROUND (C, 2t)
[Wij Z S
if Z[J * ROUND (C otherwise
8
E| + 10
// new
9
Q Z[j.:] + new
10
E Qbii W
Wh 41 + Wij ;. + -[j
12 end
13
2, + Pz, PE, PQ // new
29

---

Published as a conference paper at ICLR 2026
Algorithm 11: HPTQ
Input: W,X,P h
Output: Z,s, Q
1 8min: Smax 0, || [W]] 1l // initialize the binary search boundary
2 8 f (Smin + Smax) /2 // the scale
3 while not converge do
4
S +- s- 10xr // broadcast the scale
5
Z. Q + GPTQ (W, S, X, P, X, Z) Algorithm
6
h + average Huffman encoding bitwidth of Z
7
if < h then
8
Smax t // too few bits, try smaller scale
9
end
10
else
Smin + s too many bits, try larger scale
12
end
13
(8min + $maxx) /2 s
14
end
Algorithm 12: HRTN
Input: W,
Output: 2,8, Q
Smin: Smax 0, || W 1l.o // initialize the binary search boundary with min and max 2 8 f (Smin + Smax) /2 // the scale
while s not converge do
4
Z < ROUND (W/s, Z) // round-to-nearest
5
Q x sZ
6
h average Huffman encoding bitwidth of Z
7
h' < h then
8
Smax t // too few bits, try smaller scale
9
end
10
else
11
8min + 8 too many bits, try larger scale
end
13
s + (9min + Smax) /2
14 end
30

---

Published as a conference paper at ICLR 2026
E.2 EXPERIMENT SETUP
We work with the Qwen3 family of models, which come in a range of sizes. We focus on the Qwen3-8B model for detailed head-to-head comparisons, while the other variants, Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Qwen3-14B, help us assess how our method performs across different model scales.
We construct the calibration dataset for the GPTQ algorithm using the FineWeb-Edu dataset BluuggingFacc//Wfiieewwebbb
subset sample-10BT), The dataset is streamed and shuffled with fixed seed for reproducibility. After tokenizing the text samples, our 256 sequences are accumulated into non-overlapping sequences length 2048.
We use WikiText-2 and C4 for perplexity evaluations. For WikiText-2, the entire test split is first concatenated using two line breaks as separators and then tokenized with the default HuggingFace tokenizer for each model. For C4, we sample individual documents from the selected shard, tokenize them, and randomly extract sequences of the desired length. In both cases, sequences shorter than the target length (2048 tokens) are discarded, and sequences longer than the target length are cropped to the specified window.
31

---

Published as a conference paper at ICLR 2026
E.3 ACCURACY RESULTS FOR QWEN3 MODELS
We compare the perplexity results between RTN, GPTQ, HRTN, HPTQ, and SSQR using the Qwen3- 8B model in Table 3
In addition, the perplexity results for other variants of Qwen3 with HPTQ
shown in Table
Table shows additional zero-shot results on the Qwen3-8B model for RTN, GPTQ. HRTN, and HPTQ. Additional HPTQ results on other Qwen3 models are in Tables 6 to to 10
Table 3: Perplexity Qwen3-8B model under HPTQ, GPTQ, HRTN, RTN, and SSQR with different bitwidths.
Method
Avg. Bitwidth
Perplexity
WikiText-2
C4
BF16 Baseline
16.000
9.73
13.55
4.125
9.81
13.64
HPTQ
3.125
10.34
14.23
2.125
13.97
16.89
4.125
10.10
13.92
GPTQ
3.125
12.77
15.61
2.125
57.51
36.14
4.125
9.90
13.80
HRTN
3.125
10.75
14.63
2.125
593.05
503.00
4.125
10.30
15.20
RTN
3.125
16.30
21.08
2.125
2e10
2e10
4.445
10.00
13.83
SSQR-1%
3,445
10.64
14.71
2.445
22.30
27.07
4.765
9.96
13.76
SSQR-2%
3.765
10.57
14.56
2.765
16.55
20.80
5.085
9.92
13.76
SSQR-3%
4.085
10.42
14.32
3.085
14.05
18.57
5,405
9.84
13.71
SSQR-4%
4.405
10.34
14.29
3.405
13.12
17.60
5.725
9.80
13.67
SSQR-5%
4.725
10.32
14.22
3.725
12.88
16.85
32

---

Published as a conference paper at ICLR 2026
Table Perplexity of Qwen3 models under HPTQ for different bitwidths.
Model
Avg. Bitwidth
Perplexity
WikiText-2
C4
16.000
20.96
26.37
4.125
22.72
28.35
0.6B
3.125
31.43
37.92
2.125
156.45
171.38
16,000
16.72
19.92
4.125
18.18
20.99
1.7B
3.125
19.72
23.15
2.125
46.94
51.96
16,000
13.66
17.07
4.125
14.26
17.39
4B
3.125
14.55
18.17
2.125
24.40
26.46
16.000
9.73
13.55
4.125
9.81
13.64
8B
3.125
10.34
14.23
2.125
13.97
16.89
16.000
8.65
12.23
4.125
8.76
12.12
14B
3.125
9.06
13.97
2.125
11.36
15,50
33

---

Published as a conference paper at ICLR 2026
Table Zero-shot evaluation results (%) for Qwen3-8P B-B under different quantization methods across six benchmarks.
Method
Avg. Bitwidth
Wino
MMLU
PiQA
SciQ
TQA
MC1
MC2
BFI6 Baseline
16.000
68.11
73.02
77.80
95.7
36.35
54.50
4.125
67.17
72.28
77.42
95.6
35.01
53.36
HPTQ
3.125
66.93
70.96
77.53
95.4
36.11
54.73
2.125
59.19
52.99
72.52
86.8
31.09
49.01
4.125
68.82
71.76
77.58
95.3
36.35
54.55
GPTQ
3.125
68.35
65.80
75.46
75.46
36.11
55.21
2.125
52.25
34.25
57.83
57.83
28.40
46.91
4.125
67.56
72.15
76.99
94.2
36.47
56.46
HRTN
3.125
66.22
67.85
76.12
93.7
35.13
53.68
2.125
51.22
33.91
65.78
76.8
30.48
51.78
4.125
67.17
69.71
75.90
94.5
36.84
55.77
RTN
3.125
57.93
47.90
70.89
87.1
34.03
52.76
2.125
49.08
22.95
51.63
21.2
24.11
47.33
4.445
68.43
72.12
77.04
95.2
37.58
55.81
SSQR-19 %
3.445
68.11
68.46
75.84
95.5
38.19
55.95
2.445
51.85
26.71
61.64
69.8
28.40
43.88
4.765
67.25
72.27
77.97
95.5
35.62
53.47
SSQR-2%
3.765
67.40
69.66
76.22
95.1
33.90
53.05
2.765
55.72
37.48
66.76
83.8
27.54
45.54
5.085
67.72
71.89
77.53
95.6
36.47
54.46
SSQR-3%
4,085
65.59
69.88
77.31
94.3
37.82
55.34
3.085
59.19
49.32
69.59
86.4
29.50
48.53
5,405
69.53
72.63
77.31
95.1
36.23
53.60
SSQR-4%
4.405
67.48
69.51
76.61
94.9
37.21
54.81
3.405
61.25
54.07
72.80
89.5
31.33
50.46
5.725
68.27
72.23
77.42
95.2
35.86
53.76
SSQR-5%
4.725
67.48
70.76
76.71
95.5
35.37
52.91
3.725
62.59
58.67
73.23
90.8
31.21
50.25
Table 6: TruthfullQA (%)
zero-shot results (MC1/MC2) for Qwen3 models quantized with HPTQ.
Avg. Bitwidth
0.6B
1.7B
4B
8B
14B
16,000
27.17/42.80
29.50/45.88
37.33/54.83
36.35/54.50
40.76/58.62
4.125
26.19/41.56
28.76/45.17
36.72/54.46
35.01/53.36
40.51/58.28
3.125
25.34/41.5 .95
29.62/46.13
35.25/53.83
36.11/54.73
39.90/58.33
2.125
23.99/46.39
28.15/48.25
31.70/50.67
31.09/49.01
36.84/54.93
34

---

Published as a conference paper at ICLR 2026
Table 7: MMLU (%)
zero-shot results for Qwen3 models quantized with HPTQ.
Avg. Bitwidth
0.6B
1.7B
14B
16.000
40.34
55.44
68.38
73.02
77.10
4.125
29.84
53.95
67.45
72.28
76.27
3.125
32.92
47.49
62.70
70.96
75.53
2.125
24.58
23.87
40.83
52.99
64.31
Table 8: PiQA (%)
zero-shot results for Qwen3 models quantized with HPTQ
Avg. Bitwidth
0.6B
1.7B
4B
8B
14B
16.000
67.30
72.31
74.92
77.80
79.87
4.125
66.00
70.78
75.30
77.42
79.54
3.125
62.08
68.44
73.01
77.53
78.78
2.125
54.13
57.40
66.76
72.52
75.46
Table 9: WinoGrande (%)
zero-shot results for Qwen models quantized with HPTQ.
Avg. Bitwidth
0.6B
1.7B
4B
8B
14B
16.000
56.43
61.48
65.27
68.11
72.53
4.125
54.38
59.67
64.09
67.17
73.01
3.125
52.72
58.72
64.80
66.93
71.19
2.125
49.80
49.96
53.04
59.19
66.06
Table 10: SciQ (%) zero-shot results for Qwen3 models quantized with HPTQ, with internal reasoning disabled.
Avg. Bitwidth
0.6B
1.7B
4B
8B
14B
16.000
83.5
91.2
93.5
95.7
96.8
4.125
80.7
88.9
93.3
95.6
97.1
3.125
76.6
89.9
92
95.4
96.8
2.125
40.8
62.8
81.2
86.8
93.8
35

---

Published as a conference paper at ICLR 2026
E.4 ACCURACY RESULTS FOR LLAMA MODELS
Tables I1 to [5 report the evaluation results for Bamm--3.-3BB-nnstrue Llama-3.1-8B-Instee and
Llama-2-77 2-7B models under the same setups in Section E.3
Table 11: Perplexity of Llama-3.2-3B-Instme model under HPTQ. GPTQ, and SSQR with different
bitwidths.
Method
Avg. Bitwidth
Perplexity
WikiText-2
C4
BF16 Baseline
16.000
11.01
13.49
4.125
11.27
14.64
HPTQ
3.125
12.51
15.81
2.125
22.58
29.82
4.125
11.96
15.37
GPTQ
3.125
15.20
18.99
2.125
357.69
172.89
4,445
11.38
14.95
SSQR-1%
3,445
13,48
18.38
2.445
83.41
67.19
4.765
11,50
14.77
SSQR-2%
3.765
13.20
16.65
2.765
45.93
41.69
5.085
11.39
14.64
SSQR-3%
4.085
12.50
16.10
3.085
37.41
30.74
5,405
11.53
14.69
SSQR-4%
4.405
12.33
15.96
3.405
23.74
27.59
5.725
11.47
14.69
SSQR-5%
4.725
12.29
15.81
3.725
22.94
25.44
36

---

Published as a conference paper at ICLR 2026
Table 12: Perplexity of L model under HPTQ, GPTQ, and SSQR with different
bitwidths.
Method
Avg. Bitwidth
Perplexity
WikiText-2
C4
BF16 Baseline
16.000
7.20
9.09
4.125
7.37
9.99
HPTQ
3.125
7.84
11.04
2.125
11.89
16.37
4.125
7.56
10.46
GPTQ
3.125
9.44
13.16
2.125
148.15
71.33
4.445
7.50
10.30
SSQR-1%
3,445
8.67
12.35
2.445
57.26
39.96
4.765
7.48
10.20
SSQR-2%
3.765
8.32
11.75
2.765
25.18
25.21
5.085
7.41
10.11
SSQR-3%
4.085
8.16
11.54
3,085
17.27
20.03
5.405
7.39
10.05
SSQR-4%
4.405
8.01
11.31
3.405
13.22
17.77
5.725
7.38
10.03
SSQR-5%
4.725
7.98
11.13
3.725
12.12
16.13
Table 13: Perplexity of Llama-2-7B model under HPTQ, GPTQ, and SSQR-1% with different bitwidths.
Method
Avg. Bitwidth
Perplexity WikiText-2
C4
FP16 Baseline
16.000
5.50
6.24
4.125
5.53
6.73
HPTQ
3.125
5.77
7.04
2.125
7.45
9.43
4.125
5.70
6.90
GPTQ
3.125
6.75
8.08
2.125
28.07
26.13
4.445
5.60
6.81
SSQR-1%
3.445
6.09
7.52
2.445
14.58
15.85
37

---

Published as a conference paper at ICLR 2026
Table 14: Zero-shot evaluation results (%) for 1attma-..2----Istttiit under different quantization methods.
Method
Avg. Bitwidth
Wino
MMLU
PiQA
SciQ
HSwas wag
aCCnomm
BF16 Baseline
16.000
68.75
62.18
76.17
95.4
53.27
71.65
4.125
68.03
61.57
76.55
95.0
53.02
71.28
HPTQ
3.125
68.35
58.50
7497
95.5
51.76
70.00
2.125
60.85
42.75
69.15
89.9
44.54
60.29
4.125
68.11
59.81
75.73
95.5
52.29
70.54
GPTQ
3.125
66.06
49.13
72.58
94.0
47.25
63.93
2.125
50.59
22.96
53.65
63.4
28.06
30.58
4,445
68.19
60.94
76.12
95.7
52.37
70.88
SSQR-1%
3.445
66.93
54.10
74.92
95.6
50.55
68.86
2.445
51.70
23.97
58.22
64.5
31.14
36.90
4.765
68.03
61.17
76.33
95.2
52.49
70.99
SSQR-2%
3.765
65.51
56.37
74.43
94.4
50.88
68.85
2.765
53.12
23.91
60.01
78.3
34.12
42.99
5.085
68.27
61.68
76.82
95.4
53.03
71.29
SSQR-3%
4.085
66.69
57.65
75.03
95.0
50.98
69.00
3.085
58,48
34.20
65.61
90.5
39.87
52.43
5,405
68.90
61.11
76.28
95.5
52.80
71.03
SSQR-4%
4.405
66.77
57.73
75.03
95.3
51.08
68.79
3.405
57.85
33.74
66.49
90.1
40.66
54.44
5.725
68.35
61.67
75.57
95.3
52.88
70.97
SSQR-5%
4.725
66.69
57.02
75.52
95.3
51.32
69.70
3.725
57.38
37.24
65.56
91.5
41.37
54.96
38

---

Published as a conference paper at ICLR 2026
Table 15: Zero-shot evaluation results (%) for 1---aa...-------s----- under different quantization methods.
Method
Avg. Bitwidth
Wino
MMLU
PiQA
SciQ
HSwag wag
aCCnomm
BF16 Baseline
16.000
73.72
68.31
80.14
97.3
59.81
79.59
4.125
73.56
67.90
79.49
97.7
59.57
79.25
HPTQ
3.125
72.77
64.58
79.16
96.9
58.42
78.21
2.125
63.69
45.01
69.15
90.8
49.84
67.98
4.125
73.80
65.68
79.27
97.2
58.61
78.36
GPTQ
3.125
72.45
58.19
77.37
95.5
55.21
74.57
2.125
54.93
24.67
54.46
75.1
31.77
37.79
4,445
74.43
66.78
79.65
96.9
59.18
78.93
SSQR-1%
3.445
72.45
60.14
77.97
96.3
56.74
76.24
2.445
52.80
23.07
58.49
74.1
33.25
40.05
4.765
73.80
67.21
79.49
97.2
58.94
78.53
SSQR-2%
3.765
73.24
63.13
78.78
96.4
57.63
77.22
2.765
54.30
27.08
61.04
82.5
38.41
50.41
5.085
72.93
67.38
79.54
96.9
59.64
79.07
SSQR-3%
4.085
73.09
63.77
79.11
96.6
57.62
77.40
3.085
54.54
26.15
58.81
83.6
38.34
49.52
5,405
73.24
66.95
79.92
96.9
59.32
79.06
SSQR-4%
4.405
73.24
62.92
78.73
96.5
57.61
77.47
3.405
54.54
29.95
54.95
82.3
39.80
51.87
5.725
74.03
67.91
80.52
97.2
59,49
79.39
SSQR-5%
4.725
73.40
64.14
79.05
97.0
58.16
77.63
3.725
64.25
42.59
72.58
88.7
49.94
68.20
39

---

Published as a conference paper at ICLR 2026
E.5 COMPARISON WITH OTHER QUANTIZATION METHODS
We compare zero-shot WinoGrande and PiQA accuracies of our methods (HPTQ, SSQR) against GPTQ and state-of-the-art post-training. weight-only quantizers AQLM (Egiazarian al., 2024), QulP# (Tseng et al. (024a), and QTIP (Tssng et al., 2024b) on Llama-2-7B Results are reported Table 16 sorted by average bitwidth. Metries for AQLM, QuIP#, and QTIP are taken from their respective papers.
As shown in Table 16 for average bitwidth 4, all methods yield accuracy close the full-precision baseline. In the 3-4 bit regime, vanilla GPTQ falls behind recent methods; however, HPTQ and SSQR close this gap, bringing scalar quantization approach to parity with vector quantization methods (AQLM, QulP#, QTIP). In the 2-3 bit regime, HPTQ remains competitive with the state of the art.
Table 16: Comparing the zero-shot results of different quantization methods on Llama-2-77
Method
Avg. Bitwidth
WinoGrande
PiQA
FP16 Baseline
16.000
69.46
78.13
AQLM
5,020
67.40
78.29
SSQR-1%
4.445
68.82
78.35
HPTQ
4.125
69.61
77.75
GPTQ
4.125
68.82
77.97
AQLM
4.040
67.32
78.24
QulP#
4.000
67.60
78.40
QTIP
4.000
67.10
78.40
SSQR-19 1%
3.445
65.43
77.15
HPTQ
3.125
67.72
77.80
GPTQ
3.125
64.96
73.88
AQLM
3.040
66.93
76.88
QulP#
3,000
66.50
77.30
QTIP
3,000
66.90
78.10
SSQR-1 1%
2.445
50.04
56.15
AQLM
2.290
65.67
74.92
HPTO
2.125
65.82
73.56
GPTQ
2.125
49.64
56.20
AQLM
2.020
65.67
74.76
QulP#
2.000
64.90
75.10
QTIP
2.000
64.70
75.90
40

---

Published as a conference paper at ICLR 2026
E.6 TECHNICAL DETAILS AND PERFORMANCE OF SSQR'S CUDA KERNEL
The kernel is specialized for two regimes: in the low-batch regime, the kernel utilizes SIMT GPU cores exclusively, while tensor cores utilized when batch size is 28, the smallest outer dimension where tensor cores can be utilized without padding, and with 16-bit operands and 32-bit floating-point accumulators. For both regimes, sparse outliers are handled with SIMT cores.
To handle the dense inliers, we apply two reordering schemes here. First, the weights are reordered for memory movement involving tensor cores. Second, apply an additional reordering scheme enable batched conversion between 2-4-bit integers into their 16-bit counterparts.
To handle the sparse outliers, we group sparse outliers in groups of rows (matching the outer tensor core dimension), then store them in polumn-majo, row order with padding to account for differences between non-zero counts across rows the group.
Figure 5 shows the layer-wise speedup of the SSQR kernel on NVIDIA RTX 6000 GPU compared the PyTorch BF16 matrix multiplication baseline across different layer shapes in the Qwen3-8B model (layers with the same input are merged), inlier bitwidths, outlier rates, and batch sizes. We observe the largest gains in low-batch regime, with up to 4x speedup when <1 % outliers are present. As the outlier rate incrasss, the speedup diminishes, but the kernel consistently outperforms the BF16 baseline across all settings.
41

---

Published as a conference paper at ICLR 2026
5
Speedha (Q.K-VAA
3
16
2
16
16
Speedup (0
3
3
16
2
16
2
Speatup (Gate
Specdap (Down
3
2
2
0
16
16
Batch Size (2-Bit Inlier)
Ratch Size (3-Bit Inlier)
Batch Size (4-Bit Inlier)
Ouilier Rase
5%
Figure 5: Layer-wise inference speedup of the SSQR kernel over the PyTorch BF16 baseline on Qwen3-8B across inlier bitwidths, outlier rates, and batch sizes on A6000 GPU.
42

---

Published as a conference paper at ICLR 2026
F LLM USAGE
LLM was used to aid and polish the writing of this paper, e.g., correcting grammar and rephrasing sentences.
43


Reviews:
[{"review_id": "VgouQDtwid", "reviewer": "Reviewer_64UP", "summary": "The paper considers the GPTQ algorithm, and mentioned how running it back-to-front is identical to Babai’s nearest-place algorithm. The author show several theoretical results and also propose practical schemes for algos that utilize these insights.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The intro of the paper is clearly written\n2. Algos 1 and 2 paint a clear picture, although GPTQ is a little more involved than what Algo 1 depicts.\n3. The theoretical results are solid, i went over the theorems and corollaries. The equivalence b/w GPTQ and Babai’s nearest-plane is clean. \n4. The experiments are conducted on the Qwen3 series of models, representing near-state of the art models.\n5. HPTQ seems to be competitive in experiments", "weaknesses": "1. I dont agree with the statement that GPTQ’s inner workings are “adhoc”. GPTQ is inspired from the Optimal Brain Surgeon/Compression literature (OBS). OBS is quite principled and its update can be derived using KKT conditions, making it quite rigorous, Except the optimization bit for GPTQ which is quite sloppy, GPTQ is a pretty grounded method.\n2. The role of clipping is a bit confusing, maybe required in practice. Although the results on Qwen3 are good, would be good to extend to other model families.\n3. I am not sure if the experiment results offer something practical. I am confused about the main contribution of this paper. Are you claiming that this is a theoretical paper? Or are you claiming that there is a better version of GPTQ? Because several papers have already practically bettered GPTQ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdjvg3fblV", "reviewer": "Reviewer_fgCD", "summary": "This paper establishes a formal equivalence between the GPTQ algorithm and Babai's nearest plane algorithm from lattice theory. This connection provides a theoretical foundation for a method that has thus far been understood primarily as a sequence of heuristic, greedy updates. The core theoretical insight is that when GPTQ is executed in a back-to-front order, its process of quantizing a weight and propagating the error is mathematically identical to Babai's algorithm solving the Closest Vector Problem (CVP). Motivated by the new foundation, they propose a new method: Huffman-encoded post-training quantization (HPTQ).", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper's primary contribution is establishing a rigorous mathematical equivalence between the GPTQ quantization algorithm and Babai's nearest plane algorithm from lattice theory. \n\n2. The theoretical foundation leads to practicaly improved method, including a formal error bound for GPTQ and the design of new, improved quantization methods like HPTQ. These methods demonstrably outperform the original GPTQ.\n\n3. The presenation is clear and well-written.", "weaknesses": "1.  The experimental evaluation primarily benchmarks the proposed methods (HPTQ, SSQR) against GPTQ and Round-to-Nearest (RTN). They didn't compare with strong baseline method like QuaRot, QuIP, SpinQuant, OmniQuant. Although GPTQ is a well known quantization method, without comparing with other recent, strong baselines limits the practical value of this work. \n\n2. The paper derives a worst-case error bound, but its utility is questionable. The bound is only valid in the no-clipping regime, which is not the standard use case. Besides, the proposed \"min-pivot\" ordering is shown to provide only a modest practical improvement over the much cheaper \"act-order\" heuristic.\n\n3. The paper introduces non-trivial computational costs without a thorough analysis. HPTQ requires a binary search over scales to hit a target Huffman bitrate, and SSQR requires a similar search to meet an outlier budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuhS9fOAXM", "reviewer": "Reviewer_ej6W", "summary": "The authors identify the equivalence between GPTQ and Babai's nearest plane algorithm (Section 4.1 and 4.3), and derive a tight error bounds for GPTQ in the no-clipping regime (Section 4.4). Based on this understanding, they design two variants of GPTQ (SSQR and HPTQ; Section 5) with better accuracy than GPTQ and implement it in CUDA with considerable speedup.", "questions": "1. You are claiming your bound (Theorem 5) is \"tight\". Can you construct a \"hard case\" that attains your error bound?\n1. What's the counterpart for GPTQ for running Babai's algorithm with LLL/BKZ basis reduction?\n1. Why did everyone suddenly starts to notice the equivalence between GPTQ and Babai’s algorithm? The authors mentioned a [concurrent work](https://arxiv.org/abs/2508.01077), and I assume that's not a coincidence?\n    - You don't have to answer this if you are feeling uncomfortable; I'm just curious about how you discovered the connection between GPTQ and Babai's algorithm.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provides a geometric interpretation and theoretical guarantee for GPTQ (not quite due to order and clipping, but close enough!)\n1. This is a bridge paper that connects the modern LLM quantization algorithms with the classical CVP literature\n1. The mathematical proofs (both equivalence and error bound) are non-trivial\n1. The CUDA inference kernels for SSQR and HPTQ have impressive speedup despite GPU-unfriendly designs such as outlier storage in CSR and Huffman coding.", "weaknesses": "1. Gap between theory and practice:\n    1. People typically run GPTQ front-to-back, which is not consistent with the back-to-front order studied in the paper\n    1. Clipping is inevitable in practice due to the limited range of quantized types, but the authors focus on the no-clipping scheme\n    1. From my understanding, Babai's algorithm is most attractive when preceded by LLL/BKZ basis reduction, but GPTQ is equivalent to \"Babai’s algorithm without basis reduction\".\n1. SSQR and HPTQ only has modest improvement over GPTQ, and far from SToA weight-only quantizers like [QTIP](https://arxiv.org/abs/2406.11235).\n1. Insufficient experiments: all experiments are ran on Qwen3-8B, but ideally we want to see the performance across different scales and/or model families.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9KmBRd5kbC", "reviewer": "Reviewer_PGNw", "summary": "The paper shows how GPTQ, a well-established method for model weight quantization, is mathematically equivalent to Babai's nearest plane algorithm, a technique for tackling the closest vector problem on a lattice. In the case of GPTQ, the lattice is defined by the Hessian matrix of the layer's inputs. By leveraging the connection, the authors are able to apply the error guarantees from Babai's algorithm to the GPTQ algorithm, as well as give a geometric interpretation to its quantization steps.", "questions": "- Reading Algorithms 1-3 without the corresponding sections is a bit difficult — I wonder if captions detailing the procedures might help readability? (I leave this at authors' discretion) \n- The authors provide an optimised kernel for SSQR. It would be useful to also highlight the implications of the Huffman encoding/decoding in terms of practical efficiency/speed.\n- Minor: On line 158 the authors mention parameter \"delta\", but this is not defined/mentioned elsewhere so it wasn't quite clear to me as I wasn't deeply familiar with the LLL algorithm.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and easy to follow. I was familiar with the GPTQ algorithm and its usage in quantization, but not with the nearest vector on a lattice problem; I believe the authors were able to clearly establish the connection, and introduce both techniques in a clear and instructive way. Even if the connection might have limited algorithmic implications (in terms of improvements to GPTQ), I believe it still offers valuable insight into a widely-used quantization technique. I do not think the connection is necessarily obvious to the quantization community, and thus I believe this is a solid contribution and a helpful paper. A few specific comments worth highlighting:\n- The paper is well-structured and the mathematical notation is clear and easy to follow.\n- Figures 1-3 are very helpful in explaining the geometric interpretation of the algorithm.\n- The authors highlight the importance of \"no clipping\" for full equivalence of the two algorithms, and provide solutions using the SpQR method for detecting outliers, and Huffman encoding for lossless compression.\n- The authors helpfully provide an optimised kernel for the SSQR technique as part of their submission.", "weaknesses": "I view the paper's main contribution as the establishment of the theoretical connection between the two widely used algorithms. The algorithmic contributions (SSQR and HPTQ) are relatively incremental, as SSQR is a direct application of SpQR to the GPTQ method, and Huffman coding has indeed been applied to quantization problems previously (see e.g., https://arxiv.org/abs/1612.01543). This however is clearly indicated by the authors, and does not diminish the contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)