GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Authors: Jinuk Kim
Marwa El Halabi
Wonpyo Park
Clemens JS Schaefer
Deokjae Lee
Yeonhong Park
Jae W. Lee
Hyun Oh Song
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Quantization
Post-training quantization
One-shot quantization
Primary area: deep_learning->large_language_models
TL;DR: We propose GuidedQuant, a quantization framework that integrates gradient information from the end loss into the quantization objective while explicitly modeling inter-weight dependencies.

Abstract: Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Jinuk Kim 12f Marwa El Halabi
Wonpyo Park Jac W. Lee'
Clemens JS Schaefer Deokjae Lee Hyun Oh Song" 12
Yeonhong Park
Abstract
Table I. Summary of results of GuidedQuant applied to state-of- the-art PTQ methods on the Llama-2-78 model. Wiki2-4K and Wiki2-2K represent perplexity on Wiki'exx2 dataset with con- text size of 4096 and 2048, respectively. W4A4KV4 indicates
Post-training quantization is key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. How- ever, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output chan- nels. GuidedQuant consistently boosts the per- formance of tlate-of-the-arr quantization methods across weight-only scalar, weight-only vector, and weight-and-aactivtion quantization. Additionally, we introduce novel non-uniform scalar quantiza- tion algorithm, which is guaranteed to monoton- ically decrease the quantization objective value, and outperforms existing methods this category. We release the code at https: //gi thub. com/ wnt-mmlaa//uiieeuuue
quantization of all weight, activation, and KV cache to bits.
Method
Bits|,
Wiki2-4K,
Type
Original
16
5.12
Weight-only
SqueezeLLM
2.01
39.58
LNQ (Ours)
2.01
23.31
Scalar
LNQ GQuant (Ours)
2.01
8.83
Weight-only
2.00
Vector
GQuant (Ours)
2.00
Method
Bits]
Wiki2-2KJ
Type
Original
16
5.47
Weight-and-
SpinQuant
W4A4KV4
5.95
Activation
SpinQuant GQuant (Ours)
W4A4KV4
5,89
model weights, reducing data transfer and thus accelerating inference in memory-bound scenarios such as small-batch inference (Gholami et al., 2024; Kim al., 2024; Tseng et al., 2024b). On the other hand, weight-and-activation quantization methods quantize both the model weights and activations. In addition to reducing data transfer, these methods also speed arithmetic operations, making them particularly heneficial for large-batch scenarios such as pre-filling input tokens or generating batched samples (Ashkboos al., 2024; Liu al., 2024). Weight-only quantization techniques have used three grid types: uniform scalar (Frantar al., 2023), non-uniform scalar (Kim et al., 2024), and vector quantization (Tseng et al., 2024b; van Baalen et al., 2024), each with its own advantages (see Section for details). In contrast, bright-and-activatiie methods typically use a uniform scalar grid, as using a non-uniform grid would require dequantization before multi- plication, preventing the use of faster arithmetic operations.
1. Introduction
Large language models (LLMs) have shown remarkable capabilities across range of tasks, from text generation to complex reasoning. However, these advancements come at the cost of substantial memory usage and inference latency. Quantization provides an effective solution to these chal- lenges. Weight-only quantization methods quantize only the
!Wrk partly done during an internship at Google. 'Department of Computer Science and Engineering, Seoul National Univer- sity "Neural Processing Research Center "Samsung AI Lab, Montreal 'Google. <hyunoh aor.oh@snu.cckkr
Quantization benefits come at the cost of performance degra- dation. Alaatization-Awarr Training (QAT) methods rely on retraining the quantized model to mitigate this, which is prohibitively expensive at the scale of modern LLMss. In constrast, Post-Training Quantization (PTQ) methods quan- tize the pretrained model using a small calibration dataset or
Correspondence to: Hyun Oh Song
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
2f 3XX
E Raxdout
X Ruxd
W E Rul Xdout
W E R Xdout
Jg
Vj dout)
H = X DDag of )"X
vk (1.....))
(g < dout)
H
= Mll
1
H
Wg c
H
W
WiJe
H2
W W
Figure 1. Top: The proposed GuidedQuant's layer-wise quantization objective (4). Bottom-left: Its equivalent quadratic form (6). Bottom-right: The approximated objective (7) proposed in Scction 3.2. We denote the input, weight, and quantized weight matrices as X E Raxdn W E did ddant, and W E RRd Xdast, respectively. The groups JI,...,JJ form partition of the set {1,..., dout J. and Z, E doduut denotes the j-th column Z XW.
no data, without retraining the entire model. Most existing PTQ methods for LLMs rely on surrogate objective rather than the end loss to make quantization feasible.
computes saliency scores for layer outputs using the gra- dients of the end loss with respect to these outputs. Each layer is then quantized independently by approximately min- output errors. Unlike
imizing the sum of previous methods that assume diagonal Hessian, this ob- jective is equivalent to refined quadratic approximation assuming block-diagonal Hessian, again approximated by the empirical Fisher information matrix. While cross-layer and cross-ore cross-output channel interactions are still ignored, de- pendencies within output channels are preserved, enabling more accurate estimation of quantization? impact on the end loss.
One common PTQ strategy, which refer layer- wise output-based quantization, aims quantize each layer by minimizing the mean squared error between the layer's 's original output and the quantized one (Nagel et al., 2020; Frantar et al., 2023; Egiazarian et al., 2024; Chee et al., 2024; Tseng et al., 2024a;b; Liu et al., 2024). However, this strategy treats all hidden features equally, overlooking their varying impact on the end loss.
Alternatively, methods such as Choi et al. (2017); Kim et al. (2024) leverage gradient information from the end loss to assess the impact of individual weight errors. This is done by computing the gradient of the end loss with respect to weights via a single backpropagation step on calibration dataset. Saliency scores are then assigned to weights based on these gradients, and the model is quantized by approx- imately minimizing the sum of saliency-weight errors. This objective corresponds to a quadratic approxima- tion of the change in the end loss, based on its second-order Taylor expansion, where the Hessian is approximated by the diagonal of the empirical Fisher information matrix (Has- sibi Stork, 1992). A key limitation of this approach is that it ignores weight interactions, which are crucial for overall performance.
Computing and storing the diagonal blocks of the Fisher matrix for a given layer is too expensive for modern LLMs. To address this, we partition the layer's 's outputs into a small number of groups and average the Fisher matrix's 's blocks within each group (Figure 1). Other block-diagonal Fisher matrix approximations of the Hessian have been used for pruning CNNs (Singh & Alistarr, 2020) and BERT LLMs (Kurtic et al., 2022) with arbitrary blocks along the diagonal, and for quantizing CNNs (Li et al., 2021) with diagonal blocks corresponding to the model's residual blocks (see Appendix E.11 for more details). However, our work is the first make this approach computationally and storage- efficient the scale of modern LLMs..
weight
GuidedQuant can be applied as direct plug-in to any layer- wise output- based PTQ method. We demonstrate its effec- tiveness by integrating it into the current state-of-the-arr methods for weight-only vector quantization, QTIP (Tseng et al., 2024b), and weighh-and-activatiie quantization, Spin- Quant (Liu et al., 2024), which are both layer-wise output-
Contributions In this work, we propose GuidedQuant, a novel PTQ approach that integrates gradient information from the end loss while preserving wright dependen- cies within output channels. In particular, GuidedQuant
2

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
based PTQ methods. GuidedQuant consistently improves their performance (Table 1).
1992), is the following quadratic approximation of the change in the end loss
For weight-only scalar quantization, the current state-of- the-art methods are SqueezeLLM (Kim et al.. 2024) and GPTVQ ID (van Baalen et al., 2024). Since GPTVQ 1D a layer-wise output-based PTQ method, GuidedQuant can be applied to it. However, GPTVQ ID employs a suboptimal algorithm for minimizing layer-wise output er- rors. To address this, introduce a novel Layer-wise Non-uniform Quantization method, LNQ. which minimize layer-wise output errors using an alternating minimization algorithm, where the codebook is optimized in closed-form, and assignments are optimized via a coordinate descent (CD) algorithm. LNQ outperforms GPTVQ ID and matches or surpasses SMM. Applying GuidedQuant LNQ further improves its performance, achieving oate--ofthheann results (Table 1).
(w) ((w) ~ ]( -
w). (2)
This approximation derived from the second-order Taylor approximation of l, assuming that the trained model has converged and thus the gradient is close to zero. Since com- puting the Hessian is infeasible even for small models, a pop- ular approach first proposed in Hassibi & Stork (1992) ap- proximates the Hessian by the empirical Fisher information matrix F = , E (e,(w) V(:(w) T, which yields the following quadratic approximation, (w-w) w) F(w (w-ww ww).
SqueezeLLM (Kim et al., 2024) is weight-only non- uniform scalar PTQ method for LLMs which uses this quadratic approximation, but further approximates the Fisher information matrix by its diagonal dFag(F), ignoring off off-diagonal entries. The resulting objective is given by
2. Preliminaries
(w- w) T ""ddia((FF)) -w) = Σ Fix ( - wx)² (3)
Consider a neural network with L linear layers, trained with a loss function l and a calibration data of size n. We de- note the loss computed on the i-th data point as &z. Let ( E R4 x be the weight matrix of the I-th linear layer, where each column vector w (1) E TR corresponds to an output channel. We denote its quantized approximation as Wt), The input and output feature maps of this layer are x!) e R=* and (t) E Raxdll,, respectively. The out- put of the linear layer is computed as (9 = (OW()
For non-uniform scalar quantization, minimizing this ob- jective corresponds to solving weighted k-means problem in ID, which can be solved exactly using dynamic pro- gramming algorithm (Grønlund et al., 2017). SqueezeLLM instead employs Lloyd's algorithm with k-means++ initial- ization (Lloyd, 1982; Arthur & Vassilvitskii, 2007), which is only guaranteed to achieve a (log k) approximation in expectation, where k is the number of clusters, but is faster in practice (Hyun, 2024). However, the diagonal approxi- mation is highly inaccurate, as both the Hessian matrix and its Fisher approximation are usually strongly non-diagonal, as observed in prior work for small CNNs (Hassibi & Stork, 1992; Singh & Alistarh, 2020). We also confirm this observa- tion for the Fisher matrix of Llama-2-7B in Figures and 4.
and the output after quantization as 7, = X W() (L)(W(()))T and w [vec( W()) ..... vec (WO)TT be the vectors of weights in all T. layers before and after quantization, where vec (e) corresponds to stacking the columns of W
Let
[vec(W()))
Most existing PTQ methods for LLMs are luier-wise output- based quantization methods, which quantize each layer by approximately minimizing the objective
For additional related work, see Appendix A.
3. GuidedQuant
In this section, we introduce our PTQ approach Guid- edQuant. We first propose a layer-wise quantization ob- jective that more accurately approximates the impact of quantization on the final loss compared to surrogate objec- tives used in existing PTQ methods. We then present sim- plified version of this objective, making it computationally and memory efficient for LLMs with up to 70B parameters.
19 W WOO - W 12
2,00
2 (1)
1=1==
ignoring the varying impact of outputs on the end loss Existing methods employ various heuristics to minimize this objective, such AdaRound (Nagel al., 2020), CD meth- ods (Nair& & Suggala, 2024; Behdin et al., 2023; Egiazarian et al., 2024; Chee et al., 2024), OBQ (Frantar & Alistarh, 2022), GPTQ' (Frantar et al., 2023), GPTVQ (van Baalen et al., 2024), and AQLM (Egiazarian et al., 2024).
3.1. Objective
As discussed earlier, most existing PTQ methods treat all output features as equally important, by employing the sur- rogate objective in Eq. (1). In contrast, we propose to modify this objective to account for the varying impact of each output feature on the final loss.
more accurate proxy objective, first introduced in early pruning methods (LeCun et al., 1989; Hassibi & Stork,
1 Also referred to as OPTQ.
3

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Remark 3.1. The sum of the layer-wise objective in Eq. (4) over all layers is equal the following quadratic approxi- mation of the change in the end loss
Original-FPI6
SqueezeLLM (weighted k-means)
LNQ (layer-wise)
LNQ Guidedduant (our objective)
40
Widiled2 Porplexity
24
(0)
7
(w) W
(5)
8
6.0
The proof of Remark 3.1 follows from the chain rule, and is given in Appendix B. A similar observation was made in Molchanov et al. (2019).
5.0
2.0
3.0
4,0
Assuming that the quantization grid used is separable over layers, which typically the case, minimizing the objective in Eq. (5) is equivalent independently minimizing
Bits
Figure 2. Non-uniform scalar quamtization results on Llama-2-7B with different objectives: layer-wise output error objective (1) used in L.Q (Algorithm 2), weighted l-means objective (3) used in Squeezel..MM, and our approximated GuidedQuant objective (7) used in LNQ combined with GuidedQuant. We report perplexity WikiText2 with a context size of 4096. Results are from Table 3.
(6)
for every layer, where H = n nF or equivalently the layer-wise objective in Eq. (4).
To that end, we approximate the change in the end loss resulting from the output feature z! changing to Z after quantization, using a first-order Taylor expansion, assuming independence of output features:
Thus our proposed objective is a more accurate approxima- tion of the change in the end loss than the layer-wise output at error objective (1), which assumes 82 x I, as well as the weighted k-means objective (3) used in SqueezeLLMM which ignores all off-diagonal entries in the Fisher matrix including those within the blocks p("). As result, our approach achieves better performance, even with the ad- ditional approximation discussed in Section 3.2, as high- lighted in Figure for non-uniform sealar quantization, and later across other formats Section 5.
( ( ~
08
1
2.5 (
0z
Accordingly, we propose to scale each output error by the gradient of the end loss with respect to that output, leading to the following wase objective:
ae oz
In Figures 3 and 4, visualize submatrix of the Fisher information matrix corresponding to the first two output channels in the linear layers of the first Transformer block of 2-7B. The visualization confirms that the Fisher matrix exhibits strong off-diagonal values and a prominent Block-diggonal structure, with blocks corresponding to F for the two output channels E {1,2].
3 (x
(
X
W())
2
11) (2 1
de
(4)
where C denotes the element-wiss multiplication. This crite- rion was previously proposed in Molchanov et al. (2019) for pruning neurons and filters in vision models, where pruning the jthh neuron in layer / corresponds to setting ( 0.
3.2. Averaging Approximation
The layer-wise output error objective (1) can be written as
We note that the objective in Eq. (4) can be viewed as sim- plification of the second-order Taylor approximation of the change in the end loss given in Eq. (2), where the Hessian is approximated by the empirical Fisher information matrix, and where interactions between weights belonging to differ- ent layers or output channels of the same layer are ignored. In other words, we adopt block dlack-diagonal approximation of the Fisher matrix where we only keep the d (() blocks F 1 n Σi. dé,
(0)
H
W w
W
where HO = (X E T x Most existing heuristics for optimizing this objective, such as GPTQ (Fran- tar al., 2023) and CD (Nair & Suggala, 2024; Behdin et al., 2023), require access only to H and not X Thus, the Hessian matrix (9) is typically precomputed, which reduces the peak memory usage during optimiza- tion, since (1) E Rax is much larger than H(U), given
) DE, )T,,)T corresponding to
aw! (w((1)
interactions within each output channel j of every layer 1, and ignore all off-block entries.

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
that d0 in << n. Additionally, the precomputed Hessian can
Algorithm 1 GuidedQuant
be reused across multiple quantization configurations and bit- bit-widths, amortizing the cost of its computation.
input Layer-wise quantization algorithm Q, number of
groups g. number of linear layers L
J + Lou (k-
Our proposed objective (6) can be seamlessly integrated
1) + 1,
ont k), VI E [L] k E [g]
into any layer-wise oulput based quantization method by replacing H()) by H n. for each output channel j-
2: 5k
( f Ju
EjEJa Je
ae )² AI E [ gg]
3: for all 1 E [T,,k [g] do
However, precomputing and storing H for all j incurs memory cost of )² (dd and a time complexity of (((( pd per layer l. This is infeasible at the scale of modern LLMs, where both dl and d exceed 10², and is much larger than both.
4: H
(1)
t x Diag (s X
5: W
(1)
+ Q
W [.. [1))
end for
output W() W
To address this challenge, we partition the output channels of each layer into y distinct groups (g- < dea ) and replace the individual Hessian matrices H ( within each group k by shared matrix ! obtained by averaging H within the group. Formally, let "!!,...,_!! be a partition of the set {I...., d For each group k = 1,....9. 9. we define H = 1 Ejegge H The resulting layer-wise objective then becomes
In our implementation, we scale the gradients by a large constant (we used 10 in all experiments) while computing the averaged Hessians H₃ to prevent underflow.
GuidedQuant quantizes each layer independently by ap- proximately minimizing the layer-wise objective in Eq. (7). A complete overview of GuidedQuant is provided in Algo- rithm 1. As discussed, the layer-wise quantization algorithm Q can be any layer-wise output based quantization method. The gradient computation (Line 2) requires single back- propagation step on the calibration dataset. During this step, we only store the averaged squared gradients (1) which requires O()gL) storage.
H (1)
(
(1
w.))
(7)
Note that by the chain rule, we can write
The total memory cost of GuidedQuant (without the back- propagation step) is then OLLg(d,, + n)), and its total time complexity is O (Lg(nd + To(din (mur//))). where du dout are the largest input and output channel dimensions across all L layers and To (d d2) is the time complexity of quantizing di x d2-weight matrix using Q. Each step in the for loop (Lines 3-6) can be done in parallel for all groups and layers. As previously discussed, the Hessian matrices H 's only need to be computed once, and can be reused for different quantization configurations and bit-widths.
TD = ( Diag
aE
x(,)
Öz
where Diag( _ is the diagonal matrix whose diagonal entries are the element-wise square of the gradient of é with respect to the jth column zj
(1) of ( We can thus compute
H by averaging the squared gradients:
de 2
H X (()
1 JJH|
4. Layer-wise Non-uniform Quantization
Oz
(1)
JEJ&
The choice of the layer-wise output based quantization method Q in GuidedQuant is critical to its overall perfor- mance. For weight-only non-uniform scalar quantization, the current satte-ff-the-att layer-wise output based quan- tization method is the 1D variant of GPTVQ (van Baalen et al., 2024), which alternates between optimizing the code- book via gradient descent and the assignments via GPTQ algorithm (Frantar al., 2023). However, both of these steps can be improved. Given fixed assignments, the code- book admits an optimal closed form solution. Also, for optimizing assignments, recent works have demonstrated that coordinate descent (CD) methods outperform GPTQ in uniform weight-only quantization (Behdin et al., 2023; Nair & Suggala, 2024). this section, we introduce Layer-wise
This averaging approximation reduces the number of d x ( Hessian matrices that need to be computed for each layer from d()) to g (Figure 1). Computing and storing ! for all k requires significantly lower memory cost of (((())))) and time complexity of ((n(d()) per layer (assuming the squared gradients averages are already computed), making the method scalable. To partition the output channels, we use a simple strategy that groups ev- ery d'l) /g consecutive channels into a single group. This simple approach works well in practice, though more sophis- ticated clustering algorithms may yield additional benefits.
5

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Algorithm 2 LNQ
where 1 is the vector of all ones. Note that the optimization for each column j is independent of other columns, and can be done in parallel.
input Hessian of the objective H E 12dnxdu input weight W E Rd Xdeut, initial assignment p() E Rd Xm for each output channel j.
1: H- -LL'
{Cholesky decomposition)
4.2. LNQ Algorithm
2: for j E (I,...,,onn ,dont do
We propose LNQ, alternating minimization algorithm, which iteratively updates the codebook cli) and assignment matrix P(J) for each j. optimizing one while keeping the other fixed. Alternating mininization is a common strategy used by most non non-uniform quantization methods, including SqueezeLLM and GPTVQ. LNQ quantizes each layer inde- pendently. We present an overview of LNQ. applied to one layer with weights W IR X in Algorithm 2.
3: fort to I do
4:
c(i) -- (PO)TLL LL)) LL w,
5:
W, -- p())
6:
for k = to K do
7:
for: i 1 to din do
1 - argmin (w, w) H H(w,- w,)
8:
W, E|
12 122
9:
Wy + og
(3)
Given fixed assignment matrices pú), Problem (8) reduces to a standard least-squares problem, which admits closed- form optimal solution c = (XP())+X Xw where i denotes the PeooreePenrose pseudoinverse. We assume that the matrix p HP())) is invertible, where recall that H = X X Under this assumption, the closed-form solution is:
1 q'
10:
Vg E m. p())
{ 0 otherwise.
11:
end for
12:
end for end for
13:
clt) + (PU)TLL PPO) - PL w
14:
15:
end for
c = -(DS)THH()) "PO)THI Hw
output W [P())()) ,...,, ((d..,,(dd..)) ((dww) p(dew))(dow))
In practice, HP))H(()) is not always invertible, even when H is invertible (for example if no weight is assigned to given codebook value of)).. To address this, we add a small constant À = 10-7 to the diagonal of the matrix, as com- monly done in prior work (Frantar & Alistarh, 2022; Frantar et al., 2023; van Baalen et al., 2024). In implementation, use torch. 1 linalg....... g. lstsq function to compute the least squares solution in Eq. (9), which takes XPO) and Xw, as inputs. However, since X is not explicitly stored, we compute the Cholesky decomposition of H = X XX de- noted as H = LL T. and instead provide L [P and Lw, to the solver. Because Cholesky decomposition requires H to be positive definite, we ensure this by adding a small constant to the diagonal of H.
Non-uniform Quantization (LNQ), an alternating minimiza- tion algorithm which leverages the closed form solution for the codebook and employs CD to optimize the assign- ments. We then discuss its theoretical guarantees, as well as the memory cost and computational complexity under our efficient implementation.
4.1. Optimization Problem
We omit the layer index for notational simplicity through- out this section. Following prior work, we assign to each output channel a separate codebook, though LNQ can be easily adapted to finer-gramularity grouping. Non-uniform scalar quantization maps each scalar weight in the column Wi E dR to one of m = 2 real values (1) ...., J. where b E N is the target bit-width. The quantized weights W, can then be expressed as w = p(), where cb)) E I. is the vector containing the codebook values tof) ..... ( and p(f) E {0, djchnxm is the assignment matrix such that p(u) p(i) otherwise.
For fixed codebooks c Problem (8) can be equivalently written as
(
daut
minimize
> (w, w,) H(W, wy). (10)
Wr E(e Co
G2
j=1
iq 1 if Wi, is assigned to oy and
Even in the special case of uniform codebook, this problem corresponds to a closest vector problem with box constraints, which is NP-Hard approximate within any constant factor approximation for m > (Arora al., 1997, Theorem 1). Existing heuristics for solving it include OBQ (Frantar & Alistarh, 2022) which does not scale to LLMs with bil- lions of parameters; its faster variant GPTQ (Frantar et al., 2023); LDLQ (Chee et al., 2024), which is a more efficient implementation of GPTQ; greedy CD (Nair & Suggala, 2024); and cyclic CD (Behdin al., 2023; Chee et al., 2024;
The optimization problem for layer-wise output-based non- uniform scalar quantization can then be written follows:
doat
minimize po' ER
Xw, - xp())_0013
subject to
p
1dm
(8)
6

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Egiazarian et al., 2024). Recent works show that both greedy CD (Nair& & Suggala, 2024) and eyclic CD (Behdin et al., 2023) outperform GPTQ on this problem when using a uni- form grid. We thus adopt the cyclic CD algorithm, since it performs similarly to the greedy variant while being signifi- cantly less expensive (Nair& & Suggala, 2024, Appendix D). In Appendix E.6, we present an ablation study that further support this choice, showing that cyclic CD matches or out- performs GPTQ when used within LNQ for non-uniform scalar quantization.
Table 2. End-to-end inference throughput of I. Llama-2 models on RTX 4090 GPU. OOM indicates an Out Out-of-Memory error, meaning the GPU lacks memory to run model inference. See Appendix D. for experimental setup details.
Llama-2-99
2-BB
Llama-2-15
Llama-2-700
Type Original
Bits
Tok/st
Bits.
Tok/st
Bits.
T
16
67
16
00M
16
0OM
Uniform scalar
2,00
334
2.00
200
2.00
47
Non-uniform scalar
2.01
347
2.01
203
2.01
47
Vector
2.00
200
2.00
121
2,00
38
Cyclic CD is an iterative algorithm which iterates over coor- dinates in a fixed order, minimizing at each iteration the ob- jective with respect to one coordinate, while keeping all oth- ers fixed. The minimization for each coordinate (Line 8 in Algorithm 2) has a closed form solution, as shown in Behdin et al. (2023, Lemma 1) and Chee et al. (2024, Section B.2):
Uniform scalar
3.00
260
3.00
1.50
3.00
0OM
Non-uniform scalar
3.03
264
3.02
148
3.01
OOM
Vector
3.00
176
3,00
103
3.00
0OM
Uniform scalar
4.00
214
4.00
121
4.00
00M
Mon-uniform scalar
4.05
209
4.04
116
4.03
0OM
Vector
4.00
151
4.00
89
4.00
00M
H. [dwll (W (Wallees - Wit.)ies))
Round, WaJ
book (Line and 14) requires O(d m), and optimiz- ing the codes (Lines 6-12) requires ( in K) time com- plexity. The total time complexity of LNQ algorithm is O + di dout + K)). Here, T and K denotes the number of iterations for alternating optimization and the number of cycles in coordinate descent, respectively. We provide a detailed analysis of the time complexity in Ap- pendix C.2. We discuss in Appendix C.3 how to signifi- cantly speedup the implementation of CD on GPU, using precomputation and lazy batch-updates. Precomputation also used in Behdin et al. (2023); Chee et al. (2024), while lazy batch-updates is only used in Chee et al. (2024) (though not discussed in the paper). These tricks do no change the theoretical time complexity of CD, but they yield up to 3x speedups in practice.
Hi,
(11) where Round;()) denotes rounding to the nearest point in the grid (c())..... ((()......)))
())
CD is a descent method when initialized with a feasible solution W, E (ci () din, i.e., it monotonically decreases the objective function value. It can be used as standalone solver for problem (10) initialized with the original weights W. as in Nair & Suggala (2024); Behdin et al. (2023), or to refine the output of another quantization method, as done in the uniform quantization method QulP, which runs CD after LDLQ (Chee et al., 2024).
In LNQ, at each iteration, we initialize CD with the quan- tized weights corresponding to the current assignment and codebook w, c() for each j. For the first iter- ation, any feasible assignment matrix can be used. In our experiments, we initialize with the assignments from SqueezeL.LM. Since the codebooks are updated optimally and CD acts as descent method with feasible initialization, it follows that LNQ itself is a descent method and it converges. Refer to Appendix B for the proof.
5. Experiments
In this section, we demonstrate the versatility and effective- ness of our method across various quantization schemes. We first explore different quantization scenarios and identify the formats best suited to each setting, ultimately focusing on three main approaches: weight-only scalar, weight-only vec- tor, and weight-and and-ivivviioooo quantization. By integrating the GuidedQuant objective into existing methods, our re- sults consistently achieve state-of-the-art PTQ performance. Refer to Appendix D.2 for details on how we incorporate GuidedQuant objective into existing methods. Additional experiments and details, including the overall cost of our method, the effect of the number of groups g, and the end- to-end fine-tuning results, are provided in Appendix E.
Proposition 4.1. For any j E [dou], let f,(c,P) [Xw,- -XPel2, and let (.)) and pf") denote cli) and pli) at the t-th iteration of LNQ. Then, fz(cL") (P()) z ((C2)..(0) 2
(1) 11ch for all t, and the se- quence (f _P()))21 converges.
Since LNQ is layer-wise output based method, Guid- edQuant can be easily applied to it. In Section 5.1, we demonstrate the efficacy of LNQ both as a standalone ap- proach and in combination with the GuidedQuant objective.
5.1. Weight-only Quantization
Experimental Setup. Weight-only quantization primar- ily accelerates inference latency in low-batch scenarios, where memory bandwidth constitutes the main bottleneck
Time Complexity. Computing the Cholesky decomposi- tion of H (Line 1) requires OgL)), optimizing the code-
7

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table Weight-only scalar post-training quantization results without fine-tuning with end- ond-to-end loss. Wiki2 and C4 denotes perplexity on WikiText2 and C4, respectively. The perplexity is measured with the context size of 4096.
Llama-2-77
Llama-2-138
Llama-2-70B
Method
Bits
Wiki2
C44
Bits!
Wiki2
C4
Bits.
Wiki2,
C4
Original
16
5.12
6.63
4.57
6.05
3.12
4.97
QulP
2.00
13,48
16.16
2.01
5.90
8.17
SqueezeLLM
2.01
39.58
44.05
16.24
19.20
2.01
9.17
13.03
GPTVQ 1D
2.03
51.87
47.33
2.03
9.53
12.62
2.03
6.03
8.44
LNQ (Ours)
2.01
23.31
26.71
2.01
8.78
11.80
2.01
5.23
7.31
LNQ + GuidedQuant (Ours)
2.01
8.83
11.15
2.01
7.26
9.17
2.01
5.04
7.04
GPTQ
3.00
8.06
10.61
3.00
5.85
7.86
3.00
4,40
6.26
QulP
3.00
5.12
6.79
3.01
3.87
5.67
SqueezeLLM
3.03
5.74
7.44
3.02
4.99
6.60
3.01
3.53
5.31
GPTVQ 1D
3.03
6.17
8.02
3.03
5.13
6.76
3.03
3.55
5.35
LNQ (Ours)
3,03
5,89
7.74
3.02
5,02
6,68
3,01
3,50
5.31
LNQ GuidedQuant (Ours)
3.03
5,57
7.22
3.02
4.91
6.49
3.01
3.47
5.27
GPTQ
4,00
5.49
7.20
4.00
4.78
6.34
4,00
3.35
5.15
QulP
4.00
4.76
6,29
4.00
3.58
5.38
LQ
4.05
5,23
6.78
4.04
4.67
6.15
4.03
3,20
5.04
GPTVQ ID
4.06
5.27
6.83
4.06
4.67
6.17
4.03
3.20
5.04
INQ (Ours)
4.05
5.26
6.82
4.04
4.67
6.17
4.03
3,20
LNQ GuidedQuant (Ours)
4.05
5.21
6.75
4.04
4.65
6.14
4.03
3,20
5.03
(Gholami al., 2024). Among weight-only techniques, three quantization formats are commonly used: uniform scalar, non-umiform scalar, and vector quantization (Fran- tar et al., 2023; Kim et al., 2024; Tseng et al., 2024b). With fixed bit-width constraints, non-uniform scalar quantiza- tion generally outperforms uniform scalar quantization, as its search space encompasses that of uniform scalar quan- tization. Meanwhile, vector quantization can outperform non-uniform scalar quantization by exploiting additional redundancies across weight dimensions.
following prior work (Egiazarian et al., 2024; Tseng et al., 2024a;b), with 1024 sentences, each containing 4096 tokens. We report perplexity on the WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020) validation sets.
Scalar Post-training Quantization Results. We summa- rize the results of weight-only scalar quantization in Table 3, comparing our approach with GPTQ (Frantar et al., 2023), SqueezeLLMM without mixed precision (Kim al., 2024), QulP (Chee et al., 2024), and GPTVQ ID (van Baalen et al., 2024). For GPTQ and QulP, we report the results from Egiazarian et al. (2024), which used the same or a larger calibration dataset, while for GPTVQ ID, we reproduce the results with the same calibration data while adjusting the group size to align with the average bit-width for fair comparison (see Appendix C.4 for details).
Despite this, non-uniform scalar quantization offers advan- tages inference latency. Table 2 compares end-to-end single-batch inference latency across these formats using the state-of-the-an GPU kernels: LUT-GEMM (Park al., 2024a) for uniform scalar, Any-Precision-LLLM (Park al., 2024b) for non-uniform scalar, and QTIP (Tseng et al., 2024b) for vector quantization. Results show that vec- tor quantization incurs higher latency due to its decoding overhead (Tseng et al., 2024b), whereas uniform and non- uniform scalar quantization have similar latency with mini- mal decoding overhead. Consequently, non-uniform scalar and vector quantization remain the primary formats of inter- est for weight-only quantization. In this context, we apply our GuidedQuant to both formats, achieving state-of-th the-t-rrrtt performance in each.
We evaluate the performance of LNQ both with and without the GuidedQuant objective. Notably, LNQ combined with GuidedQuant consistently outperforms all baselines across various bit-widths and model sizes. Additionally, LNQ with the layer-wiss wise reconstruction objective surpasses GPTVQ ID in all settings, demonstrating that our approach improves upon GPTVQ ID by addressing its suboptimal optimization.
Vector Post-training Quantization Results. For vector post-training quantization (PTQ), we present the results in Table 4. We apply GuidedQuant to the state-of-the-art vector PTQ baseline, QTIP (Tseng et al., 2024b). We implement it on both the 1MAD and 3INST variants and report the variant that performs better among these two. Refer to
For our experiments, we demonstrate the effectiveness of our method on the Llama-2 model family (Touvron al., 2023), evaluating on 7B, 13B and 70B model. We use the RedPajama dataset (Computer, 2023) for calibration,
8

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table Weight-only vector post-training quantization results without fine-tuning to end-to-end loss. Wiki2 and C4 denotes perplexity on WikiText2 and C4, respectively. The perplexity is measured with the context size of 4096.
-lama-2-7B
Llama-2-13B
Llama-2-70BB
Method
Bits.
Wiki2j
C41
Bits
Wiki24
C4
Bits!
Wiki2,
C41
Original
16
5.12
6.63
4.57
6.05
16
3.12
4.97
GPTVQ
2.13
10.66
12.81
2.13
7.55
9.82
2.13
5,06
7.09
GPTVQ
2.25
7.89
10.25
2.25
6,36
8,43
2.25
4.44
6.28
QuIP#
2.00
8.22
11.01
2.00
6.06
8.07
2.00
4.16
6.01
AQLM
2.02
6.59
8.54
2.19
5.37
7.16
2.07
3.94
5.72
QПP
2,00
6.82
8.96
2.00
5.52
7.39
2.00
3.87
5.69
QTIP + GuidedQuant (Ours)
2.00
6.11
7.99
2,00
5.33
7.05
2.00
3.80
5.61
GPTVQ 2D
3.13
5.63
7.32
3.13
4.87
6.45
3.13
3.38
5.18
QulP#
3,00
5.60
7.34
3.00
4,90
6,50
3,00
3.41
5,20
AQLM
3.04
5.46
7.08
3.03
4.82
6.37
3.01
3.36
5.17
QТPР
3.00
5.38
6.99
3.00
4.74
6,28
3.00
3.27
5.09
QTIP GuidedQuant (Ours)
3.00
5.28
6.87
3.00
4.71
6.22
3,00
3.25
5.08
GPTVQ 2D
4.13
5.24
6.77
4.13
4.65
6.13
4.13
3.18
5.01
QulP#
4.00
5.22
6.79
4.00
4.65
6.15
4.00
3.18
5.02
AQLM
4.04
5.21
6.75
3.94
4.65
6.14
4.14
3.19
5,03
QТIР
4.00
5.17
6.71
4.00
4.62
6.10
4.00
3.16
5.00
QTIP + GuidedQuant (Ours)
4.00
5.16
6.68
4.00
4.61
6.09
4.00
3.15
5.00
Appendix E.10 for results on different variants. We compare approach with the following baselines: GPTVQ (van Baalen et al., 2024), QulP# (Tseng et al., 2024a), AQLM (Egiazarian et al., 2024), and QTIP (Tseng et al., 2024b). For QuIPH, AQLM, and QTIP, we report the results from their respective papers, as they used the same or larger calibration datasets than ours. For GPTVQ. we report the reproduced results using our calibration data. Our method consistently outperforms all vector quantization baselines across different bit-widths and model sizes as well.
Table 5. Weight-and-aactivttoon quantization results Llama-2
models. I-2-7B, T.-2-13B and 1-2-70B denote Tlama-2-7B, Llama-2-13B, and Llama-2-70B model, respectively. Wiki2 de- notes perplexity Wikitext2 with the context size of 2048.
L-2-7B
L-2-13B
L-2-70B
Bits
Method
Wiki2|
Wiki2,
Wiki2|
Original
5.47
4.88
3.32
QuaRot
6.08
5.39
3.80
W4A4KV4
SpinQuant
5.95
5.24
3.71
SpinQuant GQuant (Ours)
5,89
5.19
3.71
QuaRat
6.02
5.34
3.77
5.2. saahht-andd-ctivvatinn Quantization
W4A4KV16
SpinQuant
5.90
5.20
3.68
SpinQuant GQuant (Ours)
5.84
5.17
3.68
Weight-and-an and-activation quantization methods apply uniform quantization on both weights and activations to leverage the faster matrix multiplication units in the hardware (A (Ashkboos et al., 2024; Liu et al., 2024). State-of-the-art methods for secht--and-cttiattinn quantization include QuaRot (Ashk- boos al., 2024) and SpinQuant (Liu et al., 2024), which rotation matrices reduce the activation outliers be- fore applying the uniform quantization. We incorporate our GuidedQuant objective into the weight quantization process of these methods, guiding the model to quantize the weights more accurately. Specifically, we implement GuidedQuant on top of the SpinQuant using GPTQ weight quantizer and present the results in Table Following prior work, we use the Wexi2ext2 dataset (Merity et al., 2016) for calibration, with 128 sentences, each containing 2048 tokens (Ashkboos et al., 2024; Liu et al., 2024). Our objective consistently improves the perplexity compared to the baseline methods,
demonstrating its effectiveness.
6. Conclusion
We introduced GuidedQuant, novel PTQ approach that integrates gradient information from the end loss while preserving cross dependencies within output chan- nels. GuidedQuant improves state-of-the-art methods across quantization formats, including weight-only scalar, weight- only vector, and weight-and and-t--addattiitioonn quantization. Fur- thermore, identified inefficiencies in the current state- of-the-art methods for non-uniform scalar quantization and proposed LNQ, new algorithm that, when combined with GuidedQuant, improves over the state-of-thh-ar perfor- mance. These contributions advance the efficiency and accuracy of quantization for modern L.LMs.

---

GuidedQuant:
Large Language Model Quantization via Exploiting End Loss Guidance
Impact Statement
of linear equations. Journal of Computer and System Sciences, 54(2)317-331, 1997.
This work advances the compression of LLMs, in particular via post-training quantization. As discussed, quantization, and model compression more broadly, reduces the memory and computational requirements of LLMs and speeds up inference, thus reducing their environmental impact and enabling their use on resource-constraing devices and for latency-critical applications. This can also help democratize access to these models for organizations with limited re- sources and support pweeacc-rreervvigg... offline applications. On the other hand, compression methods, including quan- tization, adversely affect fairness in language models (Ramesh al., 2023). While there ongoing efforts ad- dress faimess concerns in pruned LLMs (Zayed et al., 2024), extending these mitigation strategies to quantized models remains an important direction for future research. Fur- thermore, reducing the cost of using LLMs can also lower the barrier to their use by malicious actors. Finally, the energy and resources saved through compression might be reinvested elsewhere, so the net reduction in environmental harm is not guaranteed (Jevons paradox (Alcott, 2005)).
Arthur, D. and Vassilvitskii, S. k-means++: the advantages of careful seeding. In SODA, 2007.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi. M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in ro- tated 1lms. In NeurlPS, 2024.
Bai, R., Liu, Q., and Liu, B. Skim: Any-bit quantization pushing the limits of post-training quantization. arXiv preprint arXiv:2412.04/
2024.
Behdin, K., Acharya, A., Gupta, A., Song, Q. Zhu, S.,
Keerthi, S., and Mazumder, R. Quantease: Optimization- based quantization for language models. arXiv preprint arXiv: 23309.0188,, 2023.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense natural language. In AAAI, 2020.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. In NeurlPS, 2024.
Acknowledgements
This work was supported by Samsung Electronics Co., Ltd. (0250441-122699-)) Mobile eXperience (MX) Business, Samsung Electronics Co., Ltd., Institute of Information & Communications Technology Planning & Evaluation (UTPP) grant funded by the Korea government (MSIT) [No. RS- 2020-11200882, (SW STAR LAB) Development of deploy- able learning intelligence via self-sustainable and trustwor- thy machine learning, No. RS-2021-1121 1343, Artificial Intelligence Graduate School Program (Seoul National Uni- versity), and No. 2022-0-00480, DS-2022-11220480 De- velopment of Training and Inference Methods for Goal- Oriented Artificial Intelligence Agents], and Basic Science Research Program through the National Research Founda- tion of Korea (NRF) funded by the Ministry of Education (RS-2023-00274288), Hyun Oh Song is the corresponding author.
Choi, Y., El-Khamy, M., and Lee, Towards the limit of network quantization. In ICLR, 2017.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv::803..0544 2018.
Computer, T. Redpajama: an open dataset for training large language models, 2023. URL t/ghtbb.. RedPajama -Data.
Dettmers, T. and Zettlemoyer, L.. The case for 4-bit preci- sion: k-bit inference scaling laws. In ICML, 2023.
References
Alcott, B. Jevons' paradox. Ecological economics, 54(1): 9-21, 2005.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. In ICLR, 2021.
Ansel, J., Yang, E., He, H., Gimelshein, N., Jain, A., Voz- nesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In ASPLOS, pp.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. In NeurIPS, 2023.
2024.
Ding, X., Zhou, X., Guo, Y., Han, J., Liu, J., et al. Global sparse momentum sgd for pruning very deep neural net- works, In NewrIPS, 2019.
Arora, S., Babai, L., Stern, J., and Sweedyk, Z. The hardness of approximate optima lattices, codes, and systems
10

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In ICML, 2024.
Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., and Alistarh, D. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In EMNLP, 2022.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate cest-training quantization and pruning, In NeurIPS, 2022.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. In NeurlPS, 1989.
Li, Y., Gong, R., Tan, X.,, Yang. Y., Hu, P., Zhang, Q., Yu, F, Wang, W., and Gu, S. Breeq: Pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021.
Frantar, E., Kurtic, E., and Alistarh, D. M-fac: Efficient matrix-free approximations of second-order information. In NewrIPS, 2021.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Optq: Accurate post-training quantization for generative pre- trained transformers. In ICIR, 2023.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https::////....... /znoddo. org/ records/ 12 12608602.
Lloyd, S. Least squares quantization in pem. IEEE transac- tions on information theory, 129-)77 1982.
Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv- tuning: Beyond straight-through estimation for extreme Ilm compression. In NeurlPS, 2024a.
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., and Keutzer, K. Ai and memory wall, IEEE Micro, 2024.
Malinovskii, V., Panferov, A., Ilin, I., Guo, H., Richtárik, P., and Alistarh, D. Pushing the limits of large lan- guage model quantization via the linearity theorem. arXiv preprint arXiv:2411.1755 2024b.
Gray, A. Getting started with cuda graphs, 2019.
URL wwthhhtps//// htittp:://ddeeloepet www.:eeelee....iiiiaaaa......eaa avidia. htt.bll...oooooo./////o./// cuda-graphs/.
Martens, and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In ICML, 2015.
Grønlund, A., Larsen, K. G., Mathiasen, A., Nielsen, J. S., Schneider, S., and Song, M. Fast exact k-means, k- medians and bregman divergence clustering in 1d. arXiv preprint arXiv: 002204, 2017.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint 1609.07444778 2016.
Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. In NeurIPS, 1992.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. arxiv e-prints. arXiv preprint arXiv:1512.03385, 10:9, 2015.
Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J. Pruning convolutional neural networks for resource efficient inference. In ICLR, 2017.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In ICLR, 2021.
Molchanov, P., Mallya, A., Tyree, S., Frosio. I., and Kautz, J. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11264-11272, 2019.
Hyun, J. Log-time k-means clustering for 1d data: Novel ap- proaches with proof and implementation. arXiv preprint arXiv:2412.15295, 2024.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post- training quantization. ICML, pp. 7197-7206. PMLR, 2020.
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M., and Keutzer, K. Squeezellm: Dense- and-sparse quantization. ICMI., 2024.

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Nair, P. A. and Suggala, A. S. Cdquant: Greedy coordinate descent for accurate IIm quantization. arXiv preprint arXiv: 2406.17542, 2024.
Baalen, M., Kuzmin, A., Nagel, M., Couperus, P., Bas- toul, C., Mahurin, E., Blankevoort, TT, and Whatmough, P. Gptvq: The blessing of dimensionality for Ilm quanti- zation. arXiv preprint arXiv:2402.15314 2024.
Park, G., Park, B., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. In ICMI,, 2024a.
Wang. C., Grosse, R., Fidler, S., and Zhang, G. Eigen- damage: Structured pruning in the kronecker-factored eigenbasis. In ICML, 2019.
You, Z.., Yan, K., Ye, J., Ma, M., and Wang, Gate deco- rator: Global filter pruning method for accelerating deep convolutional neural networks. In NeurIPS, 2019.
Park, Y. Hyun, J., Cho, S., Sim, B., and Lee, J. W. Any-precision Ilm: Low-cost deployment of multiple, different-sized llms. In ICML, 2024b.
Zayed, A., Mordido, G., Shabanian, S., Baldini, I., and
Chandar, S. Fairness-aware structured pruning trans- formers. In AAAI, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? ACL, 2019.
Ramesh, K., Chavan, A., Pandit, S., and Sitaram, S. A comparative study on the impact of model compression techniques fairness in language models. In ACL. pp. 15762-15782, 2023.
Zeng, W. and Urtasun, R. MIprune: Multi-laye pruning for automated neural network compression. OpenReview, 2018.
Sakaguchi, K., Bras, R. I.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv: 1907. 10641, 2019.
Sap, M.. Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interac- tions. In EMNLP. 2019.
Singh, S. P. and Alistarh, D. Woodfisher: Efficient second- order approximation for neural network compression. In NeurlPS, 2020.
Theis, L., Korshunova, I., Tejani, A., and Huszár, F. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arviv: 1rXX:v:801005781 2018.
Touvron, H., Lavril, T., Izacard, G.. Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.1399 13977 2023.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quip#: Even better 11m quantization with hadamard incoherence and lattice codebooks. In ICML, 2024a.
Tseng, A., Sun, Q., Hou, D., and De Sa, C. Qtip: Quan- tization with trellises and incoherence processing. NeurlPS, 2024b.
Tycho F. A. van der Ouderaa, Markus Nagel, M. B.B. B. The Ilm surgeon, In ICLR, 2024.
12

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
A. Additional Related Work
There's large body of work on neural network compression, even when considering only quantization for LLMs, making a complete overview infeasible. Instead, we focus here on the works most related to ours.
Hessian-based Compression Neural networks compression based on the second-order Taylor approximation of the end loss (Eq (2)) dates back to the early works of LeCun et al. (1989) and Hassibi & Stork (1992). OBD (LeCun et al., 1989) introduced this approach for pruning, under the assumption that the Hessian matrix is diagonal. OBS (Hassibi & Stork, 1992) improved upon this by dropping the diagonal assumption and instead approximating the Hessian by the empirical Fisher information matrix. However, applying OBS to large neural networks remains computationally intractable. To address this, various more efficient Hessian approximations have been proposed, including the K-FAC approximation (Martens & Grosse, 2015; Zeng & Urtasun, 2018; Wang et al., 2019; Tycho F. A. van der Ouderaa, 2024), block-diagonal Fisher approximation (Singh & Alistarh, 2020; Kurtic et al., 2022; Li et al., 2021), and diagonal Fisher approximation (Choi et al., 2017; Theis et al., 2018; Kim et al., 2024; Bai et al., 2024). Other strategies directly estimate inverse- vector products (Frantar et al., 2021). The most similar approaches to GuidedQuant are ones that employ Block-diagone Fisher approximation, which achieve a good trade-off between approximation accuracy and computation and storage cost. However, these methods remain intractable at the scale of modern LLMs (see Appendix E.11).
Gradient- Compression Various compression methods are based on a first-order Taylor approximation of the end loss, with respect to output feature maps or gates applied to them (Molchanov et al., 2017; 2019; You et al., 2019), or weights (Ding et al., 2019). The one most similar to GuidedQuant is (Molchanov et al., 2019), which employs the same criterion in Eq. (4) to prune filters and neurons in vision models. However, as explained in Section 3.2, adopting this criterion for quantizing modern LLMs is infeasible, without the averaging approximation we propose.
Non-uniform Scalar PTQ for LLMs PTQ encompasses a vast array of work, so we focus on non-uniform scalar PTQ methods for LLMs that use look-up tables (codebooks) for weight decoding, which are closely related to our LNQ algorithm. One approach is zero-shot quantization, which requires no calibration data: Dynamic Tree Quantization (Dettmers al., 2021) defines new data type with dynamic exponential bits and stores decoded values in the codebook; Quantile Quantization (Dettmers & Zettlemoyer, 2023) saves quantile values of the weight distribution; and QLoRA (Dettmers et al., 2023) introduces the NF4 data type using quantiles of standard normal distribution. These methods share a global codebook, with each layer maintaining its own scale parameters. HIGGS (Malinovskii et al., 2024b) further refines this by adopting MSE-optimal grids for the standard normal distribution and applying rotation matrices to approximate Gaussian weight distributions. Another line of work involves one-shot quantization methods that optimize the output quantization error using calibration data, For instance, Squeezel.LM (Kim al., 2024) optimizes separate channel-wist codebooks via the k-means algorithm, while ID variant of GPTVQ (van Baalen et al., 2024) alternates between optimizing assignments with the GPTQ algorithm and refining codebooks with gradient descent. The GPTVQ ID shows the strongest performance among this line of research. Although not a scalar PTQ method, the vector quantization variant of AQLM (Fgiazarian et al., 2024) also follows a similar paradigm, optimizing assignments through CD and codebooks via gradient descent.
B. Proofs
Here, we prove Remark 3.1 and Proposition 4.1, each restated here for convenience. Remark 3.1. The sum of the layer-wise objective in Eq. (4) over all layers is equal to the following quadratic approximation of the change in the end loss
7
w
(1)
F
(w
(()))
(5)
1=1
Proof. Recall that z = x(WW! Then, by chain rule we have that
06, ow?) az
0e;
[X]
(1)
Note also that
ae
at,
az
az
13

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Hence,
de (Z @ (x -XOWO-X" x
DE,
(1) bl;
((1) [W]
(0!!)) )
2z (1) :.
== i=1
(w)) x
Öw (!)
j= i=1
w (T (w (w/)) w (0))
n
where the last equality follows from the definition of the Fisher blocks F > over E [L] on both sides yields the claim.
Hc
]{
8C )
)T
Taking the sum
ōw! Ow!
Proposition 4.1. For any j E [oout]. let f, P) = Xw, XPel and let e and pnd denote ciu) and P at the t-th
iteration of LNQ. Then, f, f)(c()) p( 2 f,(c 1 ()) 2 f) converges.
()) P for all t, and the sequence (f,(c() p Mez
Proof. We first show that the objective value is non-increasing in LNQ. For all 2 1, we have PL7) 1m 1d,. and thus the corresponding quantized weights vi =P c, are feasible Hence, CD initialized with feasible solution each iteration I, so acts as a descent method. Then.
fg(cl) 10) (
(())
2 P()
(since c++1
( argmin P,)) P.))
e
2 55(021 2102
(since CD does not increase the objective value)
for all t 1. Since ff P) bounded below by 0, the sequence (f monotonically non-increasing and bounded below. Hence, converges to its infimum by the monotone convergence theorem.
C. Hyperparameters and Details
In this section, we clarify the hyperparameters and details of the methods discussed in the main paper.
C.1. GuidedQuant
The proposed GuidedQuant method has single hyperparameter: the number of group used to average the Hessian matrices H,; (see Section 3.2). For weight-only quantization experiments, we set g = for Llama-2-7B and Llama-2-13B, and = for Llama-2-70B. For weight-and bsight-and-acttinn and-activation quantization experiments, we set g = 1. For the hyperparameter g. we selected the number of groups to be as large as possible within the limits of our computational and memory constraints. Notably, GuidedQuant also maintains strong performance with smaller values of g (see Appendix E.5).
Computing the Hessian (Line 4 in Algorithm 1) and rumning the quantization algorithm Q (Line 5 in Algorithm 1) for each group and layer can be parallelized. We parallelize Hessian computation across groups. For quantization, we parallelize across groups in LNQ + GuidedQuant, while in QTIP + GuidedQuant and SpinQuant + GuidedQuant, we run this step in a sequential manner to minimally change the codebase of the original methods.
C.2. LNQ
The proposed LNQ method has two hyperparameters: (1) the number of iterations during which we alternate between optimizing e and P (T in Algorithm 2), and (2) the number of coordinate descent iterations over the output dimensions (K in Algorithm 2). For Llama-2-7B and 1lama-2-13B. we use = 2 and = 4, and for Llama-2 2-70B, we use T = 1 and K in all the experiments.
14

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Algorithm 3 Efficient CD algorithm with precomputation
input Hessian of the objective H R4 input weight W Raux current codebook c(u) I and current quantized weight W Rd Xdous Initialize Q E {1,.... my du x xdous (rounded indices).
1: H + diag H) 'H, U + StrictUpper(H)
2: for k = K do
3:
B U(W W)
4:
for to dm do
5:
W. < RounddW. - B,.), Qi f- RoundIdx (.. B..))
6: B(++):: + B(+1): + U(+1) W.,: W..)
end for 8: end for
9: Vi E j [dout) [m]: p(() -
(Extracting assignment matrix)
otherwise.
output P(t) ..... ((ooa),
We further explain a derivation of the time complexity of the proposed LNQ algorithm (Algorithm 2), discussed in Section 4.2,
First, the time complexity of the Cholesky decomposition for matrix H Rd x is O(d (Line 1).
For optimizing the codebook (Line 4 and 14), we analyze the computational cost within the loop follows:
Computing L PP)) requires O(P,,m) time.
Computing L w, requires O(d),) time.
torch. 1inalg lstsq function uses QR decomposition of L TP6) to compute least squares solution, which requires Odd,, time.
Since din >> m, the dominant cost is O m).
For computing w p() (Line 5), the cost is Od)mm).
CD, the cost of the minimizing the objective for each coordinate i (Line 8, Eq. (11)) is (dn, + m)) Since du >> m, the dominant cost (ddn). Considering loop iterations, optimizing the code (Lines 6-12) takes (dd, K) time complexity.
Therefore, the cost of Lines 4-12 is (dl (m K)). and the cost of Lines 2-15 is Odl dont T(m K)). Including the Cholesky decomposition, the total time complexity of LNQ algorithm is Odd, + di T(m K)).
C.3. Efficient Implementation of CD Algorithm in LNQ
In the LNQ algorithm (Algorithm 2), computing the solution across all output channels j E [dout] is independent and thus fully parallelizable. Therefore, we perform the coordinate descent (CD) updates for each output channel in parallel.
Ra x douss the CD update for the i-th input coordinate can be computed in parallel using the coordinate-wist closed-form solution as follows (Behdin et al., 2023, Lemma l):
Coordinate-wise Closed-form Solution. For given quantized weight matrix W E
Hs,dd.l H
W Wid..ie)))
W <- Round W
[din]Va.
(12)
where Round(.) 1x R 1xdout rounds j-th element to the nearest point in the grid (ct)) ...., R.)) We adopt this coordinate-wise closed-form solution within the CD loop.
Precomputation Trick. On GPUs, the coordinate-wise CD update in Eq. (12) can be accelerated by precomputing parts of the update that remain unchanged during previous coordinate updates. Specifically, when updating the i-th coordinate, the components of W corresponding to coordinates (i + 1) to dn remain fixed and can therefore be precomputed before
15

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Algorithm 4 Efficient CD algorithm with precomputation and lazy batch-updates
input Hessian of the objective H R input weight W R current codebook c(u) I and current quantized weight W Rd Xdous Initialize Q E (1,.... my du xdous (rounded indices).
1: H + diag H) 'H, U + StrictUpper(H)
2: for k
K do
3:
+ U(W -+U(WW-W W)
4: for = b+ 1, 26- ,,..., dm - b do
5:
for i = to
do
6:
W. t Round W. Bi.)) Qi + Roundldx W., - B;,)) ((++)))+6);; ((++))(+t)); + U ((+T))(+))) W.: W..))
8: end for
9: (a++)) + (o+b):: + (ott))m((tt)) W s:(o+)), W We((+b)):)
10: end for
11:
end for
1 Q
12: Ví E j E [dout], a E [m] pl)) =
[Etrractnng assignment matrix)
otherwise.
output ((I) (doot))
entering the CD loop:
H (.di.I
H. H11s Hi,
H ((+11 His
( Widdi/bb
- W|d [amn/sss
W
(ao1)) - W(++144))
His
Cannot be precomputed before the CD loop
Can be precomputed before the CD loop
To take advantage of this, we precompute the second term (which corresponds to future coordinates) for all E [dia] in parallel using matrix operations before entering the CD loop:
B <- StrictUpper H W),
where H is obtained by dividing each row of H by the corresponding diagonal entry Hi,,, and Striet (brir()pper()) extracts the strictly upper triangular part of the matrix.
During the CD loop, we use the precomputed B compute the noordinate-wist update in Eq. (12), and after updating the i-th coordinate, we incrementally update B to reflect the new value of W..:
W Round B:..)
B (1+1): ((++): + StrictUpper H ((41):
W 4.: W..).
The full CD algorithm incorporating this precomputation strategy is provided in Algorithm 3.
This acceleration trick has been proposed in QulP (Chee et al., 2024, Appendix B.2.) and QuantEase (Behdin et al., 2023).
is worth noting that this precomputation trick does not change the theoretical time complexity, but improves practical performance by exploiting the GPU parallelization. In particular, the CD update for the i-th coordinate Equation (12) requires 2dout - 1) FLOPs without precomputation, while with precomputation, the cost is reduced to 2dout (din - FLOPs.
Lazy Batch-updates. After incorporating the precomputation trick, we observe that the update steps within the CD loop (Lines 4-7 in Algorithm resemble the OBQ update scheme used in the GPTQ method (Frantar & Alistarh, 2022; Frantar et al., 2023). In OBQ, each iteration involves rounding single coordinate and adjusts the not-yet-rounded coordinates accordingly. Analogously, our CD update with precomputation rounds W.. B;. for the i-th coordinate and incrementally updates ((41), to reflect the new values W W
16

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 6. Hyperparameters that we used in reproducing GPTVQ (van Baalen et al., 2024) in Tahle and Table 4.
Weight bits
VQ
Codebook sharing
Scaling block size
Codebook bit-width
Table
dim
group size
Avg bits
2
1
1024
8
2.03
3
1
2048
8
3.03
Table
4
8192
256
8
4.03
4
1
4096
128
8
4.06
2
2
2048
8
2.13
2
4
32768
8
2.25
Table
3
2
16384
64
8
3.13
4
2
65536
64
8
4.13
Both OBQ and our CD update suffer from low compute-to-memory ratio: although each iteration involves relatively few FLOPs, it requires frequent reading and writing to large matrices, As result, these updates tend to be memory-boune and suffer from poor GPU utilization. To mitigate this, GPTQ introduces lazy batch-updates, in which batch of coordinates (with batch size = 128) is processed together. Within each batch, updates are applied scquentially to each coordinate, while corrections are made only for the remaining unprocessed coordinates within the batch. Once all b coordinates in the batch are updated, a global correction step is performed for the rest of the matrix. This strategy improves memory efficiency by reducing the frequency of global updates.
We adopt this lazy batch-updates approach in our CD implementation with precomputation trick. Specifically, we restrict updates to the relevant portion of B within each block of coordinates, and defer global updates to B until the entire block has been processed. This significantly reduces memory-bound operations and enhances GPU utilization. The final efficient CD algorithm incorporating both precomputation trick and lazy batch-updates is given Algorithm 4.
QulP (Chee et al., 2024) also supports lazy batch-updates their open-source code, though is not mentioned in their paper. QuantEase (Behdin et al., 2023) does not this approach in their implementation. As with the precomputation trick, lazy batch-updates do not change the theoretical time complexity. However, they substantially accelerate the overall algorithm in practice by better utilizing GPU resources.
Speedup Factor To demonstrate the speedup achieved by our optimization techniques for the CD algorithm, we report the quantization time for quantizing the Llama-2-7B model into 4-bit precision single RTX 6000 Ada GPU. Without any optimizations, adopting the naive strategy of exhaustively evaluating the objective function for all coordinate choices and selecting the option with the lowest value takes 3.9 hours to quantize the entire model. Applying the coordinate-wist closed-form solution described in Eq. (12) reduces this time to 2.7 hours. Incorporating the precomputation trick further lowers to 1.2 hours. Finally, applying lazy batch-updates brings the total quantization time down to just 0.9 hours. Overall, these optimizations yield more than a 4x speedup in end-to-end quantization time on GPU.
C.4. GPTVQ
In the original GPTVQ paper (van Baalen et al., 2024), the authors used 128 sentences from the WikiText2 dataset (Merity et al., 2016), each containing 2048 tokens, as a calibration data. For fair comparison, we reproduced their method using their open-sourced code but used 1024 sentences RedPajama dataset (Computer, 2023), each containing 4096 tokens. We adopted their default hyperparameters except for the group size and block size, which adjusted to match the average bit width when comparing with different methods in Table 3. We provide complete list of GPTVQ hyperparameters for each table in Table 6.
D. Details on Experimental Setup
This section provides detailed explanation of the experimental settings used.
17

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table End-to-end inference throughput Llama-2 models on RTX 4090 GPU, including the vector quantization kernel after fusing the query/key/value projection matrices into one linear layer and the up/gate projection matrices into another when measuring the througbput. OOM indicates an -emorry error, meaning the GPU lacks memory rum model inference.
Llama-24 2-BB
LJama-2- 13B
Llama-2-70B
Type
Bits./
Tok/st
Bits..
Tok/st
Bits
Tok/s
Original
16
67
16
OOM
16
OOM
Uniform scalar
2.00
334
2.00
200
2.00
47
Non-uniform scalar
2.01
347
2.01
203
2.01
47
Vector
2.00
200
2.00
121
2.00
38
Vector (fused)
2.00
248
2.00
153
2.00
42
Uniform scalar
3.00
260
3.00
1.50
3.00
OOM
Non-uniform scalar
3.03
264
3.02
148
3.01
OOM
Vector
3.00
176
3.00
103
3,00
OOM
Vector (fused)
3.00
209
3.00
123
3,00
OOM
Uniform scalar
4.00
214
4.00
121
4.00
OOM
Non-unifond uniform scalar
4.05
209
4.04
116
4.03
OOM
Vector
4.00
151
4.00
89
4.00
OOM
Vector (fused)
4.00
176
4.00
103
4.00
OOM
D.1. End-to-end Inference Throughput Experiments (Table 2)
In Table 2, we measure each model's 's inference throughput in generating 100 tokens on RTX 4090 GPU, after integrating the kernels with into a PyTorch-based inference pipeline optimized with the torch. compi e function (Ansel et al., 2024; Gray, 2019). For QTIP, our chosen vector quantization kernel, we adopt the HYB variant of it as its GPU kernel is publicly available, though it is possible to implement fast GPU kernels with other variants as well (Tseng et al., 2024b).
For the base model and for models quantized using uniform or non-uniform scalar formats, we fuse the query/key/value projection matrices into linear layer and the up/gate projection matrices into another when measuring the throughput. This fusion trick can be applied to QTIP as well, provided the matrices are fused before quantization and the scale parameters are shared across layers. However, in the main paper, we present QTIP results without fusion to match the original experimental setup (and reported numbers) from their work, in which they quantize the layers independently without fusing them. Meanwhile, scalar quantization methods quantize the layer in an output channel-wise manner, and this allows fusing matrices even when layers are quantized separately.
For completeness, include Table 7, which also shows QTIP's fused end-to-end throughput (measured using dummy values) to illustrate the impact of fusion, restating the relevant results from Table 2. Although the fusion boosts the throughput, it does not change the conclusion that QTIP still runs more slowly than the scalar quantization methods.
D.2. Implementation Details for Different Quantization Types
GuidedQuant employs a quantization algorithm Q subroutine (Line in Algorithm 1). In this section, clarify which specific quantization algorithm Q each method uses, which GuidedQuant builds upon in Algorithm 1. We integrate GuidedQuant with three different quantization methods: (1) LNQ for weight-only scalar quantization, (2) QTIP for weight- only vector quantization, and (3) SpinQuant for beight-and-activatiin quantization. LNQ adopts the algorithm shown in Algorithm 2, QTIP uses the BlockLDLQ algorithm proposed in Tseng et al. (2024a), and SpinQuant employs the GPTQ algorithm introduced Frantar et al. (2023).
18

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 8. Total GPU cost incurred during the quantization process for LNQ and QTI,, both with and without GuidedQuant, across various group sizes g. We specity the number and type of GPU used in the parentheses. R6A denotes the RTX 6000 Ada GPU.
LNQ
QTIP
Model
Method
GPU Cost 2 bits
GPU Cost -3 bits
GPU Cost 4 bits
GPU Cost bits
GPU Cost 3 bits
GPU Cost 4 bits
Llama-2-7B
Layee-wise (LNQ. QTIP)
0.5 (IxRA)
0.6 (1 XR6A)
0.9 (1x66A)))
(13
(12
1.2 (1x
waser-wise + GQuant (g- 1)
h
0.6 (1 (1XR6A)
(19h(1X
1.3 (1 (1xR6A)
1.2 (1x
Layer-wise GQuant (g
0.6 (1xR6A)
0.7 (1X6A)
0,9 (1xR6A)
1.5 (1xR6A)
(Lx
(1xRR)
Layer-wise GQuant (g 4)
0.7 (IxxAA)
0.7 (I
0,9 (1X)AA)
1.9 (1x)
(
1.9 (1
Llama-2-13B
Layee-wise (LNQ. (TTP))
0.9 la (1xR6A)
1.1 b (IR6A)
1.6 (1XR6A)
3.0 (1XR6A)
(1x
2.7 h (1XR6A)
Layer-wise- GQuant (1- 1)
0.9 (1xR6A)
(I
1.6 (LXRGA)
3.0 (1X)6A)
(Ix
(1x
waser-wise GQuant (g-
1.1 (1xx6A)
(1xR()))))
1.6 (1XA6A)
2.4 (1x (1xR6A)
2.2 (1x
2.3 R6AR
Layer-wise GQuant (g 4)
1.2 b (Ix)R)
(1
1.7 (1 (GAA)
3.0 (6A)
3.0 (IxRR)
3.0 (1x R6A)
1lama-2-70B
1 Layer-wise (1 NQ. QTIP)
(Ix
3.3 (Ix
5.1 h (6A)
12.0
10.8 R6A))))
11.0 (1 (6A)
Layer-wise+ GQuamt (g 1)
2.6 (1xR6A)
(1
(1XR6A)
12.0
10.8 (1xR6A)
11.0 (1xR6A)
waser-wise+ GQuant (g-
(1x
6.8h (.x6()))))
(I.06()))
11.9h(1xR6N)
12.0 (1xR6A)
Table 9. Total GPU cost and disk usage incurred during the gradient and Hessian caching processes for cach objective- bbjictive-eeigheed k-means (M), layer-wise (LNQ. QTIP), and GuidedQuant. We specify the number and type of GPU used in the parentheses. R6A and A100 denote the RTX 6000 Ada GPU and the A100 GPU, respectively. The calibration data are 1024 sentences of the RedPajama dataset, each containing 4096 tokens.
Gradient Caching
Hessian Caching
Model
Method
GPU Cost
Disk Size
GPU Cost
Disk Size
1-ama-2-78
Weighted k-means (Squeezel.IMM)
0.3 (1xA100)
13 GiB
Layer-wise (LNQ, QTIP)
(4x R6A)
27 GiB
Layer-wise + GQuant (g = 1)
0.3 h (1xA100)
2 GiB
0,3 h (4x R6A)
27 GiB
Layer-wisc GQuant (g 2)
0.3 h (1X00)
4 GiB
0.4 h (4x
53 GiB
Layer-wise GQuant (g- 4)
0.3 h (1xA100)
7 GiB
0.8 R6x)6A)
106 GiB
Llama-2-13B
Weighted k-means (SqueczeLL.M)
0.6 h (2xA100)
25 GiB
Layer-wise (LNQ, QTIP)
0,5 h R6A)
52 GiB
Layer-wise GQuant (g 1)
0.6 (2xA100)
3 GiB
0.5 (4x R6A)
52 GiB
Layer-wise GQuant (g- 2)
0.6 (2xA100)
GiB
0.9 (4x RRAA
104 GiB
Layer-wise+ GQuant (g 4)
0.6 (2xA100)
13 GiB
1.5 h R66)
208 GiB
Llama-2-70B
Weighted k-means (SqueezeLLMM)
2.7 h (6xA100)
129 GiB
Layer-wise (LNQ, QTIP)
3.5 (4xR6A)
366 GiB
Layer-wise+ + GQuant (g = 1) Layer-wise++ GQuant (y = 2)
2.7 h (6xA100)
5 GiB
3.5 (4xR6A)
366 GiB
2.7 h (6xA100)
9 GiB
5.8 h (4xR6A)
731 GiB
E. Additional Results and Discussions
E.1. Quantization Cost
In this section, we present a detailed breakdown of the computational costs associated with our method, as summarized in Table 8 and Table 9. The layer-wise quantization methods on which we build typically require two phases: (1) caching the Hessian matrices to disk, and (2) loading them to quantize weights based on these cached Hessian matrices. It is worth noting that the cost of the first phase (caching) can be amortized if one needs to quantize the same model multiple times at different bit-widths or configurations, as the Hessian matrices can be reused. We report the weight quantization cost in Table 8, and the Hessian-caching cost in Table 9.
From Table 8, observe that for g = 1, the quantization is identical to standard layer-wise quantization, since the Hessian size the same. Even for g 2 or g 4. the quantization cost does not increase by more than 50%. This is because while more Hessian matrices are employed, each weight block to be quantized becomes correspondingly smaller, leaving the total computation unchanged. All these steps can be performed in an embarrassingly parallel manner; for example, quantizing Llama-2-70B using our LNQ algorithm takes less than three hours when using 8 RTX 6000 Ada GPUs.
We further report the cost of caching Hessian matrices in Table 9, along with the number of GPU used and the disk size requirements. Note that while we used GPUs for caching, this process is also fully parallelizable; using fewer GPUs will simply take longer (it can run on a single GPU), whereas additional GPUs can shorten the total time. Finally, our method's
19

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 10. Welght-only scalar post-training quantization results on IJama-3 models. Wiki2 and C4 denotes perplexity on WikiText2 and C4, respectively. The perplexity is measured with the context size of 8192.
Llama-3-8B
Llama-3-70B
Method
Bits
Wiki2
C44
Bits.
Wiki24
C44
Original
16
5.54
7.10
16
2.59
5.78
SqueezeLLM
2.01
16322
1501
2.01
38.53
38.15
LNQ (Ours)
2.01
133.00
72.75
2.01
24.22
19.71
LNQ + GuidedQuant (Ours)
2.01
30.80
20.41
2.01
10.21
11.06
SqueezeLLM
3.03
7.39
8.84
3.02
4.12
6.44
LNQ (Ours)
3.03
7.28
8.46
3.01
4.57
6.61
LNQ + GuidedQuant (Ours)
3.03
6.99
8.10
3.01
3.90
6.27
SqueezeLLM
4.05
5.91
7.43
4.03
2.91
5.91
LNQ (Ours)
4.05
5.90
7.40
4.03
3,05
5.94
LNQ + GuidedQuant (Ours)
4.05
5.80
7.32
4.03
2.89
5.89
Table 11. Weight-only scalar post-training quantization results on Llama-2 models, including end-to-end throughput. Wiki2 and C4 denotes perplexity on WikiText2 C4, respectively. The perplexity measured with the context size of 4096. Throughput is evaluated on an RTX 3090 GPU, reported as the average of 5 runs with standard deviation in parentheses. OOM indicates an Out-of cut-of-Memory erro, meaning the GPU lacks memory to run model inference.
Llama-2-7B
1lama-2-133
Llama-2-70B
Method
Bits|
Wiki2
C41
Tok/st
Bits.
Wiki2j
C41
Tok/st
Bits|
Wiki2|
C41
Tok/st
Original
16
5.12
6.63
64.8 (0.1)
16
4.57
6.05
OOM
16
3.12
4.97
OOM
Sq
2.01
39,58
44.05
245.1 (1.8)
2.01
16.24
19.20
140.5 (0.5)
2.01
9.17
13.03
31.5 (0.0)
LNQ (Ours)
2.01
23.31
26.71
244.6 (0.6)
2.01
8.78
11.80
141.1 (0.4)
2.01
5.23
7.31
31.6 (0.1)
LNQ GuidedQuant (Ours)
2.01
8.83
11.15
244.4 (2.9)
2.01
7.26
9.17
141.2 (0.5)
2.01
5.04
7.04
31.6 (0.1)
Sq
3.03
5.74
7.44
207.3
3.02
4.99
6.60
118.0 (0.5)
3.01
3.53
5.31
0OM
LNQ (Ours)
3.03
5.89
7.74
207.3 (2.1)
3.02
5.02
6.68
118.0 (0.6)
3.01
5.31
OOM
LNQ GuidedQuant (Ours)
3.03
5.57
7.22
207.6
3.02
4.91
6.49
117.9 (0.6)
3.01
3.47
5.27
OOM
SqueczeLLM
4.05
5.23
6.78
161.8
4.04
4.67
6.15
89.8 (0.1)
4.03
3.20
5.04
OOM
LNQ (Ours)
4.05
5.26
6.82
(1.6)
4.04
4.67
6.17
89.7 (0.1)
4.03
3.20
5.04
OOM
LNQ GuidedQuant (Ours)
4.05
5.21
6.75
(1.88
4.04
4.65
6.14
89.8 (0.1)
4.03
3.20
5.03
00M
disk-space requirement proportional to the number of groups g. However, we highlight that for constrained disk space, choosing smaller number of groups can still capture most of the performance benefits (Table 13).
E.2. Results on Llama-3 Models
In this section, we present the results of evaluating LNQ and LNQ combined with GuidedQuant on Llama-3-8B and Llama-3-70B models, comparing with SqueezeLLM under a weight-only scalar quantization setting, We present the results in Table 10. We use RedPajama dataset (Computer, 2023) for calibration with 1024 sentences, each containing 4096 tokens. We set the number of groups be g 1 for Llama-3-8B and Llama-3-70B, and set the hyperparameters for LNQ (and LNQ + GuidedQuant) to be 7 2, K = for Llama-3-8B and T = 1, K = 4 for Llama-3-708 model. LNQ with GuidedQuant consistently outperforms the baselines, demonstrating the robustness and effectiveness of our approach.
E.3. Additional Inference Throughput Results
GuidedQuant leverages existing CUDA kernels Anny-Prciiion-LLLMM kernel (Park et al., 2024b) for weight-only scalar and QTIP kernel (Tseng et al., 2024b) for weight-only vector quantization) and optimizes assignment and codebook values, thus achieving improved performance without sacrificing inference throughput. To validate this, we compare weight-only scalar PTQ results on Llama-2 2 models across methods using the same CUDA kernel, as shown in Table 11. Specifically, we report perplexity and end-to-end throughput for Squeezel..MM, LNQ, and LNQ + GuidedQuant, all using the Any-Precision Kernel
20

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 12. Weight-only scalar post-training quantization results, evaluated on zero-shot and few-shot downstream tasks. Zero-shot Avg denotes the average accuracy across eight zero-shot tasks: BoolQ, PIQA, SIQA. HellaSwag, WinoGrande, ARC-easy, ARC-challenge, and OBQA. For the few-shot benchmark, MMLU (5-shot) denotes accuracy on the MMLU benchmark in 5-shot setting. We report the standard error in parentheses and bold the best results, as well those whose accuracy score falls within the top score + standard error.
Jlama-2-7B
Llama-2-13B
Method
B
Zero-shot Avgt
MMLU (5-shot)
B
Zero-shot Avgt
MMLU (5-shot)
Original
16
59.88 (0.43)
45.97 (0.41)
16
62.80 (0.43)
54.93 (0.40)
SqueezeLLM
2.01
41.80 (0.41)
24.75 (0.36)
2.01
42.44 (0.41)
24.47 (0.36)
GPTVQ ID
2.03
37.35 (0.40)
26.56 (0.37)
2.03
46.34 (0.41)
29.63 (0.38)
LNQ (Ours)
2.01
40.30 (0.40)
26.76 (0.37)
2.01
49.51 (0.42)
32.51 (0.39)
INQ + GuidedQuant (Ours)
2.01
50.39 (0.43)
31.53 (0.39)
2.01
53.98 (0.43)
40.15 (0.41)
SqueezeLLMM
3.03
57.55 (0.43)
40.59 (0.41)
3.02
61.16 (0.43)
49.94 (0.40)
GPTVQ ID
3.03
54.92 (0.43)
41.08 (0.41)
3.03
60.38 (0.43)
52.06 (0.40)
LNQ (Ours)
3.03
56.85 (0.43)
42.18 (0.41)
3.02
60.61 (0.43)
51.62 (0.40)
LNQ + GuidedQuant (Ours)
3.03
58.16 (0.43)
43.38 (0.41)
3.02
61.00 (0.43)
52.67 (0.40)
SqueezeLLM
4.05
59.41 (0.43)
44.79 (0.41)
4.04
62.32 (0.43)
54.52 (0.40)
GPTVQ ID
4.06
59.23 (0.43)
45.06 (0.41)
4.06
62.37 (0.43)
54.95 (0.40)
LNQ (Ours)
4.05
59.14 (0.43)
44.51 (0.41)
4.04
62.40 (0.43)
54.79 (0.40)
LNQ + GuidedQuant (Ours)
4.05
59.41 (0.43)
45.16 (0.41)
4.04
62.17 (0.43)
54.39 (0.40)
(Park et al., 2024b). Throughput is measured on an RTX 3090 GPU as the average of 5 runs, with standard deviation in parentheses. Results confirm that our methods (LNQ and LNQ + GuidedQuant) achieve better perplexity while maintaining the same throughput as other method using the identical kernel.
E.4. Evaluations on Zero-shot and Few-shot Downstream Benchmarks
In this section, we provide the evaluations on zero-shot and few-shot downstream tasks of our methods (LNQ and LNQ + GuidedQuant) alongside baselines (SqueezeL.LM and GPTVQ 1D) under the weight-only scalar quantization settings, using Llama-2-7B and Llama-2-13B models, in Table 12. The evaluation includes eight zero-shot tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), ARC-easy (Clark et al., 2018), ARC-challenge (Clark et al., 2018), and OBQA (Mihaylov et al., 201 8). For a few-shot benchmark, we include results on the MMLU (Hendrycks et al., 2021) benchmark in 5-shot setting. We evaluate on these tasks using version 0.4.3 of the 1m uat woon-harness library (Gao et al., 2024).
Table 12 reports both accuracy and standard error for all methods. We highlight the best-performing results, as well as those whose accuracy falls within the top score standard error, under the same bit width constraint. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with notable improvements in extreme quantization scenarios, such as 2-bit quantization.
E.5. Results on Varying the Number of Groups g
In this section, we present results on how varying the number of groups g (introduced in Section 3.2) affects performance, focusing on whether fewer groups preserve accuracy introduce trade-offs when averaging the Hessian within each group. Table 13 summarizes the impact of changing g under non-uniform scalar quantization scheme. While increasing 9 can moderately improve results in extreme cases (e.g.. quantizing models into 2 bits), performance differences across the number of groups remain minimal in other scenarios. Note that for weight-only quantization experiments, we chose = 4 for Llama-2-1 2-7B and Llama-2-13B, and = 2 for Llama-2-70B, Still, smaller number of groups are sufficient for achieving most of the performance gains, making them a practical choice for resource-constraing scenarios.
E.6. A Alation Study on Assignments Optimization in LNQ
In this section, we evaluate our choice of using cyclic CD algorithm instead of GPTQ to solve Problem (8) for a fixed codebook cli) in LNQ. In particular, we compare two variants LNQ with the GuidedQuant objective: the variant described
21

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 13. Results with different number of groups g in weight-only post-training quantization results non-uniform scalar quantization format, without fine-tuming to the end-to and-to-end loss. Wiki2 and C4 denotes perplexity on WikiTex12 and C4, respectively, which are measured with the context size of 4096.
Llama-2-7B
Llama-2-13B
Llama-2-70B
Number of groups
Method
Bits!
Wiki2
C4
Bits
Wiki2,
C44
Bits.
Wiki2]
C4
Original
16
5.12
6.63
16
4.57
6.05
16
3.12
4.97
LNQ
2.01
23.31
26.71
2.01
8.78
11.80
2.01
5.23
7.31
LNQ + GuidedQuant
1
2.01
9.00
11.35
2.01
7.32
9.29
2.01
5.11
7.06
2
2.01
8.82
11.20
2.01
7.18
9.22
2.01
5.04
7.04
4
2.01
8.83
11.15
2.01
7.26
9.17
LNQ
3,03
5.89
7.74
3.02
5.02
6.68
3.01
3.50
5.31
LNQ + GuidedQuant
1
3.03
5.55
7.23
3.02
4.92
6.49
3.01
3.46
5.27
2
3.03
5.57
7.22
3.02
4.92
6.49
3.01
3,47
5.27
4
3.03
5.57
7.22
3.02
4.91
6.49
LNQ
4.05
5.26
6.82
4.04
4.67
6.17
4.03
3.20
5.04
LNQ + GuidedQuant
1 4.05
5.21
6.75
4.04
4.65
6.14
4.03
3.20
5.03
2
4.05
5.22
6.75
4.04
4.65
6.14
4.03
3.20
5.03
4
4.05
5.21
6.75
4.04
4.65
6.14
Table 14. Ablation study on optimizing discrete assignment in Problem (8). We compare two algorithms for optimizing discrete assignments; GPTQ and coordinate descent algorithm. Wiki2 and C4 denotes perplexity WikiText2 and C4, respectvvlly, which measured with the context size of 4096.
Llama-2-7B
Llama-2-13B
Llama-2-70B
Optimization method for P
Method
Bits
Wiki2
C41
Bits
Wiki2,
C41
Bits.,
Wiki2|
C44
Original
16
5.12
6.63
4.57
6.05
16
3.12
4.97
LNQ + GQuant
GPTQ Coordinate Descent
2.01
9.65
11.83
2.01
7.96
11.65
2.01
4.92
6.93
2.01
8.83
11.15
2.01
7.26
9.17
2.01
5.04
7.04
LNQ GQuant
GPTQ Coordinate Descent
3.03
5.58
7.25
3.02
4.91
6.50
3.01
3.47
5.27
3.03
5.57
7.22
3.02
4.91
6.49
3.01
3.47
5.27
LNQ GQuant
GPTQ
4.05
5.22
6.75
4.04
4.65
6.14
4.03
3.20
5.03
Coordinate Descent
4.05
5.21
6.75
4.04
4.65
6.14
4.03
3.20
5.03
in Section 4.2, which updates the assignments using cyclic CD, and alternative variant that uses GPTQ for assignments updates. Both variants update the codebook using the closed-form solution in (9). We report the results on Llama-2-7B model, evaluated on WikiText2 and C4 datasets, Table 14. Our experiments show that consistently outperforms or matches GPTQ, validating our choice of using CD optimize the assignment matrix P())
F.7. End-to-end Fine-tuning Results
Recent weight-only quantization methods have explored fine-tuning quantized models using extensive data and compute to improve performance for low-bit models (Tseng al., 2024a:b; Malinovskii al., 2024a). In Table 15, we summarize the performance of quantized models after further fine-tuning on end loss using more data and compute for scalar weight-only quantization. We implement PV-Tuning (Malinovskii et al., 2024a) in non-uniform scalar quantization setting and report the performance of both our model and SqueezeLL.M after fine-tuning with it. For Squeezel and LNQ GuidedQuant, we obtain the results using the official open-source implementation of PV-Tuning, Our fine-tuning setup uses training data from RedPajama dataset (Computer, 2023), with a context size of 4096 tokens, a batch size of 128 sentences, and fine-tuning for 128 steps 2-bit quantization and 32 steps in 3-bit quantization. For GPTQ (uniform scalar quantization), we report the results from the PV-Tuning paper (Malinovskii et al., 2024a).
22

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 13. Weight-only quantization results on Llama-2-77 --BB model after fine-tuning with end-to-ent loss, scalar quantization methods, we report the performance after fine-tuning with PV- Tuning (Malinovskii et al., 2024a).
Method
B
Wiki24
C44
Type
Original
16
5.12
6.63
GPTQ
2.14
8.43
10.82
SqueezeLLM
2.01
6.78
8.82
-el
LNQ + GQuant (Ours)
2.01
6.53
8.53
Scalar
SqueezeLLM
3.03
5.53
7.23
LNQ + GQuant (Ours)
3.03
5.50
7.14
Table 16. Weight-and-an and-ivvaatittitiioonn quantization results on Llama-2-78 model, while quantizing weights into and 3-bits. Wiki2 denotes perplexity on Wikitext2 with the context size of 2048. WwAyKV2 indicates quantizing weights into Iz, activations into yr, and KV cache to b-bits, respectively.
Method
Bits
Wiki2
Original
16
5.12
SpinQuant
W2A4KV4
100.22
SpinQuant GQuant (Ours)
W2A4KV4
36.05
SpinQuant
W3A4KV4
6.61
SpinQuant GQuant (Ours)
W3A4KV4
6.29
The results in Table 15 show that our method remains superior, though the gap narrows larger bit-widths. We hypothesize that existing PTQ methods, which rely less accurate surrogate objectives, have smaller gaps higher bit-widths, allowing fine-tuning to narrow the difference. However, in more extreme compression settings, where the gap is wider, our method maintains its advantage even after fine-tuning.
E.8. Results on Smaller Bit-width in Weight-and--activatin Quantization
In weight-and-activation quantization, further conduct additional experiments with lower bit-widths for weights, specifically 2-bit and 3-bit, while keeping activations and KV caches at 4-bit precision (denoted as W2A4KV4 and W3A4KV4, respectively), on Llama-2-7B model. The results, shown in Table 16, demonstrate that GuidedQuant outperforms baseline methods by larger margin in these more extreme scenarios, highlighting the strength of our approach under stricter bit-width constraints.
E.9. Comparison with mixed-ppecision variant of Squeeze
The dense-and-sppan variant of SqueezeLLM (Kim et al., 2024), which preserves a small fraction of weights 16-bit precision to maintain accuracy, orthogonal to our method and can combined with Accordingly, Table 17, we report results for SqueezeLLM, LNQ, and LNQ + GuidedQuant methods, with the dense-and-spars. approach applied all of them, using the identical experimental setting with Table Following the original SqueezeLLM paper, we retain 0.45% of the weights in 16-bit and evaluate with 2-, 3-, and b-bit quantization the Llama-2-77 7B model. The results show that LNQ with GuidedQuant consistently outperforms the baselines in the dense-and-sparse setting as well, demonstrating the superiority and robustness of our method.
E.10. Results on Different QTIP Variants (1MAD, 3INST, HYB)
The original QTIP paper introduced three variants of their method: IMAD, 3INST, and HYB (Tseng et al., 2024b). Both IMAD and 3INST are look-up table-free methods, while HYB incorporates small look-up table that fits within the LI cache of modern GPUs. The authors reported post-training quantization results without fine-tuning for the 1MAD and 3INST formats, while quantization with fine-tuning was reported for the HYB format. To maintain consistency, we report the better-perfonee peater-performing variant between IMAD and 3INST Table 4 for both QTIP and our method (QTIP GuidedQuant).
23

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table 17. Weight-only scalar post-training quantization results on Llama-2-7B model, evaluated under a dense-and-sparse setting, preserving 0.45% of the weights in 16 bits. Wiki2 and C4 denotes perplexity on WikiTexi2 and C4, respectively. The perplexity is measured with the context size of 4096.
Method
Bits
Wiki2
C44
Original
16
5.12
6.63
SqueezeLLM (0.45%)
2.22
10.64
14.10
LNQ (0.45%) (Ours)
2.22
8.26
10.34
LNQ + GuidedQuant (0.45%) (Ours)
2.22
8.00
10.18
SqueezeLILM (0.45%)
3.24
5.58
7.23
LNQ (0.45%) (Ours)
3.24
5,49
7.15
LNQ GuidedQuant (0.45%) (Ours)
3.24
5,48
7.12
SqueezeLLM (0.45%)
4.27
5.22
6.75
LNQ (0.45%) (Ours)
4.27
5.20
6.74
LNQ GuidedQuant (0.45%) (Ours)
4.27
5.20
6.73
For completeness, the full performance results across IMAD and 3INST are provided in Table 18.
It is worth noting that QTIP has only open- spen-sourced the CUDA acceleration kernel for HYB, although it is theoretically possible to implement kemels for IMAD and 3INST. Therefore, we also include the post-training quantization results (without fine-tuning) for the HYB format as well, summarized in Table 18. The results show that the variations among QTIP methods have minimal impact on the results, and our method consistently outperforms all others in Table 4, regardless of the QTIP variant chosen.
E.11. Discussion on Block-diagonal Fisher Approximation
In this section, we review existing neural network compression methods that use a block-diagonal Fisher matrix approxi- mation of the Hessian and highlight their differences from GuidedQuant. In particular, we discuss WoodFisher (Singh & Alistarh, 2020) for pruning CNNs, Optimal BERT Surgeon (Kurtic al., 2022) for pruning BERT models, and BRECQ (Li et al., 2021) for quantizing CNNs.
WoodFisher and Optimal BERT Surgeon use blocks of arbitrary size B x B along the diagonal to reduce the storage cost. WoodFisher explores B size of {20, 100, 1000, 5000, 12288, 37000} in ResNet-20 (He et al., 2015), while Optimal BERT Surgeon uses B= 50, since the larger block size does not fit the memory. BRECQ leaves the blocks that correspond to the parameters within each residual block in CNNs, and further uses a first-order Taylor approximation on the residual block's outputs to estimate the second-order error for each block avoid the need handle prohibitively large matrices.
The proposed GuidedQuant maintains the blocks corresponding to each output channel, resulting the B size to be 4096 to 11008 for Llama-2-7B 7B model. Directly computing these dick-diagonaal matrices would be infeasible, requiring over 110 TB for and more than 13, 000 GPU hours on RTX 6000 Ada GPU for Llama-2-7B. To address this, GuidedQuant averages the Fisher diagonal blocks within each group. approximately preserving dependencies within each output channel at the scale of modern LLMss We present the theoretical complexity of GuidedQuant in Section 3.2, report its practical cost in Table 9, and report the performance of approximating more (opting for smaller number of groups) in Appendix E.S.
In Figures 3 and 4, we illustrate submatrices of the scaled Fisher information matrix, nF(()) x 10 for the linear layers in the first Transformer block of the Llama-2-7B model, alongside corresponding approximation results. Here, n denotes the number of calibration data, and the results are computed using calibration data from the RedPajama dataset, which consists of 1024 sentences with 4096 tokens each. Since each linear layer in the model contains din dout weights, fully visualizing its Fisher information matrix would yield matrix of size d dout dy which is computationally prohibitive. Therefore, we restrict our visualization to the submatrix corresponding to the first two output channels of each layer. Since each output channel has din weights, this results in visualizing a 2dnn x 2din matrix. Within the Transformer block of the 1-ama-2-7B model, there are seven linear layers: self_atte j.proj, self_attn. k_ j_proj, self aat..nn...... v.proj, self-atton o o.rroj mlp. gate eproj mlp. up-proj, and mip. jown_proj. For the first six layers, dn = 4096, so we visualize an 8192> x 8192 matrix, while for the final layer (mlp. down proj) with dn, = 11008, an 22016 22016
24

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Table Wnlg post-training quantization results on different QTIP variants (IMAD, 3INST, HYB), without fine-tuning to the end-to-end loss. Wiki2 and C4 denotes perplexity on Texi and C4, respectively, which are measured with the context size of 4096.
Llama-2-7B
Llama-2-13B
Llama-2-70B
Variant
Method
Bits
Wiki2
C44
Wiki24
C44
Wiki24.
C41
Original
16
5.12
6.63
4.57
6.05
3.12
4.97
IMAD
QTIP
2.00
7.05
9.14
5.59
7.46
3.87
5.70
QTIP GQuant (Ours)
2.00
6.11
7.99
5.33
7.05
3,80
5.61
QTIP
3,00
5.38
6.99
4.74
6,28
3.27
5.09
QTIP GQuant (Ours)
3.00
5.28
6.87
4.71
6.22
3.25
5.08
QTIP
4.00
5.17
6.71
4.62
6.10
3.16
5.00
QTIP + GQuant (Ours)
4.00
5.16
6.68
4.61
6.09
3.15
5.00
3INST
QTIP
2.00
6.82
8.96
5.52
7.39
3.90
5.69
QTIP GQuant (Ours)
2.00
6.16
7.99
5.33
7.04
3.82
5.61
QTIP
3,00
5.40
7.01
4.74
6.28
3.27
5.09
QTI+ GQuant (Ours)
3.00
5.30
6.87
4.70
6.22
3.26
5.08
QTIР
4.00
5.17
6.71
4.62
6.10
3.16
5.00
QTIP + GQuant (Ours)
4.00
5.16
6.68
4.61
6.09
3.15
5.00
HYB
QTIP
2.00
6.84
9.03
5.62
7.46
3.93
5.74
QTIP GQuant (Ours)
2.00
6.19
8.06
5.36
7.10
3.84
5.64
QTIP
3.00
5.39
7.03
4.76
6.31
3.28
5.10
QTIP GQuant (Ours)
3.00
5.32
6.89
4.72
6.24
3.27
5.09
QТIР
4.00
5.19
6.73
4.63
6.12
3.17
5.01
QTIP + GQuant (Ours)
4.00
5.18
6.70
4.61
6.10
3.16
5.00
matrix is visualized.
We compare two approximation strategies:
WoodFisher: This approach retains the blocks size of B x B along the diagonal. The storage requirement for this method is B dm dout-
GuidedQuant: Here, the block size is set to di, x di, and blocks averaged within groups This strategy requires g dff storage, where is the number of groups.
To ensure a fair comparison, we choose the WoodFisher block size as [g dout / d Specifically, we choose g for the GuidedQuant, which results in B for the self-attention projection layers, 2 for the m1p. gatee proj and mlp. www.uppppoooo.. layers, and B 11 for the mlw.down. www.dwwwwwooooooooo layer.
The visualizations reveal that the original Fisher information matrix exhibits strong off- off-diagonal values and prominent block-diagonal structure with blocks size dn x din- This indicates stronger interactions among weights within the same output channel compared to those across different channels. Overall, the GuidedQuant approximation captures significantly more of this structural detail than the WoodFisher-style block-diagonal approximation, which retains only arbitrarily sized diagonal blocks.
25

---

GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Fisher (Original)
WoodFisher
GuidedQuant (Ours)
self_attn 9-PEo
4096
0.0
6144
0.5
6144
-0.5
6144
005
2048
4096
6144
-1.0
2048
4096
6144
-1.0
2048
4096
6144
-10
10
1.0
2048
sel Belf_atth k-proj
005
2048
4096
6144
-1.0
2048
4096
6144
-1.0
2048
4096
6144
-10
x10
1.0
x10
1.0
x10
1.0
2048
2048
2048
0.5
self_attn V-pro
4096
-0.5
-1.0
-1.0
2048
4096
6144
2048
4096
6144
2048
4096
6144
0.0010
0.0020
0.0010
0.0005
self_attn
o-pro]
4096
0.0000
4096
0.0000
4096
0.0000
6144
6144
6144
4036
-0.0010
2048
4096
-0.0010
Figure Visualization the scaled Fisher information matrix, m) 10 for the first two output channels in se self_attn w.l.aattt.q.proo... self attn.k k proj. self attn. v proj. and self attn. proj layer of the first Transformer block in Llama-2-7B model. layers); Right: the GuidedQuant approximation (the number of groups g 1). Both approximations compared under equal storage budget.
Left: the original Fisher matrices; Middle: WoodFisher style block-diagonal approximation (block size
26

---

QuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Fisher (Original)
WoodFisher
GuidedQuant (Ours)
2048
3.0005
2048
0.0005
2048
0.0003
gate-pzo
mlp
4096
3.0000
4006
0.0000
0144
-0.0005
-0.0005
6144
-0.0005
-0.0010
-000010
-0.0010
2048
4096
6144
2048
4096
61:44
2048
4096
b144
0.0010
0.0010
0.0010
up-pro]
4096
0,0000
4096
0.0000
4096
0.0000
m_p
6144
-0.0005
0144
-0.0005
01.44
-0.0005
-0.0010
-0,0020
0.0010
5504
0.0005
5504
0.0005
5504
0.0005
mlp down-pro)
11008
11008
0.0000
11008
0.0000
16512
-0.9005
18512
16512
-0.0005
5504
11008
18512
-0.0010
5504
11000 16512
0.0010
5506
11008
16512
0.0030
Figure 4. Visualization of the scaled Fisher information matrix, nF 10 for the first output channels in the mwp. mlp. wp.proo.. and mlp. down.ppow layer of the first Transformer block in Llama-2-77 model. Left: the original Fisher matrices; Middle: the WoodFisher style block-diagonal approximation (block size
B 2, and B 11, respectively); Right: the GuidedQuant approximation (the number of groups 4). Both approximations are compared under an equal storage budget.
27


Reviews:
[{"review_id": "FOJ3mLblW9", "reviewer": "Reviewer_WqtU", "summary": "This paper proposes to use layer output combined with gradient information as the objective to minimize layer-wise quantization perturbation, along with an approximate method to solve this resource-intensive problem. By improving GPTVQ and combining it with the new objective proposed in the paper, the experiment performed well on non-uniform quantization and also showed some improvements in uniform quantization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We address below your questions.\n\n---\n**Q1. Evaluations on zero-shot and few-shot downstream benchmarks (Reviewers WqtU, i4yb, UEhr).**\n\nWe provide evaluations of our methods (LNQ and LNQ + GuidedQuant) alongside baselines (SqueezeLLM and GPTVQ 1D) under the weight-only scalar quantization settings, using Llama-2-7B and Llama-2-13B models, in *Table A1*. The evaluation includes eight zero-shot tasks: BoolQ [1], PIQA [2], SIQA [3], HellaSwag [4], WinoGrande [5], ARC-easy [6], ARC-challenge [6], and OBQA [7]. For a few-shot benchmark, we include results on the MMLU [8] benchmark in a 5-shot setting.\n\n*Table A1* reports both accuracy and standard error for all methods. We highlight the best-performing results, as well as those whose accuracy falls within the top score $\\pm$ standard error, under the same bit width constraint. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with notable improvements in extreme quantization scenarios, such as 2-bit quantization. While we currently report results on a subset of models and settings due to time constraints, we will perform a more comprehensive set of experiments and include the full results in the revision.\n\n- *Table A1: https://drive.google.com/file/d/1XnAs0V8CwTKpoul5I3cIVecE5JL8cgto/view?usp=sharing*\n\n\n---\n\n**Q2. Comparison with Dense-and-Sparse variant of SqueezeLLM (Reviewers WqtU, NTd9).**\n\nTo ensure a fair comparison, all methods in the paper were evaluated under uniform precision. The dense-and-sparse variant of SqueezeLLM, which preserves a small fraction of weights in 16-bit precision to maintain accuracy, is orthogonal to our method and can be combined with it. Accordingly, we report results for SqueezeLLM, LNQ, and LNQ + GuidedQuant methods, with the dense-and-sparse approach applied to all of them, in *Table A2*.\n\nFollowing the original SqueezeLLM paper, we retain 0.45% of the weights in 16-bit and evaluate with 2-, 3-, and 4-bit quantization on the Llama-2-7B model. The results show that LNQ with GuidedQuant consistently outperforms the baselines in the dense-and-sparse setting as well, demonstrating the superiority and robustness of our method. We will include these results in the revision.\n\n- *Table A2: https://drive.google.com/file/d/1dORQnlmUi1FPQmuSR8n1OebNQ5h0M8Gi/view?usp=sharing*\n\n---\n\n**Q3. In weight-and-activation quantization (Table 5 in the paper), the GuidedQuant does not show obvious advantage over the baseline.**\n\nThis is primarily because we present weight-and-activation results using relatively high bit-width for weight quantization (4-bit), where the benefits of our method are less evident. Our method demonstrates more significant improvements in more aggressive quantization settings.\nTo illustrate this, we conducted additional experiments with lower bit-widths for weights, specifically 2-bit and 3-bit, while keeping activations and KV caches at 4-bit precision (denoted as W2A4KV4 and W3A4KV4, respectively), on Llama-2-7B model. The results, shown in *Table A3* below, demonstrate that GuidedQuant significantly outperforms baseline methods in these more extreme scenarios, highlighting the strength of our approach under stricter bit-width constraints. We will include these new results and the corresponding discussion in the revision.\n\n- *Table A3: https://drive.google.com/file/d/1PaDjWnzsWEYVd5Lwdd_Xoath72Fwfghr/view?usp=sharing*\n\n---\n\n**References**\n\n\n[1] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. 2019.\n\n[2] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. 2020.\n\n[3] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. 2019.\n\n[4] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? 2019.\n\n[5] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. 2021.\n\n[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.\n\n[7] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. 2018.\n\n[8] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D. and Steinhardt, J., Measuring massive multitask language understanding. 2020.", "claims_and_evidence": "Yes, it is clear. The method proposed is relatively clear and simple, and the paper uses sufficient empirical study to prove the effectiveness of the proposed method.", "methods_and_evaluation_criteria": "Not sufficient. As an LLM PTQ paper, the article only involves model perplexity as the only evaluation indicator, and does not test it on any generative tasks or other few-shot benchmarks, which is inconsistent with convention.", "theoretical_claims": "Yes, I've checked the theoretical analysis in Section 3.1.", "experimental_designs_or_analyses": "Yes, I've checked the main experiment, including the weight-only PTQ and weight-activation PTQ parts. My main concern comes from the fact that the evaluation is limited to model perplexity.", "relation_to_broader_scientific_literature": "The key contribution of this paper can be viewed as an improvement of previous layer-wise error reconstruction problems. By Introducing gradient information as an importance indicator, the reconstruction process can be more accurate and the empirical study shows better results.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "# Strengths:\n1. The paper is well written, and the previous knowledge is fully explained and introduced, making it easy for readers to follow.\n2. The results on non-uniform quantization show significant improvement over existing methods\n# Weaknesses:\n1. In the weight-activation quantization experiment(i.e. table 5), it seems to show that GuidedQuant does not show obvious advantage than GPTQ\n2. It is recommended that the authors supplement the test results of generative tasks and other commonly used zero-shot tasks, such as MMLU.", "other_comments_or_suggestions": "1. The third entry in Table8 should be \"Llama-2-70B\" instead of \"Llama-2-7B\"", "questions_for_authors": "1. Why dense-sparse separated quantization is not enabled for squeezellm ? This setting may significantly improve the results according to the results reported in the original paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrV3f9dYAA", "reviewer": "Reviewer_NTd9", "summary": "This paper introduces GuidedQuant, a post-training quantization framework for large language models (LLMs) that integrates gradient information from the end-to-end loss into the quantization objective while explicitly modeling inter-weight dependencies. The authors claim that GuidedQuant improves the performance of state-of-the-art quantization algorithms across scalar and vector quantization methods. Additionally, they propose a non-uniform scalar quantization algorithm, LNQ, which outperforms existing methods in this category. The paper presents experimental results on Llama-2 models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We address below your questions.\n\n---\n\n**Q1. Lack of empirical evidence showing improved modeling after grouping approximation (Section 3.3).**\n\nWe highlight that the GuidedQuant objective, after applying the grouping approximation, consistently demonstrates a clear empirical advantage over both the layer-wise objective and the diagonal Hessian approximation (i.e., the weighted $k$-means objective). This is evidenced by the performance gains shown in Figure 2, Tables 3, 4 and 5 of the paper.\n\nTo provide further qualitative evidence, we visualize a submatrix of the Fisher information matrix corresponding to the first two output channels in the linear layers of the first Transformer block of Llama-2-7B in *Figures A1 and A2*. Since each output channel contains  $d_{\\mathrm{in}}$  weights, the visualized matrix has dimensions of $2 d_{\\mathrm{in}} \\times 2 d_{\\mathrm{in}}$ (here, $d_{\\mathrm{in}}=4096$ for all layers except `mlp.up_proj` for which  $d_{\\mathrm{in}}=11008$). \nThe visualizations reveal that the original Fisher information matrix exhibits strong off-diagonal values and a prominent block-diagonal structure with blocks of size $d_{\\mathrm{in}} \\times d_{\\mathrm{in}}$, corresponding to interactions within each of the two output channels. GuidedQuant approximately preserves the off-diagonal terms within each output channel, thereby capturing significantly more structural detail than the diagonal approximation (SqueezeLLM), which ignores the cross-weight interactions. We believe this better structural modeling helps explain the performance gains achieved by our method. We will include these results and discussions in the revision.\n\n- *Figure A1: https://drive.google.com/file/d/1ilhJarLN1u1nNBzT2ywDCxg8IRz-T2oz/view?usp=sharing*\n- *Figure A2: https://drive.google.com/file/d/19LJkHuKebYyOvPsqsjJQG_5PkJAWr7Bz/view?usp=sharing*\n\n---\n\n**Q2. Comparison with mixed-precision variant (dense-and-sparse) of SqueezeLLM.**\n\nPlease refer to **our response to Q2 from Reviewer WqtU**, where we present a comparison between our method and SqueezeLLM under a mixed-precision (dense-and-sparse) setup. The results show that our method consistently outperforms the SqueezeLLM in this setting as well.\n\n---\n\n**Q3. Results on newer models other than Llama-2 (Reviewers NTd9, i4yb, UEhr)**\n\nWe have conducted additional experiments on newer models, including Llama-3-8B and Llama-3-70B, comparing our method with SqueezeLLM under a weight-only scalar quantization setting. We present the results in *Table A4*. LNQ with GuidedQuant consistently outperforms the baselines, demonstrating the robustness and effectiveness of our approach. We plan to conduct more comprehensive experiments on these newer models and will include the results in the revision.\n\n- *Table A4: https://drive.google.com/file/d/11bgdJ5eOO5s3LjjcZgVPyAzPsXDr6JJZ/view?usp=sharing*\n\n---\n\n**Q4. The gradients are computed on calibration data, which is unaddressed in Appendix A when connecting the objective to the Fisher matrix.**\n\nIn Proposition 3.1, we define the Fisher information matrix $\\mathbf{F}$ as the *empirical* Fisher information matrix computed over the calibration data, $\\mathbf{F} = \\frac{1}{n} \\sum_{i=1}^n \\nabla \\ell_i (\\mathbf{w}) \\nabla \\ell_i (\\mathbf{w})^\\top$. Accordingly, the proof assumes that gradients are computed using the calibration data and is connecting the objective to this empirical Fisher information matrix. In the revision, we will explicitly refer to $\\mathbf{F}$ as the empirical Fisher information matrix on the calibration data and clarify that it is an approximation of the true Fisher information matrix.\n\n---\n\n**Q5. Minimal gains with a bigger number of groups, and lack of explanation on the choice of hyperparameter $g$ (e.g., $g=4$ in 7B models).**\n\nWe believe that achieving strong performance even with a small number of groups is a strength of our method, making it particularly effective and robust in resource-constrained settings. Regarding the hyperparameter choice, we selected the number of groups $g$ to be as large as possible within the limits of our computational and memory constraints. We will clarify this in the revision. \n\n---\n\n**Q6. Additional comments and suggestions.**\n\nThank you for the suggestions. We will revise the writing to improve overall clarity, including correcting typos, reducing redundant explanations, and refining figure captions for clarity.", "claims_and_evidence": "- GuidedQuant “explicitly models pairwise interactions between weights.” **Issue**: While Proposition 3.1 links the objective to second-order error estimates, the actual implementation approximates Hessian matrices via grouping (Section 3.3), which dilutes the theoretical benefit. No empirical evidence shows improved modeling of interactions.\n- LNQ “outperforms existing non-uniform scalar quantization methods.” **Issue**: Comparisons to GPTVQ 1D and SqueezeLLM are provided, but SqueezeLLM’s mixed-precision variant (Kim et al., 2024) is excluded.", "methods_and_evaluation_criteria": "- The grouping heuristic is reminiscent of block-wise quantization (Dettmers et al., 2022) but lacks justification.\n- Results are restricted to Llama-2 models. No experiments on other architectures.", "theoretical_claims": "- LNQ’s closed-form codebook update (Equation 10) is a theoretically sound improvement over gradient-based fine-tuning.\n\n- The proof (Appendix A) correctly links the objective to the Fisher matrix but assumes gradients are computed precisely. In practice, gradients are noisy due to calibration data  (Frantar et al., 2022), which is unaddressed.", "experimental_designs_or_analyses": "Table 10 (Appendix D.1) shows minimal gains with larger groups, suggesting the method is not robust to hyperparameters.", "relation_to_broader_scientific_literature": "GuidedQuant builds on second-order optimization (Hassibi & Stork, 1992) and layer-wise reconstruction (Frantar et al., 2022).", "essential_references_not_discussed": "Not applicable.", "other_strengths_and_weaknesses": "Not applicable.", "other_comments_or_suggestions": "**Typos**:  \n   - Page 3: \\(\\mathbf{z}_{j}\\in\\mathbb{R}^{d_{\\text{out}}}\\) has a mismatched bracket.  \n   - Table 1: “WAA4KV4” is undefined.  \n\n**Repetition**: Section 2 redundantly explains basics of quantization.  \n**Clarity**: Figure 1’s caption is overly technical.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tUkspd41JE", "reviewer": "Reviewer_i4yb", "summary": "The paper introduces GuidedQuant, a layer-wise quantization method that also considers its impact on the final loss. Through derivation, the authors propose a new objective that reweights the Hessian using the loss with respect to the layer outputs. To mitigate computational costs, they approximate the Hessian by averaging it over groups. Additionally, the paper presents an EM-based algorithm (LNQ) to learn quantization scalars.\n\nThe proposed methods are evaluated on LLaMA2 across various quantization settings, where GuidedQuant consistently outperforms existing approaches in nearly all scenarios.\n\n**Update after rebuttal**: My latest reply reflected my final update.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review. We address below your questions.\n\n---\n\n**Q1. Results on newer models**\n\nPlease refer to **our response to Q3 from Reviewer NTd9**, where we conduct more experiments on Llama-3-8B and Llama-3-70B under a weight-only scalar quantization setting. The results show that our method consistently outperforms the baseline methods in these settings as well.\n\n---\n\n**Q2. Evaluations on zero-shot and few-shot downstream tasks**\n\nPlease refer to **our response to Q1 from Reviewer WqtU**, where we evaluate our method alongside baselines on eight zero-shot downstream tasks and the MMLU benchmark under a 5-shot setting. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with especially notable improvements under extreme quantization scenarios.\n\n---\n\n**Q3. Clarification on the gradient / Hessian cache size**\n\nWe note that GuidedQuant only requires the Hessian averaged within groups for each layer. Since this averaged Hessian can be computed using the group-wise averaged gradient values, there is no need to cache the full gradient with respect to the output tensor. Instead, we only store one averaged scalar gradient value per group, per layer.\n\nFor example, in the case of the Llama-2-7B model with $g=4$ number of groups, each layer only needs to cache 4 scalar values per layer for averaged gradients.\nThe total gradient cache size is therefore computed as:\n\n- $1024$ (batch size) $\\times$ $4096$ (sequence length) $\\times$ $4$ (number of groups) $\\times$ $32$ (number of layers) $\\times$ $7$ (number of modules per layers) $\\times$ $2$ bytes (stored in 16-bits) = $7.5 \\text{GB}$.\n\nThis averaging based approximation also applies to the Hessian computation, substantially reducing the storage cost compared to storing the full Hessian without approximation. This is how the cache sizes reported in Table 8 in the paper are derived. We will clarify this in the revision.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. However, it would be great to evaluate on newer models and on some downstream tasks, as perplexity sometimes doesn't reflect the downstream performance.", "theoretical_claims": "Yes. I checked the proof of proposition 3.1 and I think there are no issues.", "experimental_designs_or_analyses": "Yes. \n\nHowever, I have question on how is the caching is calculated in table 8. For gradient respect to outputs, I thought the approximate cost for a LLaMA2-7B model is\n1024 (batch size) * 4096 (sequence length) * 4000 (dimension) * 32 (number of layers) * 7 (number of modules per layers) * 2 (if store in 16bits) / 10 ** 9 = 7516 GB, the same question also extends to Hessian caching.", "relation_to_broader_scientific_literature": "The findings can be applied to layer-wise pruning as well.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "[Strength]\n\n* The paper introduces an useful improvement to equip layer-wise quantization with awareness of final output loss.\n* The paper is well-written and easy to follow.\n* The experimental results are strong and generalize to different quantization methods.\n\n[Weakness]\n\nI only have some minor comments which can be found in \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\".", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0z6YCJM6S", "reviewer": "Reviewer_UEhr", "summary": "This paper introduces GuidedQuant, a novel quantization framework that integrates gradient information from the end-to-end loss into the quantization objective while explicitly modeling inter-weight dependencies. The authors first identify a critical limitation of existing quantization methods: they either treat all hidden features equally or, when incorporating end-to-end loss, fail to capture the cross-interaction between weights. \n\nTo address these challenges, the paper proposes guided quantization, which leverages gradient information from the end-to-end loss to reweight layer-wise quantization errors adaptively. Furthermore, the authors introduce Layerwise Non-Uniform Quantization (LNQ), a novel non-uniform scalar quantization method that employs a closed-form solution for optimizing the codebook and a coordinate descent algorithm for optimizing the assignment matrix.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback. We address below your questions and concerns.\n\n---\n\n**Q1. The objective is derived from changes in the end-to-end loss, which is similar to previous work & Concern on the simplicity of the approximation method.**\n\nWe believe the simplicity of our approach should not be mistaken for a lack of novelty. On the contrary, we view the simplicity of our approach as a strength rather than a limitation, as it achieves strong and robust performance gains through a simple and efficient method (see Figure 2 and Tables 3-5 of the paper).\n\nThat said, we agree that exploring alternative grouping strategies, such as incorporating sophisticated clustering algorithms for more accurate Hessian approximation, is a promising direction for future research.\n\n---\n\n**Q2. Evaluations on zero-shot and few-shot downstream tasks**\n\nPlease refer to **our response to Q1 from Reviewer WqtU**, where we evaluate our method alongside baselines on eight zero-shot downstream tasks and the MMLU benchmark under a 5-shot setting. The results show that LNQ combined with GuidedQuant consistently matches or surpasses baseline performance, with especially notable improvements under extreme quantization scenarios.\n\n---\n\n**Q3. Comparison with uniform scalar quantization methods**\n\nWe note that comparisons with uniform scalar quantization methods are included in our experiments. Specifically, Table 3 includes the baselines of weight-only uniform scalar quantization methods such as GPTQ and QuIP, and Table 5 includes comparisons for weight-and-activation uniform scalar quantization methods like QuaRot and SpinQuant. Our method consistently outperforms these baseline approaches in both weight-only and weight-and-activation settings, and shows effectiveness even when used with a uniform quantization scheme as well (SpinQuant + GuidedQuant (Ours) in Table 5).\n\n---\n\n**Q4. Evaluations on different models**\n\nPlease refer to **our response to Q3 from Reviewer NTd9**, where we conduct more experiments on Llama-3-8B and Llama-3-70B under a weight-only scalar quantization setting. The results show that our method consistently outperforms the baseline methods in these settings as well.\n\n---\n\n**Q5. Derivation for closed-form solution in Section 4.2**\n\nThe closed-form solution $\\mathbf{c}^\\ast$ in Section 4.2 (Eq. 10) is the standard least squares solution\n $\\mathbf{c = A^\\dagger y}$ for the problem $\\min_{\\mathbf{c}} \\\\| \\mathbf{y - A c} \\\\|^2$, where $\\mathbf{y = X w}$ and $ \\mathbf{A =X P}$.  We assume that $\\bf{A^\\top A}$ is invertible, hence $\\mathbf{A^\\dagger = (A^\\top A)^{-1} A^\\top}$. This is a common assumption in quantization.\nIn practice, $\\bf{A^\\top A}$ is not always invertible. To address this, we add $\\bf \\lambda I$ to $\\bf{A^\\top A}$, with a small $\\lambda = 10^{-7}$, as commonly done in prior work. \n\n---\n\n**Q6. Clarification on the perplexity evaluation setups and ensuring a fair comparison**\n\nTo ensure a fair comparison, we use the perplexity values reported in previous papers only when the evaluation setup strictly matches ours, i.e., same calibration and evaluation dataset, context length, and a calibration data size that is equal to or larger than ours (giving the baselines an advantage). When any of these criteria were not met, we reproduced the baseline methods under the exact same setup as our experiments. We will clarify this in the revision.\n\n---\n\n**Q7. Clarification on measuring GPU cost in Table 7 and 8**\n\nIn Tables 7 and 8, we report the end-to-end runtime for each quantization method, along with the number and type of GPUs used (indicated in parentheses). For example, \"1×R6A\" refers to the runtime measured using a single RTX 6000 Ada GPU, while \"4×R6A\" indicates that the method was run using four RTX 6000 Ada GPUs.\n\n---\n\n**Q8. Marginal improvements in quantization results with higher bits**\n\nThe performance gains from our method are especially prominent in more challenging quantization scenarios. In particular, our approach demonstrates clearer advantages in lower-bit settings or when applied to models that are inherently harder to quantize, such as Llama-3 models. In these settings, our method yields more substantial improvements. We will include these discussions and corresponding results in the revision.\n\n---\n\n**Q9. Non-uniform scalar quantization format for the GPTQ method**\n\nNon-uniform scalar quantization maps weights to non-uniform grids. GPTQ is designed to minimize the layer-wise output reconstruction error for a fixed quantization grid, thus it can be extended to non-uniform grids by also optimizing the choice of the grid.\nThe GPTVQ 1D baseline does this; it applies the GPTQ algorithm to optimize assignments for a fixed quantization grid while optimizing the non-uniform codebook using gradient descent. We included GPTVQ 1D as a baseline in Table 3 in the paper and showed that both LNQ and LNQ with GuidedQuant consistently outperform it across all evaluated models and bit-widths.", "claims_and_evidence": "- The authors theoretically derive an objective function that accounts for the importance of each layer in Section 3.1.\n- They provide a proof in Appendix A to support Proposition 3.1, which states that minimizing their proposed objective function is equivalent to minimizing a second-order error estimate incorporating the Fisher information matrix, including inter-weight interaction terms.\n\n- Their claims are also supported by evaluation results presented in Section 5 and Appendices C and D.", "methods_and_evaluation_criteria": "The proposed methods are well-justified for the following reasons.\n- They address the limitations of existing quantization methods, which fail to account for the varying importance of layer-wise outputs in relation to end-to-end loss.\n- They also mitigate another drawback of end-to-end loss-based approaches, which tend to overlook inter-weight dependencies.\n- The guided quantization framework is adaptable and can be integrated with various existing quantization methods.\n\nHowever, the proposed methods lack novelty.\n- The objective function is derived from changes in the end-to-end loss, which is similar to previous work.\n- The proposed approach to addressing increased computational and memory costs is relatively straightforward. More ablation studies are needed to either substantiate the claim that a simple method is sufficient or to explore ways to mitigate accuracy degradation in 2-bit quantization.\n\nAdditionally, evaluation criteria is somewhat limited.\n- The results are presented only on LLaMA 2 models using perplexity on the Wikitext-2 and C4 datasets. To better support the effectiveness of the proposed methods, evaluations on additional benchmark datasets (e.g., MMLU, zero-shot evaluations) should be included.\n- Since uniform scalar quantization is widely used, further comparisons with this approach would provide a more comprehensive assessment of the proposed method’s effectiveness.", "theoretical_claims": "In Section 4.2, the closed-form solution for optimizing the codebook is theoretically well-motivated; however, it lacks a formal proof or derivation.", "experimental_designs_or_analyses": "- The experimental designs are well established to support the paper's main claim and address potential concerns about the proposed methods' weaknesses.\n\n- However, authors mentioned they used perplexity values reported in previous papers for several baselines. However, perplexity can vary depending on evaluation setups and is rarely reproducible across different setups. If the authors aim to claim state-of-the-art performance, they should evaluate baseline methods in the same environment to ensure a fair comparison.\n\n- Moreover, while the proposed methods achieve the best results, the accuracy improvement appears marginal, particularly in 3-bit and 4-bit quantization. Broader evaluations across diverse architectures and tasks would strengthen empirical validation.", "relation_to_broader_scientific_literature": "The key contributions of this paper relate to the literature on LLM post-training quantization (PTQ), which aims to minimize layer-wise reconstruction error. While adopting the concept of layer-wise error minimization, this paper introduces a new objective function that accounts for both layer-wise importance and inter-weight interactions.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "I have discussed the strengths and weaknesses of this paper in the preceding sections.", "other_comments_or_suggestions": "- Can you clarify how GPU cost in Table7 and Table8  is measured? \n\n- In Section 5.1, you mentioned applying GuidedQuant to non-uniform scalar quantization and vector quantization. However, Table 3 includes GPTQ, which uses uniform scalar quantization. If you applied a non-uniform scalar quantization method for the GPTQ baseline, could you provide a more detailed explanation? If not, further clarification on this would be helpful.", "questions_for_authors": "Comments and suggestions section covers up my questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces GuidedQuant, a novel post-training quantization framework that integrates gradient information from the end-to-end loss and explicitly models inter-weight dependencies. After the rebuttal, it receives two accept, one weak accept, and one weak reject. Its merits, including the interesting idea, good organization and writing, extensive experiments and good results, are well recognized by the reviewers. The response well addresses the reviewers' concerns about the additional evaluation and comparison, detailed analysis, and so on. I think the current manuscript meets the requirement of this top conference and recommend for acceptance. Please incorporate the revision in the updated manuscript.
Decision: Accept (poster)