PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Authors: Vladimir Malinovskii
Denis Mazur
Ivan Ilin
Denis Kuznedelev
Konstantin Pavlovich Burlachenko
Kai Yi
Dan Alistarh
Peter Richtárik
Venue: NeurIPS 2024 oral (2024)
Keywords: Quantization
Large Language Models
Primary area: natural_language_processing
TL;DR: Improving Extreme LLM Quantization through Better Fine-tuning

Abstract: There has been significant interest in "extreme" compression of large language models (LLMs), i.e. to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constrained devices.  
Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training  approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. 
In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs.
We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases.
On the practical side, when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. 
Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama-2 family models at  2 bits per parameter.

PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Vladimir Malinovskii Yandex, HSE University
Denis Mazuri MIPT SberDevices
Ivan Ilin Al Initiative, KAUST"
Denis Kuznedelev Yandex, Skoltech
Konstantin Burlachenko AI Initiative, KAUST
Kai Yi AI Initiative, KAUST
Dan Alistarh
Peter Richtarik Al Initiative, KAUST
IST Austria, NeuralMagic
Abstract
There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constraines devices. Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training approaches are reaching diminishing returns in terms of the accuracy- vs-bit-width trade-off. State-of-the
quantization methods such QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. In this work, question the use STE for extreme LLM compression, showing that it can be sub-optimal, and perform systematic study of quantization-awwa fine-tuning strategies for LLMs. We propose PV-Tuning a representationn-anooti framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases. On the practical side, when used for 1-2 bit vector quanti- zation, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama-2 family models 2 bits per parameter.
Introduction
Recent years have seen the development of ever more capable large language models, attracting immense interest from both researchers and industry. One of the driving factors behind progress in this area is the availability of powerful open LLMs such as Llama [69], Mistral 34 355, or Phi [4]] The main advantage of open LLMs is that they be run and fine-tuned locally by end users; however, as state-of-the-art LLMs grow larger, they also become harder to run on commodity hardware. For instance, in order fit the best available Llama-3 model a consumer GPU, the model would have to be compressed to below 2 bits per parameted
To achieve such "extreme" degrees of compression accurately, researchers have proposed a variety of techniques, which can be roughly categorized into i) better quantized weight representations and úi better algorithms to learn these representations. The weight representations used for extreme quantization include group quantization 22 20 sparse high-precision outliers [7
32 incoherence processing of the weights 19, 77). additive and residual quantization (1 72]. In turn, the
calibration algorithms also vary between data-free methods [20), layer- wise calibration [22 [8],
block-wise or global fine-tuning 71 even quantization-aware training 778 75). However, the
1 Equal contribution. Equal senior authors.
Moscow Institute of Physies and Technology, Russia 1 Work performed while at Yandex. King Abdullah University of Science and Technology, Saudi Arabia
At the time of writing, the hest open model (Llama-3 70B) occupies 130GB FP16, while most consumer GPUs have 8-2GGBB DRAM, some of which must reserved for the attention cache.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

QuIP+
70
Perplexity Wiliteet-2
AQLM
Average 0-sioot 0-sbbot Avcuracy
PV
65
60-
QuIPa
AQLM
8
16
16
Model size (GiB)
Model size (GiB)
Figure 1: WikiText-2 perplexity (left) and average zero-shot accuracy (right) of 2-bit quantized LLAMA models as a function of model size (GiB). See detailed setup in Section 4.3
weight representation and the fine-tuning algorithm are largely orthogonal: most popular quantized representations could be obtained layer-wise in one-shot, fine-tuned layer-wise to variety of optimization objectives, or even trained entirely from scratch.
Surprisingly, there is clear disparity between the degree of interest shown to accurate one-shof quantization versus accurate fine-tuning. Specifically, one-shot quantization is very well-studied, to the extent that, as shown in Figure 2
improvements in this direction are clearly saturating. At strategy is largely unknown: while many recent works use
the same time, the impact of some form of fine-tuning [63 21 71) they typically consider a single fine-tuning regimen based on trrighh-through estimation (STE) 6 15]. Thus, given the multitude of representations considered,
is not at all clear whether current fine-tuning strategies are optimal.
In this work, we analyze the problem of fine-tuning over highly-comprest weights from the optimization perspective. We begin by analyzing popular line-tuning strategies for extreme LLM quantization. The key challenge in this context is that the quantized representations may contain both continuous and discrete variables: while continuous parameters, such as learnable scales or codebooks, be optimized by backpropagation, the discrete parameters (e.g., integer assignments for the weights) cannot. Existing fine-tuning techniques either do not optimize over discrete parameters at all [71 21 or fine-tune them using heuristics such as STE or stochastic rounding (3]. Unfortunately, these methods are not well-justified for weight quantization from the point of view of optimization theory, and, we show in Section 3 can provide poor practical performance.
We propose an alternative solution: instead of following heuristic gradient estimates, our approach follows the actual gradient of the objective in a small subspace of optimized parameters where it can be meaningfully improved. Following this insight, we formulate the PV-tuning framework for fine-tuning arbitrary quantized representations. We update both discrete and continuous components to minimize a global objective function, such as the KL divergence relative to the original model predictions. Our results show that this strategy leads to significant improvements across weight representations, achieving new state-of-the-art in compression-accuns
trade-offs.
The main contributions of our work can be summarized as follows:
1. We analyze the problem for training discrete quantized representations for better understanding of the limitations of existing optimization algorithms. We then propose novel algorithm inspired by compressed gradient methods that addresses these limitations. When compared to straight-through estimation and stochastic rounding, our approach 1) can be shown to converge to a stable solution; and 2) this solution is significantly more accurate in practice.
2. We generalize the proposed algorithm into the PV-Tuning framework which can minimize a global objective function over a general quantized representation, by optimizing both continuous and discrete parameters via variant of coordinate descent.
3. We demonstrate that PV-tuning can improve quantized model accuracy for leading existing approaches, including GPTQ and AQLM, on popular LLMs including Llama-2 & 3 and Mistral. Our procedure achieves state-of-ther thethrr accuracy (measured through perplexity) in 1- and 2-bit quantization regimes while using the same amount of calibration data as the original algorithms. Importantly, the PV-tuned models use the same underlying weight representations, and are compatible with existing inference kernels. In terms of accuracy per model size, PV-tuning of vector quantization outperforms all prior techniques in the 1-3 bits/parameter range, and is the first to achieve Pareto-optimal quantization for Llama-2 models at around 2 bits per parameter.
By "one-shot" we refer to methods that quantize quantize the model a single pass over calibration data. The official implementation available httppss //g1 thggthhhbb.... wwt.//oo//////eeee wtw.////////////eeeeeeeeeeeeeeee tuning

---

2 Background
Post-Training LLM Quantization (PTQ). There has been significant interest in PTQ meth- ods 49 23 that would scale to LLMs. Early work [17
78 50] used direct round-to-nearest (RTN) quantization over weight groups of well-chosen size. GPTQ [22] improved upon these results significantly via an accurate one-shot solver for minimizing layer-wise compression errors. Next, AWQ 422 improved upon these results by employing per-channel scaling to reduce the error on important weights while SqueezeLLM [36] implemented non- uniform quantization. QulP [9] pro- posed a more accurate weight representation by leveraging incoherence matrices. Another line of works [8] 39) proposes an improved quantized weight representation, which saves a small fraction of outliers in full precision. Other recent works propose augmenting quantized representations with lowrank "adapters" that compensate quantization error 28 84 Recently, BiLLMM (32) developed residual binarization that stores salient weights in progressively higher bitwidth, quantizing models
to nearly bit per parameter at non-catastrophic accuracy loss.
Currently, the state-of-the-art methods in terms of accuracy-vs-size are QuIP# 71 and AQLM (21) Both methods work roughly by mapping weight groups to points on highly-dimennsonal lattices, which are either chosen satisfy some optimality properties (for QulP#) Interestingly. AQLM showed that fine-tuning the continuous parameters (codebooks) can improve accuracy significantly relative to pure one-shot compression: a variant of this approach was also adopted by QulPH. PV-Tuning is compatible with both methods: as we show, it can lead to state-of- tae-art compression results for such representations.
learned (for AQLM).
Fine-tuning over Quantized Weights. As mentioned above, the two SOTA quantization techniques apply fine-tuning, but only update continuous parameters, such as quantization scales. When optimizing over discrete parameter sets, a standard choice in deep learning is the Straight-Through Estimator (STE) 6 733.
Prior work on LLM compression proposed to update both continuous and discrete parameters, via STE, both for post-training quantization [78 63 and for training quantized networks from scratch [32). However, it was observed early on that STE leads to instability when fine-tuning heavily quantized LLMs (78). While early results suggest that STE can perform well when training quantized models from scratch 44] this behavior is yet to be validated for highly-peefformnn multi-billion- petiiammlioon-r models, which are the focus of our work.
In summary, the two standard approaches for fine-tuning quantized LLMs are 1) fine-tuning only over the continuous parameters, such as quantization scales, which heavily limits the number of trainable parameters; and 2) optimizing all parameters via the STE, which however is known be quite noisy especially for extreme quantization. In this context, our work proposes alternative approaches in the post-training compression setting, which lead to state-of-the-ard results relative to both options.
3 Fine-Tuning Quantized Models
In this section, we study the problem of fine-tuning quantized models to minimize a global objective, such cross-entropy. Section 3.1 formulates this problem from an optimization perspective and introduces our notation. In Section 3.2] we analyze several popular strategies for solving this problem and highlight some of their limitations. To circumvent these limitations, we propose alternative optimization algorithm in Section 3.3 and discuss implementation details in Section 3.4
(.4]
3.1 Problem description
Consider the problem of minimizing objective (loss) 6,
min (xx),
ZER
where Φ IRd__ - R a differentiable function bounded from below (e.g., by zero), and Rd c Rd is set of all possible quantized weights that can be represented with a given quantization method. Without loss of generality 4 we first analyze the case of scalar nonlinear guantization. In this scenario,
e E [d] := (1,2...., d} (typically e < d). and TR C IRd is the set of all vectors in IRd whose d entries take exactly distinct values. In other words, the cardinality of the set (zz) == (I1,..., zd)
is equal c, and we can therefore write Bd = {a E Id
[(x) c).
We explain how this generalizes to other quantized representations in Appendix
3

---

Useful notation. A vector I E IRA naturally induces a partition, which we shall call P(r), of the set {I....,de .....dd] into nonempty subsets Pi (a.)....,PP characterized by
P...,.,()))
J; I Ij
3k i E Pk and j Pk-
Moreover, we shall write 2 P(a) if each element of P(x) ( is a subset of some element of P(y). For distinct i.j E [d]. let us introduce the notation otherwise. Given this notation, notice that P(y) 2 P(x)) and only if for all # j we have (x)(2) = = yi = Vj- Finally, we define I = R U U IR as the set of all vectors in R whose entries take at most c distinct values. So, if x E R and P(v)= E P(x), then ge
Let's denote (P)):=(P(=))...... (z)
Pc(x)).
(ar( 1 if there exists It such that i, Pk (x), and (1)(2)
PV method. Following this notation, we define an optimization algorithm that alternates between optimizing & with fixed P or fixed V. From practitioner's 's point of view, these represent optimizing continuous parameters (scales, codebooks, zeros) and discrete codes (assignments), respectively.
The P step (fixing P). Given Re consider the mapping
Mp(x) = (p) == arg min {d(u) : P(y) 3 P(x)}. yER
(2)
Notice that, necessarily, Mx(2) E R& and ((Mp(x)) ((Mp(z)) I d(x). Evaluating Mp
amounts to solving an unconstrained optimization problem in a c-dimensional space.
The V step (fixing V). Similarly, given R we define the mapping
Mv (v) = Mv.o(y) == arg min (Φ(x) : V(x) C V(v) ZERd
Likewise, (y) E R and (Mv & (M (v)) s dly). Evaluating My amounts to solving
difficult discrete optimization problems with search space of size [V(x) 2 < c (exponential in d).
Algorithm 1 PV algorithm
1: Initialization: starting point x0 E IRd
2: for k 0, 1.... do
3: yd Mp(x") = arg yin {Φ(v) P(y) 3 P(x*)}
(P step: continuous) (V step: discrete)
4: =++ = My = arg min {δ(x): : V((z) S v(**)}
5: end for
Our key algorithmic idea, in its simplest form, is to optimize ø by alternating the P and V steps, i.e., iteratively applying the Mp and My operators. (We will propose several more practically-uses -useuul approximations and variations later; see Sections (3. and also Appendix [B) This resulting method, which call the PV method, formalized Algorithm Our key guarantee for the PV method is formalized in the next result.
Theorem 3.1 (Convergence of the PV method), Assume á is bounded below, and let x E Rd. Then (i) R Rdd and x Rd for all le 2 0; (ii) ((ah+1) Φ(v*) S d(a*) for all k: 2 0; and (iii)
the sequence {0(**)) converges.
The proof can be found in Appendix A.I Note that we do not claim that the method converges to a minimizer of Φ; the optimization problem is too difficult for us to be able to guarantee this. However, as we shall see the numerical results, we nevertheless obtain great empirical performance, especially when coupling the PV approach with some additional algorithmic tricks.
This general approach is popular in "shallow" machine learning problems; for instance, if ((x)) 1/2 is the squared error with respect to some user-specified vector z, then the above algorithm recovers I-dimensional K K-means on the data vector z. Likewise, if Φ(-) the log-likelihood, then, depending the choice of the set Rd the approach is related to the EM algorithm [16).
|| 2|
In turn, we apply the PV method to obtaining highly-accurate quantized LLMs. Applying the PV method "as is", would be infeasible in practice: computing the P and V mappings requires solving difficult optimization problems especially due to L.L. parameter scales. However, both mappings can be approximated. The P step can be reparameterized unconstrained optimization problem on the

---

unique values in the weight matrix. Practically it means that the "codebooks" can be optimized using an automated differentiation engine (i.e. PyTorch). However, for many quantized representations. (r( can be approximated by one or more steps of GD, directly optimizing @ over the set V(z) (x) of its e unique values. The c-dimensional gradient can be computed efficiently by backprop, as described in prior works [63 71. On the other hand, the V step (-)) is more difficult to approximate as it involves searching a discrete space of size c² We dedicate the next two sections to this task.
3.2 Linearized V step & gradient- discrete updates
The V mapping 3 can be approximated by solving discrete least squares problem using an approximation of (x)) around y:
(xz) ~ (,(x) == o()) + (Vo(y), x- v) + 11 yll.
(4)
where L 0 is sufficiently large constant. Subsequently, we perform the V step using the simpler convex quadratic function dy instead of the typically more complicated function oi
My)
M My (v) arg min .ERd
(0)(z)) V(z) S VV()).
Our first lemma shows that we can replace Ty by a more convenient function by measuring the squared distance between x and y+ == y L a single GD step from with learming rate z. disregarding the constraint:
1 Voly), the latter being the point obtained after taking
Lemma 3.2. For any y Id we have Mv. (y) M v. (v), where
Mv.
x,(x) =
Vo(v)) 2 y s+1 Σ (x. v()²²
(5)
The proof can found in Appendix A.2
To summarize, the V step the PV method (Algorithm i.e., x = (v), can be approximated via the "linearized V step"
I
= Mv. M, v. (y) = ft.
Our next lemma says that the above approximation is in a certain sense natural reasonable provided that o is L-smoothfon on Rd i.c., provided that
<(x) S ((v) + (Vo(v), x - y) + 21100 yll
Vx, y E Rd Se
(7)
Lemma 3.3 (Monotonicity). Lei E R I is L-smooth on Rd <e then & (Mvioly) S x(0)= o(y), where f is the point obtained from y by the linearized V step (6.
S
Indeed, the point obtained via the lincarized V step can not have a worse loss than the previous point y. Of course, one hopes that the loss will strictly decrease so that the method makes progress. From a practical perspective, the key advantage of linearized V step is that it can be performed much faster compared to the vanilla V step. The proof of Lemma 3.3 can be found in Appendix AA3
Note that since (2)) is separable (see (8). each entry/weight of can be optimized independently of others. For sealar quantization, each individual problem can be solved in (logg(() (c)) time using binary search in sorted version of V(y). For vector quantization, there are specialized optimization procedures for efficiently minimizing the Ln2 error (see Appendix
Key challenge. The main caveat with linearized V step is that it may be impossible to make small gradient-based updates to low-bitwidth discrete weights. More specifically, in (6). one must update the discrete assignments to approximate y² 1. V. However, for low-bit weights, the desired update 1 V can be smaller than the lowest possible increment to obtain quantized vector. As result, the optimal solution to (6) is often y² itself. such a situation, the algorithm will get stuck on y², which is undesirable. This problem is especially pronounced in deep LLMs, where L can be very large, or, from a practitioner's point of view, where one needs a small learning rate. practice, as we explore in Section 4.2) the lowest learning rate where the algorithm makes any updates at all is already too large for optimization, leading to divergence.
'It is possible to consider different class of functions instead: e.g., Lipschitz functions. In such a case, we would different approximation. For simplicity of exposition, we work with L-smooth functions.
5

---

Algorithm 2 PV-Tuning: Optimization
Algorithm 3 PV-Tuning:
Implementation, one step
Require: initial parameters x0 E TRd
Require: quantized model, subspace size tau deq_model : dequantize_weighte (model) 2: for f = 1, ,,...?
objective function Φ :: I - R,
subspace size T [d]
1: for = O..... K
3: loss = deq medel (next_batch) batch ()) .loss accumulate gradients for P and V steps
2: b P Pstep: update (xx) by backprop
4: loss. (assbbckwarr()
3:
V" arg (igmi(((()) : P(y) 2 P)z*)}
5: end for
6: b P step: update codebooks by backprop
4:
V step: choose a subspace s & update P(a)
7: grad_phi deq_model www.moowwww.ww...eeeeeeee weight grad
5: S arg t [vo((")] find largest
8: grad_codebooks backprop (grad_phi)
isidd
9: model.codebooks adam
6:
= | x- y Ligh 2t (volu)))
10: > V step: choose subspace and update codes
II: update adam (grad_phi) deq_model weight
7: = arg mink
{ ( (o)))s((t) (V)))(())
12: s choose_ chhose_subspee subspace (update,, tau) (ww.aaattttaaataaaaa (update(s]) 13: model. codes [s] find_neares
8: end for
Many popular strategies for discrete fine-tuning can be seen attempts reconcile coarse low- precision weights with the need to make small updates. These include straight-through estimation, stochastic rounding, or adding regularizers that push the solution to (6) away from y² We review straight-through estimation in Appendix E. and stochastic rounding in Appendix E.2
3.3 Lincarized subspace V step
Here we ask the following question: Can we modify the PV method so as to force the V step to make larger update? In other words, we need optimization algorithm that updates quantized weights either by a sufficiently large increment, or not at all.
A natural example of such an algorithm is coordinate descent (CD)
58 or more generally, subspace descent 266 38 Instead of updating all parameters by small margin, CD in each iteration chooses single parameter, and makes a large update instead. This strategy can be generalized to updating more parameters at the same time, which leads to subspace descent methods 0 The parameters to be updated can be chosen either greedily, (e.g., several i E [d] with the largest magnitude of the partial derivative [v)((-))) or at random, or through a variety of other means.
Let S* c [d] be the set of paraaneters//eiights/oooodddddde choose =5 < d. Let z* _d + d be the linear mapping defined as follows: (Z+)x,, r if i S and (z)), = 0 if & S We now formulate the linearized subspace step:
we wish to update at iteration k. We
x+
M Mv sn (v) == arrg min ZERd {o),s(())
V(x) SV()
(1- (1-
where
(8)
LI
and Lgt > is smoothness parameter of d associated with the subspace spanned by the parameters belonging to Sk This detail is important because Lg* < L when T & d. When estimating Lipschitz constants for real LLMs, we found that it is lower by at least one order of magnitude, making it possible to train with sufficiently large step sizes (see details and Lga estimates in Appendix FF.
Note that, necessarily, x² = y; for i & She The remaining entries of x+ can be identified exactly by searching a discrete space size [V())",, which feasible if (()) and O(),, for example.
In practice, it means that the algorithm can apply large updates to quantized LLM weights, with the caveat that should only update a fraction of them time. This allows to perform the linearized step with sufficiently large "learning rate" to make non-trivial (i.e., b+1 * y² improvements to quantized weights even without straight-throus estimation or stochastic rounding.
We formulate the full procedure in Algorithm 2 The algorithm performs the P step by directly optimizing (xx) (i.e., codebooks) by backprop as described in Section 3.1 For the V step, the algorithm greedily chooses a subset of 7 quantized weights for update, then updates them using Eq. (8) The arg top 7 operator finds 7 indices with the largest absolute gradient values and builds subspace of R where only these values can be changed, and the rest must be equal to y²
'Very closely related methods include block coordinate descent and compressed gradient descent with sparsification operators such RandK TopK 77.

---

3,4 Implementation details
To speed up convergence, we use adaptive learning rates for both P and V steps. In Eq. g we replace Voly) with a single Adam [37 update, as depicted Algorithm 3 preliminary experiments, we found that this results in a significant convergence speedup. When choosing the subspace S we select weights based not on [v)(()),, but on the magnitude of Adam update for that weight. For simplicity, we greedily choose the weights with the largest update norm within each weight matrix.
This could be further improved through better techniques for choosing S* explored in Appendix Q We also found that, despite the fact that PV-tuning by itself outperforms ttraight-througe estimation, we could achieve slightly better accuracy by combining PV-tuning with straight-through estimation. We explore this in more detail in Section
We describe our approach for preparing the calibration data in Appendix G
We found that the prepro- cessing used in several recent PTQ works introduce a small bias when sampling the calibration data, leading to somewhat worse fine-tuning accuracy. For faireess, we always compare representations (Section 4.1 and algorithms (Section 4.2) using the same pre-processing.
part of PV tuning is computing the gradients Vo(-), which is done through repeated forward and backward passes on an LLM. To reduce the number of gradient accumulations, we reuse gradients for P and V steps within one iteration. We use mixed precision, gradient checkpointing and batch accumulation to train more efficiently; for larger L.MMs such as LLAMA 3 70B we also use sharding and optimizer offloading (see Appendix H Our code can train 7B LLMs on a single GPU, while larger ones (e.g. 70B) fit into single machine with 8 A 100. In terms of wall-clock time, PV-tuning takes up to 1.5x longer than the fine-tuning procedure of 71 and requires additional memory in order to hold Vo(r).
Fine-tuning efficiency. The most
Experiments
4.1 Evaluating quantized representations with finetuning
Before evaluating PV-tuning, we need to choose the quantized representation to be fine-tuned. We therefore compare popular weight representations from recent works on LLM quantization (see Section 2 To better isolate the effect of the weight representation, we evaluate them in three configurations: i) when quantizing single LLM layer, in terms of MSE, ii) full model quantization in terms of perplexity without finetuning and iii) with finetuning.
We compare several recently proposed quantized representations (see details in Appendix
1. GPTQ: scalar uniform quantization with channel-wist and block-wise scales 22
2. SpQR: an extension of block-wise GPTQ with learned sparse outliers [8].
3. VQ: basic vector quantization with single codebook 722 with multi-step training.
4. AQLM: additive vector quantization with multiple learned codebooks (21)
5. QuIP#: vector quantization with lattices and incoherence processing 7T).
6. VQ/AQ + outliers:
vector/additive quantization with sparse outliers via pruning 666
g
7. VQ/AQ + lowrank:
vector/additive quantization with Low-Rank Compensation (LoRC)
79
0.030
AQUM
Vi
6.6
VU/AQ outhor
VUrAQ
6.2
Vigia
VQLAQ ootioa
0.025
GPTO
Pemlesity on Wiletent2
QulPP
VO
Perpiciity Wikitest2
6.0
VQ/1Q
lownak
SPQR
6.2
5.8
0.015
6.0
0.010
3.6
3.6
0.005
5.4
2
2.00
2.25
2.50
2.75
3.00
2.00
2.25
2.50
2.75
3.00
wbits
wbits
whits
Figure 2: (left) L2 errors for 17th layer of LLAMA 2 7B with different representations. Full model perplexity on WikiText-2 is reported without finetuning (middle) and with fine-tuning (right).
7

---

We run all three experiments LLAMA 2 7B model [69], calibrating on the RedPajama [3 dataset that best approximates the original pre-training data. When evaluating single layer errors. we report the I.2 error in attention query projection outputs of a fixed transformer block, with other blocks exhibiting similar behavior. For full model evaluation, we report quantized model perplexity on WikiTexx-2 [45] dataset. We use the same data splits and preprocessing as in most recent PTQ works 22]122 18 70 21[ including the biased preprocessing step that we mentioned in Section For fine-tuning, we train continuous parameters only, using the approach from [71]. To compare these diverse representations, we evaluate their quantization errors as a function of average number of bits per parameter. To get diverse set of bits per parameter, vary the hyperparameters such as wbits, block size, codebook and group size for vector quantization and the rate of outliers.
Figure summarizes our findings. Overall, vector quantization methods (VQ. QulP# and AQLM) outperform their scalar counterparts. Outliers and low-rank compensation both reduce error, but this improvement comes at the cost of extra bits per parameter. Interestingly, the improvement from outierss is significantly smaller when both methods have access to fine-tuning. Likewise, the improvement from using low-rank adapters also diminishes when comparing fine-tuned models, to point where no longer justifies the increase in model size. We provide a more detailed breakdown of results and hyperparameter configurations in Appendix !
Our main takeaway is that for sub 2 bits per parameter, the vector quantization (VQ) representation can achieve near-optimal quantization accuracy, whether or not it uses outliers, LoRC or incoherence processing. Naturally, this does not reduce the value of prior works since they were designed for different scenarios, typically with a higher number of bits per parameter.
4.2 Evaluating Fine-tuning Algorithms
Next, we compare different funeng strategies and ablate our PV-tuning protocol. We design our protocol to be sntessennttiionggnottte i.e. compatible with different quantized representations. To showcase this, we pick three methods from the previous section: GPTQ, VQ and AQLM.
These methods differ not only in their weight representations, but also in how they search for the optimal codes. Namely, GPTQ can scale the target weight and round it to nearest 2-bit integer. In turn, VQ quantizes weights as a group and must find the nearest vector from its codebook, and AQLM uses a multi-step beam search procedure to choose the best combination of codes from both codebooks. Our PV-Tuning implementation uses these search algorithms during the subspace linearized V step (find hind_eeerestttt in Alg. 3 We describe the full PV configuration for each method in Appendix K
We compare PV tuning against several popular fine-tuning regimens found in the literature. Our first baseline is fine-tuning only continuous parameters, e.g., codebooks or input/output embeddings 771 74]. The second baseline is training with Straight Through Estimation (STE) 775 77). We also test stochastic rounding as described in Appendix [..2] Finally, we evaluate PV tuning combined with STE, but otherwise the same configuration. We set the subspace size 7 equal to the number of weights such that the update satisfies || 11/
| < 0.01, also known as known as trust ratio [81].
The results in Table show that PV-Tuning consistently finds better quantized models, with STE coming consistently second. We explore this further by combining subspace updates with STE, which leads to slightly better perplexity and accuracy in most (but not all) setups.
Table 1: Comparing different fine-tuning strategies for VQ, GPTQ and AQLM on LLAMA 7B in terms of perplexity on WikiText-2, C4 and average zero-shot accuracy on tasks from Section 4.3
GPTQ 214 bisww
VQ, 1.58 bit'
AQLM, 2 bith
Fine-tuning Method
Wiki2
C41
Acc.t
Wikiz|
C4
Acc.t
Wiki2,
C4
Acc.t
Calibration only (no global finc-tuning)
3290
4125
29.0
20.26
20.09
43.42
7.38
9.34
53.2
Continuous params only 7
21
16.77
17.53
46.27
8.17
10.99
52.14
6.69
8.77
56.57
Naive Linearized PV (no subspace)
16.73
17.48
47.68
8.19
10.94
52.08
6.68
8.75
56.51
Stochastic Rounding X (tuned)
11.97
13.07
49.79
8.02
10.64
52.31
6.56
8.39
56.68
Straight Through Estimation 777
8.79
11.04
50.61
7.76
10.26
$2.58
6.41
8.63
57.04
Subspace Linearized PV (ours, 7=0.01)
8.49
10.78
52.17
7.38
9.47
53.36
6.13
8.35
57.81
Subspace Linearized PV+STE (T=0.01)
8.43
10.82
51.90
7.32
9.35
55.22
5.90
7.43
58.19
8

---

PV-Tuning over QulP# In addition to these three configurations, we also apply PV-tuning to QulP# 77] modification vector quantization that applies Randomized Hadamard Transform (RHT) before quantization and uses fixed lattices instead of learned codebooks. We experiment with 1lama-2 7B model quantized with QulP# to bits per weight and found that possible to significantly improve the model through PV-Tuning. For instance, PV-tuning improves WikiText-2 perplexity from 6.19 (QulP# with built-in continuous fine-tuning) to 5.71 (PV-Tuning + STE). Since original 16-bit model has a perplexity of 5.13, this corresponds to almost halving the quantization error terms perplexity. We report additional details for QulP# with PV-Tuning and full evaluation results in Appendix and include it to Table
"QuIP#+PVV".
On the choice of hyperparameters for 1-bit vector quantization. There are several possible hyperparameter configurations for vector quantization (VQ) that fall into 1-1.1 bit range. One can either use larger codebooks for longer groups (vectors), or smaller codebooks for shorter groups accordingly. In our main evaluations, we quantized vectors of 16 consecutive weights with 14-16 bit codebooks fit into the desired bitwidth. However, we later found that it is more advantageous to choose smaller groups as well as codebooks. We found that 8-bit code per 8 weights outperforms 14-bit code per 16 weights despite having near-identical bitwidth (due smaller codebooks). We report this configuration as "PV (gs8)" in Table 2 and provide additional experiments in Appendix M
4.3 Large-scale Evaluation & Discussion
Finally, evaluate the resulting PV algorithm with vector quantization backbone and KI. objective on a range of popular LLM models. For this section, our goal to evaluate our approach holistically for different models and target bit-widths, comparing against the best known baselines in common settings. To that end, we evaluate on LLAMA 2 & 3 [69], MISTRAL 7B (34] and PHI1- Mini-4k- Instruct at 1-2.5 bits per parameter (averaged over all transformer layers).
We report perplexity on WikiText-2 45 and C4 [54] validation sets, zero-shot accuracy on Wino- Grande [60], PiQA [67]. HellaSwag, [83) ARC-easy and ARC-challenge [12] via the LM Eval
Harness 24 We follow the exact evaluation setup from GPTQ [22]. We compare against QuIP (70), BiLLM 32. PB-LLM [62]. DB-LLM [TO), AQLM [2I). OneBit (77), QuIP# [7I]. the latter three
using fine-tuning. For LLAMA 3, we use baselines from 33 and re-evaluate perplexity in our setup.
Table 2: Quantized model perplexity on WikiText-2., 45 & C4 [4 and the Average accuracy on
5 zero-shot tasks [4] for various models and bitwidths. Arrows t mean higher lower is better.
Size Method
Avg bits Wiki2] C41
Averaget
Size
Method
Avg bits Wiki2, C4
Averaget
LLAMA model family
LLAMA model family
16
5.12
6.63
64,80
5.54
7.10
68.61
BiLLM
1.08
32.48
40.52
41.68
BiLLM
1.1
28.8
257
37.90
OneBit
1.01
9.73
11.11
50.06
PB-LLM
1.7
35.68
197.56
36.00
PV-Tuning
1.02
8.28
10.37
50.66
PV-Tuning
1.01
11.17
11.67
50.01
PV (gs8)
1.00
7.62
9.73
53.77
QulP
2.01
76.95
98.47
36.8
AQLM
2.02
6,64
8,56
56.47
PB-LLM
2.00
21.74
61.04
38.80
QuIP#
2.01
6.19
8.16
57.51
DB-LLM
2.01
12.77
14.82
51.8
DB-LLM
2.01
7.23
9.62
55.12
PV-Tuning
2.01
6.99
8.29
64.36
PV-Tuning
2.02
5.84
7.62
61.35
16
2.59
5.78
75.37
QulP#+PV
2.01
5.71
7.51
61.81
BiLLM
1.1
15.26
65.07
44.2
16
4.57
6.05
67.82
PV-Tuning
1.01
8.67
9.68
51.47
70B
AQLM
1.97
5.65
7.51
60.59
QuIP
2.00
11.63
18.54
48.71
QulP#
2.01
5.35
7.20
61.45
13B
PB-LLM
10.33
28.89
46.04
DB-LLM
2.01
6.19
8.38
59.41
PV-Tuning
2.07
4.57
6.56
70.38
PV-Tuning
1.97
5.12
6.83
64.92
PV-Tuning
2.19
5.05
6.74
66.05
MISTRAL 7B v (A) and PHI 3 Mini-4k Instruct (B)
16
3.12
4,97
72.40
4.78
5,71
69.38
AQLM
2.07
3.94
5.72
68.75
7B
QulP# PV-Tuning
2.01
6.02
6.84
62.20
(4)
70B
QuIP#
2.01
3.91
5.71
68.60
2.01
5.29
6.17
66.32
DB-LLM
2.01
4.64
6.77
65.83
5.83
9.35
70.5
PV-Tuning
2.07
3.78
5.56
70.72
3.8B
AQLM PV-Tuning
2.03
8.85
12.19
60.4
(B)
PV-Tuning
1.14
5.52
7.50
64.58
2.03
6.88
10.08
65.70

---

Table 2 summarizes our findings: PV-tuning with vector quantization outperforms all known methods for 1- and 2-bit per weight. The closest competitors on LLAMA are QulPH, AQLM and OneBit, all of which use fine-tuning. The improvements on LLAMA 3 are also remarkable as this model notoriously hard to compress 33
We report additional evaluations in Appendix N
Pareto-optimality. A key practical question concerns obtaining optimal quality for the target model size, where a smaller model compressed to 3-4 bits often dominates larger model compressed to 1-bit. The best known Pareto-optimal bit-width for Llama 2 is 2.5 [2]] compressing a larger model to less than 2.5 bits per weight is inferior to smaller model quantized to the same total number of bytes. From this perspective, PV-tuning pushes the Pareto-optimal frontier for LLAMA 2 to 2.0 bits. This is easiest to see in Table 12 2-bit 13B model outperforms any 7B quantization and is comparable with the 16-bit 7B model. The same holds for the 2-bit 70B model.
Fine-tuning efficiency. One limitation of our algorithm is that it requires more compute and memory during the fine-tuning procedure. The 7B models can be fine-tuned on a single GPU, our 70B runs require server with 8x A100 or rely on RAM offloading. PV-Tuning shares this drawback with prior methods based on STE (21 as both methods need gradients w.r.t. dequantized weights. Our longest training run took 2 days 8 GPUs to outperform all baselines and 8 days to fully converge.
Inference speed. PV-Tuning does not change the underlying compressed representation, allowing us reuse existing segh-performance inference kernels. Specifically, VQ+PV can reuse efficient kernels from (1 while GPTQ+PV can use ExLlamaV2 kernels [14). We report these inference speed evaluations in Appendix O From a practitioner's s point of view, PV-Tuning can significantly improve the accuracy of extreme (1-2 bit) quantized models, making it possible to deploy large L.LMs on cssstrce-onstrriine devices. As proof of concept, developed specialized inference engines for running vector-quantized models with PV-Tuning on mobile devices or in the browser"
5 Conclusions
Limitations. We focused our effort on evaluating PV-Tuning with multiple setups and models, but spent relatively little effort tuning our algorithm for each specific setup. For instance, we always use constant learning rate and T with no schedule, and always train on the same data. While this shows robustness of PV-Tuning, it also means that our results may be improved with better hyperparameters. For instance, Appendix M shows how PV-Tuning with 1-bit vector quantization can be improved by choosing smaller vector sizes, while Appendix L suggests that PV-Tuming can dramatically improve models quantized with QulP# and may similarly be applied to other quantized representations. Furthermore, the algorithm could achieve better accuracy by simply training longer and on more data.
Future work. This work opens several new research directions. The first is about how to choose Sk: while we found that greedy strategy works in practice, there may be fundamentally better ways. Another direction is applying PV-Tuning to other quantization niches: our evaluation focuses on extreme weight-only quantization, but the proposed algorithm can be used in weight + activation setting or KV cache quantization. Overall, PV-Tuning shows how an insight from optimization theory can improve LLM quantization and we excited to see how this develops in future research.
A Acknowledgements
Authors would like to thank Vage Egiazarian, Andrei Panferov and Ruslan Svirschevski for their help and advice on AQLM codehase and running large-scale experiments. We also thank Philip 7mushko and Artem Fedorov for helpful discussions during the early stages of our research. The research of Kai Yi, Konstantin Burlachenko, and Peter Richtárik reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940. We would also like to thank our NeurIPS reviewers for their helpful suggestions, smaller codebook sizes and evaluate PV-Tuning with QulPH, both of which produced interesting findings. Finally, thank the open-source contributors from llama.cpp" and the LocalLlams community for discussions and inspirations on practical use cases of quantized language models, and in particular, Yalda Shabanzadeh and Arthur Aardvark for their help with improving the codebase.
specifically highlight p3Lv's suggestions to consider
httpes//gihhbb.com/ggeggg/////// httmt.htmulttllt........m.mmm
httg://reddd...nm///ocaaaaaa
10

---

References
[1] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree,
A. Bakhtiari, H. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, C. C. T. Mendes, W. Chen, V. Chaudhary, Chopra, A. D. Giorno, G. de Rosa, M. Dixon, R. Eldan, D. Iter,
A. Garg, A. Goswami, S. Gunasekar, E. Haider, Hao, R. J. Hewett, J. Huynh, M. Javaheripi,
X. Jin, P. Kauffmann, N. Karampatziakis, D. Kim, M. Khademi, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, C. Liang. W. Liu, E. Lin, Z. Lin, P. Madan, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker T. Portet, R. Pryzant, H. Qin, M. Radmilac, C. Rosset, S. Roy, Saied, Salim, M. Santacroce, S. Shah, N. Shang. H. Sharma, X. Song, M. Tanaka, X. Wang, R. Ward, G. Wang, P. Witte, M. Wyatt, C. Xu, J. Xu, S. Yadav, Yang, Z. Yang,
O. Ruwase, O. Saarikivi,
Yu, C. Zhang, Zhang. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
[2] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. Qsgd: Communication-efficinn sgd via gradient quantization and encoding. Advances neural information processing systems, 30, 2017.
[3] D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Randomized quantization for cummunication-cffiicee stochastic gradient descent. In Conference on Neural Information Processing Systems (Neur/PS), 2017.
[4] D. Alistarh, Hoefler, M. Johansson, Khirirat, N. Konstantinov, and C. Renggli. The convergence of sparsified gradient methods. In Advances in Neural Information Processing Systems, 2018.
[5] A. Babenko and V. Lempitsky. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 931-938, 2014.
[6] Y. Bengio, N. Léonard, and A. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint /308.3333 2013.
[7] A. Beznosikov, S. Horváth, P. Richtárik, and M. Safaryan. On biased compression for distributed learning. wrXXiv20022.241000 2020.
[8] V. Boža. Fast and optimal weight update for pruned large language models, 2024.
[9] J. Chee, Y. Cai, V. Kuleshov, and C. D. Sa. Quip: 2-bit quantization of large language models with guarantees, 2023.
[10] H. Chen, C. Lv, L. Ding, H. Qin, X. Zhou, Y. Ding, X. Liu, M. Zhang, J. Guo, X. Liu, and D. Tao. Db-llm: Accurate dual-binarization for efficient llms, 2024.
[11] S. Chen, W. Wang, and S. J. Pan. Metaquant: Learning to quantize by learning to penetrate non-differentiable quantization. Advances in Neural Information Processing Systems, 32, 2019.
[12] Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 10335478 2018.
[13] T. Computer. Redpajama: open dataset for training large language models, October 2023.
[14] E. contributors. ExLlamaV2. fast inference library for running LLMs locally on modern consumer-class GPUs. Open-source library developed by turboderp and controbutors.
[15] M. Courbariaux, Y. Bengio, and J.-P. David. Training deep neural networks with low precision multiplications. arXiv preprint arxiv: 442220244 2014.
[16] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22, 1977.
11

---

[17] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, 2022.
[18] T. Dettmers, R. Svirschevski, V. Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh. Spqr: A sparse-quantized representation for near- lossless Ilm weight compression. arXiv preprint arX1iv:330600307. 2023.
[19] T. Dettmers and von Koeller. Accessible large language models via k-bit quantization for pytorch.
[20] T. Dettmers and L. Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws, arXiv preprint arXiv: 2022.
[21] V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh. Ex- treme compression of large language models via additive quantization. arXiv preprint Stti::2401.661166 2024.
[22] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq:
Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.. 17323, 2022.
[23] Y. Freund and R. E. Schapire. Large margin classification using the perceptron algorithm. In Proceedings of the eleventh annual conference Computational learning theory, pages 209-217, 1998.
[24] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang. K. Wang, and Zou, A framework for few-shot language model evaluation, Sept. 2021.
[25] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, A survey of quantization methods for efficient neural network inference. arXiv preprint M.Kiv:2203.136633 2021.
[26] R. M. Gower and P. Richtárik. Stochastic dual ascent for solving linear systems. 15X1v:1512.06890, 2015.
[27] A. Griewank and A. Walther. Algorithm 799: revolve: an implementation of checkpoint- ing for the reverse or adjoint mode of computational differentiation. ACM Transactions on Mathematical Software (TOMS), 26(1):19 45, 2000.
[28] H. Guo, P. Greengard, E. P. Xing, and Y. Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning, 2024.
[29] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayana.. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737-1746.
PMLR, 2015.
[30] K. Helwegen, Widdicombe, L. Geiger, Z. Liu, K.-T. Cheng, and R. Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 32, 2019.
[31] G. Hinton. Neural networks for machine learning, coursera (video lectures).. 2012.
[32] W. Huang, Y. Liu, H. Qin, Y. Li, S. Zhang, X. Liu, M. Magno, and X. Qi. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arX1v:2402.04291, 2024.
[33] W. Huang, X. Ma, H. Qin, X. Zheng, C. Lv, H. Chen, J. Luo, X. Qi, X. Liu, and M. Magno. How good are low-bit quantized Ilama3 models? empirical study, 2024.
[34] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. Casas, Bressand.
G. Lengyel, G. Lample. L. Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825,, 2023.
[35] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. I. Casas, E. B. Hanna, Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401.0444 040888 2024.
12

---

[36] S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer. Squeezellm: Dense-and-spare. quantization. arXiv preprint arXiv:2306.07629 2023.
[37] D. P. Kingma and Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
[38] D. Kozak, S. Becker. A. Doostan, and L. Tenorio. Stochastic subspace descent. arXiv preprint arXiv: 1994000110 145, 2019.
[39] C. Lee, Jin, Kim, H. Kim, and E. Park. Owq: efficient fine-tuning and inference of large language models, 2024.
Outlier-aware weight quantization for
[40] S. Li, Y. Zhao, R. Varma, O. Salpekar, P. Noordhuis, T. Li, A. Paszke, J. Smith, B. Vaughan, Damania, and S. Chintala. Pytorch distributed: Experiences on accelerating data parallel training. 2020.
[41] Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.054:: 05463, 2023.
[42] Lin, Tang, H. Tang, Yang, X. Dang, and S. Han. Awq: Activation-awarg weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00974 2023.
[43] Z.-Q. Luo and P. Tseng. On the convergence of the coordinate descent method for convex differentiable minimization. Journal of Optimization Theory and Applications, 72(1):7-35, 1992.
[44] S. Ma, H. Wang. L. Ma, L. Wang. W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024.
[45] S. Merity, C. Xiong, Bradbury, and R. Socher. Pointer sentinel mixture models, arXiv preprint arXiv: 1609.07843, 2016.
[46] X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, R. Y. Y. Wong, Z. Chen, D. Arfeen, R. Abhyankar, and Z. Jia. Specinfer: Accelerating generative Ilm serving with speculative inference and token tree verification, 2023.
[47] R. Mises and H. Pollaczek- Geiringgr.. Praktische verfahren der gleichungsauflisung. ZAMM- Journal of Applied Mathematics and Mechanics/Zeitschiiff für Angewandte Mathematik und Mechanik, 91))588-77, 1929.
[48] J. Mockus. On bayesian methods for seeking the extremum. In Optimization Techniques, 1974.
[49] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
[50] G. Park, B. Park, S. J. Kwon, B. Kim, Y. Lee, and D. Lee. nuQmm: Quantized matmul for efficient inference turge-scale generative language models. arXiv preprint arXiv:2206.09557, 2022.
[51] J. Park and E. K. Ryu. Exact optimal accelerated complexity for fixed- point iterations. In Proceedings of the 39th International Conference on Machine Learning, 2022.
[52] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Veto,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), Neural Information Processing Systems Foundation, 2019.
[53] A. Polino, R. Pascanu, and D. Alistarh. Model compression via distillation and quantization. 2018.
arXiv preprint
[54] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2161 2020.
13

---

[55] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 220. IEFE Press, 2020.
[56] I. Rechenberg. Evolutionsstrategie:: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Problemata (Stuttgart).
1973.
[57] Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He. Zero-offload: Democratizing billion-scale model training, 2021.
[58] P. Richtárik and M. Takáč. Parallel coordinate descent methods for big data optimization. Mathematical Programming, 156(1-2):433-488 2016.
[59] Rosenblatt. The perceptron a perceiving and recognizing automaton. Technical Report 85-460-1, Cornell Aeronautical Laboratory, Ithaca, New York, January 1957.
[60] K. Sakaguchi, R. L.. Bras, C. Bhagavatula, and Y. Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, 2021.
[61] T. L. Seao, A. Fan, C. Akiki, E. Pavlick, Ilié, D. Hesslow, R. Castagné, A. S. Luccioni. F. Yvon, M. Gallé, et al. Bloom: A 776b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05/00 2022.
[62] Y. Shang, Z. Yuan, Q. Wu, and Dong. Pb-llm: Partially binarized large language models, 2023.
[63] W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y. Qiao, and P. Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
[64] A. Shekhovtsov and V. Yanush. Reintroducing traight-through estimators as principled methods for stochastic binary networks. In DAGM German Conference on Pattern Recognition, pages 111-126. Springer, 2021.
[65] M. Spallanzani, G. P. Leonardi, and L. Benini. Training quantised neural networks with ste variants: the additive noise annealing algorithm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 470-479, 2022.
[66] M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models, 2024.
[67] S. Tata and M. Patel. PiQA: An algebra for querying protein data sets. In International Conference Scientific and Statistical Database Management, 2003.
[68] TH UAE. The Falcon family of large language models. http::://higginggaccc./////// falcon-40b May 2023.
[69] H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1 13971, 2023.
[70] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. De Sa. Quipt: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 2rXiv:2402.04336, 2024.
[71] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. D. Sa. Quip#:: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024.
[72] M. van Baalen, A. Kuzmin, M. Nagel, P. Couperus, C. Bastoul, E. Mahurin, T. Blankevoort, and P. Whatmough. Gptvq: The blessing of dimensionality for Ilm quantization. arXiv preprint ar0i 15319, 2024.
[73] A. van den Oord, O. Vinyals, and k. kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
14

---

[83] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? In A. Korhonen, D. R. Traum, and L. Marquez, editors, Proceedings of the 57th Conference the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 4791-4888 Association for [74] H. Vanholder. Efficient inference with TensorRT. NVIDIA GTC On-Demand Slides avail- able at
efficientt-ifeeeeee--iiiitttttss 2017.
[75] H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023.
[76] B. Widrow and M. A. Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proc. IEEE, 78: 78:1411-1442, 1990.
[77] Y. Xu, X. Han, Z. Yang, S. Wang, Q. Zhu, Z. Liu, W. Liu, and W. Che. Onebit: Towards extremely low-bit large language models, 2024.
[78] Yao, Y. Aminabadi, M. Zhang, Wu, C. Li, and Y. He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint vrXiv:2206.0/8// 2022.
[79] Z. Yao, X. Wu, C. Li, S. Youn, and Y. He. Exploring post-training quantization in ilms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19377-19385, 2024.
[80] Yin, J. Lyu, S. Zhang, Osher, Y. Qi, and Xin. Understanding straight-through estimator 0,5662 2019.
in training activation quantized neural nets. arXiv preprint
[81] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
[82] P. Zamirai, J. Zhang, C. R. Aberger, and C. De Sa. Revisiting 6 training. 2020.
Computational Linguistics, 2019.
[84] C. Zhang, Cheng, G. A. Constantinides, and Y. Zhao. Lqer: Low-rank quantization error reconstruction for 1lms, 2024.
[85] P. Zhang, G. Zeng, T. Wang, and W. Lu. Tinyllama: An open-source small language model, 2024.
[86] Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, Chen, C. Dewan, M. Diab, X. Li, X. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXi:22005.01668 2022.
[87] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1606v:1606.06/1 2016.
15

---

Appendix
Contents
Introduction
Background
3
Fine-Tuning Quantized Models
Problem description)
3.2
Linearized V step & gradient-based discrete updates
3.3
Linearized subspace V step
3.4
Implementation details
Ex Experiments
Evaluating quantized representations with finetuning
4.2
Evaluating Fine-tuning Algorithms
8
Large-scale Evaluation Discussion
5
Conclusions
10
Proofs
18
A.I Proof of Theorem 3.1
18
Proof of Lemmal
18
A
3 Proof of Lemma 3.
18
Approximate PV Algorithm
19
B. Approximate V step, variant (non-accelerated)
19
B.2 Approximate step, variant 2 (accelerated)
19
Generalization to Other Quantization Algorithms
20
Efficient Linearized V Step for Vector Quantization
20
Gradient-based Strategies for Training Quantized Models
21
E.
Straight-through Gradient Estimation
22
Stochastic Rounding
................................
22
Comparing Discrete Optimization Techniques
23
L-smoothness LLM Objective in Sparse Subspaces
24
Calibration Data Matters
26
Additional Engineering Considerations
27
16

---

Additional evaluations of perplexity
28
Additional Details for Section 4.1
28
Additional Details for Section 4.2
30
PV-Tuning of QulP#
M On 1-bit Vector Quantization Options
31
Additional Details and Evaluations for Section 4.3
32
Inference Speed with Vector Quantization Kernels
32
The Choice of the Initial Point o
P.I Clipping of x²'
35
P.2 Random σ0 R
35
Q
Small-Scale Experiments and Interpretation
35
Q.1
Objective function
35
Q.2
Tiny-scale experiments (d E 1,
................................
36
Q.3
Interpretation of P(v) 3 P(**)
36
Q.4
Simple example of P(y) P(x*)
37
Q.5
Small-scale experiments (d 100, c E 1, 100)
37
Q.6
Lincarized PV
38
Linearized PV sparse updates
39
R PV Algorithm
40
Broader Impact
17

---

A Proofs
A.1 Proof of Theorem 3.1
Part (i): First, by assumption, know that x0 BR Assume that x.th R for some k 2 0. Since P()") 2 P(**), this implies that y Rg Next, since (x S V(y"), we conclude that g E B The claim now follows by induction.
Part (ii): Since
y* = arg min (Ø(v) : P(y) 2 P(x*)} yERd
and because = xh satisfies the constraint P(y) 2 P(**), we conclude that d(y s o(x*).
Further, since
arg min fo(x) V(z) S V(*))
ZER4
and because x = yh satisfies the constraint (yy) C V Vyy"), ) we conclude that ((**+1) - ((x²). In summary,
)(I S <o() S 6(x*).
Part (iii): In view of part (ii), the sequence ) kmm is non-increasing. By assumption, it is bounded below. Hence, it converges to its infimum:
lim ((x*) inf 1-0000
p(x*).
AE(U,
A.2 Proof of Lemma 3.2
Mv.5, (y) = arg min zERd
(5(x)) : V(x) v(x) [(vl
Φ(v) (Voly), x yt + - vl v(o
arg min
zExe
(V)(y), + 1 yl v(x) [ww)
arg min Rd
2 Volu). x/) + || +110-111 -11 v(x) [wo)
arg min zERd
{|
| (w- (v- 1, = V((x) V(o)cvw)
arg min zERd
T.
(2)(4) ((x) [v
arg min zERd Se
My My,5, (v).
A.3
Proof of Lemma 3.3
First, note that
((t)
(Mrra (v)
Lemia 3.2
① (M (2)
min (2)(2) V(x) C V(y)
#ERd
min ((0) + (Vo(g), x - v\ + 1| -01
w1 v()ssw)) (9)
RERd
18

---

Since R any I E R' satisfying v(z) S (vv) must also satisfy a E Bg So, in view of L-smoothness of e Ind So we can bound the last expression in ( from below via
i
min {d(x) V(x) S V())
zER
(My v (y))
Finally, since x = satisfies the constraint ((x) C Vyy), we can upper bound the same expression via
o(t) o(v) + (Vo(y). v) L 1l vll Q(y).
2
B Approximate PV Algorithm
We now introduce the pseudocode of an approximate PV metu-algorithm; the idea is to replace the and V steps with some approximate computations to be defined later.
Algorithm 4 Approximate PV Algorithm
1: Parameters: starting point x² E Rdd
2: for k 0, 1...... do
3: y" = (r.
4: x++1 = My.a(v (for example, we can use the method from Section B.1 or the method from
Section B.2
5: end for
Next, we describe two new approximations of the V step.
B.1 Approximate V step, variant 1 (non-accelerated)
We now describe an algorithm computing an approximation to My.a(y):
1. Start with some y E Rd and choose sufficiently large I. > 0, number of iterations
2. Set 20 y
3. For / O,...,T T iterate:
(i) Define (.) == ||| -1' (2==+Vo(=')
(ii) Set 20+1 Mv. (z²)
4. Output: 2"
The method is constructed so that 2T = My. If use this subroutine with T = to approximate the step PV method, we recover what we earlier called the linearized PV method. Choosing sufficiently large T 2 2 may be advantageous.
B.2 Approximate V step, variant 2 (accelerated)
We now describe different algorithm for computing an approximation My)s(y):
1. Start with some y R and choose sufficiently large L > number of iterations T
2. Choose a suitable decreasing sequence of positive scalars {a,), with og = 1 and
lim Ct: =
3. Set 2 y/
4. For t O,..., T - iterate:
19

---

(i) Define (+)()) := ||.- vV
iii) Set 2++1 = (1 -0z) Mv vv. (z²) + a2z"
5. Output: z²
The method constructed so that 2 (y). This approach is based on Halpem acceleration of fixed point methods [51] and as such, may be sometimes effective.
C
Generalization to Other Quantization Algorithms
we define R set of vectors with most unique items. This translates to the idea of k-means quantization, a scalar nonlinear quantization where each weight is rounded to one of centroids found by clustering the weights. Below, show how this can be generalized to linear quantization, vector quantization, additive quantization, and others.
In Section 3.1
Linear quantization is the most basic and widely used type of quantization where weights are stored as integers, possibly multiplied by a scale and added to a zero point. The simplest way to account for this quantization type is to declare that weight values are integers up to V(z) (0, 1, 2, .... After that, one can treat scales/ zero points as an extra non-quantized parameter, similar to biases or normalized scales. This extra parameter interacts with weights by multiplication or addition, and hence it can be updated by backprop, similarly to other non non- non-quuntized weights. Equivalently, once can declare that (zx) (0, s, 2s, ..., (c 1)) for arbitrary s E R. Both options lead to equivalent fine-tuning algorithms where the step does not change and the P step has an additional condition.
1).
Next, let us discuss vector quantization. Consider a quantization that splits x into 2-dimensional groups (non-intersccting pairs of adjacent weights) and encodes these weights one of с 2- dimensional codes that form its codebook. This can be viewed as two sets of weights (odd and even) quantized with scalar quantization, except that values in the two sets have the same partitioning P(z). In other words, two values in the odd half belong to the same partition, the corresponding values in the other half also belong to the same partition, though the values themselves can be different. Alter- natively, one can simply write down a version of Re where V--) is a set of 2-dimensional vectors, not scalars. Likewise, nigher-dimensional vector quantization translates to higher-dimensional items in V()).
Both the P and V steps for vector- qusto--quantzedd weights follow the same general principle: P-step can be approximated by backprop with slightly more trainable parameters, In turn, the V step can be done by trying all values in V(x) and selecting the one with the lowest Ø(-). A more compute-efficient version of the V step for this case is described in Appendix D
RVQ and Additive Quantization can be treated as learning two separate sets of vector rector-quantized parameters. However, more efficient way would be to run the V step to find the best combination of codes via beam search [5).
can be seen as learning two distinct matrices with different definitions of R. one quantized and the other is sparse (for outliers). Similarly, quantized weights with low-rank adapters (e.g. (28)) can treat the adapter as an additional -onattantzedd parameter for the step. This makes PV-tuning potentially extensible to neural network pruning.
Quantization with sparse outliers 17
18
12
D Efficient Linearized V Step for Vector Quantization
As we describe in Section 3.2 the linearized V step minimizes the squared error between quantized weights and the updated "target" weights. Here, explain how one compute and minimize the squared error efficiently in practice. To simplify the notation for this section, we define the objective as ||ær BI where B is the target vector set by the linearized V step. Following the definition of squared L2 norm, this objective can be re-written as follows:
|| B11
||
2(xz, B) + ||
(10)
20

---

Consider the first term: ||=|²= Di= x; Since R this term is sum of at most e unique terms. Abusing your notation, this can be rewritten as
||=|| Σ (((x)).1P(()))
where (a( is i-th unique element a and [P(())) the number of such elements.
The second term is also a sum of e unique values:
B; = -2 Σ 2 (x, B) 2 Σ
Vi(x))
Σ (EP)(
B
1=1
1=1
The third term does not depend on x.
If you know the objective for some given æ, you can efficiently compute & for all neighboring E (x) where you only change one index. For the sake of formality, let us define the such neighboring å as follows:
(1)2) = { E V( V(x), || 211o 1}
(11)
Consider one à e (z) where only k-th value changed (i.e. 2g .2). Then,
( )(x) = |||||- - 2 (# x. B) + B 2 ||
(12)
( 6(x) - - 2+x 2k) +0
(13)
Note that, for any Va E Mi (x), there are e² possible values for . ot and another e² unique values for 2- (#t= - 2R) BR, regardless of d, since there are unique values in both and 4. This allows for efficient local search algorithm:
1. Let a be the input to Mp
2. Compute and save all 2c possible red and blue values
3. Compute @n = 6(x))
4. for t= 0......
5. for E (z*)::
6. find k ft # x (there's 's only one such k))
7. compute ((+)= <(x)) + of ot 2 (it xx) B
8. g++1 arg min (
(minimum from array of pre-computed values)
(x*)
In practice, this can be extended from greedy (local) search to semi-greedy beam search. These practical algorithms are described in AQ, LSQ, and LSQ++. Algorithms for ||A(z B) are explained in AQLM and probably other works.
E Gradient-based Strategies for Training Quantized Models
In this section, we overview possible solution to the general problem of training / fine-tuning neural networks with quantized weights. We focus strategies that train by gradient descent with additional measures to deal with graiind weights.
In principle, there are also gradient- gradient-free methods for quantized training, such as Evolution Strate- gies (56), Bayesian Optimization [88 and others. However, these gradient-free methods have so far not gained popularity for large language model quantization. Adapting these methods to the scale and dimensionality of L.LMs would likely require extra effort. Thus, we leave these methods outside the scope of our work and focus on gradient-based optimization.
21

---

gradient-based training of quantized models. We describe the general framework for and 3.2] To summarize, the training algorithm computes the gradient w.r.t. de-quantized weights as though they were continuous, then uses these gradients to update continuous (P step) and discrete (V step) parameters. The core problem with this approach is that, when discrete parameters are very coarse (e.g. low-bit quantization), gradient updates are no longer large enough to make any changes and are lost to the "rounding error". We review two strategies for cireumventing this problem: straight-through estimation and stochastic rounding.
Reminder:
training quantized weights in Sections [.1 and
E.1 Straight-through Gradient Estimation
Straight-through gradient estimation is technique for training neural networks with discrete com- ponents that ignores these discrete components during backpropagation. Its usage goes back to the Perceptron introduced by Rosenblatt (59). There, the artificial neuron uses a step function as activation, but the training procedure treats this function as though it was identity. Subsequent works introduce variations of this idea, extend it to multi-layer networks [31], discuss its convergence properties 776 23 80) Other works use straight-through estimation or similar techniques to training
neural network with quantized weights 30 651
STE for LLM quantization. As we discuss in Section 2 straight- ttraight-through estimation introduces an auxiliary non non-quanin non-quantized weight tensor that is updated using the gradients Vo(y) w.r.t. quantized weights. The quantized weights are then updated to best approximate this auxiliary buffer, usually in terms of T.2 error. As result, if an update y _ Veh(y) is not large enough to change the parameter, it is still accumulated in a straight-through "buffer". Eventually, the cumulative effect of several such updates will be large enough that y² will no longer be the solution to Equation (6)
This strategy prevents the algorithm from stalling, but it does so at the cost of convergence guar- antees [80]. When applied to extreme LLM quantization (Section 4 straight-through estimation initially improves y², but then stops improving and oscillates. We also tried several a variant of straight-through estimation [65] that introduce stochasticity to forward pass. When applied to extreme LLM guantization, this variant did not diverge like naive STE, but trained much slower and did not reach the same optimum "deterministic" STE. We attribute this the fact that adding noise during training can slow down convergence, which also applies to stochastic rounding (Appendix F.2
E.2 Stochastic Rounding
Stochastic (or probabilistic) rounding 82 87 29 is one of the techniques that can circumvent
stalling when training low-precision weights. To recall, the linearized V step (6) can be seen as rounding y* := y Vo(y) to the nearest quantized weight in R. which often happens to be itself. To circumvent the problem of rounding back to y. one can instead round stochastically, to one of the two adjacent values that yt falls between. Let's denote these two adjacent values ii and x, for left and right. The probability of rounding inversely proportional to the rounding error (distance), or, in terms of the objective,
x(x1) -1/2
p(ruud to m1)
(011)-11/ 1/2 + ((I,)-1/22
This way,
1 L V Vo(y).
F
x x=
p(round to
The main drawback of stochastic rounding is introduces noise, it changes the underlying optimization problem. Intuitively, if the optimal x" is adjacent to a significantly worse solution, the method may oscillate between rounding to either side. This rounding noise increases further as consider lower quantization width. In Section 4.2 we exploit his phenomenon for real-world LLMs and find that stochastic rounding converges find significantly worse solutions, presumably because at every step, some portion of LLM weights will be rounded poorly. On top of that, when used for vector quantization, stochastic rounding is either intractable or biased.
Stochastic rounding for vector quantization. To recall, stochastic rounding for non-vector quanti- zation needs to find two quantized values: the nearest neighbor above the solution, and the nearest neighbor below it. It will then round to either of the two values inversely proportionally to their rounding errors.
22

---

However, this intuition no longer works if you consider more complex quantization schemes such as vector quantization, additive quantization, quantized low-rank adapters, and others. In vector quantization, group of weights is encoded jointly as one vector from fixed set (usually called codehook or lattice). For simplicity, let us consider the case where the weight group size equals 2, i.e. weights are quantized in pairs.
For pair of two eights, we can no longer rely on the fact that they have one neighbor from above and one from below. Instead, they may have any number of adjacent "clusters" they can be rounded to. Intuitively, a pair of weights is a point in 2-dimensional that can have neighbors from left, right, top, bottom, and any diagonals. Formally, to determine full list of neighbors, we can run Delaunay triangulation on all vectors from the codebook (or lattice) plus the target vector that needs to be rounded, then find all points that share triangle with the target vector.
Unfortunately, this procedure can be very expensive, especially for higher-dimennional vectors. A popular practical approximation to stochastic rounding for vector quantizations is to find K (e.g. 2) nearest vectors from the codebook, then use the probability formula from scalar stochastic rounding:
prrundd to r,) ((x;) 1 (51z))-1113)
However, unlike the original stochastic rounding, this approximation does not guarantee that
1 Voly).
E p(rund 2.)
24 -
(14)
For (counter)example, if there is a high density of codes on one side of the target, all K (e.g. 2) nearest neighbors will on the same side. As result, this approximate stochastic rounding biased and further changes the optimization result.
Stochastic rounding with temperature. When used for LLM quantization, the main problem with stochastic rounding is that it introduces noise to the training procedure. This is important because modern LLMs 669 68 T typically train without dropout similar noise layers. This is because the dataset is huge and the training suffers not from overfitting, but from not fitting the data enough.
Training with stochastic rounding makes optimization inherently noisy if using dropout, making harder to train. What is worse, low-bitwidth models produce more noise than high-bitwidth due to larger intervals between quantized values. This additional noise makes it difficult for the model to fit the training data tightly. For extreme 1-bit training, we often observed that the training objective (cross-entropy) would increase instead of decreasing due sheer amount of rounding noise.
To combat this issue, we introduce stochastic rounding with temperature т (hyperparameter)):
3(x)) 1/(2r)
p(round 21)
((x)) 1/(2r) + ((x.)
Setting 7 < results the algorithm keeping the original weights more often, which reduces the training noise the cost of slower updates. In Table [ (Section 4.2), try {1, 0.5, 0.1, 0.013 and choose the best result for each setup that involves stochastic rounding. It can be shown to converge as long as annealed, but changes the underlying optimization problem similarly to dropout. In principle, it is possible to gradually anneal 7 during training to make the algorithm unbiased in the limit.
E.3
Comparing Discrete Optimization Techniques
Finally, we compare the two above strategies and our proposed approach from Section
3.3
From the optimization perspective, the most popular variant of straight-through estimation is known to not converge to a stable solution. While, in practice, STE can still significantly improve model quality (see Table T). still a heuristie. In turn, stochastic rounding can be seen as an additional source of noise to stochastic gradient descent which can converge to stable solution when is
23

---

gradually reduced to O. In contrast, subspace PV does not introduce noise or instability and does not require annealing.
The most notable difference of PV-Tuning from STE is that the former tries to update all weights on every step, whereas algorithm only updates chosen subset. We give explanation why updating all weights is problematic at the end of Section 3.2
From the efficiency perspective, training with straight-through estimation requires storing additional set of buffers on device memory to accumulate weight updates. Unlike the quantized weights, these buffers need to be stored in higher precision (half or full) to accumulate smaller updates that would be lost on quantized buffers. As result, straight-through estimation requires additional memory for fine-tuning. Stochastic rounding and subspace PV (w/o STE) do not need these buffers, but they still need to accumulate high precision gradients w.r.t. de-quantized weights and store optimizer statistics for those weights in higher precision. To summarize, all methods require significantly more memory than naive (P-only) fine-tuning, but straight-through estimation has higher memory overhead.
As for the computational overhead, both STE, stochastic rounding and subspace PV introduce additional computations and therefore increase step complexity. Of the three alternatives, the subspace PV algorithm is slightly faster since only runs the discrete optimization on a small portion (subspace) of model weights per step, while stochastic rounding has somewhat higher overhead due to the complicated rounding procedure, especially for vector quantization. However, this overhead is small in practice: most of the LLM training time is spent accumulating the gradients large training batch, which is not affected by any of the three algorithms. In principle, it should be possible to reduce the compute memory overhead both with technical improvements and better optimization algorithms, but we leave this investigation to future work.
F
On L-smoothness of LLM Objective in Sparse Subspaces
The classical definition of L-smoothness for differentiable function f E + ER represented by requirement
[Vf(x) - vf(v) - L| -yll.
Vx, y E Rd.
If function f(x) is twice continuously differentiable, then it is easy to show that function f is L- smooth if and only 2 f())ll - L. If striving to find the minimum value of L then via following the definition, one has to select L max (I))f()))).
ZERd
If the domain of function f(x) is restricted to any subset S C Rd. then the global L smooth constant can only decrease for new function. It can be observed from the fact that AS C Rdd we have
max (V²f(x) */11=l) max (V2f(x) 5/1 ell Vx E Rd. #ES
VER4N0
Sparse sub-spaces satisfy this requirement; therefore, this theoretical observation is valid in this circumstance.
Another observation is that the definition of L smooth constant has a global notion. However, for practical purposes, for the first-order training algorithm what matters is L-smoothness constant for the function f f~x) with the domain restricted the trajectory of iterates only. Unfortunately, the training process iterates follow the prior unknown path in Rd space.
Below we demonstrate two approximate schemas for evaluating L-smoothness constant for function with a domain (artificially) restricted to the trajectory induced by iterates generated by Gradient Descent (GD) for functions f(x) with different subspace sizes and in general with different optimiza- tion trajectories, but with the same start iterate (model). We have performed 10 iterations of GD for training auto-regressive Llama-lite like LLama- 160M [46] and TinyLlame amllammm11 1. 1B
(55) models using
automatic tokenizer from these models. We trained all linear layers in these models. The used step
size for GD is 10-4
Schema I: Estimating L along the trajectory of iterates without capturing local curvature.
24

---

After running GD for 10 iterations the L smooth constant has been estimated along trajectory (2,,,2,.... .... F10l with approximated
[V - | x,y|
L
max Fi.Ty Ea
Results are presented in Tables From them, iterates have the same property as global I., namely during restricting subspace of training variables the L-the smooth constant non-increasing, and in practice substantially decreasing. This schema exploits available information on gradient oracles in iterates in & and iterates s itself. This schema represents an estimation of upper bound L on the true value of L.
that i. estimate along the trajectory of
Table 3: Subspace Size
Estimated L along the GD trajectory for LLama- 160m (Schema I).
Number Trainable Parameters Estimated Í.
5%
2.36M
10.01
10%
8.26M
14.40
20%
17.69M
305.72
30%
24.77M
498.16
40%
36.57M
919.82
60%
60.75M
5454.79
70%
85.52M
6915.06
85%
102.04M
7043.19
100%
113,25M
7251.50
Table 4: Estimated L along the GD trajectory for TinyLlama-1. 1..11111a1111 (Schema I).
1B
Subspace Size
Number of Trainable Parameters
Estimated L
5%
13.11M
33.24
10%
49.28M
143.00
20%
133.84M
2159.22
30%
242.75M
2369.63
40%
369.10M
2638. 11
60%
582,48M
5185.92
70%
684.20M
5901.73
85%
831.52M
6091.04
100%
968.88M
9480.57
Schema II: Estimating along the sequence of iterates with capturing local curvature.
The previous schema used a fixed sequence iterates s = {x1 ... 210} essentially estimate the L-smoothness constant along the piece-wise linear interpolated path along s. In the next schema, approximate T-ssmoothness constant as
max 2 f(x,))l)
Therefore this schema exploits a sequence of points and selects the maximum in absolute values eigenvalue for all matrices fz;( Computing any spectral information for a matrix with big dimensions can be challenging.
The approximate schema that we have used to compute V f())] leverages several observable facts. ( (x)))) where A; identify the maximum eigenvalue in the absolute value we can use the normalized Power Iteration algorithm [47] which requires execution only the hessian-vector product. Third, we use Taylor expansion and forward difference approximation for V fx +):
First, f=))]
th eigenvalue for 4 fxx). Second, to
=
f() O (1r|)3)
25

---

The K approximate hessian-vector product can be accomplished with K + gradient oracle calls. For the experiment, we run Power Iteration for prior known number of iterations equal 10. In fact Power Iteration does not converge in case of degeneracy such situation when the matrix has two maximum eigenvalues in absolute values but with opposite signs, and the convergence rate determined by the absolute value of the ratio of the second-largss--maniite eigenvalue to the first. We ignore these aspects in our heuristic approximate Algorithm [
Algorithm 5 Approximate Matrix-free Algorithm for Computing [
1: Parameters: Point a E IR, fixed 7y E R such as 7 = 10 æ for numerical stability.
2: r ~ua.r Rd
3: g = Vf(x)
4: for k = 0, =0,1.... K do
5: yd = =-=-/111 /110/
ptt1 p = 1/7 (VI( xT 5) // Approximate computation of pkt1 V2( ,
6:
7: end for
8: Output: Approximate eigenvector , * /1///1 corresponding |Amax|
||
Results are presented Tables From them, we can see that also this notion of L. estimate along the set of iterates has the same property as global L, namely during restricting subspace of training variables the L-the smooth constant non-increasing similar to previous estimation method.
Table 5: Estimated L along the GD iterates for LLama- 110m with local curvature (Schema II).
Subspace Size
Number Trainable Parameters
Estimated
5%
2.36M
10.96
10%
8.26M
791.30
20%
17.69M
878.37
30%
24.77M
1202.00
40%
36.57M
1918.04
60%
60.75M
5262.77
70%
85.52M
5325.83
85%
102.04M
5901.21
100%
113.25M
11522.45
Table 6: Estimated L along the GD iterates for TimyLlama-1. 1B with local curvature (Schema II).
Subspace Size
Number of Trainable Parameters
Estimated
5%
13.11M
40.30
10%
49.28M
146.93
20%
136.84M
4366.38
30%
242.75M
4487.38
40%
369.10M
6767.58
60%
582.48M
8983.85
70%
684.20M
15445.54
85%
831.52M
21167.00 .06
100%
968.88M
28629.24
G Calibration Data Matters
For a fair comparison, we our algorithm using the same calibration data as the baseline algorithms, typically sample from RedPajama [33. However, the way we handle this calibration data differs from most baselines 18 71..
26

---

When analyzing their codebase, we found that these algorithms resample calibration data by taking random excerpt fixed length from a random document in the calibration data, both sam- pled uniformly. However, with this approach, the data from longer documents (e.g. books) are underrepresented compared to shorter ones (e.g. news articles), which biases the calibration data.
Upon further investigation, we believe that new methods blindly copied this code from each other, going back GPTQ [22) and possibly further. This choice was harmless for GPTQ since requires relatively little calibration data; however, full model fine-tuning like in QuIP# [9] and AQLM (2]] works better on unbiased data.
To remove the bias, we use standard LM preprocessing that concatenates all documents, then splits them into evenly sized chunks that become training sequences. The benefits from debiasing range from insignificant to as large as 0.15 perplexity for some models. To compensate for that, we run experiments with the same preprocessing protocol unless explicitly stated otherwise.
H
Additional Engineering Considerations
When done naively, the longest operation is the discrete update all LLM parameters. For scalar quantization, this step does simple rounding and runs nearly instantly. In turn, applying
X that runs discrete optimization on
vector quantization requires solving a discrete optimization algorithm (e.g. beam search) for every group of weights. However, since equation (8) can only update weights within S*, we can skip discrete optimization for any weight that was not among the chosen few. As result, when training models with up to 70 billion parameters, search less than one billion times per step.
The next longest operation is computing the gradients Vo((), needed both for and V steps. This involves running LLM multiple forward and backward passes on batches of texts and accumulating the gradients. To reduce the overhead from gradient computation, we compute the gradients once using mixed precision, then reuse these gradients for one P and one V step, respectively. In other words, we switch from alternating and V steps to performing these steps simultaneously. We also reuse these gradients to update any parameters not affected by quantization: input embeddings. normalization layers, biases, etc.
To limit VRAM usage, we use gradient checkpointing (27], batch accumulation. For larger models, also use parameter sharding [55) and optimizer offloading (57). We need these techniques so that smaller 7B LLMs can be trained on a single GPU and larger ones with 70B parameters fit into single machine with 8x A000 Still, PV-tuning takes up to 1.5x longer than tuning continuous parameters and uses more memory (during training) to hold the gradients w.r.t. dequantized weights.
l1 from e.g. http:://tttu..oooeeeee/////////t
Language-modelingg// mwdeeiig/ruu.nnmmmmm..mmmmmmm clmm
We use Py Torch FullySharreddatttPaalllll 52 40 and wrap discrete weights in bitssanbyytes
27

---

I Additional evaluations of perplexity
6.5
AQLM
PV
6.0
QuiPa
Theoretical Lossless (FP16)
Peplecity Wigicat-2
5.5
5.0
4.5
4.0
3.5
3.0
16
Model
Figure 3: WikiText-2 perplexity of 2-bit quantized LLAMA 2 models function of model size (GiB) compared to a theoretical lossless 3 bit compressed model (i.e. float16 perplexity numbers paired with 3-bit model sizes).
Additional Details for Section 4.1
Here, we describe some the implementation details we used to optimize different quantized representations. For every optimizations, we check that this optimization improves the algorithm in both MSE and perplexity and does not require additional resources that would result unfair comparison.
Vector Quantziation. The original algorithm quantizes all weights single pass over input channels. We found that it works slightly better if we make multiple such passes and, between passes, update codes by Adam to minimize the same objective 722. This is resembles QulP# with no RHT lattices or AQLM with no additive quantization. For simplicity, we also use single shared codebook (e.g.
instead of groupwise codebooks).
VQ+outliers To select outlier coordinates, we use hattps://gat /ttp://ggttt...... htt.:/mommifioommmboooeo
admm-pruning: that outperforms the SpQR outlier criterion [18] in both 1.2 error and perplex- ity (when both criteria are used alongside vector quantization). We re-run the ADMM procedure multiple times during optimization, resulting in an EM-like algorithm.
VQ+lowrank. We experimented with two different initializations for low-rank correction: a) quantizing weight matrix, then training LoRC on quantization error, in [79] and b) initializing LoRC to approximate the reference weight matrix, the applying vector quantization to LoRC errors. Of these two approaches, found that the latter produces a (slightly) better solution in both MSE and perplexity.
Additional representation evaluations. Below, we report some additional quantized representation
configurations that extend our evaluations from Section 4.1 For convenience, we report them both as per-method perplexity values in Table and the combined plots in Figure
28

---

Table 7: Comparison of WikiText-2 Perplexity for each method with and without fine-tuning for
quantizing Llama 2 7B model. For each method, PPL no FT denotes its perplexity without fine-
tuning, whereas PPL w/ FT perplexity with fine-tuning. We use the same setup as in Section 4.1.
Method
Avg bits
PPL FT.
PPL w/ F
Method Avg bits PPL FTІ
PPL w/ FT
5.12
GPTQ
3290
16.77
VQ/AQ
2.01
6.64
6.17
GPTQ
3
8.52
7.26
VQ/AQ
2.29
6,31
5.92
GPTQ
5.87
5.74
VQ/AQ
3.04
5.46
5.39
SpQR
2.09
12.19
9.90
QulP#
2.00
8.22
6.19
SpQR
3.45
5.48
5.37
QulP#
3.00
5.60
5.41
SpQR
3.98
5.29
5.21
62
260
25
Figure 4: Llama2 7B perplexity on WikiText-2 after compression without fine-tuning (left) and with fine-tuning (middle). On the (right), there is a plot that combines the first two, allowing for better comparison of each method with and without fine-tuning. Compression algorithms without fine-tuning are represented with dashed lines, while algorithms with fine-tuning are represented with continuous lines.
Table 8: Evaluation of quantized LLAMA 2 models for 2.x bits per weight. We use the same setup as in the main paper (Section 4.2 with extra baseline). As requested, finetune the model in 16-bit precision for the same number of steps, then quantize it with AQLM, reported as "FT+AQLM", Finally, the "Finetuned" row corresponds without quantization. We hypothesize that fine-tuning the model has little effect since we train on dataset resembling its original pretraining data.
uncompressed 16-bit model finetuned
Size
Method
Avg bits
Wiki2, C4 AreCt ArcEt HellaSwagt
PiQA WinoGrande" Averaget
16.00
5.12
6.63
43.43
76.3
57.14
78.07
69,06
64.80
AQLM
2.02
6.64
8.56
33.28
61.87
49.49
73.56
64.17
56.47
PV-Tuning
2.02
5.84
7.62
38.40
71.17
53.50
76.99
66.69
61.35
Finetuned
16
5.13
6.59
43.46
76.47
56.96
78.14
68.91
64.79
FT + AQLM
2.02
6.48
8.36
33.29
62.45
49.31
73.49
64.82
56.67
Prolliin subeuuc ED0 Stridht Thougss Eakeabe
Pertletiooo on
20
40
60
50
100
5ep
Figure 5: Learning curve for PV-tuning and STE algorithms, when tuning tinyllama model with 2x8g8 AQLM quantization (2 codebooks with bits per code and input groupsize equal to 8).
29

---

K
Additional Details for Section 4.2
VQ: vector quantization as a simple near-optimal algorithm. We use single 16-bit codebook with group size 16 (over input dimension) and per-channel trainable scales over output dimension. During P step, we update the codebook, scales and non-quantized model layers by backprop. During V step, we try every code in the codebook and choose the one that minimizes (6)
GPTQ: scalar quantization with block-wise scales and zero points, We use 2-bit base codes and block size 128. During P step, we update the scales and non-quantized parameters by backprop. In tur, V step performs simple rounding to nearest (scaled) integer [13
we perform scalar quantization with two 8-bit codebooks and group size 8 and channel-wise steps. This was originally published in [21] as the "speed-optimizee
AQLM:
configuration capable of fast lookup-based inference. During P step, we update both codebooks, as well scales an non-quantized parameters by backprop. On V step, we run beam search with beam size 8 with gradient-updated dequamtized weight as target.
Training. We minimize
Beulback-Leiblerr divergence as our loss function for all three representations. More specifically, we fine-tune the quantized "student" model to approximate the predictions (logits) of a "teacher" - the same model prior to quantization. We fine-tune on the same RedPajama sample as in calibration. More specifically,
use the official one-billion-toke sample provided by the dataset authors [3]. We use a batch size of 2 (= 1M) tokens, split into batches of model-specific sequence length (e.g. 4096 tokens for LLAMA 2, 8192 for LLAMA 3). In early experiments. we found PV to be resilient to the choice of batch size, consistently training with up to 4x smaller batches.
Hyperparameter tuning: we tune the hyperparameters for each method individually. For all algorithmss, we tune learning rate on a logarithmic scale out of (Ic-4, 3c-4, 1c-3, 3e-3, 1c-2). For methods involving discrete optimization, optimal configuration for STE and stochastic rounding is to use learning rate 3e-4 for both codes and codebooks. The optimal configuration for subspace linearized PV and the same with STE is to use learning rate 3e-4 for P step and 3e-3 for codes. Curiously, the subspace methods are stable even with larger step sizes for codes, e.g. 1e-2, whereas unrestricted methods (e.g. pure STE) are not.
tune learning rate for P and V step separately. The
For stochastic rounding, we found that the unbiased rounding [33 causes the model quality to quickly degrade, likely due to the fact that the algorithm makes too many weight changes due to rounding. The results we reported in Table use stochastic rounding with temperature 0.2. In other words, we nearest bins and round to each bin proportionally to distance- We also tried gradually annealing the rounding temperature to 0, but achieved only insignificant improvements in accuracy and perplexity (<0.01). To simplify evaluation, we do not use annealing in Table
measure the distances
For PV-tuning and PV-tuning with STE, we always set T to maximum number of weights such that updating them satisfies ||x_ | < 0.01. We implement this by trying to update large portions of weights (in our implementation, we update 0,01 of all weights at a time) until the total change exceeds the constraint. Once it does, we rollback weights the last chunk to x* until the constraint is satisfied. As an implementation quirk, we always allow at least one weight to change even if changing just one weight already exceeds 0.01 ||**|| || However, we didn't see this in practice.
We also found that Lamt to approximately the same accuracy. We use 81 =0.9 and 32=0.95, same in most LLM training configurations [69] 86 61]. We do not use learning rate decay for simplicity. It is likely possibly to improve our results by annealing the learning rates during training using warmup. We intentionally avoid this to reduce the number of "moving parts" and simplify evaluation. Overall, we found that PV-tuning is about sensitive hyperparameters as continuous-only LLM fine-tuning
81 is more stable when training with large batch sizes, but converges
In the codebase, we release several additional PV-tuned models with a more careful choice of training hyperparameters and calibration datasets: for instance, when quantizing instruction tuned models, we found that calibrating the model on chat-like data results in better accuracy.
We also found that we can perform V step by running the entire OPTQ calibration while using y 1.2 Vo() as target weights, showing the flexibility of the general PV framework. This variant converges to the same values, but is much slower due to having re-accumulate the 1.2 error hessians for each V step. Dataa-----aaaa-p
14 "htpp://huggnngfacc wtagginnaaac../aaaattt/////eeeee
1 Lamb is variant of Adam that limits the norm of weight updates relative to the norm of weights.
30

---

L PV-Tuning of QulP#
QulP# is a popular modification of classical vector quantization for neural network compression. Unlike standard VQ. QulP# does not quantize weights directly, but instead applies a Randomized Hadamard Transform (RHT) and quantizes the projected weight representations. This is done by applying the Hadamard transform to the rows and columns of the X matrix and multiplying them by random vectors Sv ~ [=1,,,, Sy U +1 1 after each transformation. These vectors do not need be stored, but may be generated on the fly from a random seed.
The main objective of this transformation is to ensure that the adheres to normal distribution. With this in mind, QuIP# no longer needs learn the optimal quantization "codebook", but may instead use the optimal lattice for quantizing normal distribution. As result, QulP# does not need store the codebooks in memory, thereby reducing its overall bitwidth. For instance, for - 7B vector quantization with group size 8 and 16-bit bit codes, VQ produces 2.29 bit model while QulP# has slightly less than 2.01 bits per parameter.
weight matrix
We apply PV-Tuning to QulP# in a manner similar to Vector Quantization. We start with a model already quantized using the original QulP# algorithm, specifically employing the official 2-bit quantization of the LJama 2 7B mode [6 We then optimize this model using the same hyperparameters as those for Vector Quantization (see Appendix K).
During P step, we optimize quantization scales (SU, SV), as well as any non-quantized model parameters (embeddings, LM head, normalization scales and biases), but not the codebook, since QuiP# relies on predefined lattice structure that cannot be trained directly. During V step, we update the discrete codes responsible for choosing vector out of the said lattice using the same procedure as in standard vector quantization.
We compare PV-Tuning against several other fine-tuning strategies: no fine-tuning, the built-in fine- tuning procedure proposed in [71), as well as the technically improved version of that procedure using the observations described in in G (denoted as "improved FT We report the resulting perplexities and zero-shot accuracies in Table and include them in other relevant tables as "QuIPH+PV".
Table 9: Evaluation of LLAMA-2 7B quantized using QulP# with various fine-tuning strategies. We report perplexity on WikiText-2 [45) C4 (54) and zero-shot accuracy. The Average the mean accuracy of 5 zero-shot tasks. Primary metries are Wiki2 (PPL), C4 (PPL) and Average (Accuracy).
Method
Avg bits
Wiki2|
C4.
AreCT
ArcEt
HellaSwagt
PiQAt
WinoGrandet Average
16.00
5.12
6.63
43,43
76.3
57.14
78.07
69.06
64.80
QulP# (w/o FT)
2.01
8.22
11.01
28.84
55.56
42.94
71.38
62.43
52.23
QulP# (built-in in FT)
2.01
6.19
8.16
34,60
64.60
48.34
75.10
64,90
57.51
QulP# (improved FT)
2.01
5.92
7.82
37.63
72.39
53.07
76.12
65.35
60.91
QulP# + PV
2.01
5.71
7.51
39.33
72.01
54.10
77.20
66.38
61.81
In general, we found that QulP# shows similar if not better improvements from PV-Tuning VQ and AQLM. Furthermore, when compared to only fine-tuning continuous parameters, QuiP# shows better relative improvement from V steps than traditional vector quantization. We attribute this to the fact that VQ can fine-tune its codebook even without PV-Tuning, whereas QuIP# relies on fixed codebooks that cannot be learned. As result, discrete parameters (codes) take up larger fraction of the total model size in QuIPH, allowing V steps to make more significant improvements. The detailed instructions for running these experiments can be found in our official implementation [7 subsection of the common README file.
in separate
M On 1-bit Vector Quantization Options
As discussed above, there are several ways achieve the same number of bits per parameter with vector quantization. In this section, we explore the effect of varying the group size: either quantizing
16 httepp///uuggiigddeee.c//eeesess E-0-2Bi----------
http::///gghhi.occm/////sss////t tuning
31

---

groups of 16 weights with larger codebooks, groups of 8 weights with smaller ones. The results with different group and codebook sizes are reported Table
Table 10:
Evaluation of LLAMA-2 7B quantized using VQ+PV with different group size (GS, also known as vector dimension) and code bits (CB, s.t. codebook size is BC We report perplexity on WikiText-2 [4] & C4 (54) and zero-shot accuracy. The Average is the mean accuracy of 5 zero-shot tasks. Primary metrics are Wiki2 (PPL), C4 (PPL) and Average (Accuracy).
GS CB
Avg bits
Wiki2
C4
AreCt
ArcEt HellaSwagt
PiQAt WinoGrandet Average
16.00
5.12
6.63
43,43
76.3
57.14
78.07
69.06
64.80
8
1.00
7.62
9.73
28.84
61.66
44.66
72.14
61.56
53.77
9
1.13
7.15
9.20
30.55
65.11
46.68
72.63
62.04
55.40
14
1.02
8.28
10.37
25.85
57.58
40.88
68,99
57.77
50.21
16
16
1.58
7.32
9.35
29.44
64.14
46.03
73.12
63.38
55.22
Overall, we figure out that smaller groups provide better performance for the same model size. We hypothesize that PV-Tuning is better able deal with these configurations because with smaller group size, discrete codes constitute larger fraction of model parameters, which allows the V step of our algorithm to achieve more significant improvements. In the initial version of manuscript performed all 1-bit experiments with suboptimal group size 16. More recent results for group size 8 provided under separate name "PV (gs 8)" in Table 7
N Additional Details and Evaluations for Section 4.3
In this section, we report additional results for L.LAMA 3, MISTRAL and PHI-3 and discuss baselines. In this section, we always evaluate PV-tuning for vector quantization, using 14-16 bits per codebook for group of 8 or 16 weights, with each combination fitting a particualr niche. For instance, 16 bits per 8 weights is slightly over 2 bits per weight, whereas 14 bits per 16 weights is either at or below bit per weight, depending on the model size.
We use LLAMA 2 models as our main benchmark as they are well studied in the PTQ community, Here, we gather the latest state-of-the-ee algorithms at the time of publication and group them according to their target number of bits, roughly 1-1.7 bits per weight (Table IT) and 2-2.5 bits per weight (Table 12].
Both our training runs and almost all baselines use the same sample of RedPajama data from previous s [8] The only exception to this is OneBit that uses corpora of LLM outputs gathered specifically for that paper [77].
The source code for this method was unavailable until very recently. The official link com//uuuzzzuuuagg1/////iii 1/nnettg used to point to an empty repository until the code released in commit hattps://githu: com/ xuyuzhuang11 wwe11////0//t///////eeeee on May 10th. Thus, unfortunately, we did not have time to make OneBit compatible with models except LLAMA 7B and 13B that were featured in the original paper.
httpp::/gittie thub....ttu..... wwt./oooooiiiaee
For LLAMA 3, we evaluate PV-tuning of vector quantization against the baselines introduced in 33 Curiously, their paper seems to compute perplexity differently than our paper. Since our protocol matches with most prior works [22 [8 71], we chose to re-evaluate the results from our perplexity code and not the other way around. We calibrate using the official code 19 and reuse published models where available.
(33 with
Inference Speed with Vector Quantization Kernels
In this section, we demonstrate that PV-Tuning can achieve speedups by using fast inference kerness from the underlying quantized representation.
reamizzedeerreeenttattiooon Since our main experiments use vector quantization,
http:////waaaaacccccee/eeeeee/ea -aamaa--aa---- Maa---TSaamppn Sample httpp///tthhbooo///ccceolll//llt
32

---

Table 11: Evaluation quantized LLAMA 2 models for 1-1.7 bits per weight, grouped by bitwidth.
We report perplexity WikiText-2 4S & C4 [4 and zero-shot accuracy. The Average is the mean
accuracy of zero-shot tasks. Primary metries are Wiki2 (PPL), C4 (PPL) and Average (Aceuracy).
Size
Method
Avg bits
Wiki2
C44
AreCt
ArcE
HellaSwag?
PiQAt WinoGrande?
Averaget
16.00
5.12
6.63
43.43
76.3
57.14
78.07
69.06
64,80
BiLLM
1.08
32.48
40.52
24.4
36.2
34,8
60.6
52.4
41.68
OncBit
1.0
9.73
11.11
29.61
41.58
52.58
68.12
58.41
50.06
7B
PV-Tuning
1.02
8.28
10.37
25.85
57.58
40.88
68.99
57.77
50.21
PB-LLM
1.70
69.20
80.15
25.00
28.00
27.70
53.80
49.30
36.76
PV-Tuning
1.58
7.32
9.35
29.44
64.14
46.03
73.12
63.38
55.22
16.00
4.57
6.05
48.38
79.42
60.03
79.05
72.22
67.82
BiLLM
1.10
16.77
27.54
21.84
46.84
30.97
60.61
56.75
43,40
OneBit
1.00
8.76
10.15
33.62
43.10
56.43
70.13
61.72
53,00
13B
PV-Tuning
0.97
7.23
9.31
30.8
63.09
47.03
72.25
62.35
55.10
PB-LLM
1.70
151.09
144.59
21.89
35.08
24.82
54.17
52.76
37.74
PV-Tuning
1.37
6.65
8.72
34.04
67.38
49.14
73.94
65.51
58.00
16.00
3.12
4.97
54.35
82.74
64.79
82.15
77.98
72.40
BiLLM
1.08
8.41
15.19
38.91
67.3
45.71
69.7
67.64
57.85
PV-Tuning
1.01
6.09
8.20
38.31
71.80
53.98
75.24
68.43
61.55
70B
PB-LLM 1.70
28.37
32.63
39.89
49.50
36.62
61.43
62.80
50.05
PV-Tuning 1.14
5.52
7.50
42.66
74.96
56.42
77.37
71.51
64.58
Table 12: Evaluation of quantized LLAMA 2 models for 2-2.3 bits per weight, grouped by bitwidth. C4 [54] and zero-shot accuracy. The Average is the mean
We report perplexity on 2 45] accuracy of 5 zero-shot tasks. Primary metries Wiki2 (PPL., C4 (PPL) and Average (Accuracy).
Size
Method
Avg bits
Wiki2,
C4.
AreCt
AreEt
HellaSwagt
PiQA
WinoGrande"
Average
16.00
5.12
6.63
43.43
76.30
57.14
78.07
69.06
64.80
QuIP#
2.02
8.22
11.01
34.60
64.60
48.34
75.10
64.90
57.51
AQLM
2.02
6.64
8.56
33.28
61.87
49.49
73.56
64.17
56.47
7B
PV-Tuning
2.02
5.84
7.62
38.40
71.17
53.50
76.99
66.69
61.35
AQLM
2.29
6.29
8.11
34.90
66.50
50.88
74.92
65.67
58.57
PV-Tuning
2.29
5.68
7.47
38.91
72.90
53.94
77.37
67.72
62.17
16.00
4.57
6.05
48.38
79.42
60.03
79.05
72.22
67.82
QulP#
2.01
6.06
8.07
39.50
69.30
53.44
77.30
67.70
61.45
AQLM
1.97
5.65
7.51
37.80
69.78
53.74
76.22
65.43
60.59
13B
PV-Tuning
1.97
5.12
6.83
43.00
75.38
57.96
78.24
70.01
64.92
AQLM
2.19
5.41
7.21
41.98
75.04
55.49
76.99
69.53
63.81
PV-Tuning
2.19
5.05
6.74
45.65
77.57
58.00
78.07
70.96
66.05
16.00
3.12
4.97
54.35
82.74
64.79
82.15
77.98
72.40
QulP#
2.00
4.16
6.01
48.70
77.30
60.79
80.30
75.90
68.60
70B
AQLM
2.07
3.94
5.72
51.96
81.44
61.46
80.25
76.64
70.35
PV-Tuning
2.07
3.78
5.56
$1.88
81.02
63.07
80.74
76.87
70.72
we adopt simplified version of AQLM inference kernels with one codebook of size 16 for groups of 8 consecutive weights. This kernel is not written by us: was added to official AQLM implementation by an open-source contributor.
We adapt this inference code our codebase and switch using group size 16 support out 1.1- 1.58 bit models. We evaluate inference speeds on a single Nvidia RTX 3090 GPU using transformers with cuda graph...
The specific version of each library can be found in 'requirements.xx
33

---

Table 13: Evaluation quantized LLAMA 3 models for 1-2.3 bits per weight, grouped by bitwidth.
We report perplexity WikiText-2 4S & C4 [44 and zero-shot accuracy. The Average is the mean
accuracy of zero-shot tasks. Primary metries are Wiki2 (PPL), C4 (PPL) and Average (Accuracy).
Size
Method
Avg bits
Wiki2
C44
AreCt
ArcE
HellaSwag?
PiQA WinoGrande?
Averaget
16.00
5.54
7.10
50.43
80.09
60.19
79.71
72.61
68.60
BiLLM
1.10
28.80
65,00
17.70
36.00
28.90
56.10
51.00
37.90
PV-Tuning
1.01
11.13
11.63
25.43
59.09
41.01
68.26
56.27
50.01
PB-LLM
1.70
35.68
197.56
17.50
31.70
27.70
52.50
50.40
36.00
PV-Tuning
1.54
9.43
10.26
32.68
65.78
46.66
72.63
64.40
56.43
8B
QulP
2.00
76.95
98.47
21.30
29.00
29.20
52.90
51.70
36,80
PB-LLM
2.00
21.74
61.04
17.20
37.80
29.80
57.00
52.50
38,80
DB-LLM
2.00
12.77
14.82
28.20
59.10
42.10
68.90
60.40
51.80
PV-Tuning
2.00
6.99
8.29
42.75
75.84
55.52
77.75
69.93
64.36
PV-Tuning
2,30
6,76
8. 10
42.32
75.46
56.21
78.45
71.67
64.82
16.00
2.59
5.78
60.41
86.7
66.34
82,48
80.9
75,366
BiLLM
1.10
15.26
65.07
25.11
46.42
37.48
58.21
53.63
44.17
PV-Tuning 1.00
8.67
9.68
25.51
54.34
48.71
65.56
63.22
51.47
PB-LLM
1.70
16.27
$4.03
25.8
49.90
34.90
56.5
53.10
44.1
PV-Tuning
1.14
7.76
8.93
33.28
63.89
53.39
69.86
69.61
58.01
QulP
2.00
11.63
18.54
26.50
48.90
40.90
65.30
61.70
48.70
PB-LLM
2.00
10.33
28.89
25.10
40.60
42.70
65.20
56.40
46.00
PV-Tuning
2.07
4.55
6.54
50.77
80.22
63.85
79.22
78.06
70.42
Table 14: Evaluation of quantized MISTRAL v0.1 7B (A) and PHI 4-MINN-4K-INSTRUCTT 3.8B (B)
models for 1-2.3 bits per weight, grouped by bitwidth. We report perplexity WikiText-2 [45] & C4 [54] and zero-shot accuracy. The Average is the mean accuracy zero-shot tasks. Primary metrics Wiki2 (PPL), C4 (PPL) and Average (Accuracy).
Size
Method
Avg bits
Wiki2
C4
AreCt
AreEf HellaSwag?
PiQA WinoGrande?
Averaget
16.00
4.77
5.71
50.43
80.09
60.19
79.71
72.61
68.60
AQLM
1.01
70.88
34.67
19.11
27.36
26.3
52.99
48.38
34.83
PV-Tuning
1.01
7.58
8.14
27.73
60.82
44.33
69.97
60.38
52.65
7B
QulP#
2.01
6.02
6.84
39.76
72.14
52.95
76.71
69.30
62.20
(A)
AQLM
2.01
6.19
6.90
30.8
49.87
25.63
56.53
57.06
43.98
PV-Tuning
2.01
5.29
6.17
44.20
77.36
58.21
79.05
72.77
66.32
AQLM
2.27
5.78
6.55
42.06
75.17
55.09
76.93
70.24
63.90
PV-Tuning
2.27
5.22
6.10
45.31
77.57
58.61
79.22
70,96
66.33
16.00
4.77
5.71
60.41
86.7
66.34
82.48
80.90
75.37
AQLM
1.03
102.54
85.20
18.86
28.16
27.13
53.54
49.80
35.50
PV-Tuning
1.03
11.71
14.59
21.50
49.87
34.62
65.67
54.70
45.27
AQLM
1.60
34.36
35.16
19.62
35.10
29.71
$7.24
51.7
38.67
3.8B
PV-Tuning
1.60
9.21
12.16
30.89
63.55
43.09
70.35
61.4
53.86
(B)
AQLM 2.03
8.85
12.19
41.04
74.49
47.25
72.36
66.93
60.41
PV-Tuning 2.03
6.88
10.08
46.84
78.24
53.75
78.67
71.03
65.71
AQLM
2.30
8.07
11.23
47.01
78.03
50.51
75.95
70.8
64.46
PV-Tuning 2.30
6.63
9.89
50.51
79.5
55.32
79.49
73.01
67.57
We were able to acheve 47.4 tokens per second for 7B model, 32.8 tokens per second for 13B model and 7.2 tokens per second for LLAMA 70B model. Compared to 16-bit inference code, this results in speedups of 14%, 22% and 28% respectively. Note that PV-Tuning does not make the model inherently faster than, for instance, AQLM. Instead, can achieve better quality with lower bit smaller and faster models for the same accuracy target.
models, allowing practitioners
34

---

P
The Choice of the Initial Point x
Algorithm 1 converges from any initial point 00 R 3.1 but it might converge different final point (xx)) o) In this section, we discuss two possible variants of instantiating the initial point.
P.1 Clipping of
Notation: [d] = f1, } is the set of d distinct natural numbers from 1 to d.
Assume we have the vector with dimensionality d and let [d]. Let us define the clipping operator C(x)) Ied RR in the following way:
C(x) 5: V (x) and [V
(15)
We can come with different variants of clipping operators C(z). but in our experiments, we choose ((x) such that (( consists of the smallest distinct elements from V(x)). We will call this clipping operator by C- (x)). Example P.1. If x = 11,1,3.5, 5, then having = 2 the clipping operator C_(x) = E
{1, 3,,1,,3333 3). So, i consists of smallest elements from V(x) (,93.
P.2 Random d0 E
Now let us define the algorithm for generating random points from RC (Alg. 6
Algorithm 6 Generation of Random Point x² E I2d
1: Parameters: dimensionality d, number of distinct values
2: Generate a set U with unique elements: U {ur E Rd) UU| = c.
3: Sample d random elements x E U: _d = (x), =2)² such that [V(x))
Different random x E Rd provide different loss functions. To get more stable and smoothed results we ran the algorithm (T) with different initial points, generated by algorithm (6). We will denote r as the number of runs with different random initialization.
Small-Scale Experiments and Interpretation
Q.1
Objective function
Let c E [d] and consider the problem
min σ(z),
(16)
WER
where Rd > R. and Rd C Rd is the set of all vectors in Rd whose entries take most distinct values. In other words, the cardinality of the set
V(x) == (.1,-.. ,r)
(17)
most For small experiments, we aim minimize the following objective
<(x) aa(a - x²)² (s x² x² A(x x²),
(18)
where a; E Rdd, at a unique optimal point: Vo(x*)) = and
aa
A diag(a1, aa)
(19)
0
ad
We set a; = 1/a for E [d]. Hence, we have the Lipschitz continuity for (18) with L = 1. Note that the Algorithm does not guarantee to converge to xx Let å denote the vector to which the Algorithm converges.
35

---

Q.2
scale experiments (d = 6, c [1,6]
We applied PV algorithm to the problem 16 with ((x) being 18 (Fig. different random intial points
Number of runs with
50.
10a
N.c=2
N.c-i
10a
F + steps
10a
PV.c-6
10
Φxx
10a
10
10
10
2
4
8
10
Iteration 4
2
Iteration k
(a) PV algorithm with different values of E
(b) The influence of P and V steps the loss function o(x), c= 3.
[1.
d
Figure 6: PV algorithm 1 applied the very small dimensional (d 6) quadratic objective 18 The starting point is chosen randomly using the ng algorithm
When 1 converge just in step of PV (red line Gag because have only one degree
of freedom to play with and we fully utilize it on the first step. One can show that the solution for the
case 1 will be (E=10))
Di=1 0:20
Note that as we increase the maximum number of unique elements the final loss o(t) decreases up until the moment when d, when the loss is zero (the plot 6a is in logarithmic scale and that why we cannot see the last line).
We run PV algorithm 1 with different random starting points x² Id (6 Different starting points can lead to different local optimum. That is why we ran the algorithm (1) several times with different random initial points z0. A number of runs 50, used this value for all further experiments.
Each of the two steps of the algorithm contributes the convergence. the loss function over the iterates and explicitly marked the progress of both P and V steps (Fig, (bb). We can see that we have progress during each of these steps and one single and V step not enough to obtain solution even in this very small and simple case.
observe this we plotted
These simple experiments demonstrate that
1. larger e (smaller compress ratio) provides better final accuracy
2. several and V steps are needed to converge to the solution
Q.3 Interpretation of P(y) 3 (ax**)
As we mentioned before, we select the initial point x0 IR randomly such that it has c unique elements in its linear shell V==²). To understand the notation P(y) P(x*). defined in the chapter (3.1) let us consider the first step of the algorithm, specifically the step.
On the first step of the algorithm select new point by solving the following optimization problem:
y arg min (d(y) P(v) 2 P(z)))
(20)
yERd
Here we have a random point z0 with some partition P(x9) (2) Pa(x:)) sets of equal elements from z0 We find 1 such that minimizes (())) and have the same smaller number
of partitions: P(y") (P)))) PP()))) where
36

---

Strict equality of two sets, P(x") and P(y") would mean that the linear shell the number of unique elements of the vector xn is equal the number of unique elements from y (c = s). In our PV algorithm we allow y to have smaller number of unique elements than x", so we possibly merge some partitions of x."
It is worth to mention that in real experiments we observe that the number of unique elements did not change or the change significant.
Q.4 Simple example of P(y) 2 P(**)
Let d = 8 and
(1.3, 1, 1,3). Then Z E Rg where == 4 since x consists of 4 unique
floats: 1,3, 4 and We have (z) = (1,3, 7. P2(x) (2.8), P3(x) = (4, and Pa(x) {6).
So. P(x)= = {Pi (zx), (2)x), (x(x), Px(x) (11,3, 7}. (2, (4.5), P(z) is set whose
elements are four sets, forming a partition
1.,,,....0
Let y {1. 1,4,1, 1.1. Then y R where = 2. We have Pi()) {1, 2,3.7.8) and P2(y) = [4,5,6] So, P(y) = {(1,2,3, 2,8,7 5, 5.6]]- Notice that each element of P(x) is a subset of some element of P(y). For example, {2. 8) is subset of (1,2,3,7,8) 7,83 and (6) is subset
of (4, 5, 5). Because of this, by our definition, P(y) 2 P(z).
Q.5 Small-scale experiments (d 100, c E [1, 100))
The problem of the algorithm the complexity (c Even for small tasks, this becomes unpractical to solve.
is that the V step requires the full parameter search which gives us
Example Q.1. Let us take d = 100 and c 10, then the complexity one V step will be O110100). Modern computers make roughly 10 petaflops 10 calculations per second, hence will have to wait ~ 1676 years to make a single V step.
Let us consider special sets of function ((x) that we will call separable functions. This class of functions should satisfy the assumption (Q.2).
Assumption Q.2 (Separable function). The function <(x:) TRdd R can be written in the form d ((x) Σ 1 (x,),, where (-) is mapping (b(+)= : + IR.
Example Q.3. The objective function (18) is sum of squares that can be written in the following form:
d
(xx)= = Σ a1 x;)²
(21)
where (x(x,) a,(x, x²)² Hence, the objective 18 is a separable function.
One can show that for separable functions (Q.2) the algorithm can be written in the form Hence, for separable functions, we can compute the V step in Olc d) operations (instead of O which makes the algorithm practical to use.
Algorithm 7 PV Algorithm with Optimized V step
1: Initialization: starting point v0 E IRd
2: for & = 0, 1.... do
3: y arg min { (d))) P(y) P(**)}
(P step)
4: for = 0, 1,..,, do
5: x k+1 arg min,, ER (0)( : x, E ((**))
(Optimized step)
6: end for
7:
end for
Then we ran the optimized PV algorithm 7 on the quadratic objective 18] The results are presented in (Figure 7. Number of runs with different random initial points 50.
For these experiments, we see the same results as for tiny scale with d d= (Q.2) larger e (smaller compress ratio) provides better final accuracy, and several P and V steps have to be done to obtain better solution. In addition, we observe that the PV algorithm does not decrease the dimensionality (number of unique elements) of (z))
37

---

As it was mentioned before, the algorithm T is conceptual only and cannot be used in the raw form practice. That why we need move the experiments with linearized V step.
10*
Alter stop + steps
10a
PV
10a
x
101
Pv
10a
100
10a
10-1
10
10
2
4
8
10
2
8
10
Iteration k
Iteration k
(a) PV algorithm with different values of E
(b) The influence of P and V steps the loss function o(z), c= 10.
[10, 100), 100.
Figure 7: Optimized PV algorithm (7) applied on the quadratic objective 18). d 100. Number of runs with different random initial points 50.
Q.6
Linearized PV
Linearized V step B.I allows us to greatly reduce the cost of one V step. We ran Linearized PV on the quadratic objective 18 (Fig. {
10a
PV
Unearioed PV, T
After stop steps
Uncorized PV. T
Unearized T
10a
10?
(xx*)
10a
109.
10a
10
Geo -000
2
10
12
14
8
Iteration k
Iteration k
(a) Lincarized PV algorithm [B.1] with different [1, Larger T provides faster convergence rates, but same final accuracy.
(b) The influence of P and V steps the loss function d(xx), c 10, T We need make linearized steps on each iteration.
Figure 8: Experiments with Linearized PV algorithm
B.
From the first experiment (Fig.
8a we can see that
1. increasing provides slightly better convergence rates and approximately the same final accuracy. We saw the similar results on large-scale experiments. Hence, T = to save computations.
2. Linearized PV algorithm
B.. converges to a worse accuracy than the exact PV T
3. Linearized PV has to make more iterations than PV to converge
The second plot (Fig 8b) demonstrates the effect of multiple Linearized step. We can see that the largest effect comes from the first V step.
38

---

Q.7 Linearized PV + sparse updates
In previous section QQ66 we have seen that Linearized PV algorithm converges to a worse accuracy than the exact PV
Linearized PV B. with combination of sparse updates (3.3 is intended to mitigate this issue.
On (Fig. Pa we can see comparison of three methods: exact PV T red line, Linearized PV (B.1 blue line and Linearized PV with sparse updates (3.3)
green line.
From this experiment we observe
1. Linearized PV with sparse updates converges better accuracy than Linearized PV
2. inearized PV + sparse updates has to make more iterations than Linearized PV and exact PV converge
Hence, this approach helps to converge to better accuracy, but with price of larger number of iterations to converge.
10a
PV Unearized PV
10a
Lineerized
Linearized PV sparse updanes igreedy lop-()
Uncarized sparse upderes (greddy Top
Linsarized PV sparse updaces (anddm unifomi
107
10a
3-
Lineatized PV sperse updece (random pmportional)
((x*
10a
10a_
10a
10a
1
10-1
10
20
30
40
50
60
70
80
300 Iteration
Iteration
Influence of sparse updates
(b) Different choise of sampling for choosing Sk
Figure 9: Comparison of the exact PV algorithm with Linearized PV B.1 and Linearized PV +
sparse updates (3.3
We can use different rules for choosing the subspace S which produce different convergence rates and the final accuracy levels (9b).
1.0
80
random
2 madom
uniferm
0.9
random proportional
75
rendom
nendom
random
3 0.7
random
0.6
0.5
0.4
10
20
30
40
50
10
20
50
60
70
80
Iteration k
Iteration k
(a) The behaviour of Lga in Linearized PV sparce updates.
(b) Degradation of dimensionality of V(x) for Linearized PV algorithm (B.I.
Figure 10: Comparison of the exact PV algorithm Sparse Updates (3.3)
with Linearized PV B.1 and Linearized PV +
39

---

In large scale experiments we used sparse updates with S being chosen greedily based on the absolute values of the entries of gradient vector [V,((y)) green line. Here we demonstrated that other strategies based on random uniform sampling or random proportional (V,((y") sampling can be even more advanced providing better final accuracy at cost of larger number of iterations to converge purple and orange lines (Fig. 9b).
We can see that Linearized PV with sparse updates can converge to even a better accuracy than the exact PV algorithm. The problem of Linearized PV algorithm is that come to the local minimum and cannot get out of that minima because of the small value of the gradient (we are near to a real solution) and small stepsize.
Linearized PV with sparse updates allows us to mitigate this problem by reducing the subspace from R to S in which are solving this optimization problem. This allows us to greatly reduce the local Lipschitz constant from L to Lg* and hence significantly increase the stepsize 7 = 1/L We demonstrated how Lgx changes for Linearized PV with different sparse updates sampling methods (Fig. T0ah.
R PV+ Algorithm
We can have degradation of [V(=*)| during both the P and V steps. Here are examples for both of them with optimizing simple quadratic objective (xx) > (x; xt) (case when all a; E [d] in (T8) are equal to one).
1. Degradation during the P step: Let x² 2, 13, x0
(I. [r. so (V then after the step we will have y" = (1, 1, 1}. hence [V(y)) =1
2. Degradation during the step: Let
(2, 10, 0, 11]. x0 yr,y.8.27 2), so [(=*)==
3, then after the P step we will have yd = 5,, 11}. Finally, after the V step we have
x² (2, 11, 2, 11), so [(x*)l
In real experiments we observe decreasing of [V()). You can observe this phenomena with Lin- earized PV algorithm (IOb). As we can see we have a big decreasing of [V()) during the first several iterations. We can use this gap to find even better solution with improved final accuracy and faster convergence rate.
To add additional unique element in V (xx we considered the modification of V step: V step, where we allow IV (xr) become larger up to some upper bound.
Let (6-) is mapping Φ(-) : Rd + R and the vector x= E IR be the optimal point: Vø(z*) = 0. Let
the vector x E IRd_ and define the set W (x, x²,", c) such that contains at most e - (x)] unique elements from V V(**) without elements from V(x):
W(x V(x*) V(z): [W .c) = (==-c-(())
(22)
Algorithm 8 PV Algorithm
Parameters: starting point x0 E Rd the optimal point x", maximal number of distinct values
C,
2: for 0, 1.... do
3: yd arg yin (d()): P(y) ] P(**-1))
(P step)
4: x arg min (Ø(x) V(z) C W()))) x*)
(Modified V step)
5: end for
Theorem R.1. Assume & is bounded below, and let dd E Rd Then the algorithm PV+ has the following guarantees
(i) y * e Rd for all 2 O.
(it) d(akt1) )(v²) S ol2*)) for all k 2 0, and
(ii) the sequence } converges to some value, which smaller or equal to one produced by PV algorithm
40

---

Proof. Part (ii): Since
arg min (d(z): V(x) S v(v e aER
and because I = ya satisfies the constraint V(x) S v(y*) ,c),), ((x*+1) < oly)).
.c), we conclude that
The rest of the proof is identical to (A.1)
S Broader Impact
The main impact of our work, both positive and negative, the ability to deploy higher-quality LLMs run on memory-limited devices like desktops, laptops, and phones. On the positive side, this would allow practitioners to develop offline LLM applications (c.g. translate service), lower-latency chat assistants that are dependant where the user's private data never leaves their device. Furthermore, this can facilitate the creation of free open-source software based LLMs by eliminating the need to maintain costly inference servers on the backend. Since phones are everywhere and LLMs are powerful general general-purpose tools, PV-tuned models could significantly impact how the general population uses LLMs complete tasks.
network latency, or privacy-sensitive LLM applications
However, LLMs are still a dual-use technology with the potential for significant benefits and serious harm. Risks range from deliberate misuse (e.g. spam generation) and accidental misuse negative economic side-effects. An upper bound on these risks that PV tuning does not create new (potentially risky) LLM capabilities, merely making existing ones more accessible.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims are that the PV-tuning algorithm 1) achievstate-of-the-aut quantized LLM quality for 1-2 bits per parameter (backed by Section 4.3 2) Pareto optimal at around bits (also Section 4.3 3) and is compatible with various methods (Section 4.2).
In the introduction, we also claim that the advanced quantized representations, such as those having sparse outliers, do not give significant benefit on top of simple vector quantization with fine-tuning: this part is backed by 4.1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the methodological limitations of our study near the end, after
Section 4.3
We also explain limitations for practitioners in Section 3.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to addressing problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
42

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We carefully introduced the assumptions (e.g. that <(-) is L-smooth) and provided proofs in appendix. To the best of our knowledge, these proofs are both correct and complete.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided the in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We train open-access LLMs on open datasets and release our full training code. We do our best to provide instructions and hyperparameters the code, though running our algorithm in different conditions may require basic tuning.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
43

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As we state above, we release the full implementation for the PV algorithm with requisite instructions. We do not introduce new datasets and use openly available ones. We also plan to release the main quantized models the non-anonymizee version of the paper, since it would be impractical to upload them with the supplementary zip archive.
Guidelines:
The answer NA means the paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
Policy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https?
7/nips. wwtpccccppppccceeececccccccceeee /taliic/ggiess/eee httpc/gguudde//oeSuububi.iee sonddeSubiissioPPoo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our setup in Sections 3.4 and 4 with additional hyperparameters baked into the supplementary code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, the in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
44

---

Justification: We report error bars for small-scale experiments Q For full fine-tuning runs, we do include error bars since running those would be prohibitively costly for us. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar and then state that they have a 96%CC the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type computing workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: report the hardware setting, calibration setting, time, and memory require- ments in Section 4 which is sufficient for practitioners to reproduce our results. We omit some details, c.g. which runs were restarted due to unrelated server infrastructure issues.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute worker CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required mocomputingute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research is focused the base capability and accessibility of LLMs. While working on LLMs always has potential externalities, our specific work adheres to the ethics guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
45

---

The authors should make sure to preserve anonymity (eg. if there is a special consider ation due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: While our work is more concerned with fundamental matters of discrete optimization and LLM quantization, we provide brief overview of its societal impacts in Appendix S
Guidelines:
The answer NA means that there is no societal impact on the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not release any newer models, and quantizing existing models typically results less capable (and therefore less risky) model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make the best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited, and are the license and terms of use explicitly mentioned and properly respected?
46

---

Answer: [Yes]
Justification: We use academically published artifacts (datasets, models, etc) and cite their respective authors.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code and provide documentation in the form of README and detailed doestrings. Both are included in the supplementary archive.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects in our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
47

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not conduct any research on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
48


Reviews:
[{"review_id": "uRvbLgfdT5", "reviewer": "Reviewer_3n7c", "summary": "The paper proposes the use of fine-tuning over highly-compressed models to achieve better model compression. By introducing the PV-tuning algorithm, the authors have achieved superior discrete parameter optimization compared to traditional STE methods. The effectiveness of PV-tuning has been validated by the authors using various base models and datasets.", "questions": "An important question: if we need to perform large-scale fine-tuning to achieve model quantization, why not maintain full precision during this large-scale fine-tuning and only apply low-precision quantization techniques after full precision fine-tuning?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors have tested the effectiveness of PV-tuning under different experimental settings. The improvements are notably significant compared to the baselines.\n2. The authors have found that fine-tuning should be employed to achieve better model compression rather than one-shot quantization.\n3. The authors have proposed the PV-tuning method, addressing the issue of small gradient updates in discrete optimization.", "weaknesses": "1. I would appreciate it if the authors could clarify the distinction between fine-tuning and one-shot quantization as mentioned in lines 37-43. What exactly does \"one-shot\" imply? Does it refer to layer-wise quantization? Conversely, what does the authors' claim of \"fine-tuning layer-wise\" mean? As I understand, quantization typically uses a small calibration set, yet the authors have utilized the entire RedPajama for calibration in section 4.1. Is the \"fine-tuning\" mentioned by the authors simply calibration on a larger scale?\n\n2. It seems the main problem the authors aim to address (line 45) is the further fine-tuning of already highly-compressed models on a large-scale dataset (RedPajama), rather than the traditional quantization/compression methods applied to full precision models. I agree that this approach might be effective. However, if this is the case, there might be an issue with the experimental setup in 4.1. Firstly, Figure 2 should compare the effects of each method with and without fine-tuning; the current Figure 2 middle and right do not directly facilitate this comparison. Secondly, the authors should not only select quantized representations from different methods for fine-tuning but also apply different fine-tuning strategies, including STE, PV-tuning, GPTQ, etc., to demonstrate the advantages of PV-tuning.\n\n3. In Table 2, PV-tuning indeed shows a significant effect. However, I am uncertain whether this is due to PV-tuning using more data (RedPajama) or because it is inherently more effective. Did the authors use the same calibration data for different methods in this experiment?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer appreciates our technical contribution and accuracy improvements. Below, we do our best to address concerns and answer questions raised in the review.\n\n> (W1) What does “one shot” imply? Does it refer to layer-wise quantization?\n\nIndeed, in L37-38, one-shot quantization refers to the family of quantization algorithms that compress the model in a single layer-by-layer pass over the model, without fine-tuning. Typically, this is done through the use of advanced quantized representations. This includes layerwise quantization algorithms such as GPTQ [1], SPQR [2], or QUIP [3], among others, and is contrasted to algorithms that fine-tune the entire model or a subset of the layers, such as OmniQuant [4], QuIP#[5] or AQLM[6]. \n\n> (W1) Conversely, what does the authors' claim of \"fine-tuning layer-wise\" mean? \n\nThis refers to a type of algorithm that fine-tunes every transformer block individually, relative to the outputs of the original model, to minimize MSE between original and quantized outputs. This technique is implemented in, e.g., AQLM [6] and QUIP# [5].\n\n> (W1) As I understand, quantization typically uses a small calibration set, yet the authors have utilized the entire RedPajama for calibration in section 4.1.\n\nWe believe there is a misunderstanding: as we describe in the dataset configuration (Appendix H, referenced in L243), **we use not the entire RedPajama dataset, but a small sample (0.1%) for that dataset**. Thus, we use a similar amount of calibration data to prior popular quantization techniques, such as AQLM and QuIP#.\n\nWe agree that some of these terms (e.g. “one-shot”) and details can be described more accurately. We will rewrite L37-38 and L243 to make our setup easier to follow.\n\n> (W2) Figure 2 should compare the effects of each method with and without fine-tuning; the current Figure 2 middle and right do not directly facilitate this comparison.\n\nThank you for the suggestion. We conducted additional experiments for some of the missing methods (e.g. GPTQ) and reported all results in **Table 1 and Figure 1** in the attached PDF. Note that one of the algorithms (SpQR) is still missing since the algorithm’s codebase requires additional work to incorporate proper finetuning. We will rectify this by the final version of the paper.\n\nIn the next revision, we will improve Figure 2 to make it easier to compare each of these methods with and without finetuning.\n\n> (W2) The authors should not only select quantized representations from different methods for fine-tuning but also apply different fine-tuning strategies, including STE, PV-tuning, GPTQ, etc., to demonstrate the advantages of PV-tuning.\n\nWe compare different fine-tuning strategies, including STE, PV-tuning, stochastic rounding and some of their combinations, in Section 4.2, Table 1. We apply each of these strategies to both GPTQ, VQ and AQLM to better understand their behavior for different representations. If you have any further suggestions to improve this analysis, we would be happy to apply them.\n\n> (W3) In Table 2, PV-tuning indeed shows a significant effect. However, I am uncertain whether this is due to PV-tuning using more data (RedPajama) or because it is inherently more effective. Did the authors use the same calibration data for different methods in this experiment?\n\nPV-Tuning only trains on a small sample of the RedPajama data, similarly to the baselines.\nWe use an official sample of the RedPajama dataset and compare algorithms in equal conditions, with a few exceptions. Specifically, all algorithms except OneBit use samples  from the RedPajama dataset. For some of these algorithms (QuIP# and one-shot algorithms), their implementation cannot handle as much data within 1TB RAM due to storing activations in memory. For these algorithms, we used the largest possible dataset size and tested that the algorithm reaches saturation (i.e. adding data does not improve accuracy). Finally, the OneBit baseline uses a custom dataset described in its original paper. This is because the code for that specific algorithm was not available at the time and we had to use the existing results. We describe these details further in appendices H and L.\n\n> (Q1) If we need to perform large-scale fine-tuning to achieve model quantization, why not maintain full precision during this large-scale fine-tuning and only apply low-precision quantization techniques after full precision fine-tuning?\n\nQuantized fine-tuning works because it allows layers to adjust each other’s quantization errors. In contrast, fine-tuning an uncompressed model in full precision (e.g. by cross-entropy) may improve general model quality, but it does not compensate for the quantization error. Note that finetuning is much shorter than pretraining of the original model. \n\nTo better demonstrate the difference between the two strategies, we run an additional experiment where we fine-tune the model in full precision and quantize it afterward. For this experiment, we substitute the KL divergence loss with cross-entropy on the data. This is because the KL divergence between two identical full precision models is always zero. We run this experiment with the Llama 2 7B model using 2-bit vector quantization in the same setup as in Section 4.3, and then quantize the model in one shot using AQLM. The results in Table 2 (PDF) show that this achieves some improvement over AQLM, but PV-Tuning still significantly outperforms this baseline. We hypothesize that the advantage of PV-Tuning is explained by the quantization error compensation during fine-tuning.\n\n[1] Frantar, E., et al. arXiv preprint arXiv:2210.17323 (2023).\n\n[2] Dettmers, T., et al. arXiv preprint arXiv:2306.03078 (2023).\n\n[3] Chee, J., et al. arXiv preprint arXiv:2307.13304 (2024).\n\n[4] Shao, W., et al. arXiv preprint arXiv:2308.13137 (2024).\n\n[5] Tseng, A., et al. arXiv preprint arXiv:2402.04396 (2024).\n\n[6] Egiazarian, V., et al. arXiv preprint arXiv:2401.06118 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWESc2amFO", "reviewer": "Reviewer_5n4W", "summary": "The paper focuses on fine-tuning techniques over compressed weights and proposes PV-Tuning, a general framework that improves existing fine-tuning strategies and provides convergence guarantees. Experiments show that PV-Tuning outperforms prior techniques and achieves the first Pareto-optimal quantization for Llama-2 models at extreme compression. The idea is novel and reveals the importance of fine-tuning strategies.", "questions": "1. In the experiment part, besides the PPL and accuracy, is there any evidence that supports the STE is noisier than PV-Tuning? It would be more convincing if the paper could provide some experimental evidence. \n2. From Table 1, it seems the naïve Linearized PV without subspace doesn’t achieve pleasing performance while subspace Linearized PV performs well. Does this mean that subspace is the key to performance improvement? Can you combine subspace and STE to conduct some ablation experiments?", "limitations": "Please see the Weaknesses and Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The technical contribution of the work seems solid. In particular, the authors conduct extensive experiments to demonstrate the effectiveness of PV-Tuning coupled with the state-of-the-art quantized representations.", "weaknesses": "1. The paper introduces the principle and specific implementation of PV-Tuning in detail, but does not compare the differences in principle and specific implementation with the existing STE method and Stochastic Rounding method. It would be beneficial to include such a comparison for clarity.\n2. Some notations in the paper may be confusing. In part 3.1, the expression “P(y) ⊇ P(x)” is used but it does not mean that P(x) is a subset of P(y), which is confusing because the ⊇ symbol usually means to include. Also in this part, it would be clear if there are some simple examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer appreciates our technical contribution and extensive experiments. We do our best to address their concerns and answer questions below.\n\n> (W1) The paper introduces the principle and specific implementation of PV-Tuning in detail, but does not compare the differences in principle and specific implementation with the existing STE method and Stochastic Rounding method. It would be beneficial to include such a comparison for clarity.\n\nCurrently, we describe each of these approaches in appendices E and F respectively (L676–717), but we do not compare them directly in terms of the underlying principles and specific implementations.\n\nTo address this concern, we will add a dedicated comparison between these two strategies and PV-tuning. More specifically, we will highlight the additional memory requirements of STE and PV-tuning, compared to stochastic rounding, and the effects of the stochastic rounding noise on the optimization. As for the experimental differences, we compared PV-Tuning vs the two other baseline strategies in Section 4.2. We would welcome further suggestions for comparisons.\n\n> (W2) Some notations in the paper may be confusing. In part 3.1, the expression “$P(y) \\supseteq P(x)$” is used but it does not mean that P(x) is a subset of P(y), which is confusing because the $\\supseteq$ symbol usually means to include. Also in this part, it would be clear if there are some simple examples.\n\nWe do our best to formally define our notation, e.g. we explain the meaning of “$P(y) \\supseteq P(x)$” in L127-128. However, we agree that it would be better to avoid collision with existing set notation and will do so in the next revision. We currently consider changing this to $\\sqsupseteq$, but we would welcome additional suggestions. In the next revision of the paper we will also add examples that demonstrate this relation.\n\n> (Q1) In the experiment part, besides the PPL and accuracy, is there any evidence that supports the STE is noisier than PV-Tuning? It would be more convincing if the paper could provide some experimental evidence.\n\nIn preliminary experiments we observed that training STE results in model diverging after a certain number of steps, with point of divergence varying with random seed. To better quantify this effect, we have run several experiments with different random seeds and reported std and error bars in **Figure 2** in the attached PDF.\n>(Q2) From Table 1, it seems the naïve Linearized PV without subspace doesn’t achieve pleasing performance while subspace Linearized PV performs well. Does this mean that subspace is the key to performance improvement? Can you combine subspace and STE to conduct some ablation experiments?\n\nIndeed, the subspace optimization during V step is the key to our performance improvements. From a practitioner’s standpoint, we introduce the general PV framework in order to show why and how subspace descent during V step can help with the optimization. We elaborate on this reasoning in lines 208-210 and 224-227.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kefQ5K30nd", "reviewer": "Reviewer_p3Lv", "summary": "This paper proposes a new PTQ fine-tuning algorithm called PV tuning. Recent SOTA LLM PTQ works include fine tuning steps to recover the original model on top of the actual quantization step. Fine tuning has been shown to be an effective and relatively cheap to run (vs full QAT) way to improve quantization quality. However, fine tuning for LLM PTQ has not been well explored, in part due to it only being recently introduced and also LLMs being expensive to do things with. My understanding of PV tuning is that it performs alternating optimization on a quantized LLM. Specifically, it alternates between optimizing continuous parameters (codebook values, layernorms, etc) and what is essentially a form of coordinate descent on codebook assignments to weight matrix entries.", "questions": "see above", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- PV tuning appears to achieve strong empirical results on a wide variety of models. Notably, PV tuning achieves good performance on Llama 3, which GPTQ-based methods tend to do poorly on.\n- PV tuning does not seem too costly to run vs existing fine tuning methods and should be compatible with a wide range of \"base\" quantization methods.", "weaknesses": "- This paper is written in a dense and arcane way and is hard to follow. While there are some theorems, they do not appear to be very useful (eg 3.1 just claims that PV tuning converges, but without any statements on how good the final solution is or any bounds on error). Perhaps I am misunderstanding the paper completely, so it would be useful for the authors to give a TLDR version of the actual method.\n- How large is the subspace for the CD step? Iterating through all possible code assignment changes is exponential in the subspace size, so how do you perform the code assignment optimization in a tractable way?\n- The empirical results presented are based off of the AQLM quantization method. One main benefit of quantizing LLMs is to achieve speedups in memory bound scenarios. However, the AQLM configuration used for PV tuning is too slow for fast inference (relative to what the hardware *could* achieve) due to the size of the codebook. How well does PV tuning perform with a smaller codebook (eg a 10 bit 8D codebook, which would use 16KB)? The paper also mentioned PV tuning could be applied to QuIP#, which does achieve near memory bandwidth. Do you have any experiments showing this? QuIP# is based off of GPTQ which performs poorly on Llama 3, does PV tuning fix this if applied to QuIP# after GPTQ?\n- I started reviewing this paper a few weeks ago and lost the first copy I was marking up, but I remember there were some inconsistencies in the result tables with numbers reported in the baselines, and some of the baseline results in 4.1 seemed a bit suspicious. Please check that your numbers are indeed correct. The pareto optimality claim is also questionable, since the \"frontier\" is just for PV tuning. A better comparison would be PV tuning @ 2 bits vs FP16 @ 3 bits, and it doesn't seem (although I didn't plot this) that PV tuning is actually better at 2 bits than FP16 at 3 bits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and address their concerns below.\n\n> (W1) This paper is written in a dense and arcane way and is hard to follow. While there are some theorems, they do not appear to be very useful (e.g. 3.1 just claims that PV tuning converges, but without any statements on how good the final solution is or any bounds on error). Perhaps I am misunderstanding the paper completely, so it would be useful for the authors to give a TLDR version of the actual method.\n\nWhile our main results are practical, we agree that our theoretical results can be presented more clearly, and we will do our best to improve presentation. For instance, Theorem 3.1 proves that PV-Tuning converges to a stable solution in order to contrast our method against Straight Through Estimation and Stochastic Rounding, both of which lack this property. We agree that proving tight convergence bounds would be valuable, but it is difficult to prove such bounds for our mixed continuous-discrete optimization problem–intuitively, the additional constraints make the problems “harder” than standard convex/non-convex optimization. We see this as an important direction for future work.\n\n> (W2) How large is the subspace for the CD step? Iterating through all possible code assignment changes is exponential in the subspace size, so how do you perform the code assignment optimization in a tractable way?\n\nThe size of the subspace $\\tau$ is such that the update satisfies  $\\|x^{k+1} - x^k\\| / \\|x^k\\| \\leq 0.01$, also known as the trust ratio (see Section 3.3). We choose this subspace using a simple heuristic: we select $\\tau$ parameters with the largest gradient of the original loss function. Further details can be found in Section 3.4. In practice, choosing the subspace takes negligible time, compared to accumulating the gradients w.r.t. LLM parameters.\n\n\n> (W3) The empirical results presented are based on the AQLM quantization method. One main benefit of quantizing LLMs is to achieve speedups in memory bound scenarios. However, the AQLM configuration used for PV tuning is too slow for fast inference\n\nWe have also presented results for, e.g. the GPTQ format in our paper, although we indeed focused primarily on AQLM as it is one of the best methods in terms of model size vs accuracy. Fortunately, VQ and AQLM still offer speedups in memory-bound scenarios. We discuss this briefly in L334-337 and report speedups in Appendix M.\n\n> (W3) Do you have any experiments showing this? QuIP# is based off of GPTQ which performs poorly on Llama 3, does PV tuning fix this if applied to QuIP# after GPTQ?\n\n\nWe have chosen AQLM over QuIP# because it outperformed the latter in WikiText2 perplexity (fig. 2). We have looked into integrating with QuIP#’s code base, but, due to its complexity, we are afraid that it won’t be possible to evaluate QuIP#’s performance with PV-tuning due to the rebuttal’s time constraints. We will try to add results in this direction for the next revision. \n\n> (W4) The result tables with numbers reported in the baselines, and some of the baseline results in 4.1 seemed a bit suspicious. Please check that your numbers are indeed correct.\n\nWe would appreciate it if the reviewer could make this concern more specific. Below we respond based on our best guess about what the review refers to.\n\nThere are two important causes why baseline numbers may seem odd. First, some of the baseline papers, notably [1] use a different way of computing the perplexity score, which is not compatible with most prior works. To compare these approaches, we reevaluated the methods from that paper using their official code. Second, for every paper that uses a different dataset or less data, we reevaluated these methods with the same sample of RedPajama we used (excluding OneBit, see L874-879). Please see Appendices J and L for a detailed experimental configuration. We verify all numbers that we report as a standard procedure, and will verify them again in the final revision.\n\n\n\n> (W4) The pareto optimality claim is also questionable, since the \"frontier\" is just for PV. A better comparison would be PV tuning @ 2 bits vs FP16 @ 3 bits, and it doesn't seem (although I didn't plot this) that PV tuning is actually better at 2 bits than FP16 at 3 bits.\n\nWe use the same method for evaluating pareto optimality as in [2], the same method is used in follow-up works. Unfortunately, we don’t understand what “FP16 @ 3 bits” stands for. We encourage the reviewer to clarify their suggestion (e.g. by editing their review) so we can apply it in the final version of the paper.\n\n\n[1] Huang, W., et al. arXiv preprint arXiv:2404.14047 (2024).\n\n[2] Dettmers, T., Zettlemoyer L. arXiv preprint arXiv:2212.09720 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iAOXfs3EZ7", "reviewer": "Reviewer_NP8P", "summary": "This paper proposes a novel PV-tuning algorithm for extreme compression of LLMs. In each iteration, two steps (P step and V step) are performed, each aimed at reducing the loss function. Different than existing methods that use straight-through estimator (STE), PV-tuning employs a subspace search strategy at the V step to enable large updates and overcome the training stagnation caused by discreteness. Extensive experiments demonstrate that the proposed method achieve superior accuracy on models such as LLaMA and Mistral.", "questions": "1. In my view, the PV algorithm seems similar to alternating minimization. Is there a connection between these two algorithms? \n\n2. Why is the P step an unconstrained minimization problem, as there is a constraint $P(y) \\subseteq P(x)$?\n\n3. Is the linearized V step described in section 3.2 equivalent to projected gradient descent? What is the computational cost to compute the projection onto {$x: V(x) \\subseteq V(y)$} in the V step ?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed PV framework comes with theoretical guarantees. \n\n2. The subspace descent in the V step is a novel approach to overcome the stagnation issue caused by discreteness, and it appears to work better than the popular STE method.\n\n2. PV-tuning is validated through extensive experiments and consistently outperforms all existing methods for 1-bit and 2-bit vector quantization.", "weaknesses": "1. PV-tuning is computationally more expensive than existing PTQ methods and requires more GPU resources on larger models. Could you report the runtime of PV-tuning in your expiriments?\n\n2. The authors claim that STE-based optimization is not well justified and leads to poor practical performance. However, I am aware of a paper [1] that provided the theoretical justification for STE approach.\n  - [1] Yin el al. Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets, ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and we are glad that they appreciate our technical contribution and experiments. Below we do our best to address their concerns and answer questions.\n\n\n> (W1) PV-tuning is computationally more expensive than existing PTQ methods and requires more GPU resources on larger models. Could you report the runtime of PV-tuning in your experiments?\n\nWe acknowledge the fact that PV-tuning requires significant computation. PV-tuning is positioned between PTQ (post training quantization) methods (which do everything in one-shot on a small amount of data) and QAT (quantization-aware training) methods (which can require even more significant amounts of data and computation): we use little data, but need to be able to fine-tune full blocks or even the entire model. The improved accuracy should offset this additional cost.  \n\nYet, we stress that the runtime of the finetuning procedure is still negligible relative to LLM training. Specifically, PV-tuning of Llama-2-7b and Llama-2-70b on a single machine with 8 A100 GPUs took 90 hours and 171 hours, respectively (the 70b model was trained with a smaller number of iterations). This amounts to ~1000 gpu-hours.  While this is not negligible, our fine-tuning time is still orders of magnitude smaller than the time it took to train the model: for instance, training Llama-2-7b and Llama-2-70b model took 184’320 hours and 1’720’320 gpu-hours, respectively [3, 4].\n\n> (W2) The authors claim that STE-based optimization is not well justified and leads to poor practical performance. However, I am aware of a paper [1] that provided the theoretical justification for STE approach.\n\nThe referenced work presents important insights about the application of STE to the training algorithm, together with the set of prescriptions for a proper choice of the STE. However, we believe that the setup in [1]  is very different from the one considered in our paper. Firstly, **this work applies STE for non-differentiable activations, rather than quantized model weights**. Moreover, their experiments are done on small scale networks and datasets (small MLPs and CNNs on MNIST/CIFAR10) and the conclusion may not transfer to LLMs and large datasets. There are, however, works that study the STE applied to weight quantization, such as [1, 2]: these works found that the standard (deterministic) STE does not converge to the optimum and propose alternatives that do converge, but introduce noise similar to the stochastic rounding that we evaluate in Section 4.2 (see Appendix F for details).\n\n> In my view, the PV algorithm seems similar to alternating minimization. Is there a connection between these two algorithms? \n\nYes, the PV-Tuning algorithm can be seen as a particular case of alternating minimization / coordinate descent, where one alternates between optimization of discrete and continuous parameters.\n\n> Why is the P step an unconstrained minimization problem, as there is a constraint $P(y) \\subseteq P(x)$?\n\nWe meant that the optimization problem for P-step can be reparameterized as an unconstrained optimization problem on the $c$ unique values in the weight matrix. From a practical viewpoint, we declare the $c$ “clusters” or codebooks as an optimized variable, and use an automated differentiation engine (i.e. PyTorch) to minimize the loss as a function of these values. We will clarify this in (L161) in the next revision of the paper.\n\n> Is the linearized V step described in section 3.2 equivalent to projected gradient descent? What is the computational cost to compute the projection onto {$x: V(y) \\subseteq V(x)$} in the V step?\n\nThere is indeed some similarity to the projected gradient descent, applied to a discrete space of quantization codes. In this interpretation, the projection operation is finding the nearest quantized value (in terms of L2) out of a fixed-size set. For non-vector quantization (e.g. GPTQ in Section 4.2), this projection is done by simple rounding, which is a constant time per weight, after accounting for the quantization scale and zero point (see L294). As for the vector quantization, the projection is done by searching the set of all vectors in the current quantization “codebook” (L295-296). Note that this projection is only computed for a small fraction of weights in the current subspace (Section 3.3).\n\n[1] A. Shekhovtsov and V. Yanush. Reintroducing straight-through estimators as principled methods for stochastic binary networks. In DAGM German Conference on Pattern Recognition, pages 111–126. Springer, 2021.\n\n[2] M. Spallanzani, G. P. Leonardi, and L. Benini. Training quantised neural networks with STE\nvariants: the additive noise annealing algorithm. In Proceedings of the IEEE/CVF Conference\non Computer Vision and Pattern Recognition, pages 470–479, 2022.\n\n[3] Touvron, Hugo, et al. \"Llama 2: Open foundation and fine-tuned chat models.\" arXiv preprint arXiv:2307.09288 (2023).\n\n[4] Dubey, Abhimanyu, et al. \"The Llama 3 Herd of Models.\" arXiv preprint arXiv:2407.21783 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t31nrn5rqz", "reviewer": "Reviewer_APHA", "summary": "The paper provides a new QAT algorithm for extreme compression of LLMs under various discrete weight representations (including vector quantization,  uniform quantization). Traditional optimization techniques rely on straight-through estimation (STE) for optimization in the presence of discrete parameters. This paper provides an alternative to QAT called PV-tuning which resembles the EM algorithm, and performs alternating optimization.   \n\nWhen specialized to standard uniform quantization, the algorithm does the following: in the P step, the scale parameter is updated using backprop, and V step performs rounding to the nearest scaled integer. One important aspect of the algorithm is that the V step only  updates a few coordinates at a time. This is mainly done to ensure the algorithm doesn't get stuck at a sub-optimal point. \n\nExperimental results show that this new optimization algorithm leads to better quality models than existing PTQ techniques such as QuIP, AQLM, GPTQ on diverse models. Experiments on Llama 2 7B also show that this technique improves upon STE. In terms of compute efficiency, the proposed algorithm is 1.5x slower than standard fine-tuning.", "questions": "- It would be great if the authors can provide some explanation for why PV tuning outperforms STE for uniform quantization. \n   - For the case of uniform quantization, it looks like PV tuning does some form of alternating minimization for learning the scale parameters and the quantized weights. In contrast, STE jointly optimizes for both scale parameters and quantized weights. Intuitively, shouldn't joint optimization perform better than alternating optimization? \n- It looks like PV tuning only updates a few coordinates in each iteration. Doesn't this slow down the convergence of the algorithm and increase the training cost (for each backprop we would be updating very few coordinates)?\n- If tau is set dynamically (as described in line 304), wouldn't it hurt the performance of optimizers like Adam which maintain coordinate-wise gradient statistics?\n- It looks like in experiments, STE is applied on top of PV-tuning. But this algorithm is never described in the paper. It would be great if the authors can provide some details about this algorithm?", "limitations": "See weaknesses and questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Improving STE is a very important problem and is key to making progress on extreme compression of LLMs.  The current paper aims to tackle this problem. So, I believe the broad direction is interesting to the community.\n\nThe proposed algorithm has an EM flavor, which is known to work well for estimating latent variables models. I found it quite interesting that the proposed algorithm seems to out-perform STE for 2-bit compression of Llama 7B. That being said, there are certain weaknesses in the empirical evaluation that needs to be addressed.", "weaknesses": "- **Clarity** The presentation (especially the algorithmic, technical details) in the paper needs to be significantly improved. Instead of the non-uniform quantization example used in the paper, I would suggest using a more relevant example such as uniform quantization or vector quantization. In its current form, I found it hard to wrap my head around how the proposed algorithm can be used for vector quantization and uniform quantization (I had to look into the appendix to figure this out). This is probably because there is no single framework that can unify all the weight representations. Different representations require different definitions of P, V.\n  - without the clean presentation, it is very hard to really understand how the proposed technique compares with existing QAT algorithms such as STE, at a conceptual level. \n- **Experimental Evaluation** Better experimental evaluation is needed to understand the real utility of the proposed approach. To be precise, the experiments in Table 2 compare PV tuning with PTQ techniques. But it is well known that QAT outperforms PTQ techniques. So, a better baseline to compare in Table 2 is QAT using STE (or other state of the art QAT techniques).\n- **Related Work** Uniform quantization, as done by GPTQ and QuIP, has been well studied in the literature. And several QAT techniques that improve upon STE have been developed for uniform quantization (packages such as Pytorch, JAX, TF have really good implementations of QAT for uniform quantization). It's a bit surprising that none of these works are discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and address their concerns below.\n> (Clarity) The presentation (especially the algorithmic, technical details) in the paper needs to be significantly improved.\n\nWe agree with your comments on the presentation, and will try to make the algorithm definition easier to follow. Non-uniform scalar quantization was chosen for illustration as a non-trivial example with multiple continuous (1-d codebooks, locations of the quantization points) and discrete (1-d codes, codebook assignments) parameters, being at the same time simpler than multidimensional quantization. PV framework can be easily extended to various quantization formats. Specifically, for symmetric uniform scalar quantization there is only a single continuous parameter (P) - quantization scale and discrete parameters (V) are the integers on a line segment. For vector quantization, continuous parameters (P) form a set of vectors (codebooks), and discrete parameters (V) are the indices corresponding to a vector from the codebook. \n\n> (Experimental Evaluation) Better experimental evaluation is needed to understand the real utility of the proposed approach. To be precise, the experiments in Table 2 compare PV tuning with PTQ techniques. But it is well known that QAT outperforms PTQ techniques. So, a better baseline to compare in Table 2 is QAT using STE (or other state of the art QAT techniques).\n\nWe provide a comparison with QAT methods based on STE and Stochastic Rounding in Table 1, Section 4.2. PV-tuning outperforms STE for all quantization methods considered (GPTQ, VQ, AQLM). Therefore, in the following comparison in Table 2 we adopted PV-tuning as the most performant fine-tuning strategy and focused on comparing against established baselines. However, if the reviewer sees a better way to arrange these results, we will welcome their suggestion (through an update on openreview).\n> (Related Work) Uniform quantization, as done by GPTQ and QuIP, has been well studied in the literature. And several QAT techniques that improve upon STE have been developed for uniform quantization (packages such as Pytorch, JAX, TF have really good implementations of QAT for uniform quantization). It's a bit surprising that none of these works are discussed in the paper.\n\nWe agree with the concern and will update Section 2 to discuss these techniques, such as  [1, 2,3,4] and other works. To the best of our knowledge, PyTorch and TF use vanilla STE in their QAT implementation, which we have already compare against. We will investigate this further for the final version.\n\nQuestions\n\n> (Q1) It would be great if the authors can provide some explanation for why PV tuning outperforms STE for uniform quantization. For the case of uniform quantization, it looks like PV tuning does some form of alternating minimization for learning the scale parameters and the quantized weights. In contrast, STE jointly optimizes for both scale parameters and quantized weights. Intuitively, shouldn't joint optimization perform better than alternating optimization?\n\nWe do our best to explain this below and will incorporate this in the final version of the paper. When comparing linearized subspace PV (our practical algorithm) and STE, the main difference is that STE tries to update all weights on every step, whereas our algorithm only updates a chosen subset. We explain why updating all weights is problematic in L192-204. In L218-221 and Appendix G, we explain how choosing a subset of weights helps with discrete optimization. In practice, STE can improve the model during the first steps, but eventually diverges without reaching the optimal solution. Furthermore, as we describe in 787-788, PV-tuning can also do P and V steps simultaneously.\n\n> (Q2) It looks like PV tuning only updates a few coordinates in each iteration. Doesn't this slow down the convergence of the algorithm and increase the training cost (for each backprop we would be updating very few coordinates)?\n\nThe V step in PV-tuning performs large updates on the discrete parameters. Constraining updates to a subset of parameters is necessary for stable convergence of the algorithm. Nevertheless, PV-tuning with part of the parameters updated exhibits steadier and faster convergence compared to STE\n\n> (Q3)If tau is set dynamically (as described in line 304), wouldn't it hurt the performance of optimizers like Adam which maintain coordinate-wise gradient statistics?\n\nWe agree that updating a varying fraction of hyperparameters on each step makes the optimization problem more intricate. However, the preconditioning used in Adam and similar preconditioned gradient descent method is only a coarse approximation of a true curvature, thus the performance could be quite robust to inaccuracies in preconditioner estimates.\n\n> (Q4)It looks like in experiments, STE is applied on top of PV-tuning. But this algorithm is never described in the paper. It would be great if the authors can provide some details about this algorithm?\n\nPV-tuning is proposed as an alternative to STE for continuous and discrete optimization. In Table 1 we provide a comparison between PV-tuning and STE and show that PV-tuning outperforms STE.\n\n\n[1] Zhou, Shuchang, et al. (2016), arXiv:1606.06160\n\n[2] Chen, S. et al. (2019), Metaquant: Learning to quantize by learning to penetrate non-differentiable quantization. NeurIPS 2019\n\n[3] A. Shekhovtsov and V. Yanush (2020), arXiv:2006.06880 \n\n[4] M. Spallanzani et al. (2022), arXiv:2203.11323", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing valuable comments and suggestions. We are glad that the reviewers appreciate our strong empirical results (3n7c, 5n4w, p3Lv, NP8P) including the latest models (p3Lv). Reviewers also highlight our technical contribution (5n4w, 3n7c, NP8P) and theoretical guarantees (NP8P).

We provide an overview of the individual responses below:

**(R3n7c) Comparing the effects of each method with and without fine-tuning**

To facilitate this comparison, we evaluated the algorithms missing from Figure 2 of our submission (GPTQ and SpQR) and rearranged their presentation. We present these results in two ways. First, we report the performance of each method with and without fine-tuning in Table 1 (PDF). We also provide plots where we compare all methods, with and without fine-tuning, to provide global context (Figure 1 in response PDF). The reviewer also suggests additional experiments with different fine-tuning methods, which we address in the individual response.

**(R3n7c) why not maintain full precision during this large-scale fine-tuning and only apply low-precision  quantization techniques after full precision fine-tuning?**

While we detail our reasoning in the direct response, we generally agree that it is interesting to test this empirically. To that end, we conducted full-precision fine-tuning and quantized the fine-tuned model as requested. The resulting perplexity scores can be found in Table 2 in the attached PDF (3n7c). Overall, it appears that SoTA LLMs are already well tuned on general data (that we use for calibration); fine-tuning them further lead to negligible improvements on some tasks to the small detriment of others. In turn, PV-Tuning also does not improve the original model quality, but it allows the quantized layers to “re-adjust” to each other’s quantization errors, leading to better accuracy. We thank the reviewer for this interesting suggestion. 

**(R5n4W) Verifying the observation that STE is noisier than PV-Tuning**

To better quantify this effect, we ran each algorithm 5 times with different random seeds  and reported the adjusted standard deviation for each iteration. Since running with multiple random seeds takes a lot of time, we chose to run these experiments using a smaller TinyLlama-1.1B LLM using 2-bit compression.

For convenience, we arrange our results in a plot **(Figure 2 in the response PDF)**, where we report **WikiText-2 perplexity every 5 training steps**. The solid line represents the mean perplexity, and the surrounding pale “error bars” represent standard deviation (with the same scale). In this experiment, we use the same data and experimental configuration as in Section 4.2 and provide some additional description in the figure caption (see PDF). To summarize, STE shows similar early performance, but eventually diverges with an increasing noise magnitude.

**(p3Lv) How well does PV tuning perform with a smaller codebook (eg a 10 bit 8D codebook, which would use 16KB)?**

We reported a similar small 8-bit codebooks in the original submission (Section 4.2, Table 1, right). To further address the reviewer’s concern, we evaluate the requested configuration (a single 10 bit codebook with group size 8, about 1.25 bits per weight) with Llama 2 7B, using the same data and hyperparameters as in Section 4.2. We report the resulting metrics in the individual response to p3Lv and welcome further suggestions.

We hope that these new results alleviate reviewers’ concerns. If not, we encourage reviewers to comment and clarify their requests in the author-reviewer discussion phase.

Meta-review: The paper considers compression of LLMs through quantization, focusing on fine-tuning techniques over initially compressed weights so as to better approximate the performance of the unquantized model. The method PV-Tuning is developed and compared with existing baselines. 

The reviewers have noted that the method is novel, reveals the importance of discovering good fine-tuning strategies, extensive experiments validate the method and show improvement on baseline methods, and achieves good performance on Llama 3 (where GPTQ-based methods do not). 

The optimization is a mixed continuous-discrete case that complicates the solution. The PV-Tuning method is compared with and shown to outperform straight through estimation (STE), and some new insights into the problem and methods is gained as well. 

Although significant, the PV-tuning method seems to have complexity relative to existing fine-tuning methods and should be compatible with various initial quantization methods.  

The paper provides some convergence analysis and guarantees, contrasting with straight through estimation (STE) and stochastic rounding that generally lack such guarantees.  The reviewers noted that the performance analysis was not always clearly presented.

The rebuttal and discussion with reviewers has led to further interesting experiments and potential algorithmic improvements.
Decision: Accept (oral)