Title: QuIP: 2-Bit Quantization of Large Language Models With Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: Quantization

Large Language Models

Adaptive Rounding

Theoretical Guarantees
Abstract: This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from incoherent weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP.

QuIP: 2-Bit Quantization of Large Language Models With Guarantees

Abstract

This work studies post-training parameter quantization in large language models (LLMs). We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from incoherent weight and Hessian matrices, i.e., from the weights being even in magnitude and the directions in which it is important to round them accurately being unaligned with the coordinate axes. QuIP consists of two steps: (1) an adaptive rounding procedure minimizing a quadratic proxy objective; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence via multiplication by random orthogonal matrices. We complement QuIP with the first theoretical analysis for an LLM-scale quantization algorithm, and show that our theory also applies to an existing method, OPTQ. Empirically, we find that our incoherence preprocessing improves several existing quantization algorithms and yields the first LLM quantization methods that produce viable results using only two bits per weight. Our code can be found at https://github.com/Cornell-RelaxML/QuIP .

1 Introduction

Large language models (LLMs) have enabled advances in text generation, few-shot learning, reasoning, protein sequence modeling, and other tasks [2, 30, 35]. The massive size of these models-often reaching into hundreds of billions of parameters-requires sophisticated deployment methods and motivates research into efficient inference algorithms.

This work studies the post-training quantization of LLM parameters as a way to improve their runtime efficiency [4, 8, 22, 31, 33, 34]. Our key insight is that quantization can be most effective when weight and proxy Hessian matrices are incoherent -that the weights themselves are even in magnitude, and the directions in which it is important to have good rounding accuracy are not too large in any one coordinate. Intuitively, incoherence can be thought of as a principled form of outlier reduction, which makes it easier to adaptively round the weights to a finite set of compressed values. We use this intuition to develop theoretically sound two-bit quantization algorithms that scale to LLM-sized models.

Specifically, we introduce quantization with incoherence processing (QuIP), a new method motivated by the above insight. QuIP consists of two steps: (1) an adaptive rounding [20] procedure, which minimizes a quadratic proxy objective ℓ ( ˆ W ) = tr(( ˆ W -W ) H ( ˆ W -W ) T ) of the error between the original weights W and the quantized weights ˆ W using an estimate of the Hessian H ; (2) efficient preand post- processing that ensures that the weight and Hessian matrices are incoherent by multiplying

them by a Kronecker product of random orthogonal matrices. We denote 'incoherence processing' as both the pre- and post- processing steps of our procedure. Incoherence processing can be viewed as a form of outlier suppression across the weights and the activation space.

We complement our method with a theoretical analysis-the first for a quantization algorithm that scales to LLM-sized models-which analyzes the role of incoherence and shows that our quantization procedure is optimal within a general class of rounding methods. Interestingly, we find that QuIP without incoherence processing yields a more efficient implementation of an earlier algorithm, OPTQ [8]; our paper thus also provides the first theoretical analysis for that method.

Empirically, we find that incoherence processing greatly improves the quantization of large models, especially at higher compression rates, and yields the first LLM quantization method that produces viable results using only two bits per weight. For large LLM sizes (>2B parameters), we observe small gaps between 2-bit and 4-bit compression that further decrease with model size, hinting at the feasibility of accurate 2-bit inference in LLMs.

Contributions. In summary, this paper makes the following contributions: (1) we propose QuIP, a quantization method based on the insight that model parameters should ideally be incoherent; (2) we provide a theoretical analysis for a broad class of adaptive rounding methods that encompass QuIP and OPTQ; (3) we demonstrate that QuIP makes two-bit LLM compression viable for the first time.

2 Related Work

Adaptive rounding. Nagel et al. [20] are the first to motivate the 'adaptive rounding' proxy objective (Eq. (1)) in a principled way. There are many quantization methods which quantize by optimizing this proxy objective [5, 6, 9, 12, 14, 20, 32]. Many require further retraining which can be expensive, and are not evaluated on the current largest open LLMs (OPT [35], BLOOM [30]). Lybrand and Saab [15] propose a greedy per-neuron quantization procedure that is similar to ours, except they do not consider arbitrary linear functions of the error correction. Their work bounds the proxy objective, albeit on the first layer only.

Post training quantization in large models. There is a growing body of work on PTQ in LLMs such as OPT and BLOOM. The size of these models make it difficult to apply previously developed methods. The majority of these methods make quantization easier by somehow reducing the range of weights or activations, but still use nearest rounding. SmoothQuant [31] rescales between activations and weights to remove outliers from the activations and make quantization overall easier. ZeroQuant [33] proposes a per-layer knowledge distillation method. LLM.int8() [4] decompose matrix multiplications into a majority of 8 bit and a minority of 16 bit operations. LUT-GEMM [22] designs kernels to accelerate quantized matrix multiplications. RPTQ [34] reorders activations and quantizes them in groups, reducing the impact of range differences between channels.

OPTQ(Formerly known as GPTQ). OPTQ [8] is based on OBQ [7], and proposes a novel rounding method that can work on the largest OPT and BLOOM models. The method works iteratively over the weight columns in a fixed order: (1) quantize with nearest rounding and compute the error, (2) update the remaining weights with a scaled error, and (3) repeat.

Other quantization methods. There are other quantization procedures which do not round based on the proxy objective of [20], or are not designed for the largest language models [10, 11, 13, 19, 28, 29].

3 Quantization With Incoherence Processing: Adaptive Rounding Step

This section introduces quantization with incoherence processing (QuIP), a new method consisting of: (1) an adaptive rounding step; (2) efficient pre- and post-processing that ensures weight and Hessian incoherence. We define and analyze step (1) in this section; the next section focuses on step (2).

Following existing state-of-the-art post-training quantization methods, we round weights per-layer by minimizing the 'adaptive rounding' proxy objective, as in Nagel et al. [20],

Here, W ∈ R m × n is the original weight matrix for a given linear layer, ˆ W ∈ R m × n are the quantized weights, x ∈ R n is an input vector drawn uniformly at random from a calibration set, and H is the

second moment matrix of these vectors, interpreted as a proxy Hessian. Crucially, this formulation lets the quantization be run in parallel across neurons, which is tractable for large language models [8]. For simplicity, we will focus in this section on rounding to the integers; subsequent sections will extend the analysis to finite grids.

3.1 LDLQ: An Optimal Adaptive Rounding Method

Our strategy is to define a family of adaptive rounding methods for optimizing objective (1) and then define LDLQ, the optimal method within that class. Our defined methods iteratively perform the following update for k = 1 , 2 , ..., n :

where W k denotes the k -th column, W 1:( k -1) denotes the first k -1 columns, the subroutine Q denotes either nearest rounding or standard unbiased rounding to the integers (which rounds up or down such that E [ Q ( z )] = z ), and a k ∈ R k -1 is some sequence of vectors. This scheme rounds columns one at a time; at each step, it adds a 'correction' term that is a linear function of the residual from the rounding we have done so far. The final ˆ W satisfies the following matrix equation:

where U is a strictly upper-triangular matrix whose columns are the vectors a k and Q acts elementwise. Because U is upper-triangular, ˆ W k only depends on ˆ W 1:( k -1) .

If we let η = Q ( W +( W -ˆ W ) U ) -( W +( W -ˆ W ) U ) denote the quantization error of Q , we find that ˆ W -W = η ( U + I ) -1 and we can rewrite objective (1) as

The LDLQ Method How should we specify U , the linear feedback from the quantization error of preceding columns in (2)? Equation 3 provides an answer. If we choose U ← ` U such that the LDL decomposition of H is

where D is a (non-negative) diagonal matrix and ` U is upper unit triangular, then the terms ( U + I ) in Eq. (3) cancel. We denote as LDLQ the rounding procedure in Eq. (2) with U ← ` U as the LDL assignment from Eq. (4). We will now see that the LDL assignment of U is in fact optimal.

3.2 Deriving the Optimality of the LDLQ Adaptive Rounding Procedure

In order to reason about optimality, we consider weights which are worst and average-case for the proxy loss. Let A denote a rounding method, and let A ( W,H ) be the resulting quantized weights. Define the worst-case ( L worst ) and average ( L avg ) proxy losses with respect to the input weights as

Theorem 1. LDLQ is worst and average-case optimal amongst rounding methods which specify the linear feedback U as a function of H (not of W ), and when rounding to the integers. That is, for all rounding methods A in the class described by Eq. (2) , for all positive semi-definite H , and for Q as either nearest or stochastic rounding,

where D is the matrix from the LDL decomposition of H , and c = 12 for nearest, c = 6 for stochastic.

Remarks. The number of rows being quantized is m , and each quantization method operates across the n entries of each row. For all rounding methods described by Eq. (2), and for all positive semidefinite H , Q as nearest rounding achieves the same worst-case proxy loss as stochastic rounding, but achieves better average proxy loss.

0

500

1000

1500

2000

2500

10

5

10

4

10

3

10

2

10

1

10 0

Normalized eigenvalues

Block 16 k_proj

Block 20 q_proj

Block 30 fc1

0.1

0.2

0.3

0.4

0.5

Before incoherence max |

Wij

|

0.0

0.1

0.2

0.3

0.4

0.5

After incoherence max |

Wij

|

0.2

0.4

0.6

0.8

1.0

Before incoherence max |

eigvec

(

H

)

ij

|

0.2

0.4

0.6

0.8

1.0

After incoherence max |

eigvec

(

H

)

ij

|

Moving beyond a generic algorithm A within our framework, we consider the common baselines of nearest and stochastic rounding. These methods are represented within our framework by choosing the appropriate Q subroutine, and setting all entries of the linear feedback to zero.

For these baseline methods, their optimality gap to LDLQ is governed by tr ( D ) vs. tr ( H ) . For any non-diagonal ˜ H ⪰ 0 , LDLQ achieves strictly lower worst and average-case proxy loss because tr ( D ) < tr( ˜ H ) . Let B = { Near , Stoch } . Then, L worst ( LDLQ , ˜ H ) < L worst ( Stoch , ˜ H ) and L avg ( LDLQ , ˜ H ) < L avg ( B , ˜ H ) . Across OPT models 125m to 2.7b, tr ( D ) / tr ( H ) ≤ 0 . 65 -empirically verifying that the gap is not insignificant. See Supplement C for full details.

3.3 Incoherence: Optimality with a Spectral Bound

Theorem 1 gives exact expressions for the proxy loss, albeit with tr ( D ) , which can be difficult to reason about. In Figure 1, we empirically observe that H is approximately low-rank: we visualize the spectrum of several randomly chosen H from OPT-2.7b, and observe that the spectrum decays rapidly. In fact, across all layers of OPT-125m to 2.7b models, a vast majority of H matrices have fewer than a quarter of eigenvalues > 1% of the max eigenvalue; see Supplement C for full details. Given this observation about the low rank of H , can we bound the behavior of LDLQ, and thus tr ( D ) , using the spectrum of H ?

We do this building on a variant of the incoherence assumption that is specialized to our case [3, 24].

Definition 1. We say a symmetric Hessian matrix H ∈ R n × n is µ -incoherent if it has an eigendecomposition H = Q Λ Q T such that for all i and j , | Q ij | = ∣ ∣ e T i Qe j ∣ ∣ ≤ µ/ √ n . By extension, we say a weight matrix W ∈ R m × n is µ -incoherent if all i and j , | W ij | = ∣ ∣ e T i We j ∣ ∣ ≤ µ ∥ W ∥ F / √ mn .

Note that 'most' n × n matrices are incoherent with µ = O ( √ log n ) = ˜ O (1) because a random orthogonal matrix has entries with squared-magnitudes that concentrate around their mean of 1 /n . Incoherence in W can be viewed as a form of outlier reduction: a small bound on the magnitude of its entries means that we do not need to scale it as much to make it fit in the finite range of representable low-precision numbers. Figures 2 and 3 plot the max absolute weight and hessian eigenvector entries before and after our incoherence processing, on all layers in OPT-2.7b. A line with slope=1 is drawn for reference. We see that W and H are more incoherrent after our incoherence processing is applied. Making H incoherent is less intuitive, but its utility is motivated by the following lemma.

Lemma 2. Let H ∈ R n × n be a µ -incoherent positive semi-definite symmetric matrix and let H = ( ` U + I ) D ( ` U + I ) T be its LDL Cholesky decomposition, where ` U is a strictly upper triangular matrix and D is a (non-negative) diagonal matrix. Then,

To the best of our knowledge, this is a novel result using incoherence to obtain a bound on tr ( D ) that depends only on the spectrum of H . To help interpret this result, we derive explicit proxy losses for plain nearest and stochastic rounding, which we will then compare to what LDLQ gets via Lemma 2.

Algorithm 1 QuIP - Incoherence Pre-Processing

Algorithm 2 QuIP - Incoherence Post-Processing

Require: b ∈ N , H ∈ R n × n SPD, quantized W ∈ [0 , 2 b -1] m × n , s ∈ R & ˜ D ∈ R n × n (Alg 1) 1: seeded sample random two-factor orthogonal matrices U ∈ R m × m and V ∈ R n × n 2: W ← s ∗ ( ( W/ (2 b -1)) ∗ 2 -1 ) 3: W ← U T WV ; H ← V T HV ▷ revert incoherence 4: return W ← W ˜ D -1 ▷ revert diagonal rescaling

Lemma 3. Let H be symmetric positive definite. In the worst case stochastic rounding achieves L worst ( Stoch , H ) = ( m/ 4) tr ( H ) . In the average case nearest and stochastic rounding achieve L avg ( { Near , Stoch } , H ) = ( m/c ) tr ( H ) , where c = 12 for nearest, and c = 6 for stochastic.

To interpret this result, consider H rankk with µ 2 k < n . By Cauchy-Schwarz, tr( H 1 / 2 ) 2 ≤ k tr ( H ) . Combining Lemma 2 with the LDLQ proxy losses of Theorem 1 and comparing with Lemma 3,

where B ∈ { Near , Stoch } , and c is as given in Theorem 1. This shows that for sufficiently low-rank H , LDLQ is asymptotically better than plain nearest and stochastic rounding by a factor of µ 2 k/n .

Without incoherence: no improvement with a spectral bound. By assuming incoherence, we were able to show LDLQ gets an asymptotically better bound in terms of just the spectrum of H . We might ask: was the incoherence assumption necessary to get this result? The following theorem answers this question in the affirmative by showing that without incoherence, the best spectral bound for LDLQ cannot differentiate it from the nearest and stochastic rounding baselines.

Theorem 4. Consider all ˜ H with the same spectrum as H . For any positive semi-definite H , the following holds. On the worst-case loss LDLQ achieves the same error as stochastic rounding,

On the average-case loss LDLQ achieves the same error as the corresponding rounding routine. Let B = { Near , Stoch } and c = 12 for nearest, c = 6 for stochastic.

Note that the worst case for comparing LDLQ against these baselines occurs when H is diagonal, see Theorem 1 and Lemma 3. Assuming incoherence as we do is a natural way to exclude such cases.

4 Quantization With Incoherence Processing: Incoherence Processing Step

Next, we leverage the above incoherence analysis to introduce incoherence processing , the second step of the QuIP algorithm. Our strategy will be to pre-process weight and Hessian matrices to ensure

the favorable incoherence properties outlined above. One straightforward way to make a symmetric matrix incoherent is to conjugate it by a uniform random orthogonal matrix: this will result in each of its eigenvectors being a random unit vector, whose entries will concentrate around magnitude n -1 / 2 .

Specifically, let U ∈ R m × m and V ∈ R n × n be two random orthogonal matrices. (Let's temporarily ignore how these matrices are generated, or how we would efficiently perform inference.) We ensure the weight and Hessian are incoherent with high probability through random orthogonal multiplications ˜ H ← V HV T and ˜ W ← UWV T . Importantly, this transformation preserves the proxy quadratic form since tr( ˜ W ˜ H ˜ W T ) = tr(( UWV T )( V HV T )( V W T U T )) = tr( WHW T ) .

4.1 Incoherence via Efficient Orthogonal Multiplication

If all we wanted to do was to store or transmit the weights of the quantized neural network, the above procedure would introduce no overhead, since we can generate a random orthogonal matrix from a seed-making it essentially free to store. However, for running inference on a DNN, we need to multiply by the weight matrix W , and here the need to manifest and multiply by n × n random orthogonal matrices U, V would be prohibitive.

To handle this, we propose to instead use a distribution over random orthogonal matrices for which multiplication is fast. Let n = pq be a factorization of n (where p ≈ q ≈ √ n ), and set U = U L ⊗ U R where U L is sampled uniformly from the p × p orthogonal matrices and U R is sampled uniformly from the q × q orthogonal matrices. Multiplication of a vector x ∈ R n by the matrix U can be accomplished by reshaping to a p × q matrix, multiplying on the left by U L and the right by U T R , and then reshaping back: this takes O ( n ( p + q )) = o ( n 2 ) operations. Using more than two factors in this way is also possible, but using two suffices to make this preprocessing asymptotically non-dominant.

Lemma 5. Let H be a positive semi-definite matrix on R n × n and W a matrix on R m × n , and suppose that m = p 1 · p 2 · · · p k and n = q 1 · q 2 · · · q k . Let U 1 , U 2 , . . . , U k , V 1 , V 2 , . . . , V k be independent random orthogonal matrices on R p i × p i and R q i × q i respectively. Set U as the Kronecker product U = U 1 ⊗ U 2 ⊗··· ⊗ U k and V as V = V 1 ⊗ V 2 ⊗··· ⊗ V k Then V HV T is µ H -incoherent with probability at least 1 -δ , and UWV T is µ W -incoherent with probability at least 1 -δ , where

for some global constants A and C independent of n and k .

Remarks. This lemma means that multiplying by a random matrix in this family suffices to make a matrix incoherent with parameter µ only poly-logarithmic in the matrix size. In our experiments we use k = 2 factors to construct the orthogonal matrices U, V .

4.2 Additional Heuristics

We outline QuIP pre-processing and post-processing in Algorithms 1 and 2, respectively. In line 5 of Algorithm 1, we apply the aforementioned fast orthogonal multiplication procedure to ensure W and H are incoherent. We also randomly permute entries at the fast matrix multiplication step to prevent any correlation between attention heads from worsening performance. We introduce a number of additional heuristic improvements that further improve performance.

Incoherence-Based Heuristics. Line 4 diagonally rescales W and H to minimize ℓ ( ˆ W ) ≈ tr ( H ) ∥ W ∥ 2 F , effectively trading off the spectrum of these matrices to find a minimum. Motivated by the incoherence of W , Line 6 computes the quantization range depending on the spectrum ∥ W ∥ F , instead of the typical max i,j | W ij | . Our full QuIP procedure is described in Algorithm 3, which contains calls to the pre- and post-processing sub-steps in Algorithms 1 and 2.

Greedy local search. Our basic procedure yields a good initial guess with error guarantees. We can further lower the proxy loss by running coordinate descent after LDLQ (but before post-processing), updating the weights in the same order as in the initial pass. See Supplement B for full details.

Algorithm 3 QuIP: Quantization with Incoherence Processing

5 Extensions and Further Analyses

5.1 OPTQ is a Special Case of LDLQ

We prove a novel theoretical insight: QuIP without incoherence processing (i.e., LDLQ) is equivalent to a more efficient version of the OPTQ algorithm. That is, OPTQ falls under our class of adaptive rounding procedures with linear feedback, and is within-class optimal.

Theorem 6. OTPQ [8] falls within the class of adaptive rounding procedures with linear feedback as described by Eq. (2) , and is equivalent to LDLQ in Section 3.

Remarks. To the best of our knowledge, this equivalence yields the first theoretical analysis of OPTQ. Even though the two methods are equivalent, LDLQ is more efficient. OPTQ's implementation requires a matrix inversion of H , and two Cholesky decompositions. Our implementation of LDLQ performs no matrix inversion, and only one Cholesky decomposition.

Empirical Verification. The quantized outputs of the OPTQ implementation [8] are shown to be exactly identical to the outputs of our LDLQ implementation. Synthetic random data was used, with W ∼ Unif[0 , 1] 1000 × 1000 . Full details can be found in Supplement C.

5.2 A Bound for Rounding to a Finite Grid

In Section 3, we saw that LDLQ (equivalently, OPTQ) is optimal for minimizing the adaptive rounding objective. However, this analysis assumed rounding to the integers. In practice, we do not want to round W just to the integers, but instead to scale it, shift it, and round it a finite subset corresponding to a b -bit integer. To do this, the 'real' LDLQ algorithm uses a clamp operation to restrict the range of quantized values. Is LDLQ still optimal when this small change is made? It turns out that the answer is no , as the following concrete example illustrates.

10 2

10 3

10 1

10 2

10 3

10 4

tr(

W

W

)

H

(

W

W

)

T

LDLQ (nearest)

nearest

stoch

LDLQ (nearest, no clamp)

matrix size

n

Finite Grid Counterexample. Figure 4 illustrates the behavior of LDLQ and other rounding methods-when restricted via clamping to a finite 4-bit grid [0 , 15] -on a particular example where H is a (cleverly chosen) small perturbation of ( I n + 1 n × n -e n e T n ) /n , and W has m = 16 and is a small perturbation of 1 m × n / 2 . Details of the setup appear in Supplement C. The figure shows that clamped LDLQ with nearest rounding is asymptotically worse, and the clamping to the finite grid is what causes it to be worse in this case.

Note that in our experiments in practice, OPTQ has been shown to soundly beat nearest rounding. This clamping issue does not seem to arise in practice; however, since it is possible we do need to take it into account to prove useful end-to-end bounds.

A Procedure With a Bound. In order to address the above issues in theory, here we describe a method that acts to restrict the value of | ˆ W ij -W ij | , so that the rounded weights will remain inside the grid if W is sufficiently far inside. We do this via the optimization problem with hyperparameter c

Our 'fixed' algorithm solves this convex problem (e.g. with ADMM), then runs QuIP using stochastic rounding and U = R -1 -I in place of the LDL decomposition. Observe that for sufficiently large c , this is exactly equivalent to base QuIP, since the solution of that optimization problem is given by the LDL decomposition when the constraint is dropped. Doing this (the full algorithm is given in the supplemental) yields the following theorem.

Theorem 7. Suppose that we run Algorithm 5 (Supplement) to quantize a matrix W ∈ R m × n by solving the objective (7) . Then there exists an assignment of the algorithm's hyperparameters c and ρ such that with probability at least 1 -δ , all the quantized weights will be in range (no overflow or need for clipping) and

In practice, because clamping rarely causes issues, and because of the significant additional compute needed to solve this program, we always just use QuIP as described in the previous sections, which is equivalent to setting c large and using nearest rounding.

6 Experiments

Overview. We quantize the OPT [35] family of models (up to 66B parameters) and Llama 2 70B [27] using various quantization and processing methods. QuIP is superior to OPTQ and other baselines across all model sizes and evaluation tasks. Most interestingly, incoherence processing yields excellent performance using as little as two bits per weight when paired with any of the quantization methods we consider (including nearest rounding). Two-bit quantization with QuIP is viable at even moderate model sizes (1B parameters), a regime where other two-bit quantization methods fail. At the largest model sizes, the difference between 2-bit and 16-bit weight performance becomes small. We compare the throughput of QuIP with OPTQ's efficient implementation on language generation and show that it is not much slower. Additional results on the effectiveness of the proxy loss, unbiased rounding, and Algorithm 5 are presented in the Supplement C.

Setup. The experimental infrastructure is built on top of OPTQ's [8] repository which is implemented in PyTorch [23]. We quantize the HuggingFace implementations of the OPT and Llama 2 model families. All models are quantized on a single GPU, with up to 48GB of memory. Our calibration set is the same as OPTQ; 128 random 2048 token segments from the C4 dataset [25] consisting of generic text data from crawled websites. Therefore, no task-specific data is viewed when quantizing. Following OPTQ, quantization is performed one Transformer block at a time: loaded into GPU memory, the Hessian computed, and then the weights quantized. The current block's inputs are then passed through the quantized block to produce inputs for the following block. The Hessian is computed from the quantized Transformer up to that point rather than from the full precision model; like OPTQ, we find this improves quantization. Further details on the setup can be found in Supplement C, including a description of the computational resources used to perform the experiments.

Methods. We evaluate compositions of several quantization and pre/post processing methods. For quantization methods, we evaluate nearest rounding, LDLQ (or OPTQ), and two variations. LDLQRG re-orders the weights based on diag( H ) to modify the quantization order and adds further greedy updates to the proxy. 'Greedy' performs the greedy updates only. We evaluate the baseline preprocessing from OPTQ which adds H ← H + α ∗ mean(diag( H )) I for numerical stability. We also evaluate our incoherence processing in Algorithms 1 and 2, denoted as 'IncP'. With this notation QuIP = LDLQ + IncP, and QuIP-RG = LDLQ-RG + IncP.

Datasets. We evaluate on the following language generation tasks: WikiText2 [17], Penn Treebank (PTB) [16], and C4. We also evaluate on zero-shot tasks, including LAMBADA (LAMB) [21], ARC Easy (ArcE) [1], PiQA [26], and StoryCloze (SC) [18]. See Supplement C for the full set of results.

Main Results. QuIP is the first PTQ procedure to achieve good quantization at two bits per weight, across a variety of LLM sizes and evaluation tasks. In Figure 5 we compare QuIP and OPTQ when quantizing to 2 and 3 bits per weight (4-bit quantization works equally well for both methods); we evaluate OPT models (up to 66B) on PTB, C4, ARC Easy, and LAMBADA. QuIP is superior to OPTQ across the model sizes and evaluation tasks. At three bits, QuIP matches the full precision model reasonably well. At two bits and for larger LLMs (>2B parameters), QuIP begins to approach the performance of the full precision model. As model size increases, so does the quality of QuIP's

10 1

10 2

10 3

10 4

Perplexity on PTB

10 1

10 2

10 3

10 4

Perplexity on C4

10

1

10 0

10 1

# params in billions

0.3

0.4

0.5

0.6

Accuracy on ArcE

10

1

10 0

10 1

# params in billions

0.0

0.2

0.4

0.6

Accuracy on LAMB

FP16

OPTQ-W3

OPTQ-W2

QuIP-W3

QuIP-W2

Table 3: Ablating sub-steps of QuIP's incoherence processing, see Algorithm 1. Perplexities are averaged over WikiText2, PTB, and C4 for OPT-350m.

2-bit quantization. We provide plots on the remaining datasets in Supplement C. Note that the dip in OPTQ on OPT-66B is documented in their paper.

Table 1 shows the results of quantizing Llama 2 70B using QuIP and OPTQ. Again, QuIP achieves good quantization at two bits while OPTQ does not.

Incoherence Processing Ablation. Table 2 shows all combinations of quantization and processing methods evaluated on OPT-30B. At lower weight bits, QuIP's incoherence processing dramatically improves the performance of all quantization methods, across all evaluation tasks. Remarkably, all quantization methods-even nearest-are viable at two bits with our incoherence processing. Our modifications in QuIP-RG sometimes give an improvement over QuIP, but further study is required to evaluate these modifications. Figures for OPT-125M to 13B are in Supplement C.

Throughput Comparison. We evaluate the additional overhead of our incoherence processing during model inference by modifying OPTQ's efficient forward pass. OPTQ's implementation contains a quantized-matrix full-precision-vector product kernel and was shown to offer speedups over a FP16 baseline. Our incoherence processing additions are performed in PyTorch. Table 4 shows that our QuIP implementation is about 1.5 × slower than OPTQ.

Further Ablation. QuIP's incoherence processing contains several sub-steps. Table 3 shows their relative contributions; all are necessary for the full improvement. Table 5 shows that the random permutation step within the fast orthogonal multiplication also significantly reduces perplexity.

7 Conclusion

This paper introduced quantization with incoherence processing (QuIP), an algorithm consisting of (1) an optimal adaptive rounding procedure which minimizes a quadratic proxy of the weight error, and (2) efficient pre- and post-processing to ensure the incoherence of the weight and Hessian matrices by multiplying them by a Kronecker product of random orthogonal matrices. We showed that QuIP quantization is optimal in a general class of adaptive rounding methods with linear feedback; this theoretical analysis is the first for any quantization algorithm that scales to LLM-sized models.

Empirically, QuIP achieves the first viable two-bit quantization results for LLMs, especially at large model sizes, hinting at the feasibility of accurate 2-bit inference in LLMs.

References

Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, and Michael Witbrock. A systematic classification of knowledge, reasoning, and context within the ARC dataset. In Proceedings of the Workshop on Machine Reading for Question Answering , pages 60-70, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-2607. URL https://aclanthology.org/W18-2607 .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et. al. Language models are few-shot learners. In Conference on Neural Information Processing Systems , 2020.

Christopher De Sa, Kunle Olukotun, and Christopher Ré. Global convergence of stochastic gradient descent for some non-convex matrix problems. In International Conference on Machine Learning . PMLR, 2015.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Conference on Neural Information Processing Systems , 2022.

Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In International Conference on Computer Vision , 2019.

Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. In Conference on Neural Information Processing Systems , 2020.

Elias Frantar, Sidak Pal Sing, and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. In Conference on Neural Information Processing Systems , 2022.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations , 2023.

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning . PMLR, 2021.

Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. Mr.biq: Post-training nonuniform quantization based on minimizing the reconstruction error. In Conference on Computer Vision and Pattern Recognition , 2022.

Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. In Conference on Neural Information Processing Systems , 2022.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations , 2021.

Yijian Liu, Huanrui Yang, Zhen Dong, Kurt Keutzer, Li Du, and Shanghang Zhang. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Conference on Computer Vision and Pattern Recognition , 2023.

Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In Conference on Neural Information Processing Systems , 2021.

Eric Lybrand and Rayan Saab. A greedy algorithm for quantizing neural networks. In Journal of Machine Learning Research , 2021.

Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The Penn Treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994 , 1994. URL https://aclanthology.org/H94-1020 .

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-1098. URL https://aclanthology.org/N16-1098 .

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In International Conference on Computer Vision , 2019.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning , pages 7197-7206. PMLR, 2020.

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144 .

Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. arXiv preprint arXiv:2206.09557 , 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Processing Systems , 2019.

Jain Prateek, Netrapalli Praneeth, and Sanghavi Sujay. Low-rank matrix completion using alternating minimization. In Proceedings of the Forty-fifth Annual ACM STOC , 2013.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .

Sandeep Tata and Jignesh M Patel. Piqa: An algebra for querying protein data sets. In International Conference on Scientific and Statistical Database Management , 2003.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng

Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.

Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning . PMLR, 2020.

Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. In Conference on Neural Information Processing Systems , 2022.

BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, and François Yvon et. al. Bloom: A 176b-parameter open-access multilingual language model, 2023.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438 , 2023.

Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W. Mahoney, and Kurt Keutzer. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning . PMLR, 2021.

Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Conference on Neural Information Processing Systems , 2022.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Luzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089 , 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

A Checklist

A.1 Broader Impacts

Our work pushes the quantization of large language models into the 2 bits per weight regime. Our aim is to drive foundational research on theoretical and empirical aspects of quantization. The ultimate goal is to enable more powerful LLMs to run more efficiently. However our work is unaware to what ends those LLMs are used.

A.2 Limitations

The adaptive rounding [3] proxy objective considers each layer in isolation; it remains to be seen what other computationally tractable proxies could improve quantization. For example quantization methods do exist which consider interactions between layers, but so far have been too computationally expensive to be applied to the largest open LLMS.

A.3 Experiments, Reproducibility

Our code is included in the Supplement. See the included README for instructions on how to reproduce the various experiments, including random seeds. The code also downloads all datasets used to quantize or evaluate the models.

B Additional Method Clarifications

B.1 Subsection 4.2 (Incoherence-Based Heuristics)

Line 4 diagonally rescales W and H to minimize ℓ ( ˆ W ) ≈ tr ( H ) ∥ W ∥ 2 F , effectively trading off the spectrum of these matrices to find a minimum. Note to minimize tr ( D -1 HD -1 ) ∥ WD ∥ 2 F = ( ∑ n i =1 H ii /D 2 i )( ∑ n i =1 D 2 i ∥ W i ∥ 2 ) implies that D i = √ H ii / ∥ W i ∥ . Motivated by the incoherence of W , Line 6 computes the quantization range depending on the spectrum ∥ W ∥ F , instead of the typical max i,j | W ij | . The parameter ρ controls the quantization range; we tune it and find that a value of 2.4 works well across all our experiments. We use ρ = 2 . 4 consistently across all experiments. Our full QuIP procedure is described in Algorithm 3, which contains calls to the pre- and post-processing sub-steps in Algorithms 1 and 2.

B.2 Subsection 4.2 (Greedy Updates)

In this subsection, we describe the 'greedy local search' method mentioned in the main body of the paper in more detail. The basic idea is to iterate over coordinates of the weights in the same order as the initial quantization method, modifying each weight in turn-but still restricting it to be a representable quantized value-so as to minimize the proxy loss while keeping the other weights fixed. These greedy updates amount to coordinate descent on the proxy loss, but restricted to the quantization grid. Greedy updates can be performed after any initial quantization method, or as a standalone method. When performed after an initial quantization method, greedy local search is a descent method because the individual weight updates cannot increase the loss, but when performed alone, these greedy updates are not a descent method because the initial point ( ˆ W = W ) is not feasible because it contains unquantized values that are off the representable quantization grid. Concretely, a greedy update of weight ( i, j ) to the grid { 0 , 1 , . . . , 2 b -1 } does the following, where ℓ is the proxy loss:

(Note that ˆ W -e i e T j ˆ W ij + e i e T j z is the result of setting the ( i, j ) th entry of ˆ W to z .) A full pass of greedy updates constitutes mn of these updates performed in the same order as LDLQ. This algorithm is very simple, since it is just greedy coordinate descent. In the rest of this subsection, we will give a bit more intuition about this method by showing how this greedy algorithm falls within our framework of adaptive rounding with linear feedback.

Algorithm 4 Greedy Updates: A Single Pass

Require: b ∈ N , H ∈ R n × n SPD, weights W ∈ R m × n , initial guess ˜ W

1: ˆ W ← ˜ W

2: U ← ( H ⊙ M ) diag( H ) -1 ▷ M is the strictly upper triangular mask

3: V ← W -( ˜ W -W )( H ⊙ M T ) diag( H ) -1

▷

can skip if

˜

W

=

W

4: for k ∈ { 1 , . . . , n } do ˆ W k ← clamp( Q near ( V k +( W -ˆ W ) U k ) , 0 , 2 b -1)

5: return ˆ W

An application of greedy local search as a single-pass stand-alone method falls under our Adaptive Rounding with Linear Feedback framework, with the linear feedback set to U = ( H ⊙ M ) diag( H ) -1 , where M is the strictly upper triangular mask and ⊙ denotes the Hadamard (entrywise) product, as we will derive below. For ease of explanation consider a single (row) weight vector w ∈ R 1 × n . When looking only at column j , the proxy loss from setting ˆ w j to z is

This is just a quadratic function in z , and so its minimum value on the grid { 0 , 1 , . . . , 2 b -1 } will just be its minimum value on R rounded to that grid. To find this minimum over R , we differentiate to minimize, yielding

by setting

V

W

and solving for z ,

Since when we use greedy local search as a stand-alone method, we have not updated ˆ w j yet, at this point ˆ we j = we j , and so this means that a single step of greedy updates looks like

for Q referring to nearest rounding with the necessary clamping. Since ˆ w -w is zero for all entries following the j th one, this is equivalent to

where U is set as U = ( H ⊙ M ) diag( H ) -1 as above. This shows how this single-pass version of greedy updates fits into our adaptive rounding with linear feedback framework.

Analyzing greedy local search as a post-processing pass is a bit more difficult, but we will see that it can also be written as something like adaptive rounding with linear feedback. Suppose that we do a pass of greedy updates, but our quantized weights start at an initial value ˆ w = ˜ w already quantized from some previous method (e.g. LDLQ). Returning to (8), since we haven't updated ˆ w j yet, we'll have

Now, all the entries of ˆ w which come after j are still the ones from ˜ w . This means that we can split this up as

where the first part of this sum comes from the entries which we may have already updated during this pass, the second comes from the entries which are still equal to their initial values in ˜ w , and the case of w j is handled specially, cancelling it with the ˜ we j term. We can write this more compactly in matrix form as

where M is the strictly upper triangular mask and ⊙ is elementwise multiplication. This yields a final quantization step of

So, more generally, if we define U as above, and set

we can write a single pass of greedy updates in matrix form as

which is very close to our rounding with linear feedback form, albeit with the difference that here V is in place of W . This is made explicit in the included Greedy Updates Algorithm.

We can use this algorithm both as a whole quantization method (by setting ˜ W = W ) or as a postprocessing step (by setting ˜ W to the output of some other initial quantization algorithm, such as LDLQ). When we do use it as a post-processing step, we typically run multiple passes of greedy updates (e.g. 10 passes): this involves passing the output of the greedy updates algorithm back in as the input guess ˜ W to another run of the greedy updates algorithm, and repeating this multiple times.

C Additional Experimental Descriptions and Results

C.1 Subsections 3.2 and 3.3 (Empirical Properties of H Across OPT-125m to 2.7b)

Interpreting the exact proxy loss of LDLQ and nearest rounding by empirically comparing tr ( D ) vs tr ( H ) . Theorem 1 gives the average-case proxy loss for LDLQ in terms of tr ( D ) , where D is from the LDL decomposition of H . Lemma 3 gives the average-case proxy loss for standard nearest rounding in terms of tr ( H ) . We know that LDLQ is better in practice, but comparing these equations is difficult because we need to reason about tr ( D ) vs tr ( H ) . Our paper resolves this difficulty by deriving bounds on the proxy loss for LDLQ in terms of the spectrum of H (with and without incoherence). However we also perform a quick empirical check: if tr ( D ) ≪ tr ( H ) , then our theory explains the empirical superiority of LDLQ over nearest rounding (at least on these models). Table 6 gives the ratio tr ( D ) / tr ( H ) across all layers for OPTQ models 125m to 2.7b; the mean value is always less than 0 . 55 , and it falls as the model gets larger.

H is approximately low-rank. Subsection 3.3 plotted the normalized eigenvalues of H from 3 randomly chosen layers in OPT-2.7b. Table 6 gives much more evidence that H is consistently approximately low-rank. Across each model, we calculate the absolute and approximate fractional rank

of H across all layers in OPT models 125m to 2.7b (explanations in the caption). The approximate fractional rank decreases as model size increases; for OPT-2.7b the fractional rank is ≈ 0 . 02( ± 0 . 02) .

C.2 Subsection 5.1 (Empirical Verification of OPTQ Equivalence)

Weshare a python script in the supplementary code which empirically verifies that our implementation of LDLQ produces quantized values exactly matching OPTQ's [1] implementation. While we prove the equivalence between LDLQ and OPTQ's respective algorithm statements, empirically comparing ours and Frantar et al. [1]'s code ensures that the respective implementations are sufficiently close to their algorithmic statements. Therefore we can be sure that LDLQ and OPTQ are equivalent in their implementation.

C.3 Subsection 5.2 (Empirical Verification of LDLQ/OPTQ Finite Grid Counterexample)

The following code constructs a weight matrix W and Hessian matrix H where OPTQ performs worse than nearest when rounding to a finite grid.

The intuition behind this counterexample is as follows: we want to quantize many coordinates in W in such a way that OPTQ excepts there to be a very large error correction to quantize the last entry. However, the finite grid restricts this large error correction. Note that we can achieve this poor OPTQ behavior with c =0, but here nearest rounding also does poorly. We make a small perturbation ( c =0.01) to make OPTQ round in the wrong direction, but not nearest.

C.4 Additional Details on the Experimental Setup and Computational Resources

We run experiments on a university cluster managed by a Slurm workload manager which has GPUs with up to 48GB of memory, though larger GPUs are only required for some methods on larger model sizes. Note we use the LAMBADA OpenAI version. When Greedy updates are used, we perform 10 passes over the weights in the same order as LDLQ and OPTQ, except for 5 passes on OPT-30b and OPT-66b. For the incoherence-based quantization range, we tune the parameter ρ and find that a value of 2.4 works well across all model sizes and quantization methods. We use this value for all our experiments.

C.5 Section 6 (Main Results on Additional Evaluations)

Figure 6 shows additional results for QuIP and OPTQ on WikiText2, PiQA, and StoryCloze when quantizing to 2 and 3 bits per weight. The insights about our method QuIP remain the same after viewing these additional results: QuIP is the first PTQ procedure to achieve good quantization at two bits per weight, across a variety of LLM sizes and evaluation tasks. We evaluate on OPT models (up to 30B); 4-bit quantization works equally well for both methods. QuIP is superior to OPTQ across model sizes and evaluation tasks here.

On WikiText2 2-bit quantization, note that the trend in perplexity for QuIP mirrors the trend in perplexity for OPTQ. We run OPTQ's [1] implementation, though they did not report 2-bit results at this model size. Because OPTQ is equivalent to QuIP's quantization sub-procedure, it thus makes sense that worse performance in the quantization sub-procedure could result in worse overall performance. OPTQ increases perplexity when going from OPT-1.3b to OPT-2.7b. QuIP's perplexity also increases from OPT-1.3b to OPT-2.7b, and is unusually higher than the adjacent OPT-1.3b and

10

1

10 0

10 1

# params in billions

10 1

10 2

10 3

10 4

Perplexity on WikiT ext2

10

1

10 0

10 1

# params in billions

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Accuracy on PiQA

10

1

10 0

10 1

# params in billions

0.50

0.55

0.60

0.65

0.70

0.75

Accuracy on StoryCloze

FP16

OPTQ-W3

OPTQ-W2

QuIP-W3

QuIP-W2

OPT-6.7b models. However QuIP still beats OPTQ in this setting. Our observations about OPTQ and QuIP on WikiText2 and OPT-2.7b were consistent across multiple independent runs.

C.6 Section 6 (All Methods, All Model Sizes, All Bit Weights, All Evaluation Tasks)

Tables 7-13 provide results on all combinations of the following: methods, model sizes (OPT 125m30b), bit weights(4,3,2), and evaluation tasks. Across our extensive array of experiments, we see that incoherence processing always enables a step function change in quantization at 2 bits.

Table 12: Quantizing OPT-350m with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.

Table 13: Quantizing OPT-125m with all combinations of quantization and pre-post processing methods, evaluating on language generation and zeroshot tasks. Our incoherence processing enables a step function change in quantization at 2 bits, across all rounding methods.

C.7 Section 6 (Evaluating the Effectiveness of the Proxy Objective)

In Table 14 we show the proxy loss of the four quantization methods we evaluate, evaluated over OPT models 125m to 2.7b. The proxy is averaged over models proxy losses normalized by their model dimension; we use H matrices computed as a result of OPTQ and nearest rounding. We do not conduct any processing in the proxy evaluation; this is an evaluation of the rounding methods only. Trends in the proxy reflect end-to-end results. OPTQ/LDLQ, LDLQ-RG, and Greedy are roughly equivalent at 2 bits, and do better than Nearest.

C.8 Section 6 (Evaluating Unbiased Rounding in LDLQ/OPTQ)

Note in our formulation for Adaptive Rounding with Linear feedback, the Q subroutine could be biased, or unbiased. It is typical to perform biased rounding in practice; here we investigate if there is

↓

AVERAGE(Perplexity Unbiased - Perplexity Biased) on Wiki, PTB, C4 (

)

any benefit to switching to unbiased rounding schemes. Table 15 computes the average perplexity difference (i.e. unbiased -biased ) for LDLQ/OPTQ on WikiText2, PTB, and C4. That is, we run LDLQ with the Q subroutine as stochastic rounding, instead of nearest. The average difference is positive (and large for 2 and 3 bits), meaning that unbiased rounding performs worse than biased (i.e. nearest) across OPT models 125m to 2.7b. These results indicate that in practice, we want to stick with biased rounding schemes.

C.9 Section 6 (Evaluating Algorithm 5 Which Accounts for Clamping)

Table 16 shows results from using Algorithm 5 to quantize OPT models 125m to 1.3b, with incoherence processing and baseline processing. At 2 bits and incoherence processing, we observe modest improvements over QuIP in terms of perplexity on OPT models 125m and 350m. However, at the

larger OPT-1.3b QuIP beats Algorithm 5 on 2/3 language generation tasks. In addition, Algorithm 5 is computationally more work to run. Therefore we decide not to use it.

Another observation: in practice, we don't seem to encounter constructions of W and H that are bad for LDLQ/OPTQ. Therefore this 'clamping' issue seems to not be an issue in practice, especially as model size increases.

D Proofs for Section 3 (Quantization With Incoherence Processing: Adaptive Rounding Step )

Subsection 3.2 (Deriving the Optimality of the LDLQ Adaptive Rounding Procedure)

Theorem 1. LDLQ is worst and average-case optimal amongst rounding methods which specify the linear feedback U as a function of H (not of W ), and when rounding to the integers. That is, for all rounding methods A in the class described by Eq. (2) , for all positive semi-definite H , and for Q as either nearest or stochastic rounding,

where D is the matrix from the LDL decomposition of H , and c = 12 for nearest, c = 6 for stochastic.

Proof. Let X be the strictly upper triangular matrix associated with the rounding procedure A such that U ← X in Eq. (2). Let B ≡ ( X + I ) -1 ( ` U + I ) where ` U is from the LDL decomposition of H in Eq. (4). The proxy loss is then,

With the LDL assignment of U , we further have that,

First, consider the worst-case loss, L worst . The goal is to construct a particularly bad case where the entries of ˜ W are 1 / 2 ± ϵ , and thus when rounding to the integers we will always have error 1/2. Construct a weight matrix ˜ W ∈ R m × n such that each entry satisfies,

and the quantization errors η ∈ R m × n are for each entry { +1 / 2 , -1 / 2 } with equal probability. For this particular ˜ W , A achieves proxy loss L worst ( A , H ) (9) = E [ tr ( ηBDB T η T )] = m 4 tr ( BDB T ) , with Q as either nearest or stochastic rounding. It follows from the supremum in the definition of L worst in Eq. (5) that, L worst ( A , H ) ≥ m 4 tr ( BDB T ) . For the LDL assignment of U , the worst case expected quantization error rounding to the integers is 1 / 2 . Therefore, L worst ( LDLQ , H ) (10) = m 4 tr ( D ) , again for Q as either nearest or stochastic rounding. B must be a unit triangular matrix since it is the product of unit triangular matrices. Therefore tr ( BDB T ) is minimized when B = I , and

Next, consider the average loss, L avg , where W ∼ Unif [0 , 1] m × n . For Q as nearest rounding, the entries of the quantization error η are Unif [ -1 2 , 1 2 ] , because each entry is independent and uniformly distributed. It follows that for any entry of η , E [ η 2 ij ] = ∫ 1 / 2 -1 / 2 x 2 dx = 1 12 . Therefore, L avg ( A , H ) (9) = E W ∼ Unif [0 , 1] m × n [ tr ( ηBDB T η T )] = m 12 tr ( BDB T ) . For Q as stochastic rounding, the entries of the quantization error η are Unif [ -1 , 1] . It follows that for any entry of η , E [ η 2 ij ] = ∫ 1 0 x (1 -x ) dx = 1 6 . Note that for stochastic rounding, the quantization error will be x with probability (1 -| x | ) . Therefore, L avg ( A , H ) = m 6 tr ( BDB T ) . Based on these same calculations of E [ η 2 ij ] ,

we have that L avg ( LDL,H ) (9) = m 12 tr ( D ) with Q as nearest , and = m 6 tr ( D ) with Q as stochastic rounding. By the same reasoning on the minimization of tr ( BDB T ) ,

Subsection 3.3 (Incoherence: Optimality with a Spectral Bound)

Definition 1. We say a symmetric Hessian matrix H ∈ R n × n is µ -incoherent if it has an eigendecomposition H = Q Λ Q T such that for all i and j , | Q ij | = ∣ ∣ e T i Qe j ∣ ∣ ≤ µ/ √ n . By extension, we say a weight matrix W ∈ R m × n is µ -incoherent if all i and j , | W ij | = ∣ ∣ e T i We j ∣ ∣ ≤ µ ∥ W ∥ F / √ mn .

Lemma 8. Let H ∈ R n × n be a positive semi-definite symmetric matrix, and let a 1 , . . . , a n be a sequence of vectors in R n . Consider the recurrence given by Σ 0 = 0 ∈ R n × n and from k = 0 to n -1

Let ℓ ( a 1 , . . . , a n ) = tr ( H Σ n ) . Then if H = LDL T is the LDL decomposition of H , a global minimum of ℓ occurs when a k is the k th column of L , and at this minimum, ℓ = tr ( D ) .

Proof. First observe that at step k , Σ k will be 0 in all entries (Σ k ) ij if min( i, j ) ≥ k . This means that changing the last n -k entries of a k does not change Σ (or ℓ ) at all. Without loss of generality, set those entries of a k to 0 . If A is the matrix whose k th row is a k , this is equivalent to saying that A is strictly lower triangular.

Next, let η be a random Gaussian sampled from N (0 , I ) , and consider the recurrence given by x 0 = 0 ∈ R n and

It's straightforward to see that Σ k = E [ x k x T k ] . But it's also easy to see that the stepk update only modifies/assigns the k th entry of x , and does so based only on earlier entries of x . Since e T k x k = 0 , and no later step assigns the k -or-lower entries of x ,

which in vector form yields and

In particular, this immediately implies that

where B = I + A . Differentiating with respect to B in strictly lower triangular direction ∆ (the only direction in which we have degress of freedom, since the diagonal of B must be unit) yields

It's not hard to see that if H = LDL T is the LDL decomposition of H , and B T = L , that the gradient is

Since ∆ T is strictly upper triangular, but B -1 D must be lower triangular, this is 0 so we have a minimum. The uniqueness of this minimum (up to assignments of the lower-triangular elements of A or B , which have no effect on ℓ ) also immediately follows from the recurrence relation. This implies the minimum is global. This is what we wanted to show.

Lemma 2. Let H ∈ R n × n be a µ -incoherent positive semi-definite symmetric matrix and let H = ( ` U + I ) D ( ` U + I ) T be its LDL Cholesky decomposition, where ` U is a strictly upper triangular matrix and D is a (non-negative) diagonal matrix. Then,

Proof. By continuity of tr ( D ) and tr ( H 1 / 2 ) , it suffices to prove the lemma for positive definite H . First, the closure of positive definite symmetric matrices is the set of positive semi-definite symmetric matrices. Second, consider the set of H that are positive definite and satisfy µ 2 n tr ( H 1 / 2 ) 2 -tr ( D ) ≥ 0 , i.e. are non-negative. The closure of this set (i.e. H ⪰ 0 ) must also satisfy that the inequality is non-negative.

Let H = Q Λ Q T be the eigendecomposition of H . First, observe that by incoherence,

Set

and consider the recurrence from Lemma 8 with

Then

Suppose by way of induction that for some scalar the covariance Σ k ⪯ αH -1 / 2 . For the base case, this obviously holds since Σ 0 = 0 . At step k ,

Note that with this assignment,

So, by induction it follows that

and so

But from Lemma 8, we know that tr ( D ) is the global minimum of tr ( H Σ n ) for any assignment of a k . This immediately gives us the desired result.

Lemma 3. Let H be symmetric positive definite. In the worst case stochastic rounding achieves L worst ( Stoch , H ) = ( m/ 4) tr ( H ) . In the average case nearest and stochastic rounding achieve L avg ( { Near , Stoch } , H ) = ( m/c ) tr ( H ) , where c = 12 for nearest, and c = 6 for stochastic.

Proof. For nearest and stochastic rounding, set the linear feedback U in Eq. (2) to be zero. Stochastic rounding achieves worst-case loss,

For the average-case proxy loss, recall the computations of E [ η 2 ij ] from the proof of Theorem 1.

Without incoherence: no improvement with a spectral bound

Theorem 4. Consider all ˜ H with the same spectrum as H . For any positive semi-definite H , the following holds. On the worst-case loss LDLQ achieves the same error as stochastic rounding,

On the average-case loss LDLQ achieves the same error as the corresponding rounding routine. Let B = { Near , Stoch } and c = 12 for nearest, c = 6 for stochastic.

Proof. See Lemma 3 for calculations on the proxy loss for nearest and stochastic rounding.

For LDLQ, we will derive lower and upper bounds on sup ˜ Hs.t. eig( ˜ H )=eig( H ) L worst ( LDLQ , ˜ H ) and sup ˜ Hs.t. eig( ˜ H )=eig( H ) L avg ( LDLQ , ˜ H ) , and show they are equal. To construct a lower bound, consider ˜ H = I Λ I where Λ are the eigenvalues of H . This decomposition is also the LDL decomposition of ˜ H , rewritten as ˜ H = ( U + I ) D ( U + I ) -1 . It follows that tr ( D ) = tr ( ˜ H ) for this ˜ H . Combine this result with the worst and average-case losses calculated in the proof of Theorem 1. For the worst-case loss from the proof of Theorem 1, ≥ m 4 tr ( H ) . The lower bound for the average-case loss is ≥ m 12 tr ( H ) for Q as nearest, and ≥ m 6 tr ( H ) for Q as stochastic. Now upper bounds are derived using the preceding calculations in Eq. (11)-(13), and using the worst and average-case optimality of LDLQ proven in Theorem 1. The lower and upper bounds are tight, proving our result.

E Proofs for Section 4 (Quantization With Incoherence Processing: Incoherence Processing Step )

Subsection 4.1 (Incoherence via Efficient Orthogonal Multiplication)

Lemma 9 (Theorem 2.4 from Lalley [2] ) . There exist constants C and A independent of n such that for any function F from the unit sphere in n dimensions to R that is 1-Lipschitz relative to the Riemannian metric on the sphere,

Lemma 10. Let B ∈ R m × n be a matrix, and let x be a random vector uniformly distributed on the unit sphere in R n . Then there exist global constants A > 0 and C > 0 independent of m and n such that

Proof. Let

Observe that and so

Also observe that for x drawn uniformly from the sphere in n dimensions,

If we let δ be then

Trivially, then, for some modified global constants A ′ and C ′ ,

This means that i.e.

This is what we wanted to prove.

Lemma 5. Let H be a positive semi-definite matrix on R n × n and W a matrix on R m × n , and suppose that m = p 1 · p 2 · · · p k and n = q 1 · q 2 · · · q k . Let U 1 , U 2 , . . . , U k , V 1 , V 2 , . . . , V k be independent random orthogonal matrices on R p i × p i and R q i × q i respectively. Set U as the Kronecker product U = U 1 ⊗ U 2 ⊗··· ⊗ U k and V as V = V 1 ⊗ V 2 ⊗··· ⊗ V k Then V HV T is µ H -incoherent with probability at least 1 -δ , and UWV T is µ W -incoherent with probability at least 1 -δ , where

for some global constants A and C independent of n and k .

Proof. First we will prove what we want to prove about H ; then we will prove what we want to prove about W . Let Q be a matrix of eigenvectors of H . Observe that since Q is an orthogonal matrix (by the spectral theorem, because H is symmetric), Qe j is a unit vector, i.e. ∥ Qe j ∥ = 1 . Call Qe j = y . Also observe that

for some indices i j . Call e T i j U j = x T j , and observe that the x j are all independent unit random vectors. So,

for random unit vectors x 1 , . . . , x k and unit vector y . We can easily bound this with k applications of Lemma 10 and a union bound, yielding

Setting δ ↦→ δ kn 2 yields

and unioning over all the entries of the large orthogonal matrix,

Next, for W , observe that if we flatten W , then W/ ∥ W ∥ F is a unit vector. Then any entry of the resulting matrix can be written as

where x 1 , . . . , x k and y 1 , . . . , y k are k independent random unit vectors. We can easily bound this with 2 k applications of Lemma 10 and a union bound, yielding

Setting δ ↦→ δ 2 kmn yields

and unioning over all the mn entries of the large orthogonal matrix,

This is what we wanted to show.

F Proofs for Section 5 (Extensions and Further Analyses)

Subsection 5.1 (OPTQ is a Special Case of LDLQ)

Theorem 6. OTPQ [1] falls within the class of adaptive rounding procedures with linear feedback as described by Eq. (2) , and is equivalent to LDLQ in Section 3.

Proof. OPTQ works in the following way. After OPTQ has quantized the first t -1 components of the row vector w , it minimizes the proxy loss over the remaining n -t +1 elements, keeping the first t -1 elements fixed. It then quantizes the t th element using nearest rounding to the grid and clamping. It then proceeds to the next column. If we let ∆ = ˆ w -w , this proxy loss that it minimizes can be written in block form as

and its minimum over ∆ t : n will occur when

i.e.

Now, suppose that H = ˜ UD ˜ U T is the LDL decomposition of H , where ˜ U is unit upper triangular and D is diagonal. Since ˜ U is upper triangular,

Similarly,

This means that

Now, the only part of the value of ∆ t : n which matters is the first entry, since this is the one that's going to be used to make the next quantization decision. But since ˜ U t : n,t : n is unit upper triangular and so is its inverse, ( ˜ U t : n,t : n ) -1 e t = e t , and so

Finally, we quantize the t -th weight as

This update is equivalent to our adaptive rounding with linear feedback procedure in Eq. (2), with U assigned from the LDL decomposition of H .

Subsection 5.2 (A Bound for Rounding to a Finite Grid)

Algorithm 5 presents a quantization procedure which theoretically address OPTQ's clamping issue, by incorporating a restriction of | ˆ W ij -W ij | into objective (7). Note that for simplicity, here we present the explicit case where only two factors are used in each Kronecker product of orthogonal matrices; however, the proof should generalize to any number of factors.

Algorithm 5 'Fixed' Rounding via a Convex Program

Require: W ∈ R m × n , H ∈ R n × n , c > 0 , ρ > 0 Require: factorization m = p 1 p 2 , n = p 3 p 4 draw U 1 ∈ R p 1 × p 1 uniformly from the set of orthogonal matrices using seed seed ( U 1 ) draw U 2 ∈ R p 2 × p 2 uniformly from the set of orthogonal matrices using seed seed ( U 2 ) draw U 3 ∈ R p 3 × p 3 uniformly from the set of orthogonal matrices using seed seed ( U 3 ) draw U 4 ∈ R p 4 × p 4 uniformly from the set of orthogonal matrices using seed seed ( U 4 ) W ← ( U 1 ⊗ U 2 ) W ( U 3 ⊗ U 4 ) H ← ( U T 3 ⊗ U T 4 ) H ( U 3 ⊗ U 4 ) W ← 2 b -1 2 ( W ρ +1 ) elementwise W ← clamp( W, min = 0 , max = 2 b -1)) elementwise use ADMM or some other solver to solve minimize: tr ( HL T L ) over: L unit upper triangular subject to: e T i L T Le i ≤ 1 + c, ∀ i ∈ { 1 , . . . , n } . note that when c = ∞ , L -1 is the factor from the LDL decomposition of H ` U ← L -1 -I for k ∈ { 1 , . . . , n } do ˆ W k ← clamp( Q ( W k +( W -ˆ W ) ` U k ) , 0 , 2 b -1) ▷ round with LF ˆ W ← ρ ( 2 ˆ W 2 b -1 -1 ) ˆ W ← ( U T 1 ⊗ U T 2 ) ˆ W ( U T 3 ⊗ U T 4 ) ˆ

return W encoded as a tuple of the integer rounded values, the scale factor ρ , and the seeds

over: L unit upper triangular

Then the solution satisfies and so

Proof. Let η ∈ R 1 × n be a random standard Gaussian variable as a row vector, let A be a matrix, and consider the recurrence relation over x t ∈ R 1 × n given by x 0 = 0 and

We first note that since x t is supported only on { 1 , . . . , t } , if M denotes the strictly upper triangular mask, this update step is equivalent to

From here, it's fairly easy to see by induction that

or

Now, since I + A ⊙ M is a unit upper triangular matrix, its inverse is also a unit upper triangular matrix. If we let L = ( I + A ⊙ M ) -1 , then L is a unit upper triangular matrix and

We are going to choose A such that L is a feasible solution to our optimization problem and has the desired objective. Next, let Σ t = E [ x T t x t ] , and observe that

Let α > 0 be some constant to be set later, and set A = αH 1 / 2 . Suppose by way of induction that for some constant β > 0 to be set later, Σ t ⪯ βH -1 / 2 . The base case clearly holds since Σ 0 = 0 . For the inductive step,

This inductive step will hold if, letting h = max i e T i H 1 / 2 e i ,

So the constraint of our optimization problem will be satisfied if

To satisfy these constraints, set β = max( h, h/c ) and α = β -1 . Then

and

Also, the objective will be bounded by

Now, applying incoherence to bound h , where H = U Λ U T is the eigendecomposition of H ,

So this yields a whole bound of

This is what we wanted to show.

On the other hand,

Lemma 12. Suppose that we quantize the row vector w ∈ R 1 × n using L the solution to the optimization problem

over: L unit upper triangular

where Q stoch denotes elementwise unbiased stochastic rounding. Then for any u ∈ R n and any δ > 0

In particular,

Proof. Let η be the error of stochastic rounding, and observe that each entry is, conditioned on earlier steps, zero mean and supported on two values that differ by 1 . Also observe that and so

and

From a repeated application of Hoeffding's lemma, we get

And by Markov's inequality,

i.e.

Minimizing the right side over γ yields γ = 4 R ∥ Lu ∥ -2 and

Now setting the right side equal to δ ,

This is what we wanted to show. The second statement follows from the fact that

and

Setting u ↦→ γu for γ > 0 ,

By a union bound,

Lemma 13. Suppose that we quantize the row vector w ∈ R 1 × n using L the solution to the optimization problem

over: L unit upper triangular

where Q stoch denotes elementwise unbiased stochastic rounding. Suppose that for some integer b , 1 ≤ w ij ≤ 2 b -2 . Then if we set

then with probability at least 1 -δ , 0 ≤ ˆ w ij ≤ 2 b -1 and

Proof. First, from the previous lemmas, if Ue i is the i th eigenvector of H , with eigenvalue λ i since

By the union bound,

And so

which simplifies to

Now applying the other lemma,

And substituting δ ↦→ δ/ 2 ,

On the other hand, again by a union bound from the previous lemma,

Setting yields

and

And so by another union bound, the probability that

and

is no less than 1 -δ . It's clear that if this second inequality holds, the value we pass in to the stochastic quantizer will be in range, and thus so will the output. This proves what we want.

Theorem 14. Suppose that we are given an input matrix w with bounded maximum entry magnitude ∥ w ∥ ∞ and we want to quantize it using b bits. Suppose that we first re-scale the entries of w by mapping

this guarantees that 1 ≤ w ij ≤ 2 b -2 . Then, suppose we quantize using the procedure described in the previous lemma. Finally, we undo the scaling. Then then with probability at least 1 -δ , all the quantized weights will be in range (no overflow or need for clipping) and

Proof. This is a straightforward consequence of the previous lemma.

Theorem 15. Suppose that we are given an input matrix w with bounded ∥ w ∥ F and we want to quantize it using b bits. Suppose that we first multiply by two-factor orthogonal matrices, and then we re-scale the entries of w by mapping

this guarantees that 1 ≤ w ij ≤ 2 b -2 . Then, suppose we quantize using the procedure described in the previous lemma. Finally, we undo the scaling and multiplication. Then then with probability at least 1 -δ , all the quantized weights will be in range (no overflow or need for clipping) and

Proof. It is a straightforward consequence of Lemma 5, that unioning over the three bounds on the infinity norm of w , the incoherence of H , and the stochastic rounding, with probability at least 1 -3 δ ,

Substituting δ ↦→ δ/ 3 ,

And this right side is clearly less than

Theorem 7. Suppose that we run Algorithm 5 (Supplement) to quantize a matrix W ∈ R m × n by solving the objective (7) . Then there exists an assignment of the algorithm's hyperparameters c and ρ such that with probability at least 1 -δ , all the quantized weights will be in range (no overflow or need for clipping) and

Proof. This follows directly from the previous theorem, which says explicitly what the hyperparameter assignments should be.

References for the Appendix

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In International Conference on Learning Representations , 2023.

Steve Lalley. Lecture notes on measure-theoretic probability 2. http://galton.uchicago. edu/~lalley/Courses/383/Concentration.pdf , 2018.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning , pages 7197-7206. PMLR, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a unified framework for weight quantization with error feedback together with preprocessing and postprocessing transformation that makes the model more quantization-friendly. Authors derive theoretical bounds on the quantization error and investigate the failure cases of OPTQ quantization. The introduced LDLQ method with incoherence processing is evaluated on quantization of LLM for 2,3 and 4 bit quantization.\n\nSTRENGTHS:\n* Paper proposes a unified view on quantization with the objective of minimization of layer-wise MSE loss that involves well-known OPTQ as a particular case.\n* Authors derive explicit average and worst-case bounds on quantization error. \n* The introduced incoherence processing is well-motivated and supported by quantitative analysis.\n* The demonstrated LDLQ failure case example despite being very different from typical cases occurring in practice is still interesting and shows potential limitations of the optimal quantization methods.\n* Different variants of LDLQ achieve strong performance on quantization of models from OPT family at various bit widths considered. The most impressive result is that the model attains reasonable perplexity and zero-shot accuracy for 2 bit quantization, which is known to be very challenging and all the competitive methods experience massive performance drop. \n* Overall, the work is well-structured and accompanied with thorough theoretical analysis and empirical study.\n\nWEAKNESSES:\n* Method is evaluated only on a single model LLM family. To be sure that the method achieves strong performance on LLM quantization in general one should consider at least one more family of LLM. \n* Minor. $U^{\\prime}$ in formula (4) is not upper unit triangular but rather strictly upper triangular (I guess  $U^{\\prime}  + I$ is supposed to be upper unit triangular). \n* Minor.  This will result in each of its eigenvalues being a random unit vector. I guess the authors meant eigenvectors instead of eigenvalues.\n\nQUESTIONS:\nHow well does LDLQ+QuIP perform on other families of LLM? Would be interesting to evaluate QuIP on LLaMA given the popularity and impressive performance of this model family. Since it is not fully open-sourced one could consider the recent Falcon family (Falcon-7B and Falcon-40B) instead.\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presented in this paper introduces quantization with incoherence processing, which enables better quantization with fewer bits per parameter. The authors provide a theoretical analysis for adaptive rounding methods and present experimental results demonstrating performance of 2-bit quantization..To do this, the paper introduces the LDLQ adaptive rounding method and demonstrates its optimality compared to other rounding methods which specify linear feedback U for hessians, and rounding to integers (see Theorem 1). The authors define incoherence and demonstrate its effectiveness in achieving a theoretically superior asymptotic bound for LDLQ in terms of the spectrum of H. Additionally, the authors propose efficient pre and post incoherence processing techniques to transform W and H matrices, eliminating the need for nxn matrix multiplications.\n\nSTRENGTHS:\nThe paper encompasses numerous interesting new concepts, theorems, and their corresponding proofs. The extensive experimentation serves to prove the authors' claims effectively. The writing is solid, making the paper relatively accessible despite the number of theorems and proofs. Proposed incoherence seems to improve baseline methods, for example OPTQ (table 6,7 appendix). The utilization of the incoherence technique has the potential to contribute significantly to achieve usable 2-bit quantization. They provided code, which is always a plus.\n\nWEAKNESSES:\n1) One significant drawback of the paper is that despite the aforementioned enhancements, the performance of 30-bit 2-bit quantization, although much better than 2 bit OPTQ, still falls short compared to 13b model 4-bit quantization with OPTQ(and even 4 bit RTN), making it practically unusable. This limitation diminishes the practical usability surrounding the work on 2 bit quantization.\n\n2) The concept of incoherence remains unclear until page 4 of the paper, which is problematic considering it is one of the main focal points. I believe the introduction should include a clear definition and intuitive explanation of what incoherence entails. Furthermore, I found it challenging to establish a connection between the intuition provided in line 22 and the definition presented in line 134.\n\n3) It is possible I missed, but it appears that there is no mention or measurement of inference speed presented in the paper.\n\nQUESTIONS:\n1)I may have overlooked some details, but it appears that the paper only show how making H incoherent reduce loss upper bounds, nothing about W incoherence. What is benefit for making also W  incoherent?\n2) In line 169 you write that \"to make symmetric matrix incoherent is to conjugate it by uniform random orthogonal matrix\". Can you please add citation or explanation why this is true?\n3) Again I may have overlooked some details, in 172- 176 you said that the procedure described there makes the matrix H and W incoherent with high probability/ Why is that, and where is the proof?   \n\nSuggestions:\n1. In line 64, it seems that \"OBC\" might be a typographical error, and \"OBQ\" could be the correct term.\n2. It would be helpful to include a citation for LDL decomposition. \n3.In line 99, it would enhance clarity to present a equation that demonstrates the result obtained when equation (4) is applied in equation (3). \n4.In line 217, it appears that \"OTPQ\" might be a typographical error, and you mean \"OPTQ\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors coin a family of adaptive rounding methods for layer-wise quantization, establish that the state-of-the-art method GPTQ is optimal within this family, and prove quality guarantees.\n- The paper further introduces incoherence preprocessing, together with a Kronecker-factor based inference scheme, which leads to significant accuracy improvements for very low bitwidth compression.\n\nSTRENGTHS:\n- The paper introduces a non-obvious and useful optimization to GPTQ which fully eliminates the matrix inversion while obtaining equivalent results.\n- The authors present what seem like the first theoretical guarantees for an adaptive layerwise rounding algorithm.\n- The proposed incoherence preprocessing leads to greatly improved results for 2-bit compression. While the idea of multiplying with an orthogonal matrix to produce more uniform and thus easier to compress data is not new, I have not seen it applied in the context of LLM quantization before.\n- The paper also studies the impact of some additional heuristics, like greedy post-processing passes on top of GPTQ. While those are mostly small tricks, it is good to have those implemented and evaluated.\n- Code is provided for reproducabilty, including also a script to verify the equivalence of their GPTQ optimization.\n\nWEAKNESSES:\n- The method is only evaluated on OPT, which is not considered a great LLM anymore by today's standards; good results on e.g. LLaMa would be significantly stronger.\n- There do not seem to be any runtime numbers for inference via the proposed efficient Kronecker-factored scheme. I think that the tripling of required FLOPs (2 additional o(n^2) matmuls if I understood this part correctly) may be challenging to implement in practice without significant overheads (both for low- and large-batch inference), which would limit the practicality of incoherence preprocessing.\n- The algorithm family seems to be somewhat designed around how GPTQ works and the corresponding optimality is thus not exactly super surprising.\n\nQUESTIONS:\n- Given that the largest models are usually more robust to quantization, I am wondering how QuiP performs on the 66B and 175B variants, is there any reason why those results are missing?\n- How does QuiP perform with groups, which are generally very effective for standard GPTQ / RTN?\n- 221: GPTQ only requires one inverse + one Cholesky decomposition\n- 282: It would be good to reference that the official GPTQ repository also proposed a similar trick in the context of LLaMa models several months ago.\n- The incoherence post-processing code currently returns a matrix that is not quantized anymore, which is a bit confusing. Related to that, I would also suggest to provide pseudocode for the efficient Kronecker-factor inference scheme.\n\nLIMITATIONS:\nThe paper briefly discusses limitations and broader impact in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces QuIP, an algorithm for weight quantization in large language models, with theoretical guarantees. The experimental results demonstrate that QuIP achieves nearly lossless performance when using 3-bit quantization for models larger than 3B, and it shows good performance with 2-bit quantization for models larger than 7B.\n\nSTRENGTHS:\nFirst of all, QuIP is the first post-training quantization algorithm that compresses weights to 2 bits with reasonable loss of performance, significantly pushing the boundary of LLM on-device deployment and making larger LLMs available to ordinary users.\n\nSecond, the paper proposes LDLQ rounding method, which is both worst-case and average-case optimal in a family of adaptive rounding methods, i.e. iterative quantization through each column of weights. The family is carefully selected so that OPTQ (previously known as GPTQ) falls within this family.\n\nThird, solid proof is shown that LDLQ is optimal in its family assuming rounding to integers, while a rigid study, with counterexamples and more careful analysis is conducted in more realistic cases in Section 5.2.\n\nWEAKNESSES:\nThe experiments are mostly conducted on OPT, which might not be the most standard model at the time of reviewing. Models like Llama, Falcon and MPT might be more popular LLMs. However, the reviewer understand the paper was submitted months ago and there might be limited time to conduct all the experiments. Also, the main experiments are conducted on decoder-only auto-regressive language models, while there are other models/domains of interest, for example, FastChat-T5 which is encoder-decoder architecture, Vision transformers, etc.\n\nQUESTIONS:\nAs mentioned in Section 5.1., OPTQ is equivalent to LDLQ in the class of adaptive rounding methods with linear feedback (proof in Supplement E), and thus is a special case of LDLQ in general. Is there any further case study why OPTQ performs worse in general?\n\nIs there any analysis on quantization outliers under QuIP/LDLQ and how would QuIP handle those cases?\n\nLIMITATIONS:\nWe believe that considering activation quantization could lead to further improvements in the future. However, the implementation of QuIP as a weight quantization method has already overcome major obstacles and enabled the deployment of LLMs.\n\nCurrently, there is a lack of realistic experiments to assess the performance, in terms of compute speed. Nevertheless, implementing and optimizing 2/3-bit performance on CUDA would likely be a highly challenging task, and it could be worthwhile to explore this further in a separate research paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}