Title: Compressing Large Language Models using Low Rank and Low Precision Decomposition
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Models (LLMs)

Model Compression

Post-training Quantization

Low-Rank Decomposition

Low-Precision Formats

Quantization Error Analysis

Rank-Constrained Regression

Randomized Linear Algebra

Sketching
Abstract: The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces $\rm CALDERA$ -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix $\mathbf{W}$ by approximating it via a low-rank, low-precision decomposition as $\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$. Here, $\mathbf{L}$ and $\mathbf{R}$ are low rank factors, and the entries of $\mathbf{Q}$, $\mathbf{L}$ and $\mathbf{R}$ are quantized. The model is compressed by substituting each layer with its $\mathbf{Q} + \mathbf{L}\mathbf{R}$ decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, $\mathbf{L}$ and $\mathbf{R}$ are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. $\rm CALDERA$ obtains this decomposition by formulating it as an optimization problem $\min_{\mathbf{Q},\mathbf{L},\mathbf{R}}\lVert(\mathbf{Q} + \mathbf{L}\mathbf{R} - \mathbf{W})\mathbf{X}^\top\rVert_{\rm F}^2$, where $\mathbf{X}$ is the calibration data, and $\mathbf{Q}, \mathbf{L}, \mathbf{R}$ are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of $\rm CALDERA$ are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-$2$ $7$B/$13$B/$70$B and LlaMa-$3$ $8$B models obtained using $\rm CALDERA$ outperforms existing post-training LLM compression techniques in the regime of less than $2.5$ bits per parameter.

Compressing Large Language Models using Low Rank and Low Precision Decomposition

Abstract

The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces CALDERA - a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix W by approximating it via a lowrank, low-precision decomposition as W ≈ Q + LR . Here, L and R are low rank factors, and the entries of Q , L and R are quantized. The model is compressed by substituting each layer with its Q + LR decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, L and R are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. CALDERA obtains this decomposition by formulating it as an optimization problem min Q , L , R ∥ ( Q + LR -W ) X ⊤ ∥ 2 F , where X is the calibration data, and Q , L , R are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of CALDERA are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa2 7 B/ 13 B/ 70 B and LlaMa3 8 B models using CALDERA outperforms existing post-training LLM compression techniques in the regime of less than 2 . 5 bits per parameter. The implementation is available at: https://github.com/pilancilab/caldera.

1 Introduction

Large Language Models (LLMs) stand out due to their remarkable ability to generate human-like text, thereby supporting a diverse range of applications ranging from writing assistance to code generation. These models leverage vast datasets and significant computational resources to achieve their impressive functionality. The architecture of LLMs typically includes multiple layers, each with weight matrices essential for encoding various aspects of the training data - from simple syntactic patterns to complex semantic relationships. However, the substantial size of these trained models leads to high computational costs and considerable energy consumption during inference, which can be challenging for deployment in resource-constrained environments. As LLMs continue to expand in scale, compression techniques to reduce the memory and computational requirements of the models are becoming crucial to ensure their broad accessibility.

Due to the correlated nature of language syntax and semantics learned during training, often, the weight matrices of LLMs exhibit redundancy, which manifests as a low-rank structure. This redundancy suggests the potential for compression without substantial loss in performance. This work introduces CALDERA: C alibration A ware L ow-Precision DE composition with LowR ank A daptation, which compresses LLMs by leveraging the approximate low rank structure inherent in

these weight matrices. Given a matrix W ∈ R n × d , CALDERA approximates it as W ≈ Q + LR , where Q ∈ R n × d , L ∈ R n × k and R ∈ R k × d . Here, the left and right low rank factors, respectively L and R , are tall and wide matrices, and k is the target rank. Furthermore, the entries of Q , L and R are represented using low-precision formats with B Q , B L and B R bits per entry, respectively.

0

1000

2000

3000

4000

Index

10

4

10

3

10

2

10

1

10 0

10 1

Magnitude

Singular values of LlaMA-2 7B Query matrices

Query

W

d

n

Full precision

≈

L

k

n

B

L

bits

R

k

d

B

R

bits

+

Q

n

d

B

Q

bits

Since the singular value profile (aka spectrum) of the weight matrices of an LLM follow a decaying profile as shown in Fig. 1, the low-rank factors L and R capture the effect of the large singular components of W with high fidelity. Moreover, the backbone Q , which is quantized aggressively - for instance, using B Q = 2 bits, coarsely captures the essence of the moderately decaying and low singular components of W . CALDERA substitutes each weight matrix W in an LLM, with its approximate low-precision and low-rank decomposition Q + LR , resulting in a post-training quantization strategy that delivers state-of-the-art zero-shot performance. In addition, since usually k ≪ min { n, d } , implying that the total number of parameters in LR is much smaller compared to the number of entries in W (i.e., k ( n + d ) ≪ nd ), CALDERA can readily fine-tune (or "adapt" ) the low rank factors L and R in order to boost the zero-shot results.

1.1 Significance and Related Works

Recent efforts have explored various avenues for compression, including but not limited to weight pruning, quantization, and the use of parameter-efficient training methods - each approach offering distinct advantages and tradeoffs. This section briefly reviews the current methodologies, highlighting the contributions and limitations of some studies closely related to this work.

LLM Compression and Outlier Mitigation : Recent studies like SmoothQuant [42], OPTQ [8], QuIP [3], AQLM [7], and QuIP# [36] consider the challenging regime of sub4 bit post-training LLM quantization. These works collectively emphasize the need to manage the impact of outliers, i.e., weights with unusually high magnitudes. Accommodating outliers necessitates choosing the dynamic range (or scale) of a quantizer to be high, consequently increasing the quantization error. QuIP equalizes (and reduces) the weight matrices by using a randomized matrix transform, and subsequently, QuIP# employs E8 lattice to make the weights more amenable to vector quantization. Both QuIP and QuIP# use a refined variant of the column-wise quantization method proposed in OPTQ, wherein error feedback from previously quantized columns of a matrix is used to compensate for the error incurred while quantizing subsequent columns. CALDERA utilizes this diverse arsenal of strategies and builds on top of QuIP#, while capitalizing on the approximate low-rank structure of LLM weight matrices. While it is possible to obtain even more aggressively compressed LLMs [24], this approach requires training from scratch, which is computationally demanding.

Parameter Efficient Fine-Tuning (PEFT) : In a related yet distinct vein of work, PEFT methods have gained significant momentum, aiming to adapt LLMs to specific tasks without extensive computational overhead. Recent studies such as QLoRA [5], LoftQ [22], and LQ-LoRA [13] have explored the intersection of PEFT and quantization, demonstrating that fine-tuning through low-rank updates, as originally proposed in LoRA [16], can mitigate the performance losses due to quantization. Given that CALDERA yields a decomposition Q + LR , the low-rank components are particularly suitable for fine-tuning with any existing PEFT methods, thereby enhancing the zero-shot capabilities.

Low Rank Approximation : The rankk approximation of a matrix A ∈ R n × d can be represented by the factorization A ≈ LR , with L ∈ R n × k and R ∈ R k × d , where k ≤ min { n, d } . Known as the Burer-Monteiro factorization, this method substantially decreases the number of parameters,

thus reducing computational demands. Recent studies such as LoRD [18], ASVD [44], FWSVD [15], LASER [33], LQER [45], and ZeroQuant-V2 [43] have explored the efficacy of low-rank structures in LLM weights, treating low-rank factorization and quantization independently. In contrast, LPLR [31] approaches this by uniquely formulating a joint optimization problem for generic matrices, while simultaneously leveraging the equalization property of randomized transforms, as in [3]. CALDERA formally leverages this inherent low-rank structure for LLM compression alongside existing frameworks such as QuIP# [36] and LoftQ [22], providing additional flexibility for compression. Furthermore, rigorous theoretical guarantees are derived using a rank-constrained regression framework for obtaining a low precision and low-rank decomposition, thereby also analytically demonstrating its superiority over rank-agnostic strategies.

2 Problem Formulation

In a neural network layer, a weight matrix W transforms an input activation x into an output activation given by Wx . This transformation can be succinctly described using the matrix's singular value decomposition (SVD). For any matrix A ∈ R n × d , the SVD is A = ∑ i σ i u i v i , where σ i , u i , v i are the i th singular value and the corresponding left and right singular vectors, respectively. The impact of each singular component u i v i on the matrix's transformation is determined by the magnitude of σ i . Given that weight matrices exhibit a decaying singular value profile (Fig. 1), indicating an approximate low-rank structure, lesser contributing singular components can be pruned with minimal impact on the functionality of the matrix, ensuring minimal distortion in the output activations.

CALDERA approximates the weight matrix of a neural network, W , as a low-precision, low-rank decomposition, W ≈ Q + LR , with all components Q , L , R in low-precision format. Unlike previous works such as [13, 22, 44, 45], which represent the low-rank factors L and R in highprecision (16 or 32-bit floating point), this work extends their representation to low-precision. This further reduces the memory footprint while preserving performance. Alternatively, for the same memory footprint, it allows the target rank k to be higher, thereby capturing the low rank structure with higher fidelity by including more of the higher singular value components. The following paragraph formalizes this as a constrained optimization problem.

For a given quantizer, let Q denote the set of discrete quantization points in R . For B -bit quantization, the cardinality of Q satisfies log 2 | Q | ≤ B . Consider a matrix W ∈ R n × d . The goal of this work is to obtain an decomposition W ≈ Q + LR by approximately solving the minimization problem

Here, Q Q , Q L and Q R denote the lattice codebooks used to quantize Q , L and R , using B Q , B L and B R bits, respectively. Furthermore, X ∈ R m × d is a calibration matrix that aims to preserve the Frobenius norm error of the compressed layer output activations. If W is the first layer's weight matrix, X includes input embeddings from a calibration dataset, such as a subset of RedPajama [34], with the i th row representing the i th datapoint. For intermediate layers, X contains the input activations, which are the output activations of the preceding layer.

Using the Frobenius norm of the output of a layer as a proxy objective for quantizing the weight matrices of an LLM is a popular strategy, and was used in prior work of Nagel et al. [27] . This proxy objective function is particularly useful for post-training quantization of LLMs because their large size makes it difficult to apply sophisticated compression methods.

3 Proposed Algorithm: Calibration-Aware Low-Precision Decomposition with Low Rank Adaptation

This section introduces CALDERA to approximately solve (1) and get a Q + LR decomposition of a weight matrix W using the calibration matrix X . The pseudocode is provided in Alg. 1. It consists of a nested loop for alternately optimizing the variables Q , L and R . Suppose Q Q , Q L and Q R , respectively, denote quantizers used for quantizing Q , L and R . For instance, they can refer to uniformly dithered scalar quantizers, as described in App. G.2. Initially, the low-rank factors are set to 0 , and W is quantized using the LDLQ quantizer proposed in [3, §3.1]. LDLQ is an adaptive

quantization method that iteratively quantizes [8] each column of W using Q Q to get Q as

where Q ( k ) , W ( k ) denote the k th column, W (1: k -1) denotes the first k columns, Q Q has a bit-budget B Q , and a k ∈ R k -1 is a learnable sequence of vectors. Update Eq. (2) incorporates linear feedback from already quantized columns, it can be seen that Q satisfies Q = Q Q ( W +( W -Q ) M ) , where the feedback matrix M is a strictly upper triangular matrix with columns a k . Defining H ≜ 1 m X ⊤ X to be the (scaled) Hessian of the least squares objective in (1), [3] show that the optimal feedback matrix is the M obtained from the LDL decomposition of m H , given by m H = ( M + I ) D ( M + I ) ⊤ .

Subsequently, Q is fixed and the Low-Precision Low-Rank (LPLR) factorization of the residual, ( W -Q ) , is computed. This is done by the LPLRFACTORIZE submodule (Alg. 2), which is a refined version of the LPLR algorithm proposed in [31]. For a given matrix A , Alg. 2 minimizes

where Q L and Q R use B L and B R bits, respectively. In contrast to [31], the objective in (3) is calibration-data aware. Therefore, the update equations are derived using a rank-constrained regression framework, as described in App. B. Moreover, lines 7 to 14 in LPLRFACTORIZE iteratively refine the estimates of L and R , and can only yield a smaller Frobenius norm error. The left and right low-rank factor update equations are described as follows.

Initialization : In the absence of quantization constraints, a globally optimal solution to the optimization problem (3) can be found as described later in lemma 4.2. Consequently, the low-rank factors are initialized using rank-constrained regression in lines 2 -4 . Since subsequent quantization disrupts optimality of the solution, the factors are iteratively updated to minimize this distortion.

Updating L : To update the left factor L , lines 5 and 9 of Alg. 2 solves min Z ∥ ( ZR -A ) X ⊤ ∥ 2 F . For a fixed R , this is a least squares minimization, whose solution is available is closed form as ` L = ( AX ⊤ ) ( RX ⊤ ) † = AHR ⊤ ( RHR ⊤ ) -1 , as derived in App. C.1.

Updating R : Line 8 of Alg. 2, updates the right factor R by keeping L fixed and solving min Z ∥ ( LZ -A ) X ⊤ ∥ 2 F . As this is an under-determined linear system, there exist multiple solutions for Z , all attaining the same objective function value. It is shown in App. C.1 that ` R = L † AHH † is a solution. The corresponding error is also obtained, which is used in the derivation of Thm. 4.1.

Computational Complexity : A high-level calculation is provided here, and detailed discussions can be found in App. D. It is worthwhile to note that the closed form expressions of ` L and ` R , which are iteratively quantized, are functions of the Hessian H = 1 m X ⊤ X . Therefore, H can be computed offline initially, per LLM, by doing a single forward pass, and subsequently used for all model quantization experiments. For each layer, this pre-processing includes computing H and its LDL decomposition, along with computing HH † , requiring a total of O( md 2 +2 d 3 ) multiplications. Each outer iteration involves an LDLQ quantization. Quantizing the k th column has complexity O( nk ) , since feedback from k already quantized columns need to be incorporated. Hence, quantizing a matrix in R n × d entails O( n 2 +3 n ) complexity. Moreover, LPLRFACTORIZE requires O( m 2 ( n + d )) to initialize, and subsequently, each inner iteration entails O( ndk ) . Assuming n, d ≥ m ≫ k , and keeping only the dominant terms, the total complexity of CALDERA, not including the complexity of the pre-processing discussed earlier, is O ( T out ( n 2 + m 2 ( n + d ) + ndk T in )) .

Fine tuning via Low-Rank Adaptation : Once the weight matrices of each layer are replaced by its Q + LR approximation, the zero-shot performance of (post-training) quantized model can be evaluated. §5 shows that CALDERA quantized models outperform existing strategies. Additionally, if desired, the low-rank factors L and R can be further fine-tuned using low-rank adaptation [13, 16, 22] on a small task-specific dataset. While the initialization of the fine-tuning step has quantized Q , L and R , the fine-tuned factors are represented using 16 -bits ( BF16 format). Although this leads to a slight increase in the memory footprint, the performance gains from fine-tuning are substantial.

4 Approximation Error Analysis

The approximation error upper bounds are derived via a rank-constrained regression framework. Thm. 4.1 below (formally stated and proved in App. C.4) is an informal version of the main theoretical result of this paper, and provides an upper bound on the Frobenius norm error when CALDERA

approximates a weight matrix W is as W ≈ Q + LR by solving the optimization problem (1) using Alg. 1. For convenience of analysis, it is assumed that the dynamic range of Q Q , denoted as R , is chosen to be high enough, ensuring it remains unsaturated. Consequently, for a scalar input the quantization error from Q Q has zero mean and bounded variance, given by ∆ 2 4 = R 2 (2 B Q -1) 2 .

Theorem 4.1. Approximation error of CALDERA (Informal) Given W ∈ R n × d and X ∈ R m × d with m ≤ d , let D be obtained from the LDL decomposition X ⊤ X = m H = ( M + I ) D ( M + I ) ⊤ , and λ max , λ min denote the max and min eigenvalues of H . Additionally, let Q ≜ LDLQ ( W , Q Q ) , where Q Q has dynamic range R and bit-budget B Q , the quantization error be η ≜ Q Q ( Q +( W -Q ) M ) -( Q +( W -Q ) M ) , and σ 1 ≥ . . . ≥ σ k . . . be the singular values of X ( W -Q ) ⊤ . If the target rank k satisfies 0 . 25 λ 1 / 2 min ( mσ 1 ) -1 λ -3 / 2 max ∑ i>k σ 2 i ≤ k ≤ m , and the dynamic ranges of Q L and Q R are set as R L = 2 σ 1 σ k √ mλ min and R R = σ 1 , then Q , L and R returned by Alg. 1 satisfy

while utilizing an average budget of 1 2 log 2 ( kσ 3 1 mϵσ k λ max λ min √ d/n ) bits per parameter for the low-rank factors L and R , when n ≈ d . Here, the expectation is over the stochasticity of the quantizers.

An informal version of the main result is provided here, and the formal version including specific constant values, along with the derivation, can be found in App. C.4. The requirement m ≤ d is not restrictive, because when H is positive definite, (1) can be rewritten as ∥ ( Q + LR -W ) X ⊤ ∥ 2 F = ∥ ( Q + LR -W ) H 1 / 2 ∥ 2 F , ensuring m = d . This is detailed further in App. C.5. The approximation error upper bound given by Thm. 4.1 can be directly compared with the result of Chee et al. [3, Thm. 1], which states that for vanilla LDLQ without LPLRFACTORIZE,

Evidently, Alg. 1 yields a smaller error provided, ∑ i>k E λ i ( η D η ⊤ ) < ∑ k i =1 E λ i ( η D η ⊤ ) -ϵ , where ϵ can be chosen to be arbitrarily small. Furthermore, since the expression in Thm. 4.1 consists of two terms, namely, the rank-constrained regression error, which depends on the target rank k , and the additive quantization error of ϵ , which is dictated by the bit-budgets used for L and R , this upper

bound can be made arbitrarily small by ensuring that the two terms are approximately equal, i.e., E ∥ ( Q + LR -W ) X ⊤ ∥ 2 F is upper bounded by 2 ϵ . This is apparent in the following regimes:

(i) k ≪ n : In this regime, k is treated as a constant as n grows. Then, if the bit-budget B Q satisfies B Q log 2 ( 2R( πϵ ) -1 / 2 √ nmd λ max +1 ) , then E ∥ ∥ ( Q + LR W ) X ⊤ ∥ ∥ 2 2 ϵ.

(ii) k = O( n ) : For a fixed B Q , if k is allowed to grow with dimension n , then choosing k to satisfy √ 2

This implies that the upper bound can be made arbitrarily small by either (i) increasing the bit-budget of the backbone, i.e., B Q , for a fixed rank k , or (ii) increasing the rank k for a fixed B Q , for example, B Q = 2 . Alternatively stated, this provides a tunable knob for controlling the error by trading off the allocated bit-budget between the backbone Q and the low-rank factors L , R .

4.1 Analysis Outline

In this section, a brief proof sketch is presented, highlighting the major challenges in the proof and how they are addressed. For analysis, Q is assumed to be updated prior to L , R in Alg. 1. However, in practice, the update order is inconsequential, and can be swapped, depending on whichever yields a smaller error. The complete derivation of the approximation error is provided in App. C. A key ingredient of the proof is the solution of the rank-constrained regression problem, which is defined as,

Although this problem is non-convex, it can be solved to global optimality via two SVDs [41]. The following lemma characterizes the solution to the optimization problem in (5).

Lemma 4.2. Given Y ∈ R m × n , and full rank X ∈ R m × d , where m ≤ d . Let X = U ˜ ΣV ⊤ and ⊤ ` ` ⊤ ⊤

where Σ := ˜ ΣI m ∈ R m × m is a diagonal matrix consisting of the non-zero singular values of X . Moreover, denoting the non-zero singular values of Y as { σ i ( Y ) } i m =1 , the optimal value of (7) is

The complete lemma (with the case m > d ), and the derivation, are provided in App. B. Using lemma 4.2, the approximation error of LPLRFACTORIZE is analyzed in App. C.3. Specifically, lemma C.3 shows that for any input matrix A , Alg. 2 with suitably chosen B L and B R , ensures that E ∥ ∥ ( LR -A ) X ⊤ ∥ ∥ 2 F , as in (3), can be upper bounded by twice the sum of squared trailing singular values, (ref. (6)). While proving lemma C.3, it is assumed that if Q L or Q R gets saturated, a trivial output of L = 0 , R = 0 is returned. Therefore, lemmas C.1 and C.2 specify choosing the dynamic ranges R R and R L to be sufficiently high so that saturation happens with a very low probability. The proof of Thm. 4.1 is completed by using the LDL decomposition of m H as proposed in [3], along with an application of Marchenko-Pastur approximation to bound the expected eigenvalues of the quantization error, i.e., E λ i ( ηη ⊤ ) , yielding the final inequality.

5 Numerical Simulations

The efficacy of CALDERA is assessed by using it to compress four popular open source LLMs from Meta AI, namely, LLaMa-2 7B, LLaMa-2 13B, LLaMa-2 70B [35] and LLaMa-3 8B [26]. The framework is built in PyTorch on top of the QuIP# [36] and LoftQ [22], and is available at https://github.com/pilancilab/caldera.

Baselines . The full-rank matrix Q , also referred to as the backbone, is quantized to 2-bits using the LDLQ procedure from QuIP [3, 36], employing an E8 lattice quantizer [39]. For CALDERA, which allows even the low-rank factors, L and R , to be represented in low-precision, the quantization is also performed with an E8 lattice. Prior to running Alg. 1, a randomized Hadamard transform (RHT) is applied to the left and the right of the input weight matrix, as the incoherence pre-processing step, to equalize the magnitude of the entries making them more robust to quantization. In other words, CALDERA decomposition is performed on ˜ W ≜ H ⊤ L WH R , where H L and H R are Hadamard matrices, right-multiplied by a diagonal matrix with i.i.id. {± 1 } entries. In addition, the Hessian matrix obtained from the calibration data is substituted by ˜ H ≜ H ⊤ R HH R . As described in [3], this improves the quantization error incurred by LDLQ. Further details are provided in App. E.2.

Metrics . The performance of CALDERA is evaluated using perplexity on the test splits of the Wikitext2 [25] and C4 [6] datasets, as well as task-specific goodness-of-fit metrics such as zeroshot accuracy for sequence classification. Specifically, zero-shot accuracy was measured on the Winogrande [19], RTE [1, 40], PiQA [2], ARC-Easy, and ARC-Challenge [4] tasks. App. E.3 provides more details regarding these benchmarks. Perplexity was measured using a sequence length equal to the model's maximum context length, i.e., 4096 for LLaMa-2, and 8192 for LLaMa-3. Zero-shot experiments were performed using EleutherAI's Language Model Evaluation Harness [9].

5.1 Zero-shot Results

Tables 1 and 2 report the perplexities and accuracies for CALDERA with varying target rank ( k ) of L and R . Asmaller value is better for perplexity, which is defined as the exp( · ) of the training objective, while zero-shot accuracies are reported as percentages. Per-parameter bit budgets range from 2 . 1 (e.g., rank64 factors in 4 -bit precision) to 2 . 4 bits (e.g., rank64 factors in half precision or rank256 factors in 4-bit precision). For comparison, the Q + LR decomposition of weight matrices found in the QuIP# codebase was performed on each model. For the sake of direct comparison, fine-tuning of the diagonal matrices in RHT was omitted. As QuIP# does not support quantized factors, L and R are rank-64 in order to ensure that the per-parameter bit-budget remains in the 2 -2 . 4 range. As another baseline comparison, each model is quantized using QuIP# without any low-rank factors. Results for the unquantized models are also provided.

For all models, the rank256 CALDERA decomposition with 4 -bit factors had the lowest perplexity and generally had the highest accuracies. As CALDERA supports quantizing low-rank factors with minimal performance loss, more singular components can be captured compared to using half-precision factors while employing the same number of bits. Consequently, the low-rank factors

can regain the performance that was compromised when the backbone Q was quantized to 2 bits. Since zero-shot experiments have some inherent randomness and low-rank regularization effects [33], the zero-shot accuracies reported here are not as directly indicative of quantization performance as the perplexity results. In addition, §5.3, demonstrates that degradation in zero-shot accuracy can be recovered via LoRA fine-tuning. It is worthwhile to note these results substantiate the claims of [17], which report that low-bit quantization of LLaMa3 8 B, significantly deteriorates model performance across various post-training quantization techniques, more so than with the LLaMa2 series.

5.2 Fine-tuning of Randomized Hadamard Transform (RHT) Parameters

As CALDERA presents a general optimization framework for matrix decompositions of the form Q + LR , it can easily be extended with additional heuristics to improve performance. This section serves as a proof of concept, by examining one such heuristic: Fine-tuning of randomized Hadamard transform parameters. This technique, proposed in QuIP# [36], involves fine-tuning the diagonal Rademacher matrices with ± 1 entries in the RHT to minimize the cross-entropy loss between the output of the original and quantized models on the calibration dataset. Subsequently, RHT fine-tuning is performed on the models quantized using CALDERA in §5.1. 1 Details on specific fine-tuning hyperparameters can be found in App. E.4.

Perplexity and zero-shot results in Tables 3 and 4 match the trends in §5.1, i.e., CALDERA with rank256 factors typically performs best, with the exception of RTE. In addition, perplexities are substantially lower than without the fine-tuning of randomized Hadamard transform parameters.

5.3 Low Rank Adaptation (LoRA) Fine-tuning Results

In addition to RHT FT as described above, once the Q + LR decomposition with target rank k is obtained, and k takes values 64 , 128 and 256 , fine-tuning the top r ( ≤ k ) singular components on a specific downstream datasets can recover the performance lost due to quantization. We consider three such tasks (i) language modeling on Wikitext (Wiki2), (ii) recognizing textual entailment (RTE), and (iii) commonsense reasoning (WinoGrande). Throughout all experiments in Table 5, r = 64 is chosen and those singular components are fine-tuned to 16 -bit precision, i.e., BF16 format. The tasks (ii) and (iii) are sequence classification tasks, and the pre-trained LLaMa model is augmented with a linear classification head, which is fine-tuned along with the low-rank factors [29]. In other words, the approximation is written as W ≈ Q + L 1 R 1 + L 2 R 2 , where L 1 ∈ R n × r , L 2 ∈ R n × ( k -r ) , R 1 ∈ R r × d , R 2 ∈ R ( k -r ) × d , L = [ L 1 | L 2 ] , R ⊤ = [ R ⊤ 1 | R ⊤ 2 ] . The value of r is set to 64 and L 2 , R 2 are fined-tuned to L bf16 , R bf16 using low-rank adaptation similar to [13, 16, 22]. Doing this significantly on a small task-specific dataset like WikiText2, RTE, or Winogrande, can noticeably boost the zero-shot accuracy, as can be seen from Table 5. 2

Experimental details can be found in App. E.4. For each dataset, ten checkpoints are saved during the course of fine-tuning, and the best test performance is reported in Table 5. For datasets where test labels are not available, evaluation performance is reported instead.

For comparison, results from the LoftQ [22] and LQ-LoRA [13] papers are also reported, where available. As these papers were published before the release of LLaMa-3, only LLaMa-2 results are available. 3 In each case, CALDERA achieves better performance at a lower bit budget.

5.4 Autoregressive Generation Throughput

The low-rank ( LR ) component in CALDERA can recover some of the accuracy lost due to the aggressive 2 -bit quantization of Q . However, CALDERA also needs to dequantize and multiply the low-rank factors, which results in a slight (albeit, acceptable) throughput degradation compared to QuIP# (shown in Table 6). Nevertheless, CALDERA's throughput is significantly higher than that of the unquantized model. This is because compressing weight matrices results in a smaller volume of data transfer from and to the GPU's SRAM, speeding up forward passes. It is worthwhile to note that

Llama-2 70B runs into out-of-memory (OOM) as an A10G GPU only has 24 GiB of VRAM, which is not enough for 70B parameters in FP16 format (which approximately requires 140 GiB).

Notably, the throughput is higher when the LR factors are in 16 -bit compared to when they are in 4 -bit. This is because CALDERA used QuIP#'s lattice dequantizers for the low-rank factors as well, adding to the compute overhead. Moreover, QuIP# also used fused kernels, and CALDERA's throughput can be improved by leveraging such optimizations. Since this work is primarily motivated with the goal of closing the gap with respect to uncompressed models in the 2 to

2 . 5 bits per parameter regime, throughput improvement using custom kernels, paged attention, etc., is left for future work. We discuss the broader impacts of our work along with limitations in App. H.

6 Conclusions

In this work, the problem of obtaining a low-precision and low-rank decomposition of an LLM weight matrix was considered. A Q + LR decomposition efficiently captures the high singular components of the weight matrix with sufficient fidelity, while coarsely compressing the less significant moderateto-low singular components. An optimization-theoretically motivated algorithm was proposed to obtain this decomposition, which iteratively optimized the quantized backbone Q and the low-rank factors L , R . Additionally, it was shown that L and R can be efficiently fine-tuned using low-rank adaptation to boost the zero-shot performance of the quantized model. By utilizing a rank-constrained regression framework, an upper bound was established on the approximation error of the algorithm, and it was shown that this upper bound can be significantly smaller than prior bounds in the literature. Finally, the proposed method was empirically evaluated by compressing the LlaMA family of LLMs in the challenging sub2 . 5 bits per parameter regime. The proposed approach can also be used to complement existing compression strategies; thereby making it efficient to distribute compressed LLMs and deploy them on regular consumer hardware, making them more accessible to researchers.

References

L. Bentivogli, I. Dagan, H. T. Dang, D. Giampiccolo, and B. Magnini. The Fifth PASCAL Recognizing Textual Entailment Challenge, 2009.

Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence , 2020.

J. Chee, Y. Cai, V . Kuleshov, and C. D. Sa. QuIP: 2-Bit Quantization of Large Language Models With Guarantees. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457v1 , 2018.

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=OUIFPHEgJU .

J. Dodge, M. Sap, A. Marasovi´ c, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, 2021.

V. Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh. Extreme Compression of Large Language Models via Additive Quantization, 2024. URL https: //arxiv.org/abs/2401.06118 .

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=tcbBPnfwxS .

L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/ records/10256836 .

G. H. Golub and C. F. van Loan. Matrix Computations . JHU Press, fourth edition, 2013. ISBN 1421407949 9781421407944. URL http://www.cs.cornell.edu/cv/GVL4/ golubandvanloan.htm .

R. Gray and T. Stockham. Dithered quantizers. IEEE Transactions on Information Theory , 39 (3):805-812, 1993. doi: 10.1109/18.256489.

S. Gugger, L. Debut, T. Wolf, P. Schmid, Z. Mueller, S. Mangrulkar, M. Sun, and B. Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: //github.com/huggingface/accelerate , 2022.

H. Guo, P. Greengard, E. P. Xing, and Y. Kim. LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning. arxiv:2311.12023, 2023. URL https://arxiv.org/abs/2311.12023 .

N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review , 53(2):217-288, 2011. doi: 10.1137/090771806. URL https://doi.org/10.1137/090771806 .

Y.-C. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, and H. Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=uPv9Y3gmAI5 .

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

W. Huang, X. Ma, H. Qin, X. Zheng, C. Lv, H. Chen, J. Luo, X. Qi, X. Liu, and M. Magno. How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study, 2024.

A. Kaushal, T. Vaidhya, and I. Rish. LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression, 2023.

S. Keisuke, L. B. Ronan, B. Chandra, and C. Yejin. WinoGrande: An Adversarial Winograd Schema Challenge at Scale, 2019.

A. Krishnamoorthy and D. Menon. Matrix inversion using cholesky decomposition. In 2013 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA) , pages 70-72, 2013.

H. J. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'12, page 552-561. AAAI Press, 2012. ISBN 9781577355601.

Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. LoftQ: LoRAFine-Tuning-Aware Quantization for Large Language Models. arxiv:2310.08659, 2023. URL https://arxiv.org/abs/2310.08659 .

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation, 2024.

S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, J. Xue, and F. Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024.

S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer Sentinel Mixture Models, 2016.

Meta AI. Introducing Meta Llama 3: The most capable openly available LLM to date. https: //ai.meta.com/blog/meta-llama-3/ , 2024. Accessed: 2024-05-07.

M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. Up or Down? Adaptive Rounding for Post-Training Quantization. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 7197-7206, 2020.

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models, 2020.

S. Raschka. Building a gpt-style llm classifier from scratch, 2024. URL https://magazine. sebastianraschka.com/p/building-a-gpt-style-llm-classifier . Blog post.

J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '20, page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/ 3394486.3406703 .

R. Saha, V. Srivastava, and M. Pilanci. Matrix Compression via Randomized Low Rank and Low Precision Factorization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=rxsCTtkqA9 .

L. Schuchman. Dither Signals and Their Effect on Quantization Noise. IEEE Transactions on Communication Technology , 12(4):162-165, 1964. doi: 10.1109/TCOM.1964.1088973.

P. Sharma, J. T. Ash, and D. Misra. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction, 2023.

Together Computer. Redpajama: an open dataset for training large language models, October 2023. URL https://github.com/togethercomputer/RedPajama-Data .

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.

A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. D. Sa. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, 2024.

M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science , 1(1):144-160, 2019. doi: 10.1137/18M1183480. URL https://doi.org/10.1137/18M1183480 .

R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596.

M. Viazovska. The sphere packing problem in dimension 8 . Annals of Mathematics , 185(3), May 2017. ISSN 0003-486X. doi: 10.4007/annals.2017.185.3.7. URL http://dx.doi.org/ 10.4007/annals.2017.185.3.7 .

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A MultiTask Benchmark and Analysis Platform for Natural Language Understanding, 2019. In the Proceedings of ICLR.

S. Xiang, Y. Zhu, X. Shen, and J. Ye. Optimal exact least squares rank minimization. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '12, page 480-488, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450314626. doi: 10.1145/2339530.2339609. URL https: //doi.org/10.1145/2339530.2339609 .

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning , 2023.

Z. Yao, X. Wu, C. Li, S. Youn, and Y. He. ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation, 2023.

Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, 2023.

C. Zhang, J. Cheng, G. A. Constantinides, and Y. Zhao. LQER: Low-Rank Quantization Error Reconstruction for LLMs, 2024.

Contents

A Notations

This section begins by outlining key notations used in both linear algebra and probability theory. Boldface uppercase and lowercase letters, such as A and a , represent matrices and vectors respectively. I denotes the identity matrix, and its dimension is assumed to be imminent from the context. The first k columns of the identity matrix I as I k , and let I k be the submatrix formed by the last k columns. The singular values of A are denoted by σ max ( A ) = σ 1 ≥ σ 2 ≥ . . . ≥ σ r = σ min ( A ) , where r = rank( A ) . Similarly, the eigenvalues are denoted as λ 1 ( A ) , . . . , λ r ( A ) . The max-norm of A is defined as ∥ A ∥ max = max i,j | A ij | , the spectral norm of A is defined as ∥ A ∥ 2 = sup ∥ x ∥ =1 ∥ Ax ∥ =

For any matrix X , its Moore-Penrose pseudo-inverse is denoted by X † . The notations ≈ , ≲ and ≳ are used to denote approximate equality and inequalities that hold asymptotically in the limit when dimensions grow to infinity. In other words, for any two functions A ( n ) and B ( n ) ,

Notations ≈ and ≳ are defined analogously. Wherever relevant, dimension-dependent terms are highlighted blue.

B Rank-constrained Regression

Recall that the submatrix formed by the first m columns of the identity matrix I is denoted as I m , and let I m be the submatrix formed by the last m columns. The dimension of I is inferred depending on context. Consider the following:

Although this problem is non-convex, it can be solved to global optimality via two SVDs. The following lemma characterizes the solution the rank-constrained regression in (5).

Lemma B.1. (Global optimality of rank-constrained regression) Suppose Y ∈ R m × n is given, and suppose X ∈ R m × d is full rank, i.e., rank( X ) = min { m,d } , with SVD, X = U ˜ ΣV ⊤ . Furthermore, let ` U ` Σ ` V ⊤ denote the full SVD of U ⊤ Y if m ≤ d , or the full SVD of ( UI d ) ⊤ Y if m>d . Then, for k ≤ m , the solution of (5) is given by

Here, Σ is a diagonal matrix of the non-zero singular values of X , defined as Σ := ˜ ΣI m ∈ R m × m when m ≤ d , and Σ := I ⊤ d ˜ Σ ∈ R d × d when d < m . Additionally, the optimal value is min rank( Z ) ≤ k ∥ XZ -Y ∥ 2 F

Proof. Case m ≤ d : Since the full SVD of X ∈ R m × d is X = U ˜ ΣV ⊤ , where U ∈ R m × m , ˜ Σ ∈ R m × d , and V ∈ R d × d , the last ( d -m ) columns of ˜ Σ will be zero, i.e., ˜ ΣI m -d = 0 . Let Z ′ := V ⊤ Z ∈ R d × n be the transformed optimization variable. Since V is a unitary matrix, rank( Z ) ≤ k if and only if rank( Z ′ ) ≤ k . Splitting Z ′ ∈ R d × n into Z ′′ := I ⊤ m Z ′ ∈ R m × n and Z ′′ := I ⊤ d -m Z ′ ∈ R ( d -m ) × n , it can be seen that ˜ ΣZ ′ = ˜ ΣI d Z ′′ + ˜ ΣI m -d Z ′′ = ΣZ ′′ , where Σ := ˜ ΣI m ∈ R m × m is a diagonal matrix comprised of the non-zero singular values of X . Then,

Note that the objective function value is independent of Z ′′ , as Z ′′ lies in the null space of X ⊤ , and rank( Z ′′ ) ≤ k follows from the fact that rank of a submatrix cannot exceed the full matrix. Since X is full rank, Σ is invertible, and the minimization in (10) is equivalent to

where ˜ Z := ΣZ ′′ ∈ R m × n . The equality in (11) follows as a consequence of Eckart-Young-Mirsky theorem (lemma G.1), which states that ∥ ˜ Z -U ⊤ Y ∥ 2 F is minimized by taking the best rankk approximation of U ⊤ Y , and the fact that the singular values of U ⊤ Y and Y are the same as U is unitary. Moreover, as the SVD of U ⊤ Y is ` U ` Σ ` V ⊤ ,

In other words, if Z ′ ∗ denotes the solution of the middle optimization problem in (10), I ⊤ m Z ∗ = Z ′′ ∗ . Furthermore, note that setting I ⊤ d -m ˜ Z ∗ = 0 ensures rank( Z ′ ∗ ) = rank( Z ′′ ∗ ) ≤ k , while keeping the objective value in (10) unchanged. Hence, an optimal solution is given by

Case m > d : Recalling that the full SVD of X ∈ R m × d is X = U ˜ ΣV ⊤ , where U ∈ R m × m , ˜ Σ ∈ R m × d , and V ∈ R d × d , in this case, the last m -d rows of ˜ Σ will be zero, i.e., I ⊤ m -d ˜ Σ = 0 . Denote Z ′ := V ⊤ Z . This time, as Σ := I ⊤ d ˜ Σ ,

where Z ′′ := ΣZ ′ = ΣV ⊤ Z . Note that the term ∥ ∥ ( UI m -d ) ⊤ Y ∥ ∥ 2 F is the irreducible error, and (i) is, once again, a consequence of the fact that ∥ ∥ Z ′ -I ⊤ d U ⊤ Y ∥ ∥ 2 F is minimized by taking the best rankk approximation of I ⊤ d U ⊤ Y . Moreover, since the SVD of I ⊤ d U ⊤ Y is ` U ` Σ ` V ⊤ ,

Remark : It is not necessary for X to be full rank. An equivalent result can be derived with Σ -1 replaced by Σ † .

Computational complexity of rank-constrained regression : Arriving at the globally optimal solution in lemma B.1 requires computing two SVDs, namely X ∈ R m × d , which entails a complexity of O( dm 2 ) , and U ⊤ Y ∈ R m × n , with a complexity of O( nm 2 ) . Hence, the total computational complexity is O( m 2 ( n + d )) .

C Derivations for Calibration-Aware Low-Precision and Low-Rank Decomposition: CALDERA

C.1 Update Equations for Low-Rank Factors in LPLRFACTORIZE submodule

As discussed in §3, the left and right low-rank factors in the LPLRFACTORIZE sub-module (Alg. 2) are found by solving least squares minimization problem. The closed form expressions can be obtained by solving the normal equations directly. In what follows, the same expressions are also derived explicitly via the singular value decomposition, as it gives an expression for the error (for example, refer to (17)) - which consists of an additive and irreducible error term.

Updating L with fixed R : For a fixed R ∈ R k × d , the left low rank factor is computed in lines 5 and 9 of Alg. 2 by solving

Let XR ⊤ = U XR ˜ Σ XR V ⊤ XR be the full SVD of XR ⊤ , where U XR ∈ R m × m , ˜ Σ XR ∈ R m × k , and V XR ∈ R k × k . Recall from App. B that I denotes the submatrix formed by the first m columns of the identity matrix I m , and I m is the submatrix formed by the last m columns. Denoting Σ XR := I ⊤ k ˜ Σ XR and Z ′ := V ⊤ XR Z , since I ⊤ m -k ˜ Σ XR = 0 , it can be seen that

The last term of (17) is minimized by setting Z ′ ← Z ′ ∗ := Σ -1 XR ( U XR I k ) ⊤ Y . Since Z ′ ∗ = V ⊤ XR Z ∗ = V ⊤ XR ` L ⊤ , this yields the left low rank factor to be

` L ⊤ = V XR Σ -1 XR ( U XR I k ) ⊤ XA ⊤ = ⇒ ` L = ( AX ⊤ )( RX ⊤ ) † (i) = AHR ⊤ ( RHR ⊤ ) -1 , (18) where (i) follows from the explicit expression for the pseudoinverse of the wide matrix RX ⊤ ∈ R k × d .

Updating R with fixed L : For a fixed L ∈ R n × k , the right low-rank factor is computed in lines 4 and 8 of Alg. 2 by solving

Let X = U X ˜ Σ X V ⊤ X be the full SVD of X ∈ R m × d , where U X ∈ R m × m , ˜ Σ X ∈ R m × d , and V X ∈ R d × d . Then, denoting Z ′ := ZL ⊤ , the minimization (19) is equivalent to

Note that (20) is an undetermined linear system with multiple solutions. In particular, since ˜ Σ X ∈ R m × d , the last ( d -m ) columns of ˜ Σ X consist of zeros, i.e., ˜ Σ X I d -m = 0 . Therefore, a solution of this system is given by

This implies ` R = L † AX ⊤ U X ( ˜ Σ † X ) ⊤ V X .

The proof is completed by noting that †

C.2 Dynamic Ranges of Quantizers

Lemmas C.1 and C.2, stated in this section provide sufficient conditions to ensure that the quantizers Q L and Q R , used to quantize the left and right low-rank factors in the LPLRFACTORIZE submodule, remain unsaturated.

Lemma C.1. (Dynamic range of right quantizer) Given matrices A ∈ R n × d and X ∈ R m × d , let σ max denote the maximum singular value of XA ⊤ . Then, the quantizer Q R remains unsaturated if the dynamic range is chosen to be R R = σ max .

Since the input to Q L is dependent on the output of Q R , the following lemma C.2 provides an upper bound on the input to Q L , provided that Q R was unsaturated.

Lemma C.2. (Dynamic range of left quantizer) Given matrices A ∈ R n × d and X ∈ R m × n , let λ min denote the smallest eigenvalue of H = 1 m X ⊤ X , and let σ max and σ k denote the largest and the k th singular values of XA ⊤ , respectively. For some small ϵ > 0 and an absolute constant C , suppose the number of bits for the right quantizer Q R satisfies

Then, if the dynamic range of Q L is chosen to be

then Q L remains unsaturated with probability exceeding 1 -0 . 25 ϵ ∥ ∥ XA ⊤ ∥ ∥ -2 F .

Proof. In Line 5 of Alg. 2, the input to Q L is ` L 0 = ( WX ⊤ ) ( R 0 X ⊤ ) † . In the rest of the proof, the subscript 0 in ` L 0 and ` R 0 is dropped for brevity. Recall the notation for the full SVD of XR ⊤ , i.e., XR ⊤ = U XR ˜ Σ XR V XR , where U XR ∈ R m × m , ˜ Σ XR ∈ R m × k , and V XR ∈ R k × k . From Eq. (18), ` L can be expressed as ` L = V XR Σ † XR ( U XR I k ) ⊤ XA ⊤ . Consequently, ∥ ` L ∥ max is upper bounded as

Here, (i) holds because V XR is a unitary matrix, (ii) follows from submultiplicativity of spectral norm, and (iii) follows from the fact that ( U XR I k ) ( U XR I k ) ⊤ ≼ I and lemma G.4. Furthermore,

where σ min ( Σ XR ) is the smallest non-zero singular value of XR ⊤ , (i) follows from lemma G.4 and (ii) follows from lemma G.5. Here, E R ∈ R k × d is the quantization error from quantizing right low-rank factor, and consist of unbiased random variables with bounded variance, as described in lemma G.6. Since ` Σ contains the singular values of U ⊤ XA ⊤ ,

where (i) and (ii) follow because ` V and U are unitary matrices. Substituting (27) in (26) and (26) in (25),

Lemma C.1 suggests choosing the dynamic range of Q R to be ∥ XA ⊤ ∥ 2 to ensure that Q R remains unsaturated. As a result,

where ∥·∥ ψ 2 denotes the subgaussian norm (refer to lemma G.2). Subsequently, lemma G.2 yields that for some absolute constant C ,

Setting t = √ log ( 8 ∥ XA ⊤ ∥ 2 F ϵ ) yields

with probability exceeding 1 -ϵ 4 ∥ XA ⊤ ∥ 2 F . Note that if the bit budget B R is chosen to satisfy then ∥ E R ∥ 2 ≤ σ k / 2 . Consequently, from (28),

This completes the proof.

C.3 Proof of Lemma C.3: Approximation Error Upper Bound for LPLRFACTORIZE

Lemma C.3. (Approximation error of LPLRFACTORIZE ) Given A ∈ R n × d and X ∈ R m × d with m ≤ d , let λ max and λ min denote the maximum and minimum eigenvalues of H = 1 m X ⊤ X , and let

σ 1 ≥ . . . ≥ σ k denote the singular values of XA ⊤ . Suppose the target rank k satisfies

the dynamic ranges of quantizers Q L and Q R are respectively set as

and for some absolute constant C and arbitrarily small ϵ that satisfies 0 < ϵ ≤ 4 mkλ 2 max λ -1 min σ 1 , suppose the bit-budgets of Q L and Q R satisfy

Then, the factors L , R returned by Alg. 1 satisfy E ∥ ∥ ( LR -A ) X ⊤ ∥ ∥ 2 F ≤ ∑ i>k σ 2 i + ϵ , where the expectation is over the stochasticity of quantizers Q L and Q R .

Proof. Firstly, note that upper bounding the error E ∥ ( A -L 0 R 0 ) X ⊤ ∥ 2 F suffices, since the lines 7 to 13 in Alg. 1 refine the estimates of L and R and can only yield a smaller Frobenius norm error. Consider the quantized low rank factorization A ≈ Q L ( ` L 0 ) Q R ( I ⊤ k ` Σ ` V ⊤ ) . Let denote the quantization error matrices. Furthermore, let

denote the optimal value of the unquantized rank-constrained regression problem. Since the dynamic ranges of quantizers Q R and Q L are chosen according to lemmas C.1 and C.2 respectively, the entries of E L and E R are unbiased random variables with bounded variance as in lemma G.6. Let ∆ R := 2R R / (2 B R -1) , and ∆ L := 2R L / (2 B L -1) denote the quantization resolutions. Conditioned on the event that the left quantizer Q L is unsaturated, the error matrix satisfies E E L = 0 . This yields,

where (i) follows from the fact that the error matrix is unbiased.

where inequality (i) is obtained by replacing the minimizer ` L 0 with a different, but appropriately chosen matrix. Here, Σ is the diagonal matrix containing the non-zero singular values of X .

Since κ 2 := λ max /λ min , the second term in (37) is

Moreover, since E E R = 0 , term T 2 can be upper bounded as

Here, (i) follows since V is a unitary matrix, (ii) follows from lemma G.4, (iii) follows from the cyclic property of trace, (iv) follows as ` U is unitary, and (v) follows since I ⊤ k I k = I as k ≤ m .

The first term in (39) is

where (i) and (ii) follow from lemma G.4 and the fact that ` V is a unitary matrix. Similarly, the second term of (39) is

Here, (i) follows because E [ E ⊤ L E L ] is a diagonal matrix since

As a consequence of (37) to (41),

̸

Since R R = σ 1 , the second term of (43) does not exceed 0 . 25 ϵ if

This needs to be considered in conjunction with the lower bound on B R in lemma C.1, which yields the maximum of two quantities in the theorem statement. Since R L = 2 σ 1 σ k √ mλ min , the third term of (43) does not exceed 0 . 25 ϵ if

provided Q L stays unsaturated. Furthermore, if ϵ ≤ 4 kmλ 2 max λ -1 min σ 1 , choosing B R and B L as above ensures that the third term of (43) is also upper bounded by 0 . 25 ϵ .

The approximation error upper bound in (43) holds conditioned on the event that Q L was unsaturated, which according to lemma C.2, is ensured with probability exceeding 1 -0 . 25 ϵ ∥ ∥ XA ⊤ ∥ ∥ -2 F . For analysis purposes, it is assumed that when Q L gets saturated, Alg. 1 returns L = 0 and R = 0 , as a result of which, the approximation error is upper bounded by ∥ XA ⊤ ∥ 2 F . 4 Then, since Pr(Q L is unsat. ) ≥ 1 -0 . 25 ϵ ∥ XA ⊤ ∥ 2 F , using Cauchy-Schwarz inequality for expectations,

This completes the proof.

C.4 Proof of Thm. 4.1: Approximation Error Upper Bound for CALDERA

Theorem C.4. Approximation error of CALDERA (Formal) Given W ∈ R n × d and X ∈ R m × d with m ≤ d , let D be obtained from the LDL decomposition X ⊤ X = m H = ( M + I ) D ( M + I ) ⊤ , and λ max , λ min denote the max and min eigenvalues of H . Additionally, let Q ≜ LDLQ ( W , Q Q ) , where Q Q has dynamic range R and bit-budget B Q , the quantization error be η ≜ Q Q ( Q +( W -Q ) M ) -( Q +( W -Q ) M ) , and σ 1 ≥ . . . ≥ σ k . . . be the singular values of X ( W -Q ) ⊤ . If the target rank k satisfies

the dynamic ranges of Q L and Q R are set as

and for some absolute constant C and arbitrarily small ϵ that satisfies 0 < ϵ ≤ 4 mkλ 2 max λ -1 min σ 1 , suppose the bit-budgets of Q L and Q R are set so that they satisfy

B

R

{

1

2

}

1

2

1

kd

ϵ

λ

λ

min

then Q , L and R returned by CALDERA (Alg. 1) satisfy

B

,

B

,

where

B

:= log and

(

2

σ

√

+1

)

,

where the expectation is over the stochasticity of the quantizers Q Q , Q L and Q R .

Proof. As in §C.3, an upper bound is obtained on E ∥ ∥ ( Q 0 + L 0 R 0 -W ) X ⊤ ∥ ∥ 2 F , since after the first iteration, the alternating steps 3 and 4 in Alg. 2 simply refine the estimates of Q , L and R , and can only yield a smaller error. For convenience of notation, in what follows, the subscript 0 is omitted.

Since L , R is obtained after applying LPLRFACTORIZE submodule to ( W -Q ) , lemma C.3 suggests that if R L , R R , B L , and B R are chosen appropriately,

where the expectation in the upper bound on the right hand side is over the randomness in Q due to the stochasticity in quantizer Q Q . Since Q = LDLQ ( W , Q Q ) is the LDLQ quantizer of Chee et al. [3] (or, BLOCKLDLQ quantizer of Tseng et al. [36], which is a successor of LDLQ proposed by Chee et al. [3]), it is shown that

where M is a strictly upper triangular matrix. This is a consequence of the fact that LDLQ quantizes one column at a time, while simultaneously incorporating a linear feedback from the already quantized columns. Moreover, if η = Q Q ( W +( W -Q ) M ) -( W +( W -Q ) M ) denotes the quantization error, then it can be seen that

Moreover, since m H = X ⊤ X = ( M + I ) D ( M + I ) ⊤ ,

where (i) follows from (48). From (47),

Since η D η ⊤ = ∑ d j =1 D jj η j η ⊤ j , where D jj denotes the j th diagonal entry of D and η j is the j th column of η ∈ R n × d ,

Note that η consists of quantization errors, which, for uniformly dithered scalar quantization, are zero mean random variables with bounded variance upper bounded by ∆ 2 4 = R 2 (2 B Q -1) 2 . As the exact error distribution for non-subtractively dithered quantization (which is commonly used in practice), is not fully understood, a simplification is made for easier analysis, assuming the quantizer is subtractively dithered. For subtractively dithered quantizers, if Schuchman's conditions are met [11, 32], this results in a uniform error distribution in the interval [ -∆ 2 , ∆ 2 ] . Therefore, assuming quantization errors are independent and identically distributed, for large dimensions d , the eigenvalues of 1 d ∑ d j =1 η j η ⊤ j are distributed according to the Marchenko-Pastur distribution, given by f mp ( x ) := 2 d πn ∆ 2 x √ ( λ + -x ) ( x -λ -) , where λ ± := ∆ 2 4 ( 1 ± √ n d ) 2 . Furthermore, denoting q := F -1 mp ( 1 -k n ) , where F -1 mp is the inverse cumulative distribution function (CDF) of f mp ( x ) , it can be seen that

Here, ≈ denotes that the equaility holds asymptotically, (i) follows from upper bounding the integral using the fact that ( λ + -x ) ( x -λ -) is maximized at x = λ + + λ -2 . Furthermore, (ii) follows after substituting the values of λ ± and using the following linear upper bound on the inverse CDF of Marchenko-Pastur distribution,

It can be verified by inspection that F mp ( x ) is lower bounded by ( λ + -λ -2 ) x -( λ 2 + -λ 2 -4 ) , and inequality (53) is a consequence of this fact. Furthermore, from lemma G.3, max j D jj ≤ λ max . Substituting (52) in (51), and substituting ∆ 2 = 4R 2 (2 B Q -1) 2 , completes the proof.

Informal version of Thm. C.4 : Consider n ≈ d . From Thm. C.4, a (simplified) asymptotic dependence on the bit-budgets B L and B R can be obtained. In what follows, constant factors inside the log 2 ( · ) have been ignored. Comparing the expressions for B 1 and B 2 , it can be seen that the desired bit-budgets are

For n ≈ d , this yields an average bit-budget of

C.5 A simplification for positive definite Hessians

The expressions in (8) are a little involved, but they can be simplified if H is positive definite, i.e., all eigenvalues are strictly greater than 0 . Let H = UΛU ⊤ be the eigenvalue decomposition of H , and let H 1 2 = U Λ 1 2 U ⊤ be its symmetric square root. Furthermore assume that H is positive definite, i.e., all diagonal entries of Λ are strictly positive. In practice, this can be ensured by regularizing H through the addition of an identity matrix, scaled by a small amount.

Then, consider the following equivalent optimization problems:

Here, Z ′ ≜ ZUΛ 1 2 , and the constraint rank( Z ′ ) ≤ k follow from the fact that multiplying Z by an invertible matrix, UΛ 1 2 , keeps its rank unchanged. The final optimization problem can be solved optimally by considering the SVD of Y as Y = UΣV ⊤ , and the optimal solution is

where I k denotes the first k columns of the identity matrix. So the final solution is given by,

This yields a closed form expression for the optimal solution of the rank-constrained regression problem for the case of positive definite Hessians.

D Computational Complexity of CALDERA

Complexity of pre-preprocessing : For a given calibration dataset/activation X ∈ R m × d , the Hessian is computed as H = 1 m X ⊤ X , which requires O( md 2 ) compute. Subsequently, the LDL decomposition of H is computed, which entails O( d 3 ) complexity [20]. Moreover, HH † , which is used in the update equation of ` R is also computed beforehand, with a complexity of O( d 3 ) . therefore, the total complexity of pre-processing is O( md 2 +2 d 3 ) .

Complexity of LDLQ: Each iteration of CALDERA invokes a single call to LDLQ. An LDLQ call involves updating each column of an n × d iteratively. From (2), updating the k th column requires multiplying a matrix (of already quantized columns) in R n × ( k -1) with a vector in R k -1 , implying a complexity of O( n ( k -1)) . Subsequently, if Q Q is uniform scalar quantization, quantizing the (feedback incorporated) k th column entails O( n ) compute, implying a total of O( nk ) compute for the k th column. Hence, quantizing all columns from k = 1 , . . . , n would require O( n ∑ n k =1 k ) = O ( n 2 2 + 3 n 2 ) = O( n 2 ) compute.

Complexity of LPLRFACTORIZE: Every iteration of CALDERA invokes a call to LPLRFACTORIZE, which itself consists of T in inner iterations. Each inner iteration of LPLRFACTORIZE, consists of initializing the left and right low rank factors using rank-constrained regression. As seen in §B, it has a computational complexity of O ( m 2 ( n + d ) ) . Subsequently, for any matrix A , the left low rank factor as ` L = AHR ⊤ ( RHR ⊤ ) -1 can be found using successive matrix multiplications. While a gradient descent based algorithm can possibly speed up this computation, in implementation, closed form expressions are computed directly as they are computationally affordable for the hidden dimensions of the LLMs considered. Computing A ( HR ⊤ ) requires O( dk ( n + d )) , computing ( RHR ⊤ ) -1 entails O ( dk ( k + d ) + k 3 ) , and multiplying them together requires O ( nk 2 ) . Keeping the dominant term, the aggregated computational complexity for computing the left low-rank factor in each inner iteration is O( ndk ) .

Additionally, the left low-rank factor can be computed as ` R = L † A ( HH † ) , where computing L † A is O( ndk ) . Since HH † is already computed beforehand, multiplying L † A and HH † entails O ( kd 2 ) . Once again, keeping the dominant term, the complexity is O( ndk ) . Hence, the total complexity of each inner iteration of LPLRFACTORIZE is O( ndk ) .

Remark : Compared to the two-stage LPLR algorithm proposed in [31], the left and right low-rank factors are iteratively refined in Alg. 2 since this additional compute can be afforded for quantizing LLM weight matrices. Moreover, SVD computations can be sped up with randomized SVD [14].

E Additional Experimental Details

E.1 System Details

The code for this paper is available at https://github.com/pilancilab/caldera. The framework is built in PyTorch on top of the QuIP# [36] and LoftQ [22] repositories, and utilizes HuggingFace implementations of all datasets and LLMs. Experiments were performed on either NVIDIA RTX A6000, NVIDIA A10G, or NVIDIA H100 GPUs. Hessian computation for all but LLaMa-2 70B was performed on four NVIDIA A10 GPUs (provisioned as Amazon AWS EC2 instances), parallelized by distributing different layers to different GPUs. Hessian computation for LLaMa-2 70B was performed on a single H100 GPU. Model quantization was performed on four NVIDIA RTX A6000 GPUs, parallelized in the same manner as Hessian computation. All zero-shot and fine-tuning experiments were also run on A6000 GPUs, except for LLaMa-2 70B zero-shot experiments, which were run on an H100 GPU. Low-rank adaptation experiments were parallelized across four GPUs via HuggingFace Accelerate [12], with DeepSpeed ZeRO Level 2 [28, 30]. The use of Llama family of LLMs for research is guided by a community license that can be founded at https://llama.meta.com/license/ and https://llama.meta.com/llama3/license/.

E.2 Parameters of CALDERA Quantization

For all CALDERA decompositions, the number of alternating iterations between updating Q and L , R (i.e., T out in Alg. 1) is 15 . For decompositions with quantized low-rank factors, except LLaMa-2 7B and LLaMa-3 8B, the number of LPLR iterations (i.e., T in in Alg. 2) is 10 . For LLaMa-2 7B and LLaMa-3 8B, the number of LPLR iterations is 50 .

For the LLaMa-2 7B and LLaMa-3 8B models (for both CALDERA and QuIP# decompositions), the calibration dataset consists of 256 random samples from the RedPajama dataset. For all other models, the calibration dataset provided by the QuIP# [36] authors was used. This calibration dataset consists of 6144 random samples from RedPajama.

The number of tokens in each calibration data point is equal to the context length of the corresponding model, i.e., 4096 for LLaMa-2 and 8192 for LLaMa-3 8B.

An additional heuristic applied during the CALDERA decomposition is an update to the Hessian matrix based on the subspace spanned by LR :

Where H = MM ⊤ is the LDL decomposition of the Hessian and LRM = UΣV ⊤ is the singular value decomposition of the product LRM . The updated ˜ H is used in all LPLR updates, and the original Hessian is used for updating the low-rank factors via Alg. 2. This heuristic was found in the QuIP# codebase, and empirically speeds up convergence of CALDERA, as is discussed in App. F.1.

Although there is randomness in the generation of the randomized Hadamard matrices, we found that it has negligible effect on the results, due to the iterative nature of the CALDERA algorithm. As such, the effect of random seeds is minimal.

E.3 Details about Goodness-of-Fit Metrics

In addition to perplexity on the WikiText2 and C4 datasets, CALDERA was evaluated using zero-shot accuracy on the following sequence classification tasks:

Winogrande [19]: a collection of 44k problems problems inspired by the Winograd Schema Challenge [21], specifically designed to be robust to learned biases in model training datasets. Specifically, Winogrande is a two-choice fill-in-the-blank task that requires significant commonsense reasoning.

RTE [1]: Recognizing Text Entailment is a task in the General Language Understanding Evaluation (GLUE) benchmark [40]. Given two statements, the model must classify whether or not the second follows from the first.

PiQA [2]: Physical Interaction: Question Answering is a question-answering task that requires understanding about physical relationships between objects.

ArcE [4]: In general, the AI2 Reasoning Challenge is a set of multiple-choice questions that require a grade-school level of knowledge. ArcE is the subset that is labeled Easy.

ArcC [4]: The ARC-Challenge problems are in the same format as ARC-Easy, but the dataset only includes problems that were answered incorrectly by selected algorithms.

The unquantized ( 16 -bit) numbers mentioned in Tabs. 1 and 2 are either copied from [36] for the tasks available reported therein. For some ones that could not be easily found, inference was performed directly on the 16 -bit model downloaded from HuggingFace.

E.4 Fine-tuning Parameter Details

Fine-tuning of the diagonal Rademacher matrices in the RHT was performed over a calibration dataset sampled from the training split of RedPajama. Due to computational constraints, each sample contains 512 tokens, rather than the full context length of the model. The calibration dataset is 256 samples in total, with 192 data points in the training split and 64 in the evaluation split. RHT fine-tuning was performed for 5 epochs with a learning rate of 10 -3 . The epoch with the lowest evaluation loss was used for further zero-shot and fine-tuning experiments.

Parameters for low-rank adaptation experiments can be found in Table 8. Overall, the number of epochs and therefore number of training steps, was determined empirically based on when the evaluation accuracy plateaued.

For the comparison of CALDERA to LQ-LoRA in Winogrande accuracy, the LQ-LoRA result reported in [13] did not involve fine-tuning directly on the Winogrande dataset, but rather on a subset of C4 and Wikitext2.

E.5 Computation of Average Bit Budget Per-Parameter

Assume that the dimensions of the seven weight matrices in each transformer layer have dimensions n i × d i , where i ∈ { 1 , . . . , 7 } . Also, for the models we consider, each transformer layer has matrices of the same dimensions. In general, if the full-rank matrix Q has a bit budget of B Q, the factors L and R both have a bit budget of B LR, and the rank of the factors is k , the average number of bits per parameter is

For LLaMa-2 7B, all self-attention matrices are 4096 × 4096 , the gate and up projections are 11008 × 4096 , and the down projection is 4096 × 11008 . So, ( n i , d i ) = (4096 , 4096) for i ∈ { 1 , . . . , 4 } , ( n 5 , d 5 ) = ( n 6 , d 6 ) = (11008 , 4096) , and ( n 7 , d 7 ) = (4096 , 11008) .

For LLaMa-3 8B, ( n 1 , d 1 ) = ( n 4 , d 4 ) = (4096 , 4096) , ( n 2 , d 2 ) = ( n 3 , d 3 ) = (4096 , 1024) , ( n 5 , d 5 ) = ( n 6 , d 6 ) = (14336 , 4096) , and ( n 7 , d 7 ) = (4096 , 14336) .

In §5.3, r = 64 of the k factors are fine-tuned in 16-bit precision. This results in the following number of bits per parameter:

E.6 Additional Notes on Numerical Simulations

For LLaMa-2 70B, the zero-shot perplexities and accuracies reported in the QuIP# paper [36] could not be replicated. As such, the QuIP# performance reported in this paper are based on recreations of those experiments.

F Additional Numerical Simulations

F.1 Ablation Study of CALDERA Parameters with respect to Frobenius Norm Error

The per-iteration relative Frobenius norm error of CALDERA, for a few selected weight matrices of LLaMa-2 7B, is plotted in Figure 3. The data-aware relative Frobenius norm error is defined as

where W is the unquantized weight matrix, X is the calibration data, and Q + LR is the CALDERA decomposition. The following ablations of CALDERA decomposition parameters are considered:

16-Bit Factors : Q is quantized to 2 bits via LDLQ. The low-rank factors are kept in half precision, so no LPLR iterations are required after rank-constrained regression is performed.

The randomized Hadamard transform is performed on the W and H matrices are prior to quantization.

4-Bit Factors : The low-rank factors are quantized to 4 bits of precision using an E8 lattice quantizer, and 10 iterations of LPLRFACTORIZE (Alg. 2) are run for each update of the factors (i.e., T in = 10 in Alg. 1). Otherwise, the decomposition parameters are the same as in 16-Bit Factors .

4-Bit Factors (No RHT) : The parameters of CALDERA are the same as in 4-Bit Factors , except the randomized Hadamard transform step is omitted.

4-Bit Factors (No LDLQ) : The parameters of CALDERA are the same as in 4-Bit Factors , except Q is quantized to 2 -bit precision using direct E8 lattice quantization instead of the LDLQ algorithm.

16-Bit Factors (Hessian Update) : The parameters of CALDERA are the same as in 16-Bit Factors , except the Hessian update step described in App. E.2 is performed prior to LDLQ.

For each ablation, the rank of L and R varies between k ∈ { 64 , 128 , 256 } . For comparison, QuIP# with the same-size low-rank factors is performed, using the same calibration data and the low-rank decomposition from the QuIP# codebase.

Overall, CALDERA with 16-bit factors consistently achieves a lower error than QuIP#. Additionally, the Hessian update heuristic improves convergence and often, but not always, reduces the final error achieved after 50 iterations of Alg. 1. CALDERA with 4-bit factors has higher Frobenius-norm error than with 16-bit factors, but the degradation is minor. On the other hand, replacing LDLQ with a lattice quantizer significantly degrades the Frobenius norm error, and omitting the randomized Hadamard transform worsens the error for some of the weight matrices considered.

To demonstrate the convergence of LPLRFACTORIZE (Alg. 2), the per-iteration relative data-aware error of LPLRFACTORIZE for the factorization W ≈ LR is plotted in Figure 4. For all curves plotted, the randomized Hadamard transform is performed on W and H before the factorization is computed, and both factors are quantized to 4 bits of precision via an E8 lattice quantizer.

In both cases, the alternating minimization iterations reduce the error. For the Up projection matrix, this reduction is nominal, whereas, for the Key projection matrix, the alternating iterations result in a significant improvement.

F.2 Experiments on Mistral-7B

Perplexity and language modeling benchmark accuracy results for quantizing Mistral 7B via CALDERA and QuIP# are in Table 9. Results are consistent with those in Section 5.1.

G Auxiliary Results

G.1 Useful Results from Linear Algebra and Probability

This section states some useful linear algebra results as lemmas. Some of them are stated without proof, which can be easily proved or found in a linear algebra textbook such as [10].

Lemma G.1. (Eckart-Young-Mirsky theorem) For any matrix A with SVD given by A = UΣV ⊤ , the solution of A k := arg min rank( ̂ A ) ≤ k ∥ A -̂ A ∥ 2 F is given by A k = ( UΣ 1 / 2 I k )( I ⊤ k Σ 1 / 2 V ⊤ ) , and ∥ A -A k ∥ 2 F = ∑ i>k σ 2 i ( A ) .

Lemma G.2. (Vershynin [38, Thm 4.4.5]) Let X be a d × m random matrix whose entries X ij are independent, zero-mean, subgaussian random variables. Then, for any t > 0 , ∥ X ∥ 2 ≤ CK ( √ d + √ m + t ) with probability exceeding 1 -2 e -t 2 , where K = max i,j ∥ A ij ∥ ψ 2 , and ∥·∥ ψ 2 denotes the subgaussian norm, and C is an absolute constant.

Remark : Lemma G.2 states a high probability upper bound on the spectral norm of a random matrix in terms of the subgaussian norm of the entries of the matrix. The subgaussian norm of a subgaussian random variable X is defined as ∥ X ∥ ψ 2 ≜ inf { t ≥ 0 | E [ e X 2 /t 2 ] ≤ 2 } . It can be shown that any bounded random variable X is subgaussian, and satisfies ∥ X ∥ ψ 2 ≤ ∥ X ∥ ∞ log 2 .

Lemma G.3. Suppose the LDL decomposition of a matrix H ∈ R d × d is given by H = ( M + I ) D ( M + I ) ⊤ , where M is strictly upper triangular and D is diagonal. Then, max j D jj ≤ λ max , where λ max denotes the largest eigenvalue of H .

Proof. Let e j denote the j th canonical basis vector. Then,

Moreover, from properties of eigenvalues, e ⊤ j He j ≤ λ max . Therefore, for any j , D jj ≤ D jj + ∑ i>j M 2 ji ≤ λ max . Since it holds true for all j , this completes the proof.

Lemma G.4. (Loewner ordering for matrix products) For any matrix A and B ,

LemmaG.5. (Minimum singular value) For matrices A and B , σ min ( A + B ) ≥ σ min ( A ) -∥ B ∥ 2 .

G.2 Uniformly dithered scalar quantizer

Let us consider quantizing a scalar x with | x | ≤ R . Given B bits, the scalar quantizer with dynamic range R is described by first specifying the M = 2 B quantization points

where ∆ = 2R M -1 is the resolution. The quantizer operation is defined as:

where k = arg max j { q j ≤ x } , i.e., x ∈ [ q k , q k +1 ) , and r = x -q k ∆ . As shown in the following lemma G.6, such a quantizer satisfies

i.e., it is unbiased and the error variance depends on R and B . Here, the E ( · ) is over the randomness from dithering in (59). If the input x to the quantizer falls outside this range, i.e., x > R or x < -R , the quantizer is said to be saturated . To quantize any matrix X , Q R , B ( X ) is obtained by quantizing each entry independently, i.e., [Q R , B ( X )] ij ≜ Q R , B ( X ij ) .

Lemma G.6. For scalar x with | x | ≤ R , denote the quantization error of uniformly dithered B -bit scalar quantizer as ϵ = Q R , B ( x ) -x . Then, E [ ϵ ] = 0 and Var ( ϵ ) ≤ R 2 (2 B -1) 2 , where the expectation E is over the randomness due to dithering in the quantizer operation.

Furthermore, the variance can be upper bounded as

When the input x to the quantizer exceeds R , the quantizer is said to be saturated , causing the quantization error to deviate from zero mean and bounded variance. Thus, it is crucial to ensure that the quantizer operates within the unsaturated regime with high probability.

H Limitations and Further Discussions

Our proposed algorithm CALDERA exploits low-rank structure in weight matrices of LLMs, and is seen to boost the performance of existing methods in the literature of LLM compression. It does so by providing an additional degree of freedom to control the compression ratio - namely, the target rank ( k ) . Theoretical guarantees show improved performance over existing benchmarks, when the matrix being compressed is inherently low-rank to begin with - something that is observed in LLMs. This also implies that CALDERA can be used for any other application scenarios that require matrix compression, as discussed in [31]. Its effectiveness largely depends on the presence of an inherent approximate low-rank structure, which is seen in many real-world matrices [37].

Despite attaining a lower loss for LLM compression, since CALDERA is designed to tackle an optimization problem through an iterative process, it requires slightly more computational resources. For instance, compressing Llama-2 7B or Mistral-7B models (with rank256 ) took approximately 34 GPU-hours (on NVIDIA A10G GPUs provisioned from an AWS G5 instance), and compressing Llama-2 13B took 59 GPU-hours (on a locally hosted NVIDIA A6000 GPU). Moreover, LLaMa-2 70B can be quantized via CALDERA with rank256 factors in approximately 90 GPU hours (on an H100 from Lambda labs), which is on par with QuIP# (with RHT finetuning), which reports 100 GPU hours (on A100). It should be noted that these wallclock times can be reasonably afforded, and are not prohibitively large . Furthermore, since the compression of an LLM is a one-time computational expense, this cost becomes highly amortized considering the frequent use of LLMs for inference following the deployment of the compressed model.

Although CALDERA often achieves perplexities and accuracies similar to unquantized models, a gap remains, as shown in Tables 1 and 2. This indicates there is potential for enhancing quantization strategies. For example, the target rank ( k ) can be treated as a hyper-parameter, and adjusted across different layers. Sharma et al. [33] demonstrate that such a layer-specific rank reduction can improve generalization capabilities. Additionally, improved fine-tuning strategies such as those proposed by Liu et al. [23], which reduce the gap between LoRA and full fine-tuning, can be incorporated with the low-rank adaptation step of CALDERA. Detailed investigations are left for future work.

Broader Impacts : Compressing models enables their deployment in resource-constrained settings, facilitating educational and technological advancements with limited infrastructure. Deploying on edge devices for inference also enhances privacy by reducing the need for data to be sent to centralized servers for inference, thereby enhancing user privacy and data security. Additionally, lower computational requirements of inference using compressed models is a step towards adoption of environment-friendly AI strategies.

On the other hand, as the compression is often not always lossless, any technique may result in reduced accuracy or loss of nuanced understanding, potentially impacting the reliability of the models in critical applications. Conseuqently, due diligence should be exercised when deploying these models. From a broader perspective, LLMs are quite powerful, and their easier deployment can possibly lead to misuse, such as the spread of misinformation or automated generation of harmful content. Consequently, robust regulatory frameworks are necessary, as LLMs continue becoming more accessible to the general audience.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract proposes an algorithm for compression large language models (LLMs) using low-precision and low-rank approximation. The algorithm, named CALDERA, presented in this work achieves that goal. The performance of CALDERA is studied theoretically by deriving explicit approximation error guarantees, as well as numerically by compressing Llama and Mistral families of models for a variety of tasks, and shows improved performance over existing benchmarks that don't take into account low-rankness of the weight matrices.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed explicitly in Appendix Sec. H. Assumptions made with regard to theory are stated in the theorem statement. These assumptions hold in practice as our algorithm is seen to perform well in numerical simulations. Extensive experimentation were performed on as many setups as allowed within our computational budget.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions are stated in the main Theorem statement of §4. A brief outline of the analysis (i.e., proof sketch) is provided in §4.1. Complete derivations of the proof are provided in App. C. The derivation is obtained by providing lemmas that build up to the proof, and the lemmas are appropriately cross-referenced. Auxiliary lemmas and well-known results that are used the derivation are stated as auxiliary lemmas in App. G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental details are provided in §5 and App. F. Open source datasets are used, and the codebase associated with the paper is also open-sourced.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codebase developed for this paper is open-sourced at https://github.com/pilancilab/caldera. The README file provides necessary instructions to reproduce experiments from the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details are provided in Sec. 5 and App. E. Code is opensourced at https://github.com/pilancilab/caldera, along with a README file that provides instructions on how to reproduce the results from the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: As LLMs are computationally expensive to process and run inference on, error bars are not reported. However, since code is open-sourced, results can be repeated with different configurations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details are provided in Sec. 5 and App. E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader societal impacts are discussed in App. H

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: No new datasets are used in this work. All datasets are already publicly open-sourced. Models compressed using CALDERA will be open-sourced.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the creators and original owners of assets used in this paper have been properly credited, and the associated license and terms of use mentioned and respected. Details can be found in §5 and App. E.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Compressed models will be released publicly.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowd-sourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowd-sourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework that combines quantization and low-rank approximation. Given a neural net weight matrix $W\\in \\mathbb{R}^{n\\times d}$, it considers the following decomposition: $W=Q+LR$, where $Q$ is a quantized sketch of $W$ with very few bits (2 or 4 in their experiments), and $L\\in \\mathbb{R}^{n\\times k}, R\\in \\mathbb{R}^{k\\times d}$ are low-rank factors that serve as a \"correction\" to the quantization sketch $Q$. They are usually quantized with more bits (4 or 16). To compute respective $Q, L, R$, the authors propose an alternating minimization framework: starts with $L, R=0$, update $Q$ as $Q_t=Q_Q(W-L_{t-1}R_{t-1})$, then solve for $L_t, R_t$ by solving the rank-constrained regression against $W-Q_t$ and $X\\in \\mathbb{R}^{m\\times d}$, where $X$ is the calibration dataset. They further show their algorithm could be used in conjunction with LoRA and fine-tuning the randomized Hadamard transform. Extensive experiments show that their proposed algorithm gives low perplexities and high accuracies when compared against earlier algorithms.\n\nSTRENGTHS:\nThis paper proposes a method that is a good mixture of quantization and low-rank approximation, as simply using only one of quantization sketch or low-rank approximation intuitively \"misses\" an important part. The algorithm proposed in this paper is a natural and simple alternating minimization framework. This paper also contains a theoretical analysis for approximation error under assumptions on the target rank. Empirically, it is also shown the proposed algorithm has good performance under small rank and bit precision.\n\nThe paper is also well-written.\n\nWEAKNESSES:\nOverall, I don't see big weaknesses of this paper. One potential direction for improvement is theoretically, the alternating minimization framework could possibly be sped up given $k\\ll n, d$, especially the rank-constrained regression, see [1]. On the other hand, it is not clear how to integrate these possible algorithmic pieces into post-processing a model weight, so I think the approach and runtime analysis of this paper are fair.\n\n[1] Gu, Song, Yin and Zhang. Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time. ICLR'24.\n\nQUESTIONS:\nIf we allow $L$ and $R$ to be computed to full bit-precision, could the statement / result of Theorem 4.1 be simplified?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CALDERA, a new post-training compression algorithm for large language models (LLMs). CALDERA uses the inherent low-rank structure of LLM weight matrices by approximating them via a low-rank, low-precision decomposition $W \\approx Q + LR$, where $L$ and $R$ are low-rank factors with quantized entries. CALDERA provides an effective way to compress large language models by exploiting their low-rank structure, enabling efficient distribution and deployment of LLMs on resource-constrained hardware while maintaining strong performance.\n\nSTRENGTHS:\n1. CALDERA introduces a new post-training compression technique for LLMs that exploits their inherent low-rank structure, setting it apart from existing methods. Also, the paper provides theoretical upper bounds on the approximation error of CALDERA using a rank-constrained regression framework, lending credibility to the approach.\n\n2. Experimental results demonstrate that CALDERA outperforms existing post-training LLM compression techniques in the low-bit regime (less than 2.5 bits per parameter), highlighting its effectiveness. By enabling efficient distribution and deployment of LLMs on resource-constrained hardware, CALDERA can make LLMs more accessible and promote their broader adoption.\n\nWEAKNESSES:\nThis paper is generally well-written. Here are some minor weaknesses:\n\n1. The performance of CALDERA depends on the choice of target rank and quantization bit budget. The paper does not provide a systematic way to determine the optimal values for these hyperparameters. Is it possible to apply meta-learning here to decide the optimal values?\n\n2. The experiments focus on compressing LLaMa models. It would be valuable to see how CALDERA performs on a wider range of LLMs to assess its generalizability.\n\n3. As mentioned by the authors, the iterative nature of CALDERA's optimization process may require more computational resources compared to simpler compression methods, although this is a one-time cost.\n\nQUESTIONS:\nHow does CALDERA perform compared to other compression approaches? Is it possible to combine CALDERA with these techniques?\n\nLIMITATIONS:\nPlease see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CALDERA, a novel post-training method that combines quantization and low-rank decomposition techniques for compressing large language models. The primary contribution lies in the design of a combined pipeline and the application of low-rank decomposition. Experimental results demonstrate that integrating the designed low-rank decomposition technique enhances compression performance compared to using quantization alone.\n\nSTRENGTHS:\n1. This paper addresses a highly significant and widely studied research topic: compressing large language models.\n\n2. The author not only conducts experiments but also provides numerical evidence to demonstrate the effectiveness of the designed compression method.\n\n3. Experimental results indicate that the combined methods designed in this study effectively enhance the performance of large language models compressed using quantization-only methods.\n\nWEAKNESSES:\n1. **Missing Experimental Results:** The experiments conducted in this paper are inadequate. Many critical experiments conducted in previous works have not been included. For instance:\n\n- Evaluation of throughput for LLaMA models compressed by CALDERA and QuIP# under the same average bits with the same batch size and sequence length. It's crucial to assess whether combining low-rank decomposition with quantization compromises efficiency compared to uncompressed LLMs.\n- Evaluation results on LLMs with different architectures from LLaMA and on generation datasets.\n- Evaluation of time consumption between CALDERA and QuIP# for compressing LLMs.\n- Evaluation of compression performance using different calibration sets and varying numbers of calibration data.\n2. **Lack of Novelty:** The novelty of this paper is limited. The primary contributions are the design of a combination framework and a low-rank decomposition algorithm, both of which appear simplistic and resemble iterative weight compression methods with minor losses. The use of the existing quantization method LDLQ for combination and the effectiveness of the designed low-rank decomposition method in achieving minimal compression loss are questionable. Replacing it with established work like data whitening method in SVD-LLM[1] might yield better performance, given SVD-LLM's theoretical proof of achieving global optimality through data whitening.\n\n*[1] Wang, X., Zheng, Y., Wan, Z., & Zhang, M. (2024). SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. https://arxiv.org/abs/2403.07378*\n\n3. **Intuitive Experimental Results:** While it's intuitive that combining low-rank decomposition with quantization can improve performance over quantization-only methods, existing works such as SVD-LLM already demonstrate this. To add value to the proposed methods, it's crucial to include experimental results where the average bits are equal to or less than 1-bit, demonstrating capabilities that quantization-only methods cannot achieve.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article introduces CALDERA, a post-training compression algorithm for large language models (LLMs) that leverages the low-rank structure of weight matrices to achieve significant compression. CALDERA approximates a weight matrix $W$ using a low-rank, low-precision decomposition $W \\approx Q + LR$, where $Q$, $L$ and $R$ are quantized. The method aims to reduce the memory and computational footprint of LLMs, facilitating their deployment on memory-constrained edge devices.\n\nSTRENGTHS:\nInnovative Compression Technique: CALDERA combines low-rank approximation with low-precision quantization, addressing both the redundancy and precision issues in LLM weight matrices.\n\nTheoretical Foundations: The algorithm is backed by rigorous theoretical guarantees on the approximation error, which enhances its reliability.\n\nPerformance: Empirical results show that CALDERA outperforms existing post-training compression techniques in terms of zero-shot performance, particularly when compressed to less than 2.5 bits per parameter.\n\nAdaptability: The method supports low-rank adaptation, which can further enhance the performance of the compressed models on specific tasks.\n\nWEAKNESSES:\nComplexity: The algorithm involves a nested optimization process, which may introduce significant computational overhead during the compression phase.\n\nDependency on Calibration Data: The performance of CALDERA relies on the availability and quality of calibration data, which may not always be accessible or representative.\n\nLimited Empirical Comparisons: While the article claims superiority over existing methods, the empirical comparisons might be limited in scope, potentially omitting some relevant state-of-the-art techniques.\n\nPotential Issues\n\nQuantization Artifacts: Aggressive quantization, especially at very low bit budgets, might introduce artifacts that could degrade model performance in specific scenarios.\n\nGeneralization: The effectiveness of the method across a diverse range of LLMs and tasks needs thorough validation, as it might not generalize well beyond the tested models and datasets.\n\nOptimization Stability: The iterative nature of the optimization process might lead to stability issues, particularly for large models with highly non-convex loss landscapes.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
This work proposes a new decomposition procedure to achieve low-precision and low-rank compression of large language model (LLM) weight matrices. Such a new method could capture high singular components accurately while compressing less significant ones. An efficient algorithm is proposed to optimize the quantized backbone and low-rank factors, which can be fine-tuned to enhance the model performance. The work also provides a tighter approximation error bound and is validated by compressing the LlaMA LLMs to below 2.5 bits per parameter.

STRENGTHS:
1. The proposed CALDERA method can effectively compress LLMs without degrading the performance too much. And it is demonstrated through a series of experiments that the method is capable of model compression and low-rank adaptive fine-tuning at the same time. 
2. The paper also provides a rigorous approximation error analysis of the proposed CALDERA method.

WEAKNESSES:
I think the presented experiment study is limited. I mainly have the following suggestions. 
1. It would be better to supplement the experiments with LoftQ and LQ-LoRA, and add a set of experiments with full parameter fine-tuning as a baseline. 
2. Please provide an experimental comparison of model size. 
3. It would be better to add some experiments on evaluating the computation cost of the proposed CALDERA.

QUESTIONS:
1. Since the CALDERA method requires many time-consuming operations like SVD and matrix inversion, can this be improved in future work?
2. Are there any efficient ways to improve the selection strategy of the hyperparameter rank?

LIMITATIONS:
In my opinion, how to efficiently tune the  hyperparameter rank is the main limitation of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14