Compressing Large Language Models using Low Rank and Low Precision Decomposition
Authors: Rajarshi Saha
Naomi Sagan
Varun Srivastava
Andrea Goldsmith
Mert Pilanci
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models (LLMs)
Model Compression
Post-training Quantization
Low-Rank Decomposition
Low-Precision Formats
Quantization Error Analysis
Rank-Constrained Regression
Randomized Linear Algebra
Sketching
Primary area: deep_learning_architectures
TL;DR: We propose a post-training compression algorithm for Large Language Models (LLMs), that harnesses the inherent low-rank structure of LLM weight matrices, that effectively combines low-rank and low-precision matrix decompositions,

Abstract: The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces $\rm CALDERA$ -- a new post-training LLM compression algorithm that harnesses the inherent low-rank structure of a weight matrix $\mathbf{W}$ by approximating it via a low-rank, low-precision decomposition as $\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$. Here, $\mathbf{L}$ and $\mathbf{R}$ are low rank factors, and the entries of $\mathbf{Q}$, $\mathbf{L}$ and $\mathbf{R}$ are quantized. The model is compressed by substituting each layer with its $\mathbf{Q} + \mathbf{L}\mathbf{R}$ decomposition, and the zero-shot performance of the compressed model is evaluated. Additionally, $\mathbf{L}$ and $\mathbf{R}$ are readily amenable to low-rank adaptation, consequently enhancing the zero-shot performance. $\rm CALDERA$ obtains this decomposition by formulating it as an optimization problem $\min_{\mathbf{Q},\mathbf{L},\mathbf{R}}\lVert(\mathbf{Q} + \mathbf{L}\mathbf{R} - \mathbf{W})\mathbf{X}^\top\rVert_{\rm F}^2$, where $\mathbf{X}$ is the calibration data, and $\mathbf{Q}, \mathbf{L}, \mathbf{R}$ are constrained to be representable using low-precision formats. Theoretical upper bounds on the approximation error of $\rm CALDERA$ are established using a rank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-$2$ $7$B/$13$B/$70$B and LlaMa-$3$ $8$B models obtained using $\rm CALDERA$ outperforms existing post-training LLM compression techniques in the regime of less than $2.5$ bits per parameter.

Compressing Large Language Models using Low Rank and Low Precision Decomposition
Rajarshi Saha Stanford University
Naomi Sagan Stanford University
Varun Srivastava Stanford University
Andrea J. Goldsmith Princeton University
Mert Pilanci
Stanford University
Abstract
The prohibitive sizes of Large Language Models (LLMs) today make it diffi- cult to deploy them on memory-constrained edge devices. This work introduces CALDERA new post-training LLM compression algorithm that harnesses the inherent low-rank structure of weight matrix W by approximating it via a low- rank, low-precision decomposition as W = Q + LR. Here, and R are low rank factors, and the entries of Q, L and R are quantized. The model is compressed by LR decomposition, and the zero-shot perfor- mance of the compressed model is evaluated. Additionally, L and R are readily amenable to low-rank adaptation, consequently enhancing the zero-shot perfor- mance. CALDERA obtains this decomposition by formulating it as optimization problem minq.I,R IQ + LR - W) 1|2 where is the calibration data, and Q, L. R are constrained to be representable using low-precision formats. Theoreti- cal upper bounds on the approximation error of CALDERA are established using cank-constrained regression framework, and the tradeoff between compression ratio and model performance is studied by analyzing the impact of target rank and quantization bit budget. Results illustrate that compressing LlaMa-2 7B/13B/70B and LlaMa-3 8B models using CALDERA outperforms existing post-training LLM compression techniques in the regime of less than 2.5 bits per parameter. The implementation is available at:
substituting each layer with its Q +
Introduction
Large Language Models (LLMs) stand out due to their remarkable ability to generate human-like text, thereby supporting diverse range of applications ranging from writing assistance to code generation. These models leverage vast datasets and significant computational resources to achieve their impressive functionality. The architecture of LLMs typically includes multiple layers, each with weight matrices essential for encoding various aspects the training data from simple syntactic patterns to complex semantic relationships. However, the substantial size of these trained models leads to high computational costs and considerable energy consumption during inference, which can be challenging for deployment in resource-constrained environments. As LLMSs continue expand in scale, compression techniques to reduce the memory and computational requirements of the models are becoming crucial ensure their broad accessibility.
Due to the correlated nature of language syntax and semantics learned during training, often, the weight matrices of LLMs exhibit redundancy, which manifests low-rank structure. This re- dundancy suggests the potential for compression without substantial loss in performance. This work introduces CALDERA: Calibration Aware Low-Preciss DEcomposition with Low-Rank Adaptation, which compresses LLMs by leveraging the approximate low rank structure inherent in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

these weight matrices. Given a matrix W E Raxd, CALDERA approximates it as W ~ Q - LR. where Q E Innd L E Fank and R Rexd Here, the left and right low rank factors, respectively L and R, are tall and wide matrices, and & is the target rank. Furthermore, the entries of Q, L and R are represented using low-precision formats with Bq, Br, and BR bits per entry, respectively.
Singular LiaMA-2 Query mabrices
191
R
W
R
Q
+
L
d
d
Figure 2: CALDERA decomposes full pree-pini weight
trix into low-rank component (LR), which captures the con- tribution of the top singular values using Br, Bn bits, and Q for the trailing singular values with Bg bits, enabling flexible pre- cision settings for each component. Typically, Bq < BI., BR.
Index
Figure 1: Decaying spectrum of weight ma-
trices (aka, "approximate low-rank")
Since the singular value profile (aka spectrum) of the weight matrices of LLM follow decaying profile as shown in Fig. 1, the low-rank factors L and R capture the effect of the large singular components of W with high fidelity. Morcover, the backbone Q which is quantized aggressively - -orr instance, using BQ = 2 bits, coarsely captures the essence of the moderately decaying and low singular components of W. CALDERA substitutes each weight matrix W in an LLM,, with its approximate low-pree pwwepiiiiiinn and low-rank decomposition Q + LR, resulting in a post-training quantization strategy that delivers state-of-the-arr zero-shot performance. In addition, since usually k < min(n, d). implying that the total number of parameters in LR is much smaller compared to the number of entries W (i.e., k(n + d) < nd), CALDERA can readily fine-tune (or "adapt") the low rank factors L and R in order to boost the zero-shot results.
1.1 Significance and Related Works
Recent efforts have explored various avenues for compression, including but not limited to weight pruning. quantization, and the use of parameter-efficient training methods - cach approach offering distinct advantages and tradeoffs. This section briefly reviews the current methodologies, highlighting the contributions and limitations of some studies closely related to this work.
LLM Compression and Outlier Mitigation: Recent studies like SmoothQuant [42], OPTQ [8]. QulP [3], AQLM [7), and QulP# [36] consider the challenging regime of sub-4 bit post-training LLM quantization. These works collectively emphasize the need to manage the impact of outliers, i.e. weights with unusually high magnitudes. Accommodating outliers necessitates choosing the dynamic range (or scale) of quantizer to be high, consequently increasing the quantization error. QulP equalizes (and reduces) the weight matrices by using a randomized matrix transform, and subsequently, QuIP# employs ES lattice to make the weights more amenable to vector quantization. Both QulP and QulP# use a refined variant of the column- wise quantization method proposed in OPTQ, wherein error feedback from previously quantized columns of a matrix is used compensate for the error incurred while quantizing subsequent columns. CALDERA utilizes this diverse arsenal of strategies and builds on top of QulP#, while capitalizing on the approximate low-rank structure of LLM weight matrices. While it is possible to obtain even more aggressively compressed LLMs [24), this approach requires training from scratch, which is computationally demanding.
Parameter Efficient Fine- Fine-Tuning (PEFT): T): In a related yet distinct vein of work, PEFT methods have gained significant momentum, aiming to adapt LLMs to specific tasks without extensive computational overhead. Recent studies such as QLoRA [5], LoftQ [22], and LQ-LoRA [13] have explored the intersection of PEFT and quantization, demonstrating that fine-tuning through low-rank updates, as originally proposed in LoRA [16], can mitigate the performance losses due to quantization. Given that CALDERA yields a decomposition Q LR, the low-rank components are particularly suitable for fine-tume toninunnng with any existing PEFT methods, thereby enhancing the zero-shot capabilities.
Low Rank Approximation: The rank-k approximation of matrix A E Raxd can be represented by the factorization A LR, with L E Raxk and R. E Rkxdd where k S min d). Known as the Burer-Monteirg factorization, this method substantially decreases the number of parameters,

---

thus reducing computational demands. Recent studies such as LoRD [18], ASVD [44], FWSVD [15], LASER [33]. LQER [45], and ZeroQuant-V2 [43] have explored the efficacy of low-rank structures in LLM weights, treating low-rank factorization and quantization independently. In contrast, LPLR [31] approaches this by uniquely formulating a joint optimization problem for generic matrices, while simultaneously leveraging the equalization property of randomized transforms, as in [3]. CALDERA formally leverages this inherent low-rank structure for LLM compression alongside existing frameworks such QulP# [36] and LoftQ [22], providing additional flexibility for compression. Furthermore, rigorous theoretical guarantees are derived using a rank-constrained regression framework for obtaining low precision and low-rank decomposition, thereby also analytically demonstrating its superiority over rank-agnostic strategies.
2 Problem Formulation
In neural network layer, a weight matrix W transforms an input activation x into an output activation given by Wx. This transformation can be succinetly described using the matrix's singular value decomposition (SVD). For any matrix A E Rxxd the SVD A = Σ, ,,,,V, where o, u, V, are the fth singular value and the corresponding left and right singular vectors, respectively. The impact of each singular component u,v; on the matrix's transformation is determined by the magnitude of og, Given that weight matrices exhibit a decaying singular value profile (Fig. 1), indicating an approximate low-rank structure, lesser contributing singular components can be pruned with minimal impact on the functionality of the matrix, ensuring minimal distortion in the output activations.
CALDERA approximates the weight matrix of neural network, W, as low- www-precision, low-rank decomposition, W x Q + LR, with all components Q, L, R in low-precision format. Unlike previous works such as [13, 22, 44, 45], which represent the low-rank factors L and R in high- precision (16 or 32-bit floating point), this work extends their representation to low-precision. This further reduces the memory footprint while preserving performance. Alternatively, for the same memory footprint, it allows the target rank k to be higher, thereby capturing the low rank structure with higher fidelity by including more of the higher singular value components. The following paragraph formalizes this as a constrained optimization problem.
For given quantizer, let Q denote the set of discrete quantization points in IR For B-bit quantization, the cardinality of Q satisfies log, [ < B. Consider a matrix W E Raxd_ The goal of this work is to obtain an decomposition W
Q + LR by approximately solving the minimization problem
||Q LR W) Q.L.R
min
subject to
Q E Qaxd
L E Q7,*k, and R Qlxdd. (1)
Here, Qq. Q1. and Qn denote the lattice codebooks used to quantize Q L and R. using Bq. BL and BR bits, respectively. Furthermore, X E Rmmx is calibration matrix that aims to preserve the Frobenius norm error of the compressed layer output activations. If W is the first layer's 's weight matrix, X includes input embeddings from calibration dataset, such as a subset of RedPajama [34), with the (t row representing the fth datapoint. For intermediate layers, X contains the input activations, which are the output activations of the preceding layer.
Using the Frobenius norm of the output of layer as a proxy objective for quantizing the weight matrices of an LLM is a popular strategy, and was used in prior work of Nagel et al. [27]. This proxy objective function is particularly useful for post-training quantization of LLMs because their large size makes it difficult to apply sophisticated compression methods.
3 Proposed Algorithm: Calibration-Aware Low-Precision Decomposition with Low Rank Adaptation
This section introduces CALDERA to approximately solve (1) and get a Q + LR decomposition of a weight matrix W using the calibration matrix X. The pseudocode is provided in Alg. 1. It consists of a nested loop for alternately optimizing the variables Q, L and R. Suppose Qq. Qr. and QR. respectively, denote quantizers used for quantizing Q. L and R. For instance, they can refer to uniformly dithered scalar quantizers, as described in App. G.2. Initially, the low-rank factors are set to O, and W is quantized using the LDLQ quantizer proposed in [3, $3.1]. LDLQ is an adaptive
3

---

quantization method that iteratively quantizes [8] each column of W using QQ to get Q as
Q ((k) = (w(() + (W(1==1) Q (1*-)1))a))
(2)
where Q(k,, W denote the ttth column, w(1==11 denotes the first k columns, QQ has a bit-budget
Bq. and ag R is a learnable sequence of vectors. Update Eq. (2) incorporates linear feedback from already quantized columns, it can be seen that Q satisfies Q = Qq (W + (W M), where the feedback matrix M is strictly upper triangular matrix with columns ag- Deñnnng H X X to be the (scaled) Hessian of the least squares objective in (I), [3] show that the optimal feedback matrix is the M obtained from the LDL decomposition of mHH given by mH =
I
Subsequently, Q fixed and the Preeeiiciionn Low-Rank (LPLR) factorization of the residual, (W Q). is computed. This is done by the LPLRFACTORIZE submodule (Alg. 2), which is a refined version of the LPLR algorithm proposed in [31]. For given matrix A, Alg. 2 minimizes
(LR A)X IF subject to L E QE and E Qh Qxx
min L,R.
where QL and QR use B2 and BR bits, respectively. In contrast to [31], the objective in (3) is calibration-date aware. Therefore, the update equations are derived using a rank-constrained regres- sion framework, as described in App. B. Moreover, lines 14 in LPLRFACTORIZE iteratively refine the estimates of L and R. and can only yield smaller Frobenius norm error. The left and right low-rank factor update equations are described as follows.
Initialization: In the absence of quantization constraints, a globally optimal solution to the optimiza- tion problem (3) can be found described later in lemma 4.2. Consequently, the low-rank factors initialized using cank-constrained regression in lines 2 - 4. Since subsequent quantization disrupts optimality of the solution, the factors are iteratively updated minimize this distortion.
Updating L: To update the left factor L, lines 5 and 9 of Alg. solves minz (ZR. - AA IP: For fixed R, this is a least squares minimization, whose solution is available is closed form as L =- -AX )) (RX )²
AHR (RHR as derived in App. C.I.
Updating R: Line 8 of Alg. 2, updates the right factor R by keeping L fixed and solving minz|](La - A)X II.. As this is an under-determined linear system, there exist multiple solutions for Z, all attaining the same objective function value. It is shown in App. C.1 that R = L AHE is a solution. The corresponding error is also obtained, which is used in the derivation of Thm. 4.1.
Computational Complexity: A high-level calculation is provided here, and detailed discussions can be found in App. D. It is worthwhile to note that the closed form expressions of L and R. which are iteratively quantized, are functions of the Hessian H X X. Therefore, H can be computed offline initially, per LLM,, by doing a single forward pass, and subsequently used for all model quantization experiments. For each layer, this pre-pccesssing includes computing H and its LDL decomposition, along with computing HHT, requiring a total of O(md² + 2dd) multiplications. Each outer iteration involves an LDLQ quantization. Quantizing the t column has complexity O(kk), since feedback from & already quantized columns need to be incorporated. Hence, quantizing a matrix in Rn entails Onn² + 3n) complexity. Morcover, LPLRFACTORIZA requires 0(m² (n + d)) to initialize, and subsequently, each inner iteration entails ((ndk). Assuming n, d 2 m > k, and keeping only the dominant terms, the total complexity of CALDERA, not including the complexity of the pre-processing discussed earlier, is O (Tout (n² m2 (n d) ndl: T..)).
Fine tuning via Low-Rank A Adaptation: Once the weight matrices of each layer are replaced by its Q + LR approximation, the zero-shot performance of (post-training) quantized model can be evaluated. $5 shows that CALDERA quantized models outperform existing strategies. Additionally, desired, the low-rank factors L and R can be further fine-tuned using low-rank adaptation [13, 16, 22] on a small task-specific dataset. While the initialization of the fine-tuning step has quantized Q, L and R. the fine-tuned factors represented using 16-bits (BF16 format). Although this leads to slight increase in the memory footprint, the performance gains from fine-tuning are substantial.
Approximation Error Analysis
The approximation error upper bounds are derived via rank-constrained regression framework. Thm. 4.1 below (formally stated and proved in App. C.4) is an informal version of the main theoretical result of this paper, and provides upper bound on the Frobenius norm error when CALDERA

---

approximates a weight matrix W is as W = QR+LRb by solving the optimization problem (1) using Alg. 1. For convenience of analysis, is assumed that the dynamic range of Qq, denoted as R. is chosen to be high enough, ensuring it remains unsaturated. Consequently, for scalar input the quantization error from QQ has zero mean and bounded variance, given by 4 (2 1y2
R²
Theorem 4.1. Approximation error CALDERA (Informal) Given W E Raxd and X E Rmxd with m S d, let D be obtained from the LDI decomposition X = mH (M + I)D IIT, and Amaxo Amin denote the max and min eigenvalues of H. Additionally, let Q - LDLQ(W, Qq). where QQ has dynamic range R and bit-budget Bq. the quantization error be n - Qq + (W Q)M) (Q+ (W- M))),, and 01 ... 2 Ok ..... be the singular values of (( Q) the target rank k satisfies 0.25 2/2
unn ) Amax Eish of S. k S m, and the dynamic ranges of Qt. 20r
and QR are set as RT, eLvmmmmn
and RR 01, then Q, L and R. returned by Alg, I satisfy
4dAmax R²
(2 - 1)2 (1-22) k -
(1
B B LR -
Σ 1>
EAi(nDD +
+ C,
2n
while utilizing an average budget of 2 log2 factors L and when n. d. Here, the expectation over the stochasticity of the quantizers.
ko
Amaz
dif
bits per parameter for the low-rank
meer
Amin
An informal version of the main result is provided here, and the formal version including specific constant values, along with the derivation, can be found in App. C.4. The requirement m d is not restrictive, because when H is positive definite, (1) can be rewritten as [ LR wix || II(Q + LR W |2 ensuring m d. This is detailed further in App. C.5. The approximation error upper bound given by Thm. 4.1 can be directly compared with the result of Chee et al. [3, Thm. 1]. which states that for vanilla LDLQ without LPLRFACTORIZE,
( - (IQ--)x x) I s E [T = Σ EA,(DDD)))
(4)
Evidently, Alg. I yields a smaller error provided, Σi>A (A,(nDD < E Ex
where can be chosen to be arbitrarily small. Furthermore, since the expression Thm. 4.1 consists of two terms, namely, the rank- constraatd regression error, which depends on the target rank k, and the additive quantization error of e, which dictated by the bit-budgets used for L and R. this upper
Algorithm 1: CALDDRA:: Calibration Aware Low-Precision DEcomposition with Low-Rec Rank Adaptation
Input: Matrix: W Raxd Target rank: Calibration matrix: X E Rmxd Outer and inner iterations: Tout. Tin. Quantizers: Qq. Qu. QR. Flag: EnableLoRA, Fine-tune rank: r
Output: LPLR decomposition: Q Q
Q R. Qexd WX xQ+ LR)X
Initialize: + 0, Lo +- 0, Ro + 0, MinlError
2 while < Tout do
Update Q: Q++1 +- LDLQ(W - L,R,, Qq)
Update low-rank factors:
L R4+1 LPLRFACTORIZE W Q441, K X, QL, QR. Tm)
if (Q + L,41 R,+1 W)X < MinError then
Quest <- Qc+1, Libeut <- L4+1, Rbest + R
MinError + [1(Q++1 + L441 R B
end
6- 1+1
end
EnableLoRA is TRUE then
11
Further Fine-tune tune top-r singular components of Lbest and Rbest to 16-bit precision using Low-Rank Adaptation (LoRA) (as [13, 16, 22])
12 end
return Qbett Lbset, Rissst
5

---

Algorithm 2: LPLRFACTORIZE X, QL, Qu. Tia): LPLR factorization submodule
Input: Matrix: A E Rnxxd, Target rank: k, Calibration matrix: X E R Ittrations: Tin.
Quantizers: Qu, QR
Output: Low precision Low Rank factors: E Quikk k R E Qexd s.t. AX LRX
Initialize: Iteration counter: 0
2 Compute SVD of X as UΣVT.
Compute SVD of U XAT ÜΣVT
Get right low-rank factor: Rg + QR(
Get left low-rank factor: Lo & QL))),, where L. arg zeReed (ZR, A)XT|
Lheat <- Lg, Besst f- Ro, MinError - [[ A)XT F
7 while < Tin do
Update right: R,41 +- QR (R(+)))), where R arg minzez
[L, - A) X
Update left: L,+1 < (L where L,41 arg minzzeRnx (I AA) X
if (L A)X || MinError then
10
Dost +- Li+1, Resett + R4,1, MinError + [[ R,41 A)XT|
11
12
end
i i+1
end
15 return Lbest: Risst
bound can be made arbitrarily small by ensuring that the two terms are approximately equal, i.c.,
[ + LR x upper bounded by 2c. This is apparent in the following regimes:
< n: this regime, is treated as a constant as n grows. Then, if the bit-budget Bq satisfies
Bq 2 log2 (2R(Te)-1/2 / nmnd Amax + +1). then E (Q + LR -W) T | S 2e.
iii) h: = (n): For fixed BQ if & is allowed to grow with dimension n, then choosing k to satisfy
k 2 2n (e 1 (md))
(IT))
ensures E [| + LR w)x T 1 2e.
This implies that the upper bound can be made arbitrarily small by either (i) increasing the bit-budger of the backbone, ie., Bq, for a fixed rank k or (ii) increasing the rank for a fixed Bo, for example, BQ = 2. Alternatively stated, this provides tunable knob for controlling the error by trading off the allocated bit-budget between the backbone Q and the low-rank factors L, R.
4.1 Analysis Outline
In this section, a brief proof sketch is presented, highlighting the major challenges in the proof and how they are addressed. For analysis, Q is assumed to updated prior to L, R Alg. However, whichever yields
in practice, the update order is inconsequential, and can be swapped, depending a smaller error. The complete derivation of the approximation error provided in App. C. key ingredient of the proof is the solution of the rank-constrained regression problem, which is defined as, Y
min rank 2) Sk
XZ
(5)
Although this problem is non-convex, it can be solved to global optimality via two SVDs [41]. The following lemma characterizes the solution to the optimization problem (5).
Lemma 4.2, Given Y E mxn, and full rank X E RRmxd, where m S d. Let X UEVT and U = ULVT denote full SVDs of X and U Y. Then, for k S m. the solution of (5) is given by
arg min |XZ- - Y| (VIm E. -Ú (I, (I, (k
Z,

---

where Σ == Em E Rmxx is a diagonal matrix consisting of the non-zero singular values of X. Moreover, denoting the non-zero singular values of Y as (Y)))) the optimal value of (7) is
m
min Fank S
XZ
YII XZ YI = Σ of (Y).
(6)
1=k+1
The complete lemma (with the case > d), and the derivation, are provided in App. B. Using lemma 4.2, the approximation error of
is analyzed in App. C.3. Specifically, lemma C.3 shows that for any imput matrix A, Alg. 2 with suitably chosen B, and BR. ensures that |
p. as in (3), can be upper bounded by twice the sum of squared trailing singular values, (ref. (6)). W hile proving lemma C.3, it is assumed that if Qr. QR gets saturated, a trivial output of L 0, R = 0 returned. Therefore, lemmas C.I and C.2 specify choosing the dynamic ranges Rg and Rg, to be sufficiently high so that saturation happens with very low probability. The proof of Thm. 4.1 is completed by using the LDL decomposition of mH as proposed in [3], along with application of Marchenko- Pastur approximation to bound the expected eigenvalues of the quantization error, i.e., Eλ, (177 yielding the final inequality.
| - A T
5 Numerical Simulations
The efficacy of CALDERA is assessed by using it to compress four popular open source LLMs from Meta AI, namely, LLaMa-2 7B, LLaMa-2 13B, LLaMa-2 70B [35] and LLaMa-3 8B [26]. The framework is built in PyTorch on top of the QulP# [36] and LoftQ [22], and is available at http:///ituub.coo/llaaaacclaaaee
Baselines. The full-rank matrix Q, also referred to as the backbone, is quantized to 2-bits using the LDLQ procedure from QulP [3. 36], employing an E8 lattice quantizer [39]. For CALDERA, which allows even the low-rank factors, L and R. to be represented in low-precision, the quantization is also performed with an E8 lattiee Prior to running Alg. 1, a randomized Hadamard transform (RHT) is applied to the left and the right of the input weight matrix, as the incoherence pre-processing step, to equalize the magniude of the entries making them more robust to quantization. In other words,
CALDERA decomposition is performed on W A H., WHR, where Hr, and HR are Hadamard matrices, right-multiplied by a diagonal matrix with i.i.dd. {+1} entries. In addition, the Hessian matrix obtained from the calibration data is substituted by H A H HHR. As described in [3], this improves the quantization error incurred by LDLQ. Further details are provided in App. E.2.
Metrics. The performance of CALDERA is evaluated using perplexity on the test splits of the Wikitext2 [25] and C4 [6] datasets, as well as task-specific goodness-offfit metrics such as zero- shot accuracy for sequence classification. Specifically, zero-shot accuracy was measured Winogrande [19], RTE [1, 40], PiQA [2], ARC-Easy, and ARC-Challenge [4] tasks. App. E.3 provides more details regarding these benchmarks. Perplexity was measured using sequence length equal to the model's 's maximum context length, i.c., 4096 for LLaMa-2, and 8192 for LLaMa-3, shot experiments were performed using EleutherAl's Language Model Evaluation Harness [9].
the
5.1 Zero-shot Results
Tables 1 and report the perplexities and accuracies for CALDERA with varying target rank (k) of L and R. A smaller value is better for perplexity, which is defined as the exp(-) of the training objective, while zero-shot accuracies are reported as percentages. Per-parameter
budgets range from 2.1 (e.g., rank-64 factors in 4-bit precision) to 2.4 bits (e.g.. rank-64 factors in half precision or rank-256 factors in 4-bit precision). For comparison, the Q + LR decomposition each model. For the sake of direct comparison, fine-tuning
weight matrices found in
the QulP# codebase was performed of the diagonal matrices in RHT was omitted. As QuIP# does not support quantized factors, L and R are rank-64 in order to ensure that the per-parameter bit-budget remains in the 2 - 2.4 range. As another baseline comparison, each model is quantized using QulP# without any low-rank factors. Results for the unquantized models are also provided.
For all models, the rank-256 CALDERA decomposition with 4-bit factors had the lowest perplexity and generally had the highest accuracies. As CALDERA supports quantizing low-rank factors with minimal performance loss, more singular components can be captured compared to using half- half-precision factors while employing the same number of bits. Consequently, the low-rank factors
7

---

Table 1: Zero-shot perplexities (denoted by 4) and accuracies (1) for LLaMa-2. BQ =
throughout.
Method
Rank
B2( t= BR)
Avg Bits
Wiki2
C4
Wino
RTE1 1
PIQA 1
ArcE 1 ArcC
CALDERA (7B)
64
16
2.4
7.36
9.47
64.6
66.4
73.7
60.8
31.7
CALDERA (7B)
64
4
2.1
7.37
9.74
63.7
62.1
72.3
60.9
31.7
CALDERA (7B)
128
2.2
6.76
8.83
63.8
59.9
75.1
65.1
34.6
CALDERA (7B)
256
2.4
8.14
63.6
34.0
QulP# (7B, No FT)
64
16
2.4
7.73
10.0
63.1
66.8
71.7
63.2
31.7
QulP# (7B. No FT)
0
2
8.23
10.8
61.7
57.8
69.6
61.2
29.9
CALDERA (13B)
64
2.08
6.04
7.98
66.9
61.0
76.0
69.5
37.2
CALDERA (13B)
128
4
2.16
5.72
7.66
67.9
58.5
76.0
68.5
38.7
CALDERA (13B)
256
2.32
5.41
7.21
66.9
62.1
76.2
70.3
40.4
QulP# (13B, No FT)
6.06
8.07
63.6
54.5
74.2
68.7
36.2
CALDERA (70B)
128
5.95
75.5
69.3
76.9
47.7
CALDERA (70B)
256
2.2
3,98
5.76
77.6
71.5
79.8
79.5
47 4
QulP= (70B, No FT)
2
4.16
6.01
74.2
70.0
78.8
77.9
48.6
Unquantized (7B)
16
5.12
6.63
67.3
63.2
78.5
69.3
40.0
Unquantized (13B)
16
4.57
6.05
69.5
61.7
78.8
73.2
45.6
Unquantized (70B)
16
3.12
4.97
77.0
67.9
81.1
77.7
51.1
Table 2: Zero-shot perplexities (denoted by 1) and accuracies (f) for LLaMa-3 8B. Bq = 2 bits throughout.
Method
Rank
By (= Bn)
Avg Bits
Wiki2
C4
Wino f
RTE
PIQA
ArcE
AreC 1
CALDERA
64
16
2.4
9.22
10.5
68.9
63.9
72.9
69.9
36.5
CALDERA
64
2.1
10.6
11.8
66.9
58.5
71.8
68.2
34.3
CALDERA
2.2
9.21
67.6
69.7
74.4
71.8
CALDERA
256
4
2.4
8.22
9.56
69.7
65.0
75.1
73.2
40.0
QulP# (No FT)
64
16
2.4
10.9
11.8
66.5
57.0
69.6
63.8
31.0
QulP# (No FT)
0
2
13.8
15.6
63.2
52.7
67.6
57.6
28.2
Unquantized
0
16
5.54
7.01
73.5
68.6
79.7
80.1
50.2
can regain the performance that was compromised when the backbone Q was quantized to 2 bits. Since zero-shot experiments have some inherent randomness and low-rank regularization effects [33], the zero-shot accuracies reported here are not as directly indicative of quantization performance as the perplexity results. addition, $5.3, demonstrates that degradation in zero-shot accuracy can be recovered via LoRA fine-tuning. It is worthwhile to note these results substantiate the claims of [17], which report that low-bit quantization of L.LaM--3 8B, significantly deteriorates model performance across various post-training quantization techniques, more so than with the LLaM--2 series.
5.2 Fine-tuning of Randomized Hadamard Transform (RHT) Parameters
As CALDERA presents general optimization framework for matrix decompositions of the form Q + LR, it can easily be extended with additional heuristics to improve performance. This section serves proof of concept, by examining one such heuristie: Fine-tuning of randomized Hadamard transform parameters. This technique, proposed in QuIP# [36], involves fine-tuning the diagonal Rademacher matrices with +1 entries in the RHT to minimize the cross-entropy loss between the output of the original and quantized models on the calibration dataset. Subsequently. RHT fine-tuning performed on the models quantized using CALDERA in $5.1. Details on specific fine-tuning hyperparameters can be found in App. E.4.
Perplexity and zero-shot results in Tables and 4 match the trends in $5.1, i.e., CALDERA with rank-256 factors typically performs best, with the exception of RTE. In addition, perplexities are substantially lower than without the fine-tuning of randomized Hadamard transform parameters.
Since this is primarily a proof of concept, the fine-tuning not as extensive as in [36] limits. While [36] performs fine-tun the-ttnnng layer-by-layer prior to doing so on the end-to-end objective, experiments in this section only include the end-to-en and-o--end fine-tuning tuning. Furthermore, the fine min-tunnng is performed with sequence length of 512, opposed to 4096 in [36]. As such, the QulPH numbers reported here different from [36].
computational

---

Table 3: Zero-shot perplexities and accuracies LLaMa-2 7B, with end-to-end fine-tuning randomized
Hadamard transform parameters. Bq bits throughout. 'See Footnote 1.
Method
Rank (=(= BR)
Avg Bits
Wiki2
C4 4
Wino
RTE
PIQA f ArcE
AreC
CALDERA
64
16
2.4
6.22
8.23
64,2
63.2
76.1
63.4
34.7
CALDERA
64
2.1
6.30
8.32
64.6
65.7
75.4
63.3
35.4
CALDERA
128
2.2
6.09
8.06
76.5
65.1
CALDERA
256
2.4
5.84
7.75
65.7
60.6
76,5
64.6
35,9
QulP#
64
2.4
6.32
8.31
64,9
66.4
75.0
65.2
34.5
QulP#
0
2
6.58
8.62
64.4
53.4
75.0
64.8
34.0
Table 4: Zero-shot perplexities and accuracies for LLaMa-3 8B, with end-to fine-tuning of randomized Hadamard transform parameters. Bq bits throughout. 'See Footnote 1.
Method
Rank
(=(= BR.)
Avg Bits
Wiki2
C4 4
Wino
RTE
PIQA t
ArcE f
ArcC
CALDERA
64
16
2.4
7.63
8.9
70.3
70.8
75.4
72.4
39.0
CALDERA
64
4
2.1
8,06
9.34
69.5
64.3
76.0
71.5
40.0
CALDERA
128
2.2
7.76
9.02
69.4
63.9
76.0
73.7
41.8
CALDERA
256
2.4
7.34
8.68
73.6
42.3
QulP#
64
2.4
7.92
9.15
68.4
58.1
74.9
72.3
40.4
QulP#
0
2
8.44
9.75
67.5
57.8
72.9
67.6
37.3
5.3 Low Rank Adaptation (LoRA) Fine-tuning Results
In addition to RHT FT described above, the Q + LR. decomposition with target rank is obtained, and & takes values 64, 128 and 256, fine-tuning the top r (S k) singular components specific downstream datasets can recover the performance lost due quantization. We consider three such tasks (i) language modeling on Wikitext (Wiki2), (ii) recognizing textual entailment (RTE), and (iii) commonsense reasoning (WinoGrande). Throughout all experiments in Table 5, r = 64 chosen and those singular components are fine-tuned to 16-bit precision, i.e., BF16 format. The tasks (ii) and (iii) are sequence classification tasks, and the pre-trained LLaMa model is augmented with linear classification head, which is fine-tuned along with the low-rank factors [29]. In other words, the approximation is written as W ~ Q L,R + LyR,, where L1 E Raxr, - Lg E (*x((tr), R. E rx R2 E ((--r)xd L= L L2]. R = [ R The value of is set to 64 and L2, R2 are fined-tuned to Lurie, RyTIE using low-rank adaptation similar to [13, 16, 22]. Doing this significantly on small task-specific dataset like WikiTex12, RTE, or Winogrande, can noticeably boost the zero-shot accuracy, as can be seen from Table 5.2
Experimental details can be found in App. E.4. For each dataset, ten checkpoints are saved during the course of fine-tuning, and the best test performance is reported in Table 5. For datasets where test labels are not available, evaluation performance is reported instead.
For comparison, results from the LoftQ [22] and LQ-LoRA [13] papers are also reported, where available. As these papers were published before the release of LLLMa-3,, only LLaMa-2 results are available.3 In each case, CALDERA achieves better performance at a lower bit budget.
5.4 Autoregressive Generation Throughput
The low-rank (LR) component in CALDERA can recover some of the accuracy lost due to the aggressive 2-bit quantization of Q. However, CALDERA also needs to dequantize and multiply the low-rank factors, which results in slight (albeit, acceptable) throughput degradation compared to QulP# (shown in Table 6). Nevertheless, CALDERA' 's throughput is significantly higher than that of the unquantized model. This is because compressing weight matrices results in smaller volume of data transfer from and to the GPU's SRAM, speeding forward passes. It is worthwhile to note that
There is element of stochasticity in fine-tuning so the final accuracies listed in Table for different CALDERA parameters are not directly indicative the smaller error (i.e.. ( + LR W)X I1:) of the initialization. Rather, they show that low-rank adaptation can recover accuracy degradation from quantization.
"ooe LLaMa-3 results for LoftQ are present its GitHub repository, but none for the tasks evaluated here.

---

Table CALDERA fine-tuning results LLaMa-2 and LLaMa-3 8B. Br., BR are the bit-budgets and R for the low-rank initialization, Rank-64 fine-tuned factors are represented in BF16 precision.
LLaMa-2 2 7B
LLaMa-3 8B
Method
Rank
Bq
(1 Bg)
RHT FT
Avg Bits
Wiki2
RTE
Wino
Wiki2 4
RTE f
Wino
CALDERA
64
2
16
No
2.4
6.06
82.31
84.06
7.91
84.48
85,56
CALDERA
64
16
Yes
2.4
5.89
85.19
85.32
7.88
86.28
88.16
CALDERA
64
4
No
2.4
6.01
81.23
84.06
8.33
85.56
88.40
CALDERA
64
4
Yes
2.4
5.91
85.56
83.42
7.96
87.00
88.40
CALDERA
128
4
No
2.5
5.84
83.75
85.32
7.84
84.84
88.63
CALDERA
128
Yes
2.5
5.77
84.12
85.00
7.68
86.6
88.00
CALDERA
256
4
No
2.7
5.61
83.75
85.4
7.44
86.28
88.08
CALDERA
256
4
2.7
5.55
86.28
84.93
7.44
85.20
89.19
LoftQ
16
2.4
7.85
LoftQ
64
2.5
16
2.9
5.78
LQ-LoRA
64
2.75
8
2.95
5.67
72.4
Llama-2 70B runs into out-of-memory (OOM) an A10G GPU only has 24 GiB of VRAM, which not enough 70B parameters FP16 format (which approximately requires 140 GiB).
Notably, the throughput is higher when the LR fac- tors are in b6- compared when they are 4-bit. This is because CALDERA used QulP#'s s lattice dequan- tizers for the low-rank fac- tors as well, adding to the compute overhead. More- over, QulP# also used fused kernels, and CALDERA's throughput can be improved by leveraging such optimiza- tions. Since this work is primarily motivated with the goal of closing the gap with respect to uncom- pressed models in the 2 to 2.5 bits per parameter regime, throughput improvement using custom kernels, paged attention, etc., is left for future work. We discuss the broader impacts of our work along with limitations App. H.
Table 6: Throughputs for meta-Ilama/LLaan 2- 2-17,70Jb-hf A10G GPU for a batch size and sequence length of (Bq 2 for all rows)
NVIDIA
Method
Rank B, BR)
Throughput (tok/s)
Uncompressed (7B, FP16)
31.75
CALDERA (7B)
64
16
61.68
CALDERA (7B)
64
4
46.29
CALDERA (7B)
128
46. 46.19
CALDERA (7B)
256
45.89
QuiP# (7B)
0
87.74
Uncompressed (70B, FP16)
OOM
CALDERA (70B)
128
5.33
CALDERA (70B)
256
4.66
QuiP# (70B)
18.18
Conclusions
In this work, the problem of obtaining low-precision and low-rank decomposition of an LLM weight matrix was considered. A Q+ LR decomposition efficiently captures the high singular components of the weight matrix with sufficient fidelity, while coarsely compressing the less significant moderate- to-low singular components.
optimization- taaattionnnthorriccllllll motivated algorithm was proposed to obtain this decomposition, which iteratively optimized the quantized backbone Q and the low-rank factors R. Additionally, it shown that and R can be efficiently fine-tuned using low-rank adaptation to boost the zero-shot performance of the quantized model. By utilizing rank cank-contrrained regression framework, an upper bound was established on the approximation error of the algorithm, and it was shown that this upper bound can be significantly smaller than prior bounds in the literature. Finally, the proposed method was empirically evaluated by compressing the LlaMA family of LLMMs in the challenging sub-2.5 bits per parameter regime. The proposed approach can also be used to complement existing compression strategies; thereby making it efficient distribute compressed LLMs and deploy them regular consumer hardware, making them more accessible to researchers.
10

---

Acknowledgements
This work was supported part by the National Science Foundation (NSH) under Grant DMS- 2134248, CCF-1908308; in part by the NSF CAREER Award under Grant CCF-2236829; in part by the U.S. Army Research Office Early Career Award under Grant W91 in part by the Office of Naval Research under Grant N00014-24-1-21644 and in part by the AFOSR award #002484665.
References
[1] L. Bentivogli, I. Dagan, H. T. Dang. D. Giampiccolo, and B. Magnini. The Fifth PASCAL Recognizing Textual Entailment Challenge, 2009.
[2] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. PIQA: Reasoning about Physical Com- monsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence. 2020.
[3] Chee, Cai, V. Kuleshov, and C. Sa. QulP: 2-Bit Quantization of Large Language Models With Guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[4] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge. drXiv:1803.054570 2018.
[5] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In seirth- Conference on Neural Information Processing Systems, 2023. URL https::/// /pwprevveneeeeeeeeeeeeeeee net wttp://www.eeettttiiieeeeee fttp
[6] J. Dodge, M. Sap. A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, 2021.
[7] V. Egiazarian, A. Panferov, D. Kuznedelev, Frantar, A. Babenko, and Alistarh. Extreme Compression of Large Language Models via Additive Quantization, 2024. URL https: //arxiv. pprg/abs/24011 06118.
[8] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. The Eleventh International Conference on Learning Representations, 2023. URL https://openrevier wwtp:orunttdndneeeeeeeee....
7id tcBBPdd..
[9] L. Gao, Tow, B. Abbasi, S. Biderman, Black, DiPofi, C. Foster, L. Golding, Hsu, A. Le Noac'h, h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, Sutawika, E. Tang, A. Thite, B. Wang. Wang, and A. Zou. framework for few-shot language model evaluation, 12 2023. URL https://zenodo org/ records/10256836.
[10] G. H. Golub and C. F. van Loan. Matrix Computations. JHU Press, fourth edition, 2013. ISBN 1421407949 9781421407944, URL http:://www cornell. edu/cv/VVL// golubandvanlow waaanddaanoan....mmmmm htm.
[11] R. Gray and T. Stockham. Dithered quantizers. IEEE Transactions on Information Theory, 39 (3):805-8122 -12, 1993. doi: 10.1109/18 10.1109/18.256480 8.256499.
[12] Gugger, Debut, T. Wolf, P. Schmid, Z. Mueller, Mangrulkar, M. Sun, and B. Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: gratthub.co//uuggingg wttpc////cceeeaatt..mmmm 2022.
[13] H. Guo, P. Greengard, F. P. Xing, and Y. Kim. LQ-LORA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning. arxiv:2311.1200 2023. URL https:/ /atp///axvvvv/////////eee//aaav org/ abs/2311, 12023.

---

[14] N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randommess: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-288, 2011. doi: 10. 10.1137/ 1137/090771800 URL. htwps:tt//s//tt //doii///ooio/ org/10.
10. 1
[15] Y-C. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, and H. Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL httpp:///ppeeeeee http:/:://///eee/eeeeeeee nwt http///rrmnidddtt/aeeee
[16] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li,
S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning
Representations, 2022. URL http:///ppppprrrreeeeeeerrrooraa wwt.:rrrmmm..dn/eeeeeeeee.......
[17] W. Huang, X. Ma, H. Qin, X. Zheng, C. Lv, H. Chen, J. Luo, X. Qi, X. Liu, and M. Magno. How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study, 2024.
[18] A. Kaushal, T. Vaidhya, and I. Rish. LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression, 2023.
[19] S. Keisuke, L. B. Ronan, B. Chandra, and C. Yejin. WinoGrande: An Adversarial Winograd Schema Challenge Scale, 2019.
[20] A. Krishnamoorthy and D. Menon. Matrix inversion using cholesky decomposition. In 2013 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), pages 70-72, 2013.
[21] H. J. Levesque, E. Davis, and Morgenstern. The winograd schema challenge. In Proceedings of the Thirleenth International Conference Principles of Knowledge Representation and Reasoning, KR'12, 12, page 552-561. AAAI Press, 2012. ISBN 978157735560.
[22] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. LoftQ: LoRA- Fine-Tuuing Aware Quantization for Large Language Models. arxiv:2310.086599 2023. URL https: /ttp//aavvvvv.///////////...a..a org. abs /ts/aa10.0800000
[23] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation, 2024.
[24] S. Ma, H. Wang. I.. Ma, I.. Wang, W. Wang. S. Huang, Dong, R. Wang, Xue, and F. Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024.
[25] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer Sentinel Mixture Models, 2016.
[26] Meta AL. Introducing Meta Llama 3: The most capable openly available LLM to date. https:
/ //ai meta. www/blog///ccoocccoccoocc..ooo.o /wt.amttaaaaaa amama 3/, 2024. Accessed: 2024-05-07.
[27] M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort. Up or Down? Adaptive Rounding for Post-Training Quantization. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 7197-7206, 2020.
[28] S. Rajbhandari,, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models, 2020.
[29] S. Raschka. Building gpt-style Ilm classifier from scratch, 2024. URL https: htttp::/magazine: sebastianraschier cww/ httpp//pbulinnggi///ee a --g-gggtttlleeeeee 11m- classifier. Blog post.
[30] Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20. page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 1450379944. doi: 10.1145/3394486.3400 3406700 URL attps://dot http:///ooooo/////// org/10. 1145/
3394486. 3406703.
[31] R. Saha, V. Srivastava, and M. Pilanci. Matrix Compression via Randomized Low Rank and Low Precision Factorization. In seirth- Conference on Neural Information Processing Systems, 2023. URL
12

---

[32] L. Schuchman. Dither Signals and Their Effect on Quantization Noise. IEEE Transactions on Communication Technology, 12(4):162-165, 1964. doi: 10.1109/TCOM.1144.1000
[33] Sharma, J. T. Ash, and D. Misra. The Truth is in There: Improving Reasoning Language Models with Layer-Selective Rank Reduction, 2023.
[34] Together Computer. Redpajama: open dataset for training large language models, October www.:/ogeeeeeecccopteeeeeocooeee 2023. URL https: tig.......... www/ coom/togethae wwwettthrrrrmccuteeee CeaeitteerrRPPaaaaae RedPaj ama Data.
[35] H. Touvron, Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, Saladi, Schelten, Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, Zarov, Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.
I. Molybog, Y. Nie, A. Poulton, Reizenstein, R. Rungta,
[36] A. Tseng, J. Chee, Q. Sun, V. Kuleshov, and C. D. Sa. QulP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks, 2024.
[37] M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-160, 2019. doi: 10.1137/18M1 183480. URL 1137/18M1183480
[38] R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi:
000999881112111111 108231596.
[39] M. Viazovska. The sphere packing problem in dimension 8. Annals of Mathematics, 185(3), May 2017. ISSN 0003-486X. doi: 10.40077ammas URL.htpp://xx.do doi org/ 10.
2017 185.3.7.
[40] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A Multi- Task Benchmark and Analysis Platform for Natural Language Understanding, 2019. In the Proceedings of ICLR.
[41] S. Xiang, Y. Zhu, X. Shen, and J. Ye. Optimal exact least squares rank minimization. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '12, page 480-488, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 97814503 111550314422 14626. //doi orgo.orrg/100 1145/2339530. 2339609.
doi: 10. 10.1145/2339530.233666 URL https:
[42] G. Xiao, Lin, M. Seznec, H. Wu, Demouth, and S. Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[43] Z. Yao, X. WU, C. Li, Youn, and He. ZeroQuant-Vi V2: Exploring bost-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation. 2023.
[44] Z. Yuan, Y. Shang, Song, Q. Wu, Y. Yan, and G. Sun. ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models, 2023,
[45] C. Zhang, J. Cheng, G. A. Constantinides, and Y. Zhao. LQER: Low-Rank Quantization Error Reconstruction for LLMs, 2024.
13

---

Contents
Introduction
1.1 Significance and Related Works
2 Problem Formulation
3 Proposed Algorithm: Calibration-Aware Low-Precision Decomposition with Low
Rank Adaptation
4 Approximation Error Analysis
4.1
Analysis Outline
5
Numerical Simulations
7
5.1
Zero-shot Results
5.2
Fine-tuning of Randomized Hadamard Transform (RHT) Parameters
5.3
Low Rank Adaptation (LoRA) Fine-tuning Results
5.4
Autoregressive Generation Throughput....
6
Conclusions
10
Notations
15
Rank-constrained Regression
15
Derivations for Calibration-Aware Low-Precision and Low-Rank Decomposition:
CALDERA
17
C.1 Update Equations for Low-Rank Factors in I.P.FAATOOIIE submodule
17
C.2 Dynamic Ranges of Quantizers
18
C33 Proof of Lemma C.3: Approximation Error Upper Bound for LPLRFACTORIZE
19
C.4 Proof of Thm. 4.1: Approximation Error Upper Bound for CALDERA
22
C.5 simplification for positive definite Hessians
Computational Complexity of CALDERA
25
E
Additional Experimental Details
25
E.1
System Details
25
E.2
Parameters of CALDERA Quantization
26
E.3
Details about Goodness-of-Fil Metrics
26
E,4
Fine-tuning Parameter Details
26
E.5
Computation of Average Bit Budget Per-Peraetter
27
E.6
Additional Notes on Numerical Simulations
27
Additional Numerical Simulations
27
F.1
Ablation Study of CALDERA Parameters with respect to Frobenius Norm Error Experiments Mistral-7B
27
F22
28
G Auxiliary Results
28
G.1 Useful Results from Linear Algebra and Probability Uniformly dithered scalar quantizer
28
G.2
30
H Limitations and Further Discussions
31
14

---

A
Notations
This section begins by outlining key notations used in both linear algebra and probability theory. Boldface uppercase and lowercase letters, such as A and a, represent matrices and vectors respectively. I denotes the identity matrix, and its dimension is assumed to be imminent from the context. The first columns of the identity matrix I as Ig. and let I& be the submatrix formed by the last columns. The singular values of A are denoted by max
σ1 2 02 2 ... 2 o, min (A), where r = rank(A) Similarly, the eigenvalues are denoted as X1 (A).... ),()),, The max norm A is defined as max maxi,j Aigl the spectral norm of A is defined as |All2 sup |||==1 |Ax|
1/2
1/2
maxx (onax(A), (A), and the Frobenius norm is || (Ei, A2)
Tr (AT(ATA)
(ErerH of
For any matrix X, its Moore-Penrose pseudo-invers inverse is denoted by x The notations R,, X and 2 used to denote approximate equality and inequalities that hold asymptotically in the limit when dimensions grow to infinity. In other words, for any two functions A(n) and Bn)).
A(n) < Bn))
lim A(n) < lim B(n).
n-+00
n-20
Notations and 2 are defined analogously. Wherever relevant, dimension-dependenn terms highlighted blue.
Table 7: Notations used in this paper
Notation
Description
Remarks
LLM weight matrix of any layer Backbone, left and right low rank factors
W Raxd
Q. L. R
Q E Ruxd L ga*k and R 1
represented using Bq. Br. and BR bits, respectively. Approximation:
Q
LR
X
Calibration data matrix
X mx Input activation for each layer. Computed once offline for each LLM.
H
H x
(Scaled) Hessian of least-squares objectives
(1) and (3). Computed offline once.
Target rank low-rank factors
k 64, 128, 256 in our expts.
Qq Bq, R
Quantizer. bit bit-budget and dynamic range for the backbone
Operates on matrices in 1e*xa
Qu, BL, Rr.
Quantizer, bit bit-bt and dynamic
Operates on matrices in x
range for the left low-rank factor Quantizer, bit budget and dynamic range for the right low-rank factor Strictly upper triangular matrix from the LDL decomposition mH Diagonal matrix obtained from the LDL decomposition of m.H
QR, BR, RR
Operates on matrices Loood
M
mH (M IJD(M + +I)
D
m (M + I)D(M 1)
Quantization error of LDLQ, given
7 E x x is assumed to consist of i.idd.
by & Qq + (W (W-Q)MM Q)M) Q (W- Q)M)
entries for analytical tractability.
E1, Ex
Quantization error matrices from quantizing left and right factors
E, m and ER E Rxd consists of zero-mean random variables with bounded variance.
B Rank-coostrained Regression
Recall that the submatrix formed by the first m columns of the identity matrix I denoted as I and let I, be the submatrix formed by the last m columns. The dimension of I inferred depending on context. Consider the following:
min
XZ Y|
(7)
rank Z
15

---

Although this problem is non-convex, it can be solved to global optimality via two SVDs. The following lemma characterizes the solution the rank-constrained regression in (5).
Lemma B.1. (Global optimality of rank-constrained regression) Suppose Y E R mm is given, and suppose X E R is full rank, i.e., rank(X) min(m, d}, with SVD, X UEVT. Furthermore, let UÊVT denote the full SVD of Y if m S d, or the full SVD of (UL,) Y if m d. Then, for S m, the solution of (5) is given by
VI Σ UL (VE UI,
(IVV
if dd
Z, arg min |XZ - YIF rank(Z)sk
(8)
I EVV)
otherwise.
Here, is diagonal matrix of the non-zero singular values X,, defined as = ΣIn R Rmxm
when m S d, and Σ I e gaxd when d m. Additionally, the optimal value is
min rank(Z) Zi
XZ - -Y
Th o7 (Y), Em of ((UL)) + (UIm-c) Ty
if m S d, otherwise.
- YI
99)
Proof. Case m S d: Since the full SVD of X E Raxd is X UÊVT, where U E R=xm Σ E mxd and V E Rdddd the last (d m) columns of Σ will be zero, i.e., ΣIm-d 0. Let Z' = V E Rdx= be the transformed optimization variable. Since V is a unitary matrix, (Znk(Z) S if and only if rZnk S k. Splitting Z' Rdxa into Z" = I,T Z' E | and
z" T -I- Z' [g(dmm)xn, it can be seen that ΣZ' ΣI,Z" + EIm-aZ
EZ", where
Σ = ΣI E Rmxm is a diagonal matrix comprised of the non-zero singular values of X. Then, |
min Sk
XZZ-
Y
min SEk
min )Sn)
2 F (10)
F
Note that the objective function value is independent of Z", as Z" lies in the null space of XT, and ra") < k follows from the fact that rank of submatrix cannot exceed the full matrix. Since X is full rank, Σ is invertible, and the minimization in (10) is equivalent to
min
2
= Σ of (Y),
(11)
rank S
i=k+1
where Ž ; EZ" E -gmx The equality in (11) follows consequence of Eckart- Young-
theorem (lemma G.1), which states that 7 U U Y is minimized by taking the best rank-k approximation of U Y, and the fact that the singular values of U Y and Y are the same U is unitary. Moreover, the SVD of UT Y is ÚΣVT.
Z. arg min
Ż U
Y
||= F
(ÜIe) (I.VV)
rank (k)
and, In other words, if Z' denotes the solution of the middle optimization problem in (10), IT = z" Furthermore, note that setting ___ Z. = 0 ensures rank (z.)) = rank (Z!)) < k. while keeping the objective value in (10) unchanged. Hence, an optimal solution given by
Z := İ. - (-----i
(I.
(12)
UT (1 IVEVV)
)}
and, z, = VZ, (VI (I. EVT) (13)
z'=
(dd-m)Xn
Case d: Recalling that the full SVD of X E Rmxd is X UEVT where U Rmxm
E E mm and V E Rdxd, in this case, the last m - d rows of Σ will be zero, i.c., - Denote Z' == V Z. This time, as Σ Iī Σ.
E = 0.
Žz'-UTY 2
min (Z
XZ- YIE-
min rank(Z') Sk
16

---

ΣZ' 1 U Y + |
min mnk(Z')sk
- Y 2 | + l
min rank(Z")SR
m
of ((UL)) + [((mma))) T Yl
(14)
(=++1
where z" = EZ' EV Z. Note that the term | m T Y is the irreducible error, and (i) is,
once again, consequence the fact that || L Y [| is minimized by taking the best rank-k
approximation of U Y. Moreover, since the SVD of I U i UÊVT.
Z" == arg min rank(Z")sk
Z I U Y (ÜIs)
[ (I.V)).
and Z, VE -E'""" (VE UT (I.
(15)
Remark: It is not necessary for X be full rank. An equivalent result can be derived with Σ replaced by D
Computational complexity rank-constrained regression: Arriving at the globally optimal solution in lemma B.1 requires computing two SVDs, namely X E R= which entails complexity of (dmm), and U E Rmx*, with a complexity of O(nm²). Hence, the total computational complexity is O(m² + d)).
C Derivations for Calibration-Aware Low-Precision and Low-Rank Decomposition: CALDERA
C.1 Update Equations for Low-Rank Factors in LPLRFACTORIZE submodule
subbooo
As discussed $3, the left and right low-rank factors the LPLRFACTORIZE sub-module (Alg. 2) are found by solving least squares minimization problem. The closed form expressions can be obtained by solving the normal equations directly. In what follows, the same expressions are also derived explicitly via the singular value decomposition, it gives expression for the error (for example, refer to (17)) - which consists of an additive and irreducible error term.
Updating L with fixed R: For fixed R E R the left low rank factor is computed in lines 5 and of Alg. by solving
XR Z - xIfe where == XA E mMX
min ZEREXa
(16)
Let XR EXR VT be the full SVD of X where UXR Rax ΣXR mg and V XR E RRac Recall from App. B that I denotes the submatrix formed by the first columns of the identity matrix I,, and I the submatrix formed by the last m columns. Denoting ΣXR = I Exn
and Z
V since T EXR= 0, it can be that
UXR XR VT XR Z ||
EXR ZRZ UUR Y|
min ZER
min XEREXR
-UXR Ty
EXR (UxnIa) Tpp (17)
min
Z' zee
The last term of (17) minimized by setting Z' < Z,
Ex (UxnI:) TY. Since Z VT VRR Z.
V VRR L this yields the left low rank factor to be
L T VxREx (UXRL)) TXA = L (AXT
(RX yt
AHR (RHRT)-! (18)
where (i) follows from the explicit expression for the pseudoinverse of the wide matrix RX E g
17

---

Updating R with fixed L: For fixed L E R the right low-rank factor is computed in lines 4 and of Alg. 2 by solving
min ZEROXE
XZL
Yll,, where Y == XA E m
(19)
Let X UxEXVV X be the full SVD of E Rm where Ux E Rmxm, Σx E Rnx and
Vx E Rdxdd Then, denoting Z' == ZL the minimization (19) is equivalent to
Z' min
V Y min Z"ER-X"
|xx"" Y where Z= V Z'
(20)
Z'
Note that (20) is an undetermined linear system with multiple solutions. In particular, since Σx E Rmxdd the last (d m) columns of Σx consist of zeros, i.e., ExIa-m 0. Therefore, solution of this system is given by
Z" Y or, Viz = U or, Z VXETU
or, Z. U
(21)
This implies R L'XX Vv.
The proof is completed by noting that X UX Vx HH! because.
HH = x x Xx
X
V V
=X
=XTU&&&VVV [xxix Vx
($x) T
=XTUXEx Ex
Vx = Ux VX
(22)
C.2 Dynamic Ranges of Quantizers
Lemmas C.I and C.2, stated this section provide sufficient conditions to ensure that the quantizers QL and Qn, used to quantize the left and right low-rank factors in the LPLRFACTORIZE submodule, remain unsaturated.
Lemma C.1. (Dynamic range of right quantizer) Given matrices A E Baxd and X E Rmxxd let mmax denote the maximum singular value of XAA... T. Then, the quantizer Qn remains unsaturated if the dynamic range is chosen to be Rn. 0max:
Proof. Note that the input to the right quantizer QR i.e., EVV satisfies || ΣVT max
[ EVT = I Σ E = | XA
||2 = |XA 112.. This completes the proof.
Since the input QL is dependent on the output of QR, the following lemma C.2 provides an upper bound on the input to Qr., provided that QR was unsaturated.
Lemma C.2. (Dynamic range of left quantizer) Given matrices A Raxd and X E R let = x, and let Fmax and σ% denote the largest and the w singular values of XAA,, respectively. For some small e >> 0 and an absolute constant C, suppose the number of bits for the right quantizer QR satisfies
Amin denote the smallest eigenvalue H = X
4Comaax
8 8|XAT 12
BR 2 loga
Vd++ Vi Vios log
(23)
log
Then, if the dynamic range of QL is chosen to be
20
R
(24)
then QL remains unsaturated with probability exceeding 1 0.25 XXA -1=22
18

---

Proof. In Line 5 of Alg. 2, the input to Qn is Ln (WXT) (RoXT)*. In the rest of the proof, the subscript 0 in Lo and R4 is dropped for brevity. Recall the notation for the full SVD XR T i... XR = UXR ExR VXR. where UXR E Ruxm EXR R and VXR R From Eq. (18), can be expressed as L Vxr D (UxRI&)
TAT Consequently, ||L]|max is upper bounded as
|| < | (
E (UXRIa) XA
(i)
xx
2 [UXRI+))
XA
XR
||XAT|]
(25)
Here, (i) holds because Vxr is unitary matrix, (ii) follows from submuliipicatiiitty of spectral norm, and (iii) follows from the fact that (UxR 4 (+ < I and lemma G.4. Furthermore,
xn = =0 EXR) S (mdmin)) 1/2 min (R)
(mdmin)
I (EVT
|ER|22
(26)
where miin (XXR) is the smallest non non-zero singular value of XR (i) follows from lemma G.4 and (ii) follows from lemma G.5. Here, ER E IR kxd is the quantization error from quantizing right low-rank factor, and consist of unbiased random variables with bounded variance, as described in lemma G.6. Since Σ contains the singular values of U XA
Amin [EVV
min (I, Σ) = o ( = oR (U (UXAA] XA Ok (XAT)
(27)
where (i) and (ii) follow because V and U are unitary matrices. Substituting (27) in (26) and (26) in (25), [L] max
XXA 1/2 S (m-dmaa) (ax (XA
-|EE)]2))
|XAT|] 1/2
(28)
XX
Lemma C.1 suggests choosing the dynamic range of Qn to be unsaturated. As result,
XAT 22 to ensure that Qn remains
2|XATI 1|2
XA 1|2
A
|ERll
(29)
2BR
(2 - 1) log2'
where I1-11a2 denotes the subgaussian norm (refer to lemma G.2). Subsequently, lemma G.2 yields that for some absolute constant C,
2C XA XAA 112
log ERll22 < (2Ba
(Va + vk+
with probability exceeding 2e -
(30)
Setting t
log
8 8|XAA 1i:
yields
2C XAT 1l2
[8|XXAT] T
1
ER l1
Vd+ Vk
log
(31)
(2Bx.
1) log
with probability exceeding
Note that if the bit budget BR chosen satisfy
4Coi
8 [8XATI] T
BR log2
Vd+ vT
1 loog
(32)
0k log2
then |Rnl/2 S /2. Consequently, from (28),
201
max <
(33)
0 m
This completes the proof.
C.3 Proof of Lemma C.3: Approximation Error Upper Bound for LPLRFACTORIZE
Lemma C.3. (Approximation error I.P..AACTOR Given A Rad and X E Rxdd with d, let Amax and Amin denote the maximum and minimun eigenvalues of H 1 X. and let
19

---

01 2 2 denote the singular values of XAA.A.
Suppose the target rank k satisfies
1/2
min
of
S k m
3/2
Amar Aaxx
1>k
the dynamic ranges of quantizers QL and QR are respectively set as
201
RL
and RR 01
ogymmmin
and for some absolute constant C and arbitrarily small that satisfies O < S 4mmk2 max A O1 suppose the bil-budgets of Qr. and Qn satisfy
o
nk Amax
Amin + +1).
Br 2 logz
Ok
kd Amax
Amin +1):
and
BR max B2h.
where B1 == log2
201
40o1 log?
VV+ Vlogs
(8
and B2 loga
Then, the factors L, R returned by Alg. I satisfy E - expectation is over the stochasticity of quantizers Qr. and QR.
Eisa of + where the
Proof. Firstly, note that upper bounding the error EEI((A - LoRo)X 1 suffices, since the lines 7 to 13 in Alg. 1 refine the estimates of L and R and can only yield a smaller Frobenius norm error. Consider the quantized low rank factorization
Qr. (in) QR (IEVV)) Let
Eg = QL (Lo) Lo. and ER = QR [I VVVVT
I EVV
(34)
denote the quantization error matrices. Furthermore, let
E. 2 min (A Z)x rank(Z)sk
(35)
denote the optimal value of the unquantized rank constronst regression problem. Since the dynamic ranges of quantizers Qn and QL chosen according to lemmas C.1 and C.2 respectively, the entries of E, and En are unbiased random variables with bounded variance as lemma G.6. Let An /Rn B2 1), and A, 2R (2 1) denote the quantization resolutions. Conditioned on the event that the left quantizer QL. is unsaturated, the error matrix satisfies EE, 0. This yields,
E Qu (LO) QR [I. (LEVT)
A) X
E
((1o EI.) QR [1 (VVVT)
A
X
F
① (IoQu (I (VVVV) A) x E
2
ELQR
(I.
x
(36)
T2
where (i) follows from the fact that the error matrix is unbiased.
Since Lo = arg minZERexd
[|ZR
A)X
term T1 can be upper bounded as
E
QR
[VVV
1
A x
E
[ T
A
X
= E
V.V + ER
A
X
2
DV A
+E
VΣ
1
ÚT
Eg
(37)
F
20

---

where inequality (i) is obtained by replacing the minimizer Lo with different, but appropriately chosen matrix. Here, is the diagonal matrix containing the non-zero singular values X.
Since x² = Amax Amin. the second term in (37) is
E VΣ UTE
i
E
Σ
UR
E
Tr
UI, En x
XE
mXmax
Tr Σ UU
mXmax II
Tr Σ U.EER
R²a E
Tr ER
E Tr ER x² E ER kd A
x²
(38)
Here, (i) follows since V is unitary matrix, (ii) follows from lemma G.4, (ii) follows from the cyclic property of trace, (iv) follows U is unitary, and (v) follows since I L I as A: m. Moreover, since EER 0, term T2 can be upper bounded as
E Ei Qn
[ ΣV x
E
EL
I
EVI
ER
X
E
E
EV
X
+ E
E
E
(39)
The first term in (39) is
E
ELI x
Tr E XVS I,E,
mdmax Tr E ΣΣ I E
mdmax of Tr EE E S nk Af mXmax of
(40)
where (i) and (ii) follow from lemma G.4 and the fact that V is unitary matrix. Similarly, the second term of (39)
E E
X
1 E [Tr EE
X XE E.,))
< mmmax
E
Tr ER E E, E1)
md E
ER E )
E EL E
i
nd?
&
S mAmax
IE
ER it
n
ndi dd
mmmmx
E E 11 < k
mXmax-
(41)
4
4
Here, (i) follows because E [.T Er.] is diagonal matrix since
n Var (E%) < n1
E E EL))
E E[E
for i = j.
(42)
En E EN E Eby = for + j.
As consequence of (37) to (41),
E | (QL (Lo) Qn
I ΣVV
- A x x
2
R
2 nk A
mdmax of RdZ dA3
S. + kd
mAmax-
(43)
4
4
21

---

Since RR = o1, the second term of (43) does not exceed 0.25c if
kd +1
BR 2 log2 201K
(44)
This needs to be considered in conjunction with the lower bound on BR in lemma C.I, which yields the maximum of two quantities in the theorem statement. Since R₁, mmmmm the third term of (43) does not exceed 0.25e if
2
nk Amax
+1)
B1 2 log2
Amin +
(45)
provided Qr. stays unsaturated. Furthermore, if S 4kmm max X-1 0s, choosing BR and Br. as above ensures that the third term of (43) is also upper bounded by 0.25e.
The approximation error upper bound in (43) holds conditioned on the event that QL was unsaturated.
which according to lemma C.2, is ensured with probability exceeding 0.25c XA T For analysis purposes, it is assumed that when Qr. gets saturated, Alg. I returns L 0 and R O, as a result of which, the approximation error is upper bounded by ||XA
T IP Then, since
Pr (Qu is unsat.) 2 0.25
[A T I|2 using Cauchy-Schwarz inequality for expectations,
E
(LR A)X E
[
A)X
T
QL is unsat.
E [|(0- A)x
A)X
1|
QL is sat.
5.
+
3E 4
+ Pr (QL is sat.)
Eo? S &a t..
(46)
This completes the proof.
C.4 Proof of Thm. 4.1: Approximation Error Upper Bound for CALDERA
Theorem C.4. Approximation error of CALDERA (Formal) Given W Raax and X E Rmxd
with m I d, let be obtained from the LDL decomposition X mH = (M I)D(M+ and Amax. Amin denote the max and min eigenvalues of H. Additionally, les Q - LDLQ(W, Qq). where QQ has dynamic range R and bit-budget Bq, the quantization error be n - Qq(Q +(W
Q) (Q + (W M)),, and targel rank satisfies
2 Jk be the singular values of X(W - -Q) If
1/2
min
Σ
S S m,
4yna Amax i>
3/2
the dynamic ranges of QL and QR are set as
201
RL
and,
RR = 01,
and for some absolute constant C and arbitrarily small that satisfies 0 < e S 4mk A 01 suppose the bit- budgets of QL and QR. are set so that they satisfy
4 of
nk Xmax
)
Br 2 log2
Amin +
0k
kd Amax +1
and
BR 2 max B1. B2}.
where B1 == =log
201 1
Amin
4Co1
8 (8 o?
and Bg =
B2:=10ge lo8g
Vil+ Vk + V ylog
0xk log
then Q L and R returned by CALDERA (Alg. 1) satisfy
4mdAmax R2
E [Q + LR W)X
F < m V EA, + +
(2 - 1)2
1
n
2 + e,
i>
'In practice, it can be easily checked if Q, was saturated, and if so. Alg. 1 is repeated with a fresh realization of the stochastic quantizer QR. Since lemma C.2 guarantees that the saturation probability is low, it implies that "few" realizations suffice.
22

---

where the expectation is over the stochasticity of the quantizers QQ. Qr and QR.
Proof. As $C.3, an upper bound is obtained on E Q0 + LgR. W)X since after the first iteration, the alternating steps and in Alg. simply refine the estimates of Q, and R, and can only yield a smaller error. For convenience of notation, in what follows, the subscript O is omitted.
Since L, R is obtained after applying LPLRFACTORIZE submodule to (W- Q), lemma C.3 suggests that if RL. RR. BL, and BR are chosen appropriately,
(Q + LR W)X f = (LR (W Q))
Σ Eof ((Q - w)X + (47)
1>
where the expectation in the upper bound on the right hand side is over the randomness in Q due to the stochasticity in quantizer Qq- Since Q L.LQ((W, Qq) is the LDLQ quantizer of Chee et al. [3] (or, BLOCKLDLO quantizer of Tseng et al. [36], which is successor of LDLQ proposed
by Chee et al. [3), it is shown that
Q = QQ + (w- - Q)M),
where M is a strictly upper triangular matrix. This is a consequence of the fact that LDLQ quantizes one column at time, while simultaneously incorporating linear feedback from the already quantized columns. Moreover, if 1 Qq + (W- M) - (W + (W- - Q)M) denotes the quantization error, then it can be seen that
Q W 7M I)=
(48)
Moreover, since mH X = (M (M + T)T
Eo, ((Q w)X = EA; ((Q () W)
EA; (n (L++I)D(+MD (T0+++I)++T]
EA,(nDD)) ))
(49)
where (i) follows from (48). From (47),
E + LR W)X
Σ EA,(DDD)) +
(50)
Since nDD1 = ΣS D D, where Djj denotes the jth diagonal entry of D and nj is the jth
column of 17 Raxdd
1 Σ
X; (nDn d
D S d (ma X f=1
n
(51)
j=l
Note that 17 consists of quantization errors, which, for uniformly dithered scalar quantization, are zero mean random variables with bounded variance upper bounded by A 11"2 As the exact error distribution for non-subtractively dithered quantization (which commonly used in practice), is not fully understood, a simplification is made for easier analysis, assuming the quan- tizer subtractively dithered. For subtractively dithered quantizers. if Schuchman's 's conditions are met [11, 32], this results in uniform error distribution in the interval [- : assuming quantization errors are independent and identically distributed, for large dimensions d, the eigenvalues of d E)= n are distributed according to the Marchenko-
R²
A Therefore,
distribution, given
by Smp (r) == F V(+ - xx - )_ where X - (1 / Furthermore, denoting q = F-1 (1 *). where F, is the inverse cumulative distribution function (CDF) of (mp()), can be seen that
2d
V(+ (x - X-) I
Σ EA: 1>
En n
(n k)
dx
2 (n/ (A2/4)
nd
k
1
F-1
- -
π
n
mp
2
(52)
23

---

Here, = denotes that the equaility holds asymptotically, (i) follows from upper bounding the integral using the fact that (A+ x)) (a X-) is maximized at --+x- Furthermore, (ii) follows after substituting the values x and using the following linear upper bound on the inverse CDF of Marchenko- Pastur distribution,
^+ - -~) -)).) 2
( S x+
(53)
It can be verified by inspection that F (I) is lower bounded by (-ーーس= 22 -2
and I max
inequality (53) is a consequence of this fact. Furthermore, from lemma G.3, max, Djs
Substituting (52) in (51), and substituting A (284-17
41t²
completes the proof.
L
Informal version of Thm. C.4: Consider n s d. From Thm. C.4, a (simplified) asymptotic dependence on the bit-budgets Br, and BR can be obtained. In what follows, constant factors inside the logg (.) have been ignored. Comparing the expressions for B₁ and By, can be seen that desired bit-budgets are
o
nk Amax
kd Amax
BL 2 log2
and,
BR loga
2021
Amin For d, this yields an average bit-budget of
Okk
Amin
8h: of Amax Vnd
BB + BR) = 2
loga
bits per parameter.
2
Amin
C,5 simplification for positive definite Hessians
The expressions in (8) are a little involved, but they can be simplified if H is positive definite, i.e., all eigenvalues are strictly greater than Let H UAUT be the eigenvalue decomposition of H, and let H U U be symmetric square root. Furthermore assume that is positive definite, i... all diagonal entries of A are strictly positive. In practice, this can be ensured by regularizing H through the addition of an identity matrix, scaled by a small amount.
Then, consider the following equivalent optimization problems:
min
(A - Z)x 1
min <ank(Z)<k
Tr ((A Z)H(A - T)
<Z
(A Z)H²
min rank(Z)sk
F
min rank(Z) Sk
(A Z)UA
2
min (ank(25sk <k
Y ZUA
(where Y A
min
Y z'll
(54)
rank[2')sk
Here, Z' A Ž and the constraint rank (Z') S 1: follow from the fact that multiplying Z by an invertible matrix, UA keeps its rank unchanged. The final optimization problem can be solved optimally by considering the SVD of as = UEVT, and the optimal solution
Z = (UT+) (I IV
(55)
where I, denotes the first & columns of the identity matrix. So the final solution is given by,
Z, (UIs) (I. ΣV A AUT)
(56)
This yields a closed form expression for the optimal solution the rank-constrained regression problem for the case of positive definite Hessians.

---

D Computational Complexity of CALDERA
Complexity of pre-preprocessing: For a given calibration dataset/activation X E g the Hessian is computed as H _ X, which requires O(md²) compute. Subsequently, the LDL decomposition of is computed, which entails ((d") complexity [20]. Moreover, HH which used in the update equation of R also computed beforehand, with a complexity of O(d). therefore, the total complexity of pre-processing is O + 2d
Complexity of LDLQ: Each iteration of CALDERA invokes a single call to LDLQ. An LDLQ call involves updating each column of n x d iteratively. From (2), updating the ttth column requires multiplying matrix (of already quantized columns) in TR with a vector in lg*11, implying a complexity of On - 1)). Subsequently, if Qq uniform scalar quantization, quantizing the (feedback incorporated) k column entails O(n) compute, implying total of O(nk) compute for the bth column. Hence, quantizing all columns from k 1,...; n2 would require O (n Ek= k)
3m
O compute.
Complexity of LPLRFACTORIZE: Every iteration of CALDERA invokes a call to LPLRFACTORIZE, which itself consists of Tin inner iterations. Each inner iteration of LPLRFACTORIZE, consists of initializing the left and right low rank factors using rank-constrained regression. As seen in $B, it has computational complexity of O (m² (n d)). Subsequently,
for any matrix A, the left low rank factor as i = AHR T (RHR T)-1 can be found using succes- sive matrix multiplications. While gradient descent based algorithm can possibly speed up this computation, in implementation, closed form expressions are computed directly as they are com- putationally affordable for the hidden dimensions of the LLMs considered. Computing A (HR)) requires O (dk(n + d)), computing (RHR entails O (dki(k: + d) + k²)), and multiplying them together requires O (nk²). Keeping the dominant term, the aggregated computational complexity for computing the left low-rank factor in each inner iteration O (ndk).
Additionally, the left low-rank factor can computed as R = L' A (HH'), where computing L A O (ndk). Since HH is already computed beforehand, multiplying L A and HH entails (kd*). Once again, keeping the dominant term, the complexity O (ndk). Hence, the total complexity of each inner iteration of LPLRFACTORIZE is O (ndk).
Remark:
Compared to the two-stage LPLR algorithm proposed in [31], the left and right low-rank factors are iteratively refined Alg. 2 since this additional compute can be afforded for quantizing LLM weight matrices. Moreover, SVD computations can be sped up with randomized SVD [14].
E
Additional Experimental Details
E.1
System Details
The code for this paper is available at https://gtthu..com/ http://hhhhh..omm/iiallallaaaaae The framework is built in PyTorch top of the QulP# [36] and LoftQ [22] repositories, and utilizes HuggingFace implementations of all datasets and LLMS.. Experiments were performed on either NVIDIA RTX A6000, NVIDIA A10G, or NVIDIA H100 GPUs. Hessian computation for all but LLaMa-2 70B was performed on four NVIDIA AIO GPUs (provisioned as Amazon AWS EC2 instances), parallelized by distributing different layers different GPUs. Hessian computation for LLaMa-2 70B was performed on single H100 GPU. Model quantization was performed on four NVIDIA RTX A6000 GPUs, parallelized in the same manner Hessian computation. All zero-shot and fine-tuning experiments were also run on A6000 GPUs, except for LLaMa-2 70B shot experiments, which were run on an H100 GPU. Low-rank adaptation experiments were parallelized across four GPUs via HuggingFace Accelerate [12], with DeepSpeed ZeRO Level 2 [28, 30]. The use of Llama family of LLMSs for research is guided by a community license that can be founded https://llama, meta. com/license/ and https://llama, www../mm.....aaaaammmm wwt..///mmmm/////cssee...//////
25

---

E.2 Parameters of CALDERA Quantization
For all CALDERA decompositions, the number of alternating iterations between updating Q and L, R (i.e., Tour in Alg. 1) is 15. For decompositions with quantized low-rank factors, except 2 7B and LLaMa-3 8B, the number of LPLR iterations (i.e., Tin in Alg. 2) is 10. For LLaMa-2 7B and LLaMa-3 8B, the number of L.LR iterations is 50.
For the LLaMa-2 7B and LLaMa-3 8B models (for both CALDERA and QulP# decompositions), the calibration dataset consists of 256 random samples from the RedPajama dataset. For all other models, the calibration dataset provided by the QuIP# [36] authors was used. This calibration dataset consists of 6144 random samples from RedPajama.
The number of tokens in each calibration data point is equal to the context length of the corresponding model. i..., 4096 for LLaMa-2 and 8192 for LLaMa-3 8B.
An additional heuristic applied during the CALDERA decomposition is update to the Hessian matrix based on the subspace spanned by LR:
H H MVVTM.
(57)
Where H MM is the LDL. decomposition of the Hessian and LRM UEVT is the singular value decomposition of the product LRM. The updated H is used all LPLR updates, and the original Hessian is used for updating the low-rank factors via Alg. 2. This heuristic was found in the QuiP# codebase, and empirically speeds up convergence of CALDERA, as is discussed in App. F.1.
Although there is randomness in the generation of the randomized Hadamard matrices, we found that it has negligible effect on the results, due to the iterative nature of the CALDERA algorithm. As such, the effect of random seeds is minimal.
E.3 Details about Goodness-of-Fit Metrics
In addition to perplexity on the WikiText2 and C4 datasets, CALDERA was evaluated using zero-shot accuracy on the following sequence classification tasks:
1. Winogrande [19]: collection of 44k problems problems inspired by the Winograd Schema Challenge [21], specifically designed be robust to learned biases in model training datasets. Specifically, Winogrande is a two-choice flll-i-thhe-blan task that requires significant commonsense reasoning.
2. RTE [1]: Recognizing Text Entailment is a task in the General Language Understanding Evaluation (GLUE) benchmark [40]. Given two statements, the model must classify whether or not the second follows from the first.
3. PiQA [2]: Physical Interaction: Question Answering
task that requires understanding about physical relationships between objects.
4. ArcE [4]: general, the A12 Reasoning Challenge is a set of multiple-choice questions that require grade-school level of knowledge. ArcE is the subset that is labeled Easy.
5. AreC [4]: The ARC-Challenge problems are the same format as ARC-Easy, but the dataset only includes problems that were answered incorrectly by selected algorithms.
The unquantized (16-bit) numbers mentioned in Tabs. 1 and 2 are either copied from [36] for the tasks available reported therein. For some ones that could not be easily found, inference was performed directly on the 16-bit model downloaded from HuggingFace.
E.4 Fine-tuning Parameter Details
Fine-tuning of the diagonal Rademacher matrices in the RHT was performed over calibration dataset sampled from the training split of RedPajama. Due to computational constraints, each sample contains 512 tokens, rather than the full context length of the model. The calibration dataset is 256 samples in total, with 192 data points in the training split and 64 in the evaluation split. RHT fine-tuning was performed for 5 epochs with learning rate of 10-3 The epoch with the lowest evaluation loss was used for further zero-shot and fine-tuning experiments.
26

---

Parameters for low-rank adaptation experiments can be found in Table 8. Overall, the number of epochs and therefore number of training steps, was determined empirically based on when the evaluation accuracy plateaued.
Table Hyperparameter settings for low-rank adaptation', fine-tuning experiments are parallelized across four GPUs.
Batch size refers to the per-device batch size. All
Dataset
Block Size
Batch Size"
Grad. Ace.
Epochs
I.R
Weight Decay
LR Sched.
Warmup
Steps
Wikitext2
512
2
step
3e-6
0.001
Linear
42 steps
2091
RTE
20
2 steps
3e-5
Linear
100 steps
Winogrande
256
10
step
le-5
0.01
Linear 100 steps
3030
For the comparison of CALDERA to LQ-LoRA in Winogrande accuracy, the LQ-LoRA result reported in [13] did not involve fine-tuning directly the Winogrande dataset, but rather subset of C4 and Wikitex12.
E.5 Computation of Average Bit Budget Per-Parameter
Assume that the dimensions di, where i E .1,...,7 7). Also, for the models we consider, each transformer layer has matrices of the same dimensions. general, if the full-rank matrix Q has a bit budget of Bo, the factors L and R both have bit budget of BLR. and the rank of the factors is k the average number of bits per parameter is
seven weight matrices each transformer layer have dimensions
E (Bon,d, kBIR( + d:))
El= n,d,
For LLaMa-2 7B, all self-attention matrices are 4096 4096, the gate and up projections are 11008 4096, and the down projection is 4096 x 11008. So, (n;, d,) = (4096, 4096) for {1...., 4), (ns, ds) = (n6, ds) = (11008, 4096), and (17, d7) = (4096, 11008).
(4096, 4096), (n2, d2) (ms, ds) = (Ne, do) = (14336, 4096), and (17,, d7) (4096, 14336).
For LLaMa-3 8B, (mi di) (n4,d)
(na, da) (4096, 1024).
In $5.3, 64 of the factors fine-tuned in 16-bit precision. This results in the following number bits per parameter:
Ei (Bon,d, + (k - (rrrBBBRR (n, + d,) + 16r(n; + d,))
Σl n,d,
E.6 Additional Notes on Numerical Simulations
For LLaMa-2 70B, the zero-shot perplexities and accuracies reported in the QulP# paper [36] could be replicated. As such, the QulP# performance reported in this paper are based recreations of those experiments.
F
Additional Numerical Simulations
F.1
Ablation Study of CALDERA Parameters with respect to Frobenius Norm Error
The per-iteration relative Frobenius norm error CALDERA, for few selected weight matrices of I.aMa-2 7B, is plotted in Figure 3. The data-aware relative Frobenius error is defined
[I( LR
W)X
[WTTI 1|r
where W is the unquantized weight matrix, X is the calibration data, and Q LR is the CALDERA decomposition. The following ablations of CALDERA decomposition parameters are considered:
16-Bit Factors: Q quantized to bits via LDLQ. The low-rank factors are kept in half precision, LPLR iterations required after cank-constrained regression is performed.
27

---

The randomized Hadamard transform is performed on the W and H matrices are prior to quantization.
2. 4-Bit Factors:
The low-rank factors are quantized to 4 bits of precision using an E8 lattice quantizer, and 10 iterations of L.LRFACTORIZE (Alg. 2) are run for each update of the factors (i.e., Ti = 10 in Alg. 1). Otherwise, the decomposition parameters are the same as in 16-Bit Factors.
3. 4-Bit Factors (No RHT): The parameters of CALDERA A are the same as in 4-Bit Factors, except the randomized Hadamard transform step omitted.
4. 4-Bit Factors (No LDLQ): The parameters of CALDERA are the same as in 4-Bit Factors, except Q is quantized 2-bit precision using direct E8 lattice quantization instead of the LDLQ algorithm.
5. 16-Bit Factors (Hessian Update): The parameters of CALDERA are the same as in 16-Bit Factors, except the Hessian update step described in App. E.2 is performed prior to LDLQ.
For each ablation, the rank of L and R varies between k E (64, 128, 256}. For comparison, QulP# with the same-size low-rank factors performed, using the same calibration data and the low-rank decomposition from the QulP# codebase.
Overall, CALDERA with 16-bit factors consistently achieves lower error than QulP#. Additionally, the Hessian update heuristic improves convergence and often, but not always, reduces the final error achieved after 50 iterations of Alg. CALDERA with 4-bit factors has higher Frobenius-norm error than with 16-bit factors, but the degradation is minor. On the other hand, replacing LDLQ with lattice quantizer significantly degrades the Frobenius norm error, and omitting the randomized Hadamard transform worsens the eтrоr for some of the weight matrices considered.
To demonstrate the convergence of L.P.RFATOORIZE (Alg. 2), the per-iteration relative data-aware error of PPPRFACTOIZEE for the factorization W ~ LR is plotted in Figure 4. For all curves plotted, the randomized Hadamard transform is performed W and H before the factorization is computed, and both factors are quantized to bits of precision via an ES lattice quantizer.
In both cases, the alternating minimization iterations reduce the error. For the Up projection matrix, this reduction nominal, whereas, for the Key projection matrix, the alternating iterations result in a significant improvement.
F.2 Experiments on Mistral-7B
Perplexity and language modeling benchmark accuracy results for quantizing Mistral 7B via CALDERA and QulP# are in Table 9. Results are consistent with those in Section 5.1.
Table 9: Evaluations of Wikitex12 and C4 perplexities. as well as percent accuracies on some common language modeling benchmarks, CALDERA-compese on Huggingface by the authors of QulP#. Bq = 2 bits throughout, and Bt BR bits where low-rank
Mistral 7B. All quantizations calibration datasets released
factors are present. For faimeess of comparison, QulP# numbers reported not include RHT finetuning.
Model
Method
Rank
Avg Bits
Wiki2
C4 L
Wino f (acc)
PiQA f
ArcE f (acc_norm)
ArcC
(acc_nora)
(acc_norm)
Mistral 7B
CALDERA
128
2.19
6.01
8.92
71.51
78.02
75.21
46.59
Mistral 7B
CALDERA
256
2,38
5.71
8.44
70.24
79.92
74.92
46.93
Mistral 7B
QuIP#
0
2
6.19
9.09
69.30
78.45
72.90
44.80
G Auxiliary Results
G.1
Useful Results from Linear Algebra and Probability
This section states some useful linear algebra results as lemmas. Some them are stated without proof, which can be easily proved or found in linear algebra textbook such [10].
28

---

LLaMa-2 7B Layer 25 Key Proj., Rank-64 Factors
LLaMa-2 7B Layer 25 Up Proj., Rank-64 Factors
6.100
Quie
Factors
48 Factors (No RHF)
4B Pacters (No (DI0)
Reletive Fro. Norm Error
0.27
OuPe
168 Factors
481 Factors lNfe RHT)
Pactors (DLO)
6.135
0.26
6.10
0.125
0.23
0.24
0.110
6.105
0.22
30
20
30
40
50
0
10
20
40
50
Iteration
Iteration
Layer Key Proj., Rank-128 Factors
LLaMa-2 Layer Up Proj.,
Factors
0.27
Relative Error
OuPt
168 Factors
48 Factors (No RHT)
48 Factors INo (DLO)
Relative Fro. Norm Errer
168 Factors
OuFe
48 Pactors lNfe (DLO)
Factors INo RHT))
0.13
40 Fectors
160 Factors Ihfsssian Update)
Fectors
160 Factors (Hfssian Update)
0.25
0.24
0.23
0.22
Iteration
Reeation
LLaMa-2 7B Layer 25 Key Proj.. Rank-256 Factors
LLaMa-2 7B
25 Proj., Rank-256 Factors
0.26
OuPt
(No
OuPr
Factors
168 Factors
48 Factors (No (DLO)
Relative Fre Norm Error
0.25
16B Factors
Factors LDLO)
0.12
4B Factors
16B Factors IHessian Update)
48 Factors
168 Pactors lHesslnn Update)
0.24-
0.13
0.23
0.22
0.10
Relative
0.09
0.08
Iteration
Iteration
Figure 3: Relative data-aware Frobenius norm error per iteration of CALDERA for selected matrices of LLaMa-2 7B layer is the same used in $5. The first iteration of CALDERA with the Hessian update is omitted, it has a large error, inhibiting plot readability.
25. For all experiments, the bit precision of Q is 2, and the calibration dataset
LLaMa-2 78 Layer 25 Key Proj.
LLaMa-2 78 Layer 25 Up Proj.
Bank: 236
Bank 236
Bank 12B
Relative Fro. Norm Error
Bank 120
Park 54
Bank 64
Relative Norm
0.82
0.80
20
30
40
LPLR Iteration
LPLR Iteration
Figure 4: Relative data-aware Frobenius norm error per iteration of L.LRFACTORIZE, for the decomposition W x LR, for two matrices in LLaMa-2 7B layer 25.
29

---

Lemma G.1. (Eckart-Young-Miin theorem) For any matrix A with SVD given by A UEV
the solution of A == arg min rank(()) All is given by A = (UX* [1. (1T and || A& = - (A).
Lemma G.2. (Vershynin [38, Thm 4.4.51) Let be a x m random matrix whose entries Xi independent, zero-mean, subgaussian random variables. Then, for any t O. ||||| S
CK (Vd+ vm +t) with probability exceeding 1 - 2e- where K maxi, ||Assl and I1-1laz denotes the subgaussian norm, and C is absolute constant.
Remark: Lemma G.2 states a high probability upper bound on the spectral norm of random matrix in terms of the subgaussian norm of the entries of the matrix. The subgaussian norm of a subgaussian
random variable X is defined as || جb A inf(t 2 0 Blex"/e < 2}. It can be shown that any
bounded random variable X is subgaussian, and satisfies || I || 1
log2
Lemma G.3. Suppose the LDL decomposition of matrix H E [d is given by H (M I)D(++I)' where M is strictly upper triangular and D is diagonal. Then, max, S Amax where Amax denotes the largest eigenvalue of H.
Proof. Let e, denote the jth canonical basis vector. Then,
e, Hej e, (M I)D(M+I) ej Diy+ M3.
(58)
Moreover, from properties of eigenvalues, e, Hej S Amax- Therefore, for any j. DJs S Djs Eiss M3 S Amax. Since it holds true for all j. this completes the proof.
Lemma G.4. (Loewner ordering for matrix products) For any matrix A and B
oinn (A) B B < B A A AB & o (A) B
B. B.
Lemma G.5. (Minimum singular value) For matrices A and B, m (A+B) 2 Omin (A) |B]12
G.2 Uniformly dithered scalar quantizer
Let us consider quantizing a scalar with |xr] < R. Given B bits, the scalar quantizer with dynamic range R is described by first specifying the = 2 quantization points
Q1 = - R, f2 = - + A, f3 = - R + 2 -ar = - + (M - 1)4,
where A
2R is the resolution. The quantizer operation is defined as: M
[g++1 with probability r,
QR,((x)
(59)
ql with probability 1 1- r; a-th
where h: arg max, S 1 i.e., x E [gk, 9c+1), and r lemma G.6, such a quantizer satisfies
As shown in the following
A
A
R² 1)2
IE [QRB(<))== and E (QR,B(<)) x)²
(60)
4
(2B
ie., it is unbiased and the error variance depends on R. and B. Here, the E(-) is over the randomness from dithering in (59). the input x to the quantizer falls outside this range, i.e., x > R x < -R, the quantizer is said to be saturated. To quantize any matrix X, QR, is obtained by quantizing each entry independently, i.c., [QR,B (X)),; & (X,)...g).
Lemma G.6. For scalar with |x|| < R, denote the quantization error of uniformly dithered B-bit n²2
scalar quantizer as = QR,B(x) v. Then, Bie] and Var (4) S (28-153, where the expectation E. is over the randomness due to dithering in the quantizer operation.
Proof. Suppose & E [gk, (1+1) and gk+1 = f1 + A, where
2R 1-
Then,
EE QR.B(I)= = gu+1
E 9k
+ ql
(1
1 qk
[Tk A) (a qk) + 9k (A gk) T.
A
30

---

Furthermore, the variance can be upper bounded as
Var (QR,B(Z) x²² (9k+1 x)² (x 9h.) A
+ (4k x)² (1
qk
A
- (9k+1 2) (x - 9k)
sup "c (0k. 1
(gk+1 x) (xr 9k))
9k + 9k+1
4l: + 9k-1
A
R. (2B
gk+1
2
4
1)²
When the input x to the quantizer exceeds R, the quantizer is said be saturated, causing the quantization error to deviate from zero mean and bounded variance. Thus, it is crucial to ensure that the quantizer operates within the unsaturated regime with high probability.
H Limitations and Further Discussions
Our proposed algorithm CALDERA exploits low-rank structure in weight matrices of LLMs, and is seen to boost the performance existing methods the literature of LLM compression. It does so by providing additional degree of freedom to control the compression ratio namely, the target rank (k). Theoretical guarantees show improved performance over cxisting benchmarks, when the matrix being compressed is inherently low-rank to begin with something that is observed in LLMs. This also implies that CALDERA can be used for any other application scenarios that require matrix compression, as discussed in [31]. Its effectiveness largely depends the presence of an inherent approximate low-rank structure, which is seen in many real-world matrices [37].
Despite attaining a lower loss for LLM compression, since CALDERA is designed to tackle an optimization problem through an iterative process, requires slightly more computational resources. For instance, compressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours (on NVIDIA A10G GPUs provisioned from an AWS G5 instance), and compressing Llama-2 13B took 59 GPU-hours (on locally hosted NVIDIA A6000 GPU), Morcover, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in approximately 90 GPU hours (on an H100 from Lambda labs), which is par with QulP# (with RHT finetuning), which reports 100 GPU hours (on A100). It should be noted that these wallclock times can be reasonably afforded, and are prohibirively large. Furthermore, since the compression of LLM is one-time computational expense, this cost becomes highly amortized considering the frequent use of LLMs for inference following the deployment of the compressed model.
Although CALDERA often achieves perplexities and accuracies similar to unquantized models, a gap remains, shown in Tables 1 and 2. This indicates there is potential for enhancing quantization strategies. For example, the target rank (k) can be treated as a myper-parameter and adjusted across different layers. Sharma et al. [33] demonstrate that such a layer-specific rank reduction can improve generalization capabilities. Additionally, improved fine-tuning strategies such as those proposed by Liu et al. [23], which reduce the gap between LoRA and full fine-tuning, be incorporated with the low-rank adaptation step of CALDERA. Detailed investigations are left for future work.
Compressing models enables their deployment in resource-constrained settings, facilitating educational and technological advancements with limited infrastructure. Deploying on edge devices for inference also enhances privacy by reducing the need for data to be sent to centralized servers for inference, thereby enhancing user privacy and data security. Additionally, lower computational requirements of inference using compressed models is step towards adoption of environment-frientt AI strategies.
Broader Impacts:
On the other hand, as the compression is often not always lossless, any technique may result in reduced accuracy or loss nuanced understanding, potentially impacting the reliability of the models in critical applications. Conseugently, due diligence should be exercised when deploying these models. From a broader perspective, LLMs are quite powerful, and their easier deployment can possibly lead to misuse, such as the spread of misinformation or automated generation of harmful content. Consequently, robust regulatory frameworks are necessary, as L.LMs continue becoming more accessible to the general audience.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract proposes algorithm for compression large language mod- els (LLMs) using low-precision and low-rank approximation. The algorithm, named CALDERA, presented in this work achieves that goal. The performance of CALDERA is studied theoretically by deriving explicit approximation error guarantees, as well as numerically by compressing Llama and Mistral families of models for variety of tasks, and shows improved performance over existing benchmarks that don't take into account low-rankness of the weight matrices.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed explicitly in Appendix Sec. H. Assumptions made with regard to theory are stated in the theorem statement. These assumptions hold in practice algorithm is seen to perform well in numerical simulations. Extensive experimentation were performed on as many setups as allowed within our computational budget.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
32

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the assumptions are stated in the main Theorem statement of $4. A brief outline of the analysis (i.e., proof sketch) is provided in $4.1. Complete derivations of the proof are provided in App. C. The derivation is obtained by providing lemmas that build up to the proof, and the lemmas are appropriately cross-referenced. Auxiliary lemmas and well-known results that are used the derivation are stated as auxiliary lemmas in App. G.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details provided in $5 and App. F. Open source datasets used, and the codebase associated with the paper is also spurcsoureed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
33

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The codebase developed for this paper is open-sourced at https//gihhbb.com/piaanicile com/pilanciila/ccader...a The README file provides necessary instructions to reproduce experiments from the paper.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
srridee//oodSuumiisee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//wpps.cc///////// https/cccppbc/g/iiieeeeee wwwbbliccguideess.e. /CodeSummiiieiiic onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are provided in Sec. and App. E. Code is open- along with README file that provides
sourced at https://github.com wttptttub..co/iillaannt instructions how reproduce the results from the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
34

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As LLMs are computationally expensive to process and run inference error bars are not reported. However, since code open open-sourcens sourced, results can be repeated with different configurations.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details are provided in Sec. 5 and App. E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
cc. wwtpp//lcc/.ciccceiiiiiieeeeeeie ltw./eeeeeeiieeeeeeeeeeellllllee iines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
35

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader societal impacts are discussed in App. H
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification: No new datasets are used in this work. All datasets are already publicly open-sourced. Models compressed using CALDERA will be open-sourced.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
36

---

All the creators and original owners of assets used in this paper have been properly credited, and the associated license and terms of use mentioned and respected. Details can be found in $5 and App. E.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswith thcode.ho.... cww/ datasateess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Compressed models will be released publicly.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowd-sourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
37

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowd-sourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
38


Reviews:
[{"review_id": "0kJp82zGe1", "reviewer": "Reviewer_ZG6L", "summary": "This work proposes a new decomposition procedure to achieve low-precision and low-rank compression of large language model (LLM) weight matrices. Such a new method could capture high singular components accurately while compressing less significant ones. An efficient algorithm is proposed to optimize the quantized backbone and low-rank factors, which can be fine-tuned to enhance the model performance. The work also provides a tighter approximation error bound and is validated by compressing the LlaMA LLMs to below 2.5 bits per parameter.", "questions": "1. Since the CALDERA method requires many time-consuming operations like SVD and matrix inversion, can this be improved in future work?\n2. Are there any efficient ways to improve the selection strategy of the hyperparameter rank?", "limitations": "In my opinion, how to efficiently tune the  hyperparameter rank is the main limitation of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed CALDERA method can effectively compress LLMs without degrading the performance too much. And it is demonstrated through a series of experiments that the method is capable of model compression and low-rank adaptive fine-tuning at the same time. \n2. The paper also provides a rigorous approximation error analysis of the proposed CALDERA method.", "weaknesses": "I think the presented experiment study is limited. I mainly have the following suggestions. \n1. It would be better to supplement the experiments with LoftQ and LQ-LoRA, and add a set of experiments with full parameter fine-tuning as a baseline. \n2. Please provide an experimental comparison of model size. \n3. It would be better to add some experiments on evaluating the computation cost of the proposed CALDERA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> experimental comparison of model size\n\nWe have performed ablations over multiple sizes of the Llama-2 family. Please refer to Table 2 of the global response. The PPLs obtained using CALDERA are consistently better than QuIP# (without fine-tuning) for comparable average bits.\n\n> evaluating the computation cost \n\nThank you for this suggestion. While a comprehensive evaluations would require more time, we report (and compare) some numbers here. Compressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours, when done on NVIDIA A10G GPUs provisioned from a G5 instance on AWS. Additionally, compressing Llama-2 13B took 59 GPU-hours, when done on NVIDIA A6000 GPUs.\n\nAs noted in the paper, CALDERA is expected to take more time that QuIP# (without RHT finetuning) as it is an iterative algorithm. However, as seen above, the wallclock times are not prohibitively large. Furthermore, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in about 90 GPU hours on an H100 cluster, which is on par with QuIP# (with RHT finetuning), which reports 100 GPU hours. CALDERA is also faster than some other state-of-the-art methods such as AQLM, which reports 192 to 384 GPU hours, and LLM-QAT, which requires 960 GPU hours. Additionally, each layer is quantized independent of other layers -- hence, the wallclock time for CALDERA can be reduced by processing the layers in parallel (i.e., scaling horizontally by using more GPUs). Moreover, it should be kept in mind that the cost of compressing a model is a one-time cost, and it can be reasonably afforded as long as it is not prohibitively huge.\n\n> time-consuming operations like SVD and matrix inversion, can this be improved in future work?\n\nThank you for raising this question. It is indeed possible to reduce the computational complexity of CALDERA. For instance, the SVD computation in the LPLRFactorize submodule can be replaced with randomized LPLR submodule from (ref [30] in the paper), which leverages Gaussian sketching matrices to reduce the complexity from $O(nd^2)$ to $O(ndm)$, where $m \\ll \\mathrm{min}\\\\{n,d\\\\}$ is the sketch size.\n\nFurthermore, we would like to clarify that matrix inversion is not necessary to obtain the left and right low-rank factors. These factors can be obtained by directly solving the corresponding least-squares minimization problem (in lines 8 and 9 of Alg. 2) using a conjugate gradient descent based solver, which will be significantly faster. The closed form expressions in our paper are used to facilitate analysis and derive Thm 4.1.\n\nMore generally, the constrained optimization problem (1) is NP-hard, and designing efficient algorithms to solve it is an interesting research avenue, with applications much broader than LLM compression.\n\n> selection strategy of the hyperparameter rank?\n\nThank you for this very interesting question! At a high level, it is possible to adaptively select the rank of each layer during fine-tuning using a strategy similar to AdaLoRa [1]. The initial rank can be chosen to be high, for instance 256. Subsequently, during fine-tuning for a downstream task, the singular values of the low-rank component can be analyzed such that smaller singular values can be truncated to adaptively reduce the rank of each layer. Finding efficient and optimal ways to do this warrants a deeper investigation.\n\n[1] Zhang et. al., Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR, 2023)\n\n> experiments with LoftQ and LQ-LoRA ... full parameter fine-tuning as a baseline.\n\nThe results of LoftQ and LQ-LoRa are restated in Table 5 (copied from respective papers). Fine-tuning the low-rank factors obtained from CALDERA shows that we get lower PPLs for fewer average number of bits when compared to LoftQ or LQ-LoRA.\n\nLoftQ and LQ-LoRA report PPLs that are at par or better than vanilla LoRA on WikiText2 PPLs. Moreover, it is know that for several tasks, LoRA performs at par with full fine-tuning. Therefore, we have prioritized utilizing our limited computational budget for other ablation experiments on CALDERA (please refer to the global response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dg3Ks5zGBa", "reviewer": "Reviewer_xRtR", "summary": "The article introduces CALDERA, a post-training compression algorithm for large language models (LLMs) that leverages the low-rank structure of weight matrices to achieve significant compression. CALDERA approximates a weight matrix $W$ using a low-rank, low-precision decomposition $W \\approx Q + LR$, where $Q$, $L$ and $R$ are quantized. The method aims to reduce the memory and computational footprint of LLMs, facilitating their deployment on memory-constrained edge devices.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Innovative Compression Technique: CALDERA combines low-rank approximation with low-precision quantization, addressing both the redundancy and precision issues in LLM weight matrices.\n\nTheoretical Foundations: The algorithm is backed by rigorous theoretical guarantees on the approximation error, which enhances its reliability.\n\nPerformance: Empirical results show that CALDERA outperforms existing post-training compression techniques in terms of zero-shot performance, particularly when compressed to less than 2.5 bits per parameter.\n\nAdaptability: The method supports low-rank adaptation, which can further enhance the performance of the compressed models on specific tasks.", "weaknesses": "Complexity: The algorithm involves a nested optimization process, which may introduce significant computational overhead during the compression phase.\n\nDependency on Calibration Data: The performance of CALDERA relies on the availability and quality of calibration data, which may not always be accessible or representative.\n\nLimited Empirical Comparisons: While the article claims superiority over existing methods, the empirical comparisons might be limited in scope, potentially omitting some relevant state-of-the-art techniques.\n\nPotential Issues\n\nQuantization Artifacts: Aggressive quantization, especially at very low bit budgets, might introduce artifacts that could degrade model performance in specific scenarios.\n\nGeneralization: The effectiveness of the method across a diverse range of LLMs and tasks needs thorough validation, as it might not generalize well beyond the tested models and datasets.\n\nOptimization Stability: The iterative nature of the optimization process might lead to stability issues, particularly for large models with highly non-convex loss landscapes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Quantization Artifacts\n\nWe agree that compression introduces artifacts. However, PPLs on language modeling datasets like Wikitext and C4 are generally considered to be reasonably good indicators of the performance of an LLM, as can be seen in the existing literature. A comprehensive evaluation over the complete spectrum of LLM evaluation tasks is beyond the scope of our computational budget. In addition, we have performed additional model quantization experiments on different sizes of LLaMa-2 model, as well as Mistral 7B, for which CALDERA generalizes well. These results can be found in Table 2 in the global response PDF.\n\n> Limited Empirical Comparisons\n\nPlease refer to additional experiments in the global response PDF.\n\n> Optimization Stability\n\nWhile it is possible that the iterative nature of our algorithm may oscillate, it is not a cause of concern as Algs. 1 and 2 keep track of (and return) the best matrices $\\mathbf{Q}$, $\\mathbf{L}$ and $\\mathbf{R}$ seen across all the iterations. Moreover, as can be seen from our Frobenius norm vs. iteration plots in Appendix G, the convergence of our algorithm is pretty well-behaved when decomposing LLM weight matrices. Furthermore, our theoretical guarantee in Thm 4.1 serves as an assurance that stability is not a cause of concern.\n\n> Complexity\n\nWe have discussed the computational complexity of CALDERA in Appendix D, and acknowledge it as a price paid for the improved error guarantees (as stated in Thm 4.1, as well as our numerical evaluations).\n\nAdditionally, we report (and compare) some numbers here. Compressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours, when done on NVIDIA A10G GPUs provisioned from a G5 instance on AWS. Additionally, compressing Llama-2 13B took 59 GPU-hours, when done on NVIDIA A6000 GPUs.\n\nAs noted in the paper, CALDERA is expected to take more time that QuIP\\# (without RHT finetuning) as it is an iterative algorithm. However, as seen above, the wallclock times are not prohibitively large. Furthermore, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in about 90 GPU hours on an H100 cluster, which is on par with QuIP# (with RHT finetuning), which reports 100 GPU hours. CALDERA is also faster than some other state-of-the-art methods such as AQLM, which reports a range between 192 to 384 GPU hours, and LLM-QAT, which requires 960 GPU hours. It should be kept in mind that the cost of compressing a model is a one-time cost, and it can be reasonably afforded as long as it is not prohibitively huge.\n\n> Dependency on Calibration Data\n\nWhile this is true, the availability of good quality data is a central premise for the success of any data-centric algorithm. It is possible to derive an entirely data-agnostic variant of CALDERA, by simply replacing the calibration data matrix $\\mathbf{X}$ by an identity matrix, $\\mathbf{I}$. However, using a data-aware version yields better results as validated by ours, as well as several prior works, including QuIP# and LQ-LORA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HsOMqUGQhL", "reviewer": "Reviewer_32tN", "summary": "This paper introduces CALDERA, a novel post-training method that combines quantization and low-rank decomposition techniques for compressing large language models. The primary contribution lies in the design of a combined pipeline and the application of low-rank decomposition. Experimental results demonstrate that integrating the designed low-rank decomposition technique enhances compression performance compared to using quantization alone.", "questions": "See above.", "limitations": "N.A.", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 1, "strengths": "1. This paper addresses a highly significant and widely studied research topic: compressing large language models.\n\n2. The author not only conducts experiments but also provides numerical evidence to demonstrate the effectiveness of the designed compression method.\n\n3. Experimental results indicate that the combined methods designed in this study effectively enhance the performance of large language models compressed using quantization-only methods.", "weaknesses": "1. **Missing Experimental Results:** The experiments conducted in this paper are inadequate. Many critical experiments conducted in previous works have not been included. For instance:\n\n- Evaluation of throughput for LLaMA models compressed by CALDERA and QuIP# under the same average bits with the same batch size and sequence length. It's crucial to assess whether combining low-rank decomposition with quantization compromises efficiency compared to uncompressed LLMs.\n- Evaluation results on LLMs with different architectures from LLaMA and on generation datasets.\n- Evaluation of time consumption between CALDERA and QuIP# for compressing LLMs.\n- Evaluation of compression performance using different calibration sets and varying numbers of calibration data.\n2. **Lack of Novelty:** The novelty of this paper is limited. The primary contributions are the design of a combination framework and a low-rank decomposition algorithm, both of which appear simplistic and resemble iterative weight compression methods with minor losses. The use of the existing quantization method LDLQ for combination and the effectiveness of the designed low-rank decomposition method in achieving minimal compression loss are questionable. Replacing it with established work like data whitening method in SVD-LLM[1] might yield better performance, given SVD-LLM's theoretical proof of achieving global optimality through data whitening.\n\n*[1] Wang, X., Zheng, Y., Wan, Z., & Zhang, M. (2024). SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression. https://arxiv.org/abs/2403.07378*\n\n3. **Intuitive Experimental Results:** While it's intuitive that combining low-rank decomposition with quantization can improve performance over quantization-only methods, existing works such as SVD-LLM already demonstrate this. To add value to the proposed methods, it's crucial to include experimental results where the average bits are equal to or less than 1-bit, demonstrating capabilities that quantization-only methods cannot achieve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Novelty\n\nWe propose an idea which is simple in its execution; that does not mean it lacks novelty. CALDERA is the first work that combines quantization with a low-rank decomposition -- both of which are usually treated independently in existing works on LLM compression. Additionally, a significant contribution of this work lies in the novel theoretical guarantees bounding the expected approximation error with respect to the bit budget. The quantization of the low-rank factors, as well as the complexity of the LDLQ quantization method, make these bounds challenging to obtain. Thank you for pointing us out to the nice work, SVD-LLM. Specifically, our work is different in the following aspects:\n\n1. For a given weight matrix $\\bf A$, SVD-LLM considers solving the rank-constrained regression problem, ${\\rm min}_{\\bf L,R}|| {\\bf (LR-Z)X} ||_F^2$ to global optimality. This is a known result (ref. [1] below). We use an equivalent whitening process in Lemma 4.2 of our paper. Additionally, our theoretical guarantees upper bounds the perturbation error to this optimal solution due to the **joint quantization constraints** on $\\bf L$ and $\\bf R$ -- something not considered in the SVD-LLM paper. Quantization of low-rank factors in SVD-LLM is done independent to the low-rank decomposition, (i.e., one after the other) in Sec. 4.4, which is suboptimal compared to the joint decomposition of our LPLRFactorize module. This is reflected in the numerical results, where SVD-LLM reports a perplexity of 13.29 (Table 8 for SVD-LLM + GPTQ-4bit), which is significantly higher that our reported PPLs in Table 1 (for example, 6.19 for CALDERA Rank-256, $B_L = B_R = 4$).\n\n2. Moreover, CALDERA considers a $\\bf Q + LR$ decomposition. This additive matrix $\\bf Q$ (not considered in SVD-LLM), coarsely captures the effect of trailing singular values, which are entirely truncated in SVD-LLM. The $\\bf Q$ matrix enables CALDERA to achieve PPLs close to the unquantized model in the extreme compression regime considered.\n\n*[1] Xiang, et. al., Optimal exact least squares rank minimization (ACM SIGKDD, 2012)*\n\n> Intuitive Experimental Results\n\nOne of the core motivations of post-training quantization of LLMs is to ensure minimal loss in PPLs or accuracies compared to the uncompressed model. Current methods that compress an LLM in the regime of 2 to 2.5 bits per parameter, fail to close the gap to uncompressed performance. While contemporary works like BiLLM (Huang et. al, 2024) do consider the lower ~1 bit per parameter regime, the associated PPLs are relatively high compared to the uncompressed model. In contrast, our work is motivated with **closing the gap with respect to uncompressed models** in the 2 to 2.5 bits per parameter regime.\n\nWhile it is indeed intuitive that low-rank + quantization is expected to perform better than quantization only methods, our work rigorously formalizes this intuition with theoretical backing and experimental results. Although both SVD-LLM and our work explore the benefits of combining quantization with low-rank decomposition, SVD-LLM directly applies LLM quantization methods to the low-rank factors in an orthogonal fashion (which is suboptimal), whereas CALDERA additively combines a full-rank, aggressively-quantized matrix with a **quantized low-rank correction** (which is obtained by treating the quantization and low-rank constraints **jointly**).\n\n> Evaluation of throughput\n\nWe agree that it is important to ensure CALDERA does not degrade generation throughput. We have performed some throughput experiments, which can be found in Table 1 of the global response PDF. CALDERA achieves a slightly lower throughput than QuIP#, as it requires dequantization of 3 matrices -- $\\bf Q, L$ and $\\bf R$ (instead of just one in the case of QuIP#), and perform additional matrix multiplications during inference. Despite this, it is worthwhile to note that the throughput is quite higher than the uncompressed model. Furthermore, it should be kept in mind that designing specialized CUDA kernels that are aware of the $\\bf Q + LR$ decomposition can improve the throughput even further -- for example, by computing $\\bf Qx$ and $\\bf LRx$ in parallel. Writing custom kernels is left for future work.\n\n> Evaluation ... different calibration sets \n\nOur choice of sampling from RedPajama to get our calibration dataset is motivated from the fact that OpenLlama uses it to pre-train the model from scratch, and it is important to ensure that the calibration data is from the same distribution as the pre-training data. While ablations over calibration datasets is worthwhile to explore, we have decided to prioritize other experiments (cf. global response PDF) within our current computational budget.\n\n> different architectures\n\nWe have performed additional experiments (specifically, we compressed Llama-2 13B and Mistral-7B), which can be found in Table 2 of the global response PDF.\n\n> Evaluation ... time consumption\n\nWhile a comprehensive evaluations would require more time, we report some numbers here. Compressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours, when done on NVIDIA A10G GPUs provisioned from a G5 instance on AWS. Additionally, compressing Llama-2 13B took 59 GPU-hours, when done on NVIDIA A6000 GPUs.\n\nCALDERA is expected to take more time that QuIP# (no RHT finetuning) as it is an iterative algorithm. However, as seen above, the wallclock times are not prohibitively large. Also, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in ~90 GPU hours on an H100 cluster, which is on par with QuIP# (with RHT finetuning), which reports 100 GPU hours. CALDERA is also faster than some other state-of-the-art methods such as AQLM, which reports a range between 192 to 384 GPU hours, and LLM-QAT, which requires 960 GPU hours. It should be kept in mind that the cost of compressing a model is a one-time cost, and it can be reasonably afforded as long as it is not prohibitively huge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hnsvpQxW0K", "reviewer": "Reviewer_GVVM", "summary": "This paper introduces CALDERA, a new post-training compression algorithm for large language models (LLMs). CALDERA uses the inherent low-rank structure of LLM weight matrices by approximating them via a low-rank, low-precision decomposition $W \\approx Q + LR$, where $L$ and $R$ are low-rank factors with quantized entries. CALDERA provides an effective way to compress large language models by exploiting their low-rank structure, enabling efficient distribution and deployment of LLMs on resource-constrained hardware while maintaining strong performance.", "questions": "How does CALDERA perform compared to other compression approaches? Is it possible to combine CALDERA with these techniques?", "limitations": "Please see weaknesses.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. CALDERA introduces a new post-training compression technique for LLMs that exploits their inherent low-rank structure, setting it apart from existing methods. Also, the paper provides theoretical upper bounds on the approximation error of CALDERA using a rank-constrained regression framework, lending credibility to the approach.\n\n2. Experimental results demonstrate that CALDERA outperforms existing post-training LLM compression techniques in the low-bit regime (less than 2.5 bits per parameter), highlighting its effectiveness. By enabling efficient distribution and deployment of LLMs on resource-constrained hardware, CALDERA can make LLMs more accessible and promote their broader adoption.", "weaknesses": "This paper is generally well-written. Here are some minor weaknesses:\n\n1. The performance of CALDERA depends on the choice of target rank and quantization bit budget. The paper does not provide a systematic way to determine the optimal values for these hyperparameters. Is it possible to apply meta-learning here to decide the optimal values?\n\n2. The experiments focus on compressing LLaMa models. It would be valuable to see how CALDERA performs on a wider range of LLMs to assess its generalizability.\n\n3. As mentioned by the authors, the iterative nature of CALDERA's optimization process may require more computational resources compared to simpler compression methods, although this is a one-time cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> wider range of LLMs \n\nWe have performed quantization experiments on the Mistral 7B model, which can be found in Table 2 of the global response PDF. The PPLs obtained using CALDERA are consistently lower than QuIP# (no RHT fine-tuning) for comparable average bits. For future work, with additional compute, we intend to continue quantizing more such models.\n\n> other compression approaches ... possible to combine CALDERA\n\nThank you for another interesting question. It is indeed possible to combine CALDERA with other existing model compression techniques.  A primary motivation of CALDERA of is to jointly solve two popular approaches in LLM compression, namely, low-rank decomposition and quantization -- both of which are usually treated as independently in existing works. The same motivation can indeed be used if one wants to adapt CALDERA to incorporate other compression approaches, such as sparsity, quantization-aware training (QAT), knowledge distillation, etc. For instance, the quantized $\\mathbf{Q}$ matrix can be additionally pruned, making it sparse in each CALDERA iteration, further compressing the model. Moreover, provided additional compute, the quantized entries of our CALDERA decomposition can also be subjected to QAT using a straight-through estimator. QAT can either be done for fine-tuning on downstream tasks. Alternatively, QAT can also be done by performing knowledge distillation with an uncompressed teacher model, similar to what is done in LLM-QAT (ref. [1] below).\n\n*[1] Liu et. al., LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (arXiV 2023).*\n\n> systematic way to determine the optimal values for these hyperparameters ... apply meta-learning here\n\nThank you for this very interesting question. Meta-learning can indeed be useful. For instance, each matrix decomposition problem along with a given calibration set, can be treated as a separate task. The objective would be to learn how to choose the target rank and bit budget for the decomposition. The input to the meta-learning algorithm could be input characteristics of the matrix, such as dimensions, dynamic range of the entries, spectrum of the matrix, etc. Furthermore, the measure of performance could be the Frobenius norm error, possibly regularized with some metrics for the computation time required for obtaining the decomposition, or inference time with the decomposed representation. This is a direction worth exploring!\n\n> iterative nature of CALDERA's optimization process may require more computational resources\n\nYes, we acknowledge this additional computational cost of compressing models iteratively. We will release open-source our compressed models, so that this one-time compression cost need not be incurred for someone who wishes to align or fine-tune our compressed models for downstream evaluations.\n\nCompressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours, when done on NVIDIA A10G GPUs provisioned from a G5 instance on AWS. Additionally, compressing Llama-2 13B took 59 GPU-hours, when done on NVIDIA A6000 GPUs. As noted in the paper, CALDERA is expected to take more time that QuIP# (without RHT finetuning) as it is an iterative algorithm. However, as seen above, the wallclock times are not prohibitively large. Furthermore, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in about 90 GPU hours on an H100 cluster, which is on par with QuIP# (with RHT finetuning), which reports 100 GPU hours. CALDERA is also faster than some other state-of-the-art methods such as AQLM, which reports a range between 192 to 384 GPU hours, and LLM-QAT, which requires 960 GPU hours. It should be kept in mind that the cost of compressing a model is a one-time cost, and it can be reasonably afforded as long as it is not prohibitively huge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CgRqKtWfJz", "reviewer": "Reviewer_23tr", "summary": "This paper proposes a framework that combines quantization and low-rank approximation. Given a neural net weight matrix $W\\in \\mathbb{R}^{n\\times d}$, it considers the following decomposition: $W=Q+LR$, where $Q$ is a quantized sketch of $W$ with very few bits (2 or 4 in their experiments), and $L\\in \\mathbb{R}^{n\\times k}, R\\in \\mathbb{R}^{k\\times d}$ are low-rank factors that serve as a \"correction\" to the quantization sketch $Q$. They are usually quantized with more bits (4 or 16). To compute respective $Q, L, R$, the authors propose an alternating minimization framework: starts with $L, R=0$, update $Q$ as $Q_t=Q_Q(W-L_{t-1}R_{t-1})$, then solve for $L_t, R_t$ by solving the rank-constrained regression against $W-Q_t$ and $X\\in \\mathbb{R}^{m\\times d}$, where $X$ is the calibration dataset. They further show their algorithm could be used in conjunction with LoRA and fine-tuning the randomized Hadamard transform. Extensive experiments show that their proposed algorithm gives low perplexities and high accuracies when compared against earlier algorithms.", "questions": "If we allow $L$ and $R$ to be computed to full bit-precision, could the statement / result of Theorem 4.1 be simplified?", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a method that is a good mixture of quantization and low-rank approximation, as simply using only one of quantization sketch or low-rank approximation intuitively \"misses\" an important part. The algorithm proposed in this paper is a natural and simple alternating minimization framework. This paper also contains a theoretical analysis for approximation error under assumptions on the target rank. Empirically, it is also shown the proposed algorithm has good performance under small rank and bit precision.\n\nThe paper is also well-written.", "weaknesses": "Overall, I don't see big weaknesses of this paper. One potential direction for improvement is theoretically, the alternating minimization framework could possibly be sped up given $k\\ll n, d$, especially the rank-constrained regression, see [1]. On the other hand, it is not clear how to integrate these possible algorithmic pieces into post-processing a model weight, so I think the approach and runtime analysis of this paper are fair.\n\n[1] Gu, Song, Yin and Zhang. Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time. ICLR'24.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> If we allow $\\mathbf{L}$ and $\\mathbf{R}$ to be computed to full bit-precision, could the statement/result of Theorem 4.1 be simplified? \n\nYes, in Thm. $4.1$, the error from quantizing the low-rank factors is captured in the additive $\\epsilon$ term. When $\\mathrm{B_L} \\to \\infty, \\mathrm{B_R} \\to \\infty$, this error $\\epsilon \\to 0$, and we get $\\frac{1}{nm}\\mathbb{E}\\left\\lVert\\mathbf{(Q + LR - W)X}\\right\\rVert_F^2 \\lesssim \\frac{4d\\lambda_{max}\\mathrm{R}^2}{\\pi(2^{\\mathrm{B_Q}}-1)^2}\\left(1 - \\frac{k}{2n}\\right)^2$. Here, the only source of error is the finite precision of the $\\mathbf{Q}$ matrix, determined by $\\mathrm{B_Q}$. Furthermore, if the low-rank factors $\\mathbf{L, R}$ are not required to be quantized, CALDERA simplifies to LoftQ -- the only difference being CALDERA uses LDLQ quantizer, whereas LoftQ uses NF4 quantizer.\n\n> One potential direction for improvement is theoretically, the alternating minimization framework could possibly be sped up given, especially the rank-constrained regression\n\nThank you for pointing out this interesting work. We agree that solving rank-constrained regression can be sped up using sketching techniques from Gu et. al. [1]. A careful analysis is required to ensure that the error from sketching does not accumulate across the alternating iterations. A study of the efficacy of sketching in solving the non-convex optimization problem (1) is worth investigating.\n\n*[1] Gu et. al., Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time (ICLR, 2024).*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We are very grateful for the valuable time you spent in reading our paper and sharing your concerns, and greatly appreciate the voluntary nature of the review process. In this global response, we have have summarized the major points from our individual responses.

**Additional experiments compressing other models**: We used CALDERA to compress some additional popular LLMs -- namely, **Llama-2 13B, 70B**, and **Mistral 7B**, which can be found in Table 2 of the global response PDF. We also recompressed Llama-2 70B using the Hessians provided by QuIP# for a fairer comparison. The perplexities obtained using CALDERA are consistently lower than QuIP# (without RHT finetuning). Similar trend can be seen for the zero-shot task accuracies as well. 

**Additional experiments to evaluate throughput**: We also did additional evaluations on the generation throughput of our compressed models, which can be found in Table 1 of the global response PDF. It is noteworthy that the throughput of our CALDERA-compressed model is quite higher than the uncompressed model. CALDERA does achieve a slightly lower throughput than QuIP# -- however, this is expected, as it requires dequantization of three matrices -- $\bf Q, L$ and $\bf R$.  It should be kept in mind that designing specialized CUDA kernels that are aware of the $\bf Q + LR$ decomposition can improve the throughput even further.

**Computation cost of compressing using CALDERA**: We conducted some more experiments to compare the wall-clock times of using CALDERA. Compressing Llama-2 7B or Mistral-7B models (with rank-256) took approximately 34 GPU-hours, when done on NVIDIA A10G GPUs provisioned from a G5 instance on AWS. Additionally, compressing Llama-2 13B took 59 GPU-hours, when done on NVIDIA A6000 GPUs.

CALDERA is expected to take more time that QuIP\# (no RHT finetuning) as it is an iterative algorithm. However, as seen above, the wallclock times are not prohibitively large. Also, LLaMa-2 70B can be quantized via CALDERA with rank-256 factors in ~90 GPU hours on an H100 cluster, which is on par with QuIP\# (with RHT finetuning), which reports 100 GPU hours. CALDERA is also faster than some other state-of-the-art methods such as AQLM, which reports a range between 192 to 384 GPU hours, and LLM-QAT, which requires 960 GPU hours. It should be kept in mind that the cost of compressing a model is a one-time cost, and it can be reasonably afforded as long as it is not prohibitively huge.

In summary, we reiterate that our work proposes and rigorously analyzes a simple optimization-centric point of view to the problem of obtaining quantized and low-rank decompositions of the weight matrices of an LLM. We evaluate the success of our $\bf Q + LR$ style of matrix decomposition algorithm for post-training LLM quantization in the challenging regime of 2 to 2.5 bits per parameter. Our LLM compression scheme reduces the model distribution and deployment costs, and the low-rank component $\bf LR$ provides good initializations for further fine-tuning using popular low-rank adaptation methods. We hope our responses clarify the reviewers' concerns.

Meta-review: This paper introduces a new method for compressing LLM's by taking advantage of both quantization and low-rank approximation of weight matrices. In contrast to prior work, they take advantage of recent approaches for *jointly* optimizing a quantized, low-rank approximation, which surprisingly performs quite a bit better than treating the problems separately from each other (e.g., finding a low-rank approximation first, then quantizing the factors). There were some concerns with the paper's experimental setup, but my feeling is that the concerns were sufficiently addressed during the response period. Without a clear benchmark for LLM compression, decisions have to be made in terms of exactly how to conduct the experiments and which ones to conduct, and the paper's choices seem reasonable enough. For the experiments they do run, the results show a clear improvement over prior approaches that treat low-rank approximation and quantization as separate steps in the compression process. For this reason, we recommend accepting the paper.
Decision: Accept (poster)