Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Authors: Yoonjun Cho, Soeun Kim, Dongjae Jeon, Kyelim Lee, Beomsoo Lee, Albert No
Accepted at: acl25

Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition

Yoonjun Cho1  Soeun Kim2  Dongjae Jeon1  Kyelim Lee2  Beomsoo Lee3  Albert No2†

1Department of Computer Science, Yonsei University
2Department of Artificial Intelligence, Yonsei University
3Department of Electronic & Electrical Convergence Engineering, Hongik University

Abstract
Decomposing weight matrices into quantization and low-rank components (\( \mathbf{W} \approx \mathbf{Q} + \mathbf{LR} \)) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component’s unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers’ negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.

1 Introduction
Quantization (Polino et al., 2018; Jacob et al., 2018; Nagel et al., 2021) and weight matrix factorization (Golub et al., 1987; Saha et al., 2023) are two widely used techniques for compressing large language models (LLMs) to enable efficient inference on resource-constrained hardware. Post-training quantization (PTQ) reduces model size and computational cost by mapping high-precision weights to lower-bit representations (Tao et al., 2022; Bai et al., 2022; Dettmers et al., 2022; Shao et al., 2024), while matrix factorization approximates weight matrices with compact factored representations (Li et al., 2023; Gao et al., 2024b). Recent joint optimization approaches combine these methods to achieve extreme compression, representing weights as the sum of a quantization matrix and a low-rank component: \( \mathbf{W} \approx \mathbf{Q} + \mathbf{LR} \) (Saha et al., 2024; Li et al., 2024b; Guo et al., 2024).

Joint optimization approaches minimize representation error by iteratively alternating between quantization and low-rank approximation. These methods typically adopt either a quantize-first strategy (Saha et al., 2024; Li et al., 2024b) or a low-rank-first strategy (Guo et al., 2024). While these differ in iteration ordering, they can be equivalently understood as distinct initialization choices for the low-rank components: the quantize-first approach initializes \( \mathbf{LR} \) to zero, while the low-rank-first approach initializes them using factorized weights.

Critically, existing methods treat these different orderings merely as preparatory steps, assuming iterative updates will naturally converge to optimal solutions. However, viewing this process through the lens of initialization reveals an underexplored aspect of how initialization strategies might fundamentally affect weight decomposition quality.

In this work, we investigate the role of initialization in joint optimization. Our analysis shows that different initializations lead to distinct solution spaces, with components maintaining persistent roles throughout optimization. Quantize-first methods treat low-rank components as error correction terms, while low-rank-first methods preserve them as the primary weight representation. This finding highlights that initialization fundamentally determines the role assignment between quantization and low-rank components, raising a key question:

What is the optimal initialization strategy for decomposing weights into quantized and low-rank matrices?

Recent works have shown that quantization errors are pronounced for weights associated with activation outliers, as extreme activations amplify weight sensitivity (Dettmers et al., 2023b; Lin et al.,
2024; Lee et al., 2024; Huang et al., 2024; Kim et al., 2024). Building on this insight, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns a specific role to the low-rank component to capture these salient weights while using the quantized matrix to express the residuals. By handling outlier-sensitive weights through the low-rank component, our approach stabilizes quantization and enables more precise weight decomposition.

Through extensive experiments on Llama2 (7B, 13B, 70B) (Touvron et al., 2023), Llama3-8B (Dubey et al., 2024), and Mistral-7B (Jiang et al., 2023), we demonstrate that incorporating ODLRI into the joint optimization framework consistently improves perplexity and zero-shot accuracy across extreme low-bit settings. Our analysis shows that ODLRI reduces activation-aware error, minimizes quantization scale, and enhances model performance, highlighting the importance of structured initialization in low-bit quantization. These results present a principled approach for stable and efficient LLM compression, and provide new insights into the effective decomposition of quantization and low-rank matrices.

We summarize our contributions as follows:

• We propose a unified framework for expressing iterative joint optimization algorithms by introducing the concept of initialization of low-rank components.

• We analyze the impact of initialization on iterative joint optimization algorithms, revealing the suboptimality of conventional approaches.

• We propose Outlier-Driven Low-Rank Initialization (ODLRI), which assigns a specific role to the low-rank component \( \mathbf{LR} \) to capture salient weights while using the quantization matrix \( \mathbf{Q} \) for the remaining weights.

• Comprehensive experiments demonstrate that incorporating ODLRI reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy.

2 Preliminaries

2.1 Post-Training Quantization

Early post training quantization (PTQ) methods (Jacob et al., 2018; Nagel et al., 2021; Dettmers et al., 2022; Shao et al., 2024) mainly focus on minimizing the direct quantization error for a given weight matrix \( \mathbf{W} \in \mathbb{R}^{m \times n} \), optimizing:
\[
\arg\min_{\mathbf{Q}} \| \mathbf{W} - \mathbf{Q} \|_F^2,
\]
where \( \mathbf{Q} \) is obtained by rounding weights to the nearest discrete values (Banner et al., 2019; Stock et al., 2020; Wu et al., 2020). However, this naive rounding approach often results in significant accuracy degradation. This problem is particularly pronounced in large-scale LLMs, where even small perturbations in weights can propagate across layers, gradually accumulating and worsening errors.

To mitigate this issue, activation-aware PTQ methods (Dong et al., 2019; Nagel et al., 2020; Li et al., 2021, 2024a) incorporate a calibration dataset \( \mathbf{X} \in \mathbb{R}^{n \times d} \) to account for the interaction between weight quantization and activations:
\[
\arg\min_{\mathbf{Q}} \| (\mathbf{W} - \mathbf{Q}) \mathbf{X} \|_F^2.
\]
This formulation ensures that weight quantization preserves the statistical behavior of activations, and leads to improved model performance. OPTQ (Frantar et al., 2023) refines activation-aware PTQ by introducing error feedback, which reduces cumulative quantization errors for effective low-bit quantization. QuIP (Chee et al., 2023) and QuIP# (Tseng et al., 2024) further improve robustness through incoherence processing, applying orthogonal transformations to \( \mathbf{W} \) and its local Hessian to reduce correlations, making 2-bit quantization more effective.

In addition, recent studies show that effectively controlling a small portion of activation outlier-sensitive weights can significantly enhance quantization performance. SpQR (Dettmers et al., 2023b) retains these critical salient weights in higher precision, while AWQ employs per-channel scaling to protect them, effectively mitigating the adverse effects of activation outliers during quantization.

2.2 Weight Matrix Factorization

Matrix factorization decomposes a matrix into low-rank components for efficient representation and computation (Golub et al., 1987; Saha et al., 2023). Recently, it has been applied to LLM compression by approximating weight matrices with low-rank representations (Li et al., 2023; Gao et al., 2024b). Given a weight matrix \( \mathbf{W} \in \mathbb{R}^{m \times n} \), these methods decompose it into lower-dimensional matrices, \( \mathbf{L} \in \mathbb{R}^{m \times r} \) and \( \mathbf{R} \in \mathbb{R}^{r \times n} \), by minimizing:
\[
\arg\min_{\mathbf{L}, \mathbf{R}} \| \mathbf{W} - \mathbf{LR} \|_F^2.
\]
By reducing the rank \( r \), low-rank decomposition, such as Singular Value Decomposition (SVD) effectively reduces storage and computational costs while preserving key weight structures.

Activation-aware factorization further incorporates activation statistics to refine weight representation, keeping the decomposition aligned with the model’s actual computational behavior. ASVD (Yuan et al., 2023) introduces a diagonal scaling matrix to normalize activations, improving numerical stability, while SVD-LLM (Wang et al., 2025) applies truncation-aware data whitening via Cholesky decomposition.

2.3 Quantization Error Reconstruction

Empirical studies suggest that quantization errors \( \mathbf{W} - \mathbf{Q} \) often exhibit a low-rank structure (Yao et al., 2024). This observation has led to hybrid approaches that first quantize the weight matrix and then approximate the resulting quantization error using a low-rank term, which serves as an error compensation mechanism:

\[
\mathbf{Q} = \arg\min_{\mathbf{Q}} \| \mathbf{W} - \mathbf{Q} \|_F^2 \\
\mathbf{L}, \mathbf{R} = \arg\min_{\mathbf{L}, \mathbf{R}} \| \mathbf{W} - \mathbf{Q} - \mathbf{LR} \|_F^2.
\]

Here, \( \mathbf{Q} \) represents the quantized weight matrix, forming the major representation of \( \mathbf{W} \), while \( \mathbf{LR} \) provide a low-rank approximation of the residual quantization error (Liu et al., 2024).

ZeroQuant-V2 (Yao et al., 2024) estimates \( \mathbf{L} \) and \( \mathbf{R} \) by applying SVD to the residual error \( \mathbf{W} - \mathbf{Q} \). Subsequent methods further refine this process by incorporating activation-aware adjustments: LQER (Zhang et al., 2024) introduces a diagonal scaling matrix derived from activations, while QERA (Zhang et al., 2025) improves error reconstruction by leveraging an input-space autocorrelation matrix, and provides theoretical guarantees for enhanced performance.

Unlike Q-LoRA (Dettmers et al., 2023a) and other quantization-based parameter-efficient fine-tuning (Q-PEFT) methods that focus on weight representations optimized for fine-tuning, quantization error reconstruction methods solely aim for efficient weight representation.

2.4 Jointly Optimized Quantization and Low Rank Approximation

A natural extension of quantization error reconstruction is to jointly optimize both the quantized component \( \mathbf{Q} \) and the low-rank component \( \mathbf{LR} \), leading to the following formulation:

\[
\mathbf{Q}, \mathbf{L}, \mathbf{R} = \arg\min_{\mathbf{Q}, \mathbf{L}, \mathbf{R}} \| (\mathbf{W} - \mathbf{Q} - \mathbf{LR}) \|_F^2.
\]

Unlike the two-stage approach that applies quantization followed by low-rank error correction once, joint optimization methods often iteratively alternate between quantization and low-rank approximation, which generally leads to better performance. These methods typically adopt either a quantize-first strategy (Saha et al., 2024; Li et al., 2024b) or a low-rank-first strategy (Guo et al., 2024), distinguished by their iteration ordering.

Rather than distinguishing them by iteration sequences, we propose interpreting these approaches through their initialization of low-rank components. Specifically, the quantize-first approach initializes \( \mathbf{LR} \) to zero, whereas the low-rank-first approach initializes them to factorized weights. Our perspective enables us to reframe joint optimization methods within a unified view in Algorithm 1.

Algorithm 1 Joint \( \mathbf{Q} + \mathbf{LR} \) Optimization

Input: Pretrained weight \( \mathbf{W} \), Num of iterations \( T \)
Output: \( \mathbf{Q}_T, \mathbf{L}_T, \mathbf{R}_T \)
1: \( \mathbf{L}_0, \mathbf{R}_0 \leftarrow \textbf{Initialize} \)
2: for \( t = 1 \) to \( T \) do
3:   \( \mathbf{Q}_t \leftarrow \textbf{Quantize}(\mathbf{W} - \mathbf{L}_{t-1}\mathbf{R}_{t-1}) \)
4:   \( \mathbf{L}_t, \mathbf{R}_t \leftarrow \textbf{LRApprox}(\mathbf{W} - \mathbf{Q}_t) \)
5: end for
Return: \( \widehat{\mathbf{W}} = \mathbf{Q}_T + \mathbf{L}_T \cdot \mathbf{R}_T \)

Through Algorithm 1, we gain deeper insights into the joint optimization procedure. This framework not only allows us to explore algorithmic variations by examining different initialization strategies but also suggests the potential role of initialization choices in weight decomposition.

CALDERA (Saha et al., 2024), a weight-only PTQ method jointly optimizes \( \mathbf{Q}, \mathbf{L}, \) and \( \mathbf{R} \) in an activation-aware manner, has shown significant improvements in quantization quality, particularly in extreme low-bit settings. Despite its outstanding performance, CALDERA solely examines its approach by initializing low-rank components to zero, neglecting the potential impact of initialization on performance. Within this CALDERA framework, we further concentrate on how different initialization strategies for the low-rank component affect performance, aiming to shed light on its role in the overall optimization process.
Figure 1: Outlier-Driven Low-Rank Initialization (ODLRI) Framework. ODLRI decomposes the weight matrix \( \mathbf{W} \) by first identifying salient weights, corresponding to activation outliers, using the diagonal of the Hessian. These salient weights are then approximated via low-rank decomposition, producing \( \mathbf{L}_0 \) and \( \mathbf{R}_0 \), while the remaining weights are quantized. This decomposition serves as initialization for the iterative joint optimization of \( \mathbf{Q} + \mathbf{LR} \).

3 Rethinking Joint Q+LR Optimization

3.1 Dependence on LR Initialization
To examine the role of initialization in joint optimization, we evaluate CALDERA (Saha et al., 2024) under two distinct initialization strategies for \( \mathbf{LR} \): zero initialization and matrix factorization-based initialization. For analyzing the contribution of quantized representation and low-rank components to \( \mathbf{WX} \), we measure \( \| \mathbf{QX} \| \) and \( \| \mathbf{LRX} \| \) at both the initial and final stage of iteration.

Surprisingly, as shown in Table 1, when \( \mathbf{L}, \mathbf{R} \) are initialized to zero, we observe that \( \mathbf{Q} \) persistently reconstructs \( \mathbf{W} \), while \( \mathbf{LR} \) serves as a residual correction, closely resembling quantization error reconstruction. Conversely, initializing \( \mathbf{L}, \mathbf{R} \) with a matrix factorization of \( \mathbf{W} \) leads to a reversed role assignment, where \( \mathbf{LR} \) captures most of \( \mathbf{W} \), and \( \mathbf{Q} \) quantizes the residuals throughout the iteration.

<table>
  <tr>
    <th>LR Initialization</th>
    <th>0</th>
    <th>LRApprox(W)</th>
  </tr>
  <tr>
    <th></th>
    <th>\( \| \mathbf{QX} \| \)</th>
    <th>\( \| \mathbf{LRX} \| \)</th>
    <th>\( \| \mathbf{QX} \| \)</th>
    <th>\( \| \mathbf{LRX} \| \)</th>
  </tr>
  <tr>
    <td>First Iteration</td>
    <td>0.999</td>
    <td>0.014</td>
    <td>0.158</td>
    <td>0.915</td>
  </tr>
  <tr>
    <td>Last Iteration</td>
    <td>0.961</td>
    <td>0.073</td>
    <td>0.401</td>
    <td>0.664</td>
  </tr>
</table>

Table 1: Effect of LR Initialization in CALDERA. Activation norms \( \| \mathbf{QX} \| \) and \( \| \mathbf{LRX} \| \) are reported at the first and last iterations for the Layer 1 Key Projection matrix of Llama2-7B over 15 iterations. Norms are normalized by \( \| \mathbf{WX} \| \) (i.e., \( \| \mathbf{LRX} \| / \| \mathbf{WX} \| \)). More results are provided in Appendix C.4.

This finding reveals that joint optimization outcomes are highly sensitive to initialization choices, which ultimately determine whether quantization or matrix factorization dominates the final representation. While existing methods have defaulted to either zero initialization or low-rank approximation of \( \mathbf{W} \) (i.e., LRApprox(\( \mathbf{W} \))), the optimality of these conventional approaches remains unexplored. Rather than being restricted to these established choices, we ask:

How should we initialize \( \mathbf{L}, \mathbf{R} \) to achieve optimal decomposition of \( \mathbf{W} \) into \( \mathbf{Q} + \mathbf{LR} \)?

3.2 Outlier-Driven Low-Rank Initialization
For optimal decomposition of \( \mathbf{W} \approx \mathbf{Q} + \mathbf{LR} \), we assign distinct roles to quantization and low-rank approximation based on their properties. Quantization is highly sensitive to activation outliers, as extreme activations amplify weight sensitivity, leading to discretization errors that degrade model performance (Lin et al., 2024; Dettmers et al., 2023b). In contrast, the low-rank component utilizes a product formulation of two low-bit factors, effectively yielding a higher bit representation than quantization. To leverage the enhanced representational capacity of the low-rank component, we structure \( \mathbf{LR} \) to explicitly capture salient weights before quantization, rather than treating it as a post-hoc correction term. This results in salient weights being absorbed into the low-rank component, allowing \( \mathbf{Q} \) to operate on a smoother, more uniform residual, ultimately improving quantization efficiency.

To explicitly assign the low-rank component to capturing salient weights, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), as illustrated in Figure 1. More precisely, we first decompose the activation matrix \( \mathbf{X} \) into two components:

\[
\mathbf{X} = \mathbf{X}_o + \mathbf{X}_r,
\]
where \( \mathbf{X}_o \in \mathbb{R}^{n \times d} \) contains only the top-\( k \) activation channels (outliers) with the highest norms, the remaining channels are set to zero. These top-\( k \) activation channels are identified by analyzing the diagonal entries of the Hessian, computed as \( \mathbf{H} = \mathbf{X} \mathbf{X}^\top \). The remaining activations (non-outlier components) are then captured by \( \mathbf{X}_r \).

Although setting \( k = r \) (where \( \mathbf{L} \in \mathbb{R}^{m \times r} \) and \( \mathbf{R} \in \mathbb{R}^{r \times n} \)) would maximize the use of low-rank approximation, we intentionally choose \( k < r \) to focus aggressively on outlier-related structures rather than broadly approximating the entire weight distribution. This targeted selection ensures that the low-rank component prioritizes the most activation-sensitive elements, refining the decomposition and enhancing quantization robustness.

We then initialize \( \mathbf{L} \) and \( \mathbf{R} \) via outlier-aware matrix factorization, focusing on high-variance activation directions that are likely to induce quantization errors. Specifically, we solve the following truncated optimization problem:

\[
\mathbf{L}_0, \mathbf{R}_0 = \arg\min_{\mathbf{L}, \mathbf{R}} \| (\mathbf{W} - \mathbf{LR}) \mathbf{H}_o (\mathbf{W} - \mathbf{LR})^\top \|,
\]

where \( \mathbf{H}_o = \mathbf{X}_o \mathbf{X}_o^\top \) captures the covariance of outlier-sensitive channels. This objective ensures that \( \mathbf{L}_0 \mathbf{R}_0 \) prioritizes reconstructing weight directions that interact strongly with outlier channels in the activation distribution. A detailed algorithm for solving this optimization is provided in Appendix B.1, and the specific selection of \( k \) is described in Appendix B.2.

4 Experiment

4.1 Experimental Setup

To evaluate the effectiveness of Outlier-Driven Low-Rank Initialization (ODLRI), we integrate it into CALDERA (Saha et al., 2024), an iterative joint optimization framework for extreme low-bit quantization of \( \mathbf{Q}, \mathbf{L}, \mathbf{R} \). By default, CALDERA first quantizes the weight matrix and then optimizes a low-rank component, effectively treating \( \mathbf{L}, \mathbf{R} \) as zero-initialized correction factors.

In contrast, ODLRI replaces this zero-initialization with an outlier-aware initialization, ensuring that salient weights are explicitly modeled in the low-rank component before quantization. We conduct a comprehensive comparison against CALDERA’s standard setup to quantify the impact of LR initialization on final accuracy and stability.

Quantization Setup. We integrate ODLRI into CALDERA while keeping the quantization configuration largely unchanged. For Quantize, we use a 2-Bit quantization scheme implemented via QuIP# (Tseng et al., 2024), which employs an E8 lattice codebook for stable 2-Bit quantization. For LRAprox, we experiment with both 16-Bit and 4-Bit precision, where the 16-Bit setting remains unquantized, while the 4-Bit setting undergoes quantization. For 4-Bit precision, we adjust \( \mathbf{L} \) and \( \mathbf{R} \) via the LPLR iterative algorithm (Saha et al., 2023).

CALDERA performs outer iterations alternating between Quantize and LRAprox, along with inner iterations for the LPLR procedure. We follow CALDERA’s default configuration, running 15 outer iterations, and 10 inner iterations when the \( \mathbf{L}, \mathbf{R} \) components are quantized to 4-Bit.

Models. We conduct experiments on a range of large language models, including Llama2 (7B, 13B, and 70B parameters) (Touvron et al., 2023), Llama3-8B (Dubey et al., 2024), and Mistral-7B (Jiang et al., 2023).

Evaluation Metrics. We evaluate quantization performance using perplexity and zero-shot accuracy benchmarks. Perplexity is measured on the test splits of WikiText-2 (Merity et al., 2017) and C4 (Raffel et al., 2020), following each model’s predefined context length (e.g., 4096 tokens for Llama2 and 8192 tokens for Llama3). Zero-shot accuracy is assessed using the EleutherAI lm-evaluation-harness (Gao et al., 2024a), covering a range of NLP benchmarks, including Winogrande (Sakaguchi et al., 2020), RTE (Bentivogli et al., 2009), PiQA (Bisk et al., 2020), ARC Easy (Clark et al., 2018), and ARC Challenge (Clark et al., 2018). To ensure statistical reliability, most results are averaged across two independent random seeds (Details are in Appendix A).

4.2 Effect of LR Initialization on Joint Q+LR Optimization

We compare three initialization methods for \( \mathbf{L}, \mathbf{R} \) to evaluate their impact on performance. The first method, Zero Initialization (0), follows CALDERA’s default setup, where \( \mathbf{L}_0 = 0 \) and \( \mathbf{R}_0 = 0 \). The second method, LRAprox(W), initializes \( \mathbf{L}, \mathbf{R} \) using a low-rank approximation (LPLR) of \( \mathbf{W} \). The third method, ODLRI, is our proposed approach, which explicitly focuses salient weights for low-rank approximation.
Figure 2: Quantization Scale across different initialization strategies. We present the quantization scale over 15 iterations, where both \( \mathbf{L} \) and \( \mathbf{R} \) are quantized to 4-Bit at rank 256. The three subplots display results for the Key (left), Value (middle), and Down (right) projection layers in Layer 10 of Llama2-7B. ODLRI (red stars) consistently achieves the lowest quantization scale, highlighting its effectiveness in low-bit quantization. Additional results are provided in the Appendix C.5

Figure 3: Activation-aware Error across different initialization strategies. We present the normalized activation-aware error \( \|(\mathbf{W} - \mathbf{Q} - \mathbf{LR})\mathbf{X}\|_F^2 / \|W\mathbf{X}\|_F^2 \) over 15 iterations, where both \( \mathbf{L} \) and \( \mathbf{R} \) are quantized to 4-Bit at rank 256. The three subplots display results for the Key (left), Value (middle), and Down (right) projection layers in Layer 10 of Llama2-7B. ODLRI (red stars) consistently achieves the lowest error, demonstrating its effectiveness in accurately representing weights. Additional results are provided in the Appendix C.5

Quantization Scale. We first examine the quantization scale, which reflects the dynamic range of the weights and directly impacts low-bit quantization efficiency. A lower quantization scale indicates a more compact weight distribution, enabling finer representation in low-bit precision and reducing overall quantization error.

Figure 2 shows that ODLRI significantly reduces the quantization weight scale, ensuring that weights are mapped into a range more suitable for accurate quantization. In contrast, baseline methods exhibit higher scales, making quantization more challenging and increasing the risk of performance degradation. This reduction in quantization scale with ODLRI is a key factor in achieving superior performance in extreme low-bit settings.

Additional results on other layers and different projection types confirm consistent scale reduction by ODLRI (see Appendix C.5).

Activation-aware Error. To further validate that ODLRI effectively minimizes our objective, we measure the normalized activation-aware error for each layer’s weight component. This metric evaluates how well the decomposition preserves activation-dependent weight structure:

\[
\frac{\|(\mathbf{W} - \mathbf{Q} - \mathbf{LR})\mathbf{X}\|_F^2}{\|\mathbf{W}\mathbf{X}\|_F^2}.
\]

As illustrated in Figure 3, the default CALDERA configuration with zero initialization results in significantly higher activation-aware errors. Even when using a low-rank approximation of \( \mathbf{W} \), the error remains consistently higher across layers compared to ODLRI. The substantial reduction in activation-aware error achieved by ODLRI confirms that it represents weight approximations more effectively. Additional results are provided in Appendix C.5.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Avg Bits</th>
    <th colspan="2">PPL ↓</th>
    <th colspan="5">Zero-Shot Accuracy ↑</th>
  </tr>
  <tr>
    <th>Wiki2</th>
    <th>C4</th>
    <th>Wino</th>
    <th>RTE</th>
    <th>PiQA</th>
    <th>ArcE</th>
    <th>ArcC</th>
  </tr>
  <tr>
    <td rowspan="4">7B</td>
    <td>CALDERA</td>
    <td>64</td>
    <td>2.1</td>
    <td>7.34</td>
    <td><b>9.50</b></td>
    <td>63.85</td>
    <td>55.23</td>
    <td>72.69</td>
    <td>62.25</td>
    <td>32.04</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td>2.1</td>
    <td><b>7.20</b></td>
    <td>9.52</td>
    <td><b>65.04</b></td>
    <td><b>62.45</b></td>
    <td><b>72.91</b></td>
    <td><b>65.49</b></td>
    <td><b>33.70</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>2.2</td>
    <td>6.90</td>
    <td>9.01</td>
    <td><b>65.27</b></td>
    <td>57.58</td>
    <td>72.80</td>
    <td><b>63.89</b></td>
    <td>34.21</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>2.2</td>
    <td><b>6.72</b></td>
    <td><b>8.82</b></td>
    <td>64.33</td>
    <td><b>57.76</b></td>
    <td><b>75.30</b></td>
    <td>62.63</td>
    <td>33.19</td>
  </tr>
  <tr>
    <td rowspan="4">13B</td>
    <td>CALDERA</td>
    <td>256</td>
    <td>2.4</td>
    <td>6.47</td>
    <td>8.47</td>
    <td>65.19</td>
    <td>60.11</td>
    <td>74.43</td>
    <td><b>66.35</b></td>
    <td>34.25</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>2.4</td>
    <td><b>6.33</b></td>
    <td><b>8.27</b></td>
    <td><b>66.42</b></td>
    <td><b>61.01</b></td>
    <td><b>74.67</b></td>
    <td>65.49</td>
    <td><b>36.09</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>64</td>
    <td>2.08</td>
    <td>5.92</td>
    <td><b>7.96</b></td>
    <td>66.37</td>
    <td>58.84</td>
    <td>75.27</td>
    <td><b>69.00</b></td>
    <td><b>38.18</b></td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td>2.08</td>
    <td><b>5.91</b></td>
    <td><b>7.97</b></td>
    <td><b>67.32</b></td>
    <td><b>63.18</b></td>
    <td><b>75.90</b></td>
    <td>65.82</td>
    <td>37.20</td>
  </tr>
  <tr>
    <td rowspan="4">70B</td>
    <td>CALDERA</td>
    <td>128</td>
    <td>2.16</td>
    <td>5.77</td>
    <td>7.71</td>
    <td>67.4</td>
    <td><b>62.45</b></td>
    <td>75.70</td>
    <td><b>69.65</b></td>
    <td>38.91</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>2.16</td>
    <td><b>5.73</b></td>
    <td><b>7.71</b></td>
    <td><b>69.69</b></td>
    <td>61.37</td>
    <td><b>76.50</b></td>
    <td>68.69</td>
    <td><b>39.32</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>256</td>
    <td>2.32</td>
    <td>5.56</td>
    <td>7.39</td>
    <td><b>69.13</b></td>
    <td><b>64.08</b></td>
    <td>75.98</td>
    <td>70.26</td>
    <td>39.88</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>2.32</td>
    <td><b>5.46</b></td>
    <td><b>7.28</b></td>
    <td>68.19</td>
    <td>62.63</td>
    <td><b>76.65</b></td>
    <td><b>71.48</b></td>
    <td><b>39.96</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>2.1</td>
    <td>4.09</td>
    <td>5.91</td>
    <td><b>75.93</b></td>
    <td>71.84</td>
    <td>79.38</td>
    <td>76.98</td>
    <td>47.10</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>2.1</td>
    <td><b>4.06</b></td>
    <td><b>5.89</b></td>
    <td>75.30</td>
    <td><b>72.56</b></td>
    <td><b>79.98</b></td>
    <td><b>78.45</b></td>
    <td><b>49.06</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>256</td>
    <td>2.2</td>
    <td>3.99</td>
    <td>5.78</td>
    <td>75.69</td>
    <td><b>72.92</b></td>
    <td>80.41</td>
    <td><b>78.28</b></td>
    <td>49.15</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>2.2</td>
    <td><b>3.94</b></td>
    <td><b>5.73</b></td>
    <td><b>75.91</b></td>
    <td><b>70.75</b></td>
    <td><b>80.73</b></td>
    <td><b>77.96</b></td>
    <td><b>49.91</b></td>
  </tr>
  <tr>
    <td>Uncompressed (7B)</td>
    <td>—</td>
    <td>16</td>
    <td>5.12</td>
    <td>6.63</td>
    <td>67.3</td>
    <td>63.2</td>
    <td>78.5</td>
    <td>69.3</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Uncompressed (13B)</td>
    <td>—</td>
    <td>16</td>
    <td>4.57</td>
    <td>6.05</td>
    <td>69.5</td>
    <td>61.7</td>
    <td>78.8</td>
    <td>73.2</td>
    <td>45.6</td>
  </tr>
  <tr>
    <td>Uncompressed (70B)</td>
    <td>—</td>
    <td>16</td>
    <td>3.12</td>
    <td>4.97</td>
    <td>77.0</td>
    <td>67.9</td>
    <td>81.1</td>
    <td>77.7</td>
    <td>51.1</td>
  </tr>
</table>

Table 2: Comparison of our method with CALDERA on zero-shot perplexities (↓) and accuracies (↑) of Llama2 models on WikiText-2 and C4. Q is quantized 2-Bit and L, R are quantized 4-Bit. Lower perplexity and higher accuracy values are bolded.

4.3 Zero-shot Evaluation

Following the setup in the previous section, we conduct zero-shot evaluations of various models with a fixed 2-Bit Q component and an LR component set to either quantized 4-Bit or unquantized 16-Bit. The goal is to assess how applying ODLRI initialization compares against the baseline CALDERA under different quantization constraints.

4-Bit LR on Llama2. Table 2 reports results where the L and R are quantized to 4-Bit, with LPLR applied for iterative updates. Despite this additional optimization step, ODLRI consistently improves perplexity (WikiText-2, C4) and zero-shot accuracy across multiple tasks (e.g., PiQA, RTE) for most configurations. These results highlight that a principled initialization strategy enhances performance, even under aggressive quantization.

Importantly, the only modification from standard CALDERA is the LR initialization strategy, demonstrating that even in highly constrained quantization settings, ODLRI retains crucial distributional information that would otherwise be lost. Additional results under more extreme compression, specifically at lower ranks (\( r \leq 32 \)), can be found in Appendix C.2.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Avg Bits</th>
    <th colspan="2">PPL ↓</th>
  </tr>
  <tr>
    <th>Wiki2</th>
    <th>C4</th>
  </tr>
  <tr>
    <td rowspan="4">7B</td>
    <td>CALDERA</td>
    <td>64</td>
    <td>2.40</td>
    <td>7.25</td>
    <td>9.52</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td>2.40</td>
    <td><b>7.17</b></td>
    <td><b>9.41</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>2.80</td>
    <td>8.20</td>
    <td>8.95</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>2.80</td>
    <td><b>6.70</b></td>
    <td><b>8.79</b></td>
  </tr>
  <tr>
    <td rowspan="4">13B</td>
    <td>CALDERA</td>
    <td>256</td>
    <td>3.60</td>
    <td>6.42</td>
    <td>8.43</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>3.60</td>
    <td><b>6.18</b></td>
    <td><b>8.23</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>64</td>
    <td>2.32</td>
    <td>5.93</td>
    <td><b>7.95</b></td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td>2.32</td>
    <td><b>5.90</b></td>
    <td><b>7.96</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>2.64</td>
    <td>5.77</td>
    <td>7.69</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>2.64</td>
    <td><b>5.64</b></td>
    <td><b>7.54</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>256</td>
    <td>3.28</td>
    <td>5.48</td>
    <td>7.32</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>3.28</td>
    <td><b>5.38</b></td>
    <td><b>7.14</b></td>
  </tr>
  <tr>
    <td>Uncompressed (7B)</td>
    <td>—</td>
    <td>16</td>
    <td>5.12</td>
    <td>6.63</td>
  </tr>
  <tr>
    <td>Uncompressed (13B)</td>
    <td>—</td>
    <td>16</td>
    <td>4.57</td>
    <td>6.05</td>
  </tr>
</table>

Table 3: Zero-Shot Perplexity (↓) of ODLRI with CALDERA for Llama2 models on WikiText-2 and C4. Q is 2-Bit quantized, and L, R are 16-Bit. Lower values are bolded. Additional zero-shot accuracy results are provided in Appendix C.1.

16-Bit LR on Llama2. We further evaluate the models in a 16-Bit LR setting, where the LR component is left unquantized (Table 3).

Without the need for low-bit refinement steps on LR component, this configuration allows for a direct assessment of ODLRI’s effectiveness. It en-
ables \( \mathbf{L}_0, \mathbf{R}_0 \) to fully capture the benefits of outlier-driven initialization, preserving fine-grained weight structures that are critical for performance under extreme low-bit settings. As expected, perplexity and zero-shot accuracy improve compared to the more constrained 4-Bit LR scenario, providing clearer evidence of ODLRI’s effectiveness without the confounding effects of additional LR quantization. As in the 4-Bit LR setting, ODLRI achieves more pronounced gains, particularly at higher ranks. The corresponding zero-shot accuracy measurements for 16-Bit LR setting are provided in Appendix C.1.

These results demonstrate that the effectiveness of ODLRI holds across both aggressive 4-Bit LR settings and relaxed 16-Bit LR conditions without additional refinement, highlighting its robustness across diverse compression regimes.

4-Bit LR on Llama3-8B and Mistral-7B. To assess the generalizability of ODLRI beyond Llama2 models, we evaluate its performance on Llama3-8B and Mistral-7B. Table 4 presents results for these models with 4-Bit quantization applied to the LR component. Under a range of rank configurations, ODLRI consistently improves over original CALDERA by more effectively capturing salient weight directions, resulting in lower perplexity on both WikiText-2 and C4. These results demonstrate that the effectiveness of ODLRI generalizes beyond Llama2 models. We also provide results on non-LLaMA models and alternative quantizers in Appendix C.3.

<table>
  <tr>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Llama3-8B</th>
    <th colspan="2">Mistral-7B</th>
  </tr>
  <tr>
    <th>Wiki2 ↓</th>
    <th>C4 ↓</th>
    <th>Wiki2 ↓</th>
    <th>C4 ↓</th>
  </tr>
  <tr>
    <td rowspan="2">64</td>
    <td>CALDERA</td>
    <td>10.58</td>
    <td>11.35</td>
    <td>6.37</td>
    <td>7.11</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td><b>10.35</b></td>
    <td><b>11.15</b></td>
    <td><b>6.37</b></td>
    <td><b>7.10</b></td>
  </tr>
  <tr>
    <td rowspan="2">128</td>
    <td>CALDERA</td>
    <td>9.41</td>
    <td>10.21</td>
    <td>6.11</td>
    <td>6.89</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td><b>9.35</b></td>
    <td><b>10.32</b></td>
    <td><b>6.08</b></td>
    <td><b>6.86</b></td>
  </tr>
  <tr>
    <td rowspan="2">256</td>
    <td>CALDERA</td>
    <td>8.70</td>
    <td>9.77</td>
    <td>5.77</td>
    <td>6.59</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td><b>8.12</b></td>
    <td><b>9.33</b></td>
    <td><b>5.69</b></td>
    <td><b>6.53</b></td>
  </tr>
</table>

Table 4: Zero-Shot Perplexity (\( \downarrow \)) of ODLRI with CALDERA for Llama3-8B and Mistral-7B on WikiText-2 and C4. Q is 2-Bit quantized, while \( \mathbf{L}, \mathbf{R} \) are 4-Bit quantized. Lower values are bolded.

4.4 Number of Outlier Columns (\( k \))

In ODLRI, we determine salient weight components by explicitly targeting activation outliers, selecting top-\( k \) activations corresponding to outlier-sensitive weights. Instead of following the rank-based selection that picks the top-\( r \) components based solely on low-rank dimension \( r \), we set \( k < r \) to intensively focus on outliers.

<table>
  <tr>
    <th rowspan="2">ODLRI</th>
    <th colspan="2">L, R 16-Bit</th>
    <th colspan="2">L, R 4-Bit</th>
  </tr>
  <tr>
    <th>Wiki2 ↓</th>
    <th>C4 ↓</th>
    <th>Wiki2 ↓</th>
    <th>C4 ↓</th>
  </tr>
  <tr>
    <td>H<sub>o</sub> (\( k = r \))</td>
    <td>6.38</td>
    <td>8.43</td>
    <td>6.46</td>
    <td>8.52</td>
  </tr>
  <tr>
    <td>H<sub>o</sub> (\( k < r \))</td>
    <td><b>6.18</b></td>
    <td><b>8.23</b></td>
    <td><b>6.33</b></td>
    <td><b>8.27</b></td>
  </tr>
</table>

Table 5: Comparison of ODLRI with various values of \( k \), specifically H<sub>o</sub> (\( k = 256 \)) and H<sub>o</sub> (\( k = 16 \)) by perplexities (\( \downarrow \)) of Llama2-7B on WikiText-2 and C4. Q is 2-Bit, and \( \mathbf{L}, \mathbf{R} \) are either 16-Bit or quantized 4-Bit with a rank of 256. Lower perplexity values are bolded. Details regarding selection of \( k \) are provided in Appendix B.2.

To evaluate the impact of this selection, we measure perplexity (PPL) under different configurations. Our results show that choosing \( k \) based on activation outliers consistently outperforms selecting the top-\( r \) components through standard low-rank approximation. This improvement highlights the importance of leveraging activation statistics to guide low-rank approximation, ensuring that the most outlier-sensitive weights are efficiently modeled in LR. These findings validate that ODLRI’s targeted selection enhances low-rank approximation, leading to better quantization performance in extreme low-bit settings.

5 Conclusion

In this work, we investigated the role of initialization strategies in iterative joint optimization approaches for LLM compression. We introduce a unified framework that reformulates joint optimization algorithms through the lens of low-rank component initialization. Our analysis shows that the choice of initialization determines the entire trajectory of weight decomposition by assigning persistent roles to the components. Based on these insights, we proposed ODLRI, which assigns a distinct role to the low-rank component to capture activation-sensitive weights while using quantization for the remaining weights. Through extensive experiments across various LLM architectures, we demonstrated that incorporating ODLRI significantly improves model performance and compression stability. Consequently, our approach advances the practical implementation of efficient LLM compression and steers us toward optimal weight decomposition.
Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2024-00457882, National AI Research Lab Project), IITP grant funded by the Korean Government (MSIT) (No. RS-2020-II201361, Artificial Intelligence Graduate School Program (Yonsei University)), and K-CHIPS(Korea Collaborative & High-tech Initiative for Prospective Semiconductor Research) (RS-2024-00405946, 24052-15TC) funded by the Ministry of Trade, Industry & Energy (MOTIE, Korea).

Limitations

Our work focuses on weight-only quantization, optimizing the decomposition of model weights into quantized and low-rank components. While this approach enhances low-bit quantization performance, it does not address the quantization of activations or KV cache, which are critical for further improving inference efficiency. Joint weight and activation quantization introduces additional challenges, such as distributional shifts and increased sensitivity to outliers, requiring specialized calibration techniques. Similarly, KV cache quantization is essential for reducing memory overhead in long-context inference but remains outside the scope of this study.

Additionally, while we evaluate ODLRI within CALDERA, we believe that our approach can be applied to other joint Q + LR optimization algorithms beyond CALDERA. Exploring how ODLRI integrates with different iterative quantization frameworks presents an interesting direction for future research.

Ethical Considerations

We focus on efficient compression of LLMs through quantization and low-rank decomposition. While these techniques improve computational efficiency and enable broader deployment, they also present ethical considerations.

First, compressed models may inherit and amplify biases present in the original model. Ensuring fairness and preventing unintended distortions due to quantization artifacts remain critical challenges.

Second, model compression enhances accessibility but may also facilitate misuse, including deployment in applications without adequate ethical oversight. Responsible usage and regulation are essential.

We emphasize the importance of ethical AI development and encourage continued evaluation of the societal impact of model compression.

References

Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. 2022. Towards efficient post-training quantization of pre-trained language models. In NeurIPS.

Ron Banner, Yury Nahshan, and Daniel Soudry. 2019. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NeurIPS.

Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI.

Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. 2023. Quip: 2-bit quantization of large language models with guarantees. In NeurIPS.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Bita Darvish Rouhani, Ritchie Zhao, Venugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. 2023. With shared microexponents, a little shifting goes a long way. In ISCA.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In NeurIPS.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023a. Qlora: efficient finetuning of quantized llms. In NeurIPS.

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznecov, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoeffer, and Dan Al-istarth. 2023b. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078.

Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In ICLR.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach', Haonan Li, Kyle McDonell, Niklas Muenninghof, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024a. A framework for few-shot language model evaluation.

Shangqian Gao, Ting Hua, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. 2024b. Adaptive rank selections for low-rank approximation of language models. In NAACL-HLT.

G.H. Golub, Alan Hoffman, and G.W. Stewart. 1987. A generalization of the eckart-young-mirsky matrix approximation theorem. Linear Algebra and its Applications.

Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. 2024. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In ICLR.

Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. 2024. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. SqueezeLm: Dense-and-sparse quantization. In ICML.

Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. 2024. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI.

Liang Li, Qingyuan Li, Bo Zhang, and Xiangxiang Chu. 2024a. Norm tweaking: High-performance low-bit quantization of large language models. In AAAI.

Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2024b. Loftq: LoRA-fine-tuning-aware quantization for large language models. In ICLR.

Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Losparse: structured compression of large language models based on low-rank and sparse approximation. In ICML.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecc: Pushing the limit of post-training quantization by block reconstruction. In ICLR.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In MLSys.

Shih-Yang Liu, Huck Yang, Chien-Yi Wang, Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muradidharan, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, et al. 2024. Eora: Training-free compensation for compressed llm with eigenspace low-rank approximation. arXiv preprint arXiv:2410.21271.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR.

Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In ICML.

Markus Nagel, Marios Fourmarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295.

Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. In ICLR.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.

RelaxML. 2025. Hessians-llama-2 models (7b, 13b, 70b) - 6144. Accessed: 2025-02-09. Models available at: https://huggingface.co/relaxml.

Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussonet, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, et al. 2024. Gemma 2: Improving open language models at a practical size. In CoRR.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea Goldsmith, and Mert Pilanci. 2024. Compressing large language models using low rank and low precision decomposition. In NeurIPS.

Rajarshi Saha, Varun Srivastava, and Mert Pilanci. 2023. Matrix compression via randomized low rank and low precision factorization. In NeurIPS.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI.

Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR.

Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. 2020. And the bit goes down: Revisiting the quantization of neural networks. In ICLR.

Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre-trained language models via quantization. In ACL.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks. In ICML.

Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. 2025. SVD-LLM: Truncation-aware singular value decomposition for large language model compression. In ICLR.

Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaoqzhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Re, Irina Rish, and Ce Zhang. 2024. Redpajama: an open dataset for training large language models. In NeurIPS Datasets and Benchmarks Track.

Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. 2020. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602.

Jaewoo Yang, Hayun Kim, and Younghoon Kim. 2024. Mitigating quantization errors due to activation spikes in glu-based llms. arXiv preprint arXiv:2405.14428.

Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2024. Exploring post-training quantization in llms from comprehensive study to low rank compensation. In AAAI.

Mengxia Yu, De Wang, Qi Shan, and Alvin Wan. 2024. The super weight in large language models. arXiv preprint arXiv:2411.07191.

Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. 2023. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821.

Cheng Zhang, Jianyi Cheng, George Anthony Constantinides, and Yiren Zhao. 2024. Lqer: Low-rank quantization error reconstruction for llms. In ICML.

Cheng Zhang, Jeffrey T. H. Wong, Can Xiao, George Anthony Constantinides, and Yiren Zhao. 2025. QERA: an analytical framework for quantization error reconstruction. In ICLR.
A Additional Experimental Setup

Calibration Data and Hessian Computation. We use precomputed Hessians from RelaxML (2025) for Llama2 models. For Llama3-8B and Mistral-7B, Hessians are computed directly to ensure alignment with model-specific architectures. Hessians are computed using 256 randomly sampled examples from the RedPajama dataset (Weber et al., 2024). Context lengths are set to 4096 tokens for Llama2 and 8192 tokens for Llama3, maintaining consistency with each model’s native configuration.

Hardware Environment. Experiments were conducted on both consumer- and enterprise-grade GPUs. The primary pipeline was executed on NVIDIA GeForce RTX 3090, RTX 4090, and RTX A6000 GPUs, while the quantization and evaluation of the Llama2-70B model were performed on NVIDIA L40s GPU clusters due to its large size.

Computation time for quantization and evaluation varied depending on the model size and hardware configuration. For Llama2-7B and Mistral-7B, quantization was completed in 4 GPU hours on a cluster of six GPUs. For Llama2-13B, quantization required 8 GPU hours on a six-GPU cluster. Besides, for Llama3-8B, quantization was achieved in 5 GPU hours on a cluster of six GPUs. For Llama2-70B, quantization was conducted using four NVIDIA L40S GPUs, completing within 48 hours. In the absence of parallel processing, the estimated runtime would have increased proportionally to the number of GPUs used. Evaluation phase was carried out on the same GPUs utilized for quantization. Perplexity measurement required approximately 0.5 GPU hours, while zero-shot evaluation consumed around 1.5 GPU hours. Since pre-trained models were used, no additional training time was required.

All experiments were repeated using two different random seeds, with the exception of Llama2-70B, for which only one seed was used due to resource constraints. Overall, the quantization and evaluation processes accumulated approximately 4000 GPU hours in total.

CALDERA’s Default Configuration. In our experiments, the CALDERA model was configured with the following settings: hadamard_transform was set to true, outer_iter to 15, inner_iter to 10, rand_svd to false, Q_hessian_downdate to false, and update_order to Q LR.

Summary of Model and Dataset. We present a summary of the models and datasets employed in this paper. The detailed specifications, including sources, access methods, and licensing terms, are summarized in Table 6. Also, the datasets and their corresponding metadata are detailed in Table 7.

B Outlier-Aware Initialization Detail

B.1 ODLRI Method in Detail

To process outlier-sensitive weights by low-rank approximation, we decompose our optimization objective

\[
\mathbf{W} \mathbf{X} = \mathbf{W} (\mathbf{X}_o + \mathbf{X}_r) = \mathbf{W} \mathbf{X}_o + \mathbf{W} \mathbf{X}_r
\]

into two distinct components. Here, \( \mathbf{X}_o \) represents the channels containing outliers (with other channels set to zero), while \( \mathbf{X}_r \) indicates the remaining channels (with outlier channels zeroed). Since the non-zero entries of \( \mathbf{X}_o \) and \( \mathbf{X}_r \) are mutually exclusive along the channel dimension, both \( \mathbf{X}_o \) and \( \mathbf{X}_r \) retain the original matrix dimensions.

Standard activation-aware low-rank factorization, which solves

\[
\arg\min_{\mathbf{L}, \mathbf{R}} \|(\mathbf{W} - \mathbf{L}\mathbf{R})\mathbf{X}\|_F^2.
\]

Using the empirical second-moment matrix \( \mathbf{H} = \mathbf{X} \mathbf{X}^\top \), this objective is equivalent to:

\[
\arg\min_{\mathbf{L}, \mathbf{R}} \|(\mathbf{W} - \mathbf{L}\mathbf{R})\mathbf{H}(\mathbf{W} - \mathbf{L}\mathbf{R})^\top\|.
\]

However, this formulation treats all activations equally, leading to a low-rank approximation that does not explicitly focus on outliers, which are often the primary bottleneck in quantization.

To ensure that the low-rank component captures only the most challenging weight structures, we define a restricted activation covariance matrix \( \mathbf{H}_o \), a submatrix of \( \mathbf{H} \) that prioritizes outlier-sensitive channels:

\[
(\mathbf{H}_o)_{ij} = \begin{cases}
\mathbf{H}_{ij}, & \text{if } i, j \in \mathcal{I}, \\
0, & \text{otherwise}
\end{cases}
\]

where the subset of indices \( \mathcal{I} \subset \{1, \ldots, d\} \) corresponds to the top-\( k \) (\( k < r \)) channels with the highest diagonal values of \( \mathbf{H} \). These channels correspond to the most dominant activation patterns, which often align with weight outliers that distort quantization performance.
<table>
  <tr>
    <th>Model</th>
    <th>Source</th>
    <th>Accessed via</th>
    <th>License</th>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>(Touvron et al., 2023)</td>
    <td>Link</td>
    <td>Llama 2 Community License</td>
  </tr>
  <tr>
    <td>Llama2-13B</td>
    <td>(Touvron et al., 2023)</td>
    <td>Link</td>
    <td>Llama 2 Community License</td>
  </tr>
  <tr>
    <td>Llama2-70B</td>
    <td>(Touvron et al., 2023)</td>
    <td>Link</td>
    <td>Llama 2 Community License</td>
  </tr>
  <tr>
    <td>Llama3-8B</td>
    <td>(Dubey et al., 2024)</td>
    <td>Link</td>
    <td>Meta Llama 3 Community License</td>
  </tr>
  <tr>
    <td>Mistral-7Bv0.1</td>
    <td>(Jiang et al., 2023)</td>
    <td>Link</td>
    <td>Apache license 2.0</td>
  </tr>
</table>

Table 6: Summary of models used in this paper.

<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th>Accessed via</th>
    <th>License</th>
  </tr>
  <tr>
    <td>RedPajama-Data-1T-Sample</td>
    <td>(Weber et al., 2024)</td>
    <td>Link</td>
    <td>Apache License 2.0</td>
  </tr>
  <tr>
    <td>Hessians-Llama-2-7b-6144</td>
    <td>(RelaxML, 2025)</td>
    <td>Link</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Hessians-Llama-2-13b-6144</td>
    <td>(RelaxML, 2025)</td>
    <td>Link</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Hessians-Llama-2-70b-6144</td>
    <td>(RelaxML, 2025)</td>
    <td>Link</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Wikitext-2-raw-v1</td>
    <td>(Merity et al., 2017)</td>
    <td>Link</td>
    <td>CC-BY-SA-3.0</td>
  </tr>
  <tr>
    <td>C4</td>
    <td>(Raffel et al., 2020)</td>
    <td>Link</td>
    <td>ODC-BY</td>
  </tr>
  <tr>
    <td>Im-eval-harness</td>
    <td>(Gao et al., 2024a)</td>
    <td>Link</td>
    <td>MIT License</td>
  </tr>
  <tr>
    <td>Winogrande</td>
    <td>(Sakaguchi et al., 2020)</td>
    <td>Link</td>
    <td>Apache License 2.0</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>(Bentivogli et al., 2009)</td>
    <td>Link</td>
    <td>Apache License 2.0</td>
  </tr>
  <tr>
    <td>PiQA</td>
    <td>(Bisk et al., 2020)</td>
    <td>Link</td>
    <td>Apache License 2.0</td>
  </tr>
  <tr>
    <td>ARC Easy</td>
    <td>(Clark et al., 2018)</td>
    <td>Link</td>
    <td>CC-BY-SA-4.0</td>
  </tr>
  <tr>
    <td>ARC Challenge</td>
    <td>(Clark et al., 2018)</td>
    <td>Link</td>
    <td>CC-BY-SA-4.0</td>
  </tr>
</table>

Table 7: Summary of datasets used in this paper.

We then solve the outlier-aware optimization problem:

\[
\mathbf{L}_0, \mathbf{R}_0 = \arg\min_{\mathbf{L}, \mathbf{R}} \| (\mathbf{W} - \mathbf{LR}) \mathbf{H}_o (\mathbf{W} - \mathbf{LR})^\top \|.
\]

This is achieved by applying a Cholesky decomposition to the selected Hessian submatrix \( \mathbf{H}_o \), which encodes the activation covariance information of high-variance channels:

\[
\mathbf{H}_o = \mathbf{S}_o \mathbf{S}_o^\top,
\]

where \( \mathbf{S}_o \) is a lower triangular matrix.

Unlike SVD-LLM (Wang et al., 2025), which applies data whitening to the entire Hessian matrix \( \mathbf{H} \), our transformation performs selective whitening on the outlier subset \( \mathbf{H}_o \) corresponding to \( \mathbf{X}_o \). This selective whitening improves the numerical conditioning for the subsequent SVD while ensuring that the low-rank component effectively captures the salient weight information.

Once the whitening transformation is applied, we perform SVD on the transformed weight matrix \( \mathbf{WS}_o \), truncating the decomposition to rank \( r \) to obtain the outlier-focused low-rank components:

\[
\mathrm{SVD}(\mathbf{WS}_o) = \mathbf{U}_{:,r:r} \Sigma_{r:r,r} \mathbf{V}_{r:r,:}^\top.
\]

Here, the Python-style slicing notation \( \mathbf{U}_{:,r:r} \) indicates that we select all rows and the first \( r \) columns of \( \mathbf{U} \). Similarly, \( \Sigma_{r:r,r} \) denotes the upper-left \( r \times r \) block of \( \Sigma \), and \( \mathbf{V}_{r:r,:} \) selects the first \( r \) rows of \( \mathbf{V} \).

To ensure that the decomposition remains truncation-aware, we initialize the low-rank and quantization components as follows:

\[
\mathbf{L}_0 = \mathbf{U}_{:,r:r} \sqrt{\Sigma_{r:r,r}}
\]
\[
\mathbf{R}_0 = \sqrt{\Sigma_{r:r,r}} \mathbf{V}_{r:r,:}^\top \mathbf{S}_o^{-1}.
\]

By ensuring that the residual weight matrix \( \mathbf{W} - \mathbf{L}_0 \mathbf{R}_0 \) has been preconditioned to remove outlier effects, we significantly reduce quantization error compared to conventional activation-aware quantization approaches. This allows the quantized component \( \mathbf{Q} \) to operate on a more uniform weight distribution, improving numerical stability:

\[
\mathbf{Q}_1 = \mathrm{Quantize}(\mathbf{W} - \mathbf{L}_0 \mathbf{R}_0).
\]
B.2 Rank-Dependent Outlier Selection of \( k \)
In our experiments, we denote the rank of the low-rank components \( \mathbf{L} \) and \( \mathbf{R} \) as \( r \), and we set the number of outlier-sensitive columns \( k \) in proportion to this rank. Specifically, we define:

\[
k = p \times n,
\]

where \( n \) is the dimension of the calibration Hessian \( \mathbf{H} \in \mathbb{R}^{n \times n} \) and \( p \) is a rank-dependent percentage.

We adopt the following settings:

• For \( r = 64 \): \( p = 0.1\% \)

• For \( r = 128 \): \( p = 0.2\% \)

• For \( r = 256 \): \( p = 0.4\% \)

For example, consider the key projection matrix of Llama2-7B. Its corresponding Hessian matrix has a shape of \( 4096 \times 4096 \). When we set the rank \( r = 256 \), we use \( p = 0.4\% \). Thus, the number of outlier-sensitive columns is computed as:

\[
k = p \times n = 0.4\% \times 4096 \approx 16.
\]

This means that in this example, 16 outlier-sensitive columns are selected for the key projection.

B.3 ODLRI’s Impact on Salient Weights

<table>
  <tr>
    <th>Hessian</th>
    <th>\( \|LRX_o\| \)</th>
    <th>\( \|E_{LR}X_o\| \)</th>
    <th>\( \|LRX_r\| \)</th>
    <th>\( \|E_{LR}X_r\| \)</th>
  </tr>
  <tr>
    <td>H</td>
    <td>0.997</td>
    <td>0.073</td>
    <td>0.920</td>
    <td>0.392</td>
  </tr>
  <tr>
    <td>H<sub>o</sub></td>
    <td>0.999</td>
    <td>0.001</td>
    <td>0.903</td>
    <td>0.430</td>
  </tr>
</table>

Table 8: Effect of Hessian selections in ODLRI. Results are shown for Layer 10 Key Projection matrix of Llama2-7B when using ODLRI initialization. We compare normalized norm of \( \|X_o\| \) and \( \|X_r\| \) for LR and \( E_{LR} \) when using ODLRI initialization. L2-norm is denoted by \( \| \cdot \| \). And \( E_{LR} = W - LR \).

This section validates that ODLRI effectively captures and preserves salient weights by explicitly targeting outlier activations in \( X \). Unlike standard low-rank approximations that distribute capacity across all activations, ODLRI selectively focuses on outlier activations, ensuring that the low-rank component is structured to represent the most outlier-sensitive weights before quantization.

To demonstrate this, we compare two hessian formulations for guiding the low-rank approximation. The first uses the full activation matrix \( \mathbf{H} = \mathbf{X} \mathbf{X}^\top \), while the second employs only the top-\( k \) outlier activations, \( \mathbf{X}_o \), forming the restricted hessian matrix \( \mathbf{H}_o = \mathbf{X}_o \mathbf{X}_o^\top \). The effectiveness of each approach is assessed by evaluating the low-rank representation LR through the activation norm \( \|LRX_o\| \) compared to \( \|WX_o\| \).

Table 8 presents that using \( \mathbf{H}_o \) yields a significantly closer approximation of \( WX_o \) than using \( \mathbf{H} \), confirming that ODLRI’s outlier-driven initialization better preserves salient weights. Moreover, the approximation for non-salient weights (\( \mathbf{X}_r \)) remains stable, demonstrating that prioritizing activation outliers does not degrade overall representation.

C Additional Results

C.1 16-Bit LR on Llama2
Table 9 presents the results of zero-shot accuracy under the setting of 2-Bit Q and 16-Bit LR. Overall, the results show that the method incorporating ODLRI outperforms the baseline CALDERA approach.

C.2 Lower Rank, Extreme Compression on Llama2
To further evaluate the robustness of our method under more aggressive compression settings, we conduct additional experiments at significantly lower ranks, specifically \( r = 16 \) and \( r = 32 \). These settings simulate extreme compression scenarios, where the capacity of the low-rank residual becomes severely constrained.
We evaluate on the Llama2-7B model using the same setup as our main experiments: Q is quantized to 2 bits, and L, R to 4 bits. Table 10 reports perplexity and accuracy across a range of zero-shot evaluation benchmarks.
Despite the reduced rank and further limited representational capacity, our proposed ODLRI initialization continues to yield improvements over CALDERA. These findings confirm that ODLRI remains effective under severe rank constraints, preserving performance.

C.3 ODLRI on non-Llama Models and Alternative Quantizers.
Abnormal outliers have been observed in Llama-style LLMs (Yang et al., 2024; Yu et al., 2024), including the Llama family and Mistral, possibly due to their model architecture. To assess whether our outlier-aware method remains effective beyond this setting, we evaluate the robustness of ODLRI in two distinct scenarios: (1) non-Llama architectures
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Rank</th>
    <th colspan="6">Zero-Shot Accuracy ↑</th>
  </tr>
  <tr>
    <th>Wino</th>
    <th>RTE</th>
    <th>PiQA</th>
    <th>ArcE</th>
    <th>ArcC</th>
  </tr>
  <tr>
    <td rowspan="4">7B</td>
    <td>CALDERA</td>
    <td>64</td>
    <td>63.73</td>
    <td><b>55.59</b></td>
    <td>73.4</td>
    <td>61.93</td>
    <td>31.27</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td><b>65.75</b></td>
    <td>55.23</td>
    <td>72.91</td>
    <td><b>64.44</b></td>
    <td><b>31.74</b></td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>63.97</td>
    <td><b>59.03</b></td>
    <td>73.53</td>
    <td>64.58</td>
    <td>32.94</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>63.93</td>
    <td><b>59.03</b></td>
    <td><b>73.64</b></td>
    <td><b>64.86</b></td>
    <td><b>33.75</b></td>
  </tr>
  <tr>
    <td rowspan="2">CALDERA</td>
    <td>256</td>
    <td>66.1</td>
    <td>60.47</td>
    <td><b>74.45</b></td>
    <td>64.62</td>
    <td>34.00</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td>64.57</td>
    <td><b>61.46</b></td>
    <td>65.12</td>
    <td><b>71.58</b></td>
    <td><b>36.31</b></td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>0</td>
    <td></td>
    <td>61.7</td>
    <td>57.8</td>
    <td>69.6</td>
    <td>61.2</td>
    <td>29.9</td>
  </tr>
  <tr>
    <td rowspan="4">13B</td>
    <td>CALDERA</td>
    <td>64</td>
    <td>67.36</td>
    <td>58.84</td>
    <td>75.07</td>
    <td>68.11</td>
    <td><b>37.58</b></td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>64</td>
    <td><b>69.85</b></td>
    <td><b>67.51</b></td>
    <td><b>75.41</b></td>
    <td><b>69.91</b></td>
    <td>37.37</td>
  </tr>
  <tr>
    <td>CALDERA</td>
    <td>128</td>
    <td>69.85</td>
    <td>64.98</td>
    <td>75.9</td>
    <td><b>70.75</b></td>
    <td>38.82</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>128</td>
    <td>68.98</td>
    <td><b>65.83</b></td>
    <td><b>76.14</b></td>
    <td>69.59</td>
    <td><b>39.93</b></td>
  </tr>
  <tr>
    <td rowspan="2">CALDERA</td>
    <td>256</td>
    <td>67.12</td>
    <td>59.02</td>
    <td>76.24</td>
    <td>70.28</td>
    <td>39.08</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>256</td>
    <td><b>70.40</b></td>
    <td><b>67.15</b></td>
    <td><b>76.55</b></td>
    <td><b>71.63</b></td>
    <td><b>41.04</b></td>
  </tr>
  <tr>
    <td>QuIP#</td>
    <td>0</td>
    <td></td>
    <td>63.6</td>
    <td>54.5</td>
    <td>74.2</td>
    <td>68.7</td>
    <td>36.2</td>
  </tr>
  <tr>
    <td>Uncompressed (7B)</td>
    <td>—</td>
    <td></td>
    <td>67.3</td>
    <td>63.2</td>
    <td>78.5</td>
    <td>69.3</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>Uncompressed (13B)</td>
    <td>—</td>
    <td></td>
    <td>69.5</td>
    <td>61.7</td>
    <td>78.8</td>
    <td>73.2</td>
    <td>45.6</td>
  </tr>
</table>

Table 9: Comparison of our method with CALDERA by zero-shot accuracies (↑) of Llama2 models. Q is quantized 2-Bit and L, R are 16-Bit. Higher accuracy values are bolded.

<table>
  <tr>
    <th rowspan="2">Rank</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Avg Bits</th>
    <th colspan="2">PPL ↓</th>
    <th colspan="5">Zero-Shot Accuracy ↑</th>
  </tr>
  <tr>
    <th>Wiki2</th>
    <th>C4</th>
    <th>Wino</th>
    <th>RTE</th>
    <th>PiQA</th>
    <th>ArcE</th>
    <th>ArcC</th>
  </tr>
  <tr>
    <td rowspan="2">16</td>
    <td>CALDERA</td>
    <td>2.025</td>
    <td>7.88</td>
    <td>10.16</td>
    <td><b>62.64</b></td>
    <td>51.26</td>
    <td>72.09</td>
    <td><b>60.63</b></td>
    <td><b>31.48</b></td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>2.025</td>
    <td><b>7.79</b></td>
    <td><b>10.02</b></td>
    <td>61.8</td>
    <td><b>59.57</b></td>
    <td><b>72.36</b></td>
    <td>59.43</td>
    <td>30.46</td>
  </tr>
  <tr>
    <td rowspan="2">32</td>
    <td>CALDERA</td>
    <td>2.05</td>
    <td>7.85</td>
    <td>10.18</td>
    <td>62.9</td>
    <td>56.31</td>
    <td>71.7</td>
    <td>59.00</td>
    <td>29.52</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td>2.05</td>
    <td><b>7.47</b></td>
    <td><b>9.75</b></td>
    <td><b>65.03</b></td>
    <td><b>62.45</b></td>
    <td><b>72.52</b></td>
    <td><b>62.28</b></td>
    <td><b>31.65</b></td>
  </tr>
</table>

Table 10: Comparison of our method with CALDERA on zero-shot perplexities (↓) and accuracies (↑) of Llama2 models on extreme low bit setting. Q is quantized 2-Bit and L, R are quantized 4-Bit. Lower perplexity and higher accuracy values are bolded.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Llama2-7B</th>
    <th colspan="2">Gemma2-2B</th>
  </tr>
  <tr>
    <th>r = 32</th>
    <th>r = 64</th>
    <th>r = 32</th>
    <th>r = 64</th>
  </tr>
  <tr>
    <td>FP16</td>
    <td>8.71</td>
    <td></td>
    <td>13.08</td>
    <td></td>
  </tr>
  <tr>
    <td>MXINT-base</td>
    <td>10.62</td>
    <td>10.38</td>
    <td>18.72</td>
    <td>17.65</td>
  </tr>
  <tr>
    <td>+ODLRI</td>
    <td><b>10.57</b></td>
    <td><b>10.26</b></td>
    <td><b>18.52</b></td>
    <td><b>17.17</b></td>
  </tr>
</table>

Table 11: Comparison of OLDRI with MXINT-base on zero-shot perplexities (↓) of Llama2-7B and Gemma2-2B on low bit setting. Q is quantized 3-Bit and L,R are 16-Bit. Lower perplexity values are bolded.

and (2) alternative quantization methods. Specifically, we conduct experiments on both Gemma2-2B (Rivière et al., 2024) and Llama2-7B using the MXINT (Darvish Rouhani et al., 2023) quantizer. We replace QuIP# (Tseng et al., 2024) with MXINT (3-bit, block size 32) for both models.

We define MXINT-base as a baseline that follows the same quantize-then-approximate structure as CALDERA: it first applies MXINT quantization and then performs low-rank approximation using the same activation-aware SVD used in CALDERA, without any outlier-driven refinement.

Gemma2-2B serves to test whether ODLRI generalizes to architectures outside the Llama family. In contrast, Llama2-7B is included to isolate and examine the impact of changing the quantizer while keeping the model architecture fixed. For both models, we adopt the same activation-aware low-rank approximation method as CALDERA and perform 15 outer iterations alternating between quantization and low-rank refinement.

Perplexity is measured using lm-eval-harness, which may yield different values from those in the main text. Hence, we also report FP16 perplexities for reference. We note that the LR component is kept in 16-Bit precision for all experiments. As shown in Table 11, ODLRI consistently reduces perplexity compared to MXINT-base. These findings confirm that ODLRI remains effective across both non-Llama models and alternative quantiza-
C.4 Effect of LR Initialization in CALDERA

<table>
  <tr>
    <th rowspan="2">Weight Type</th>
    <th rowspan="2">Iteration</th>
    <th colspan="2">Initialization Method</th>
    <th colspan="2">LRApprox(W)</th>
  </tr>
  <tr>
    <th>0</th>
    <th>||QX||</th>
    <th>||LRX||</th>
    <th>||QX||</th>
    <th>||LRX||</th>
  </tr>
  <tr>
    <td>Key Proj.</td>
    <td>First</td>
    <td>0.999</td>
    <td>0.014</td>
    <td>0.158</td>
    <td>0.915</td>
    <td>0.961</td>
    <td>0.073</td>
    <td>0.401</td>
    <td>0.664</td>
  </tr>
  <tr>
    <td>Query Proj.</td>
    <td>First</td>
    <td>0.999</td>
    <td>0.014</td>
    <td>0.148</td>
    <td>0.924</td>
    <td>0.956</td>
    <td>0.073</td>
    <td>0.408</td>
    <td>0.657</td>
  </tr>
  <tr>
    <td>Value Proj.</td>
    <td>First</td>
    <td>0.993</td>
    <td>0.072</td>
    <td>0.378</td>
    <td>0.885</td>
    <td>0.970</td>
    <td>0.264</td>
    <td>0.622</td>
    <td>0.676</td>
  </tr>
  <tr>
    <td>O Proj.</td>
    <td>First</td>
    <td>0.993</td>
    <td>0.038</td>
    <td>0.373</td>
    <td>0.886</td>
    <td>0.958</td>
    <td>0.149</td>
    <td>0.495</td>
    <td>0.734</td>
  </tr>
  <tr>
    <td>Up Proj.</td>
    <td>First</td>
    <td>0.978</td>
    <td>0.064</td>
    <td>0.605</td>
    <td>0.766</td>
    <td>0.966</td>
    <td>0.198</td>
    <td>0.666</td>
    <td>0.684</td>
  </tr>
  <tr>
    <td>Gate Proj.</td>
    <td>First</td>
    <td>0.986</td>
    <td>0.048</td>
    <td>0.443</td>
    <td>0.824</td>
    <td>0.951</td>
    <td>0.165</td>
    <td>0.584</td>
    <td>0.626</td>
  </tr>
  <tr>
    <td>Down Proj.</td>
    <td>First</td>
    <td>1.000</td>
    <td>0.013</td>
    <td>0.071</td>
    <td>0.976</td>
    <td>0.996</td>
    <td>0.049</td>
    <td>0.181</td>
    <td>0.869</td>
  </tr>
</table>

Table 12: Effect of LR initialization strategies in CALDERA. Results are shown for Layer 1’s weight matrices from Llama2-7B over 15 iterations. We compare \(||QX||\) and \(||LRX||\), both normalized by \(||WX||\) (i.e., \(||QX||/||WX||\)) at first and last iterations. L2-norm is denoted by \(||\cdot||\).

<table>
  <tr>
    <th rowspan="2">Weight Type</th>
    <th rowspan="2">Iteration</th>
    <th colspan="2">Initialization Method</th>
    <th colspan="2">LRApprox(W)</th>
  </tr>
  <tr>
    <th>0</th>
    <th>||QX||</th>
    <th>||LRX||</th>
    <th>||QX||</th>
    <th>||LRX||</th>
  </tr>
  <tr>
    <td>Key Proj.</td>
    <td>First</td>
    <td>0.995</td>
    <td>0.040</td>
    <td>0.278</td>
    <td>0.869</td>
    <td>0.960</td>
    <td>0.106</td>
    <td>0.552</td>
    <td>0.575</td>
  </tr>
  <tr>
    <td>Query Proj.</td>
    <td>First</td>
    <td>0.992</td>
    <td>0.049</td>
    <td>0.324</td>
    <td>0.847</td>
    <td>0.960</td>
    <td>0.127</td>
    <td>0.549</td>
    <td>0.606</td>
  </tr>
  <tr>
    <td>Value Proj.</td>
    <td>First</td>
    <td>0.976</td>
    <td>0.094</td>
    <td>0.570</td>
    <td>0.792</td>
    <td>0.966</td>
    <td>0.220</td>
    <td>0.652</td>
    <td>0.722</td>
  </tr>
  <tr>
    <td>O Proj.</td>
    <td>First</td>
    <td>0.978</td>
    <td>0.102</td>
    <td>0.644</td>
    <td>0.733</td>
    <td>0.970</td>
    <td>0.265</td>
    <td>0.725</td>
    <td>0.680</td>
  </tr>
  <tr>
    <td>Up Proj.</td>
    <td>First</td>
    <td>0.978</td>
    <td>0.077</td>
    <td>0.621</td>
    <td>0.740</td>
    <td>0.978</td>
    <td>0.182</td>
    <td>0.676</td>
    <td>0.686</td>
  </tr>
  <tr>
    <td>Gate Proj.</td>
    <td>First</td>
    <td>0.987</td>
    <td>0.057</td>
    <td>0.475</td>
    <td>0.786</td>
    <td>0.975</td>
    <td>0.139</td>
    <td>0.606</td>
    <td>0.608</td>
  </tr>
  <tr>
    <td>Down Proj.</td>
    <td>First</td>
    <td>0.971</td>
    <td>0.090</td>
    <td>0.769</td>
    <td>0.591</td>
    <td>0.971</td>
    <td>0.208</td>
    <td>0.814</td>
    <td>0.552</td>
  </tr>
</table>

Table 13: Effect of LR initialization strategies in CALDERA. Results are shown for Layer 10’s weight matrices from Llama2-7B over 15 iterations. We compare \(||QX||\) and \(||LRX||\), both normalized by \(||WX||\) (i.e., \(||QX||/||WX||\)) at first and last iterations. L2-norm is denoted by \(||\cdot||\).

Table 12 and Table 13 present how different LR initialization strategies affect the final weight distributions in Layer 1 and Layer 10, respectively. In both layers, we observe that varying the initialization leads to significantly different outcomes across all weights. This finding reveals that joint optimization outcomes are highly sensitive to initialization choices, which ultimately determine whether quantization or matrix factorization dominates the final representation.

C.5 Effect of LR Initialization on Joint Optimization for Q+LR

Figure 4 examines the quantization scale across various layers, demonstrating that our method, ODLRI, effectively maintains optimal scale control throughout the network. The results indicate that ODLRI consistently outperforms baseline approaches in ensuring robust and stable quantization across diverse layers.

Similarly, Figure 5 presents the activation-aware error measured across multiple layers, confirming that ODLRI reliably minimizes activation-aware error in the preservation of intermediate activations. These findings underscore the efficacy of our approach in reducing activation-aware error, thereby enhancing overall model performance compared to baseline methods.
Figure 4: **Quantization Scale across different initialization strategies**. We present the quantization scale over 15 iterations, where both \( \mathbf{L} \) and \( \mathbf{R} \) are quantized to 4-Bit at rank 256. Subplots display results for the Key, Value, O, Gate, Up, and Down projection layers in Layer 0 and Layer 30 of Llama2-7B. ODLRI (red stars) consistently achieves the lowest quantization scale, highlighting its effectiveness in low-bit quantization.
Figure 5: Activation-aware Error across different initialization strategies. We present the normalized activation-aware error \( \| (W - Q - LR) X \|_F^2 / \| WX \|_F^2 \) over 15 iterations, where both L and R are quantized to 4-Bit at rank 256. Lower values indicate better preservation of activation information after quantization. Subplots display results for the Key, Value, O, Gate, Up, and Down projection layers in Layer 0 and Layer 30 of Llama2-7B. ODLRI (red stars) consistently shows the lowest error, demonstrating its effectiveness in reducing quantization error.

Reviewer paper summaries: This paper introduces a novel perspective on joint quantization and low-rank decomposition for LLM compression. The key idea is that the initialization of low-rank components is decisive in determining how the model balances between quantization and matrix factorization during fine-tuning. Based on this observation, the paper proposes a novel initialization to assign low-rank components the specific role of capturing activation-sensitive weights.
Summary of strengths: 1. The observation that initialization affects long-term role assignment in joint Q+LR optimization is novel and well-argued, challenging the assumption that iterative optimization "washes away" initialization effects.
2. The proposed method directly targets a key known issue in post-training quantization—activation outliers—and provides a solution by assigning those weights to the low-rank component at initialization time.
3. Experiments are comprehensive. The experiments span multiple model sizes (7B–70B), architectures (LLaMA2/3, Mistral), tasks (Wiki2, C4, zero-shot NLP tasks), and settings (4-bit, 16-bit, different ranks). Evaluation is done both with activation-aware proxy metrics (quantization scale, activation-aware error) and end-task metrics (perplexity and zero-shot accuracy).
Summary of weaknesses: 1. People have been aware of abnormal outliers in Llama-style LLMs [1][2], including Llama family and Mistral. These outliers could be caused by the Llama-style model architecture or training recipe. I wonder whether the proposed method can be generalized to other LLMs, such as Gemma, Phi-3, OLMo. It would be helpful if the authors can implement the method on one or more of the mentioned models.

[1] https://arxiv.org/abs/2405.14428
[2] https://arxiv.org/html/2411.07191v1#bib.bib32
Comments, suggestions and typos: The paper is well written.
Overall assessment scores: 4
Metareview score: 3
Metareview: This paper proposes a joint quantization and low-rank decomposition method for LLM compression. The key idea is to initialize a low-rank component to balance the model, maintaining quantization and matrix factorization. The method is evaluated on many datasets and shows its effectiveness.
Summary of reasons to publish: 1. The idea is interesting and novel. 
2. The proposed method shows effectiveness in empirical studies.