LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

## Abstract

Abstract The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed KV vectors. However, its memory consumption scales linearly with sequence length and batch size, posing a significant bottleneck in LLM deployment.
Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific. This paper introduces an orthogonal approach to KV cache compression.
We propose a low-rank approximation of KV weight matrices, allowing for plug-in integration with existing transformer-based LLMs without model retraining.
To effectively compress KV cache at the weight level, we adjust for layerwise sensitivity and introduce a progressive compression strategy, which is supported by our theoretical analysis on how compression errors accumulate in deep networks. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
Extensive experiments with LLaMA models ranging from 8B to 70B parameters across various tasks show that our approach significantly reduces the GPU memory footprint while maintaining performance.

## 1 Introduction

Autoregressive large language models (LLMs) such as GPT (Achiam et al., ), PaLM (Chowdhery et al., ), and LLaMA (Touvron et al., ), built upon transformer architectures (Vaswani et al., ), have shown remarkable capabilities across a wide range of tasks.
However, the attention mechanism underpinning those models poses significant challenges to the efficiency of their deployment, particularly the management of the Key-Value (KV) cache.
The KV cache is originally designed to accelerate the generation process by storing intermediate attention KV vectors, thus avoiding recomputation of shared prefixes for each autoregressively generated token.
Despite reducing computational overhead, the KV cache significantly increases memory footprints, as its size scales
linearly with both sequence length and batch size.
This drives the need for KV cache compression to enable cost-effective deployment of LLMs across various devices and platforms.

To address the overhead of the original attention mechanism, one prominent line of work aims to design more efficient attention variants, such as multi-query attention (MQA) (Shazeer, ) and group-query attention (GQA) (Ainslie et al., ), which inherently reduce the corresponding KV cache.
Nevertheless, those techniques typically require upcycling existing models. Without proper training, their direct application often results in degraded performance (Ribar et al., ; Ainslie et al., ; Liu et al., ), thereby making them unsuitable for deployment in resource-constrained environments. Recently, Liu et al. () design a multi-head latent attention (MLA) for efficient inference, utilizing low-rank key-value union compression to reduce KV cache. However, similar to MQA and GQA, MLA is also integrated during the model’s training cycle, thus not directly applicable to pre-trained LLMs.

In contrast, another line of work focuses on KV cache compression at test time, primarily achieved by dropping tokens while leaving the backbone model intact. Several works design the token eviction policy based on accumulated attention scores (Sheng et al., ; Zhang et al., ; Liu et al., ), or heuristics such as special tokens or and relative distance between tokens (Ge et al., )
However, these methods either ignore inter-layer dependencies or require attention pattern analysis, and the resulting eviction policy can be task-specific.

In this paper, we propose to compress KV cache from an orthogonal perspective, i.e., the KV weight matrices. As the KV weight matrices are typically characterized by low-rank properties, we perform a low-rank approximation to reduce their dimension and thus compress the resulting KV cache.
Recognizing that compressed KV caches inevitably introduce information loss to subsequent layers, and that sensitivity to input changes varies across layers, we introduce a
progressive compression strategy.
This approach is grounded in the calculation of cumulative condition numbers for KV weight matrices across different layers, reflecting their sensitivity and guiding the compression strategy.
Theoretically, we derive error bounds for both individual layer compression and error propagation through the network. These theoretical results reveal that errors introduced in earlier (shallower) layers are amplified more significantly than those in deeper layers, and informs our progressive compression strategy.

Our method is designed for straightforward implementation, requiring neither model profiling nor detailed inspection of the attention structure.
It can be directly applied to pre-trained LLMs by extracting weight matrices and leveraging their inherent properties to swiftly determine optimal layer-wise compression. This approach offers a practical and efficient solution for enhancing LLM inference performance in memory-constrained deployment scenarios, without the need for model retraining or complex eviction strategy composition.

We evaluate our method on 8B, 13B, and 70B LLaMA models that built upon multi-query attention or group-query attention.
Experiments across tasks such as commonsense reasoning, reading comprehension, text summarization, and mathematical reasoning, demonstrate that our approach can reduce substantial GPU memory footprint while maintaining minimal impact on performance.

## 2 Related Works

### 2.1 Attention Mechanism

Attention mechanisms in Transformer models have evolved to enhance efficiency and effectiveness (Vaswani et al., ). Multi-Query Attention (MQA) (Shazeer, ) reduces memory requirements during decoding, while Grouped-Query Attention (GQA) (Ainslie et al., ) balances efficiency and performance by sharing key and value heads among query groups. Recently, Liu et al. () introduced Multi-head Latent Attention (MLA), using low-rank key-value union compression to optimize inference. However, these approaches are typically integrated during model training, limiting their applicability to pre-trained LLMs.
Parallel research efforts have targeted inference efficiency improvements. For example, Pope et al. () developed multi-dimensional partitioning techniques, and de Jong et al. () optimized the Fusion-in-Decoder (FiD) approach (Izacard & Grave, ) for more efficient inference.
Holmes et al. () introduces SplitFuse which leverages dynamic prompt and generation decomposition and unification to further improve continuous batching and system throughput.
In this paper, we contribute to this line of research by improving inference efficiency through the compression of KV cache. Our approach leverages the low-rank property of the attention weight matrices, offering a plug-and-play method to reduce the memory footprint of LLMs during inference without requiring model retraining.

### 2.2 KV Cache Compression

As Large Language Models (LLMs) continue to grow in size and complexity, efficient management of their memory usage during inference has become a critical challenge.
Early efforts to compress token hidden states (Guan et al., ; Sun et al., ; Zhou et al., ) are limited to non-autoregressive models and require retraining, thus motivating research into pruning tokens in the KV cache of auto-regressive LLMs. For instance, Mu et al. () learns to compress prompts into a few special tokens to reduce memory pressure during caching, but this token prediction requires model retraining and could be an expensive overhead during inference.
Several methods design token eviction policies based on accumulated attention scores (Sheng et al., ; Zhang et al., ; Liu et al., ), or heuristics such as special tokens and relative distance between tokens (Ge et al., ). However, these approaches often overlook inter-layer dependencies, potentially resulting in task-specific eviction policies that may not generalize well across different applications.
In contrast to token-dropping methods, our study takes a different tack. We focus on compressing the KV cache from the perspective of weight matrix dimension reduction. Importantly, our progressive compression strategy carefully addresses the issue of error propagation across compressed layers, a consideration often ignored in previous methods.

A few studies have explored customized cache budgets across different layers in the context of token dropping, yet no definitive consensus has been reached on the most effective strategies. Zhang et al. () suggest increasing compression intensity in higher layers based on the assumption that these layers contain less critical information. Conversely, Liu et al. () argue that significant tokens exhibit greater variability at higher layers, thus larger caches are required to reduce cache misses. While these approaches demonstrate understanding of layer-specific requirements, they depend heavily on task-specific attention patterns.
Our approach diverges fundamentally by adopting an orthogonal perspective to compression, focusing on weight matrix dimension reduction rather than token eviction. This approach enables us to establish error propagation bounds across the network and to guide our progressive compression strategy effectively. It eliminates the need to analyze attention patterns for eviction policy design, simplifying implementation and enhancing general applicability across different LLMs.

Concurrently, Liu et al. () and Yu et al. () modify attention mechanisms to manage KV caches more efficiently during inference. While these methods align with our philosophy of altering attention dynamics, they require either pretraining adjustments or extensive model finetuning to accommodate the modified attention schemas, limiting their practicality in deployed systems. In contrast, our method requires no such training or fine-tuning, offering a plug-and-play solution that seamlessly integrates with pre-trained models to deliver efficient compression without compromising the model’s integrity or performance.

## 3 Preliminary: Attention Mechanism and KV Cache

Transformer-based language models use self-attention to weigh the importance of different tokens, thus allowing for the model to focus on different parts of the input sequence.
Given an input $X\in\mathbb{R}^{N\times D}$, where $N$ is the sequence length and $D$ is the dimensionality of each token’s embedding, we compute the Query ($Q$), Key ($K$), and Value ($V$) matrices by multiplying $X$ with their respective weight matrices: $Q=XW_{q},K=XW_{k},V=XW_{v}$.

Then the attention mechanism is as follows:

$$ $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{k}}} \right)V.$ (1) $$

Multi-head attention allows the model to jointly attend to information from different representation
subspaces at different positions

$$ $\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}^{1},\ldots,\text{head}^{h})W _{o},$ (2) $$

where

$$ $\text{head}^{i}=\text{Attention}(X(W^{i}_{q})^{T},X(W^{i}_{k})^{T},X(W^{i}_{v} )^{T}).$ (3) $$

Here, $W^{i}_{q}$, $W^{i}_{k}$, and $W^{i}_{v}$ are the weight matrices for the $i$-th attention head, and $W_{o}$ is the weight matrix for the output linear transformation.

In autoregressive transformers, the computation of attention scales quadratically (i.e., $\mathcal{O}(N^{2})$) with the sequence length $N$, as every token in the sequence computes interactions with every other token. Such scaling is impractical for very large inputs or real-time applications, where speed and efficiency are crucial.

To address this computational bottleneck, KV caches store the results of previous computations of the KV matrices. When processing subsequent tokens, the model can retrieve keys and values from the cache rather than recomputing them, thereby reducing the number of operations to a linear scale with respect to the sequence length. This method trades off increased memory usage for a reduction in computational overhead. The size of KV cache per layer is defined as below:

$$ $C_{k,v}=b\times N\times h\times d,$ (4) $$

where $b$ is the batch size, $N$ is the max sequence length in the batch, $h$ is the number of K/V head and $d$ is the head dimension.
This linear relationship between cache size and sequence length, as well as batch size, underscores the critical need for efficient compression methods.
As described, existing works that can reduce KV cache consumption either require expensive model training in upcycling stages or empirical token eviction policy design at test time. In the following section, we present a novel method for KV cache compression from the perspective of low-rank weight approximation.

## 4 Method

Figure: Figure 1: LoRC compresses KV-cache by decomposing the KV weight matrices in attention heads. The progressive compression strategy retains more dimension for KV weights in shallow layers and compresses the KV weights in deep layers more aggressively.
Refer to caption: extracted/5900565/figure/overview.png

We structure this section as follows. In Section 4.1, we detail the process of compressing the KV cache for a single layer using Singular Value Decomposition (SVD) on weight matrices. Section 4.2 introduces our progressive compression strategy, which determines adaptive compression dimensions for each layer.
Finally, Section 4.3 covers additional considerations for handling various attention mechanisms, and Section C addresses the implementation details specific to the rotary position embedding. Figure 1 presents an overview of our method, illustrating the low-rank approximation of the weight matrix and the progressive compression strategy across layers.

### 4.1 KV Cache Compression via Low-rank Approximation of Weight Matrices

Unlike previous approaches that focus on token-level eviction strategies or require model retraining, we propose a novel method that operates at the weight matrix level in the attention mechanism. This approach leverages the inherent low-rank properties of these matrices (as shown in Appendix B), allowing for significant compression without the need for complex token selection algorithms or time-consuming model tuning. By applying a low-rank approximation to the weight matrices, we effectively reduce the dimensionality of the KV cache while preserving the essential information flow through the network.

Key Matrix Compression:
Figure 1 presents how we implement SVD on the key weight matrices. Specifically, for the $i$-th head in the MHA attention, we decompose its key matrix $W^{i}_{k}\in\mathbb{R}^{D\times d}$ to:

$$ $\text{SVD}(W^{i}_{k})_{D\times d}=U_{D\times d_{c}}\Sigma_{d_{c}\times d_{c}}V ^{T}_{d_{c}\times d}=U_{D\times d_{c}}(\Sigma V^{T})_{d_{c}\times d}.$ (5) $$

For MHA, there are $h$ attention heads, then the decomposition becomes:

$$ $\text{SVD}(W^{H}_{k})_{D\times hd}=U_{D\times d_{c}}(\Sigma V^{T})_{d_{c} \times hd}=U_{D\times d_{c}}\left[\begin{array}[]{cccc}(A^{1})_{d_{c}\times d} &(A^{2})_{d_{c}\times d}&\cdots&(A^{h})_{d_{c}\times d}\end{array}\right],$ (6) $$

where $(A^{i})_{d_{c}\times d}$ is the $i$-th block in the matrix $(\Sigma V^{T})_{d_{c}\times hd}$.

We have now decomposed the key matrix $W^{i}_{k}$ to the multiplication of $U_{D\times d_{c}}$ and $(\Sigma V^{T})_{d_{c}\times hd}$. We will multiply $X$ with $(\Sigma V^{T})^{T}_{hd\times d_{c}}$ as the compressed key, which is stored in the KV cache. Through this implementation, we effectively update the size of key cache from $hd$ to $d_{c}$, where $d_{c}$ is smaller than $hd$, reducing the memory footprint while keeping the essential information intact.

For $U_{D\times d_{c}}$, we incorporate it to the query calculation by updating the original query matrix $W^{H}_{q}\in\mathbb{R}^{D\times hd}$ as follows:

$$ $W^{H}_{q^{\prime}}=(W^{H}_{q})_{D\times hd}U_{D\times d_{c}}.$ (7) $$

Note that the embedding dimension $D$ is equal to the product of the number of attention heads $h$ and the dimension per head $d$, i.e., $D=hd$. Consequently, the updated query matrix $W^{H}_{q^{\prime}}\in\mathbb{R}^{D\times d_{c}}$.

Value Matrix Compression:
The decomposition for the value matrix follows a similar structure to that of the key matrix, with the difference that we integrate its left singular vectors to the output matrix $W_{o}$. Specifically, the value matrix is decomposed as:

$$ $\text{SVD}(W^{H}_{v})_{D\times hd}=U_{D\times d_{c}}(\Sigma V^{T})_{d_{c} \times hd}=U_{D\times d_{c}}\left[\begin{array}[]{cccc}(B^{1})_{d_{c}\times d} &(B^{2})_{d_{c}\times d}&\cdots&(B^{h})_{d_{c}\times d}\end{array}\right]$ (8) $$

where $(B^{i})_{d_{c}\times d}$ is the $i$-th block in the matrix $(\Sigma V^{T})_{d_{c}\times hd}$. After multiplication with $X$, the dimension of the value cache shrinks from $hd$ to $d_{c}$, thus reducing memory consumption.

In contrast to the key matrix operation, we incorporate $U_{D\times d_{c}}$ to the output matrix. To achieve this, we update the output matrix $W_{o}\in\mathbb{R}^{D\times D}$ as follows:

$$ $W_{o^{\prime}}=(U^{\top})_{d_{c}\times D}(W_{o})_{D\times D},$ (9) $$

resulting in an updated output matrix $W_{o^{\prime}}\in\mathbb{R}^{d_{c}\times D}$.

Compression Ratio:
The compression strategy effectively reduces the dimensions from $N\times d\times h$ for both keys and values to $N\times d_{c}$, ensuring data integrity and minimizing overhead. This results in a layer compression ratio $\rho=\frac{d_{c}}{h\times d}$, which quantifies the extent of the reduction.

### 4.2 Progressive Compression Strategy

Figure: Algorithm 1 LoRC Algorithm

Having established low-rank approximation for compressing weight matrices, we now address its dynamic application across network layers. This approach is necessary due to the varying sensitivity of different layers, which significantly affects overall model efficacy and efficiency.

To tackle this challenge, we propose a progressive compression strategy for our low-rank approximation of KV weight matrices. Our intuition is that the compressed shallow layers could lead to cascading errors that propagate and amplify through the network. Therefore, we measure the layer sensitivity by the condition numbers of KV matrices to determine layer-wise compression dimensions. This approach accounts for each layer’s sensitivity to perturbations caused by previously compressed layers, ensuring output variations remain within acceptable ranges.
This progressive nature allows for more conservative compression in shallow layers and more aggressive compression in deeper layers, minimizing the risk of error accumulation throughout the network.
By carefully balancing compression across layers, we maintain model integrity while achieving significant memory savings.

#### Condition Number and Sensitivity Analysis

To ensure that the change in the output $\mathbf{b}_{l}=\mathbf{A}_{l}\mathbf{x}_{l}$ remains within a specified range when the input $\mathbf{x}_{l}$ changes due to compression in previous layers, we need to consider the sensitivity of the output to such changes. Given a weight matrix $\mathbf{A}_{l}$, its condition number plays a crucial role in determining the allowable change in $\mathbf{x}_{l}$. The condition number $\kappa(\mathbf{A}_{l})$ is defined as:

$$ $\kappa(\mathbf{A}_{l})=|\mathbf{A}_{l}|_{2}\cdot|\mathbf{A}_{l}^{-1}|_{2}= \frac{\sigma_{\text{max}}(\mathbf{A}_{l})}{\sigma_{\text{min}}(\mathbf{A}_{l})},$ (10) $$

where $\sigma_{\text{max}}(\mathbf{A}l)$ and $\sigma_{\text{min}}(\mathbf{A}_{l})$ are the largest and smallest singular values of $\mathbf{A}_{l}$, respectively.
To keep the relative change in the output $\mathbf{b}_{l}$ within a tolerance $\epsilon$, we utilize the standard definition of the condition number to relate it to the allowable relative change in the input $\mathbf{x}_{l}$:

$$ $\frac{|\Delta\mathbf{b}_{l}|_{2}}{|\mathbf{b}_{l}|_{2}}\leq\kappa(\mathbf{A}_{ l})\cdot\frac{|\Delta\mathbf{x}_{l}|_{2}}{|\mathbf{x}_{l}|_{2}}\leq\epsilon.$ (11) $$

Solving for the allowable relative change in $\mathbf{x}_{l}$, we obtain:
$\frac{|\Delta\mathbf{x}_{l}|_{2}}{|\mathbf{x}_{l}|_{2}}\leq\frac{\epsilon}{
\kappa(\mathbf{A}_{l})}$.
This inequality indicates that the acceptable change in the input $\mathbf{x}_{l}$ is inversely proportional to the condition number $\kappa(\mathbf{A}_{l})$ of the layer’s weight matrix. Layers with higher condition numbers are more sensitive to input perturbations, requiring smaller changes in $\mathbf{x}_{l}$ to maintain the output within the desired range.
Given the multi-layer structure of transformers, it is essential to consider not just the condition number of a single layer but the cumulative effect of condition numbers from all preceding layers. This cumulative measure gives a more holistic view of how perturbations might propagate and amplify as data passes through successive layers.

Cumulative Condition Number: To effectively manage this across the network, we calculate the cumulative condition number as an estimated layer sensitivity, which we then use to derive the compression dimension. For a model with $L$ layers, we calculate the cumulative condition number for each layer $l$ by multiplying the condition numbers of the current layer and all subsequent layers:

$$ $\tilde{\kappa}_{l}=\prod_{j=l}^{L}\kappa(W_{k}^{j})\cdot\kappa(W_{v}^{j}),$ (12) $$

where $W_{k}^{j}$ and $W_{v}^{j}$ denote the key and value weight matrices of the $j$-th layer, respectively. This cumulative condition number $\tilde{\kappa}_{l}$ reflects the total amplification of input perturbations from current layer to the final output layer, encompassing the effects of layers from $l$ to $L$.

Compression Dimension:
Based on the cumulative condition number, we then adjust the compression dimensions for each layer to balance the fidelity and compression rate. More sensitive layers (those with higher cumulative condition numbers) will have less aggressive compression to preserve information, whereas layers with lower sensitivity can be compressed more substantially without significantly affecting the overall network performance.
We compute the compressed dimension $d_{c}^{l}$ for each layer by scaling $\tilde{\kappa}_{l}$ using the following function:

$$ $d_{c}^{l}=d_{\max}\times\left[1-\left(\frac{\max_{i\in[1:L]}\log(\tilde{\kappa }_{i})-\log(\tilde{\kappa}_{l})}{\max_{i\in[1:L]}\log(\tilde{\kappa}_{i})-\min _{i\in[1:L]}\log(\tilde{\kappa}_{i})}\right)\times\left(1-\frac{d_{\min}}{d_{ \max}}\right)\right],$ (13) $$

where $d_{\max}$ is the maximum allowable compressed dimension, and $d_{\min}$ is the minimum one.
The logarithmic scale mitigates the effect of large variations in the cumulative condition numbers, providing a more balanced sensitivity metric across layers.
This equation ensures that layers with higher sensitivity (larger $\tilde{\kappa}_{l}$) retain more dimensions (larger $d_{l}$), while less sensitive layers can be compressed more aggressively.

### 4.3 Multi-head Attention and Group-query Attention

The above derivation in Section 4.1 holds for standard MHA, where the model dimension $D$ equals to the multiplication of number of head and head dimension $h\times d$. For GQA, the number of KV heads is reduced as shown in Table 3. To adapt such implementation, we can still follow the above procedure for cache compression. After fetching the key and value from cache, we just need to repeat them according to the number of the total attention heads.

## 5 Error Bounds for KV Cache Compression

In this section, we derive error bounds for our KV cache compression method, considering both individual layer errors and their propagation through a deep network. These theoretical results provide insights into how the matrix decomposition-based compression affects the network’s performance and guide the progressive compression strategy to balance model efficiency and performance.

### 5.1 Error Bound for Key/Value Matrix Approximation

###### Theorem 1

Let $W\in\mathbb{R}^{m\times n}$ be a weight matrix (either key or value), and let $\tilde{W}\in\mathbb{R}^{m\times n}$ be its rank-$k$ approximation obtained via truncated singular value decomposition (SVD). For any input vector $x\in\mathbb{R}^{n}$, the error introduced by the approximation is bounded by:

$$ $\|Wx-\tilde{W}x\|_{2}\leq\sigma_{k+1}\|x\|_{2},$ (14) $$

where $\sigma_{k+1}$ is the $(k+1)$-th singular value of $W$.

The proof is provided in Appendix A.1.
This theorem quantifies the error introduced at a single layer due to compressing the weight matrix. The bound indicates that the error is directly proportional to the $(k+1)$-th singular value of $W$ and the norm of the input vector $x$. Larger singular values correspond to directions of significant variance in the data, so truncating smaller singular values (which represent less significant features) minimizes the error introduced by compression.

###### Theorem 1

### 5.2 Single Layer Error Bound Including Nonlinearities

We now extend the analysis to include the effect of nonlinearities within a single layer. We derive an error bound that accounts for both the approximation of the weight matrix and the layer’s nonlinear activation function. For simplicity, we analyze the error introduced by compressing each weight matrix (key or value) individually.

###### Theorem 2

Consider a single layer applying a linear transformation $W$ followed by a nonlinearity $\phi$ with Lipschitz constant $L_{\phi}$. Let $\tilde{W}$ be the compressed version of $W$ obtained via truncated SVD with rank $k$. For any input vector $x\in\mathbb{R}^{n}$, the error at the output of the layer is bounded by:

$$ $\left|\phi(Wx)-\phi(\tilde{W}x)\right|\leq L_{\phi}\sigma_{k+1}\|x\|_{2}.$ (15) $$

The proof is straightforward by using Theorem 1 and the Lipschitz property of $\phi$, we present it as the base case in the proof of Theorem 3, which is detailed in Appendix A.2.

This theorem shows that the error introduced by the compressed weight matrix propagates through the nonlinearity, scaled by the Lipschitz constant of the activation function.
While considering both matrices simultaneously complicates the bounds due to their interactions within the attention mechanism, it is still feasible to derive combined error bounds because the attention mechanism allows us to mathematically bound these interactions. The total error due to simultaneous compression can be bounded by the sum of their individual approximation errors, scaled by a constant. However, for simplicity and clarity in the following derivation, we use the simplified version that considers each matrix individually.

###### Theorem 2

### 5.3 Error Propagation Bound

###### Theorem 3

Consider an $L$-layer network where each layer $i$ applies a linear transformation $W_{i}$ followed by a nonlinearity $\phi$ with Lipschitz constant $L_{\phi}$. Let $\tilde{W}_{i}$ be the compressed version of $W_{i}$ obtained via truncated SVD with rank $k_{i}$. The error at the output of the network is bounded by:

$$ $\|x_{L}-\tilde{x}_{L}\|_{2}\leq\sum_{i=1}^{L}\left(\sigma_{k_{i}+1}^{(i)}L_{ \phi}^{L-i}\prod_{j=i+1}^{L}\|W_{j}\|_{2}\right),$ (16) $$

where $x_{L}$ and $\tilde{x}_{L}$ are the outputs of the original and compressed networks, respectively; $\sigma_{k_{i}+1}^{(i)}$ is the $(k_{i}+1)$-th singular value of $W_{i}$; $\|W_{j}\|_{2}$ denotes the spectral norm of $W_{j}$; and $L_{\phi}$ is the Lipschitz constant of the activation function $\phi$.

We detail the proof in Appendix A.2.
Until now, we have established an upper bound on the cumulative error at the network’s output due to compression of weight matrices across multiple layers. It is important to note that the nonlinearities characterized by the Lipschitz constant
$L_{\phi}$ represent a simplification.
In practice, transformer models like LLaMA incorporate complex nonlinear components, so the exact error propagation may deviate from this simplified bound due to intricate nonlinearities.
Despite these complexities, the theorem still offers insights into how compression errors may accumulate in deep networks.
Specifically, it reveals that errors introduced in earlier (shallower) layers are amplified more significantly than those in deeper layers because they pass through more subsequent transformations and nonlinearities.

This understanding supports our design of a progressive compression strategy, where we compress shallow layers less aggressively than deeper ones. By preserving more information in the early layers (i.e., retaining more singular values), we minimize the initial errors that could be significantly amplified throughout the network. This approach helps maintain overall model performance while still achieving substantial compression in deeper layers, where the impact on the final output is less pronounced due to reduced error amplification.

###### Theorem 3

## 6 Experiment

### 6.1 Models

We conduct experiments using two attention mechanisms, Multi-Head Attention (MHA) (Vaswani et al., ) and Graph Query Attention (GQA) (Ainslie et al., ), across three models: LLaMA-2-13B, LLaMA-3-Instruct-8B, and LLaMA-3-Instruct-70B. The LLaMA-2 family incorporates the MHA mechanism, while the LLaMA-3 family is based on the GQA framework. We list the model specifications in Table  3. Note that for the models based on MHA, the number of KV heads is equal to the number of attention heads, so the weight matrices of KV are square matrices. The models based on GQA use an intermediate number of key-value heads to group the query heads, with an adjustment on the shape of KV weight matrices.

### 6.2 Implementation Details

In practice, we set thresholds to exclude compression on layers with high cumulative condition numbers: 30 for LLaMA-3-Instruct-8B, and 90 for LLaMA-2-13B and LLaMA-3-Instruct-70B. The $d_{max}$ equals to the original head dimension, while $d_{min}$ varies based on the target compression ratio. For baseline methods, we have the same refrained layers while applying the uniform compression ratios across compressed layers instead of using a progressive compression strategy.

### 6.3 Dataset

We follow Touvron et al. () to evaluate our methods on the following tasks:
BoolQ (Clark et al., ) for reading comprehension, XSum (Narayan et al., ) for text summarization. Openbook QA (Mihaylov et al., ) for commonsense reasoning, and GSM8K (Cobbe et al., ) for mathematical reasoning.
We use ROUGE score (Lin, ) as the evaluation metric for XSum and accuracy for the other tasks.
We report 2-shot results for LLaMA-2 models on BoolQ, and 0-shot results for other settings.

### 6.4 Main Results

Figure: Figure 2: Performance of KV cache compression on LLaMA models. LoRC compresses the KV weights with a progressive strategy, while the baselines compress each layer with the same ratio. The horizontal dashed line indicates the performance with a full-cache model.
Refer to caption: x1.png

Figure 2 presents our main results on four datasets with different KV cache budgets. Compared to the full-cache model, LoRC achieves on-par performance with a significant compression ratio, and the performance degradation is still nearly negligible with a $60\%$ compression ratio on most datasets. When slightly compressed, LoRC could even enhance model performance in some cases. Note that our method requires no model training or model profiling, the only efforts are SVD on weight matrices which requires minimal computational cost compared to the LLM inference. Such plug-and-play merits make our method easily integrable in resource-constrained environments, enabling efficient model deployment with limited KV cache budgets.

In Figure 2, one interesting observation is that in some cases the model with a compressed KV cache leads to better performance.
Particularly, on the GSM8K dataset, performing KV cache compression leads to more than 10% performance improvement.
This phenomenon aligns with findings reported in the literature (Ge et al., ).
Also, similar effects have been documented in the context of improving reasoning by applying low-rank decomposition on the MLP layers (Sharma et al., ).
We believe this phenomenon demonstrates the feasibility of conducting task-specific profiling for better performance, or adapting our proposed method in model finetuning.

### 6.5 Single Layer Profiling

Figure: Figure 3: Single-layer compression results. This experiment uses LLaMA-3-Instruct-8B on the OpenBookQA dataset.
Refer to caption: x3.png

To investigate the impact of compression at different layers, we conduct experiments on single-layer compression as shown in Fig. 3. We use LLaMA-3-Instruct-8B on OpenBook QA for this experiment. The original dimension of the KV head is 1024, and we select compression dimensions from $[256,384,512]$ to compress each single layer while keeping all other layers untouched.

Figure 3 shows clear layer-specific variability, indicating that some layers are more susceptible to compression than others, particularly in the shallow layers.
It is observed that the deep layers (i.e., layers 15–31 of the 32-layer LLaMA-3-Instruct 8B model), despite the reduction in dimensions, maintain performance closely approaching the full KV Cache baseline. This suggests that these layers can sustain robust performance even when subjected to significant parameter reduction. This finding supports our progressive compression strategy for optimizing model efficiency without significantly compromising the model’s effectiveness.

### 6.6 Curse of shallow layers

**Table 1: Performance comparison between compression on shallow layers and deep layers on OpenBookQA. For our progressive compression strategy, we report the performance at the $60\%$ overall compression ratio. For layer-0 compression and shallow blocks compression, we use a $50\%$ layer compression ratio within the chosen strategy. Hence, the overall compression ratio is $98.44\%$ for the layer-0 compression, and $93.75\%$ for the shallow blocks compression.**
| Model | Baseline | Ours | Layer 0 | Shallow Blocks (1/8) |
| --- | --- | --- | --- | --- |
| LLaMA-2-13b | 76.6 | 77.4 ($\uparrow 0.8$) | 77.2 ($\uparrow 0.6$) | 74.8 ($\downarrow 1.8$) |
| LLaMA-3-Instruct-8b | 78.0 | 77.4 ($\downarrow 0.6$) | 67.2 ($\downarrow 10.8$) | 61.4 ($\downarrow 16.6$) |
| LLaMA-3-Instruct-70b | 91.2 | 91.2 ($\uparrow 0.0$) | 84.2 ($\downarrow 7.0$) | 23.2 ($\downarrow 68.0$) |

To validate the intuition of the progressive compression strategy that the noise caused by shallow compressed layers will be amplified more after propagation, we compare it to compressing the first layer and the shallow blocks (i.e., the first 1/8 layers in a model) on 3 LLaMA models.

Table 1 shows how the compressed shallow layers impact the model performance, taking the baseline full-cache model and our method as reference. The results indicate that compressing only the first layer can lead to a performance decline, with reductions ranging from minimal to moderate. For instance, the LLaMA-3-70B gives a $7.0\%$ decrease, while the LLaMA-3-Instruct-8b shows a more substantial drop of $10.8\%$.
When compressing the shallow blocks, the impact is more pronounced. The LLaMA-3-Instruct-8B suffers a $16.6\%$ reduction. Notably, the LLaMA-3-Instruct-70b model shows a drastic $68.0\%$ decline, highlighting a significant sensitivity to shallow layer compression.

These findings underscore the importance of careful layer selection in compression strategies and validate the effectiveness of our progressive compression method, as the choice of layer to compress can have a substantial impact on model performance, particularly in larger or more complex models.

### 6.7 Memory footprint reduction analysis

**Table 2: Summary of Model Sizes, KV cache usage and performance drop. Experiments were conducted with a batch size of 64 and a sequence length of 2048 for all models.**
| Model | KV Cache | Average Performance Drop |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Full | dim | dim_c | Ours | Compression Ratio |  |
| LLaMA-2-13B | 50G | 5120 | 2048 | 27.5G | 55% | 0.47% |
| LLaMA-3-8B | 8G | 1024 | 512 | 4.8G | 60% | 0.92% |
| LLaMA-3-70B | 20G | 1024 | 512 | 11G | 55% | 0.22% |

We report the memory footprint reduction in Table 2. By controlling the performance drop averaged on the four tasks less than 1%, we can achieve a considerable compression ratio from 55%-60%. For the LLaMA-3 models in which the GQA has already been employed to save the KV cache, we further achieve a significant compression ratio. Note that we have excluded the GSM8k results for the performance drop calculation for a fair comparison.

## 7 Conclusions

In conclusion, we proposed LoRC, a novel approach to KV cache compression that capitalizes on the inherent low-rank properties of weight matrices. Our method employs a progressive layer-wise compression strategy, implementing a post-hoc low-rank approximation to circumvent the complexities and limitations associated with token-level eviction strategies and model retraining. Moreover, we provide theoretical analysis, deriving error bounds for layer compression and error propagation in deep networks, supporting our design of progressive compression strategy.
This theoretically grounded and universally applicable approach preserves model integrity and performance across diverse tasks, attention mechanisms, and model scales.
Our comprehensive experimental results demonstrate that LoRC significantly reduces GPU memory requirements while minimally impacting performance. This approach offers a robust and efficient solution of KV cache compression, without requiring attention pattern analysis or model tuning.

## Appendix A Detailed Proofs

### A.1 Proof of Theorem 1

Proof.

Let $W=U\Sigma V^{\top}$ be the full SVD of $W$, where $U\in\mathbb{R}^{m\times m}$ and $V\in\mathbb{R}^{n\times n}$ are orthogonal matrices, and $\Sigma=\operatorname{diag}(\sigma_{1},\ldots,\sigma_{n})$ with singular values $\sigma_{1}\geq\sigma_{2}\geq\cdots\geq\sigma_{n}\geq 0$.

The rank-$k$ approximation $\tilde{W}$ is given by:

$$ $\tilde{W}=U_{k}\Sigma_{k}V_{k}^{\top},$ $$

where $U_{k}$, $\Sigma_{k}$, and $V_{k}$ are truncated versions of $U$, $\Sigma$, and $V$, respectively, keeping only the first $k$ singular values and corresponding vectors.

We have:

$$ $\displaystyle\|Wx-\tilde{W}x\|_{2}$ $\displaystyle=\|(W-\tilde{W})x\|_{2}$ $\displaystyle=\|U(\Sigma-\Sigma_{k})V^{\top}x\|_{2}$ $\displaystyle=\|(\Sigma-\Sigma_{k})V^{\top}x\|_{2},\quad\text{since }U\text{ is orthogonal}$ $\displaystyle=\|\operatorname{diag}(0,\ldots,0,\sigma_{k+1},\ldots,\sigma_{n}) V^{\top}x\|_{2}$ $\displaystyle\leq\sigma_{k+1}\|V^{\top}x\|_{2}$ $\displaystyle=\sigma_{k+1}\|x\|_{2},\quad\text{since }V\text{ is orthogonal}.$ $$

$\square$

### A.2 Proof of Theorem 3

Proof.

Let $x_{i}$ and $\tilde{x}_{i}$ denote the outputs of the $i$-th layer in the original and compressed networks, respectively. We prove by induction that:

$$ $\|x_{i}-\tilde{x}_{i}\|_{2}\leq\sum_{s=1}^{i}\left(\sigma_{k_{s}+1}^{(s)}L_{ \phi}^{i-s}\prod_{j=s+1}^{i}\|W_{j}\|_{2}\right).$ (17) $$

Base Case ($i=1$).

Using Theorem 1 and the Lipschitz property of $\phi$:

$$ $\displaystyle\|x_{1}-\tilde{x}_{1}\|_{2}$ $\displaystyle=\|\phi(W_{1}x_{0})-\phi(\tilde{W}_{1}x_{0})\|_{2}$ $\displaystyle\leq L_{\phi}\|W_{1}x_{0}-\tilde{W}_{1}x_{0}\|_{2}$ $\displaystyle\leq L_{\phi}\sigma_{k_{1}+1}^{(1)}\|x_{0}\|_{2}.$ $$

Inductive Step.

Assume the inductive bound holds for layer $i-1$. For layer $i$:

$$ $\displaystyle\|x_{i}-\tilde{x}_{i}\|_{2}$ $\displaystyle=\|\phi(W_{i}x_{i-1})-\phi(\tilde{W}_{i}\tilde{x}_{i-1})\|_{2}$ $\displaystyle\leq L_{\phi}\|W_{i}x_{i-1}-\tilde{W}_{i}\tilde{x}_{i-1}\|_{2}$ $\displaystyle\leq L_{\phi}\left(\|W_{i}(x_{i-1}-\tilde{x}_{i-1})\|_{2}+\|(W_{i }-\tilde{W}_{i})\tilde{x}_{i-1}\|_{2}\right)$ $\displaystyle\leq L_{\phi}\left(\|W_{i}\|_{2}\|x_{i-1}-\tilde{x}_{i-1}\|_{2}+ \sigma_{k_{i}+1}^{(i)}\|\tilde{x}_{i-1}\|_{2}\right).$ $$

We can bound $\|\tilde{x}_{i-1}\|_{2}$ using the triangle inequality:

$$ $\|\tilde{x}_{i-1}\|_{2}\leq\|x_{i-1}\|_{2}+\|x_{i-1}-\tilde{x}_{i-1}\|_{2}.$ $$

Assuming that $\|x_{i-1}\|_{2}$ is bounded (which is reasonable in practice due to normalization techniques), and applying the inductive hypothesis, we can express $\|x_{i}-\tilde{x}_{i}\|_{2}$ in terms of the accumulated errors up to layer $i$.

By recursively applying this inequality and summing over all layers, we obtain the bound stated in Theorem 3.

$\square$

### A.3 Note on Activation Functions and the Lipschitz Constant

It is important to note that Theorem 3 assumes the activation function $\phi$ has a Lipschitz constant $L_{\phi}$, which reflects how much the function can amplify differences in its input. For activation functions like ReLU, which are 1-Lipschitz, the error bound simplifies and indicates minimal error amplification through the activation layers.

However, the LLaMA model family uses activation functions such as SwiGLU and GELU, whose derivatives can exceed 1, making them not 1-Lipschitz. For networks employing such activation functions, the error propagation bound in Theorem 2 is adjusted by incorporating a Lipschitz constant $L_{\phi}$, which may be greater than 1. This adjustment accounts for the potential additional error amplification introduced by the activation functions.

## Appendix B Reconstruction Error of Matrix SVD

In our approach, we conduct layer-wise weight matrix decomposition and reconstruction. In this section, we show that these matrices are low-rank and therefore can be reconstructed with low-dimension matrices, resulting in negligible reconstruction error. This suggests that instead of designing complex eviction policies at the token level, we can focus on the weight matrix level to develop a KV cache compression method. This approach eliminates the need to scrutinize attention patterns to determine which tokens should be dropped.

Figure: Figure 4: Layerwise relative reconstruction errors. $wk_{err}$ and $wv_{err}$ denote the relative difference between the original key/value matrices and their corresponding low-rank approximations measured using the Frobinus norm. The compression ratio is computed as $r={d_{c}\over N_{h}\times d_{h}}$, where $N_{h}$ is the number of attention heads and $d_{h},d_{c}$ is the original and compressed hidden dimensions respectively.
Refer to caption: x4.png

We present the relative reconstruction error in Figure 4, which is computed using the Frobenius norm. For a matrix $M$, the Frobenius norm is defined as:

$$ $\|M\|_{F}=\sqrt{\sum_{i=1}^{m}\sum_{j=1}^{n}|m_{ij}|^{2}}.$ (18) $$

The relative reconstruction error $\varepsilon$ is calculated as:

$$ $\varepsilon=\frac{\|M-\hat{M}\|_{F}}{\|M\|_{F}}$ (19) $$

where $M$ is the original matrix and $\hat{M}$ is the reconstructed matrix obtained through truncated SVD.

This approach enables us to quantify the accuracy of our low-rank approximation for each matrix. It is important to note that although Figure 4 demonstrates that reconstruction errors are similar across all layers, with shallow layers exhibiting even lower errors, this does not imply that we can directly compress shallow layers aggressively or compress all layers uniformly. In fact, compression errors propagate and amplify throughout the network as we illustrated in Section 4.2. To this end, we propose the progressive compression strategy and it is theoretically and empirically effective in minimizing the overall error accumulation.

## Appendix C Adjusted Position Embedding

Su et al. () propose a rotary position embedding (RoPE) and it has been used in most recent LLMs.
Applying RoPE to self-attention gives

$$ $q_{m}^{T}k_{n}=(R_{\Theta,m}^{d}W_{q}^{T}x_{m})^{T}(R_{\Theta,n}^{d}W_{k}^{T}x _{n})=x^{T}W_{q}R_{\Theta,n-m}^{d}W_{k}^{T}x_{n},$ (20) $$

where $\Theta$ is a pre-defined rotary matrix, $m$ and $n$ denotes the token position. In practice, the rotation matrix $R_{\Theta,n-m}^{d}$ is decomposed as $(R_{\Theta,m}^{d})^{T}$ and $R_{\Theta,n}^{d}$ to rotate the query and key separately, and the KV cache stores the rotated keys.
To ensure that our compressed keys are compatible with the rotary operation, we adjust the position embedding pipeline. Specifically, we store the compressed keys $X(\Sigma V^{\top})^{\top}_{D\times d_{c}}$ in cache, while incorporating the rotation and key projection into the query computation to streamline the process.

## Appendix D Computational details

For all experiments except those involving the LLaMA-3-70B model, we utilize a single node equipped with 4 A100 GPUs. For the LLaMA-3-70B model, we employ a node with 8 V100 GPUs.

The calculation of SVD is efficient based on the Numpy library. For LLaMA-3-Instruct-70B, the largest model used in our experiments, the all-layer (80 layers in total) SVD takes only 40 seconds.

**Table 3: Model Architectures.**
| Model | Attention | Layers | Heads | KV Heads | Head Dimension | Model Dimension | Weight Shape |
| --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-2-13B | MHA | 40 | 40 | 40 | 128 | 5120 | 5120 $\times$ 5120 |
| LLaMA-3-Instruct-8B | GQA | 32 | 32 | 8 | 128 | 4096 | 4096 $\times$ 1024 |
| LLaMA-3-Instruct-70B | GQA | 80 | 64 | 8 | 128 | 8192 | 8192 $\times$ 1024 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: The primary concern is related to the weaknesses above. If these concerns are adequately answered, I am willing to consider increasing the score to 6.
soundness: 3
strengths: 1. The process of calculating sensitivity and condition number for each layer in LoRC is novel.

2. Mathematically robust proofs support the proposed method.

3. Compared to a uniform compression rate, the proposed progressive compression strategy achieves higher accuracy at the same compression rate.
summary: The paper introduces a low-rank compression approach (LORC) for large language models (LLMs) to efficiently manage the Key-Value (KV) cache, crucial for faster inference but memory-intensive. Traditional KV compression strategies either require model retraining, limiting practical deployment. LORC tackles this with a low-rank approximation of KV matrices, integrating progressively across layers to minimize error propagation. Experiments demonstrate substantial memory savings with minimal performance loss, providing a scalable, plug-and-play solution adaptable to pre-trained models without needing task-specific tuning.
weaknesses: **1. Latency and Throughput Experiments**

   The primary purpose of reducing KV Cache size is to decrease memory usage during inference, which ideally should lower latency or increase throughput. However, the LoRC method may introduce additional computational overhead due to:

- The need to recover the compressed KV cache, which requires extra computations.
- Frequent reading and writing of the large restored KV cache per layer for attention calculations, potentially impacting latency or throughput.
- The time required to compute sensitivity and perform SVD.

Please provide experimental results showing LoRC’s latency and throughput, including the time required for sensitivity calculations and SVD processing.

**2. Comparison with Other KV Cache Compression Methods**
   - The paper would be strengthened by a comparative analysis with other existing KV cache compression methods, including:
      - KV Cache eviction [1]
      - KV Cache quantization [2]
      - SVD-LLM [3]

Please compare LoRC's accuracy and latency at the same compression rates against these methods to highlight its unique benefits and trade-offs.

**3. (minor) Different trends in sensitivity by compression between training and inference.**

 Section 6.6 analyzes that compression in earlier layers negatively impacts inference due to the amplification of input-induced errors through forward propagation. In contrast, it is known that during training, compression in later layers tends to negatively impact accuracy [4-6]. Although the results are opposite, I think the author’s argument is reasonable as input errors amplify forward during forward propagation while input gradient errors amplify backward during backpropagation. Reflecting this point in Section 6.6 or the appendix could further strengthen the argument of the paper.

[1] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, Neurips, 2023.
[2] ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification, arxiv, 2024.
[3] SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression, arixv, 2024.
[4] GACT: Activation Compressed Training for Generic Network Architectures, ICML, 2022.
[5] ALAM: Averaged Low-Precision Activation for Memory-Efficient Training of Transformer Models, ICLR, 2024.
[6] DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation, Neurips, 2024.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How do you handle RoPE?
2. How much of this system is implemented and what speedups are you able to attain?
soundness: 3
strengths: The progressive compression strategy effectively minimizes accuracy segregation with clear proof to demonstrate the error propagation under certain assumptions.
summary: This proposes a method to reduce the memory demands of large language models (LLMs) by compressing KV caches using low-rank matrix decomposition on weights. LoRC designs a progressive compression strategy that adjusts compression rates across layers based on sensitivity to error propagation, thus providing better preservation of model performance.

This paper effectively designs a method for determining specific compression rates for each layer. However, it lacks critical measurements regarding practical acceleration and potential overheads in deployment. Without this information, it’s challenging to assess its suitability for real-world applications.
weaknesses: 1. Limited Comparison with Related Work: The paper only compares LoRC to uniform compression, omitting relevant methods that leverage perplexity evaluation [1] or Hessian-based scoring [2] for deciding layer-wise compression rate for performing low-rank decomposition.
2. Lack of Measurements on Latency or Throughputs improvements: While LoRC demonstrates memory savings, it does not discuss potential gains in speed or latency—key metrics for assessing the real-world efficiency of compression methods.
3. Unclear Handling of Rotary Position Embedding (RoPE): The paper needs more clarity on the compatibility with RoPE. The author provides a brief section in Appendix C; the detailed workflow seems ambiguous, particularly on whether it must be reapplied at each step and what overhead this entails. Further explanation is crucial for models that rely on RoPE.

[1] ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
[2] Palu: Compressing KV-Cache with Low-Rank Projection

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In the previous work on SVD for LLM compression, directly performing SVD on the weight matrix without updating the weight matrix results in a huge performance loss. Can the author explain why there is no performance loss in this work?

2. Could the authors explain why using the Cumulative Condition Number as a criterion for rank selection is reasonable?

3. The progressive compression across layers seems questionable. Previous works on automatic rank pruning (e.g., ASVD, FWSVD) do not indicate that rank should decrease with increasing layer depth. Moreover, the final layers of a model are often crucial and should not have their rank excessively reduced. While the paper presents single-layer analysis, it does not consider inter-layer dependencies. Could the authors compare the Progressive Compression Strategy with other automatic rank pruning approaches to establish its validity or superiority?

4. Could the authors conduct experiments on more models (beyond Llama) and compare with other KV cache compression methods?
soundness: 4
strengths: 1. The core idea of the paper is insightful, as it proposes compressing the KV cache by performing low-rank decomposition on the KV values and absorbing them into the Q and O matrices.

2. Theoretical analysis and proofs are sufficient, as the paper derives upper and lower bounds for the error through mathematical formulations.

3. The paper demonstrates effective compression results, achieving high compression ratios of the KV cache with minimal impact on performance.

4. The writing is clear, making the paper easy to understand.
summary: This paper focuses on compressing the KV cache in long-context large language models. It cleverly utilizes Singular Value Decomposition (SVD) to decompose the KV cache and employs the Q and O matrices to absorb the decomposition, thereby reducing the required cache. The writing is clear, the proposed method is highly innovative, and experiments demonstrate the effectiveness of compressing the KV cache.
weaknesses: 1. The author directly performs SVD decomposition and reconstruction on the K and V matrices. However, in previous SVD for LLM compression work, this original method of decomposing the weight matrix without re-training it usually leads to a large performance loss (as mentioned in Pufferfish, ASVD). The paper does not clearly explain why the performance loss is small under the same approach.

2. The paper does not prove the superiority of the proposed Progressive Compression Strategy. The increase in compression ratio with layer depth does not align with findings in existing work on SVD for LLM compression (e.g., ASVD, SVD-LLM, FWSVD).

3. The rationale for using the cumulative condition number as a compression metric for each layer is not clearly demonstrated.

4. The experiments are insufficient, as the paper lacks comparisons with previous KV cache compression methods and does not evaluate models other than Llama.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see the above.
soundness: 3
strengths: 1. The authors provide a theoretical analysis of error bounds for KV cache compression to examine a progressive compression strategy. The study indicates that compressing shallow layers less aggressively than deeper layers helps preserve model performance.

2. The authors propose a KV cache compression method using low-rank approximation of KV weight matrices for transformer-based (LLMs) without requiring model retraining.

3. The paper is well-organized and the motivation is clear.
summary: This paper introduces LoRC, a novel approach to KV cache compression that leverages the inherent low-rank properties of weight matrices. It employs a progressive layer-wise compression strategy to approximate the KV weight matrices without requiring model retraining. Experimental results demonstrate that LoRC significantly reduces GPU memory usage with minimal to no performance degradation.
weaknesses: 1. By using low-rank approximation, the computational overhead should also be reduced compared to dense computation. In the experiments, I only found comparisons of compression ratio and performance. Could you provide details on your implementation for computing the attention layer? Did you use pure PyTorch? Additionally, have you tested the speedup compared to the full KV cache method?

2. The authors claims that the proposed methods enhances LLM performance in memory-constrained deployment scenarios. Are there any experiments to support it? For example, can this method be deployed on consumer GPUs with limited memory (24 GB or less) where the full KV cache model cannot fit? are there models that cannot be deployed on consumer GPUs but can be deployed using the proposed method?

Meta-review: The paper introduces LoRC (Low-Rank Compression), a method for compressing the Key-Value (KV) cache in large language models (LLMs) using low-rank matrix decomposition of the KV projection matrices. Key contributions include:

- A progressive compression strategy that varies compression rates across layers based on layer sensitivity to error propagation.

- Theoretical analysis of error bounds to justify the compression strategy and its impact on preserving model performance.

- Demonstration of practical applicability: LoRC is compatible with existing transformer-based LLMs and does not require retraining.

The reviewers praised the novelty of the idea and particularly the associated theoretical analysis of error bounds that guided the design of the progressive compression strategy. In addition, the method seamlessly integrates with existing LLMs without retraining which makes it practical for real-world deployment. The paper is also well organized and presented. 

A major concern comes from reviewer 9w6H who raised questions on the lack of a kernel implementation for handling ROPE embeddings, which has important practical implications. The authors acknowledged this limitation but argued that their contribution lies in the modeling change and the theoretical justifications. However, upon reviewing the paper I found that the modeling contributions may have important shortcomings.

- The paper overlooked the closely related works ASVD, FWSVD, SVD-LLM (as raised by the reviewers), and LoSparse (see below). Many of these works provided improved versions of SVD based model compression, and their absence diminishes the contextual grounding of the proposed approach.

- Sensitivity analysis underpinning the key theoretical results of the paper lacks discussion of related studies, particularly those in network robustness [2, 3]. In particular, the analysis of the paper only provides the naive upper bound but more precise calculations of the Lipschitze constant [4, 5] exist in the literature. Such existing results may also benefit the method presented in this paper. 

- There is no clear justification for the cumulative condition number in Eq. (12), which is the key factor in the proposed algorithm. While this definition only considers key and value matrices, the propagation of error is affected by all the other layers as well hence should be taken into account? 

Given these shortcomings, I recommend a weak rejection of the current version. Meanwhile, I strongly recommend the authors to revise their manuscript and further develop their method based on reviewer comments. I believe the paper has significant potential to make a valuable contribution to the field. 

[1] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

[2] Estimating Neural Network Robustness via Lipschitz Constant and Architecture Sensitivity

[3] Formalizing Generalization and Adversarial Robustness of Neural Networks to Weight Perturbations

[4] Lipschitz regularity of deep neural networks: analysis and efficient estimation

[5] LIPSCHITZ CONSTANT ESTIMATION OF NEURAL NETWORKS VIA SPARSE POLYNOMIAL OPTIMIZATION

Decision: Reject