Palu: KV-Cache Compression with Low-Rank Projection
Authors: Chi-Chih Chang
Wei-Cheng Lin
Chien-Yu Lin
Chong-Yan Chen
Yu-Fang Hu
Pei-Shuo Wang
Ning-Chi Huang
Luis Ceze
Mohamed S. Abdelfattah
Kai-Chiang Wu
Venue: ICLR 2025 Poster (2025)
Keywords: KV-Cache
Low-Rank Compression
Large Language Model
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a novel  low-rank KV-Cache compression method to reduce the memory footprint and accelerate decoding efficiency

Abstract: Post-training KV-Cache compression methods typically either sample a subset of effectual tokens or quantize the data into lower numerical bit width. However, these methods cannot exploit redundancy in the hidden dimension of the KV tenors. This paper presents a hidden dimension compression approach called Palu, a KV-Cache compression framework that utilizes low-rank projection to reduce inference-time LLM memory usage. Palu decomposes the linear layers into low-rank matrices, caches compressed intermediate states, and reconstructs the full keys and values on the fly. To improve accuracy, compression rate, and efficiency, Palu further encompasses (1) a medium-grained low-rank decomposition scheme, (2) an efficient rank search algorithm, (3) low-rank-aware quantization compatibility enhancements, and (4) an optimized GPU kernel with matrix fusion. Extensive experiments with popular LLMs show that Palu compresses KV-Cache by 50% while maintaining strong accuracy and delivering up to 1.89× speedup on the RoPE-based attention module. When combined with quantization, Palu’s
inherent quantization-friendly design yields small to negligible extra accuracy degradation while saving additional memory than quantization-only methods and achieving up to 2.91× speedup for the RoPE-based attention. Moreover, it maintains comparable or even better accuracy (up to 1.19 lower perplexity) compared to quantization-only methods. These results demonstrate Palu’s superior capability to effectively address the efficiency and memory challenges of LLM inference posed by KV-Cache. Our code is publicly available at: https://github.com/shadowpa0327/Palu.

Published as a conference paper at ICLR 2025
PALU: KV-CACHE COMPRESSION WITH LOW-RANK PROJECTION
Chi-Chih Chang 1.++ Wei-Cheng Lin'
Yu Lin
Chong-Yan Chen' Yu-Fang Hu'
Pei-Shuo Wang' Ning-Chi Huang'
Luis Ceze² Mohamed Abdelfattah
Kai-Chiang Wu
National Yang Ming Chiao Tung University University of Washington Comnell University
ABSTRACT
Post-training KV-Cache compression methods typically either sample a subset of effectual tokens quantize the data into lower numerical bit width. However, these methods cannot exploit redundancy in the hidden dimension of the KV tensors. This paper presents a hidden dimension compression approach called Palu, a KV-Cache compression framework that utilizes low-rank projection to reduce inference-time LLM memory usage. Palu decomposes the linear layers into low-rank matrices, caches compressed intermediate states, and reconstructs the full keys and values on the fly. To improve accuracy, compression rate, and efficiency, Palu further encompasses (1) a medium-graine -rained low-rank decompo- sition scheme, (2) an efficient rank search algorithm, (3) low-rank-aware quan- tization compatibility enhancements, and (4) optimized GPU kernels with oper- ators fusion. Extensive experiments with popular LLMs show that Palu com- presses KV-Cache by 50%, while maintaining strong accuracy and delivering up 1.89 speedup on the RoPF--based attention module. When combined with quan- tization, Palu's inherent extra accuracy degradation, while saving additional memory than quantization- only methods and achieving up to 2.91 x speedup for the RoPE-based attention. Moreover, it maintains comparable or even better accuracy (up to 1.19 lower perplexity) compared to quantization-only methods. These results demonstrate Palu's s superior capability to effectively address the efficiency and memory chal- lenges of LLM inference posed by KV-Cache, Our code publicly available at: https:/ / gi thub. com httmm//hhhooo002//a.llll
conntizaaiio-frrennll design yields small negligible
INTRODUCTION
Large language models (LLMs) have propelled Al into new applications and capabilities, providing high-level intelligence that previous machine learning (ML) models could not achieve. To speed up inference, caching the Key-Value states (K (V-Cachee) in memory is simple yet effective technique. However, the size of the KV-Cache can grow rapidly, straining memory capacity and bandwidth especially with long context lengths [u 2024); further, the memory-bounded nature of the decoding stage limits inference speed when loading KV-Cache data (Gholami al.) (024)). Therefore, KV- Cache compression has become a central research topic for running L.MMs efficiently.
Although emerging attention mechanisms such Multi-Query Attention (MQA) (Shazeer 2019 Sroup-Query Attention (GQA) (Ainslie et al. 2023) and Multi-head Latent Attention (MLA) (DeepSeek-Al et al., 2024) can reduce KV-Cache size. it either requires model pre-training or has a significant impact on model's accuracy when converting from traditional Multi-Head Attention (MHA) (Cen et al.] 2024). In contrast, post-training KV-Cache compression techniques offer an alternative approach to advance efficiency for existing models. Among various KV-Cache com- pression methods, quantization (Liu al.) [2024b] Hooper al. (0244) and token eviction (Zhang et al. 2024 Xiao et al. 2024) stand out as effective strategies to reduce the memory footprint of KV-Cache.
Quantization methods aim to reduce the bit-width used to represent each piece of data, while token eviction techniques focus on retaining a partial set of KV-Cache. However, both methods neglect the
" Equull contribution

---

Published as a conference paper at ICLR 2025
hidden dimensions of the KV-Cache, where substantial redundancy often resides. To capitalize on this untapped potential, we introduce Palu, a post-training KV-Cache compression framework that leverages low-rank projection to reduce the hidden dimension of KV tensors, offering an additional and orthogonal compression dimension to existing quantization and token eviction methods.
A naive way utilize low-rank projection for compressing the KV-Cache is by directly mapping cached matrices into low-rank space (Jolliffe & Cadima 2016) Zhao et al. (024). However, this approach imposes an unacceptably heavy overhead of computing the decomposition matrices during runtime. To avoid this, Palu statically decomposes the Key and Value-projection weight matrices
and caches the latent representations of the low-rank decomposition (see Fig. IT. This innovative design enables Palu to reduce memory while mitigating the runtime overhead of KV-Cache low-rank decomposition.
In designing an effective decomposition strat- egy for attention modules with multiple attention heads, we observed clear trade-off between accu- racy and reconstruction overhead. Decomposing the projection matrices across all attention heads together improves accuracy by preserving global information, but this approach significantly in- creases reconstruction costs. On the other hand, decomposing each head separately reduces recon- struction overhead but leads to higher loss in ac- curacy. To address this, Palu introduces a medium- grained, group-head low-rank decomposition that strikes a balance between accuracy and reconstruc- tion efficiency.
Original KV
X
W
Decompose (Offline)
H
B
Cache instead of
Figure 1: Palu's low-rank projection method for KV-Cache reduction. A weight matrix W of linear projection is decomposed into two low-rank matrices. Input X is down-projectedd to latent representation H, which is cached. Y can be reconstructed from H using the up- projection matrix B.
For LLMs, each linear projection module has a dif- ferent sensitivity to compression (Sharma et al.) 2023 Yuan et al., (023). To exploit the sensitiv- ity and improve accuracy, we design an efficient rank search algorithm based on Fisher information (Ly et al. 2017: Liu et al. 2021). Our algorithm automatically assigns higher rank for important matrices and lower ranks for less critical ones, boosting accuracy at the same overall KV-Cache compression rate.
In addition to its low-rank decomposition, Palu is compatible with quantization techniques. We found that low-rank decomposition can introduce severe outliers in the latent representation, which significantly hinders accurate low-bit quantization. Although the Hadamard transformation has been shown to be effective for outlier elimination in recent studies (Tssng et al., 2024| Ashkboos et al. 2024b. Liu et al. 2024a; Chiang et al. 2024), its integration often introduces computational over- head during runtime. However, Palu's s inherent matrix pair structure makes it highly compatible with this technique, allowing the transformation matrices to be seamlessly fused into the forward and backward matrices, effectively mitigating outliers without impacting runtime efficiency.
We evaluate Palu on widely used LLMs and benchmarks. Our experiments demonstrate that Palu maintains strong zero-shot accuracy and perplexity with up to 50% low-rank compression. More- over, when combining low-rank compression with quantization, Palu achieves an impressive over 91.25 % compression (11.4x reduction) and yields significantly lower perplexity of 1.19 than KVQuant (Hooper et al.] 2024), a state-of-the-ar KV-Cache quantization method, which only achieves an 87.5% compression rate.
For latency evaluation, under a 50% KV-Cache compression rate without quantization, Palu demon- strates up 1.89x and 2.2x speedup for RoPE-based and non-RoPE attention modules. When integrated with quantization. Palu achieves up to 2.91 x and 6.17x acceleration on RoPE-based and Ron-RoPE attention, respectively. These results underscore Palu's ability to significantly reduce KV-Cache memory footprint while boosting inference efficiency for LLMs.
Our key contributions include:
Palu, a new post-training KV-Cache compression framework that caches low-rani latent repre- sentations of Key and Value states.
2

---

Published as a conference paper at ICLR 2025
Group-head low-rank decomposition (G-LRD), optimization for balancing accuracy and re- construction efficiency.
An automated rank search algorithm for adaptively assigning ranks to each decomposed matrix, given target compression rate.
A co-designed quantization compatibility optimization that eliminates low-rank-induced outliers and imposes zero runtime overhead.
BACKGROUND
2.1 MULTI-HEAD ATTENTION MECHANISM
The multi-head attention (MHA) mechanism (Vaswani et al. 2017) is a core component of the transformer architecture. Given new input token K", an MHA with n heads projects the input into multiple queries, keys, and values using weight matrices W7, W and W., respectively, for each head i, as shown by
q: = 8 k; = V, =
Here, k; and V; represent the key and value at time step for head i. We can then compute the attention score for each head and the corresponding attention output as
g.K.
Pt.
Softmax
a. V.
where K; and V, denote the concatenation of current and all previous keys and values corresponding to the i-th head. The final MHA output is obtained by concatenating the outputs of all heads and then applying the out-projection layer Wa. as shown by
h
MHA(x)
Σ a. W: = Σ (....)W..
(3)
where W E Rdd xd represents the submatrices of the out-projection matrix for each head i.
2.2 SINGULAR VALUE DECOMPOSITION (SVD)
SVD (Jolliffe & Cadima/ 2016) is commonly used technique for computing the low-rank approx- imation for a given matrix. SVD decomposes a given matrix W E Rmxxn into three matrices: W UΣVT. Here, U and V are orthogonal matrices containing the left and right singular vectors, respectively. The matrix Σ is a diagonal matrix that consists of singular values. After decomposition, the low-rank approximation of W can be described
W AB, A= U,VE, B B= B=
(4)
where A E Rexr E Σ, Re Σ, is diagonal matrix containing the largest singular values, and U,.. V are corresponding singular vectors truncated from U and VT. This truncation and subsequent matrix formation let us approximate matrix W with two low-rank matrices A and B. thereby reducing the storage by mr-rr
12
3 THE PALU FRAMEWORK
3.1
COMPRESSING THE KV-CACHE VIA LOW-RANK PROJECTION
na've approach to compress the KV-Cache with low-rank projection to apply SVD directly on the KV-Cache and store the top-r singular vectors. However, this approach poses significant computational overhead during runtime that makes impractical for deployments (see Appendix H
To apply low-rank projection more efficiently than directly decomposing the KV-Cache during run- time, Palu uses SVD decompose the Key and Value projection matrices. This approach is based on the observation that low-rank decomposition rewrites the linear projection layer from y = xW into y xAB.
3

---

Published as a conference paper at ICLR 2025
Absorb
W
RoPE
W
(GEW))
Offline Fusion
Brae (Paned)
Output
Input
Score
Softmax
(GEMV)
Low-Rank
Reconstruct on-the-fly
Projection
Original (Not cached)
(Cached)
Projection
(Cached]
Figure 2: Palu uses low-rank decomposition (W = AB) to project the key (or value) lower- dimensional latent representation (h), thereby reducing the size of the KV-Cache. The original key (K,) is reconstructed on-the-fly with B and B is fused into W avoid reconstruction overhead. The fusion also reduces the computational burden for output projection.
Here, A E Rddr is the low-rank projection matrix, and B E Rrxd derived by SVD. The forward process first down-pun nown--rojects the input token Rd into low- dimensional latent space h E R' and then up up-projects it back to the original space:
x is the reconstruction matrix
h = Ax,
Bh
(5)
This two-step process lets Palu (1) store the lower dimension latent representation instead of the origin key and value states, and (2) reconstruct them during decoding.
3.1.1 INTEGRATION WITH THE ATTENTION MECHANISM AND OFFLINE MATRIX FUSION
We now describe how Palu decomposes the key and value linear layers for the attention mechanism. For each attention head Palu applies SVD and maps the key-projection matrix W and value- projection matrix WT into A. and A
Based on the formula of attention output in Eq. Palu absorbs the reconstruction matrix B into the output projection matrix Wi offline:
a W (p.V. = (, (BH,
(6)
Such fusion lets Palu skip the explicit reconstruction of the full value vectors, reduce the number of matrix multiplications, and improve efficiency. A similar approach applics for calculating attention scores. Matrix B can be fused into the query projection matrix W offline, shown by
q. k.KT = q. (HH x. (H -
(HŞ)*.
Here, B rxd, and W dxd,, the fused matrix (W. has size Rxx,, This fusion boosts computational efficiency by reducing the matrix dimension during attention score calculation.
3.1.2 COMPABILITY WITH POSITIONAL EMBEDDING
Recent LLMs, such as the Llama family, apply Rotary Positional Embedding (i.e., RoPE (Su et al. 2021 onto the Query and Key states prior their multiplication.
The non-linear nature of these positional embeddings prevents the matrix fusion of attention scores, as outlined in Eq. 7 To address this, Palu dynamically reconstructs the keys from latent repre- sentations on the fly. Specifically. Palu employs custom GPU kernel that efficiently integrates key reconstruction, RoPE application, and subsequent Query-Key multiplication into a single fused operation. By transferring only the low-rank latent representations and performing reconstruction directly within GPU shared memory. By doing so, Palu substantially reduces the off-chip mem- ory footprint, optimizing the memory-bound L.M decoding Yuan et al. (2024) process through
4

---

Published as a conference paper at ICLR 2025
memory-computation trade-off. Detailed implementation specifics of this kernel are provided in Appendix 4.1
Note that for some positional embedding methods, such as ALiBi (Press et al. 2022), positional embedding is not directly applied to the Key states. Consequently, the fusion described in Eq. remains valid. For these non-RoPE attention modules, Palu achieves greater speedup compared to RoPE-based attention, as their reconstruction can be avoided with matrix fusion.
3.2 DECOMPOSITION GRANULARITY
3.2.1 MULTI-HEAD LOW-RANK DECOMPOSITION
We name the per-head decomposition scheme in Sec. 3.1.1 as multi-head low-rank decomposition (M-LRD). We found M-LRD often causes a non-negligible accuracy degradation (discussed further in Sec. (.2)) possibly because SVD fails to capture the common information shared across heads. Therefore, alternative approaches needed to preserve model accuracy.
3.2.2 JOINT-HEAD LOW-RANK DECOMPOSITION
An alternative approach is to jointly decompose weight matrices for all heads. By considering the combined weight matrix W joint [W1, W, W.. E (dd we can perform a single low-rank decomposition W joint x Ajuin where Ajoint E [Rdd and Bjuint E RR'm mii x (d). We call this scheme joint-eoointheer joint-head low-rank decomposition (J-LRD).
J-LRD has the advantage of preserving the common principal components shared among different heads. This occurs because SVD is particularly effective at capturing the dominant components when applied to larger, combined matrix, resulting in more accurate approximation.
For J-LRD, the joint latent representation shared among all heads can be computed with hjsint x.joiin- During decoding, the original states for each head can be reconstructed via
,y,...,, ynl hjoin Bjoint-
Despite better-preserving model accuracy, J-LRD introduces significant compulational and memory overhead during decoding. Specifically, the total number of floating point operations (FLOPs) to reconstruct the Key or Value state of all heads now becomes L Tjjoitt dy n. Assuming the same size as the total low-rank latent representations (i.e., Tjoint = > x== ri), the total reconstruction cost n times higher than M-LRD, whose total FLOPs is L F3 dn n.. When considering the matrix fusion in Sec. 3.1.1 the fused matrix of J-LRD has size of rjoint" d n,, which is also times larger
than M-LRD, leading to substantial higher memory consumption.
Decomposition Granularity
Reconstruction
Matrix Fusion
A. Multi-head
Mutsoly
Bu
Ty
WWW.WWMM
Welepenseee.a.. Haad
Bed Accuracy X
Low Computational Cost
Law Weightt Memory
B. Group-head
Ha.
Fax
Greeped
Good Acpurecy S
Medium Computational Cost
Medium Weight Numory
C. Joint-head
Tree
Apas
Hyse
Yymr
All AalHeadd
Good Accursty
Pale
High Computational Cost
X
Recenchaatios Hatio
High Weight Memory X
Figure 3: Performing decomposition at different granularities. Jointly decomposing multiple heads can achieve higher accuracy. Assuming the same total size of the latent representations (i.e., 4-ri 2 Tg = Tjoin), the FLOPs for reconstruction overhead in joint-head decomposition schemes are times larger than those in ones.
5

---

Published as a conference paper at ICLR 2025
3.2.3 CROUP-HEAD LOW-RANK DECOMPOSITION
To balance the trade-off between accuracy and reconstruction cost, we propose group-head low- rank decomposition (G-LRD). G-LRD decompases the matrices for group of heads together. With combined weight matrices, it captures shared information within each group while limiting compu- tational overhead and preserving accuracy.
To illustrate the G-LRD process, consider the weight matrices for a group of s heads, W
[W,,1-. W where W g; E Rdx(da=s), We low-rank decompose W 0 s Ag, Ba where Ag, E id and gg, [ddxx( The latent representation shared among attention heads in the same group can be computed as hg, = xUg, During decoding, the original key or value for each head can be reconstructed via
yy.1... hg Bg
The FLOPs for reconstructing the keys and values for all heads in G-LRD is L. -l -n. Comparing the cost to J-LRD and assuming the same total rank size (rn fg Tjoins), G-LRD reduces the reconstruction cost by ng. Similarly, G-LRD also reduces the fused matrix size by ng. To sum up, G-LRD offers middle ground between computation overhead and approximation accuracy. We Please refer to Appendix [for for further discussions
illustrate M-LRD, J-LRD and G-LRD Fig. 3 on the costs of different decomposition granularities.
3.3 AUTOMATIC RANK ALLOCATION
To allocate ideal rank size to the decomposition target, it is crucial to accurately estimate the importance of the target matrix (e.g., grouped weights). In Palu, we identify Fisher information (Ly et al. 2017 7 Liu et al. 2021) as accurate approximator since it can quantify the amount of information for each parameter. We then employ the sum of Fisher information to estimate the importance of the weight matrix of each linear layer (Abdelfattah al. 2021).
Assuming that the compression sensitivity proportional to Fisher information, we determine the rank for each weight matrix by computing the ratio of its Fisher information the total Fisher information across all decomposition targets. We use this ratio to allocate the compression rate (i.e., rank level r), ensuring that more important layers retain higher rank levels. For detailed ablation study on our Automatic Rank Allocation, please refer to Appendix F3
3.4 QUANTIZATION COMPATIBILITY
Hard quantize
Easy to quantize
Bannhat Wile
We integrate quantization into Palu compress the KV-Cache further. We ob- serve that low-rank compressed latent rep- resentations have severe outliers, which limit quantization applicability in Palu. Unlike natural outliers described in previ- ous KV-Cache quantization literature (Liu) et al., 2024b; Hooper et al.) (2244) these outliers are induced by SVD-based low- rank factorization.
02
5.1
Asto
1500
120 200 Gue
200
1300
200
100k
(a) Key
(b) Low Key (with Hadamard)
Figure 4: Activation distribution of the low-rank key caches the 4th Llama-2 attention layer.
Fig. (a) shows the distribution of low-rank compressed key states from a layer of Llama-2 with G-LRD. Repeating outlier patterns appear at the beginning of each decomposed group because SVD arranges larger eigenvalues the initial rows or columns, resulting rapidly descending values in the latent representation. This pattern stretches the data distribution and hurts guantization accuracy.
Inspired by recent LLM quantization literature (Ashkboos et al. [2024b) Tseng et al. 2024), we apply the Hadmaard transform (WHT, Fino & Algazi) to eliminate outliers (Fig. 4 (b)), enabling
high quantization accuracy. However, this transformation introduces an extra matrix multiplication with associated runtime overhead. Unlike earlier methods (Ashkboos et al. 2024b) that must apply online WHT when quantizing KV-Cache, we optimize this process by integrating the Hadamard matrix into low-rank decomposed weights with additional compute overhead, described by

---

Published as a conference paper at ICLR 2025
W ~ AB (AR) = AB,
(8)
where R is the Hadamard matrix. This optimization allows Palu to integrate the proposed low-rank compression technique with low-bit quantization. Our experiments show that, on top of the low-rank compression, our quantization method only negligibly increases perplexity, even at extreme levels such as 3-bil or 2-bil with simple per-token quantization scheme (see Sec. 4.3)
4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Models and Tasks. We evaluate Palu on four LLM families, Llama-2 ( Touvron et al., 2023). Llama-3 (Dubey et al. 2024), Mistral (Jiang et al. 2023) and LongChat (Li et al. 2023). For accu- racy evaluation, we mcasure perplexity on the WikiText- (Merity ct al. 2016) and C4 (Raffel et al.) EEvalluaionn-Hrree
2020) datasets and use six common sense tasks. We also evaluate long context accuracy on 16 tasks LongBench (Bai et al. 2023). Unless specification, we refer to baseline model with non-compressed KV-Cache. See Appendix for further details on the dataset and settings.
(Gao al. (023) to measure zero-shot accuracy
Compression Settings. We implemented Palu based on the Huggingface library (Wolf et al., 2020 Decomposition of the Key and Value projection layers was performed using the truncation- aware SVD method proposed by SVD-LLM (Wang et al.] (024). Unless otherwise specified, Palu's 's results are G-LRD with group size (gs-4), with equal rank size for each group. To caleu- late Fisher information in rank searching, we used 2048 random samples from Wikitext-2, each with a sequence length of 1024. For quantization integration in Palu, we use a simple per-token, asymmetric integer quantization. For evaluation quantization results,
compare Palu to ad- vanced KV-Cache quantization methods, including Atom (Zhao et al.] 2023), KVQaunt (Hooper al. 2024), and KIVI (Liu et al. 2024b). Refer to Sec. 5 for a brief summary of these methods.
GPU Kernels Implementation. We implemented customized kernel for attention score with reconstruction in Triton (Tillet et al. 2019) (See Appendix(B) (). For quantization integration, we im- plemented kernels in CUDA for attention output and non-RoPE attention score, where matrix fusion can be applied (refer to Sec. [3I.] and Fig. 2). Our low-precision kernel fuses the dequantization process and the follow-up multiplication with low-rank compressed keys or values, enabling effi- cient processing compare to the non-compressed baseline and KIVI (Liu et al. 2024b), which we use their official code in our experiments.
quantized latent KV-Cache. When evaluating speedup with quantization, we
4.2 RESULTS WITH DIFFERENT DECOMPOSITION GRANULARITY
We evaluate perplexity and zero-shot accuracy of Palu with 50% low-rank compression rate using M-LRD, G-LRD, and J-LRD on Llama2-7B and -lama3-8B-Innnn I--ntruttt, and present the results in Table
Table 1: Perplexity and zero-shot accuracy of Palu at 50% compression rate.
Perplexity J. Wiki2
Zem Shot Accuracy (%% ? PIQA
Model
Method
C4
OBQA
Hella
ARC-e
ARC-e
Wino
Avg.
Baseline
5.47
7.26
44.20
76.00
78.07
76.30
46.42
69.30
65.05
Llama2-7B
J-LRD
5.62
7.75
45.40
75.57
77.48
75.97
45.31
69.22
64.82
G-LRD
6.01
9.82
43.60
73.39
76.33
73.02
42.57
66.77
62.61
M-LRD
6.75
12.01
39.60
65.35
74.76
67.17
35.24
64.64
$7.79
Baseline
8.28
13.01
43.20
75.80
78.62
81.61
56.83
71.90
67.99
Llama3-888 Inst
J-LRD
9.12
15.90
43.40
73.20
76.50
79.63
$1.96
72.45
66.19
G-LRD
10.11
17.87
42.60
70.36
76.06
76.30
48.99
72.38
64.45
M-LRD
12.38
23.02
38.80
63.04
73.67
69.78
42.58
62.51
58.40
Perplexity Evaluation. As Table 1 shows, for the Llama2-7B model, Palu's M-LRD method fails
maintain low perplexity a 50% compression rate. In contrast, despite having a high recompu- tation cost, J-LRD significantly outperforms M-LRD and achieves a 5.62 perplexity on WikiText-2.

---

Published as a conference paper at ICLR 2025
For G-LRD, which still maintains low computation cost, yields 6.01 perplexity Wikitext-2, showing a great balance between model accuracy and compression overheads. The same trend is observed in the Llama-3-8B model as well. More results 1-ama-2-13B can be found in Appendix
Zero-shot Evaluation Results. Similar to the perplexity evaluation, the J-LRD method demon- strates the best performance for the zero-shot accuracy on Llama-2-7B, with only a 0.23% average accuracy degradation. M-LRD method results in the lowest average performance, with 7.26% drop in accuracy compared to the baseline. In comparison, G-LRD only has a 2.4% average ac- curacy decline, offering sweet spot between model accuracy and compression overheads again.
Table 2: Quantization perplexity and KV-
4.3 RESULTS OF QUANTIZATION INTEGRATION
Cache size for Llama2-7B on WikiText- For perplexity, sequence length is 4096. KV-Cache size is demonstrated for 128K
Table 2 showcases the impact of quantization on per- plexity and KV-Cache size when combined with Palu. With 3-bit quantization, Palu incurs only slight 0.08 and 0.23 perplexity increase at 30% and 50% low- rank compression rate. These demonstrate minimal accuracy trade-off for significant compression gains compared to the 16-bit baseline. Notably, 2-bit quantization, Palu decisively outperforms the state- of-the-art KVQuant method, reducing perplexity by 1.19 and 0.54, while further slashing memory us- age by 30% and 50%. These results establish Palu with quantization as superior KV-Cache compression method.
sequence length.
KV-Cache Size (GB)
Comp.
Method
Bit
PPL
Rate
Baseline
Palu-30%
16
5.25
44.8
30%
Palu-50%
16
5.63
32.0
50%
Atom
6.15
12.6
80.32%
KVQuant
5.35
12.0
81.25%
Palu-30%
5.33
86.87
Palu
3
5.77
6.0
90.639
Atom
117.88
86.56%
KYQuant
6.95
8.0
87 50%
Palu-30%
5.76
91,25 %
Pulu-50%
6.41
4.0
93.75% %
4.4 EVALUATION ON LONG CONTEXT DATASETS
Table 3: Experiment Results LongBench: The average bit widths represent the total storage cost per element in the compressed KV-Cache, including the overhead of quantization parameters. These values are calculated for each approach, assuming context length of 10K.
Model
Method
Avg. Bits
Comp. Ratio
Multi- QA
Single- QA
Summa- rization
Few-Shot
Code
Synthetic
Avg.
Baseline
16
1.00x
29.63
36.43
66.71
54.16
42.54
305u-30%
16
1 1.43x
29.83
36.52
27.48
65.70
55.16
37.92
41.55
Mistral- 7B v0.2
Palu-50%
16
2.00x
26.92
26.01
64.04
44.54
16.88
36.23
KTV1-2
3.16
5.05x
28.81
35.07
27.60
66.45
54.47
40.28
41 41
Palu bits)
3.13
7.5%x
53.19
40.77
Palu-50% (3 bits)
3.13
10.6x
26.73
32.72
25.73
63,25
44.43
18.57
3
Baseline
16
1.00x
23.95
31.12
26.74
63.80
56.91
15.25
36.32
Palu-3058
16
1.43x
22.42
29.43
25.52
62.87
58.99
14.25
35.45
LongChat-7B-vll.
Palu Palu-50% 50%
16
2.00x
22.61
25.33
22.73
60.12
43.52
6.84
30.82
KIV1-2
3,16
5.06x
23.24
30.19
26.47
63.54
53,51
16.13
35.60 35.60
Palu-30% bits)
3.13
7.59x
23.12
29.21
25.04
61.99
54.38
11.25
34.33
Palu-50% bits)
3.13
10.6x
18.56
24.14
22.35
58.76
40.50
6.02
29.03
To access Palu's ability for long-context scenarios, we evaluate baseline, KIVI and Palu's accuracy on LongBench (Bai al. 2023) Here, we evaluate the Mistral-7B and LongChat-7B models, which have up to 32K context length. We report the average score for each task type separately, well as the overall average across all 16 tasks. The results are shown in Table We report the accuracy of KIVI using the configuration with a group size of 32 and 128-element fp16 residual (Liu et al. 2024b).
As Table inddicates, we find that at 50% low-rank compression level, Palu is relatively difficult to fully preserve accuracy. However, at a 30% compression level, Palu achieves only a minor average accuracy drop (< 1%) compared to the baseline for both models. Furthermore, Palu can quantize the low-rank latent KV-Cache down to 3 bits, with less than 1% further accuracy degradation. Overall, Palu maintains strong 40.77% and 34.33% average accuracy for Mistral- and LongChat-788 with an impressive 7.59x compression ratio. Compared to KIVI (Liu et al.] 2024b), Palu achieves similar accuracy, while having additional 30% compression rate from low-rank. Notably, Palu
8

---

Published as a conference paper at ICLR 2025
does not require the complex grouped quantization and mixed-precisioe techniques employed by KIVI, resulting in a high inference efficiency (see Sec. 4.5 for details).
4.5 LATENCY EVALUATION
In this section, we provide latency and speedup evaluation, using Llama-2-7b as the base model. We measure decode latency on a single RTX 4090 GPU and compare Palu to the FP16 and KIVI-4-bit baselines. We evaluate Palu's latency for key and value to 75% and 25%, respectively. This allocation is based on our observations from the rank allocation results (see Appendix F.3 for details). For the FP16 baseline, we use the default implementation from HuggingFace. For KIVI, we use the CUDA kernels from its official repository. Due to the small memory capacity of RTX 4090 GPU, we adopt a 4-bit quantization (Frantar al, 2024) for the weights of all linear layers. Our results are the average of 100 runs.
50% compression rate, where we compression rates
Norm Normalizer dizee Speedt App
Kinee Pob-i2il-tt-an
Norralieed Speced
Spreed--00
Nom Homallzed
aloodd
Sequence 15k (4) Attention Modale (w/ RoPE)
Sequence 32A
15c
Sequence
(b) Allention Module (volo RoPE)
(c) EuditoEadd Model (wi RoPE)
(4) End-ttxEid Medel (wis RoPE)
Figure 5: Normalized speedup for both the attention module and end-to-end model decoding, Solid lines represent exact measurements, while dashed lines indicate the FP16 baselines are out of mem- ory, and the speedups are compared to the estimated baseline's 's latency.
4.5.1 SPEEDUPS OF ATTENTION MODULE AND END-TO-ENT DECODING.
Attention module speedup. We compare latency against standard attention without compression or quantization and show the speedups of Palu and KIVI-4-bit in Fig. (a) and (b) for RoPE-based and non-RoPE attention. For RoPE-basee based attention, we applied our online reconstruction kernel for key and employed offline fusion for value as described in 3.1.2
As shown in Fig. 5 (a), Palu has minimal to no speedup when the sequence length is short, e.g. 4K. However, as sequence length increases, Palu delivers substantial performance gains. At 64K input length, Palu achieves 1.89x speedup over the FP16 baseline when using low-rank projection alone. By further applying 4-bit quantization to the Value states, the speedup rises to 2.91x x for the same 64K context length, owing times. This performance notably surpasses KIVI-4-bit, which only achieves a 1.89x speedup at 64K, hindered by the overheads of its fined-grained group quantization. Notably, for RoPE-based attention, Palu-4-bit does not quantize key, as our online reconstruction kernel only supports FP16 precision for now.
optimized oww-precision kernel and reduced memory loading
For non-RoPE attention, we apply matrix fusion to both the Key and Value states (Eq. 7 effectively eliminating all reconstruction overhead. At a 64K sequence length with 50% compression rate, Palu achieves a 2.20x speedup over the FP16 baseline. By further applying 4-bit quantization to both the Key and Value states, Palu boosts the speedup to 6.17x for 64K input length. These results demonstrate that combining low-rank compression and quantization significantly enhances inference efficiency, particularly in long-context scenarios.
End-to-end speedup. We present the end-to-end speedups in Fig. 5 (c) and (d), measuring the decoding latency of generating the next token at various input lengths. Similar to the attention performance results, Palu shows minimal or no speedup for short sequences but delivers signifi- cant acceleration for longer sequences. Without quantization, Palu achieves up to 1.71x and 2.05x speedups for RoPE-based and non-RoPE models, respectively. With a 50% compression rate, Palu runs up to 32K input length on an RTX 4090 GPU. By incorporating 4-bit quantization, Palu han- dles even longer sequences and delivers 2.59x and 5.53x end-to-end speedups at 64K sequence length. Palu integrated with quantization provides substantial speed advantage over KIVI-4-bit, which only reaches 1.78x and 1.81x speedups at 32K sequence length for RoPE and non-Ro RoPE scenarios, respectively, and is out-of-memory for longer sequences.

---

Published as a conference paper at ICLR 2025
4.5.2 KERNEL FOR BEPE-BASED ATTENTION SCORE
In this section, we evaluate the performance of our online reconstruction kernel for RoPE-based attention scores. We measure latency from the pre-RoPE query vector to post-GEMV attention score, and compare with PyTorch's s GEMV, which is used in the baseline attention (see Fig.
We present speedups for group size 1, 4, and 32 at different se- quence lengths in Fig. [ For gs-32 (J-LRD), the highest accu- racy decomposition, the high reconstruction cost causes signif- icant slowdown across all sequence lengths. For gs-1 (M-LRD), our kemnel achieves up to 3.56x speedup at sequence length 16K, showing strong performance when moderate accuracy loss is acceptable. For gs-4 (G-LRD), our kernel reaches up to 1.95x speedup. These results emphasize the need to explore various de- composition granularities for better accuracy and speed tradeoffs.
1g5-32)
Palu (qc-4)
Palu Igs-
Speed
1-
We also observe that speedup decreases for sequence lengths be- yond 16K due rising reconstruction costs, shifting the online reconstruction from memory- to compute-bound bound. A potential op- timization is quantize the decomposed weight matrices fur- ther and leverage high-throughput, low-precision hardware (e.g., INT4 Tensor Cores) for online reconstruction, which we leave for future work. Despite the speedup drop at longer lengths, Palu's 's overall attention speedup increases with longer imput, thanks matrix fusion on the Value state and the reduced memory footprint.
2k
4k
8k
16k
32k
64k
Sequence Length
Figure 6: Speedup of Palu's at- tention score kernel with online reconstruction.
5 RELATED WORK
SVD for LLM Compression. Several works have explored using SVD to compress LLMs. An early approach (Noach & Goldberg 2020) applied standard SVD weight matrices, resulting in significant compression errors. FWSVD (Hsu et al., 2022) addressed this by using Fisher informa- tion to prioritize parameters, while ASVD Yuan et 20233 considered activation outliers. SVD- LLM (Wang al. 2024) further minimized compression loss for each singular value. Unlike these methods, which compress model weights, Palu focuses on reducing KV-Cache size.
KV-Cache Quantization. Quantization is widely used technique for compressing KV-Cache. Atom (Zhao et al. (223)) applies simple per-token quantization, while WKVQuant (Yue et al. 2024) introduces two-level scheme to enhance accuracy. KIVI (Liu al. 2024b) uses per-channel and per-token quantization for Keys and Values, combined with ultra fine-graing group quantization. KVQuant (Hooper al., 2024) employs similar setup but incorporates non-uniform quantization and sparse matrices to handle outliers. On top of these approaches, GEAR (Kang et al. (024) adds a low-rank matrix to compensate for quantization errors. In Pulu, we leverage low-rank techniques to exploit hidden dimension redundancy and achieve outstanding compression through simple per- token quantization.
MLA. The recently released DeepSeek- V2 model (DeepSeck-Al al. 2024) introduces the MLA mechanism, which reduces KV-Cache size by down-projecting Key and Value to a low-rank space and reconstructing them to full rank at runtime. Although MLA may seem similar Palu a high level, particularly with J-LRD, our design and derivation processes are fundamentally different. Unlike MLA, a new attention mechanism requiring pre-training, Palu is specifically designed for post-training integration. Palu focuses on converting existing models with MHA or GQA to support low-rank compressed KV-Cache, preserving high accuracy while enhancing inference efficiency.
6 CONCLUSION
We introduce Palu, novel KV-Cache compression framework that decomposes linear projection weight matrices and caches the compressed latent representations. We propose various optimiza- tions, including group-head low-rank decomposition, automatic rank allocation algorithm, quanti- zation compatibility enhancement, and customized kernels with operator fusion. With these op- timizations, Palu can maintain accuracy while achieving significant memory reduction and high inference speedup.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research is supported in part by Taiwan's NSTC under Grant No. 113-2640-E-A49-00 and the National Science Foundation under Grant No. 2339084. Luis Ceze supported by the Lazowska Endowed Professorship. We would like to express our appreciation to Sandy Kaplan from the University of Washington for her invaluable assistance editing this paper.
REFERENCES
Mohamed Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero- cost proxies for lightweight NAS. 2021. URL
net/ forum? ?id=
0cmMMy8J5q
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Pu- rushotham Kamath. Keyformer: Ky cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 7, 2024.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv: 13245, 2023.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference Learning Representations, 2024a. URL https:
7 T/opereeviw..o fatt/foormmmee vowmmidd-xxaxrdaddd
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian I. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXivv2404.00455 2024b.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence volume 34, pp. 7432-7439, 2020.
Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data- aware low-rank compression for large nlp models. In M. Ranzato, A. Beygelz- imer, Y. Dauphin. P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 29321-29334. Curran Associates, Inc.,
2021. URL /ttpp:://procceednns.... neurips cc/paper files/paper/2021/
Ei
Yuang Chen, Cheng Zhang, Xitong Gao, Robert D. Mullins, George A. Constantinides, and Yiren Zhao. Optimised grouped-query attention mechanism for transformers, 2024. URL https: 7/ //axx/w.or//ab//266..6 14963
Hung-Yueh Yueh Chiang, Chi-Chih Chang, Natalia Frumkin, Kai-Chiang Wu, and Diana Marculescu.
Quamba: A post-training quantization recipe for selective state space models, 2024. URL
hittps://ariin http://aawiv..orr////////1110...
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. (50.::/800005557777 2018.
Pradeep Dasigi, Kyle Lo. Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-seelee seekingg questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy. Steven Bethard, Ryan Cot- terell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of
11

---

Published as a conference paper at ICLR 2025
the North American Chapter of the Association for Computationnal Linguistics: Human Lan- guage Technologies, pp. 4599-4610, Online, June 2021. Association for Computational Linguis- tics. doi: 10.18653/w1/2021..1 Lnaacl- mo...........6..... URL http:p/s//aaaaatacooorooooo/////
naac ma i...in.3.. 365
DeepSeek-AI, Aixin Liu, Bei Feng. Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi
Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song. Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang. Meng Li, Miaojun Wang, Mingchuan Zhang. Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei. Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao. Wentao Zhang. X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu. Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng, Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang. Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaar, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney
12

---

Published as a conference paper at ICLR 2025
Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic, Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presami. Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute. Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang. Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov. Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- -aattttt Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal. Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chemoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Ritmerr Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan. Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar. Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe. Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,o Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL /tttss://taaatt org/ abs/2407 21783
13

---

Published as a conference paper at ICLR 2025
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 1074-1084, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v//1191111 URL https::/accaathhnr
1102
Fino and Algazi. Unified matrix treatment of the fast walsh-hadamard transform. IEEE Transactions on Computers, (225(1))11122-111) 1976. doi: 10.1009TCC.197..164455
Elias Frantar, Roberto L. Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed- precision auto-regressive parallel inference on large language models, 2024. URL https:// arxiv.org/abs//2 1177
11743
Yao Fu. Challenges in sis, 2024. URL
transfoommsssss A theoretical peak performance analy-
08944
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. Utt.ttps::// /ttp::///zeodo..ooo org records/ 10256836
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024.
net/ forum?
|d=uNrFpDPMyo
Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt 14123
Keutzer. Ai and memory wall, 2024.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset for abstractive summarization. CoRR, abs/1911.12237, 2019. URL
http:://axivv/ org/ abs wttp:///////abbb/.1..222 12237
Daya Guo, Canwen Xu. Nan Duan, Jian Yin, and Julian J. McAuley. Longcoder: A long-range pre- trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 12098-12107. PMLR, 2023. URL https:
/pproceddings.o mlr www.:sss.....r..p............... vv2/2/222/////2/2/ /202///22222//2333 guo23j. html
Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint arXiv:2401. 18079, 2024.
Yen- Yen-Chang Hsu, Ting Hua, Sungen Chang. Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022. URL http::://perreeee
net/
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL /ettps://operrest wtt.riiwwweeeeeeeeeee net hwt.:/orrttfftmmot//// forum?
1ddnneeVkeeFyrs
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Amn Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Ian T Jolliffe and Jorge Cadima. Principal component analysis: review and recent developments. Philosophical transactions of the royal sociery A: Mathematical, Physical and Engineering Sci- ences, 374(2065).2010000 2016.
14

---

Published as a conference paper at ICLR 2025
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10. (0.18655/11/111--1447 URL https://aacac antho how.hhhooogy..oo org/
P17-1141
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of 11m, 2024.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, lon Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Xin Li and Dan Roth. Learning question classifiers. In 19th International Conference on Computa- tional Linguistics. COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24 September I, 2002, 2002. URL https: /tal:://aclanee
C02 CO2-11507
Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. In The eightt Annual Conference on Neural Information Processing Systems, 2024. http:ppe/oooeeeee.ss..eeeeeeeeee httpp///w...aeeeeeeeeeee.s..cerp (ttp::iddppp5c5cccc 4G0q21
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 7021-7032. PMLR, 2021. URL http::/prroceedingg wtt..pppes///1//////22/2
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems, 2023.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant- Ilm quantizatiin with learned rotations. arXiv preprint arXiv:2405.16440 2024a.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b.
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information, 2017.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner, On the structural pruning of large language models, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Carl Dean Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, 2000.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arX1v:1809.02789, 2018.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decom- position. In Kam-Fai Wong, Kevin Knight, and Hua Wu (eds.), Proceedings of the 1st Confer- ence of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference China, December 4-7, 2020, pp. 884 889, Association for Computational Linguistics, 2020. URL https: https://aclaatte ant. logy. www.antwwwwoooooo////2/02/// aacl -ma n. 88/1
Natural Language Processing, AACL/JCCNPP 2020, Suzhou,
15

---

Published as a conference paper at ICLR 2025
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL
httts::/cpenreviee net/ www.mmdddduuuupaaaaeo
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 1-77 2020.
Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficienl LLM inference. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL hattps:///penrre
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Charbel Sakr and Brucek Khailany. Espace: Dimensionality reduction of activations for model compression. arXiv preprini urX1v:2410.05437, 2024.
Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction, 2023.
Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019. URL https:
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding. CoRR, 090421044,9864, 2021. URL /RKLhttps://arxi..e..
abs/2104 09864
Jaaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-Aware Sparsity for Efficient Long-Conter LLM Inference. In Proceedings of the Interna- tional Conference on Machine Learning (ICML), 2024.
Philippe Tillet, Uisiang-ssung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet. Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipft: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 0008,6008,00000
URL / nww.eeeeww....e........e........ http://pp...pppppp//////aaaaaa/
16

---

Published as a conference paper at ICLR 2025
Xin Wang. Yu Zheng. Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-awart singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07378, 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- ofrtt---rttttttt natural language processing. In Qun Liu and David Schlangen (eds.). Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18653/1/2020. emnlp-demos.6. URL /ttps:://aclaane htt///aaaanntthoooo org/ 2020
delll..ddem.66 6
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Rep- resentations, 2024. URL
net forum? SttpuummmcNNNN5551111
Zhihang Yuan, Yuzhang Shang, Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models, 2023.
Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. LLM inference unveiled: Survey and roofline model insights. CoRR, 0bs/2402.166663 2024. doi: 10.4855/AAXXV22402..666 URL https:/ /woi
16363
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:240..22065, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1905:09900007300 2019.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuan- dong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. CoRR, abs/2403.03507 2024. doi: 10.48500/ARXIV.205..00000 URL lhttps: orq/10. 48550/arXiv 00000003077
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zhenga, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. arXiv preprint arXiv:2310.19102 2023.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadal- lah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based based
meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905-5921, Online, June 2021, Association for Computational Linguistics. doi: 18 2021.naa maimmaain...... URL ht http::: http:///aaaaattotoooeee org/2021 naacl marll-mainn.. 472
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A QUANTIZATION BASICS
Quantization techniques use discrete low-bit values to approximate high-precision floating points. The general asymmetric uniform quantization function is defined as:
X
X = clamp
2, 0, - 1),
(9)
where X denotes the approximated tensor with low-bit representations (i.e., 4-bit integers), X is the floating-pon point tensor, is the rounding operation.
X X is the scaling factor, and 1-x Xain is zero-point. The [-]
2
B KERNEL IMPLEMENTATION DETAILS
Split 1 2
SplitN-1
d
H
Lette
H2
K
RoPE
Cos/ Sin
Position id
atmmul
K2
Matmul
Figure 7: Illustration of our fused GPU kernel for computing attention scores with online reconstruc- tion. In this figure, represents the query vector, H denotes the low-rank compressed key states, and B stands for the reconstruction matrices.
Kernel for attention score calculation with reconstruction. The central idea of Palu is to lever- age low-rank latent representations accelerate the attention mechanism by reducing data transfer overhead. Instead of working directly with the full-sized key matrix, we store and transfer a com- pressed low-rank latent representation, denoted H R2xr, During computation, our custom GPU kernel performs an on-the-fly reconstruction using a reconstruction matrix B E & pro- ducing a restored key matrix K E Lxd,, where L is the sequence length, d,, the hidden di- mension, and denote the remaining rank after performing low-rank projection. The query vector, represented as q xd,, then multiplies with the reconstructed keys to obtain the attention scores.
To efficiently leverage parallelism, we perform tiling along the sequence length dimension L. Specifically, we split the sequence into smaller tiles of size Trile assigning each tile dedicated thread block. Each thread block independently reconstructs submatrix H; IR Lme from the low-rank latent representation H. then applies the positional embedding using RoPE, and finally per- forms the matrix-vector multiplication between q and H; produce partial attention scores. This design ensures that all intermediate computations, from reconstruction to embedding and final mul- tiplication, remain entirely in on-chip memory (i.e. share memory), thus minimizing high-latency memory access and taking full advantage of the GPU's parallel processing capabilities to achieve significant speedups.
18

---

Published as a conference paper at ICLR 2025
C DISCUSSION REGARDING MEMORY USAGE
In this work, the experimental results focus on the compression rate of the KV-Cache as a key metric. However, it is crucial consider overall memory savings as a more significant factor. For instance, as demonstrated in Sec. [2.2] a typical compression rate of 30% can lead to an increase in weight size by approximately 40%. This increase calculated under the assumption that m n and = 0.7n, resulting in the equation mrttnr = 1.4. Such an increase indicates substantial extra memory usage.
This issue primarily arises in J-LRD decomposition schemes, where the projections of all heads are decomposed jointly. In contrast, our M-LRD decomposition schemes and optimized G-LRD schemes involve non-square target matrices. For example, in the G-LRD scheme with group size of 4, the target matrix formed by concatenating the original projection matrices of each attention head in the group. In the Llama-2-7b model, with embedding dimension of 4096 and head dimensions 128, each projection matrix is 4096x 128, resulting in a concatenated matrix of size 4096x512. In this case, the dimensions should be considered m 8n. Applying the referenced equation pattne with = 0.7n, we find that mr+nr = 0. 0.7875, indicating no additional storage cost and, in fact, achieving an additional 21.25% memory savings.
Furthermore, it is important to highlight that the weights associated with the K and V projections account for only out of 7 linear layers within transformer blocks, comprising merely 16% of the parameters in Llama-2-70 models. This limits the overall impact memory usage. Thus, while J-LRD may incur overhead, the M-LRD and G-LRD schemes provide efficient alternatives that do not lead to increased memory usage, making them viable options for practical applications.
Table 4: Evaluation integrating LoRA with Palu Llama2-7B.
Zero-Shot Accuracy (%) +
Comp. Rate
Method
OBQA
HellaSwag
PIQA
ARC-e
ARC-c
WinoGrande
Avg.
Diff.
Rate 0%
baseline
44.20
76.00
78.07
76.30
46.42
69.30
65.05
J-LRD
45.40
75.57
77.48
75.97
45.31
69.22
64,83
-0.22
Rate 50% w/o LoRA
G-LRD
43.60
73.39
76.33
73.02
42.57
66.77
62.61
-2.44
M-LRD
39.60
65.35
74.76
67.17
35.24
64.64
$7.79
-7.26
J-LRD
44.20
78.51
77.27
48.81
71.03
65.65
+0.60
Rate 50% w/ LoRA
G-LRD
43.40
78.56
75.72
47.10
69.85
64.62
-0.43
M-LRD
70.78
78.02
73.86
43.86
69.22
62.92
-2.12
D INTEGRATING Palu WITH LORA FINETUNE
LoRA (Hu et al. 2022) has become of the most widely used efficient fine-tuning techniques for adapting models to particular tasks or domains with limited data. It has also been applied with LLM compression approaches (Wang et al. 2024 Ma al. 2023) as a post-compression recovery technique to recover information loss after compression. In Palu, LoRA is also applicable to boost the accuracy further.
To integrate LoRA with Palu, introduce additional low-rank matrices A Rdxr' and B', g refine the original low-rank projection as below:
h Ax A, B,
(10)
Here, A will be fixed parameters derived from low-rank decomposition from pre-trained weights of linear layers, while A, and B', are trainable parameters capture the task-specific nuances and recovers the information lost during the compression.
Setup. Following Ashkboos et al. 2024a we sample 8k samples from the Alpaca training dataset fine-tuning dataset and apply LoRA with rank rd 32 and 32. All other dyper-parameters are aligned with Ashkboos et al. (2024a), except for the learning rate 2e 4, and the use of a cosine learning rate scheduler.
19

---

Published as a conference paper at ICLR 2025
Table 5: Perplexity and zero-shot accuracy of Palu, with different decomposition strategies at 50%
Perplexity Wiki2
Zero-Shot Accuracy (%) 1
Model
Method
C4
OBQA
Hclla
PIQA
ARC-e
ARC-c
Wino
Avg.
Baseline
4.88
6.70
45.20
79.39
79.11
79.42
49.06
72.38
67.43
Llama2- 13B
J-LRD
4.97
6.92
46.40
79.48
78.62
79.29
49,91
70.56
67.38
G-LRD
5.31
7.76
77.29
77.42
76.05
45.99
72.45
65.80
M-LRD
5.65
8.34
43.20
74.34
77.53
75.76
45.39
64.20
Experiment Results. We present the experiment results with LoRA in Table Following Ashk boos al. 2024a With LoRA incorporated, J-LRD continues show minimal performance degra- dation with an average drop of 1.00%. G-LRD (gs=4) and M-LRD show improved results compared to their non-LoRA counterparts, with average drops of 2.01 % and 5.14%, respectively. Notably, with LoRA integration, G-LRD shows only a 1.03% accuracy difference compared to J-LRD.
E MORE RESULTS ON ZERO-SHOT ACCURACY
Following up Sec. 4.2 we further report the perplexity and zero-shot evaluation results of Palu on the Llama-2-13B at 50% compression rate. As shown in Table. [5] we observe that Palu achieve competitive accuracy drops around 3% or less across different using either J-LRD, G-LRD, or M- LRD. Thus, the users may adopt M-LRD first to optimize the efficiency further.
Table 6: Ablation study of low-rank decomposition group size perplexity for the Llama2-7B model at a 50% compression rate using Wikitext-2.
Method
Group Size
Perplexity
Baseline
5.47
J-LRD
5.62
5.74
5.88
G-LRD
6.01
6.42
M-LRD
6.81
F ABLATION STUDY
F.I INFLUENCE OF DIFFERENT GROUP SIZE
Since our proposed G-LRD method allows for balancing performance and efficiency by adjusting the group size, we conducted an ablation study on group size. As seen in Table as the group size increases, the amount of shared information also increases, leading improved performance.
F.2 INFLUENCE WALSH-HADDMARR TRANSFORM
We conduct the ablation study to profile the benefits of applying the Walsh-Hadamann Transform (WHT). Experiment results are reported at Table On the 3-bit quantization level, we observe that the Hadamard Transform only brings a slight amount of perplexity. However, when we quantize low-rank representation more extremely (i.e., 2-bit), we can observe notable 4. 4.17 perplexity enhancements. It's worth re-emphasizing that Hadamard Transform will not bring extra overhead during inference, as Palu optimizes the WHT process via offline preprocessing. The reader may refer to Sec. for more details.
F.3 AUTOMATIC RANK ALLOCATION VS. UNIFORM RANK ALLOCATION
Table X presents the ablation study on the impact of different rank allocation schemes on the model's accuracy. Applying rank searching results in a notable performance improvement. For instance, at
20

---

Published as a conference paper at ICLR 2025
compression rate of 50%, there is significant reduction in perplexity by 2.18. Fig. visualizes the rank allocation across different transformer blocks for key and value projection layers. The results clearly demonstrate non-uniform allocation result. Specifically, ally allocated a higher rank than the key. Additionally, the first half of the layers are assigned higher ranks, indicating their greater importance in preserving model performance. This visualization un- derscores the effectiveness of our rank search algorithm in identifying and allocating appropriate ranks different components, thereby optimizing the balance between compression and accuracy.
observe that the value is gener-
Table 7: Ablation Study on different quantization settings for quantizing low-rand latent represen- tations. Same as Sec. 4.3 benchmark.
use the WikiText-2 with sequence length set 4096 the evaluation
Method
Wikitext-2 PPL +
Llama2-7B
5.12
Palu-30% (FP16)
5.25
3-bits w/o Hadamard
5.52
3-bits Hadamard
5.33 (0.191)
2-bits w/o Hadamard
9.48
+ 2-bits w Hadamard
5.77 (3.714)
Palu-50% (FP16)
5.63
+ 3-bits w/o Hadamard
5.99
3-bits Hadamard
5.77 (0.22l)
2-bitt w/o Hadamard
10.58
+ 2-bits w Hadamard
6.41 (4.17L)
Compression
0.6
0.4
0.2
value
20
25
30
Layer Index
Figure 8: Visualization of layer-wise low-rank compression rate Llama-2-7B with 50% of over- all compression rate. Here, compression rates (i.e. rank) are allocated using the proposed Fisher Information-based automated rank allocation algorithm.
Table Ablation study on w/ and w/o rank search. We use Llama2-7b and Wikitext-2 with sequence length 2048 as the benchmark.
Ratc=30%
Rate=50%
Ratc=70%
Uniform
6,34
7.36
10.77
Automatic (ours)
5.62 (0.724)
6.02 (1.361)
8.59 (2.18
G EXPERIMENT DETAILS
G. ZERO-SHOT EVALUATION DETAILS
We selected six zero-shot tasks from the LM-eval benchmark to evaluate Palu:
OpenBookQA (accuracy, Mihaylov al.)
21

---

Published as a conference paper at ICLR 2025
HellaSwag (acc_norm, Zellers al.)
PIQA (accuracy, Bisk et al.)
ARC-Easy (accuracy, Clark et al.)
ARC-Challenge (acc_norm, Clark et al.)
WinoGrande (accuracy, Sakaguchi al.)
report accuracy for WinoGrande, PIQA, and ARC-Easy, and accuracy normalized by sequence length (acc_norm) for HellaSwag and ARC-Challenge.
G.2 LONGBENCH EVALUATION DETAILS
For the LongBench evaluation in this manuscript, we conducted tests on all available English tasks.
These comprise sixteen tasks categorized into six subgroups, ensuring a comprehensive evaluation of Palu. The tasks and their corresponding metrics are detailed below:
Single-Document QA:
Qasper (FI score, Dasigi et al.)
NarrativeQA (FI score)
MultiFieldQA-en (FI score)
Multi-Document QA:
HotpotQA (F1 score, Dasigi et al.)
2WikiMultihopQA (F1 score)
MuSiQue (F1 score)
Summarization:
QMSum (ROUGE score, Zhong et al.)
MultiNews (ROUGE score, Fabbri et al.)
GovReport (ROUGE score)
Few-shot Learning:
TREC (classification score. Li & Roth
TriviaQA (F1 score, Joshi et al.
SAMSum (ROUGE score, Gliwa et al.)
Code Completion:
LCC (similarity score, Guo et al.)
RepoBench-PP (similarity score, Liu et al.)
Synthetic:
PassaageContt (Accuracy)
PassageRetrevial (Accuracy)
During the evaluation, we set the maximum sequence length 31500 for both the Mistral and LongChat model.
H DISCUSSION OF DIRECTLY PERFORMING SVD ON KV-CACHE DURING RUNTIME
As discussed at the beginning of Sec. 3.J a straightffrward approach to compress the KV-Cache with low-rank projections is to apply Singular Value Decomposition (SVD) directly to the KV- Cache. To evaluate its feasibility, we compare the latency required for performing SVD to the time taken for forward pass through decoder block, as illustrated in Tab The results clearly demon- strate that performing SVD on the fly introduces significant computational overhead. Specifically, runtime SVD is approximately 5-10x slower than single forward pass of the decoder block.
22

---

Published as a conference paper at ICLR 2025
Seqlen
32k
64k
128k
Decoder Block (fp16)
0.231s
0.65s
1.93s
SVD
9.24s
9.90s
11.44s
Latency comparison between performing SVD on the KV-Cache and a single forward pass through the transformer decoder block, using the model configuration of Llama-2-7B
Table 9:
Given these findings, we adopt static weight decomposition techniques and modify the caching mechanism to store the owwer-dimensionll latent representations, thereby reducing the memory foot- print while avoiding the runtime costs associated with on-the-fly SVD.
I MORE RELATED WORK
Token Eviction. One prominent direction for reducing the memory footprint of KV-Cache is KV- Cache eviction Adnan et al.) (2024); Ge et al. (2024); Xiao et al. (2024); Zhang et al. (2024). KV- Cache eviction techniques selectively retain parts of the KV-Cache and discard less important tokens to maintain the use of a fixed-size size KV-Cache to control memory usage. Representative works, such as AttentionSink (Xiao et al. 2024), employ a fixed eviction policy by preserving the tokens in the very beginning, which is also called attention sink, together with recent KV pairs. H2O (Zhang al.) 2024) selected the tokens based on accumulative attention scores. SnapKV (Li et al.. 2024 evicts non-important tokens of each head based the local observation window of prompts. While these methods reduce the memory footprint of KV-Cache and data to be transferred, they permanently discard KV pairs deemed less important, leading accuracy degradation some complex tasks that may require information from those eviction parts of the sequence.
Token Selection. To address the limitations of token eviction methods, another line of research focuses retaining the entire KV-Cache while employing sparse attention mechanisms process only selected parts the KV-Cache, thereby reducing latency, Notable examples include SparQ (Ribar et al. 2024) and Quest (Tang et al., 2024). These methods achieve significant improvements in latency and accuracy preservation. However, since the full KV-Cache retained, memory re- quirements remain unaddressed. As result, techniques like CPU offloading often necessary to execute inference requests.
In Palu, take orthogonal approach by compressing the hidden dimensions of the KV-Cache via low-rank projections. A potential direction for future work could be combining Palu with to- ken selection methods reduce memory usage further. This approach could involve compressing non-salient or infrequently accessed tokens to enhance memory efficiency and overall system per- formance further.
J DISCUSSION OF ALTERNATIVE STRATEGIES FOR DERIVING LOW-RANK PROJECTION MATRICES
In Palu, low-rank matrices W AB are derived applying Singular Value Decomposition (SVD) to the weight matrices. Aside from this approach, alternative strategies (Chen et al., 2021 Sakr Khailany 2024) for obtaining low-rank projection matrices also exist. For instance, ESPACE Sakr & Khailany (2024) performs calibration on the input x and subsequently applies eigenvector decomposition compute the low-rank projection matrix P. During inference, ESPACE operates as h xA' and hB', where A' P and B' WP are pre-computed. This approach ultimately achieves the same down- down-projection and reconstruction mechanism the forward pass as the weight decomposition method used in Palu.
Future research could investigate accuracy differences between deriving low-rank projection ma- trices through direct weight decomposition, implemented in Palu, and alternative techniques like ESPACE. Additionally, Palu's GPU kernel optimizations and quantization -iriiiattooo-rinnllyy enhance- ments- such the fusion of the Hadamard Transform-are agnostic to the method used to derive
23

---

Published as a conference paper at ICLR 2025
low-rank projection matrices. This adaptability provides robust framework for integrating ES- PACE or other approaches future work.
K SVD DECOMPOSITION DETAILS AND ERROR BOUND ANALYSIS
In the proposed Palu KV-Cache compression framework, we perform low-rank decomposition onto the Key and Value projection matrices of each layer. The output enror introduced by this decompo- sition directly corresponds to the error induced in the KV-Cache. To minimize this error, we employ the truncation-awarr SVD technique introduced by (Wang et al. 2024p. This method enhances the standard SVD approach by incorporating a transformation that accounts for the statistical proper- ties of the activation data, thereby reducing the output error caused by decomposition. Below, we outline the algorithm details and provide error analysis for the resulting KV-Cache output emror, demonstrating that the introduced error can bounded.
K. TRUNCATION-AWARE SVD WITH WHITENING TRANSFORMATION
Weight Transformation. The primary idea is to adjust the weight matrix W E R by introducing invertible transformation matrix S E Rmxm This transformation aligns W with the activation data X gbxm, making the subsequent SVD more effective in capturing the essential components.
We express the transformed output as:
XW = XS- = XW,
where
X XS E Rxm.
W SWSW E R
Following Wang et al. 2024), the transformation matrix S is derived using the Cholesky decompo- sition (Meyer 2000) of the covariance matrix of the activation data:
SS X
X AI,
with À 0 being small regularization parameter to ensure numerical stability, and is the identity matrix. This choice of S ensures that the transformation accounts for the correlations the input activations, thereby minimizing the output error introduced by the decomposition.
2. SVD on Post-transformed Weights. Following the transformation, we perform standard Sin- gular Value Decomposition (SVD) the transformed weight matrix W:
W U
3. Deriving Low-Rank Matrices.
To achieve compression, retain the top singular values and their corresponding singular vectors, truncating the remaining s = 2-r singular values:
E, = diag(a o2....,o,)) U, E Rmxr, V, E Raxr
where / min(m, n) represents the number singular values the decomposition. The com- pressed weight matrix W' is then reconstructed by reversing the transformation:
W' W'=
This can be expressed as the product of two low-rank matrices A and B:
w = AB, where A = S E B= B=VE,V; Rrx..
K.2 ERROR BOUND ANALYSIS
We now analyze the error bound of the output error resulting from the compression. The analysis is divided into two parts:
24

---

Published as a conference paper at ICLR 2025
1. Expressing the Output Error in Singular Values. Let the output of the original linear layer and low low-rank approximated counterpart be Y and Y', individually. The output error is defined as:
- XW XW XW -((--XWWW-- W)
Substituting the SVD decompositions:
-VV' -,,,,,, = U 1..:::/VV......... :1:
where Σr = (iaa((r+11..... 0t) contains the truncated singular values, and Ui;,r + 1 and V(arr + i| denote the columns corresponding the truncated singular vectors. Therefore, the output error becomes:
Y Y' XU : V +1::
Assuming that X X error is:
X x I (which holds when À is small), the squared Frobenius norm of the output
Y Y 3 = Tr ((Y Y') YY))
-Tr -Tr( U XU
Tr (ETEr) = Σ of.
1=++
Here, Uy U|.rr + 1: 1. VT
Virr +1: +1:+1:10
and Tr(:) refer to matrix trace.
2. Bounding the Singular Values.
Each truncated singular value or satisfies:
0, S 01 = 0 max (W) max (SW).
The largest singular value of SW can bounded by:
mmax Tiaxx(SV (SW) S ||||| W||2.
Since obtained the Cholesky decomposition C T + AI, its largest singular value
is:
|S|| V //maxx (C) max +
Therefore, the largest singular value T'ax (SW) is bounded by:
T max (SW)= I mmax (X
W/|2
Combining these results, the Frobenius norm of the output error bounded by:
1/2
Y Y S
Σ
o?
S v Tmax (SW).
where s = - r is the number of truncated singular values.
Thus, the output error is bounded by:
Y -- S V V Fmax (X + / S Vt (X + X |||
K.3
SUMMARY
25

---

Published as a conference paper at ICLR 2025
10
Frobenlus Emor
By applying whitening based SVD compression, obtain a low-rank approximation of W in the form W 2 AB, consistent with the standard SVD ap- proach outlined in Section 2.2 Our theoretical anal- ysis demonstrates that the output error of the decom- posed linear layer can be effectively bounded and is influenced by both the number of truncated singu- lar values and the properties of X and W. Further- more, we empirically validate these theoretical error bounds using randomly generated data (m 100, = 80, 50). As demonstrated in Fig. g em- pirical results confirm that the observed errors are consistently bounded by predicted bounds, which in- crease proportionally to the square root of the num- ber of truncated singular values (v8).
10a
10a
Empirical
101
Theoretical Bound
Theoretical Upper Bound ($ [==00 t)
109
40 50
60
70 80
Number of Truncated Singular Values (s)
Figure 9: Empirical and theoretical error
bounds for the output of low-rank decom- posed linear layer with respect to the number of truncated ranks (singular values
L COMPUTE AND MEMORY FOOTPRINT ANALYSIS FOR ATTENTION SCORE COMPUTATION
Table 10: Summary of Complexity for Compute and Memory Requirements.
Configuration
Compute Complexity
Memory Complexity
Baseline
OLL .n)
OLL d...)))))
Palu (M-LRD)
O(L T4 dn n)
O(L Ti. n))
Palu (G-LRD)
O dn n.
O(I. ngg
Palu (J-LRD)
O Fjoint" dp. TL)
O -rjonnt)
This section provides detailed analysis of the compute and memory requirements for attention score computation in RoPE-based attention module with a baseline approach and the Palu. The analysis considers various decomposition granularities, including M-LRD, G-LRD, and J-LRD, plicitly accounting for reconstruction, positional embedding (e.g.. RoPE) re-application, and atten- tion score computations (GEMV). We summarize the memory and compute complexity in Table.
10
L.1
DERIVATIONS
General Formulation for Compute Complexity: The total FLOPs differ between the baseline and Palu configurations. For the baseline:
Baseline total
FGEMV-
For Palu configurations:
PPlu
F Precoms
+ FROPE + FGEMV.
total
Here:
Preconsi FLOPs to reconstruct the full key matrix K from low-rank latent representations.
FROPE: FLOPs to re-apply positional embeddings onto K.
FGEMV: FLOPs to compute attention scores via GEMV between K and the query q.
General Formulation for Memory Complexity: The total memory complexity includes both
memory reads and writes:
Moual Mieads + Murites:
where:
Mreads Mkeys MR + Mqp
Mwrite M Moaputt
26

---

Published as a conference paper at ICLR 2025
Memory terms:
Mays Memory to read the full keys in the baseline, or low-rank latents for Palu.
Mg: Memory to read the reconstruction matrices.
Mg: Memory to read the query vectors.
Moutput Memory to write the computed attention scores.
In the baseline configuration, the post- Rost Key K R and query vector q 1xd,, are directly used for GEMV computations. Reconstruction and RoPE re-application are not required.
Baseline:
Compute complexity:
F Baseline
= L dn n2
F Baselliee
~ O(L dn nt).
total
Memory complexity:
M Baselline total
keys
My + Moutpu (L dp n) (dh, n) (I.n),
MBBaliine
MiBaelliee
O(L dn n).
Palu (M-LRD): the M-LRD configuration, each attention head has its own low-rank latent H. E LxT, and reconstruction matrix B, E grexda, Reconstruction and RoPE are required.
Compute complexity:
M-LLD total
J.M-LRD + FROPE + F-GEVV
(L dn n2) + (L dn n) + (L dp n),
(L ri du + 2 L dn) n,
M-LRD total
O Ti dn. n)).
Memory complexity:
M
M-RRD
Mays + -MM + Mg + Moutput
M.LRD
M-IRD
(L. n) + (ri da n) + (dn, .n) + (L- n), (r. dn n))
O([L
O(L-r, ....., n) when > dr
In the G-LRD configuration, groups of s heads share a low-rank latent H., e I Lxx, and reconstruction matrix B,, E (dz:s))
Palu (G-LRD);
Compute complexity:
FG-LRD
FG-RD + FROPE + FGEMV
recons
(L rg dn n)) (L dr, n) + (L dr n),
(L.rg dn + 2. L dn) n,
Lotal
O(L Tg dr n).
Memory complexity:
GGLLDD total
MG-RD + MG-LRDD + Mg + Moatuut
= (I. ng) + (rg dr n) + (da . n.) + (L v)),
O(LL Tg. -gg)) + (rg dr n))
~ O(L Tg pgg))s when L > (da n)
27

---

Published as a conference paper at ICLR 2025
Palu (J-LRD): In the J-LRD configuration, all n heads share single low-rank latent H joint I Lo and reconstruction matrix Bjoint IR. "xin x (d)-n).
Compute complexity:
F LRD
EJIRD recuns
+ FRRPE + FGEMV
tiaal
rjoint dn n) + (L dn)n)) + (L dp .n).
(L Tjoint dn + 2- L dn) n,
O(L ijoine dn n).
Memory complexity:
MERRD +Ma + Moutue (I. rjoins) + (Fjoiis dn n) + (dn. n) + (I .n);
MELRD
total
2 O(L Tjoin)) + (Fjoin dn -n)). O Tjoin))
when L >> (dg, n))
28


Reviews:
[{"review_id": "myFLjM7dq8", "reviewer": "Reviewer_12bo", "summary": "This paper addresses the issue of high GPU memory consumption in KV Cache during LLM inference by proposing a novel approach that differs from traditional KV Cache compression methods (such as quantization and token-level compression). Specifically, it employs Singular Value Decomposition (SVD) to reduce the rank of the KV Cache matrix, thereby decreasing its memory usage. Building on this method, the paper further explores several enhancements, including: 1) merging multiple heads into a group for SVD to minimize precision loss; 2) adaptively adjusting the compression ratio based on the redundancy of information at different layers; and 3) integrating the Walsh-Hadamard transform to facilitate quantization of the compressed results.", "questions": "1.\tSuggestion: The authors might consider adding a mathematical derivation section to demonstrate that the SVD method has an upper bound on the accuracy error resulting from the rank reduction of parameter matrices. This would enhance the generalizability and applicability of the method across different models and datasets.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe approach proposed in this paper, which utilizes Fisher information to allocate compression ratios for each linear layer, is both effective and scalable. It offers valuable guidance for other KV Cache and model parameter compression efforts.\n2.\tThis paper implements a CUDA kernel to maximize throughput efficiency.\n3.\tThis paper proposes and experimentally validates the compatibility of its optimization methods with traditional techniques such as quantization and LoRA fine-tuning, ensuring their practical applicability in real-world environments.\n4.\tThis paper conducts comprehensive ablation experiments to validate the effectiveness of each optimization module individually.", "weaknesses": "1.\tThe method proposed in this paper, which utilizes SVD for rank reduction of parameter matrices, is not novel, as it has been employed in various studies (e.g., Sharma et al., \"The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction,\" 2023). The corresponding optimization of KV Cache storage is consequently straightforward given the rank reduction of the parameter matrices.\n2.\tThe paper does not provide open-source code, which raises concerns about the authenticity and reproducibility of their experimental results.\n3.\tThe paper compares a limited number of related works; aside from the baseline, it only includes quantization-based methods such as Atom and KVQuant. However, token-level compression methods are also a key focus in the KV Cache compression field, which this paper completely overlooks. The authors should at least conduct comparative experiments and method combination studies with one of the state-of-the-art approaches in this area, such as H2O (Zhang et al., \"H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models,\" 2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GD44zoBTL9", "reviewer": "Reviewer_Fsn9", "summary": "The paper introduces Palu, a novel framework designed to compress the KV-cache of large language models (LLMs) using low-rank projection. Unlike existing techniques, such as quantization or token eviction, which may overlook redundancies in hidden dimensions, Palu effectively captures these latent efficiencies. Notably, it provides a post-training compression solution, in contrast to low-rank methods like GQA and MLA that require training from scratch. This approach allows Palu to strike a balance between memory efficiency, accuracy, and inference speed.", "questions": "Is it possible to compress position embedding by similar way to remove the key cache reconstruction in attention computing? Or do some better tricks in attention computing rather than reconstructing key states?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Post-Training Solution:** Unlike MLA, which requires retraining from scratch, Palu can be applied post-training. This versatility allows for the compression of existing models without additional, costly training cycles.\n2. **Balanced Accuracy and Efficiency:** The group-head low-rank decomposition (G-LRD) balances accuracy and computational efficiency. It preserves essential information while reducing reconstruction overhead compared to per-head and joint-head decompositions.\n3. **Automated Rank Allocation:** By using Fisher information to dynamically assign ranks to matrices based on their importance, Palu ensures compression without significant accuracy loss. This approach optimizes each layer's sensitivity to compression.\n4. **Quantization-Friendly Design:** Palu addresses quantization-induced outliers with a Hadamard transformation, effectively supporting low-bit quantization. This further enhances compression without compromising accuracy, yielding competitive perplexity scores compared to quantization-only methods.", "weaknesses": "1. **Inefficiency with Positional Encoding**: Palu requires reconstructing the original key cache when calculating positional encodings, which may reduce computational efficiency.\n2. **Performance on Complex Tasks**: Palu-50% does not outperform lower-compression 2-bit quantized KV-Cache methods, particularly on complex tasks like Code and Synthetic. This limitation suggests that Palu’s effectiveness may decrease in more challenging scenarios, where high compression ratios can impact accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KtcgPx9TSx", "reviewer": "Reviewer_2i3G", "summary": "The paper introduces Palu, a KV-Cache compression framework that reduces memory usage in LLM inference by applying low-rank projection to the hidden dimension. Palu decomposes linear layers into low-rank matrices, caching compressed intermediate states and reconstructing KV values on the fly. Key features include medium-grained low-rank decomposition, automated rank search, quantization compatibility enhancements, and a GPU-optimized matrix fusion kernel. Experiments show that Palu compresses KV-Cache by 50% and boosts attention module speed by up to 1.87×.", "questions": "Q1: Corresponding to W1, provide theoretical or empirical evidence for the statement in Section 3.1 that \" However, this approach poses significant computational challenges during runtime that make it impractical for deployments \"\n\nQ2: Corresponding to W2, add a theoretical derivation of the algorithm's error bounds.\n\nQ3: Corresponding to W3, include an analysis of the algorithm's time complexity.\n\nQ4: Corresponding to W4, clarify the relationship between \"compression rate\" and \"Avg. Bits\" in Section 4, along with definitions of related metrics.\n\nQ5: It is recommended to open-source the code.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1: The method creatively combines SVD algorithms with KV-cache compression, optimizing speed and grouping different attention heads with varied compression rates, which is insightful.\n\nS2: The experiments are comprehensive, measuring both the algorithm’s accuracy and speed.\n\nS3: The figures and tables are clear, and the descriptions are precise.", "weaknesses": "W1: In Section 3.1, the statement \"However, this approach poses significant computational challenges during runtime that make it impractical for deployments\" lacks experimental results and theoretical support, even though this claim is foundational to the proposed method.\n\nW2: There is no theoretical proof for the accuracy of the proposed algorithm, such as bounded error analysis.\n\nW3: The time complexity of the algorithm is not analyzed.\n\nW4: In Section 4, the relationship between \"compression rate\" and \"Avg. Bits\" is unclear. Additionally, definitions of metrics such as \"compression rate\" are not specified.\n\nW5: The code is not open-sourced, making reproducibility uncertain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WpbHmzF4Mc", "reviewer": "Reviewer_pJHR", "summary": "Palu is a new framework for compressing the key-value cache in large language models, reducing memory usage and improving inference speed. It uses low-rank projection, group-head low-rank decomposition, and automatic rank allocation to achieve up to 50% KV-cache compression with minimal accuracy loss. Palu also integrates quantization compatibility and optimized GPU kernels, resulting in significant speedups, especially for long-context scenarios. The framework outperforms state-of-the-art quantization methods and demonstrates effectiveness across various LLMs and benchmarks.", "questions": "- In Table 1, why does Llama2-7B have better perplexity and accuracy than Llama3-8B-Inst? There is definitely something wrong in the experimental setup.\n- Rather than compressing weights, since KV cache compression is desired, can a method such as ESPACE [1] be used to project activations to a low rank directly?\n[1] Sakr, Charbel, and Brucek Khailany. \"ESPACE: Dimensionality Reduction of Activations for Model Compression.\" arXiv preprint arXiv:2410.05437 (2024).", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- Novelty: unlike common quantization and pruning approaches, Palu uses low rank decomposition to lower the size of KV caches, which is more mathematically sound and doesn't require hardware changes.\n- Significant compression is claimed, and it is to be expected that KV caches can be significantly compressed. However, I do have some concerns about the achieved compression which I cover in the weaknesses.\n- Extensive empirical results spanning accuracy studies and speed-up analyses are provided.", "weaknesses": "- While caching latent keys and values makes sense from the memory efficiency angle, it introduces significant computational overheads during decoding. Indeed, while the prefill phase can allow for GEMMs to occur at every layer, the decoding phase is all about GEMVs. However, in the palu framework, a full GEMM is required for to rematerialize each key and value matrix to be used in the attention module. This adds a lot of complexity, particularly when decoding a large output sequence length.\n- What the paper really is doing is not KV cache compression, but really it is compressing the QKV WxA layer preceding attention by applying SVD on two thirds of the weight matrix. \n- The experimental setup doesn't make a lot of sense. It is mentioned that FlashAttention is not used for apples-to-apples comparison. But FlashAttention anyway is there to speed-up the prefill phase where the KV cache doesn't exist yet. Why does it then matter? Why are you including prefill latency when your solution is supposed to be addressing KV cache compression, which is used in the decoding phase?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)