Efficient Low Rank Attention for Long-Context Inference in Large Language Models
Authors: Li Tenghui
Guoxu Zhou
Xuyang ZHAO
Yuning Qiu
Qibin Zhao
Venue: NeurIPS 2025 poster ()
Keywords: LLM
KV cache
Low rank decomposition
Long context inference
Primary area: deep_learning
TL;DR: Use lighweight low rank (q K) to help indexing offloaded KVCached

Abstract: As the length of input text increases, the key-value (KV) cache in LLMs imposes prohibitive GPU memory costs and limits long-context inference on resource constrained devices.
  Existing approaches, such as KV quantization and pruning, reduce memory usage but suffer from numerical precision loss or suboptimal retention of key-value pairs.
  In this work, Low Rank Query and Key attention (LRQK) is introduced, a two-stage framework that jointly decomposes full-precision query and key matrices into compact rank-\(r\) factors during the prefill stage, and then employs these low-dimensional projections to compute proxy attention scores in \(\mathcal{O}(lr)\) time at each decode step.
  By selecting only the top-\(k\) tokens and a small fixed set of recent tokens, LRQK employs a mixed GPU-CPU cache with a hit-and-miss mechanism where only missing full-precision KV pairs are transferred, thereby preserving exact attention outputs while reducing CPU-GPU data movement.
  Extensive experiments on the RULER and LongBench benchmarks with LLaMA-3-8B and Qwen2.5-7B demonstrate that LRQK matches or surpasses leading sparse-attention methods in long context settings, while delivering significant memory savings with minimal accuracy loss. Our code is available at \url{https://github.com/tenghuilee/LRQK}.

Efficient Low Rank Attention for Long-Context Inference in Large Language Models
Tenghui Li Guangdong University of Technology
Guoxu Zhou
Guangdong University of Technology Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing. Ministry Education, Guangzhou, CHINA gx zhou@gduu edu.
RIKEN AIP
tenghui.. bhe@foxmail.com
Xuyang Zhao
Yuning Qiu RIKEN AIP
Qibin Zhao RIKEN AIP
RIKEN iTHEMS RIKEN IMS Chiba University xuyang. zha.grikeaarri.en....
yuning qiu@riken.
Tibbi..hao@riken jp
Abstract
As the length of input text increases, the key-value (KV) cache in LLMs imposes prohibitive GPU memory costs and limits long-context inference resource con- strained devices. Existing approaches, such as KV quantization and pruning, reduce memory usage but suffer from numerical precision loss or suboptimal retention of key-value pairs. In this work, Low Rank Query and Key attention (LRQK) is introduced, two-stage framework that jointly decomposes full-precision query and key matrices into compact rank-r factors during the prefill stage, and then employs these low-dimensional projections to compute proxy attention scores OIr) time at each decode step. By selecting only the top-k tokens and a small fixed set of recent tokens, LRQK employs mixed GPU-CPU cache with a hit- and-miss mechanism where only missing full-precision KV pairs transferred, thereby preserving exact attention outputs while reducing CPU-GPU data move- ment. Extensive experiments on the RULER and LongBench benchmarks with LLaMA-3-88 and Qwen2.5-7B demonstrate that LRQK matches or surpasses leading pearse-attention methods in long context settings, while delivering sig- nificant memory savings with minimal accuracy loss. Our code available at https:/ thub. http//eenguieeeeeReeee
Introduction
Large language models (LLMs) have shown their remarkable capabilities across diverse tasks. Recent LLMs have extended their capabilities to support long-context processing, enabling them leverage extended sequences for improved performance in applications such as document level understanding and long-form text generation [1-5]. Despite these advances, efficient processing long-contexts presents substantial challenges. The primary constraint stems from computational resource limitations, particularly regarding memory consumption. The KV cache stores historical key-value pairs to prevent redundant computations. However, its size grows linearly with sequence length and quickly imposes prohibitive memory requirements when handling long inputs. Consequently, as context windows expand, the KV cache correspondingly increases in size, eventually becoming the system bottleneck.
"corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

4:
Agg
Light attention score
q
k,
ke
miss
TopK
Fetch CPU
Kg kt
V
Attention
q:
Ve
bit Unchange
k//v:
Figure 1: Brief overview of the proposed Low Rank Query and Key attention (LRQK) method. Subscript S2 denotes the selected tokens, denotes the current token. qr. k, are the original query and key, qr, k, are the approximated query and key, Ax, is the low rank key matrix. K 12,-11 V - are GPU cache K Va,--1 merged with fetched CPU keys and values.
Existing approaches addressing the KV cache memory bottleneck can be systematically categorized into three fundamental methodologies: (1) quantization techniques [6-9], which reduce numerical precision while preserving semantic integrity; (2) pruning mechanisms [10-12], which selectively retain critical tokens while eliminating redundant ones; and (3) offloading strategies [13-15], which redistribute memory across heterogeneous storage hierarchies. Additionally, hybrid approaches [16] combine multiple techniques maximize efficiency.
While these methods have made significant strides in addressing the KV cache memory bottleneck, they each come with inherent limitations. Quantization methods will sacrifice numerical precision, and the evincing techniques risk climinating potentially crucial key-value pairs, particularly prob- lematic when tokens deemed unimportant in current contexts later become critical for subsequent reasoning. Pruning methods may inadvertently discard important key-value pairs, leading to sub- optimal performance when previously unimportant tokens become critical in subsequent reasoning. Offloading strategies introduce substantial latency overhead due to frequent PCIe data transfers, severely impacting real time processing capabilities.
Consequently, an ideal solution should balance the trade-offs between memory efficiency, numerical precision, and computational latency. A key observation in decoder-only transformer LLMs is the inherent sparsity attention activation patterns during inference. had been shown that only a small subset of tokens contributes significant attention weights at each generation step [11, 10, 16, 12]. This sparsity also shows that the possibility of offloading of inactive key-value pairs to cheaper CPU memory while retaining active pairs in GPU memory. However, fundamental technical challenge emerges when implementing this strategy. Full attention computation requires complete access to key tokens, which are not efficiently accessible after offloading.
To mitigate these challenges, a hybrid attention mechanism proposed to exploit the intrinsic sparsity of transformer attention patterns while preserving exact computation of attention outputs. At its core lies the observation that full qucry-key interaction matrix in decoder-only LLMs admits a close low rank approximation. Therefore, the query matrix and key matrix are jointly factorized into compact bases. In the decoding phase, a "proxy" attention score is computed via these factorized query and key matrices, and identifies the top-k relevant tokens in the GPU cache. Thus, the ground truth attention score can be carried out on the retrieved subset k and v, guaranteeing numerical fidelity. An overview of the proposed method is shown in Figure and the time comparison is represented in Figure 2. The main contributions of this paper can be summarized as:
Joint Low Rank Approximation: Rather than performing computationally expensive SVD operations on pre-RoPPE keys as implemented in ShadowKV [16], the proposed method jointly optimizes low rank approximations of both query and key matrices, and reduces computational complexity while maintaining representation accuracy.
Precision- Priisiiv-rrreeviigg Attention Computation: The low rank approximated keys and values serve independently as computational proxies for lightweight attention score estimation. Critically, the subsequent attention layer operations utilize the original query, key, and

---

GPU
Attention
FFN
Atlention
ITN
time
(b) Full CPU
Attention
FFN
Attention
Load Cache
(c) L.QK
Attention
FFN
Attention
FFN
time
-Tin Cache
Time comparison with full GPU, full CPU and the proposed LRQK methods. The orange block is the selection operation of KV pairs. The black blocks are cache loading operations. The blocks above line mean GPU operations and the blocks below are CPU operations.
Figure 2:
value vectors without any approximation or reconstruction, thereby preserving mathematical fidelity and model performance.
Mixed Cache Management: A sophisticated hybrid GPU-CPU storage system featuring active token retention mechanisms and hit/miss buffer architecture that minimizes cross device data transfer is implemented. Additionally, a specialized buffer maintains recently accessed keys and values, which empirical analysis confirms consistently receive high attention scores, further optimizing cache hit rates under operational conditions.
The paper is organized as follows. In Section two key insights presented: the low rank nature of the key matrix and the high attention scores between a current token and its neighbors. Section introduces the Low Rank Query and Key (LRQK) method, detailing both prefill and decode stages. In Section 5, the performance of the LRQK method is assessed across various models and datasets.
2 Related Works
2.1 Token Eviction
Token eviction methods seek to bound the memory footprint of the KV cache by discarding less
informative key-value pairs while preserving generation quality [11, 17, 10, 12, 18, 19]. Early work such as StreamingLLM [11], maintains a fixed "attention sink" of initial tokens together with sliding window of recent tokens, stabilizing decoding for arbitrarily long sequences. Building on this, H₂O [10] formulates eviction as dynamic submodule optimization problem to retain both recent tokens and "heavy hitters". Rather than relying on global scoring, SnapKV [12] leverages per-head clustering of attention patterns within small observation window to select and compress the most relevant KV positions in one pass. More recently, Ada-KV [18] derives theoretical upper bound on post-eviction attention error and proposes head-wise adaptive budget allocation strategy, yielding consistent improvements. However, these existing eviction strategies universally trade off either long-range context fidelity or computational simplicity by relying on rigid windows, coarse heuristics, or expensive per-head clustering, therefore resulting in non-negligible approximation етror or management overhead. Our approach adopts a joint low-rank proxy to select only the top-k full-precision KV pairs and small hit/miss recency cache load them, guaranteeing exact attention outputs and less CPU-GPU data movement.
2.2 Dynamic Sparse Attention
Dynamic sparse attention mechanisms represent significant advancement in transformer based models by selectively computing attention scores on subset of tokens while maintaining complete key-value cache storage, thereby substantially reducing computational requirements. Contemporary approaches implement diverse token selection strategies with varying degrees of efficacy: QUEST [20] employs a page based selection methodology, while Loki [21] leverages dow-diiensional attention approximation through principal component analysis (PCA) on key matrices. PALU [22] and LPA (23) also utility the low rank capability in attention. Notably, two recent techniques, InfiniGin [14]
and ShadowKY KV [16] utilize singular value decomposition (SVD) for key-value selection. InfiniGen
3

---

pre-fetches essential entries using predefined projections via SVD, whereas ShadowKV offloads the entire V cache to CPU memory after SVD decomposition of the key matrix (K). While effective, these SVD) based approaches incur substantial computational overhead and fundamentally alter the original query/key representations, potentially compromising model fidelity. contrast, the proposed method combines joint low rank Q, K projection and retrieves the relevant full-precision KV pairs on demand, ensuring exact attention computation with less approximation error.
3 Preliminary
3.1 Low Rankness of Query and Key
As demonstrated in prior works [16, 14], the key matrix in decoder-only transformers exhibits pronounced low rank structure. Motivated by this observation, the full-precision query and key matrices are jointly factorized into rank-r components, enforcing
rank(QK < min (rank(Q), rank)))ann
min (rank(Q). rank(K)).
(I)
This implies that if K is effectively of rank r. then its interaction with any query also lies close to an r-dimensional subspace. Consequently, it is possible to approximate QK by a low rank factorization AQA k with a low upproximation error. An illustrative example of this phenomenon appears in Figure 3, which plots the average singular value spectrum of the per-head query and key matrix K on Qwen2.5-7B [2] and LLAMA-3-8B-IN [3, 24] under the Wikitext-2-vl test set [25]. In both cases, the singular values decay rapidly beyond small rank, confirming that K admits a low rank approximation with minimal loss.
135
1553
100
17503
500
350
(a) Qwen2.5-7B Query
(b) Qwen2.5-7B Key
100
(c) LL 3-8BB11
Query
(d) LLLAMA---BB-1 Key
Figure 3: Examples of the mean of singular values of the query and key matrix over different layers on Qwen2.5-7B and LLAMA-3 1M models. The singular values are summed over batches and attention heads.
3.2 Attention Scores Near Current Token
To quantify the locality of self-attention during decoding, let qe be the query at step and k, the key at position i, so that the attention weights are given by
softmax (q: [... k,-
k
k k:)) [-.- 0r-3 a,-2 ar1 orl-
The average her-head scores are evaluated on the Wikitext-2-vl test set using both Qwen2.5-7B and LLaMA-3-8B-B M.. Results are shown in Figure 4. In every case, the curves exhibit pronounced higher attention scores the current token and its near neighbors, confirming a strong recency bias. This empirical finding is also observed by StreamingL_LM [11], which motivates our inclusion of a compact recency buffer the GPU cache that persistently holds the most recent tokens, thereby maximizing cache hit rates with minimal overhead.

---

313
300
308
Qwen2.5-7B
(b) Llama-3-88-
Figure 4: Examples of the attention scores of the neighbors current token. The window size is 16. The z-llbel "current" is the index of the current token, -8 means the tokens at t 8, and The attention scores are averaged over batches and attention heads.
Proposed Method
The inference process of LLM can be divided into two distinct phases: prefill and decode. When prompt is provided by the user, the long prompt is first prefilled once by the LLM, and then the token decoded autoregressively. Specific cache mechanisms are required for these two phases, the LRQK will be introduced for prefill in Section 4.2 and decode in Section 4.3, respectively.
The preliminaries and formulation used in the proposed method are clarified below. In practical implementations, transformer models typically process input tensors with a shape (batch size, num heads, sequence length, hidden size). However, when methods such GQA [26] are employed, the number attention heads used for the query tensor is usually different from those used for the key and value tensors. Therefore, repeated key/value (repeat KV) mechanism is utilized, ensuring that the number of heads for the query and the key/value tensors are aligned. In addition, to simplify notation, 'batch size" and 'num heads' dimensions batches and heads are independent. Thus, the query, key, and value matrices as Q, K, V E Rixdd where denotes the sequence length and d the hidden size of each head. Moreover, since the prefill and decode process is identical across all transformer layers, the layer indices are omitted for clarity.
also omitted, computations across different
4.1 Mixed Caching
To address memory constraints and optimize runtime efficiency, mixed caching strategy is adopted that leverages both fast yet limited GPU KV cache and a larger but slower CPU cache. To further reduce the data transfer overhead between GPU and CPU, a hit-and-mise cache mechanism is introduced. In this scheme, when the required tokens are already present in the GPU cache (i.c., cache hit), direct access provided without incurring additional transfer cost. When the required cache is missing in the GPU, only the missing tokens are fetched from the CPU cache, therefore avoiding redundant data movement and improving overall system efficiency.
Based on the observation that recent tokens are more likely to receive high attention scores (as shown in Figure 4), a dedicated buffer is incorporated cache is structured as active tokens lite tokens ]. where the 'active tokens' are the top-k tokens corresponding to the 94 with the approximated highest attention scores selected by the low rank approximations of query and key matrices, and the 'lite tokens' corresponding to the set Ω capture a fixed number of neighboring recent tokens. The total window size of the GPU KV cache is thus the sum of the active and lite tokens, allowing for balance between relevance and recency in cache composition.
the most recent tokens. The GPU KV
4.2 Prefill
At this stage, the input consists of a long prompt represented as matrix X E Rixd where denotes the sequence length and the hidden dimension. By applying linear projections to X, the query, key, and value matrices: Q K, V E lixd will be obtained.
According (1), the attention score matrix is usually low rank, thus can be approximated using jointly low rank decomposition:
QK A
(2)
5

---

where Aq. Ak RR and r < d denotes matrix rank. In addition to the joint low rank approxima- tion. query and key matrices are approximated for memory efficiency:
Q A
K = ARBK.
where Bq. Bx Br Note that the approximation for attention QK' and Q. K shares the same column/row space spanned by AQ and Ak. For solving (2) and (3), can be formulated as
arg min Aq Ax Bx
QOK - A s.t. Q AqBo. K AgBK.
To solve the constraint optimization problem (4), is relaxed by deriving a Lagrangian:
Spre 1
2 QKQ AQA
Xpe
QQ -
Apx
|K- KAABBRIl
(5)
where Xpo and Apk are two scaling factors indicating the importance of the low rank approximation of the query and key matrices, respectively.
By solving the Lagrangian (5), with setting the partial derivative be zero, the solution of the query and key matrices can be obtained (see Appendix A. for details):
Aq Q (KJ Ax XO) (A AOO
Aς K (Q AQ + >pk (ADAq AxkBBBR)
(6)
Bą = AAQ
B --
From (6), it is notice that the operation of matrix inverse is required. The matrix inverse oper- ation is expensive O(r*). However, since the rank r is a small number, and A A A A AqAQ. B& k. BQ are all with shape (r, r). which are small matrices. Therefore, the computation costs of matrix inverse are relatively low. In addition, when solving Aq and Ax, the memory cost can further be reduced by changing the multiplication order. For example, by changing the order of operations from (QK A to Q(K AK). the computational complexity is reduced from O (rld)).
The iterative procedure in Algorithm 1 can be interpreted as a Block Coordinate Descent (BCD) method, where each step optimizes the Lagrangian with respect subset of variables while holding the others fixed.
Algorithm Prefill of LRQK, alternating updates for Aq, Ax. Bq, BR
Require: Q K, rank App. Apr
1: Initialize Aq, A ~ ~N(
2: while max iteration not converge do
3: Update Bę + (ATAQ) QQQ
4: Update Bg + (A K
5: Update Ak <- K (Q AQ + ApK (ATAQ + AxxBBBB -1
6: Update AQ < Q (KTA) A + AD) (AJAK + Bo
7: end while
8: return Aq. Ak, BQ, BR
4.3 Decode
Following the prefilling phase, language model generates the remaining tokens sequentially, adhering an autoregressive decoding process where each token is predicted based on the previously generated context. Suppose the current stage f,, the input a row vector x, Rixd and its projections are q: E g glxd and k; e gixd In the decode stage, the goal is to compress the qe and k,

---

to much smaller ones q E Rixr and k, IR Rxx,, therefore the memory footprint of KV cache can be greatly mitigated.
From the previous stage, the low rank approximations of the query and key matrices are obtained in the form of left and right factors: Aq. Ax R and Bq, Bx Rexd To maintain consistency in the modeling setup for new tokens, the following optimization problem is formulated for estimating the approximated query and key vectors, q and k,, respectively,
arg min l. BQ,t- qelli- + k K,BR,-1 - K.BK,.I-KKK -k ã. s.t. q.k. qik, G.
1
The matrix A denotes submatrix of Ax that includes only the rows indexed by Ω. Similarly, K is the corresponding submatrix of K formed by selecting the rows indexed by Q. Here, $2 represents the index set of KV cache retained in GPU memory at time step t 1.
The Lagrangian of (7) can be written as:
1 la. g 1
Lac 2
k. K,BK,-I kr
(8)
+ Adi
(q.k. -ikk
A 2
Iqq |q - 9:
+
where dr and An are regularization parameters. For more details, please refer to the Appendix A.2. By solving the Lagrangian (8), update rules for q and k, can be obtained as:
q. mlg M M rg.tt
k, (k T k + An q.) (Br,-I BR,.-1 + Pa)
mig. (q. dnqik, k, |
(9)
Mrg.t= 1 daik k, A AR--AAA-A
Similar to prefill stage, since the matrices require taking inverse is of size (r,r), which is small, and the computation cost is relatively small.
For BQ. and B k, one step gradient descent is applied op the objective function. The update rules are given by:
B
BR, B 1/K VBK.-11
(10)
where n/o and уe are the learning rates for BQ and B respectively. Two learning rates 1/ and 7i are computed as:
(q. q:) (q. VB......
n ( VBor (q: VBQ.-1)
(11)
(K.BK.II- k.) ks (k. T
1K (K.VBK,-II (K. (K.VBK.)))
(12)
Algorithm 2 summarizes the update rules for qr. k,, BQ,,, and BR,,, the computation proxy atten- tion, and the update for selected key and value. The graphical representation of the thorough decoding process of LRQK is shown in Figure 1. To avoid random initialization, the initial guess of k, is utilized by setting An The algorithm iterates until convergence reaches the maximum number of iterations. The attention is computed with selected key and value. Attention (qr, Ko Va,)).
7

---

Algorithm 2 Decode of LRQK
Require: qr, kt. Vr. BQ B,,--I, AaI, Aaz, A Kn,th, V9
Initial Guess Kg f- (,, B (k,t-1
2: while < max iteration not converged do
3: Update q: - M
4: Update ky + (k, K + dn q.) (BK,-IBK,,t Aai (. Gi)
5: end while
6: B < BQ 110 VBQ.-11 with 7g (11) 1K VB I. with 7/K in (12)
7: Bx, BK
8: AK, AK,--1;k;
concatenate I> compute proxy attention
9: top-k index S2g. f- top (A .k)
10: Fetch missing (k;), [vi] from CPU K, V according 2
11: K Vh,.-1 f- merge fetched {ki) {vi) with GPU K Va.-1
12: KR,1 f ( [i]. Va, + [
13: Async kG, Va to CPU
14: return K V9,t-
5 Experiments
In this section, series of experiments are conducted rigorously evaluate the effectiveness and performance of our proposed method. The experiments are performed on NVIDIA A100 GPUs. The number of max iteration and the tolerance introduced in Algorithm and 2 are chosen and 0.01, respectively. All scaling parameters are set as XpQ = >pp = Ar Aas 1. All LLM evaluations are empowered by OpenCompass [27]. Additional results can be found Appendix B, the time cost analysis is presented in Appendix C, and a guideline for hyperparameter tuning is provided in Appendix D.
5.1 Accuracy Evaluation
The accuracy of the proposed method is evaluated on the RULER dataset [28], using long-context setting with a sequence length 128K tokens. For the low rank approximation, to 32. The number of top-k tokens selected based on attention scores is set to 2048 (1.56% of 128K), while the number of lite tokens (the most recently generated tokens) is set to 64. The backbone language model used in the evaluation is LLaMA-3-8B 1-M-A----B-IN 1M [24]. This method is compared against four recent dynamic sparse attention baselines: Loki [21], InfiniGen [14], Quest [20], and ShadowKV [16]. Results are shown in Table 1.
rank set
Table 1: Comparison of different models and methods on RULER (left) and LongBench (right).
Methods
SI
$2
MK1
MK2
MQ
MV
QA-1
QA-2
VT
FWE
NQA
MQA
GRep
SAM
PRetr
LCC
IJama-3-8R
100.00
100.00
98.96
98.96
98.96
95.57
75.00
48.96
78.54
71.85
18.98
41.84
34.18
35.96
81.50
56.07
Loki
18:75
1.04
2.08
0.00
1.56
0.78
4.17
13.54
26.04
25.35
2.26
10.19
28.97
7.84
40.52
31.4
InfiniGen
54.95
38.58
Quest
100.00
100.00
98.96
77.08
97.65
93.49
60.42
50.00
77.08
65.63
20.13
36.63
27.11
35.63
79.00
53.64
97.92
52.08
17.17
31.62
Proposed
81.00 100.00
97.00
42.00 99.25 98.00
75.00
53.00
80.20
69.67
16.52
40.48
20.40
26.35
89.00
66.13
Table I presents the comparative performance of various models and methods on both the RULER and LongBench benchmarks. Our proposed method demonstrates competitive results across a wide range of tasks, particularly excelling in several key domains.
the RULER benchmark, our model achieves perfect accuracy on the S2 (NIAH Single 2) task, matching the performance of larger models like ShadowKV and Quest. Notably, our method outperforms all baselines on QA-1 (QA SQuAD) and QA-2 (QA HotpotQA). the MQ (NIAH MultiQuery) and MV (NIAH MultiValue) tasks, which evaluate a model's ability process multiple

---

queries or values within single instance, our method even surpasses the original LLaMA-3-8B-IN model, achieving results of 99.25% and 98 08., respectively.
For the LongBench evaluation, our approach demonstrates notable gains in tasks such as PRetr (Passage Retrieval) and LCC, outperforming all other methods with 89.00% and 66.13%, respectively, These tasks are partieularly demanding due to their emphasis on retrieving and reasoning over long-range dependencies, suggesting that our method is well-suited for these kinds of tasks.
While some tasks like MK2 (NIAH MultiKey 2) and NQA (NarrativeQA) show slightly lower performance compared to leading baselines, the overall consistency and high scores across multiple complex tasks underscore the effectiveness our approach. These results affirm the potential of our method in addressing long-context language understanding challenges.
The proposed method further evaluated on two backbone models: LLaMA-3-8B-IN [24] and Swen2.5-7B-Instruu [2], using the same configuration of rank r = 16, top-k 256 (6.25% of 4K), and 16 lite tokens on subset of RULER-4K. As shown Table 2, our method consists the performance of both models.
Table 2: Results on subset of RULER 4K of two models
Methods
QA-1
QA-2
VT
Methods
QA-1
QA-2
VT
-LLMMA---B-IIM
82.00
58.00
99.00
Qwen2.5-7B
90.00
64.00
99.20
LRQK
84.00
57.00
98.80
LRQQ
91.00
65.00
96.60
5.2 Impact of Rank and Top-k Selection
In this subsection, the impact of two key hyperparameter' in the proposed method investigated the rank used in the low rank approximation, and the number of top-k k active tokens selected based on attention scores.
Rank Selection. To isolate the effect of the rank, we fix the number of top-k tokens at 256 and the number of lite tokens at 16. The rank are vary over the set {16, 24, 32), allowing us to examine how the capacity of the low rank representations affects model performance.
Top-k Selection. To analyze the influence of the number of active tokens, we fix the rank at 8 and the number of lite tokens at 16. The top-k parameter is varied across {256, 512, 1024] to evaluate how size of the active set impacts retrieval effectiveness and downstream task accuracy. All experiments are conducted on a representative subset of the RULER-4K dataset using the L.LMA-3-8B-IM model as the backbone.
Tables 3 and show that increasing the rank generally improves performance on QA-1 and VT, with r = 32 even surpassing the original LLaMA-3-SBIIN on QA-1. In contrast, QA-2 performs best at lower rank r 16, suggesting some tasks benefit from more compact representations. For top-k, larger values consistently lead to better accuracy across all tasks. The configuration with k = 1024 achieves the highest scores, indicating that broader active token window improves context modeling. However, larger top-k tokens will increase both computation and memory costs, necessitating trade-off between performance and resource usage. Overall, while higher ranks and larger top-k values enhance performance, smaller settings (e.g., 16, k 512) still offer strong results with reduced resource usage.
Table 3: Accuracy across different rank values with top-k 256 and 16 lite tokens
Table 4: Accuracy across different top-k val- ues with r = 8 and 16 lite tokens
Models
QA-1
QA-2
VT
Models
QA-1
QA-2
VT
Llama-3-8B-11
82.00
58.00
99.00
Llama-3-88 3-8B-IM LM
82.00
58.00
99.00
8
79.00
50.00
56.80
top-256
79.00
50.00
56.80
16
80.00
60.00
98.80
top-512
78.00
61.00
95.60
r 24
83,00
56.00
98.80
t0p-1024
83.00
63.00
100.00
32
84.00
57.00
99.00

---

5.3 Miss Rates
The performance of the proposed strategy for memory management evaluated by computing the miss rate. The rate is a ratio Cmiss/ Grotal, where Cmiss denotes the number of KV cache rows that must be transferred from CPU GPU, and Gutal the number of all selected indices. This metric quantifies the efficiency of the token selection mechanism in reducing memory transfers.
Experiments conducted the summarization task using the wikitext-2-vl test set [25]. A grid search is performed over different hyperparameters, including the rank r E {8, 16, 32, 64), the number of active tokens E {128, 256, 512). and the number of lite tokens {4, 8). The distribution of miss rates is illustrated in Figure 5, which reveals an approximately Gaussian shape with a mean miss rate of around 0.40 and standard deviation of approximately 0.10. This implies that, on average.
the proposed hit- ait-and-miss miss mechanism reduces the amount of data transferred from CPU to GPU by about 60% in average.
16:00
1430
1330
3300
1890
200
0mer22503 24:4600 2006
Figure 5: Histogram of miss on wikitext-2-v1.
6
Conclusion
Low Rank Query and Key (LRQK) attention, a two stage inference algorithm that enables long- context processing through joint low rank decomposition of query and key matrices combined with mixed GPU-CPU cache management is presented. By computing proxy attention scores on compact rank-r factors and selectively fetching only the top-k active tokens and most recent tokens' full- precision kay-value pairs, LRQK preserves exact attention outputs while reducing CPU-GPU data transfers.
Extensive evaluations on RULER (up to 128K tokens) or LongBench benchmarks demonstrate that LRQK matches exceeds state-of-the-ard sparse attention methods across diverse long-context tasks. The method achieves substantial memory savings, enabling processing of contexts that would otherwise cause out-of- of-mmoory errors, without significant accuracy degradation. Ablation studies confirm the effectiveness of each component: low rank approximation, active token selection, and lite token retention all contribute to the method's robust performance.
Limitations. While LRQK significantly reduces data transfer overhead, further observation reveals that CPU-side indexing operations constitute the primary performance bottleneck rather than PCIe bandwidth limitations, and hoping can addressed in future work. Additionally. LRQK's hyper- parameters require task-specific tuning. While practical guidelines and default configurations provided, optimal settings still demand empirical validation for new domains.
Societal Impact. LRQK's reduced inference costs could enable broader access to long-contexs language models, benefiting research and education. However, this efficiency may also lower barriers to deploying harmful applications such as large-scale disinformation campaigns. We believe democratized access to advanced Al capabilities provides net benefits when coupled with appropriate safeguards and responsible deployment practices.
Acknowledgments and Disclosure of Funding
This study was supported in part by Natural Science Foundation of China under Grant 62203124, JSPS KAKENHI Grant Number JP23K28 109 and JP24K20849, and Moonshot R&D Grant Number JPMJMS2239. Yuning Qiu was supported by the RIKEN Special Postdoctoral Researcher Program. Tenghui Li was supported by RIKEN's IPA Program.
10

---

References
[1] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, al. GPT-4o system card. arXiv preprint arXiv:2410.21227, 2024.
[2] Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm. heaiiiggggguu.... github.io/bloo............oooo.e http://wee2.////// 5/.
[3] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The Llama 3 herd of models. CoRR, abs/2407.21783, 2024, doi: 1048550/ARXI.:40722177.878 URL cattpp://dol: org/ 48550/arXiv 2407 21783.
[4] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[5] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.1944 19437, 2024.
[6] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length LLM inference with KVeache quantization. Advances in Neural Information Processing Systems, 37:1270- 1303, 2024.
[7] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances Neural Information Processing Systems, 1-22116--7718, 2022.
[8] Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. KVcache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3331, 2024.
[9] Shibo Jie, Yehui Tang, Kai Han, Zhi-Hong Deng. and Jing Han. Specache: Speculative key-value caching for efficient generation of lims. arXiv preprint arXiv:2503.16166 2025.
[10] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H₂O: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710 2023.
[11] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Sun, editors, International Conference on Representation Learning, volume 2024, pages 21875 21895, 2024. URL https: /ttpc//e/occeeeessseeeeee iclr. cc/ cc/paper paapr_ffllesllll files/ paper/2024/ wtt.a///ee...eeee..eeoeeeeeeeee. /
Paperr----------- Conference pdf.
[12] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-22970, 2024.
[13] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with single GPU. In International Conference on Machine Learning, pages 31094-31116 116.. PMLR, 2023.
[14] Wonbeom I Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 155-172, 2024.
11

---

[15] Reza Yazdani Aminabadi, Samyam Rajbhandari. Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, al. Deepspeed- inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2022.
[16] Hanshi Sun, Li- Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. In Forty-second International Conference on Machine Learning, 2025. URL
[17] Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li. CAKE: Cascading and adaptive KV cache eviction with layer preferences. arXiv preprint 2Xiv:2503311291,, 2025.
[18] Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. AVa-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. arXiv preprint arXiv:2407. / 1550, 2024.
[19] Minghui Liu, Tahseen Rabbani, Tony O'Halloran, Ananth Sankaralingam, Mary-Anne Hartley, Brian Gravelle, Furong Huang, Comella Fermüller, and Yiannis Aloimonos. Hashevict: A pre- are-attention KV cache eviction strategy using locality-sensitive hashing. arXiv preprint 12X1::2412.16187, 2024.
[20] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint 4r01::2406.10774 2024.
[21] Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low- rank keys for efficient sparse attention. Advances in Neural Information Processing Systems, 37:16692-16723, 2024.
[22] Chi-Chih Chang, Wei-Cheng Lin, Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed Abdelfattah, and Kai-Chiang Wu. Palu: Compressing KV-cache with low-rank projection. arXiv preprint 24XXiv:207.21112 2024.
[23] Xingtai Lx, Ning Ding, Kaiyan Zhang, Ermo Hua, Ganqu Cui, and Bowen Zhou. Scalable efficient training of large language models with low-dimensional projected attention. In Yaser AI-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14588-14599, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2022 emnlp-main.803 URL https: http:p::///aaaaaaii http///aaaaatttooooooooooooo hwwpechoologg.oog//222222 org/ www.p-main 808/.
[24] Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. Llama 3 gradient: series of long context models, 2024. URL. https::/grradiett.aat ai/blg//
[25] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.07843, 2016. URL http:tp:/// /ttttp::////xvv../// wwwpxiiiorr/abbb/eeeeee http///..111090074....... 07843.
[26] Joshua Ainslie, James Lee-Thee Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from meati-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4895 4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.18653/v1/221 1/2023. 11t.55//1112222..mmmmmmma......
[27] OpenCompass. Opencompass: A universal evaluation platform for foundation models, 2023. URL https:
com/ open- comppsss..
[28] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's 's the real context size of your long-context language models? arXiv preprint arXiv:2404.0665.. 2024.
[29] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint 3ttt::2309000777.. 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly describe the two-stage LRQK frame- work, namely, the joint low rank decomposition of query/key matrices and the hierarchical GPU-CPU cache with hit-and-miss token selection-ande and claim both exact attention preser- vation and substantial memory reduction. These points are rigorously supported by the derivations in Sections 3 and 4, the complexity analysis, and the empirical results on RULER and LongBench, with full details the appendix. Thus, the stated contributions match the theoretical development, algorithms, and experiments without overstatement.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification; The paper includes dedicated "Limitations" paragraph that acknowledges two key drawbacks the need to carefully tune the rank and active set size k, whose optimal values can vary across different models and tasks, and the inference is slightly slow.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
13

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results-iiccucde the low-rank prefill objective and the decod- ing Lagrangian are stated with their full assumptions (e.g. rank-r factorizations, invertibility of r matrices) and are followed by complete, step-by-step derivations of the closed-form update rules in Appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper gives full details for reproduction, including backbone models (LLaMA-3-8B, Qwen2.5-7B), datasets (RULER, LongBench, WikiText-2), context lengths (128K, 4K), hyperparameter settings (rank r, top-k, lite tokens), iteration counts and toler- ances, hardware setup (NVIDIA A100 GPUs; NVIDIA 3090s), and the evaluation toolkit (OpenCompass). also provides clear algorithm pseudocode and detailed derivations the appendix, ensuring all components needed to replicate the experiments are specified.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
14

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The authors release a public GitHub repository (https://githen) com/ cenghuilee/Lnne containing all implementation codes, and evaluation scripts. The supplemental material includes detailed environment specifications (library versions), exact command line invocations for evaluation, and processing instructions for datasets, ensuring that others can faithfully reproduce the reported results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.cm cc/
htt///gddes/oddeeeeeeee ceabbbiissiooooicyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.comm wttpccccppppccccccccccccccccccct weciccgguiiiii gewdee/iieeeee het////ooeuuiisseeeeeeeee miionniicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Section 5 and Appendix C give complete details: hardware (NVIDIA A100
GPUs), backbone models (LLaMA-3-8B-IM Qwen2.5-7B), datasets and context lengths 4K subset, WikiTeext22 for summarization), hyperparameter settings (rank r, top-k, lite tokens, max iterations 2, tolerance 1e-2), grid-search ranges, and evaluation toolkit (OpenCompass). Algorithms pseudocode provide full pseudocode for reproducing the inference procedure, ensuring all necessary information is specified. Guidelines:
(128K for RUL.RR/LongBenee
The answer NA means that the paper does not include experiments.
15

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper provides extensive experimental results across multiple datasets, models, and settings, which offer broad statistical perspective, While it may not include explicit error bars, the consistent performance trends and comprehensive comparisons serve as strong empirical evidence.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eиrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper reports that all experiments were conducted on NVIDIA A100 GPUs, specifies the batch sizes 1, context lengths (4K, 128K), and hardware setup in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http//eurip...e httppppp...pppppieitttciiccaaaaa
16

---

Answer: [Yes]
Justification: This work proposes an algorithmic optimization for LLM inference without involving human subjects, personal data, or ethically sensitive applications. It presents transparent methods and evaluations, poses no privacy or fairness risks, and adheres to NeurlPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The proposed method is technical improvement in LLM inference, and does not involve any sensitive content, human data, or high-risk applications. The proposed method reduces inference costs could enable border access to long-context language models. However, this efficiency may also lower barriers to deploying applications such as large-scale disinformation campaigns. We believe democratized access advanced AI capabilities provides net benefits when coupled with appropriate safeguards and responsible deployment practices.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
The paper does not release new high-risk models or datasets. It builds on existing pretrained models (e.g., Qwen, LLaMA) and standard datasets, which do not require additional safeguards beyond those already established by their original authors. However, the proposed method reduces the cost of LLM inference, and it may be wrongly used.
Justification:
17

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits the original sources of all models, datasets, and methods used, including citations to Qwen, LLaMA and others. There is no indication of license violations, and usage aligns with standard academic practice.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, sepperswithcode www/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The newly proposed method, LRQK, is well documented. The paper provides theoretical foundations, implementation details, and extensive experimental settings. The re- leased code includes instructions to reproduce results, ensuring the assets are accessible and usable.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
18

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does involve any crowdsourcing experiments or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study does involve human participants or subject data, so there are no associated risks or requirements for IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Although the paper evaluates methods on pretrained LLMs such as LLaMA and Qwen, these are standard models used for benchmarking. The core methodological contribution does not involve original or non-standard usage of LLMs. We use LLM only for writing, editing purposes and does impact the core methodology, scientific rigorousness, or originality of the research, so no special declaration is required.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaadd components.
Please refer to our LLM policy (https: : wctpe/ooeeeennce/22/////11111 for what should or should be described.
19

---

Appendix
A Details of Derivatives
A.1 Prefill Derivatives
The Lagrangian (5) is formulated as:
Cpre QOK AlA Ape |Q 2B-ABoll
Apk |K AxBulli
Compute partial derivative:
ppre (QKJ AQAR) AK - Ane (Q ArBB)
aAQ
dh
0AK (QKT AQAR) AQ APk (K A&BK) B
(13)
0pr
i A (Q - AQBo)
0p ApR (K ARBK)
0B
Setting the partial derivative to be zero,
Ocpre == O,
dLrre = 0, dAk
0h =0 0. ²
0C == 0.
(14)
dAq
0B
Solving the equations (14), we get:
AQ = Q (KT AK + AO) (AKAK + XpQ -1
Aς = K (QT Ao + +Bx (ATAQ + BB
BQ -AAAA))~A00
Bą = AA) AA
A.2 Decode Derivatives
The Lagrangian (8) is formulated as:
Ldce
q111 1
k K - -ki ko]l
2
Adi
@ack, qik, )²
2 +
2
Compute partial derivative:
dLscc
(Q.BQ.-1-Qu B
oq.
+ Au (qite. - qik," ki + daz (q.A T 1 - q.KT..-)) AK (15)
HL (k,BK,t-I- k1) B B ddi (qckk gike qr
Ok.
By setting the partial derivative to be zero, we get:
q B + Aark, kg A A Aka = q. Aai K 1
20

---

i - (-. dai A ( (Bq,s-I + +dikk, + A
Similarly, we have:
G.Bx,--IBk. 1 + Aa1 ki TT
9i k, + Aen k q:
k, [Bx,t-BBa. BK..-I P
T qt) (k,B + dPy q qt)
k [k,BBK,-1 AP k. q:) (Bx,-IBKKt-
TT Da qr)
Update BQ and B First, compute the partial derivative with respect to B and B B,t-1,
dLox
qi (q:BQ,1-1 qi) = (0) G.q.... - q. qr.
@BQ,--1
dLeec kj (k.BKI11 k.) = k, B,B - kj k.
0B
Since the outer product of row vector, q E xR is rank 1 matrix, it is not invertible. Therefore, there is no closed-form solution for both BQ,t-1 and B
To update BQ.-- and Bx,-- gradient descent is utilized. The gradient of Lave with respect to Bo.t- and B are:
VBQ = q. T (q.Bor - -q)
(16)
VBKI-1 kg (k,BK.-1 k1)
The gradient descent update rules are:
B BQ - 110 VBQ,t-1,
(17)
BRG < B 7ΙK V K
where 1i and vj are the learning rates for BQ and B, respectively, Plugging the update rule of BQ into the Lagrangian (8), have,
1 l|q. (BQ.-1 nQ VBQ q 1l
arg min
(18)
2 To
This is convex quadratic optimization problem with respect to the learning rate nq. Therefore, the optimal 11 is given by,
(aBQrr1 q: 12 VBQ,t-1) ( (Q. 0.
(19)
And the optimal n2 will be.
( (Q. 9:) (q: VB VB..-1
n
(20)
( VQ,,t1 (q: V VBQ.1-11
Similarly, for B the optimal 1/k can be computed as,
(k.BK.o1 - ki) (K.BK.,-I)
V
(21)
(k.VBK--I) (k.BBK.-))
B Additional Results
B.1 Results on RULER 128K And LongBench
Since the inverse computation in PyTorch operates in float32, the parameters are temporarily cast to float32 before executing Algorithm and Algorithm 2. The models are otherwise run with bfloat16 for inference.
The CPU used an AMD EPYC 7742 64-Core Processor, featuring 64 cores with threads per core. For the RULER 128K experiment, the model runs on single NVIDIA A100 GPU with 80 GB of memory. For the LongBench experiment, the model executed on a single 1100 GPU with GB of memory. The batch size is set to one. Further results are provided in Table 5.
21

---

Table 5: Comparison of different models and methods on RULER 128K (left; t0p-2048, lite to4ens=64) and LongBench (right; rank r = 16, lite tok
Methods
51
S2
MK1
MK2
MQ
MV
QA-1
QA-2
VT
FWE]
NQQ
MQA
GRep
SAM
PRetr
LCC
Llama- 3-8B-IM
100.00
100.00
98.96
95.57
48.96
78.54
18.98
41.84
34.18
35.96
81.50
56.07
Loki
18.75
1.04
2,08
0.00
1.56
0.78
4.17
13:54
26.04
25.35
2.26
10.19
28.97
7.84
40.52
31,44
IndiniGon
100.00
98.96
84.38
53.13
63.28
54.95
65.63
48.96
81.67
50.35
14.39
31.46
27.38
21.97
74.30
38.58
Quest
100.00
100.00
98.96
77.08
97.65
93.49
60.42
50.00
77.08
65.63
20.13
36.63
27.11
35.63
79.00
53.64
ShadowKV
100.00
100.00
97.92
98.96
96.88
95.83
72.92
52.08
81.67
72.57
17.17
39.73
31.62
35.87
80.00
63.93
LROK 32
81.00
100.00
97. 00
42,00
99.25
98.00
75.00
53.00
80.20
69.67
1op1
16:52
40.48
20.40
26.35
89 00
66.13
LROK = 16
63.67
69.00
50.00
90.00
80.25
81.50
70.00
56.00
64.00
63.67
1op2018
15.86
40.15
34.15
36.64
91.50
66.53
Llama 3. LBB
100.00
100.00
98.96
91.67
98.96
95.31
82.29
47.92
68.96
71.18
31,56
55.10
34.45
29.84 84
100.00
67.31
Loki
68.75
32.29
32.29
20,83
42.71
28.
41.67
33.33
24.79
29.86
2.31
18.89
31.16
15.91
94 34.88 44.60
InfiniGon
100.00
77.08
78.13
13.54
58.07
47.4
37.40
65
65.63
41.67
60.83
50.35
27.23
52.72
29.61
24,42
98.93
56,89
Quest
100.00
98.96
97.92
34.38
93.49
88.54
70.83
44.79
65.63
68.40
29.70
49.04
30.43
29.85
98.50 57.35
ShadowKV
100.00
100.00
100.00
83.33
92.19
81.25
48.96
67.08
64.93
30.93
55.20
32.79
30.40
99.50 66.03
LRQK
32 87.00
100.00
98.00
80.00
98.25
96.25
81.00
41.00
70.00
44.67
1op1
|
29.01
55.85
19.50
12.02
99.50 24.60
B.2 Results on Llama And Qwen
More results of the proposed methods on two different models, Llama-3-8B-IM and Qwen2.5-7BB are shown in Table 6. The model is running on a single A 100 GPU with 40G memory. The configuration for LRQK are: rank=16, 20-2556 active tokens, and 16 lite tokens.
Table 6: More results on subset of RULER 4K
of two models.
RULER 4K
QA-1
QA-2
VT
QA-1
QA-2
VT
LLaMA-3-8B-IN
82.00
58.00
99.00
Qwen2,5-7B
90.00
64.00
99.20
+ LLRQ
84.00
57.00
98.80
+ -LRQK
91.00
65.00
96.60
RULER 8K
QA-1
QA-2
VT
RULER 8K
QA-1
QA-2
VT
LLaMA-3-8B-IAM
82.00
57.00
99.60
Qwen2,5-7B
82.00
$7.00
97.40
+ -LRK
79.00
47.00
94.00
+ -LROK
78.00
56.00
70.00
RULER 16K
QA-1
QA-2
VT
RULER 16K
QA-1
QA-2
VT
LLaMA-3-8B-II
80.00
57.00
99.80
Qwen2.5-7B
76.00
63.00
98.80
+ LRQK
77.00
52.00
57.20
+ LRQK
71.00
62.00
64.20
B.3 Results on Mistral and Phi-3
Two additional models, 'mistralai/Misttall ----sssttrr-------nuttt------- (Mistral) and Sitcroosott/hi---miie 128k-instruct (Phi-3), are evaluated under the default LRQK hyperparameters (rank=32, top-2048. and 64 lite tokens). Due to the 48 GB memory constraints of the NVIDIA A6000 GPU, Mistral is evaluated RULER 32K and Phi-3-mini is evaluated RULER 16K. Table 7 summarizes the results, demonstrating that LRQK generalizes effectively to architecturally diverse models.
B.4 Results on Larger Models
To evaluate the scalability of LRQK to larger models, additional experiments are conducted on "Qwen'Qwen2.5-142" Instruct' Innsttruuttt (Qwen 14B) with 64K context length and "Qwen/Qwen2.5-23 Instruct' (Qwen 32B) with 16K context length under A100 80G GPU. Following the official guidance from Qwen2.5, YaRN [29] is applied for positional encoding extension in the 64K context setting, All experiments use the default LRQK hyperparameters (rank=32, top-2048, and 64 lite tokens).
Table 8 presents the results on the RULER benchmark. For Qwen 14B at 64K context, LRQK yields substantial improvements on retrieval tasks, demonstrating its effectiveness at extended context lengths. For Qwen 32B at 16K context, the baseline model already achieves near-perfect performance on most tasks, with LRQK maintaining comparable results, indicating that the method does not degrade performance on models with strong native long- long-context capabilities.
22

---

Table 7: Evaluation results of Mistral and Phi-3 RULER benchmarks with varying context lengths.
RULER 32K
Mistral
+LRQK
RULER 16K
Phi-3
+LRQK
FWE
67.00
93.00
FWE
92.00
91.00
S1
98.00
97.00
S1
100.00
100.00
S2
91.00
100.00
S2
100.00
100.00
MK1
83.00
97.00
MK1
96.00
96.00
MK2
63.00
51.00
MK2
100.00
100.00
MV
85.75
95.00
MV
90.00
90.25
MQ
83.75
93.00
MQ
90.50
87.50
QA-1
59.00
63.00
QA-1
80.00
81.00
QA-2
45.00
47.00
QA-2
51.00
51.00
VT
98.40
96.40
VT
99.60
99.60
Table 8: Performance comparison on Qwen2.5-144 (16K context) with and without LRQK on the RULER benchmark. Columns '+LRQK' indicate the performance of the model with LRQK.
(64K context) and Qwen2.5-32B-Insttee Instrucct
RULER 64K
Qwen 14B
+LRQK
RULER 16K
Qwen 32B
+LRQK
FWE
90.33
78.33
FWE
94.67
94.00
S1
57.00
99.00
S1
100.00
100.00
S2
55.00
96.00
S2
100.00
100.00
MK1
46.00
80.00
MK1
100.00
100.00
MK2
24.00
22.00
MK2
99.00
99.00
MV
54.00
82.25
MV
100.00
100.00
MQ
49.75
88.50
MQ
100.00
100.00
QA-1
64.00
50.00
QA-I
86.00
86.00
QA-2
41.00
36.00
QA-2
66.00
66.00
VT
57.80
97.20
VT
86.40
82.60
B.5 Results With KVQuant
To investigate whether LRQK is compatible with quantization based compression methods, the combination of LRQK with KVQuant [6], a representative KV cache quantization approach, is (Llama-3.1-8B) and using the RULER 32K benchmark. The LRQK hyperparameters set to 32, top-k 2048, and 64 lite tokens.
evaluated. Experiments conducted 'meta- Llama-3.1-888 (wven//wen2.5577--mmee
(Qwen
Table presents the results comparing models with KVQuant alone versus the combination of KVQuant and LRQK. For 1lama-3.1-88 the combined approach maintains competitive performance across most tasks. For Qwen2.5 BB, combining both methods yields mixed results: while some tasks show improvements (e.g., MK1, MV, MQ), others exhibit slight degradation (e.g.. MK2). Overall, the results demonstrate that LRQK can be effectively combined with quantization methods without catastrophic performance loss, suggesting that the proposed method and quantization operate complementary aspects of KV cache compression.
B.6 Impact of Initialization Strategies
Three initialization strategies are investigated the low rank factors Aq R and Ak Rixr. where is the sequence length, and is the chosen rank, and the batch size or the number of attention heads are ignored as discussed Section 4. The design rationale these strategies stems from goal of finding effective initialization that either minimize reconstruction error provide sufficient representational capacity for the low rank approximation.
Random Gaussian Initialization (randn). The first strategy initializes both factors with random Gaussian noise:
Aq. Ax N(,,
(22)
23

---

Table Performance comparison on RULER 32K with KVQuant applied alone and in combination with LRQK. Columns '+KVQuant' indicate K VVQuant only, and columns '++LRQK' indicate both KVQuant and LRQK applied.
1lama-3.1-88
Qwen2.5-7B-
RULER 32K
+ KVQuant
++LRQK
+ +KVQuant
++RQQK
FWE
89.58
85.00
94.00
90.33
S1
100.00
100.00
100.00
100.00
S2
100.00
100.00
100.00
99.00
MK1
100.00
100.00
96.00
100.00
MK2
95.00
96.00
76.00
56.00
MV
95.31
96.75
70.50
93.50
MQ
89.00
89.00
97.00
99.25
QA-1
83.00
82.00
73.00
74.00
QA-2
56.25
51.00
52.00
53.00
VT
87.50
95.40
95.40
90.60
This approach provides a simple, parameter-free initialization that avoids any dependence on the input matrices. While it does not leverage information from Q and K, it offers computational efficiency and serves as a neutral baseline.
Independent Top-r Selection (top). The second strategy aims to minimize reconstruction error by selecting the most salient dimensions from Q and K independently. The L1 norm is computed of each dimension across the sequence:
SQ
Qli ll
SK
K ER
(23)
where is the head dimension. With these importance scores, the initialized low rank factors constructed as:
Aq + (sp)) Ax + K (spr)ss)
(24)
This strategy independently optimizes each factor capture the most significant features of its respective matrix.
Joint Top-r Selection (topeol). The third strategy selects dimensions based on the combined importance in both Q and K, ensuring that Aq and A x share the column indices:
SQK SQ + SKs
Ω top-r(sqk);
(25)
AQ
Qe,
Ak K
(26)
This joint selection promotes alignment between the query and key subspaces, potentially facilitating more coherent attention patterns in the low rank approximation.
Experimental Setup and Results. These three initialization strategies are evaluated on 'meta- Ilama/Llama' 3.--a--.--8B----- 1------Istrucc--N and D0ven//Qwen22557B-Innnnee mark with default LRQK hyperparameters (r 32, top-k = 2048, 64 lite tokens).
using the RULER 16K bench-
Table 10 presents the results. Remarkably, all three initialization strategies achieve nearly identical performance across most tasks, with differences typically within 1-2 percentage points. This robust- ness suggests that the specific choice of initialization has minimal impact on the final performance, indicating that the Algorithm is resilient to the initialization the low rank factors. The consistency across strategies, from uninformed random initialization to carefully selected subspaces, demonstrates the method's 's inherent stability.
Given this empirical equivalence, the randn initialization is recommenced as default due to its computational efficiency: it avoids the overhead of computing importance scores and performing top-k selection, making it particularly suitable for large-scale deployments.

---

Table 10: Performance comparison of initialization strategies for AQ and Aς on RULER 16K.
Despite their different design rationales, all three methods achieve comparable results.
1lama-3.1-888 MM
Qwen2.5-7B-Instreee
RULER 16K
randn
top
topcol
randn
top
topcol
FWE
86.67
86.67
87.00
85.67
86.33
86.33
S1
100.00
100.00
100.00
100.00
100.00
100.00
S2
100.00
100.00
100.00
100.00
100.00
100.00
MK1
100.00
100.00
100.00
99.00
99.00
99.00
MK2
99.00
99.00
99.00
91.00
91.00
91.00
MV
98.75
98.50
98.50
97.75
98.00
97.75
MQ
94.75
94.50
94.75
99.75
99.75
99.75
QA-1
89.00
89.00
89.00
76.00
76.00
76.00
QA-2
61.00
61.00
61.00
61.00
62.00
62.00
VT
83.80
83.00
85,40
98.80
98.80
98.40
C Runtime Performance Analysis
C.1 Throughput Results
Runtime performances of LRQK are evaluated on 'meta-llama/LLlma-3/. ----maaa-..-------- L------InttutttII1 IM' compar- ing it against standard GPU-only and CPU offloading approaches. Experiments are conducted using the Hugging Face transformers ormers library² on single NVIDIA A100 GPU (40 GB memory) with batch size 1. We utilize the RULER QA-2 (HotpotQA) dataset with context lengths ranging from to 64K tokens. The parameters of LRQK are set as = 16, top-k = 1024 active tokens, and 16 lite tokens.
Experimental Configurations:
GPU only: All KV caches stored in GPU memory (baseline).
CPU offload: KV caches offloaded to CPU memory and transferred to GPU during decod- ing.
LRQK default: Full LRQK implementation with hit/miss buffer enabled for optimized cache management.
LRQK no hit/miss: LRQK without hit/miss buffer to isolate the impact of buffer optimiza- tion.
Table 11 presents the tokens processed per second during prefill (P) and decode (D) stages. The GPU-only method achieves the highest throughput for shorter contexts (4K to 32K) but encounters out-of-memory (OOM) errors at 64K tokens due to the growth of KV cache size.
CPU offload enables processing of longer contexts but suffers severe throughput degradation, particu- larly during decoding at 32K to 64K tokens. This performance drop stems from the need to transfer the entire KV cache between CPU and GPU for each decoding step, creating bottleneck that scales linearly with context length.
In contrast, LRQK default maintains relatively stable decoding throughput across all context lengths, demonstrating consistent performance even at 64K tokens. This stability arises from LRQK's selective transfer mechanism: instead of moving the entire KV cache, only the top-k active tokens and lite tokens are transferred from CPU to GPU, significantly reducing data movement overhead. At 64K context, LRQK achieves faster decoding than CPU offload while avoiding the OOM issues of GPU-only execution.
The LRQK no hit/miss configuration shows reduced decoding performance compared to LRQK default, highlighting the importance of the hit/miss buffer optimization. Without this buffer, the method incurs additional overhead from repeated top-k selection operations, which are not well- optimized for CPU cache locality. Notably, prefill performance remains comparable across LRQK variants, this stage is less sensitive to cache management strategies.
http:///wgggggaaeeooo/////c///// v4. 52. wttp://eeeennnaeeeeeeeaaeeeeeeee wlwssee/text_geeeeeeeee
25

---

Table 11:
Throughput comparison (tokens/s) for Imeta-llama/Llmma 1------nnsttuct-. IM' on RULER QA-2 across context lengths. P: prefill stage; D: decode stage. LRQK default maintains stable decode performance while avoiding OOM at 64K tokens.
4K
8K
16K
32K
64K
D
P
D
GPU only
37500.29
35.40
37734.19
35.64
33649.79
79
35.36
32111 144
35.77
OOM
OOM
CPU offload
6945.10
4.31
6984.35
4.32
7073.53
2.40
4849.19
0.98
4131.00
0.50
LRQK default
7120.49
5.68
7280.39
5.72
6379.91
5.49
5436.66
5.36
4103.05
6.80
IRQK no hit/miss
7180.64
2.50
6747.19
2.53
5317.06
2.31
4920.02
2.48
4163.86
1.95
C.2
Comparison with Baseline Methods
To provide comprehensive performance analysis, LRQK is compared against vanilla attention and Shadow K [V666 on a text summarization task using 20 samples from LongBench with 32K context and up 128 output tokens. Method ShadowKY is configured with its default parameters (sparse budget= = 2048, rank=160, chunk size=8), and the definition of 'rank' is not the same as this paper, please refer to the original paper for more details. Experiments conducted on NVIDIA GeForce RTX 3090 (24 GB) GPUs. Due to memory constraints, vanilla attention requires GPUs, while ShadowKV and LRQK operate on a single GPU.
Table 12 presents detailed performance metrics. Vanilla attention achieves the highest throughput but requires dual-GPU deployment with large memory usage across both two devices. ShadowKV reduces throughput compared to vanilla. LRQK variants achieve 489 to 646 tokens/s depending on hyperparameters, representing trade-off between memory efficiency and speed. Basically, LRQK with less rank and smaller & can achieve larger tokens/s. The GPU power of LRQK smaller than Vanilla and ShadowKV, which means the proposed method is not fully utilizing the GPU resources. This is one limitation of the proposed method. It requires more time to wait data indexing in CPU.
Table 12: Performance comparison NVIDIA GeForce RTX 3090 (24 GB, 250W) for text sum- marization (LongBench, 32K context). Vanilla attention uses 2 GPUs; other methods use GPU. Tokens/s is computed via (number of all tokens total time).
Vanilla
ShadowKV
LRQK
2048
k
1024
GPU0
GPU1
32
r 16
32
16
Average GPU Util (%)
38.61
48.38
58,00
63.64
65.29
56.86
51.15
Max GPU Util (%)
99.95
100.00
100.00
100.00
100.00
100.00
100.00
GPU Memory (GB)
16.94
17.56
17.93
19.82
18.85
19.51
19.33
GPU Power (W)
187.24
233.21
243.85
203.98
211.96
199.59
199.85
Tokens/s
1775.70
1083.55
489.39
556.22
603.05
646.33
Time (s)
17.87
29.29
64.84
57.04
52.62
57.04
D Hyperparameter Selection Guidelines
LRQK introduces several hyperparameters that require configuration: rank for low rank approxima- tion, top-k for active token selection, number of lite tokens, and convergence parameters (iterations and tolerance). While optimal values vary across model architectures and tasks, practical guidelines provided based on our extensive experiments minimize tuning effort.
Default Configuration. It is recommended starting with the following default configuration, which achieves strong performance across diverse settings: rank r = 32, active tokens top-k = 2048, lite tokens 64, iterations 2, and tolerance 10-2, This configuration provides balanced trade-off between approximation quality, memory efficiency, and computational overhead, and serves as an effective starting point for task-specific tuning.
26

---

Rank r. The rank controls the dimensionality of the low rank factors AQ: Ak E R R for each attention head independently. Since typical attention heads have dimension dhead 128. the theoretical range for r is [1, 128). In practice, it is found that E {8, 16, 16,32, 48] can provide a favorable balance between approximation quality and computational cost. Lower ranks (e.g.. r = 8) reduce memory usage but may sacrifice accuracy, while higher ranks (e.g., 48) improve approximation at the cost increased computation. For most applications, 32 offers an effective compromise.
Top-k Active Tokens. The top-k parameter determines how many high-attention tokens are retained in GPU memory and directly influences the memory-contee length trade-off. The optimal & scales with context length: short contexts << 4K) k 256 suffices for most tasks, medium contexts 512 to 1024 may provide good coverage, long contexts (> 32K) k 2048 is
(8K to 16K): recommended. Smaller & values reduce memory footprint but may miss important context, while larger & improves recall the cost of increased GPU memory usage and data transfer overhead. The relationship between l and context length should be considered when deploying LRQK for production cases.
Lite Tokens.
The number of lite tokens specifies how many recent tokens are always retained in GPU memory capture local context. Recommend choices are {16, 32, 64), with minimal impact on overall computational cost. This parameter is relatively insensitive; recommended 64 lite tokens conservative default that ensures sufficient local context coverage.
Convergence Parameters.
The iterative updates Algorithms and 2 require two convergence
parameters:
Iterations: Number of update cycles for low rank factors. Typically, or 3 iterations may suffice, the approximation would converge.
Convergence threshold defined as the mean squared error between successive low rank matrices. Values of 10-2 or 10 10-3 work well practice. Tighter tolerance (e.g.. 10-3) improves accuracy slightly but increases computation.
Tolerance:
Tuning Strategy.
The search space for LRQK hyperparameters is relatively compact compared to other optimization based methods. It is recommended the following tuning protocol:
1. Start with the default configuration (r 32, 2048, 64 lite tokens, 2 iterations, tolerance 10-2).
2. If memory is limited, reduce i proportionally the context length or decrease
3. If accuracy is insufficient, increase k (up to hardware limits) or r.
In our experiments, this strategy typically requires evaluating fewer configurations to identify near- optimal settings for a given task and model architecture.
Implementation and Integration Overhead. LRQK introduces modest computational overhead relative to standard attention:
Prefill stage: Additional cost for computing low rank projections, partially offset by reduced attention computation over selected tokens.
Decode stage: Incremental updates to low rank matrices add computation, but attention over fewer tokens (k instead of full sequence) reduces overall cost.
Cache management: CPU-GPU transfers for selected tokens and cache lookup operations add latency, though this is mitigated by transferring only & tokens rather than the entire KV cache.
27


Reviews:
[{"review_id": "eaGblzXoTk", "reviewer": "Reviewer_GeRa", "summary": "The paper proposes Low-Rank Query and Key attention (LRQK), a two-stage framework for reducing the KV-cache memory footprint during long-context inference in decoder-only large language models (LLMs). For a long prompt, the full-precision query (Q) and key (K) matrices are jointly factorised into rank-r bases by minimising a reconstruction loss (Eq. 3). At each generation step, low-rank projections estimate “proxy” attention scores in O(ℓr) time; only the top-k tokens plus a small recency buffer are kept on-GPU, while the rest of the cache resides on CPU. A hit/miss mechanism fetches full-precision KV rows for missed indices, guaranteeing exact attention on the selected subset. Experiments with LLaMA-3-8B-1M and Qwen-2.5-7B on RULER-128 K and LongBench show that LRQK matches or beats shadow-KV, InfiniGen, QUEST and Loki on most tasks, e.g., +2 pp on QA-SQuAD versus the 8B baseline while cutting CPU↔GPU transfers by ≈60 % on WikiText-2 . Hyper-parameter sweeps reveal favourable rank/top-k trade-offs (r = 32 and k = 1024 give near-lossless accuracy)", "questions": "1. Section 5 reports “≈ 60 % reduction in CPU↔GPU transfers” but gives no tokens-per-second or latency figures on the A100 node you mention (lines 210-216). Could you provide end-to-end throughput (tokens/s) and peak GPU memory for LRQK vs. ShadowKV, InfiniGen and vanilla attention at 128 K tokens?\n\n2. All experiments use 7-8 B parameters. Are there bandwidth or PCIe limits that prevent LRQK from scaling to 30 B+ or mixture-of-experts LLMs where KV memory dominates? Even a profiling statement would help assess significance.\n\n3. Since LRQK already uses low-rank factors, can the retained full-precision KV rows be post-quantised to 8-bit or 4-bit (à la KVQuant) without harming accuracy? Discussing complementarity with quantisation would broaden impact.", "limitations": "The authors candidly notes that rank r and active-set size k require task-specific tuning and that optimal values differ across architectures. However, the societal-impact section is missing. Please add a brief discussion on potential misuse (e.g., enabling cheaper deployment of disinformation bots) and mitigation strategies.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper provides a novel and innovate approach to solving the memory constraints in long context inference. Unlike many eviction or sparsity methods, the hit/miss fetch ensures identical outputs to full attention on the retained tokens—no approximation error. \n2. The paper is generally well written and each technique introduced comes with proper motivation. This paper is timely and looks at some key memory issues with kv-caching. \n3. Authors provide strong results. With LLaMA-3-8B-1M at 128 K tokens, LRQK matches or beats strong baselines (ShadowKV, Quest, InfiniGen, Loki) on many RULER tasks and improves Passage-Retrieval and LCC on LongBench (Table 1). I also found the memory-transfer savings results interesting like on WikiText-2 the hit/miss cache cuts CPU ↔ GPU KV transfers by ≈60 % across a broad grid of settings (mean miss-rate ≈0.40, Fig. 5) Moreover, authors have done extensive ablations on rank r, active-set size k, miss-rate, and cache policy across two architectures and multiple long-context benchmarks.\n4. Authors promised to open source the code which will be valuable to the community. \n\nWeaknesses:\n1. Experiments list the hardware (A100) but omit tokens/s throughput, wall-clock latency, and peak/resident GPU memory, making it hard to judge real deployment benefits\n2. All results use LLaMA-3-8B-1M and Qwen-2.5-7B; the method’s scalability to 30 B + or MoE LLMs, where KV caches are a bigger bottleneck, remains unverified. The authors themselves flag that rank r and active-set size k require dataset/model-specific tuning, and optimal values can vary across tasks. I am concerned about the ease of adoption of this technique for more general use cases. \n3. Equation 3 formulation is internally inconsistent. The hard constraints $ Q=A_QB_Q, K=A_KB_K $ in Eq. 3 cannot hold for low rank $r<d$ and are immediately relaxed in the Lagrangian; also, the interaction term omits $B_QB_K^T$. This is mostly cosmetic but weakens the mathematical presentation.", "was_revised": true, "final_justification": "The authors have sufficiently addressed all the concerns I have raised during the rebuttal. I think this paper tackles a timely challenge of solving the memory constraints in long context inference. The novel techniques and the findings presented in the paper will be valuable for the entire community.", "rebuttal": "Thank the reviewer for insightful comments. \n\n> W1. Experiments list the hardware (A100) but omit tokens/s throughput, wall-clock latency, and peak/resident GPU memory, making it hard to judge real deployment benefits\n\nThank you for this valuable feedback. We provide comprehensive timing results on single NVIDIA A100 40GB with Llama-3-8B-1M.\nThe following table shows the results. Further details can be found in the supplemental material.\n\n|           Method |       4K |    4K |       8K |    8K |      16K |   16K |      32K |   32K |     64K |  64K |\n| ---------------: | -------: | ----: | -------: | ----: | -------: | ----: | -------: | ----: | ------: | ---: |\n|                  |     TTFT |  TPOT |     TTFT |  TPOT |     TTFT |  TPOT |     TTFT |  TPOT |    TTFT | TPOT |\n|         GPU only | 37500.29 | 35.40 | 37734.19 | 35.64 | 33649.79 | 35.36 | 32111.45 | 35.77 |     OOM |  OOM |\n|      CPU offload |  6945.10 |  4.31 |  6984.35 |  4.32 |  7073.53 |  2.40 |  4849.19 |  0.98 | 4131.00 | 0.50 |\n|     LRQK default |  7120.49 |  5.68 |  7280.39 |  5.72 |  6379.91 |  5.49 |  5436.66 |  5.36 | 4103.05 | 6.80 |\n| LRQK no hit/miss |  7180.64 |  2.50 |  6747.19 |  2.53 |  5317.06 |  2.31 |  4920.02 |  2.48 | 4163.86 | 1.95 |\n\n\n> W2. All results use LLaMA-3-8B-1M and Qwen-2.5-7B; the method’s scalability to 30 B + or MoE LLMs, where KV caches are a bigger bottleneck, remains unverified. The authors themselves flag that rank r and active-set size k require dataset/model-specific tuning, and optimal values can vary across tasks. I am concerned about the ease of adoption of this technique for more general use cases.\n\nThank you for this insightful question. We provide further validation on larger models: `Qwen/Qwen2.5-14B-Instruct` (64K context) and `Qwen/Qwen2.5-32B-Instruct` (16K context).\nYaRN is applied on 64K context. LRQK use default settings.\nResults are shown below.\n\n|       RULER 64K | Qwen 14B | Qwen 14B +LRQK |       RULER 16K | Qwen 32b | Qwen 32B +LRQK |\n| --------------: | -------: | -------------: | --------------: | -------: | -------------: |\n|             fwe |    90.33 |          78.33 |             fwe |    94.67 |          94.00 |\n|   niah_single_1 |    57.00 |          99.00 |   niah_single_1 |   100.00 |         100.00 |\n|   niah_single_2 |    55.00 |          96.00 |   niah_single_2 |   100.00 |         100.00 |\n| niah_multikey_1 |    46.00 |          80.00 | niah_multikey_1 |   100.00 |         100.00 |\n| niah_multikey_2 |    24.00 |          22.00 | niah_multikey_2 |    99.00 |          99.00 |\n| niah_multivalue |    54.00 |          82.25 | niah_multivalue |   100.00 |         100.00 |\n| niah_multiquery |    49.75 |          88.50 | niah_multiquery |   100.00 |         100.00 |\n|        qa_squad |    64.00 |          50.00 |        qa_squad |    86.00 |          86.00 |\n|     qa_hotpotqa |    41.00 |          36.00 |     qa_hotpotqa |    66.00 |          66.00 |\n|              vt |    57.80 |          97.20 |              vt |    86.40 |          82.60 |\n\nThe table shows that the proposed method can be applied to larger models.\n\n> W3. Equation 3 formulation is internally inconsistent. The hard constraints $Q=A\\_Q B\\_Q, K = A\\_K B\\_K$ in Eq. 3 cannot hold for low rank $r<d$ and are immediately relaxed in the Lagrangian; also, the interaction term omits $B\\_Q B\\_K^T$. This is mostly cosmetic but weakens the mathematical presentation.\n\nWe acknowledge the reviewer's concern and provide the following clarification:\nYou are correct that the hard constraints $Q = A\\_Q B\\_Q$ and $K = A\\_K B\\_K$ cannot hold exactly when $r < d$. The formulation should indeed reflect that these are approximation constraints. We will revise the formulation to be more mathematically precise:\n\n$$\n\\mathop{argmin}_{A\\_{Q}, B\\_{Q}, A\\_{K}, B\\_{K}} \\dfrac{1}{2} ||\n  Q K^T - A\\_{Q} A\\_{K}^T\n  ||_F^2,\n  \\text{where }\n  Q \\approx A\\_{Q} B\\_{Q},\n  K \\approx A\\_{K} B\\_{K}.\n$$\n\nRegarding the interaction term omitting $B\\_Q B\\_K^T$: The key insight is that we are **not** approximating $Q K^T$ as $(A\\_Q B\\_Q)(A\\_K B\\_K)^T = A\\_Q B\\_Q B\\_K^T A\\_K^T$.\n\nInstead, our approach recognizes that:\n1. Both $Q$ and $K$ have low-rank structure individually\n2. Their product $Q K^T$ can be approximated more efficiently by directly learning compact representations $A\\_Q$ and $A\\_K$ such that $Q K\\^T \\approx A\\_Q A\\_K^T$\n\nWe will revise the manuscript to clarify these mathematical details and ensure the formulation is both precise and well-motivated.\n\n> Q1. Section 5 reports “≈ 60 % reduction in CPU <-> GPU transfers” but gives no tokens-per-second or latency figures on the A100 node you mention (lines 210-216). Could you provide end-to-end throughput (tokens/s) and peak GPU memory for LRQK vs. ShadowKV, InfiniGen and vanilla attention at 128 K tokens?\n\nThank you for this important question. \nWe provide a timing results on NVIDIA 3090 24GB under text summarization task (20 samples from LongBench, context 32K, output up to 128 tokens) with `meta-llama/Llama-3.1-8B-Instruct` as the base model.\nDue to time limit, we evaluate the vanilla attention (2 GPUs) and ShadowKV (1 GPU), and the proposed method (1 GPU).\nAverage results are shown below:\n\n|             Method | Time (s) | tokens/s | Average GPU Util (%) | Max GPU Util (%) | GPU Memory (GB) |  GPU power (W) |\n| -----------------: | -------: | -------: | -------------------: | ---------------: | --------------: | -------------: |\n|            vanilla |    17.87 |  1775.70 |       38.61;   48.38 |     99.95; 100.0 |    16.94; 17.56 | 187.24; 233.21 |\n|           shadowkv |    29.29 |  1083.55 |                58.00 |            100.0 |           17.93 |         243.85 |\n| LRQK(r=32, k=2048) |    64.84 |   489.39 |                63.64 |            100.0 |           19.82 |         203.98 |\n| LRQK(r=16, k=2048) |    57.04 |   556.22 |                65.29 |            100.0 |           18.85 |         211.96 |\n| LRQK(r=32, k=1024) |    52.62 |   603.05 |                56.86 |            100.0 |           19.51 |         199.59 |\n| LRQK(r=16, k=1024) |    57.04 |   646.33 |                51.15 |            100.0 |           19.33 |         199.85 |\n\n\n> Q2. All experiments use 7-8 B parameters. Are there bandwidth or PCIe limits that prevent LRQK from scaling to 30 B+ or mixture-of-experts LLMs where KV memory dominates? Even a profiling statement would help assess significance.\n\n**Bottle neck**:\nPCIe limits and GPU bandwidth is not the main limitation, since the data need to be transfered between CPU and GPU is relatively small.\nBased on our observation, the main time consumption is the indexing operation in CPU. It takes time to fetch and collect hitted tokens in CPU. We are looking for a better method to reduce this time consumption.\n\n> Q3. Since LRQK already uses low-rank factors, can the retained full-precision KV rows be post-quantised to 8-bit or 4-bit (à la KVQuant) without harming accuracy? Discussing complementarity with quantisation would broaden impact.\n\nThank you so much for this sugessions, the quantization methods are valuable to explore.\nFollowing the your suggestion, KVQuant is evaluated to both the vanilla model (`meta-llama/Llama-3.1-8B-Instruct` and `Qwen/Qwen2.5-7B-Instruct`) and the model with proposed method. The results are shown in the following table.\nThe hypterparameters of LRQK are $r=32$, top-$k$ = 2048, and 64 lite tokens. \n\n|       RULER 32K | llama +KVQuant | llama +LRQK +KVquant | Qwen +KVQuant | Qwen +LRQK +KVQuant |\n| --------------: | -------------: | -------------------: | ------------: | ------------------: |\n|             fwe |          89.58 |                85.00 |         94.00 |               90.33 |\n|   niah_single_1 |         100.00 |               100.00 |        100.00 |              100.00 |\n|   niah_single_2 |         100.00 |               100.00 |        100.00 |               99.00 |\n| niah_multikey_1 |         100.00 |               100.00 |         96.00 |              100.00 |\n| niah_multikey_2 |          95.00 |                96.00 |         76.00 |               56.00 |\n| niah_multivalue |          95.31 |                96.75 |         70.50 |               93.50 |\n| niah_multiquery |          89.00 |                89.00 |         97.00 |               99.25 |\n|        qa_squad |          83.00 |                82.00 |         73.00 |               74.00 |\n|     qa_hotpotqa |          56.25 |                51.00 |         52.00 |               53.00 |\n|              vt |          87.50 |                95.40 |         95.40 |               90.60 |\n\nFrom this table, we can find that in general applying KVQuant with the proposed method does not siginificantly harm the performance.\n\n> L1. The authors candidly notes that rank r and active-set size k require task-specific tuning and that optimal values differ across architectures. However, the societal-impact section is missing. Please add a brief discussion on potential misuse (e.g., enabling cheaper deployment of disinformation bots) and mitigation strategies.\n\nThank you so much for this valuable sugessions.\n\n**Hyperparameter Tuning**: Different models and datasets may require unique set of optimal hyperparameters. \nHowever, there is a default choice that can help reduce the tuning effort.\nBased on our experiments, we can further turn from a default choice: $r=32$, top-$k$ = 2048, and 64 lite tokens.\nFor instance, the rank $r$ can be chosen from $\\{8, 16, 32, 48\\}$, and the top-$k$ value can be adjusted to accommodate different context lengths.\n\n**Societal impact**:\nLRQK's reduced inference costs could enable broader access to long-context language models, benefiting research and education. However, this efficiency may also lower barriers to deploying harmful applications such as large-scale disinformation campaigns.\nWe believe democratized access to advanced AI capabilities provides net benefits when coupled with appropriate safeguards and responsible deployment practices."}, {"review_id": "9vUvRp5pQh", "reviewer": "Reviewer_UTLG", "summary": "This paper proposes Low Rank Query and Key attention (LRQK), a two-stage framework for efficient long-context inference in large language models. LRQK jointly factorizes the query and key matrices into low-rank bases during the prefill stage, and then uses these to compute proxy attention scores for top-k token selection during decoding. A hierarchical GPU–CPU caching mechanism with a hit/miss policy transfers only missing key-value pairs, preserving exact attention outputs while reducing memory and data movement costs. Extensive experiments on RULER and LongBench benchmarks with LLaMA-3 and Qwen2.5 models show LRQK achieves comparable or superior accuracy to strong sparse attention baselines while substantially lowering GPU memory consumption.", "questions": "List in the Weakness part", "limitations": "1.Although presented algorithms are clear, actual running times, cache sizes (absolute MB/GB), and detailed hardware utilization are not given beyond high-level statements about GPU/CPU. Code is not referenced as available, and implementation details are not provided.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1. The paper addresses a real and pressing bottleneck in LLM inference: the GPU memory consumption of KV caches for long contexts. Efficient memory strategies are highly relevant for widespread LLM deployment, especially on resource-constrained devices.\n\n\nWeakness：\n\n1. The empirical comparisons primarily involve sparse/dynamic attention and offloading methods, but the selection of baselines does not cover some of the most recent quantization methods (e.g., PALU [9], which is only referenced).\n\n[1] Efficient Low Rank Attention for Long-Context Inference in Large Language Models\n\n[2] Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention\n\n2. While sensitivity to rank and top-k is shown (Tables 3, 4), there is limited discussion of the overhead and tuning effort required, particularly when considering different LLM architectures\n\n3. Results are only shown for LLaMA-3-8B and Qwen2.5-7B (Tables 1-2), lacking validation on other popular architectures.\n\n4. Please elaborate further on the differences with other methods focusing on efficient attention for long-context inference.\n\n5. While the paper claims to balance trade-offs between memory efficiency, numerical precision, and computational latency, it lacks experimental validation of these specific trade-offs.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I would like to retain my original positive rating.", "rebuttal": "We thank the reviewer for the detailed and insightful comments. \n\n> W1. The empirical comparisons primarily involve sparse/dynamic attention and offloading methods, but the selection of baselines does not cover some of the most recent quantization methods (e.g., PALU [9], which is only referenced).\n\nThank you for this kindness suggestion. We acknowledge that direct comparison with PALU requires more careful consideration due to the differences in methodology. PALU decomposes $\\mathbf{K} \\in \\mathbb{R}^{l \\times d}$ in dimension $d$, while our method and the comparison methods reduce both dimension $d$ and $l$. \nFollowing the your valuable suggestion, KVQuant is applied to both the vanilla model (`meta-llama/Llama-3.1-8B-Instruct`, `Qwen/Qwen2.5-7B-Instruct`) and the proposed method with default parameters.\nThe results are shown in the table:\n\n|       RULER 32K | llama +KVQuant | llama +LRQK +KVquant | Qwen +KVQuant | Qwen +LRQK +KVQuant |\n| --------------: | -------------: | -------------------: | ------------: | ------------------: |\n|             fwe |          89.58 |                85.00 |         94.00 |               90.33 |\n|   niah_single_1 |         100.00 |               100.00 |        100.00 |              100.00 |\n|   niah_single_2 |         100.00 |               100.00 |        100.00 |               99.00 |\n| niah_multikey_1 |         100.00 |               100.00 |         96.00 |              100.00 |\n| niah_multikey_2 |          95.00 |                96.00 |         76.00 |               56.00 |\n| niah_multivalue |          95.31 |                96.75 |         70.50 |               93.50 |\n| niah_multiquery |          89.00 |                89.00 |         97.00 |               99.25 |\n|        qa_squad |          83.00 |                82.00 |         73.00 |               74.00 |\n|     qa_hotpotqa |          56.25 |                51.00 |         52.00 |               53.00 |\n|              vt |          87.50 |                95.40 |         95.40 |               90.60 |\n\nThe suggested reference paper [2] is a great paper on applying low rank features on LLM training, which inspires us to explore the training side of the LLM. \nWe will consider this in the future work. Thank you for this suggestion.\n\n> W2. While sensitivity to rank and top-k is shown (Tables 3, 4), there is limited discussion of the **overhead** and **tuning effort** required, particularly when considering different LLM architectures\n\nWe appreciate this concern and provide comprehensive overhead analysis:\n\n**Overhead and Tuning Efforts**: Different model and dataset should have their own optimal hyperparameters. \nHowever, based on our experiments, there is an default choisece: $r=32$, top-$k$ = 2048, and 64 lite tokens.\nWe can further tune the hyperparameters based on this default choice.\n\n> W3. Results are only shown for LLaMA-3-8B and Qwen2.5-7B (Tables 1-2), lacking validation on other popular architectures.\n\nThank you for this valuable feedback. We have extended our evaluation to include two additional popular models: `mistralai/Mistral-7B-Instruct-v0.3` and `microsoft/Phi-3-mini-128k-instruct`. For these experiments, we used the default hyperparameters of LRQK.\nDue to memory constraints on the A6000 (48GB), the Phi-3-mini model is evaluated in 16K.\nThe results are summarized in below.\n\n|       RULER 32K | Mistral | Mistral +LRQK |       RULER 16K |  Phi-3 | Phi-3 +LRQK |\n| --------------: | ------: | ------------: | --------------: | -----: | ----------: |\n|             fwe |   67.00 |         93.00 |             fwe |  92.00 |       91.00 |\n|   niah_single_1 |   98.00 |         97.00 |   niah_single_1 | 100.00 |      100.00 |\n|   niah_single_2 |   91.00 |        100.00 |   niah_single_2 | 100.00 |      100.00 |\n| niah_multikey_1 |   83.00 |         97.00 | niah_multikey_1 |  96.00 |       96.00 |\n| niah_multikey_2 |   63.00 |         51.00 | niah_multikey_2 | 100.00 |      100.00 |\n| niah_multivalue |   85.75 |         95.00 | niah_multivalue |  90.00 |       90.25 |\n| niah_multiquery |   83.75 |         93.00 | niah_multiquery |  90.50 |       87.50 |\n|        qa_squad |   59.00 |         63.00 |        qa_squad |  80.00 |       81.00 |\n|     qa_hotpotqa |   45.00 |         47.00 |     qa_hotpotqa |  51.00 |       51.00 |\n|              vt |   98.40 |         96.40 |              vt |  99.60 |       99.60 |\n\n\n> W4. Please **elaborate** further on the differences with other methods focusing on efficient attention for long-context inference.\n\nThank you for this important suggestions. \nA comparison table highlighting key methodological differences is provided below:\n\n|           Methods |           Loki            |                                  InfiniGen                                   |                   ShadowKV                   |                   Proposed                   |\n| ----------------: | :-----------------------: | :--------------------------------------------------------------------------: | :------------------------------------------: | :------------------------------------------: |\n|         Decompose |     K(0:s) <- [K P](0:s)      |                               _, _,A = SVD(Q)                                |                A B = SVD(K)                 |      $Q K^{T} \\approx A_{Q} A_{K}^{T}$       |\n|          Device K |       GPU(low rank)       |                                     GPU                                      |                  GPU(A, B)                   |             CPU(all) + GPU(hit)              |\n|          Device V |       GPU(low rank)       |                                     GPU                                      |                   CPU(all)                   |             CPU(all) + GPU(hit)              |\n| Selection Prefill | top-k of $(qP) (K P)^{T}$ | $W'\\_Q, W'\\_{K}$ <- top-k columns in $\\lvert W\\_Q \\rvert, \\lvert W\\_K\\rvert$ |                 top-k chunks                 |           top-k of $q' K'^{T}; k'$           |\n|  Selection Decode | top-k of $(q P) (KP)^{T}$ |              attention score of $q' K'^{T}$ > (max - $\\alpha$)               |        top-k chunks for each KV head         |          top-k of $q' [K'; k']^{T}$          |\n|       Attention q |           $q P$           |                                      q                                       |                      q                       |                      q                       |\n|       Attention K |     $[K P]\\_{\\Omega}$     |                                 $K_{\\Omega}$                                 | $[ A\\_{\\Omega} B; K\\_{decode}; K^{outlier}]$ |                $K\\_{\\Omega}$                 |\n|       Attention V |       $V\\_{\\Omega}$       |                                 $V_{\\Omega}$                                 |  $[ V\\_{\\Omega}; V\\_{decode}; V^{outlier}]$  |                $V\\_{\\Omega}$                 |\n|        CPU <- GPU |                           |                     selected $K\\_{\\Omega}, V\\_{\\Omega}$                      |            selected $V_{\\Omega}$             | hited $K\\_{\\Omega, hit}$, $V\\_{\\Omega, hit}$ |\n|             Notes | P: PCA projection matrix  |                           modified $W\\_Q$, $W\\_K$                            |  Extra tokens $K^{outlier}$, $V^{outlier}$   |                                              |\n\n**Distinctions:** 1. Joint decomposition, 2. Hierarchical caching, 3. Exact attention preservation.\n\n> W5. While the paper claims to balance trade-offs between memory efficiency, numerical precision, and computational latency, it lacks experimental validation of these specific trade-offs.\n\nThank you for the feedback.\nWe evaluated the trade-offs on a text summarization task using 20 samples from LongBench (context: 32K, output: up to 128 tokens), with `meta-llama/Llama-3.1-8B-Instruct`.\nThe time and resource usage are measured (Nvidia 3090 24G) and shown as below:\n\n|       Method | Time (s) | tokens/s | Average GPU Util (%) | Max GPU Util (%) | GPU Memory (GB) |  GPU power (W) |\n| -----------: | -------: | -------: | -------------------: | ---------------: | --------------: | -------------: |\n|      vanilla |    17.87 |  1775.70 |       38.61;   48.38 |     99.95; 100.0 |    16.94; 17.56 | 187.24; 233.21 |\n| r=32, k=2048 |    64.84 |   489.39 |                63.64 |            100.0 |           19.82 |         203.98 |\n| r=16, k=2048 |    57.04 |   556.22 |                65.29 |            100.0 |           18.85 |         211.96 |\n| r=32, k=1024 |    52.62 |   603.05 |                56.86 |            100.0 |           19.51 |         199.59 |\n| r=16, k=1024 |    57.04 |   646.33 |                51.15 |            100.0 |           19.33 |         199.85 |\n\n\n> L1. Although presented algorithms are clear, actual running times, cache sizes (absolute MB/GB), and detailed hardware utilization are not given beyond high-level statements about GPU/CPU. Code is not referenced as available, and implementation details are not provided.\n\nWe acknowledge this limitation and will provide detailed implementation metrics:\n\n- **Cache in GPU**: 1. The most recently accessed KV cache, 2. Low rank matrices $\\mathbf{A}\\_{K}, \\mathbf{B}\\_{K}, \\mathbf{B}\\_{Q}$, 3. The weights of LLM.\n- **Cache in CPU**: 1. The full KV cache.\n- **Actual running times**: Please refer to response W5.\n- **Implementation**:  A lightweight version of our code is provided in the supplementary material. Key aspects include:  \n  - Built on the standard Hugging Face `transformers` library.  \n  - The original attention mechanism is replaced with our proxy attention mechanism.  \n  - A self-defined buff session is introduced to manage the cache.\n  - Two core algorithms are further compiled with `torch.compile`."}, {"review_id": "XRqn3IqEAi", "reviewer": "Reviewer_7jqo", "summary": "This work proposes Low Rank Query and Key attention (LRQK), a two-stage framework decomposing query and key matrices into low-rank factors during prefill and using them for proxy attention scores in decode, with a hierarchical GPU-CPU cache to reduce data movement. It balances memory efficiency, precision, and latency by selecting top-k and recent tokens, preserving exact attention outputs.", "questions": "Mention in weakness", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Joint Low-Rank Approximation: Unlike prior methods that focus solely on key matrix decomposition, LRQK jointly factorizes both query and key matrices into low-rank components. This approach reduces computational complexity per decode step while maintaining representation accuracy, avoiding expensive SVD operations.\n2. Precision Preservation: LRQK uses low-rank proxies for attention score estimation but retains original query, key, and value vectors for final attention computation. This ensures mathematical fidelity and minimal accuracy loss compared to full-precision attention.\n3. Hierarchical Cache Management: The hybrid GPU-CPU cache with a hit-and-miss mechanism selectively transfers only missing full-precision KV pairs, reducing data movement by ~60% on average. A recency buffer in the GPU cache further optimizes hit rates by leveraging attention’s locality bias.\n\nWeakness:\n1. Hyperparameter Sensitivity: The method requires careful tuning of rank and top-k token counts, which vary across model architectures and tasks. Optimal values demand extensive grid searches, increasing implementation overhead.\n2. The paper lacks performance analysis. It would be better if the authors could compare their method with baselines in terms of throughput and TTFT.", "was_revised": true, "final_justification": "Although most of my concerns have been addressed, I prefer to raise my rating.", "rebuttal": "We appreciate the reviewer's thoughtful evaluation and recognition of the proposed method. Below, we address your specific concerns:\n\n> W1. Hyperparameter Sensitivity: The method requires careful tuning of rank and top-k token counts, which vary across model architectures and tasks. Optimal values demand extensive grid searches, increasing implementation overhead.\n\nThank you for pointing out the sensitivity to hyperparameters. To achive better performance, careful tuning is indeed required.\nNevertheless, the search space is relatively small, and there is a recommend configuration: $r=32, k=2048$, 64 lite tokens and 2 iterations with tolerance $10^{-2}$. We can further turn based on this.\nThe detailed guideline for hyperparameter are as follows:\n\n**Rank $r$**: The matrices $\\mathbf{Q}$ and $\\mathbf{K}$ are approximated using low-rank decomposition for each attention head independently. Given that each head usually has a dimension of $128$, the rank $r$ can vary from $1$ to $128$. In practice, common choices for $r$ are $\\{8, 16, 32, 48\\}$, which balance between the quality of approximation and computational efficiency.\n\n**Top-$k$ Tokens**: This hyperparameter involves selecting and retaining the $k$ tokens with the highest attention scores in the GPU cache. The selection of $k$ is crucial for balancing the sequence length and memory usage. Shorter sequences generally require a smaller $k$, while longer sequences may need a larger $k$ for efficient memory management. Experimentally, $k=256$ is suitable for a 4K token context, with $k=2048$ used for contexts exceeding 32K tokens.\n\n**Number of Lite Tokens**: This parameter specifies the count of the most recent tokens maintained in the GPU. Common values are 16, 32, or 64, and this choice does not substantially impact computational overhead.\n\n**Iterations and Tolerance**: These parameters are integral to Algorithms 1 and 2. The iteration count denotes the number of cycles required to update the low-rank matrices. Generally, 2 or 3 iterations suffice. Tolerance is the convergence threshold for the low-rank matrices, calculated as the mean squared error between successive matrices, typically set to a small value like $10^{-2}$ or $10^{-3}$.\n\n> W2. The paper lacks performance analysis. It would be better if the authors could compare their method with baselines in terms of throughput and TTFT.\n\nWe provide comprehensive timing results on single NVIDIA A100 40GB with Llama-3-8B-1M.\n\n**Main Results:** The GPU only method can provide the fastest TTFT (Time To First Token; Prefill time) and TPOT (Time To First Token; Prefill time), but will reach out of memory (OOM) when context is too long (64K).\nCPU offload method can handle longer context but will suffer from limited GPU bandwidth and result in slow decoding speed, expecially for long context. \nThe default proposed method can support longer context and provide a stable throughput.\nThe proposed method *without* hit/miss mecanism is slower than the default in decoding.\nFurther details can be found in the supplemental material.\nThe following table shows the results.\n\n|           Method |       4K |    4K |       8K |    8K |      16K |   16K |      32K |   32K |     64K |  64K |\n| ---------------: | -------: | ----: | -------: | ----: | -------: | ----: | -------: | ----: | ------: | ---: |\n|                  |     TTFT |  TPOT |     TTFT |  TPOT |     TTFT |  TPOT |     TTFT |  TPOT |    TTFT | TPOT |\n|         GPU only | 37500.29 | 35.40 | 37734.19 | 35.64 | 33649.79 | 35.36 | 32111.45 | 35.77 |     OOM |  OOM |\n|      CPU offload |  6945.10 |  4.31 |  6984.35 |  4.32 |  7073.53 |  2.40 |  4849.19 |  0.98 | 4131.00 | 0.50 |\n|     LRQK default |  7120.49 |  5.68 |  7280.39 |  5.72 |  6379.91 |  5.49 |  5436.66 |  5.36 | 4103.05 | 6.80 |\n| LRQK no hit/miss |  7180.64 |  2.50 |  6747.19 |  2.53 |  5317.06 |  2.31 |  4920.02 |  2.48 | 4163.86 | 1.95 |"}, {"review_id": "k5xm3HiPYQ", "reviewer": "Reviewer_43cG", "summary": "The paper introduces LRQK, a low-rank attention method that reduces KV cache memory in LLMs during long-context inference. It approximates attention using low-rank query/key projections and retrieves only necessary full-precision KV pairs via a GPU–CPU cache. LRQK achieves strong memory savings with minimal accuracy loss, outperforming existing sparse attention methods on long-context benchmarks.", "questions": "Please refer to the Weakness.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- This paper is well-written and easy to follow. \n- The proposed LRQK framework introduces a low-rank approximation of query and key matrices during the prefill stage. This reduces the attention computation to a lightweight proxy form during decoding, enabling efficient token selection with significantly lower overhead.\n- The approach is empirically validated on strong long-context benchmarks (RULER and LongBench) with language models like LLaMA-3-8B and Qwen2.5-7B.\n\n### Weaknesses\n- The observations regarding singular values and key-value decomposition are not novel; similar findings have already been reported in prior work such as Loki [1].\n\n- The proposed method introduces a two-stage approximation involving proxy attention scores and hierarchical cache management, which adds notable complexity to the inference pipeline. However, the paper does not adequately address the implementation or integration overhead.\n\n- Although the authors acknowledge in the Limitations section that the method requires careful hyperparameter tuning, no concrete strategies or guidelines are provided—this significantly limits its practical usability.\n\n[1] Loki: Low-rank Keys for Efficient Sparse Attention", "was_revised": true, "final_justification": "Most of my concerns have been addressed. The authors have provided a detailed response clarifying the distinction between their work and a related prior study, and have also included practical guidelines for real-world use.", "rebuttal": "We thank the reviewer for the constructive feedback. Below are our detailed responses to each concern raised.\n\n> W 1: The observations regarding singular values and key-value decomposition are not novel; similar findings have already been reported in prior work such as Loki [1].\n> \n> [1] Loki: Low-rank Keys for Efficient Sparse Attention\n\nThank you for highlighting this important context. You are correct that low-rank properties of attention matrices have been explored in prior work, such as Loki [1]. However, the key distinction lies in **how we operationalize this property**:  \n\n- **Loki** decomposes the key matrix ($\\mathbf{K}$) individually to exploit its low-rank structure.  \n- **Our method** performs **simultaneous low-rank decomposition of both $\\mathbf{Q}$ and $\\mathbf{K}$**, leveraging the property of matrix multiplication:  \n$$\n  \\text{rank}(\\mathbf{Q}\\mathbf{K}^{\\top}) \\leq \\min(\\text{rank}(\\mathbf{Q}), \\text{rank}(\\mathbf{K})).\n$$  \nThis enables a tighter approximation:  \n$$\n  \\mathbf{Q}\\mathbf{K}^{\\top} \\approx \\mathbf{A}\\_{Q} \\mathbf{A}\\_{K}^{\\top},\n$$  \nwhere $\\mathbf{A}\\_Q$ and $\\mathbf{A}\\_K$ are low-rank proxies for $\\mathbf{Q}$ and $\\mathbf{K}$.  \n\nBy jointly approximating $\\mathbf{Q}$ and $\\mathbf{K}$, we avoid the limitations of isolated decomposition. This design, combined with a **proxy approximation strategy** (rather than direct filling), leads to superior performance in both efficiency and accuracy compared to Loki.  \n\n> W 2: The proposed method introduces a two-stage approximation involving proxy attention scores and hierarchical cache management, which adds notable complexity to the inference pipeline. However, the paper does not adequately address the implementation or integration overhead.\n\nWe acknowledge this important concern and will provide comprehensive implementation details in the revision.\nThe implementation and integration overhead analysis are as follows:\n\n**Implementation overhead**: \n- *Prefill stage*: Need to add a cost to compute low-rank projections on top of standard attention (increase computation). After projection, we can pick the most relevant tokens and compute attention based on the selected tokens (reduce computation).\n- *Generation stage*: Additional computation for low-rank matrices updates (increase computation). Since the number of selected tokens are limited, the computation cost of attention is reduced (reduce computation).\n- *Cache management*: The KV caches are offloaded to CPU, which requires additional lookup in memory and CPU-GPU transfer (increase computation). Additional computation is required to manage the cache (increase computation).\n\n**Integration overheads**:\nThere is a brief version of the code in supplementary material, which shows the basic idea of our implementation. A brief overview of the implementation is as follows:\n- The implementation is based on the standard huggingface transformers library.\n- The original attention mechanism is replaced by our proxy attention mechanism.\n- The cache management is following `transformers.cache_utils.Cache`, which makes it easier to integrate into existing transformer architectures.\n\n\n> W3: Although the authors acknowledge in the Limitations section that the method requires careful hyperparameter tuning, no concrete strategies or guidelines are provided—this significantly limits its practical usability.\n\nThank you for this valuable comment. \nWe will address this concern by providing guidelines for hyperparameter selection.\n\n**Rank $(r)$**:\nThe low-rank approximation of matrices $\\mathbf{Q}$ and $\\mathbf{K}$ is calculated for each attention head separately.\nTypically, the dimension of each head is $128$, allowing for a range of $r$ values between $1$ and $128$.\nFor practical purposes, common choices for the rank $r$ are $\\{8, 16, 32, 48\\}$, offering a balance between approximation quality and computational efficiency.\n\n**Top-$k$ Tokens**: Select and retain the $k$ tokens with the highest attention scores in the GPU cache.\nThe choice of $k$ is pivotal in balancing the sequence length and memory usage.\nShort sequences typically require a smaller $k$, while longer sequences necessitate a larger $k$ to manage memory efficiently.\nIn the experiment, $k=256$ is used for a 4K token context, and the maximum is $k=2048$, applying for context longer than 32K.\n\n**Number of Lite Tokens**: This hyperparameter determines the number of most recent tokens that are kept in the GPU. It has some flexibility in its choice; common values are 16, 32, or 64. This choice does not significantly increase the computation.\n\n**Number of Iterations and Tolerance**: These two parameters are introduced in Algorithm 1 and 2. The number of iterations refers to the number of loops for updating all related low-rank matrices. \nTypically, a few iterations, such as 2 or 3, is sufficient for the task.\nThe tolerance is the threshold for the convergence of the low-rank matrices, which is defined as the mean square error between the current and previous matrices. The tolerance is usually set to a small number, for example $10^{-2}$ or $10^{-3}$.\n\nTips: If no prior knowledge, it is recommended to start with the default values, $r=32, k=2048$, 64 lite tokens and 2 iterations with tolerance $10^{-2}$. Then, we can adjust the parameters based on the performance and memory usage."}]

Author rebuttal:

Meta-review: LRQK is a novel approach with exact attention preservation. It shows strong experimental results, outperforming sparse-attention methods and achieving significant memory savings.

Initial paper lacked performance metrics and scalability validation for larger models. Hyperparameter guidelines were insufficient and implementation overheads unclear.

The paper offers an Innovative solution to a critical LLM memory bottleneck. Authors' thorough rebuttal addressed all major initial weaknesses with new experiments and clarifications, strengthening the paper.

Reviewers questioned metrics, scalability, hyperparameter tuning, and implementation. Authors added detailed timing/memory results, validated on larger models, and provided hyperparameter guidelines/implementation details. All concerns resolved, leading to acceptance.
Decision: Accept (poster)