LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Authors: Dachuan Shi
Yonggan Fu
Xiangchi Yuan
Zhongzhi Yu
Haoran You
Sixu Li
Xin Dong
Jan Kautz
Pavlo Molchanov
Yingyan Celine Lin
Venue: ICML 2025 poster (2025)
Keywords: KV Cache Storage
Long-Context Modeling
Large Language Models
Primary area: deep_learning
TL;DR: We propose LaCache, a training-free KV cache optimization framework featuring a ladder-shaped storage pattern for accurate and efficient generative inference of LLMs.

Abstract: Recent advancements in Large Language Models (LLMs) have spurred interest in numerous applications requiring robust long-range capabilities, essential for processing extensive input contexts and continuously generating extended outputs. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, creating a significant efficiency bottleneck.
In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory (OOM). Specifically, LaCache integrates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), providing an extended span for capturing long-range dependencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size. This token distance-based dynamic compression enables more effective continuous generation under constrained cache budgets.
Experiments across various tasks, benchmarks, and LLM models consistently validate LaCache's effectiveness in enhancing LLMs' long-range capabilities. Our code is available at https://github.com/GATECH-EIC/LaCache.

LaCache: Ladder-Shaped KV Caching for Modeling of Large Language Models
Efficient
Dachuan Shi!
Yonggan Ful 12
Xiangchi Yuan'
Zhongzhi Yu' Haoran You' Sixu Li Xin Dong? 2
Jan Kautz² 2
Pavio Molchanov Yingyan (Celine) Lin 12
Abstract
1. Introduction
Large Language Models (LLMs) have significantly ad- vanced natural language processing tasks (Achiam et al., 2023; Touvron et al., 2023; Jiang et al., 2023; Hurst et al., 2024; Team et al., 2024; OpenAI, 2025; Guo et al., 2025; Gemini, 2025; Anthropic, 2025), but they also face ma- jor challenges due to their substantial computational costs. To mitigate this, key-value (KV) caching has been used to avoid recomputing attention keys and values during the auto auto-regressive decoding of LLMs. However, this approach introduces significant memory overhead that scales linearly with sequence length, leading to out-of-memory (OOM) issues on long sequences.
Recent advancements in Large Language Mod- els (LLMs) have spurred interest in numerous applications requiring robust long-range capabili- ties, essential for processing extensive input con- texts and continuously generating extended out- puts. As sequence lengths increase, the number of Key-Value (KV) pairs in LLMs escalates, cre- ating a significant efficiency bottleneck. In this paper, we propose a new KV cache optimization paradigm called LaCache, a training-free method for efficient and accurate generative inference of LLMs. LaCache enables LLMs to simultaneously address both of the critical challenges in long- range modeling: robust long-range capabilities and continuous generation without running out- of-memory (OOM). Specifically, LaCache inte- grates two key innovations: (1) a ladder-shaped KV cache pattern that stores KV pairs not only sequentially (left-to-right within each layer) but also across layers (from shallow to deep), provid- ing an extended span for capturing long-range de- pendencies under a fixed storage budget, thereby boosting long-range capabilities; and (2) an iter- ative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within fixed cache size. This to- ken distance-based dynamic compression enables more effective continuous generation under con- strained cache budgets. Experiments across var- ious tasks, benchmarks, and LLM models con- sistently validate LaCache's effectiveness in en- hancing LLMs' long-range capabilities. Our code is available at https//ggithbb.com//ATTTT EIC/LaCache.
Existing KV cache eviction strategies attempt to address these challenges by pruning cached KV states enhance memory efficiency. However, these strategies often struggle to balance two critical requirements for long-range LLMs in real-world applications: robust long-range capabilities and continuous generation without OOM. For example, StreamingLLM (Xiao et al., 2023b) prioritizes continuous generation but compromises accuracy on long-context tasks. Quest (Tang al., 2024) maintains high accuracy but at the cost of substantial memory usage due to the need to cache the entire KV cache, eventually leading to OOM on long sequences. H2O (Zhang et al., 2024) reduces memory costs and achieves better accuracy than Streaming but its reliance on attention maps makes it incompatible with the efficient attention implementation FlashAttention (Dao, 2023), leading slow attention computation.
In response to the aforementioned limitations, propose LaCache, training-free KV cache optimization framework that employs ladder-shaped storage pattern for accurate and efficient generative inference in LLMs. Our contribu- tions are summarized as follows:
We propose LaCache, which introduces a novel ladder- shaped KV cache pattern designed for accurate and cost- effective long-context generation. This strategy stores KV pairs not only sequentially (left to right within each layer) but also across layers (from shallow to deep). This configuration extends the span for capturing long-range
'Georgia Tech PNVIDIA. Correspondence to: Yingyan (Celine)
Lin <celine lin @ gatech.edu>>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
dependencies under a constrained storage budget, thereby enhancing long-range capabilities. Specifically, it pre- serves the KV states of early tokens in earlier layers and progressively shifts the focus to later tokens in subsequent layers, forming a stepwise, ladder-like structure. As ana- lyzed later, the ladder-shaped pattern improves the lower bound of overall information retention across all tokens.
detects repetitive patterns estimate token significance, while Wang et al. (2024); Shi al. (2024); Yu et al. (2024) merge similar tokens. However, these methods depend on full attention weights, making them incompatible with state- of-the-art (SOTA) efficient inference frameworks such as FlashAttention (Dao et al., 2022), which do not explicitly compute attention maps. This constraint limits their real- device efficiency. To overcome this, LaCache introduces attention-fres KV cache eviction strategy, ensuring high accuracy while maintaining real-device efficiency.
We further integrate LaCache with an iterative compaction mechanism to support continuous generation for infinitely long sequences without OOM. This approach periodically applies a ladder-based compression pattern previously condensed KV states, freeing up space for new tokens. It compresses older tokens more aggressively while apply- ing less compression to newer tokens, enabling the model to prioritize recent information while efficiently managing memory for incoming tokens.
Efficient LLM inference. While traditional efficiency tech- niques such as quantization, pruning, and distillation (Fran- tar et al., 2022; Lin et al., 2024; Fu et al., 2024b; Shi et al., 2023; Zhang et al., 2023; Yuan et al., 2025) remain valu- able, system-level optimizations, such as FlashAttention (Dao et al., 2022; Dao. 2023; Shah al., 2024) and mem- ory offloading (Sheng et al., 2023), have emerged as key enablers for large efficiency gains in LLM inference. Un- like most dynamic KV cache eviction methods, LaCache can be seamlessly integrated with these system-level ap- proaches, enhancing real-device efficiency with SOTA in- ference frameworks.
We evaluate and validate the effectiveness of LaCache through a series of experiments and ablation studies. Re- sults across multiple benchmarks consistently demon- strate that LaCache enhances long-range capabilities and supports continuous generation. Additionally, due its compatibility with FlashAttention, it outperforms importance-based methods such H2O in terms of achievable accuracy- throoughhut trade-offs.
3. The Proposed LaCache Framework
In this section, we present the proposed LaCache framework. We begin with an overview of LaCache in Sec. 3.1, followed by an introduction to the proposed ladder-shaped KV cache patterns in Sec. 3.2. Finally, to support efficient continuous generation without OOM, even for tasks involving infinite- length generation, we further enhance LaCache with an iterative compaction strategy, as described in Sec. 3.3.
2. Related Work
Long-context LLM. The demand for long-context model- ing has surged due to its ability to handle complex, multi- step tasks and maintain coherent interactions. This has spurred extensive research on enhancing long-context gen- eration (Li al., 2023; Peng et al., 2023; Ye et al., 2025), enabling models to process more tokens per forward pass. While approximate attention mechanisms (Beltagy et al., 2020; Kitaev et al., 2020; Wang et al., 2020) have improved efficiency, they usually lead to degradation in task accuracy. Recent advances in positional embeddings, such as position interpolation and fine-tuning (Chen et al., 2023; Peng & Quesnelle, 2023), have further extended context windows. However, inference efficiency remains bottleneck for long input sequences. In our proposed LaCache, we leverage token eviction to enhance efficiency in long-context genera- tion, including continuous or infinite generation tasks.
3.1. LaCache: Motivation and Methodology Overview
Limitations of existing methods. Existing efficient gen- eration methods for LLMs face limitations in achieving both generation accuracy and memory efficiency, which are crucial for continuous long-context generation without running out of memory. Specifically, as shown in Fig. I (a), recency-based methods like StreamingLLM (Xiao et al., 2023b). which only maintain the KV cache of the latest tokens within fixed-length sliding window with an ((1) memory complexity, can support infinite-length generation without OOM but may compromise generation accuracy. In contrast, retrieval-based methods like Quest (Tang et al., 2024), as shown in Fig. 1 (b), store the full KV cache of all tokens and retrieve the most relevant ones for each new token on the fly to improve computational efficiency. This strategy can achieve high accuracy across tasks due to the maintenance of the entire KV cache, but suffers from mas- sive cache storage with an O(T)) memory complexity, lead- ing to potential OOM issues when handling long contexts.
KV cache eviction. KV cache eviction techniques mitigate excessive cache growth by removing non-essential tokens. Early methods (Xiao et al., 2023b; Han et al., 2024) rely on static, naive retention strategies that overlook model processing patterns and input context, leading to accuracy degradation. To improve long-context handling, dynamic approaches (Adnan et al., 2024; Liu et al., 2024; Zhang et al., 2024; Wan et al., 2024) utilize attention weights to identify important tokens. For example, Liu et al. (2024)
2

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Contees Modeling of Large Language Models
(a) Recency-Based KV Cache
(b) Retrieval-Basso KV Cache
(c) Ladder-Shape KV Cache
Tokens
Recent
Cyclic
KV
Layer N-1
Compacting
Layer 0 KV Cache
Complete
Long-Range
Iterative Compaction for New Space
V Continuous Generation
X Limited Generation (OOM at 160K) Robust Long-Range Capabilitiss
v Continuous Generation (10M+ length)
Weak Long-Range Capabilities
/ Robust Long -Range Capabilities
Dropped Cache
Stored Cache
Selected Cache
Incoming Cache
Figure Illustrative comparisons among (a) recency- based KV cache (Xiao al., 2023b), (b) retrieval serieval-based KV cache (Tang al., 2024), and (c) our proposed LaCache featuring a ladder-shaped pattem. Previous KV cache storage strategies struggle to simultancously balance the needs both continuous generation without OOM and robust long-range capabilities. contrast, our proposed LaCache allows LLMs to simultaneously satisfy the two requirements.
In light of the limitations both approaches, as shown in Fig. (c), we propose LaCache, a training-fres KV cache optimization featuring ladder-shaped pattern, designed to balance accuracy and storage cost, enabling accurate and continuous generation without suffering from OOM.
Identify Pattern & Compact
KV Cache Memory
KV Cache Memory
T'verlap
Saved Space for Additional Tokens
Layer ldx
6
6
Overview. Motivated by the need for both accurate and effi- cient KV caching, our proposed LaCache features a ladder- shaped KV cache compression and storage pattern. We illus- trate LaCache's pattern in Sec. 3.2. Specifically, to achieve effective KV compression while preserving important in- formation of past tokens, rather than uniformly keeping the KV cache for the same set of tokens across all layers in Streaming (Xiao et al., 2023b), we preserve the KV states of early tokens in earlier layers and then progressively shifting the focus to later tokens in the subsequent layers, forming stepwise, ladder-like structure.
6
6
Span
Position Idx
Position Idx
Figure 2. An illustration of LaCache's KV cache storage pat- tern. LaCache leveraged compact the original full KV cache into compressed, ladder-shaped pattern, allowing for the storage of information from longer-range tokens compared StreamingLLM (Xiao et al., 2023b) under the same KV cache budget, thereby providing stronger long-range sequence modeling.
Furthermore, support continuous generation without suf- fering from OOM even for infinite-length generation, we augment LaCache with an iterative compaction strategy, as depicted in Sec. 3.3. Specifically, each time the KV cache reaches its capacity, we apply our LaCache with the ladder- shaped pattern to the already-compacted KV cache. This strategy ensures that older token information is progres- sively compressed further, while newer incoming tokens compressed less. In the following subsections, elaborate the ladder-shaped pattern and the iterative com- paction strategy, which are the two key enablers of our LaCache framework.
of most recent tokens across all layers, our key insight is that while the information from recent tokens is critical for generation accuracy, their KV cache can be maintained and processed by fewer layers. In other words, different layers can maintain the KV cache corresponding to different sets of tokens. The key advantage of this approach is that, under the same KV cache budget, more tokens can retained in the KV cache, effectively enlarging the context length and preserving more past information.
will
The proposed ladder-shaped pattern. The aforementioned insight inspires the design of our ladder-shaped KV cache pattern. As shown in Fig. (c), our LaCache adopts sim- ple yet effective strategy to cache the KV states of varied tokens across different layers: it preserves the KV states of early tokens in earlier layers and then progressively shifts
3.2. LaCache: Ladder-Shaped KV Cache Pattern
The key insight. Unlike StreamingLLM (Xiao et al., 2023b), which maintains the KV cache of the same set
3

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Identify Pattern Compact
Potterns Applied
KV Cache Memory
Cache Memory
Random
(Lower is Betttrr)
LaCeche
Saved Space
Saved Space Enlarged with Iteratively Compacting
Last Compaction Cycle
6
Position Idx
Position
Position
750
Cache Size
Figure 4. An illustration of LaCache's iterative compaction. Iter- ative compaction is introduced to support continuous generation without running out of memory, even for infinite-length genera- tion. Once the KV cache reaches the predefined size, LaCache's s ladder-shaped pattern is applied to the comppacd KV cache, frecing up space for new tokens.
Figure 3. Visualize the trade-off between PPL and cache size LaCache and over 1,500 randomly sampled KV cache pattern..
focus to later tokens in the subsequent layers, aligning with the temporal and sequential processing nature. This ap- proach results in our ladder-shaped pattern, ensuring both storage efficiency and the retention of essential information across past tokens. More specifically, as shown Fig. 2, to implement the ladder-shaped pattern denoted by the green box, we discard the KV states falling outside this pattern and then condense the original 2D KV cache into more compact structure with reduced cache size.
Implementation details. To balance storage efficiency and generation accuracy, we need to ensure that our method properly eliminates redundancy in the stored KV states while accurately preserving past context information by retaining sufficient past KV states.
To satisfy these principles, two key design factors help bal- ance both aspects and achieve an optimal trade-off: (1) the span S across consecutive layers, i.e., the number of layers used to preserve the KV state corresponding to the same token, and (2) the overlap O of each layer, i.e., the num- her of tokens with their KV states preserved in each layer. The larger the span S, the more layers are used to record the context for the same token, thereby improving context preservation with an increased storage cost. Similarly, the larger the overlap O, the more KV states are preserved by each layer, allowing for more accurate context recording at the cost of reduced storage efficiency. As demonstrated in Sec. calibrate these two design factors to minimize cache redundancy and maximize generation accuracy.
Further analysis. We note that the ladder-shaped pattern can effectively cover potentially important tokens by im- proving the lower bound of information retention. Our work intentionally does not rely on attention maps identify important tokens, thereby avoiding confficts with existing optimizations designed for efficient attention calculation. Two rationales behind our ladder-shaped KV pattern are:
Firstly, continuously extending a repetitive pattern and as- signing coverage as equally as possible to each layer, which ensured by our ladder-shaped pattern, improves the lower bound of information retention. This is because, in the worst case, important token sets may appear in the layer with the least coverage, and an unequal coverage strategy would lead to an accuracy drop.
3.3. LaCache: Iterative Compaction for Continuous Infinite-Length Generation
Secondly, since neighboring tokens natural language typ- ically have higher semantic relevance, our ladder-shaped pattern incorporates a smooth transition for each preserved cache segment. As result, the ever-expanding ladder pat- tern with partial overlaps enables smoother fade-out of older tokens, maintaining stable information retention.
To enable continuous generation in LLMs without encoun- tering OOM issues, even with an infinite generation length, it is highly desirable to maintain a constant KV cache size. To achieve this, we need to augment our LaCache with an eviction mechanism that removes the KV states of past tokens when the predefined cache size is fully utilized.
To empirically verify the benefits of these rationales, we randomly generate over 1500 patterns under different KV cache sizes to explore various configurations and visualize the achieved trade-off between Perplexity (PPL) and eache size in Fig. 3. As shown, our ladder- pattern lies the Pareto optimality boundary.
Rationale and advantage of our method. In response to the aforementioned need, we propose an iterative com- paction strategy. The rationale behind our method is simple: once the KV cache, already condensed using LaCache, is full, we apply LaCache again to further condense it.
'To avoid bubbles, slightly more positions are preserved for tokens located at the beginning and end of ladders.
The advantages of this approach include: (1) Thanks to La-

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Table Language modeling experiments are conducted the concatenated Wikitext-2-raw-vll dataset. We use decoding lengths ranging from IK to 16K for each tested model and cache budget. When the decoding length exceeds the pre pre-training length, models using a full cache encounter a perplexity explosion issue. The numbers in brackets indicate the cache size.
Decoding Length
Decoding Length
Model
Model
1K
2K
4K
8K
16K
1K
2K
4K
8K
16K
Llama2-7B (100%)
4.02
4.18
5.12
nan
nan
Llama2 7B-Chat (100%)
4.94
5.32
6.52
nan
nan
w/ StreamingLLM (512)
5.54
5.84
6.32
6.93
5.36
w/ StreamingLLM (512)
6.67
7.41
7.95
8.97
6,98
w/ LaCache (512)
4.53
5,00
5,81
6.61
5.19
w/ LaCache (512)
5.20
6.01
7.06
8,35
6.64
w/ StreamingL.LM (256)
6.08
6.38
6.90
7.52
5.92
w/ StreamingL.LM (256)
7.68
8.45
8.98
9.89
7.88
w/ LaCache (256)
5.57
5.98
6.60
7.34
5.77
w/ LaCache (256)
7.16
7.91
8.47
9.57
7.53
Llama2-13B (100%)
3.89
3.70
4.67
134.25
nan
Llama3-8B (100%)
4.28
4.39
5.82
6.16
109.94
w/ StreamingL.LM (512)
4.92
5.02
5.66
6.28
4.82
w/ Streaming (512)
5.46
5.33
6.73
6.99
5.52
w/ LaCache (512)
4.40
4.68
5,42
6.09
4.69
w/ LaCache (512)
4.61
4.89
6.40
6.78
5.40
w/ StreamingLLMM (256)
5,64
5.52
6.17
6.79
5.29
w/ StreamingLLM (256)
6.39
5.97
7.40
7.66
6.06
w/ LaCache (256)
5.22
5.17
5.88
6.50
5.08
w/ LaCache (256)
5.71
5.61
7.11
7.38
5,86
Cache's ladder-shaped pattern design introduced in Sec. 3.2, early KV caches are discarded first when applying LaCache on the
modeling and long-context understanding. Specifically, for long-context modeling, we use Wikitext-2 (Merity, 2016) and PG19 (Rae et al., 2019) datasets for evaluation. For long-context understanding, we employ LongBench (Bai et al., 2023), Beeell--n-A-Haystaee (Fu et al., 2024a), and RULER (Hsieh et al., 2024) benchmarks to thoroughly as- sess LaCache's achievable performance.
KV cache, as shown in Fig. 4. This use of larger/smaller compression ratios on early/late KV caches aligns with the design principles of recency- based methods (Xiao al., 2023b); (2) From deployment perspective, iterative compaction using LaCache provides unified solution and a clean interface, facilitating wider use.
Baselines. We benchmark LaCache against the standard full KV cache settings and prior KV cache compression methods, including StreamingLLM (Xiao et al., 2023a), H2O (Zhang et al., 2024), TOVA (Oren et al., 2024), Pyra- midInfer (Yang et al., 2024), and SnapKV (Li et al., 2024) under various cache budgets.
Implementation details. We provide a more detailed illus- tration of how our iterative compaction works. As shown in Fig. 4, parts of the stored KV states are highlighted to demonstrate their changes after iterative compaction, while the non-highlighted parts are also occupied by other KV states. When the KV cache reaches its capacity, LaCache is applied to the stored KV states, which have already been condensed by LaCache when first enqueued into the KV cache. Consequently, KV states that fall outside the ladder- shaped pattern are discarded, and the freed space is allocated for new incoming tokens, thus enabling continuous infinite- length generation. In the second iteration of Fig. 4, older KV states are compressed more while newer ones are com- pressed less, thus better preserving recent information.
Implementation Details. We implement LaCache in Py- Torch (Paszke, 2019). For all tasks, we use a batch size of for evaluation. Specifically, for language modeling experiments, we follow the implementation in StreamingLLM (Xiao et al., 2023b) and H2O (Zhang et al., 2024), employing regular token-by-token generation for Wikitext-2 and sliding window approach with window length of 256 tokens for PG-19. For all 21 datasets within the LongBench benchmark, following LongBench's 's default setting, we retain the first 128 tokens unchanged for both LaCache and baseline methods, as these initial tokens pri- marily consist of system prompts and questions. For the Needle-In- Maelle-In-A---yyaaa A-aastackk (Fu et al., 2024a) benchmark, we adopt 50 repetitions for each test unit and context length up to 128k. For the RULER (Hsich et al., 2024) benchmark, we adopt 100 repetitions for each test unit and a context length of 16k.
4. Experimental Results
4.1. Experimental Settings
Models. To validate LaCache's general effectiveness, we apply it to LLMs with varying sizes and functionalities, including Llama2-7B/13B (Touvron al., 2023), Llama3- (Touvron
8B (Dubey et al., 2024), Mamma-7BB11B-OO0 et al., 2023), Alama3.2-3B- In (Dubey et al., 2024), 1oooLLM2-1.7B-Instttue (Allal et al., 2024), and LongChat- 7b-v1.5 (Li et al., 2023).
4.2. Long-Context Modeling Benchmarks
Benchmark on Wikitext-2. We first evaluate LaCache on the concatenated Wikitext-2-raw-v/ dataset in standard
Datasets. We evaluate LaCache on two tasks: long-context
5

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
token-by-token generation setting. Four models, Llama2- 7B, Llama2-7B-Char Llama2-13B, and Llama3-8B, are tested with KV cache budgets of 256 and 512.
12
Full Cache LaCache
Log PPL (Lower is Belter)
10
Out of Memory
As summarized in Tab. 1, our experimental results demon- strate that our LaCache consistently shows stronger ca- pabilities capturing long-range dependencies compared to the recency-basee based method StreamingLLMM (Xiao al., 2023b), under the same KV cache budget, across decod- ing lengths ranging from 1K to 16K. Specifically, with KV cache budget of 512 and a K-length input, LaCache only degrades perplexity by (5.20- - 4.94) x 5% on Llama2-7B-Chan compared to using the full cache. In con- trast, StreamingLLM (Xiao et al., 2023b), under the same cache budget, results in a (6.67 - 4..44 dation in perplexity. This indicates that when targeting 2x KV cache compression, LaCache experiences significantly less degradation in language modeling performance (5% vs. 35%) compared to StreamingI.LM (Xiao et al., 2023b).
100,000
200,000
300 000
400,000 500,000 600,000
Input Length
Figure 5. Evaluate LaCache on the first ten books of concate- nated PG19 dataset, corresponding to length of 600K tokens.
Rz 35% degra-
11.00
w/ StreamingLLM
10.75
w/ LaCache
PPL (Lower is Better)
10.50
Benchmark on PG19. To assess LaCache's capabilities on language modeling tasks with extremely long inputs, we compare it against full cache and StreamingL.LM on the concatenated PG19 dataset, which comprises 100 books to- taling 10 million tokens. We adopt a sliding window of 256 tokens for higher efficiency, following the settings in (Wolf, 2019). The compacted KV cache output from each window is then passed to generate subsequent tokens, allowing the evaluation of the model's long-context capabilities.
10.25
10.00
9.75
9.50
1M
2M
3M
4M
SM
6M
7M
8M
9M
10M
Input Length
Figure 6. Evaluate LaCache on the entire concatenated PG19 dataset, corresponding to length of 10 million tokens.
As shown in Fig. 5, after an 8K input length, the perplexity of the Llama3-8B model using full cache quickly esca- lates; after 160K input length, an OOM issue arises on single NVIDIA A A100 GPU. In contrast, with our LaCache, the Llama3-88 8B model supports continuous generation with up to a 600K input length while maintaining reasonable perplexity.
consistently achieves lower PPL than Streaming LLM and the full cache setting under the same decoding length.
4.3. Long-Contess Understanding Benchmarks
Benchmark on LongBench. The LongBench bench- mark (Bai et al., 2023) evaluates LLMs bilingual long- context understanding capabilities, with most task lengths averaging between 5K and 15K tokens. Evaluation results on the 21 LongBench datasets for the LLaMA2-7B/13B Chat models are presented in Tab. 3, and results for the 10oLLLM2-177B-Insttree model are shown in Tab. 4.
Furthermore, Fig. 6 summarizes the comparison between La- Cache and StreamingLLMM (Xiao et al., 2023b) on the fully concatenated PG19 dataset. The consistent improvements achieved by LaCache further validate its strong long-range capabilities with inputs exceeding 10 million tokens.
Benchmark under an extremely small cache budget. To demonstrate LaCache's effectiveness under an extremely small cache budget, we further apply it to LLaMA3-8B-SK with cache budget of 80. As shown in Tab. 2, LaCache
Our experimental results demonstrate that LaCache, requir- ing no additional computation or storage costs, outperforms StreamingLLMM under the same KV cache budgets. For in- stance, with a 50% KV cache budget, LaCache reduces the average performance degradation from StreamingLLM'ss 2.4 to 1.5 on the LLaMA2- 1ee--11B-hhaah model, from 2.5 to 1.7 on the LLaMA2-7B-CR model, and from 1.5 to 1.0 on the Smoll 1.77BBIn-ttttu model.
Table Evaluate LaCache on the Llama3-8B -B--KK model using cache size equal to 1% of the pre-training sequence length (i.e., 80 tokens).
Decoding Length
1K
2K
4K
8K
16K
32K 64K 128K
Llama3-88 8B
4.28 4.39 5.82
6.16
109.94 nan
nan nan
Benchmark with more KV cache eviction methods. To further compare with attention based KV cache eviction methods, we evaluate the trade-offs between task perfor-
w/ StreamingLLM
7.28 7.78 8.31 8.73
8.88
8.88
9.94 15.68
LaCache
7.13 7.44
7.99
8.36
8.46 8.43 9.53 15.08
6

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Table Evaluate LaCache on Llama2-78/13B-CC models across 21 LongBench datasets under 50% and 25% KV cache budgets.
Model
Llama2-7B--a
Maama2-13B-Chne
Streaming
LaCache
Streamingl LIM
LaCache
Cache budget
100%
100%
50%
25%
50%
25%
50%
25%
50%
25%
HotpotQA
33.84
29.98
30.74
32.62
30.60
38.86
37.09
37.16
38.08
36.49
2WikiMultihoPQA
26.83
24.75
24.99
26.22
25.19
34.19
32.30
32.15
34.11
31.17
MuSiQue
8.82
8,48
6.58
7.72
7.94
14.19
12.12
11.99
13.67
12.97
DuRcader
24.06
18.30
19.11
22.64
21.05
27.34
20.03
21.07
24.07
20.78
MultiFieldQA-en
35.72
27.02
24.74
31.55
25.34
36.63
27.03
24.56
31.73
26.09
MultiFieldQA-zzh
33.32
23.84
20.13
25.69
22.55
34.13
24.90
23.11
27.49
25.44
N
16.78
15.97
13.46
16.27
15.18
19.38
17.50
14.78
19.30
17.95
Qasper
17.33
15.79
15.90
16.55
26.84
23.02
21.44
24.51
21.05
GovReport
26.25
22.54
20.73
22.84
20.47
26.21
23.89
21.78
23.76
21.54
QMSum
20.89
19.72
19.30
20.34
19.58
20.12
19.03
18.76
19.61
19.16
MultiNews
25.83
25.07
23.32
25.16
23.27
26.06
25.38
23.71
25.33
23.79
VCSUM
14.33
13.12
12.31
13.16
12.18
16.89
15.53
14.37
16.03
13.87
TriviaQA
83.01
83.13
80.31
83.28
81.09
88.45
88.25
85.78
88.56
85.95
SAMSum
41.28
39.97
38.46
39.47
38.86
36.77
36,50
36.01
36.99
35.85
TREC
64.50
62.50
59.00
65.00
58.00
68.5
65.50
62.00
66.50
60.00
LSHT
17.75
17.25
15.00
16.25
15.75
20.25
20.00
18.75
19.00
18.75
PassageRetrieval-enn
11.50
6.50
6.50
5,00
6.50
8,50
7.50
10.00
7.00
PassageCount
4.50
5,00
5.00
5,50
5.00
3,92
3,50
2.00
2.50
3.50
PassagcRetrieval-zt
12,00
7.50
7.00
7.60
5.50
14.50
13.00
8.00
10.50
9.00
LCC
47.74
47.97
47.18
47.97
45.59
39.31
39,22
39.00
40.12
39.09
RepaBench-P
44.35
43.44
43.82
43.41
43.54
42.98
41,14
39.46
41.70
38.33
Average
29.08
26.56
25.41
27.34
25.68
30.69
28.30
26.82
29.22
27.04
Average Score-Thruughout Trade-ofts All Tasks
Average Score-Throughut Trade-ofts @ OA
Average Score-Throughpt Trad-oofts @ Summarization
Athention-tree
25
Attent on free
Allentionn//ee
Score, higher better
Score higher is better
-4
28
mton on recuired
tart lon- inn-nquinnd
Partal Attento
-33
Portial Attention
higher is
20.5
Amren Arter inn-equuieed
27
Mathod
Method
Nathod
420
22
H2O
20.0
H20
TOVA
TOYA
TOVA
26
Snapty
SeaptV
19.5
Snapsty
Pyrtmist
21
Pyroowetinder
Pyrnmittnter
alade
19.0
20
50
60
70
20
30
40
50
70
20
30
40
S0
60
70
Throughput, higher is bettor (Thkonsis)
Throughput, higher is botter (Takans2s)
Throughput, higher is better (Tolknngss)
Average Scruuhhhroughput trade-offs tew-Shot
Average Score-Throughpue trade-ofts Synthetic
Average Scare-Throuchou Trade-off's Code
Attention-rre
Alttent on-free
46.5
Attention-fres
Score, higher better
Score, higher is better
6
Score, higher is better
06.0
Portial Attention
30
Partial Attention
sant on--reuirred
85.0
80
7.
$4.5
TOWA
Snaptly
Seapity
$4.0
TOVI
Snappty
Jenan VMAM
Pyronechnterr
43.5
47
LaCache
LaCacte
LaCache
20
50
20
40
50
20
30
40
50
60
Throughppt, higher
better (Tokens/s)
Throughput, higher is better (Tokens/5)
Throughput, higher better (Tokens)))
Figure Evaluate the score-throughput trade-off's on a single H200 GPU with StreamingLLM (Xiao et al., 2023b), H2O (Zhang al., 2024). TOVA (Oren et al., 2024). PyramidInfer (Yang et al.. 2024). and SnapKV (Li al., 2024) on LongBench (Bai et al., 2023). The top subfigure presents the average performance across 21 tasks, while remaining subfigures demonstrate the sub-task performance Question Answering, Summarization, Few-shot Learning, Synthetie Task, and Code Completion, respectively.
mance and throughput on LongBench using the LLaMA- 2-7B-Chat model. As shown in Fig. 7, the importance- based KV cache eviction methods H2O (Zhang et al., 2024), TOVA (Oren al., 2024), PyramidInfer (Yang al., 2024),
and SnapKV (Li et al., 2024) maintain good scores but suf- fers from low throughput, while the recency-based method StreamingLLMM (Xiao et al., 2023b) experiences lower scores. Our method achieves better trade-off between task

---

LaCache: Ladder-Shaped KV Caching for Efficient Long- Cong-Context Modeling of Large Language Models
Streamingl LM with 50% Iuxtgett: Srt 54., Apc
LeCactee with 50% Suadget pe 10..8% Age
Table Evaluate LaCache on Smoll -M2-1.78-Instrruu across 21
LongBench Datasets under 50% and 25% KV cache budgets.
Streaming .LM
LaCache
Cache Budget
100%
50%
25%
50%
25%
HotpotQA
24.08
24.22
22.21
24.32
22.81
2WikiMultihopQA
24.15
22.71
21.71
22.28
21.34
MuSiQue
8.33
9.47
9.30
10.28
8.43
1000 10
DuReader
20.24
14.56
14.58
16.73
15.56
MultiiielQQAeen
38.78
33.39
30.28
33.35
30.09
MultiFieldQA-zh
16.82
14.63
12.12
13.94
12.27
NarrativeQA
12.65
12.62
11.68
11.98
10.73
Qasper
17.22
16.52
14.47
16.48
16.79
Figure 8. Benchmark LaCache and StreamingLLM on Needle-
GovReport
26.74
21.38
18.94
21.80
18.86
In-A-Haystack (Fu et al., 2024a) using Llama3.2-3B-Insse
QMSum
21.86
21.37
21.04
21.40
20.87
128k (Dubey et al., 2024) under a 50% cache budget setting. Greener indicates better performance.
MultiNews
25.67
25.42
24.75
25.54
24.88
VCSUM
10.56
11.76
10.43
11.97
10.52
TriviaQA
80.59
78.87
78.83
80.36
78.62
SAMSum
22.05
24.15
26.12
24.63
27.80
buzget:
TREC
56.00
54.50
$2.50
54.50
53.00
LSHT
9.00
6.00
S.25
7.00
6.00
PassageRetrieval-en
9.50
5,50
5.00
6.50
5.50
PassageCount
1.50
1.00
1,50
1,50
2.00
PassageRetrievalzzt zh
7.77
4.86
4.92
7.42
4.17
LCC
37.76
37.85
37.75
37.99
37.83
RepoBench-P
34.37
33.67
34.14
34.12
33.82
Average
24.07
22.59
21.79
23.05
22.00
performance and throughput compared to these baselines.
Conrest Length
Langth
Extend to small LMs. To demonstrate LaCache's 's effec- tiveness across varying model sizes, we further apply it to the small LM SmooLLM--1.B--Inse
Figure 9, Bennchmark LaCache and StrcamingLLM on Needle-In- A Haystack (Fu et al., 2024a) using 7bb (Li et al.. 2023) under 25% cache budget setting. Greener indicates better performance.
As shown in Tab, 4, LaCache consistently achieves higher accuracy than the baseline StreamingLLM under the same cache budgets.
The Needle-In- A-Haystack benchmark (Fu et al., 2024a) assesses LLMss capabilities of retrieving specific information ("the nee- dle") embedded within extremely long text ("the haystack"), which is crucial for applications requiring precise infor- mation retrieval from long context. We benchmark La- Cache and StreamingLLM on 2etaaa..-----sntttee 128k and 7bb as shown in Fig. 8 and Fig. 9.
Benchmark on Moalle-InAA-Hasttaan
The experimental results further validate the consistently better performance of LaCache under the same KV cache conditions. Specifically, LaCache achieves a 5.06% higher average accuracy across 13 different tasks, with particu- larly large improvements on the v/ and owe tasks, where it significantly outperforms the baseline.
4.4. Ablation Studies
The results demonstrate that LaCache nearly doubles the test accuracy compared to Streaming under the same cache budget- for example, from 54.54% to 99.16% on the -lama..2-3B--nmm..
Hy perpaaaareeerr Span S. In long-context understanding tasks such as LongBench, Span S is set as an integer ap- proximately equal to the number of layers multiplied by the overall compression ratio, aiming for a uniform compression ratio distribution. For example, under a 50% cache budget, setting S equal to half the number of model layers results in ~500 compression ratio across different positions, avoid- ing situations where some locations while others are under-compressed. In language modeling tasks, S is set to 1/4 of the number of model layers, which was given by the empirical results from our ablation studies, as shown in Fig. 10, where Llama2-7B-CL model under a 256 KV eache budget and the Wikitext-2 dataset are used
128k model under a 50% cache budget and from 33.40% to 65.30% on the LongChat-7b- v1.5-32k model under 25% cache budget.
Benchmark on RULER. The RULER (Hsieh et al., 2024) benchmark utilizes synthetic examples to evaluate long- context LLMs. It encompasses four task categories, includ- ing retrieval, multi-hop tracing, aggregation, and question answering. We benchmark LaCache and StreamingLLMM on the LongChat- So-CChat-bb-l... 7b-51.5 model under a 50% cache setting, as shown Tab. 5.
over-compressed
8

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Table Evaluate LongChat-7b-vl.--22 model (I. al., 2023) the RULER benchmark (Fu al., 2024a) under a 50% cache budget
setting. A higher number indicates better performance.
Task
single single> singles multikey: multikey2 multikeys multivalue multiquery
vL
cwe
fwe
qa
qa2
Avg.
SLreamingLLM
45.00
49.00
45.00
53.00
50.00
45.00
47.00
42.00
29.40 17.20 42.00 75.00 43,00 44.82 82 60.80
LaCache
57.00
43.00
26.00
52.00
64.00
31.00
62.75
50.25
61.00 45.67 67.00
41.00
50.88
PPL Under 4K Longth (Lower is Better)
that rely more on local information, such as QA tasks (Narra- tiveQA, Qasper, MultiFieldQA-en, and MultiFieldQA-zh),
Span
5. Limitations and Future Work
55
While LaCache demonstrates advantages for accurate and efficient long-contex generation in LLMs, it also has lim- itations that present opportunities for future exploration: Although the ladder-shaped KV cache pattern is effective, it may not be optimal for every scenario. Alternative patterns could further enhance memory efficiency and performance. Future work can explore diverse KV storage configurations based on core insight: recent tokens crucial for generation accuracy, but fewer layers might suffice for ef- fectively processing and storing their KV caches. Addition- ally, LaCache is implemented in a training-free setting to ensure efficiency and ease of deployment. Incorporating fine-tuning could further improve performance by adapting the KV cache pattern to specific tasks. Future work can extend LaCache to support fine-tuning and benchmark its performance against muining-dependee methods.
10
Overlap
Under Length (Lower Better)
10.0
Span
9.7
Overlap
PPL Under 16K Length (Lower is Better)
8.0
Span
7.9
1.8
77
7.6
7.5
Overlap
Figure 10. Ablation studies the hyperparameters for language modeling. Perplexity (lower is better) is reported in the figure.
6. Conclusion
In this work, we introduce LaCache, a novel, training-free free, and easy-to-deploy KV cache optimization framework de- signed enhance both the efficiency and effectiveness of LLMs in long-context generation tasks. LaCache addresses the limitations of existing KV caching methods through ladder-shaped KV cache storage pattern and an iterative compaction mechanism. These innovations enable LLMs to better capture long-range dependencies, optimize memory usage, and sustain continuous generation even under fixed storage constraints. Specifically, by sequentially storing KV pairs both within and across layers, the ladder-shaped structure allows the model to preserve crucial information at different levels of context. Additionally, the iterative compaction mechanism dynamically manages memory, en- suring that essential information is prioritized. Our re- sults show that LaCache significantly improves memory efficiency while maintaining high generation quality, outper- forming baseline methods across various benchmarks. By offering scalable and storage-efficiest solution, LaCache enhances the capacity of LLMS to manage extended con- texts and continuous generation, paving the way for further innovations in long-range LLM optimization.
Table 6. Ablation studics on the hyperparameters for long-context understanding. Scores (higher is better) are reported in the table.
Setting
0= O= S/4 O = 5/2 A(S/4 0) A(S/2 0)
QA tasks
19.48
18.94
18.48
-0.54
-1.00
Synthetic tasks
5.17
5.67
6.17
+0.50
+1.00
for examining the impact of hyperparameters.
Hyperparameter Overlap O. The choice of Overlap O depends on the task type. Specifically, larger O allows the information of a single token to be distributed across more positions, which is better suited for tasks requiring complex semantic understanding and greater global context. In con- trast, a small overlap concentrates the information in fewer positions, which is more appropriate for tasks where the an- swers appear in very narrow window. For language model- ing tasks, O is set to 1/2 of S for achieving better semantic continuity. For long-context understanding tasks, as shown in Tab. 6, a larger overlap consistently improves perfor- mance on tasks that require more global information, such as synthetic tasks (PassageCount, PussageRetrieval-enn and PassageRetrieval-zzh) while reducing performance on tasks

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Acknowledgment
Chen, S., Wong, S., Chen, L.,, and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint (xXxx:2330.115900 2023.
This work was partially supported by the National Science Foundation (NSF) through the Computing and Communi- cation Foundations (CCF) program (Award ID: 2400511), the Division of Information & Intelligent Systems (IIS) pro- gram (Award ID: 2403297), and CoCoSys, one of the seven centers in JUMP 2.0, a Semiconductor Research Corpo- ration (SRC) program sponsored by DARPA. It was also supported by the Department of Health and Human Services Advanced Research Projects Ageney for Health (ARPA-H) under Agreement Number 140D042490003. The views and conclusions contained herein those of the authors and should not be interpreted necessarily representing the of- ficial policies or endorsements, either expressed or implied, of the Advanced Research Projects Agency for Health or the U.S. Government.
Dao, T. Flashattention-2: Faster attention with bet- ter parallelism and work partitioning. arXiv preprint arXiv:2307.086%) 2023.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io- so-awareness. Advances in Neural Information Process- ing Systems, 35: 16344-16359, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Frantar, E., Ashkboos, S., Hoefter, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210113323, 2022.
Impact Statement
This work aims to enhance the generation efficiency of LLMs on long-context tasks, achieving continuous genera- tion without OOM while maintaining long-context accuracy. As such, can facilitate wider use of L.MSs and does not suffer from additional societal consequences if LLMs are properly used.
Fu, Y., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y., and Peng. H. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.1017, 2024a.
Fu, Y., Yu, Z., Li, J.. Qian, J., Zhang, Y., Yuan, X., Shi, D., Yakunin, R., and Lin, Y. C. Amoeballm: Constructing any-shape large language models for efficient and instant deployment. Advances in Neural Information Processing Systems, 2024b.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Gemini. Gemini 2.5:
Our most intelligent
Adnan, M., Arunkumar, A., Jain, G., Nair, P., Solovey- chik, I., and Kamath, P. Keyformer: Kv cache reduction through key tokens selection for efficient generative in- ference. Proceedings of Machine Learning and Systems, 6:114-127,200
ai model, 2025. googgle/technology/ wwtp::/ggooollllooo deepmind/ gemini-model-thint-e -apt martteess-arhh-aaee 2025/ Sgemini-2-5-thiee
URL https://blog.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P., Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Tunstall, L.,, Piqueres, A., Marafioti, A., Zakka, C., von Werra, L., and Wolf, T. Smollm2 with great data, comes great performance, 2024.
Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings the the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers), pp. 3991-4008, 2024.
Anthropic. Introducing claude 4, 2025. URL https: //www. ant.roaiihi...... com/ news het//llaaddeeeeeeee
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng. A., Hou, L., Dong, Y., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding, arXiv preprint arXiv:2308. 14508, 2023.
Hsich, C.-P. Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg. B. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404 06654, 2024.
Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.0 /50, 2020.
10

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00077 2023.
Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long- range sequence modelling. arXiv preprint, 2019. URL https: / www.aiiwi...oooaeeeeeeeeeoeeeeee abs/////////// / 911. 05507.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C.,, Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel. G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint 1oXiv:2310006825, 2023.
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low low-precision. Advances in Neural Information Processing Systems, 37:68658 686888688885..0 2024.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445/ 2020.
Sheng. Y., Zheng, L.,, Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang. P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp. 31094-311 31116. PMLR, 2023.
Li, D., Shao, R., Xie, A., Sheng, Y., Zheng, L., Gonzalez, J., Stoica, I., Ma, X., and Zhang, H. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
Shi, D., Tao, C.,, Jin, Y., Yang, Z., Yuan, C., and Wang. J. Upop: Unified and progressive pruning for compressing vision-language transformers. In International Confer- ence on Machine Learning, pp. 31292-31311. PMLR, 2023.
Li, Y., Huang, Y., Yang. B., Venkitesh, B., Locatelli, A., Ye, H. Cai, T., Lewis, P., and Chen, D. Snapky: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947- 22970, 2024.
Shi, D., Tao, C.,, Rao, A., Yang. Z., Yuan, C., and Wang, J. Crossget: Cross-guided ensemble of tokens for ac- celerating vision- lision-lnguaage transformers. In Forty-first International Conference on Machine Learning, 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aweer aware weight quantization for on on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikei, B., and Han, S. Quest: Query-aware sparsity for efficient long-context Ilm inference. arXiv preprint arXiv:2406. 10774, 2024.
Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V. Xu, Z., Kyril- lidis, A., and Shrivastava, A. Scissorhands: Exploiting the persistence of importance hypothesis for 1lm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, I.., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint UxXiv:2403.08295 2024.
Merity, S. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 9, 2016.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 2023.
OpenAL. URL
Openai gpt-4.5 system card, 2025. https:// cdn opw.ai. com/
ggt-4-5-systemm cwddd
Wan, Z., Wu, X., Zhang, Y., Xin, Y., Tao, C., Zhu, Z., Wang, X., Luo, S., Xiong, J., and Zhang, M. D2o: Dynamic dis- criminative operations for efficient generative inference of large language models. arXiv preprint arXiv:2406.13035. 2024.
Oren, M., Hassid, M., Adi, Y., and Schwartz, R. Transform- ers are multi-state rnns. arXiv preprint arXiv:2401.06104, 2024.
Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.017000 2019.
Wang. S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint 2006::0000604776. 2020.
Peng, B. and Quesnelle, J. Ntk-aware scaled rope allows Ilama models to have extended (8k+) context size with- out any fine-tuning and minimal perplexity degradation, 2023.
Wang, Z., Jin, B., Yu, Z., and Zhang, M. Model tells you where to merge: Adaptive kv cache merging for llms on long- tasks. arXiv preprint arXiv:2407.0845. 2024.

---

LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
Wolf, T. Huggingface's transformers: State-of-the-
natural language processing. arXiv preprint arXiv: 1911v:1910003771. 2019.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099. PMLR. 2023a.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXhv:2309.17433 2023b.
Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. Pyramidinfer: Pyramid kv cache compres- sion for high-throughput I1m inference. arXiv preprint arXiv:2405. 125322 2024.
Ye, Z., Xia, K., Fu, Y., Dong, X., Hong, J., Yuan, X., Diao, S., Kautz, J.. Molchanov, P., and Lin, Y. C. Long- mamba: Enhancing mamba's long-contex. capabilities via training-free free receptive field enlargement. In The Thir- teenth International Conference Learning Representa- tions, 2025.
Yu, H. Yang, Z., Li, S., Li, Y., and Wu, J. Effectively com- press kv heads for Ilm. arXiv preprint 20060:2406.07056 2024.
Yuan, X., Zhang, C., Liu, Z.. Shi, D., Vosoughi, S., and
Lee, W. Superficial self ceff-inpooved reasoners benefit from model merging. arXiv preprint arXiv:2503.02/03: 103, 2025.
Zhang. C., Song, D., Ye, Z., and Gao, Y. Towards the law of capacity gap distilling language models. arXiv preprint arXiv:2311.0705.. 2023.
Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
12


Reviews:
[{"review_id": "EtMI23QfFx", "reviewer": "Reviewer_fB6E", "summary": "The paper introduces LaCache, a scheme for progressive cache eviction for more efficient long context processing. Rather than evicting the same tokens at each layer, LaCache evicts tokens using a ladder-like scheme so that earlier layers maintain tokens from earlier in the context and later layers retain tokens from later in the context (with the immediate local context being fully preserved). They show that this scheme reduces the performance degradation from cache eviction and is more efficient than attention-based patterns for cache eviction; the size of each ladder \"rung\" and the overlap between rungs across layers are hyperparameters, which are analyzed in the analysis.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions and analysis offered by our work, as well as for the constructive suggestions! We have addressed all your comments and suggestions as follows.\n\n---\n\n**Q1. Experiments on non-perplexity tasks with a long context regime (>>16k)**\n\nFollowing your suggestions, we have added new experiments with context lengths up to 128k on both Llama3.2-3B-Instruct-128k and LongChat-7b-v1.5-32k to verify the consistent effectiveness of LaCache.\n\n(a) On the Needle In A Haystack (NIAH) benchmark, we evaluated both Llama3.2-3B-Instruct-128k (in [figure https://ibb.co/1GjhX0b5](https://ibb.co/1GjhX0b5)) and LongChat-7b-v1.5-32k (in [figure https://ibb.co/s9qp1s2K](https://ibb.co/s9qp1s2K)) under both 50% and 25% cache budget settings. Our results demonstrate that LaCache nearly doubles the test accuracy compared to StreamingLLM under the same cache budget—for example, from 54.54% to 99.16% on the Llama3.2-3B-Instruct-128k model under a 50% cache budget, and from 33.40% to 65.30% on the LongChat-7b-v1.5-32k model under a 25% cache budget.\n\n| Task               | StreamingLLM | LaCache |\n|--------------------|--------------|---------|\n| niah_single_1      | 45.0         | 57.0    |\n| niah_single_2      | 49.0         | 43.0    |\n| niah_single_3      | 45.0         | 26.0    |\n| niah_multikey_1    | 53.0         | 52.0    |\n| niah_multikey_2    | 50.0         | 64.0    |\n| niah_multikey_3    | 45.0         | 31.0    |\n| niah_multivalue    | 47.0         | 62.75   |\n| niah_multiquery    | 42.0         | 50.25   |\n| vt                 | 29.4         | 60.8    |\n| cwe                | 17.2         | 61.0    |\n| fwe                | 42.0         | 45.67   |\n| qa_1               | 75.0         | 67.0    |\n| qa_2               | 43.0         | 41.0    |\n| **Mean**           | **44.82**    | **50.88** |\n\n\n(b) Similarly, on the RULER benchmark, we evaluated the LongChat-7b-v1.5-32k model under a 50% cache setting. The experimental results above verify the advantageous performance of LaCache under the same KV cache. Specifically, LaCache achieves a 5.06% higher average accuracy across 13 different tasks, especially in task cwe and fwe, where LaCache outperforms the baseline by a large margin.\n\n\n---\n\n**Q2. More analysis on the costs of compression patterns**\n\nThank you for your insightful comments! We agree that selecting an ideal set of \"important tokens\" to maximize performance is not the primary goal of our method. \n\nThe key point we aim to convey in this analysis is that LaCache improves over baselines by leveraging the insight that it is not necessary to maintain the same set of KV tokens across all layers. In particular, LaCache alleviates this redundancy through the proposed ladder pattern, which can more effectively cover potentially important tokens within a given budget and thus improve the lower bound of information retention. We will incorporate your suggestion and clarify this more clearly in the final version.\n\n---\n\n**Q3. Add more baselines on LongBench**\n\nThank you for your suggestion! We have added benchmarks with recent baselines, including both SnapKV [1] and PyramidInfer [2], as shown in [figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ). This set of results consistently validates that our LaCache achieves better score-throughput trade-offs across various tasks on the LongBench benchmark. \n\n\n---\n\n**Q4. Benchmark LaCache with H2O on the remainder of the LongBench tasks**\n\nFollowing your suggestion, we have updated the overall performance comparison between LaCache and H2O on the entire 21 tasks of the LongBench benchmark as shown in [figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ), which demonstrates that LaCache achieves better F1-score-throughput trade-offs across various tasks on the full LongBench benchmark. We will also report results by categories in the final version.\n\n---\n\nThank you for your thoughtful suggestions to help strengthen this paper! If you have any further questions or updated comments, we would really appreciate it and be happy to address them.\n\n---\n\n**References**\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference", "claims_and_evidence": "Yes; the paper demonstrates the strong performance relative to reasonable baselines in terms of both downstream metrics and latency, compared with a fixed storage (KV cache size) budget.", "methods_and_evaluation_criteria": "Yes; I think the use of perplexity alone would not be convincing, so I appreciate the use of a diverse set of tasks from LongBench. It would be nice to show a non-perplexity task in the very long (>>16k) context regime; however, I understand there are not all that many tasks that fit this description.", "theoretical_claims": "Not exactly a proof, more an informal optimization argument-- but I'm not fully convinced by the argument about the costs of compression patterns. In particular, since we are not working with tokens, but with contextual embeddings, it seems that some information about the \"important tokens\" will be present even if the embeddings of those tokens are not chosen; in addition, I'm not convinced that you can choose an ideal set of \"important tokens\" to maximize performance in the first place. This is not central to the paper, so it wasn't a major factor in my score; however, I don't think this section adds much to the argument.", "experimental_designs_or_analyses": "Yes; I think the experimental design is reasonable and compared against good baselines. \n\nI do think it would be nice to see results on the baselines other than StreamingLLM in the main table, especially as it seems that H2O might sometimes outperform StreamingLLM in downstream performance if not in latency.", "relation_to_broader_scientific_literature": "This is a crowded space, but I think this is a nice contribution and argues its point well. This is a type of fixed pattern I haven't seen before-- most methods evict the same tokens across layers, or use an attention-based decision at each layer. Using earlier layers for earlier tokens is an interesting idea.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "I particularly like the analysis described by Figure 3, where the caching pattern is compared to randomly chosen caching patterns across cache sizes. I think this is a nice extra that makes a strong argument for this technique.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1. Can you report how LaCache compares to H2O on the remainder of the LongBench tasks you use?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gj6H9dpcqL", "reviewer": "Reviewer_cTxa", "summary": "* Paper proposed a training-free KV Cache compression which stores KV pairs not only sequentially (left-to-right within\neach layer) but also across layers (from shallow to deep), giving deeper capabilities to capture long-range dependencies.\n*  Proposes iterative compaction mechanism that progressively compresses older caches, freeing up space for new tokens within a fixed cache size.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the insightfulness and promising results of our work, as well as for the constructive suggestions! We have addressed all your comments and suggestions as follows.\n\n---\n\n**Q1. More ablation studies and analyses on hyperparameters**\n\nThank you for the suggestion! We highlight that the ladder pattern can be fully determined by the two hyperparameters: Span and Overlap (i.e., S and O in Section 3.2 of our manuscript).\n\n(a) In long-context understanding tasks such as LongBench, S is set as an integer approximately equal to the number of layers multiplied by the overall compression ratio, aiming for a uniform compression ratio distribution. For example, under a 50% cache budget, setting the span equal to half the number of model layers results in a ~50% compression ratio across different positions, helping to avoid situations where some locations are over-compressed while others are under-compressed. In language modeling tasks, S is set to 1/4 of the number of model layers, which was given by the empirical results from our ablation studies, as shown in Figure 8. \n\n(b) The choice of O depends on the task type. Specifically, a larger overlap (O) allows the information of a single token to be distributed across more positions, which is better suited for tasks requiring complex semantic understanding and greater global context. In contrast, a small overlap concentrates the information in fewer positions, which is more appropriate for tasks where the answers appear in a very narrow window. \n\n|                 | Overlap=0 | Overlap=Span/4 | Overlap=Span/2 | $\\Delta$ (Span/4 - 0) | $\\Delta$ (Span/2 - 0) |\n|-----------------|------------|------------------|------------------|----------------------|----------------------|\n| QA tasks        | 19.48      | 18.94            | 18.48            | -0.54                | -1.00                |\n| Synthetic tasks | 5.17       | 5.67             | 6.17             | +0.50                 | +1.00                 |\n\n\nFor language modeling tasks, S is set to 1/2 of Span because of better semantic continuity. For long-context understanding tasks, following your suggestion, we have added above experiments on LongBench to demonstrate the impact of the overlap parameter O. As shown in the table, a larger overlap consistently improves performance on tasks that require more global information, such as synthetic tasks (PassageCount, PassageRetrieval-en, and PassageRetrieval-zh), while reducing performance on tasks that rely more on local information, such as QA tasks (NarrativeQA, Qasper, MultiFieldQA-en, and MultiFieldQA-zh).\n\n\n---\n\nThank you for your thoughtful suggestions to help strengthen this paper! If you have any further questions or updated comments, we would really appreciate it and  be happy to address them.", "claims_and_evidence": "* Different layers can maintain the KV Cache corresponding to different sets of tokens. This is a good insight.\n* The ladder-shaped method is both intuitive to follow and well-developed.\n* Iterative method to remove old KV Cache states to ensure memory compaction is a good idea.\n* Figure 4 is useful visualization of the compaction technqiue.", "methods_and_evaluation_criteria": "* Experiments are carried out on standard long context benchmarks  (LongBench) and LaCache shows very promising results.\n* Comparison against both training-based methods (StreamingLLM) and training-free methods (H20, TOVA) is very helpful.\n* Evaluation has been carried out on timely set of models", "theoretical_claims": "None", "experimental_designs_or_analyses": "* \"Continuously extending a repetitive pattern and assigning coverage to each layer as equally as possible, which is ensured by our ladder-shaped pattern, improves the lower bound of the above minimax optimality than an unequal coverage strategy.\" This is a good insight.\n* \"An ever-expanding ladder pattern with partial overlaps creates a smoother fade-out of older tokens, helping to maintain stable information retention\", also shows a deeper understanding of the problem.\n* The ablation studies section is a little weak, considering only two hyperparameters. It is not clear yet if the parameters are both necessary and complete?", "relation_to_broader_scientific_literature": "* Both contributions presented are extremely relevant for practical LLM inference systems, the ladder-based compression and the memory compaction, both seem extremely practical.", "essential_references_not_discussed": "All references are discussed and most relevant works have been used appropriately as baselines.", "other_strengths_and_weaknesses": "* The effort for building LaCache's practical considerations is very helpful as it could be deployed easily with existing systems. Great to see the compatibility with systems such as Flash Attention. Very promising work.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Minor concern: It is not clear if span and overlap are the only parameters which should be considered in the ladder-based design. Would be useful to see some additional analysis on that. Overall, this paper was a very interesting read.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xyAL8596gJ", "reviewer": "Reviewer_QVQG", "summary": "This paper proposes a method to compress KV cache with the goal of storing different sets of tokens in different layers, termed Ladder-Shaped KV Cache (LaCache). The idea is to keep earlier tokens in the sequence in the lower layer and the later tokens in the deeper layer, which intuitively makes sense.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive suggestions! We have addressed all your comments and suggestions as follows.\n\n---\n\n**Q1. Experiments on longer-context models trained on longer inputs**\n\nThank you for the suggestion! We have added experiments using both Llama3.2-3B-Instruct-128k and LongChat-7b-v1.5-32k to verify the consistent effectiveness of LaCache.\n\nOn the Needle In A Haystack (NIAH) benchmark, we evaluated both Llama3.2-3B-Instruct-128k (in [figure https://ibb.co/1GjhX0b5](https://ibb.co/1GjhX0b5)) and LongChat-7b-v1.5-32k (in [figure https://ibb.co/s9qp1s2K](https://ibb.co/s9qp1s2K)) under 50% and 25% cache budget settings. Our results demonstrate that LaCache nearly doubles the test accuracy compared to StreamingLLM under the same cache budget—for example, from 54.54% to 99.16% on the Llama3.2-3B-Instruct-128k model under a 50% cache budget, and from 33.40% to 65.30% on the LongChat-7b-v1.5-32k model under a 25% cache budget.\n\n---\n\n**Q2. Add more KV cache compression baselines**\n\nFollowing your suggestions, we have added comparisons with both SnapKV [1] and PyramidInfer [2] as shown in [figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ). This set of results consistently validates that LaCache achieves better score-throughput trade-offs across various tasks on the LongBench benchmark.\n\n---\n\n**Q3. LaCache implementation and pseudocode**\n\nThank you for the constructive suggestion! We will include a more comprehensive version along with additional implementation details in our final manuscript.\n\n---\n\n**Q4. When LaCache is applied**\n\nYes, you are correct -  it is applied at every step of decoding. Specifically, after prefilling the entire input, LaCache is used to reduce the KV cache and maintain it at a constant size by applying LaCache after each decoding step.\n\n---\n\n**Q5. Clarifications on the hyperparameters S and O**\n\nThe definitions of S and O are introduced at the end of Section 3.2 in our submitted manuscript. \n\n(a) In long-context understanding tasks such as LongBench, S is set as an integer approximately equal to the number of layers multiplied by the overall compression ratio, aiming for a uniform compression ratio distribution. For example, under a 50% cache budget, setting the span equal to half the number of model layers results in a ~50% compression ratio across different positions, helping to avoid situations where some locations are over-compressed while others are under-compressed. In language modeling tasks, S is set to 1/4 of the number of model layers, which was given by the empirical results from our ablation studies, as shown in Figure 8. \n\n(b) The choice of O depends on the task type. Specifically, a larger overlap (O) allows the information of a single token to be distributed across more positions, which is better suited for tasks requiring complex semantic understanding and greater global context. In contrast, a small overlap concentrates the information in fewer positions, which is more appropriate for tasks where the answers appear in a very narrow window. \n\n|                 | Overlap=0 | Overlap=Span/4 | Overlap=Span/2 | $\\Delta$ (Span/4 - 0) | $\\Delta$ (Span/2 - 0) |\n|-----------------|------------|------------------|------------------|----------------------|----------------------|\n| QA tasks        | 19.48      | 18.94            | 18.48            | -0.54                | -1.00                |\n| Synthetic tasks | 5.17       | 5.67             | 6.17             | +0.50                 | +1.00                 |\n\n\nFor language modeling tasks, S is set to 1/2 of Span because of better semantic continuity. For long-context understanding tasks, following your suggestion, we have added above experiments on LongBench to demonstrate the impact of the overlap parameter O. As shown in the table, a larger overlap consistently improves performance on tasks that require more global information, such as synthetic tasks (PassageCount, PassageRetrieval-en, and PassageRetrieval-zh), while reducing performance on tasks that rely more on local information, such as QA tasks (NarrativeQA, Qasper, MultiFieldQA-en, and MultiFieldQA-zh).\n\n---\n\n**Q6. Missing citations and related work**\n\nThank you for pointing this out! We have added experiments on these two baselines as shown in [figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ) and will include these two related works in our final manuscript.\n\n---\n\nThank you for your thoughtful suggestions that aim to help strengthen this paper! If you have any further questions or updated comments, we would really appreciate it and be happy to address them.\n\n---\n\n**References**\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference", "claims_and_evidence": "Experiments are conducted on the language modeling task (Wikitext and PG19) and tasks from LongBench for Llama-2-7B and Llama-3-8B. Results show that LaCache performs better than StreamingLLM on both tasks. However, I think the experiment is a little light and can be strengthened with more recent long-context models (see section on \"Experimental Designs Or Analyses\") and more baselines (see section on \"Essential References Not Discussed\").", "methods_and_evaluation_criteria": "The benchmark datasets (PG19 and Longbench) make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper primarily evaluated short context models (Llama-2-7B and Llama-3.1-8B) with up to 8K context length. These models inherently do not perform well on LongBench as most tasks have input exceeding the context length (e.g. for the PassageRetrieval task). As the proposed method aims to compress KV cache, it will be more suitable for evaluate on long-context models that can take longer input (e.g. Llama-3.1 and Qwen-2 models).", "relation_to_broader_scientific_literature": "This work contributes to the line of research on memory efficiency of long-context inference through KV cache compression, which is an active research area.", "essential_references_not_discussed": "Missing citation and related work: \n* [SnapKV](https://arxiv.org/abs/2404.14469) (NeurIIPS 2024) is a relevant method which compresses KV cache based on input tokens' attention pattern, which is a stronger and more up-to-date baseline compared to H2O and StreamingLLM.\n* [PyradmicInfer](https://aclanthology.org/2024.findings-acl.195.pdf) (ACL 2024) also proposes a layer specific KV cache eviction-based method and is a relevant baseline.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* Clarity of the proposed method: While figure 2 and figure 4 illustrate the procedure for LaCache, it would be helpful to include a pseudocode or algorithm to formally describe how the algorithm works.", "questions_for_authors": "* While it is not mentioned in the paper, I assume the compression happens at decoding time -- i.e. the model encodes the entire input for LongBench and perform LaCache compression. Is the understanding correct?\n* What are the $S$ and $O$ used in the experiments of language modeling and LongBench experiments and how are they decided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zMwUrubrH8", "reviewer": "Reviewer_gKML", "summary": "the paper presented a training-free KV cache optimization, named LaCache, for long-text generation tasks.  the proposed framework employs ladder-shaped KV cache storage pattern and an iterative compaction mechanism to enable LLMs to better capture long-range dependencies, optimize memory usage, and sustain continuous generation. the authors evaluated the proposed framework by some experiments.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and constructive suggestions! We have addressed all your comments and suggestions as follows.\n\n---\n\n**Q1. Evaluation on more datasets: Needle-in-a-Haystack (NIAH) & RULER**\n\nFollowing your suggestions, we have added experiments on the NIAH and RULER datasets. Please check our [response to Reviewer fB6E's Question 1](https://openreview.net/forum?id=SDjZtxDo35&noteId=jrewAScmqH).\n\n---\n\n**Q2. Add more KV cache compression baselines**\n\nFollowing your suggestions, we have added comparisons with SnapKV[1] and PyramidInfer[2] on the entire LongBench dataset, as shown in [this figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ). This set of results consistently validates that LaCache achieves better score-throughput trade-offs than various baselines.\n\n---\n\n**Q3. Comparison with TOVA in Figure 7**\n\nFirst, we humbly clarify and emphasize that one highlight of LaCache is its simplicity and seamless integration with the existing FlashAttention implementation, as validated by our method’s improved accuracy over baselines under the same real-device efficiency.\n\nFollowing your suggestion, we conducted experiments comparing with TOVA under more various cache budgets in [figure https://ibb.co/q3GrqCZQ](https://ibb.co/q3GrqCZQ). These experiments demonstrate that LaCache achieves better average scores than attention-based methods by a large margin under the same real-device throughput.\n\n---\n\n**Q4. The KV cache budget for LaCache in Figure 7**\n\nWe benchmarked our method against different baselines under budgets ranging from 20% to 50% in the original Figure 7. Additionally, our [updated Figure 7](https://ibb.co/q3GrqCZQ) includes more baselines and a wider range from 20% to 99%.\n\n---\n\n**Q5. The rationale and theoretical analyses of the ladder-shaped KV cache pattern**\n\nWe humbly clarify that the analysis on Page 4 of our manuscript is intended to provide a high-level rationale for why LaCache is effective rather than serving as theoretical proof. This analysis demonstrates that the ladder pattern can cover potentially important tokens more effectively under the same token budget, thus enhancing the lower bound of information retention, particularly when compared to assigning the same KV tokens across all layers.  \n\nThis analysis has been appreciated by @Reviewer cTxa, who recognized it as offering “good insights” and showing “a deeper understanding of the problem.” Additionally, as noted by @Reviewer fB6E - “I particularly like the analysis described by Figure 3, where the caching pattern is compared to randomly chosen caching patterns across cache sizes”, we have empirically verified that the proposed KV cache is robust and close to optimal. Specifically, in Figure 3 of our manuscript, we randomly generated over 1,500 patterns under different KV cache sizes to explore all possible configurations where the proposed ladder pattern lies on the Pareto optimality boundary.  \n\nIn summary, this work primarily aims to (1) empirically demonstrate that our method lies on the Pareto frontier, consistent with the evaluation protocols of most KV cache compression methods, and (2) provide the generalizable insight that the same KV token does not need to be maintained across all layers. A more rigorous theoretical proof of our method is planned for future work.\n\n---\n\n**Q6. Regarding “incremental”**\n\nWe humbly clarify that, given the increasing need for LLMs’ long-context capability in more real-world AI serving systems and applications, simplicity is a key highlight and design consideration of our proposed method. In particular, our proposed technique can (1) be seamlessly compatible with FlashAttention without introducing extra overhead during inference and (2) offer generalizable insights to guide future KV cache designs—i.e., the same KV token does not need to be maintained in all layers, and the ladder pattern is an effective way to achieve this by improving the lower bound of information retention to cover potentially important tokens. This has been positively recognized by @Reviewer cTxa, who noted, “The effort for building LaCache’s practical considerations is very helpful as it could be deployed easily with existing systems. Great to see the compatibility with systems such as Flash Attention. Very promising work.”\n\nFinally, we wish to emphasize that developing a simple yet effective technique like LaCache is nontrivial, given the extensive research already conducted in this area. This sentiment is echoed by @Reviewer fB6E, who noted, “This is a crowded space, but I think this is a nice contribution and argues its point well.”  \n\n---\n\nThank you for your thoughtful suggestions and comments that aim to help strengthen this paper! If you have any further questions or updated comments, we would really appreciate and be happy to address them.\n\n---\n\n**References**\n\n[1] SnapKV: LLM Knows What You are Looking for Before Generation\n\n[2] PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference", "claims_and_evidence": "the experimental results show that the proposed framework is effective. However, the improvement is not significant.", "methods_and_evaluation_criteria": "More datasets should to be used in the experiments. For example, Needle-in-a-Haystack, and RULER benchmarks.", "theoretical_claims": "No theoretical claims is shown in this paper.", "experimental_designs_or_analyses": "the authors should evaluate the proposed method on more benchmarks, such as Needle-in-a-Haystack, and RULER.and on recent baselines.", "relation_to_broader_scientific_literature": "incremental work", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "the main weakness of the paper is that it lack of theoretical  analysis and discussion.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. the rationale of Ladder-Shaped KV Cache Pattern is not convincing.  Theoretical analysis and discussion should be provided.  I don't fully understand why the authors presented the two formulas in page 4. The formulas don't provide convincing theoretical support for the proposed method.  \n2. What is the KV cache budget for LaCache in Figure 7?\n3. As shown in Figure 7，LaCache is not superior to  TOVA. TOVA have remarkable advantage on  F1 score while its throughtput is about 17.5, which is not much smaller than 30 (that of LaCache).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper present LaCache to compress the KV cache with comprehensive empirical validation. The ladder-shaped eviction strategy appeared to be novel. The reviews were quite diverse, with 2 accept and 1 weak reject and 1 reject. I believe the authors did a reasonable good job to address the concerns including both reviewer QVQG and gKML. One remaining concern was about the empirical comparison with SnapKV, whose accuracy appeared to be better than LaCache in the figure with slower speed. I recommend the authors to finish the remaining Pareto figure (as well as have a balanced discussion) to fully address gKML's accuracy concern.
Decision: Accept (poster)