SCBench: A KV Cache-Centric Analysis of Long-Context Methods

Authors: YUCHENG LI, Huiqiang Jiang, Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Long-context, Benchmark, KV Cache Reuse, Long-context Methods, Shared Context

Abstract:
Long-context Large Language Models (LLMs) have enabled numerous downstream applications but also introduced significant challenges related to computational and memory efficiency. To address these challenges, optimizations for long-context inference have been developed, centered around the KV cache. However, existing benchmarks often evaluate in single-request, neglecting the full lifecycle of the KV cache in real-world use. This oversight is particularly critical, as KV cache reuse has become widely adopted in LLMs inference frameworks, such as vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft, Google, and Anthropic. To address this gap, we introduce SCBENCH (SharedContextBENCH), a comprehensive benchmark for evaluating long-context methods from a KV cache centric perspective: 1) KV cache generation, 2) KV cache compression, 3) KV cache retrieval, and 4) KV cache loading. Specifically, SCBench uses test examples with shared context, ranging 12 tasks with two shared context modes, covering four categories of long-context capabilities: string retrieval, semantic retrieval, global information, and multi-task. With SCBench, we provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs (Codestal-Mamba), Mamba-Attention hybrids (Jamba-1.5-Mini), and efficient methods such as sparse attention, KV cache dropping, quantization, retrieval, loading, and prompt compression. The evaluation is conducted on six Transformer-based long-context LLMs: Llama-3.1-8B/70B, Qwen2.5-72B/32B, Llama-3-8B-262K, and GLM-4-9B. Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n^2) pre-filling computation perform robustly. Dynamic sparsity yields more expressive KV caches than static patterns, and layer-level sparsity in hybrid architectures reduces memory usage with strong performance. Additionally, we identify attention distribution shift issues in long-generation scenarios.

Decision: accept (poster)
Reviewer scores: 8, 3, 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730391722229

thread_type:
official_review

<thread_object>
creator: reviewer_tui2

<content>
summary:
The paper introduces SharedContextBench, a comprehensive benchmark designed to evaluate long-context Large Language Models (LLMs) in scenarios where multiple requests share the same input context. The benchmark covers 12 tasks across four categories of long-context abilities: string retrieval, semantic retrieval, global information processing, and multi-tasking. It includes two shared context modes: multi-turn and multi-request. The paper evaluates five categories of long-context methods on six transformer-based long-context LLMs, highlighting the impact of sparsity in encoding and decoding, task complexity, and more.

strengths:
1. The benchmark includes tasks that reflect real-world applications, such as multi-turn conversations, self-play Chain-of-Thought reasoning, and repository-level code debugging.
2. The extended retrieval tasks of key-value retrieval, prefix-suffix retrieval, and multi-hop retrieval are interesting to benchmark the LLMs.
3. The paper provides a detailed analysis of various long-context methods, highlighting their strengths and weaknesses in different scenarios, which offers valuable insights for future research and practical applications.
4. The taxonomy of different long-context methods in Table 1 is comprehensive and help to illustrate their advantages and disadvantages.
5. The paper highlights the challenges faced by sub O(n) memory methods in maintaining accuracy in multi-turn scenarios, providing an important insight into the limitations of these methods.
6. The study shows that sparse encoding methods with O(n) memory and sub $(O(n^{2})$ computation in prefilling generally perform well, offering a practical solution for efficient long-context processing.
7. The paper finds that dynamic sparse patterns in prefilling often produce more expressive memory compared to static methods, suggesting a direction for future research in optimizing long-context methods.
8. The breakdown results (Table 8 and 9) help to show more fine-grained insights.

weaknesses:
1. The paper provides configuration details for the long-context methods, but it lacks a detailed analysis of how hyperparameters were tuned or their impact on performance.
2. A more thorough analysis of hyperparameter sensitivity could provide insights into the robustness of the proposed methods.
3. Table 4 is a little confusing. Why some highest scores are not bold? And what is the meaning of the underscore?

questions:
Could you provide the source codes for the community to verify other methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490300208

thread_type:
official_review

<thread_object>
creator: reviewer_wddv

<content>
summary:
The paper presents **SharedContextBench**, a benchmark for evaluating long-context Large Language Models (LLMs) in realistic, multi-turn, shared-context scenarios requiring KV cache reuse. It includes 12 tasks across four abilities—string retrieval, semantic retrieval, global information processing, and multi-tasking—tested in multi-turn and multi-request modes. Evaluated across five categories of open-source LLMs, results show sub-O(n) memory methods struggle in multi-run scenario while sparse and full attention methods perform well.

strengths:
1. SharedContextBench fills an important gap by providing tasks that reflect the performance of LLM in realistic, multi-turn interactions.
2. The benchmark includes diverse tasks that thoroughly test LLM's long-context processing abilities, such as retrieval and global information aggregation, which matches the needs in research community.
3. By testing five different long-context methods on a range of LLMs, the benchmark offers valuable comparative insights, making it a robust tool for LLM evaluation in shared-context scenarios. The visualization in the paper is also very intuitive and helpful.

weaknesses:
1. It would be more convincing to include a table listing the source of the dataset of each task.

questions:
1. The benchmark is primarily designed to evaluate the accuracy, but with some adjustment people may also use it for performance evaluation on multi-run scenario?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730513959256

thread_type:
official_review

<thread_object>
creator: reviewer_jf9i

<content>
summary:
This paper introduces a long-context benchmark SharedContextBench, to reveal how lossy are long-context methods in KV cache reuse scenarios. SharedContextBench includes 12 tasks with two shared context modes, covering four categories of long-context abilities: string retrieval, semantic retrieval, global information processing, and multi-task capabilities. The proposed benchmark is evaluated with several long-context methods and six LLMs.

strengths:
1.	The paper introduces the long-context benchmark SharedContextBench, to reveal how lossy are long-context methods in KV cache reuse scenarios. The proposed benchmark includes 12 tasks with two shared context modes.
2.	The proposed benchmark is evaluated with several long-context methods and six LLMs. The experimental evaluation is comprehensive.
3.	The paper is easy to follow.

weaknesses:
The paper did not analyze other long-context benchmarks. It would be benefical, if related long-context benchmarks are summarized and compared in a table.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730532234704

thread_type:
official_review

<thread_object>
creator: reviewer_ftbz

<content>
summary:
This paper introduces a new benchmark designed to evaluate model performance on tasks at the intersection of long-context and shared-context settings. The authors argue that existing benchmarks inadequately address this domain, which is critical because long-context methodologies facilitate applications with substantial context reuse. Using this benchmark, the authors assess state-of-the-art open-source transformer models, their adaptations for long-context processing—such as sparse attention, KV caching, and prompt compression techniques—as well as gated linear RNN models. Additionally, the paper contributes a new static sparse attention mechanism that performs better than existing sparse attention methods. Finally, they analyze the effectiveness of various long-context strategies.

strengths:
*Originality*
This benchmark measures the performance of long context language models in settings that are currently unaddressed by existing benchmarks.

*Quality*
The benchmark covers a wide range of tasks that vary in reasoning complexity and memorization difficulty. All tasks are performed in two shared context modes that are commonly used in real applications. Additionally, the paper demonstrates that the benchmark can be used to draw novel insights about the effectiveness of different long context approaches by evaluating state-of-the-art language models with a wide range of long context methods.

*Clarity*
The goal and contribution of the paper are very clear. Experimental results are clearly visualized.

*Significance*
Many applications enabled by long context models involve context sharing. Thus, this benchmark enables other researchers to make more informed decisions when building long context language models that are effective in real use cases.

weaknesses:
1. The authors listed prefix caching methods in the related works section but they were missing from the experiments.
2. The paper does not explicitly highlight the unique insights that they were able to obtain with this benchmark that do not surface with other benchmarks. For example, it does not specify whether the relative performances of these methods consistent when they are measured with other benchmarks.
3. There does not seem to be a discussion on how well the synthesized data reflects real use cases. In particular, the retrieval use cases artificially vary the positions of the target strings to ensure long context utilization. There is no discussion on its importance in real life use cases, where queries may obey a power law distribution.
4. It is unclear how the proposed benchmark offered insights that led to the creation of the proposed Tri-shape sparse attention mechanism or how the benchmark uniquely highlights the benefits of this new mechanism, especially since it only significantly outperforms A-shape sparse attention in the first turn.

questions:
1. Why do the experiments exclude prefix caching methods?
2. What unique insights did the proposed benchmark offer that guided the design of the proposed Tri-shape sparse attention mechanism?
3. Are there inconsistencies between the results obtained with the proposed benchmark and existing long-context or multi-turn benchmarks that can demonstrate the unique insights that this benchmark can provide?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730551782693

thread_type:
official_review

<thread_object>
creator: reviewer_ww1y

<content>
summary:
This paper introduces SharedContextBench, a benchmark specifically designed to evaluate the efficacy of long-context methods in scenarios that utilize KV cache reuse. It features an array of 12 tasks that test four long-context abilities across two modes of shared context. Evaluations are performed on five different categories of long-context methods using eight leading LLMs, with a detailed analysis of the results to uncover insights into the management of KV caches.

strengths:
- SharedContextBench represents a significant innovation in benchmarking by addressing the evaluation gap in shared long-context scenarios, providing a robust assessment tool for practical applications. 
- The benchmark comprises a wide array of tasks across various capabilities and domains, ensuring a comprehensive evaluation of long-context methods. 
- A diverse range of these methods are tested against multiple large language models, offering an extensive overview of their performance. The findings bring to light considerable variations in KV cache management, emphasizing the critical importance of multi-turn and shared-context scenarios in method development. 
- The paper also delves into detailed analyses concerning memory usage and the sparsity observed in encoding and decoding processes, yielding valuable insights for future research endeavors.

weaknesses:
- Certain methods, such as KV cache compression, demonstrate limited effectiveness in shared contexts, which may restrict their practical deployment. 
- The paper points out that many of the long-context methods tested are essentially extensions of existing models adapted for multi-turn dialogues, such as StreamingLLM. 
- The tasks within the benchmark may not capture all possible real-world application scenarios, possibly overlooking specific needs within certain domains. 
- Moreover, while the performance of these methods is thoroughly evaluated, the paper does not explore the deeper internal mechanisms of the models, such as the dynamics of information propagation in sparse attention mechanisms.

questions:
- How were the real-world use cases that influenced the design of the benchmark tasks selected, and what impact did they have on the weighting of the evaluation criteria? 
- What improvements could be implemented in sub-linear memory methods to enhance their applicability and efficiency in multi-turn interactions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731419897498

thread_type:
official_review

<thread_object>
creator: reviewer_5eba

<content>
summary:
This paper proposed a new (set of) dataset focusing on the multi-round capability of LLMs under a long context setting.

strengths:
1. The proposed dataset fills a major gap in long context evaluation — where there is often only a single question after a long input, with the question being answerable by simply retaining a few pieces of information within the long context — this makes a lot of eviction-based methods seemingly able to achieve strong performance on many long context datasets, although they are not actually long context capable.

2. Aside of the contribution of collecting this new dataset, the author also conducted a decent benchmark of different methods on their dataset.

weaknesses:
1. The author should consider highlight the source and origin of each dataset a bit better. Many of the featured datasets looks similar from InfiniteBench. I am not sure if they are direct adoptions or modified; in either case, this deserve a better noting.

2. While the proposed dataset focused on multi-round QA under a "shared context" scenario. It looks like few or none of its question demand information from a previous QA. The author should consider adding some datasets (or new setting of existing datasets) covering this gap, or at least highlight it in its limitation.

questions:
Recently, there have been many newly proposed long context evaluations (HELMET, Michelangelo, etc.) I wonder if the author can provide a discussion regarding such works so that future users will have some guidance on what to adopt.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732174868406

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We greatly appreciate the comprehensive reviews and insightful feedback provided by each reviewer. In response to your feedback, we've made the following updates to our paper draft:

1. _**"...the source of dataset..."**_

We have already indicated the source datasets in Table 3. Out of the 12 datasets, three are from InfiniteBench, one from RepoQA, one from RULER, one from ManyshotICL, one for Lost in the Middle, and the remainder were developed by us. To better acknowledge the non-original datasets used in SharedContextBench, we have added direct citations in Table 3 in the updated paper (also show in RTable 1).
We would like to emphasize that although some original data originated from other datasets, these previous datasets only provided single-turn data. We have regenerated and developed corresponding multi-request datasets.

|Task|Source|
|-|-|
|Retrieval.KV|Lost in the Middle|
|Retrieval.Prefix-Suffix|Ours|
|Retrieval.Multihop|RULER|
|Code.RepoQA|RepoQA|
|En.QA|InfiniteBench|
|Zh.QA|InfiniteBench|
|En.MultiChoice|InfiniteBench|
|Math.Find|Ours|
|ICL.Manyshot|ManyshotICL|
|En.Sum|Ours|
|Mix.Sum+NIAH|Ours|
|Mix.RepoQA+KV|Ours|

RTable 1. The source of SharedContextBench.

2. _**"...compared with others long-context benchmark..."**_

Thank you for your suggestion. We have added a new table (RTable 2) to provide a more **direct comparison of our proposed SharedContextBench against existing long-context benchmarks** in terms of long-context capability, request mode, and implementation, highlighting the novel contributions of our benchmark. We have also provided new experimental results of previous benchmarks such as **LongBench and InfiniteBench and contrasted them directly with our proposed SharedContextBench**, as shown in RTable 3. This will help readers better understand the unique insights our benchmark provides, which prior benchmarks have overlooked:  
1) **SharedContextBench** offers better differentiation between methods, even in summarization tasks;  
2) The limitations of KV cache compression methods in handling multi-request and multi-turn modes on SharedContextBench;  
3) The improved retrieval accuracy of sparse attention mechanisms, such as A-shape and Tri-shape attention.

|Benchmarks|Precise Retrieval | Semantic Retrieval | Global Information | Multi-tasking | Single-request|Multi-turn|Multi-request|KV Cache Reuse|
|-|-|-|-|-|-|-|-|-|
|LongBench|x|√|√|x|√|x|x|x|
|InfiniteBench|√|√|√|x|√|x|x|x|
|RULER|√|√|√|x|√|x|x|x|
|HELMET|√|√|√|x|√|x|x|x|
|Michelangelo|√|√|x|x|√|x|x|x|
|SharedContextBench|√|√|√|√|√|√|√|√|

RTable 2. Comparison of Long-Context Benchmarks.

|Summarization|InfiniteBench|LongBench|Ours:Multi-Request|Ours:Turn-1|Turn-2|Turn-3|Turn-4|Turn-5|
|-|-|-|-|-|-|-|-|-|
|Llama-3.1-8B-Instruct|28.5|36.6|38.3|44.2|42.1|35.8|37.6|42.3|
|A-Shape|24.5|33.5|28.8|26.1|30.8|33.8|40.8|40.4|
|Tri-Shape|27.4|33.9|30.2|32.1|30.0|34.0|41.0|40.3|
|Minference|28.9|33.9|36.7|40.6|36.1|39.7|43.5|43.7|
|StreamingLLM|27.3|32.0|30.2|29.4|26.1|27.7|27.3|26.9|
|SnapKV|28.3|33.2|29.9|36.2|29.4|28.6|28.1|31.0|
|LLMLingua|23.1|32.0|30.1|32.5|22.5|26.6|25.7|26.6|


RTable 3 (a). Comparison of efficient long-context methods on the summarization task of prior benchmarks and ours.

|Retrieval|InfiniteBench|LongBench|Ours:Multi-Request|Ours:Turn-1|Turn-2|Turn-3|Turn-4|Turn-5|
|-|-|-|-|-|-|-|-|-|
|Llama-3.1-8B-Instruct|57|100|56|62|59|68|66|70|
|A-Shape|0|42|3|0|12|22|28|33|
|Tri-Shape|21|100|5|14|19|25|32|38|
|Minference|33|100|14|31|35|46|56|50|
|StreamingLLM|0|84|0|2|1|0|0|0|
|SnapKV|4|100|0|0|0|0|0|0|
|LLMLingua|0|90|0|0|1|2|0|0|


RTable 3 (b). Comparison of efficient long-context methods on the summarization task of prior benchmarks and ours.

3. _**"...different sparisty budget..."**_

We have supplemented the performance of different efficient methods under various compression budgets on Llama-3.1-8B, as shown in Figure 6 and 7 in updated paper. From the results, we can derive the following insights:  

1) Most methods show minimal performance degradation at a 1/2 budget (e.g., A-shape and Tri-shape drop by 5-6%, SnapKV drops by 11%). However, as sparsity increases, performance declines significantly. For example, StreamingLLM and SnapKV drop by 26% and 19%, respectively, under a 1/4 budget.  
2) More accurate sparse methods can maintain performance even under higher sparsity. For instance, MInference achieves performance at a 1/32 budget comparable to A-shape and Tri-shape at a 1/4 budget.  
3) While some methods exhibit similar performance in single-turn scenarios, they diverge significantly in multi-turn and multi-request scenarios. For example, SnapKV outperforms StreamingLLM in turn-1 but performs significantly worse in turn-2. In some tasks, changing the budget has little impact on turn-1 performance but substantially affects turn-2 and subsequent turns, such as in Long Document QA tasks and summarization.

We will include this analysis in the main text and highlight it in the next version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces SharedContextBench, a benchmark specifically designed to evaluate the efficacy of long-context methods in scenarios requiring KV cache reuse. The benchmark comprises 12 tasks across four categories of long-context capabilities, tested in two shared context modes. By assessing diverse methods—including sparse attention, prompt compression, and hybrid architectures—on various state-of-the-art long-context models, the authors reveal the limitations of sub-linear memory methods and highlight the strengths of dynamic sparse patterns. The key strength lies in filling an essential gap in long-context evaluation by focusing on multi-turn, shared-context scenarios critical for real-world applications. The submission also provides detailed analyses and comparative insights, demonstrating the practical utility of sparse attention mechanisms like Tri-shape in KV cache reuse. However, some reviewers noted areas for improvement, including clearer attribution of dataset sources, better discussion of synthesized versus real-world scenarios, and hyperparameter sensitivity analysis. The decision to accept is based on the benchmark’s novel contributions, robust methodology, and its potential to drive future research in scalable and efficient long-context LLMs.