SCBench: A KV Cache-Centric Analysis of Long-Context Methods
Authors: YUCHENG LI
Huiqiang Jiang
Qianhui Wu
Xufang Luo
Surin Ahn
Chengruidong Zhang
Amir H. Abdi
Dongsheng Li
Jianfeng Gao
Yuqing Yang
Lili Qiu
Venue: ICLR 2025 Poster (2025)
Keywords: Long-context
Benchmark
KV Cache Reuse
Long-context Methods
Shared Context
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Long-context Large Language Models (LLMs) have enabled numerous downstream applications but also introduced significant challenges related to computational and memory efficiency. To address these challenges, optimizations for long-context inference have been developed, centered around the KV cache. However, existing benchmarks often evaluate in single-request, neglecting the full lifecycle of the KV cache in real-world use. This oversight is particularly critical, as KV cache reuse has become widely adopted in LLMs inference frameworks, such as vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft, Google, and Anthropic. To address this gap, we introduce SCBENCH (SharedContextBENCH), a comprehensive benchmark for evaluating long-context methods from a KV cache centric perspective: 1) KV cache generation, 2) KV cache compression, 3) KV cache retrieval, and 4) KV cache loading. Specifically, SCBench uses test examples with shared context, ranging 12 tasks with two shared context modes, covering four categories of long-context capabilities: string retrieval, semantic retrieval, global information, and multi-task. With SCBench, we provide an extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs (Codestal-Mamba), Mamba-Attention hybrids (Jamba-1.5-Mini), and efficient methods such as sparse attention, KV cache dropping, quantization, retrieval, loading, and prompt compression. The evaluation is conducted on six Transformer-based long-context LLMs: Llama-3.1-8B/70B, Qwen2.5-72B/32B, Llama-3-8B-262K, and GLM-4-9B. Our findings show that sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding with O(n) memory and sub-O(n^2) pre-filling computation perform robustly. Dynamic sparsity yields more expressive KV caches than static patterns, and layer-level sparsity in hybrid architectures reduces memory usage with strong performance. Additionally, we identify attention distribution shift issues in long-generation scenarios.

Published as a conference paper at ICLR 2025
SCBENCH: A KV CACHE-CENTRIC ANALYSIS OF LONG-CONTEXT METHODS
Yucheng Li!, Huiqiang Jiang', Qianhui Wu, Xufang Luo, Surin Ahn, Chengruidong Zhang, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu Microsoft Corporation, Uniiversity of Surrey yucheng 1 @esurrey............. wwtrreyyccuuuu. .c.uk, uk, (h (hjiang, yugyang) emicrosoft.com htpss//aka.ms/SCBenchh
ABSTRACT
Long-context Large Language Models (LLMs) have enabled numerous downstream applications but also introduced significant challenges related computational and memory efficiency. To address these challenges, optimizations for long-context inference have been developed, centered around the KV cache. However, existing benchmarks often evaluate in single-request, neglecting the full lifecycle of the KV cache in real-world use. This oversight particularly critical, as KV cache reuse has become widely adopted in LLMs inference frameworks. such as vLLM and SGLang, well as by LLM providers, including OpenAI. Microsoft, Google, and Anthropic. To address this gap, we introduce SCBENCH (SharedContextBrNe) a comprehensive benchmark for evaluating long-context methods from KV cache- centric perspective: 1) KV cache generation, 2) KV cache compression, 3) KV cache retrieval, and 4) KV cache loading. Specifically, SCBench uses test examples with shared context, ranging 12 tasks with two shared context modes, covering four categories of long-contex: capabilities: string retrieval, semantic retrieval, global information, and multi-task. With SCBench, provide extensive KV cache-centric analysis of eight categories long-context solutions, including Gated Linear RNNs (Codestal-Mamba), Mamba-Attention hybrids (Jamba-1.5-Mini), and efficient methods such as sparse attention. KV cache dropping, quantization, retrieval, loading, and prompt compression. The evaluation conducted six Transformer-basee long-context LLMs: Llama-3. 1-88/70 Qwen2.5-72B/32B, (lama-3-8B-2622 and GLM-4-9B. Our findings show that sub-O(() memory methods suffer in multi-turn scenarios, while sparse encoding with ((n) memory and sub-O(n²) pre-filligg computation perform robustly. Dynamic sparsity yields more expressive KV caches than static patterns, and layer-level sparsity in hybrid architectures reduces memory usage with strong performance. Additionally, we identify attention distribution shift issues in oong-eeneration scenarios.
KV Cache Generation
KV Cache Compression
KV Cache Retrieval
KV Cache Loading
LLMs
Requesta
Now
Full
Rie quqquett
Extract Match
Attention
KV Cache
KV Cacher
Previous Long-context
Prompt Compression, oug Miniinge
Semantic
Banchmarks
Cache
Retrieval,
LLMs decoding
Multi
LLMs
Quantization, ag
or Hybird.
Sparse Attention,
Write
KV
Read
KV Cache
Loading
KV
Drapping, 2 Skappnnint
Cache
#g Ouenc
Mivipenno
Response
SCBench
Figure I: KV Cache lifecycle. Prior benchmarks focus on single-request, while real-world applica- tions KV cache across requests. We propose SCBench and categorize long-context methods into KV Cache Generation, Compression, Retrieval, and Loading from a KV-cache-centric perspective.
Work during internship Microsoft. CCorrespondngg author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Long-come Long-context capability is becoming a standard for Large Language Models (LLMs), with many of them supporting context windows ranging from 128K to 10M tokens (Reid et al., 2024; Lieber et al., 2024; Dubey et al., 2024; Gradient, 2024). These extended context windows unlock wide range of real-world applications, such as repository-level code understanding and debugging (Baini et al., 2024; Park et al., 2023; Liu et al., 2024c; Jimenez et al., 2024), long-document question- answering (Caciularu et al., 2023; Li et al., 2024b), many-shot in-context learning (Agarwal et al., 2024), and cong-generation Chain-of-Thougee (CoT) reasoning (OpenAI, 2024a; Snell et al., 2024).
Despite the benefit, Long-context inputs also present unique challenges for LLM inference due to high computational costs and memory demands. This has led to the development of efficient long-context solutions leveraging sparsity in various stage of KV cache. In this paper, we introduce an unified analysis framework for efficient long-context methods in KV cache centric perspective, consisting of the four essential stages of KV cache: 1) KV cache generation. 2) KV cache compression, 3) KV cache retrieval, and 4) KV cache loading. First, KV cache generation, aka prefilling, processes the input prompt and produce the KV cache to be used in decoding. In this stage, sparse attention methods are proposed to reduce the complexity of the attention operation (Child et al., 2019; Beltagy et al., 2020; Jiang et al., 2024). Second, KV cache compression techniques prune KV states to reduce the memory costs in decoding (Xiao et al., 2024; Li et al., 2024c). Third, KV cache retrieval aims skip KV cache generation of an incoming request, and instead retrieves and reuses KV cache from history KV cache pool for more efficient inference (Zheng et al., 2024; Yao et al., 2024a). At last, KV cache loading aims to load only partial of the KV cache for each decoding step to save the memory and computing cost (Tang et al., 2024; Liu et al., 2024b).
However, these methods are only evaluated on single-request benchmarks (Hsieh al., 2024; Zhang et al., 2024a; Kamradt, 2023; Li et al., 2024a), which fail to covers the full lifecycle of KV cache in real applications. Typically, real-world applications often require reusing prompt memory (i.e., KV cache) and involving multiple requests or multi-round interactions (Qin et al., 2025). The reuse of KV cache, known as prefix caching. is already crucial component in popular inference frameworks (Zheng et al., 2024; vLLM, 2024) and used by LLM providers (Gemini, 2024; Claude, 2024; OpenAI, 2024b; Azure, 2024). In addition, testing with multiple requests especially crucial for the long-context methods mentioned earlier, as many achieve efficiency through guery-conditioned compression. For instance, Arora et al. (2024) reports that Mamba's compression of previous information based on the current query can prevent it from answering follow-up queries.
To address this gap, we introduce SCBench, a benchmark designed to evaluate efficient long-context methods that covers the full lifecycle of KV cache in real-world scenarios, particularly for shared context and multi-round interactions where KV Cache is reused for follow-up queries. As shown in Fig. 2b, SCBench assesses four key long-context abilities across 12 tasks with two shared context modes. Each test example includes a shared context and multiple follow-up queries. The four long-context capabilities and their corresponding tasks are:
1. String Retrieval Capability: A fundamental requirement for long-context LLMs is retrieving relevant context with exact matches from long inputs. We extend previous retrieval tasks like NIAH and Multi-NIAH (Kamradt, 2023; Hsich et al., 2024) by introducing three comprehensive
KV Cache, GEU/CRU
Compression by
Cache, GPUJSPU
Long-context Capability
Shared Context Modes
Rebrivaal Capability
Global Information Capability
Multi-tosks Capabilty
Long: context
All
Hit KV Cachel
Level tng Beitrevat Rebtevai
Mutti-uurn mode
long-context- Q1
AZ
Hit KV Cache
02
A2
Motch, M MUll-nup
Monystoot
*
Reuse
Long- pong-contoxt-- + Al C2
(1) long-conteet multi turn mode
A2-003 AS
Hit Cache
(2)
A3
Level Samantic Rutt: inporisliing (ooo
ICL orton.
Mathirin
otion w/
Mutti-re unots mode
request
(a) Two Shared Context Modes
(b) Overview SCBench
Figure 2: Long- Long-context tasks often involve contexts sharing, e.g., multi-tum dialogues, multi-step reasoning, and repository-level tasks. (a) Illustration of two common shared-context patterns. (b)
Overview of tasks and scenarios covered by our benchmark, encompassing four categories of long- context abilities and two shared-context modes.
2

---

Published as a conference paper at ICLR 2025
Fullrbention
LUMLingua
FulAttention
shish
LLMLingua
jamba
Minference
SnapkV
Quest
A-shape
Streaming
Quest
TrisPape
Snapky
RetrievalAttenton
50
Multi-Ta
Ret L2
Mult--uum
Maititurns
Long
Cadhe
Accuracy
30-
Multi- -reti-re
20
Long-context Methods w/ Giko KV Cache
Ret Multi-re
T-TTaks Mult-rea
10
1st
2nd
4th Number of Requests
5th
Global Multi-req
(a) Performance Across Different Requests
Performance Different Abilities
Figure 3: Overview of performance results for SCBench. (a) Performance trends of various long- context methods across multiple requests. Methods with O(n) memory cost in decoding show improving performance as requests increase. In contrast, methods with sub-O(n) KV cache in decoding, like KV cache dropping methods, perform well only in the first request. (b) Specific performance of different long-context methods across various long-context capability tasks. All evaluated long-context methods exhibit some loss in Retrieval capability while largely maintaining Global Information processing capability.
string retrieval tasks: key-value retrieval, prefix-suffix retrieval, and multi-hop retrieval, measuring capability at different levels of granularity.
2. Semantic Retrieval Capability: Real-world applications often require long-context LLMs to understand semantic meaning before succeeding in retrieval. We considered various semantic retrieval scenarios across different domains, building four distinct tests: RepoQA (Liu et al., 2024c) and long-form QA (covering English, Chinese, and multiple-choice questions) (Zhang et al., 2024a).
Global Information Capability: We also assess the capability of long-context LLMs to process and aggregate global information through three tasks: many-shot in-context learning (Agarwal et al., 2024), summarization, and long array statistics (Zhang et al., 2024a).
Multi-tasking Capability: In real applications, LLMs often handle multiple tasks with shared long-context input. Our benchmark evaluates this capability through two tasks: RepoQA with NIAH and summarization with KV retrieval.
In addition, as shown Fig. 2a, our bench- mark includes two typical shared context modes: Multi-turn Mode, where the context is cached within single session, and Multi-request Mode, where it is cached across multiple sessions.
40
Mirforence
SCBench Acc(%)
Aishape
Tisshape
30-
Steaming
O SiapkV
With SCBench, conduct extensive KV cache-centric analysis in the four stages as shown in Fig. (details in $2). Specifically, we evaluate 13 long-context methods across four stages and eight categories on eight open- source long-context LLMs, including Llama- 3,1-8B/70B (Dubey et al., 2024), Qwen2.5- 72B/32B (Team, 2024), Llama-3-880 3-8-2-2----66KK (Gra- dient, 2024), GLM-4-9B- IM (GLM et al., 2024), Codestal Mamba (team, 2024), and Jamba-1.5-
@ Pynemidez
KM
Quest
1 1
32 16
2
Compression Rate
Figure 4: Performance of various long-context
methods different compression rates on
SCBench using Llama-3.1-4 3-BB (Dubey et al., 2024).
mini (Lieber et al., 2024). These methods span gated linear RNNs (e.g.. Codestal Mamba), hy- brid models (e.g., Jamba- sparse attention (e.g., A-shape, Tri-shape, MInference (Jiang et al., 2024)), prompt compression (c.g., LLMLingua-2 (Pan et al., 2024)). KV cache dropping (e.g.. Streamingl.LMM (Xiao et al., 2024), SnapKV (Li al., 2024c)), KV cache quantitation (e.g., KIVI (Liu
3

---

Published as a conference paper at ICLR 2025
al., 2024e)), semantic retrieval (e.g., CacheBlend (Yao et al., 2024a)), and KV cache loading (e.g.. Quest (Tang et al., 2024), RetrievalAttention (Liu et al., 2024b)), as detailed in Table 1. Additionally, introduce Tri-shape, a novel, training-free free sparse attention method that demonstrates improved first-turn performance in our evaluations.
Our experimental results reveal the following insights: 1) (nb-O(n) memory is almost infeasible in multi-turn decoding, as shown in Fig. 3. Sparse decoding methods (sub-O(n) memory) perform well on the first query but lose accuracy in subsequent requests. In contrast, sparse encoding methods (O(n) memory with O(n²) computation during pre-filling) can approximate full attention accuracy across multiple queries. 2) Task performance declines varying rates, as illustrated in Fig. 3b. Sparse KV cache methods excel in tasks requiring global information. while (n) memory crucial for exact match retrieval tasks. 3) All long-context methods degrade as the budget decreases, as shown in Fig. 4. However, sub-O(n) memory methods experience sharp performance drop 1/4 compression rate. Methods like RetrievalAAttention and KIVI, which maintain ( memory with sparse decoding. sustain higher performance even under higher compression rates. 4) Long-generation scenarios exhibit distribution shift issues. As generation length and the number of rounds increase, the KV cache's 's importance distribution changes significantly. This -f-ttri-iitiittiinnn (OOD) issue leads to performance degradation, even for O(n) memory methods like RetrievalAttennion shown in Fig.
Our contributions are as follows:
We propose new benchmark, SCBench, to evaluate long-context methods on multi-round and multi-request scenarios in two typical KV cache reuse scenarios, providing a more realistic assessment.
We design extensive set of downstream tasks, covering four long-context capabilities across 12 subtasks in various domains.
We systematically categorize long-context methods from a KV-cache-centric perspective and evaluate 13 different long-context methods (including our newly proposed sparse attention method, Tri-shape) on eight state-of-the-arr open-source long-context LLMs using SCBench. Our comprehensive analysis reveals key insights into the effects of sparsity in encoding and decoding, task complexity, and more.
2 A KV CACHE-CENTRO PERSPECTIVE ON LONG-CONTEXT METHODS
Table 1: We evaluated long-context methods on SCBench, where represents the token size of the input prompt and m represents the generation token size, with n > m.
P-stage Efficient
D-stage Efficient
KV Cache
Prefilling Complexity
Decoding Complexity
Methods
Taxonomy
Stage
Size
Codestral Mamba (team. 2024)
Gated Linear RNN
/
O(k)
O(km)
O(km)
Jamba (1.ieber et al., 2024)
Gated Linear RNN
/
O(n)
O(a²)
0(mm)
Full Attention
LLMLingua-2 (Pan et al., 2024)
Prompt Compression
/
x
O(an)
O(a?n²
O(anm)
A-shape (Xiuo et al., 2024)
Tri-shape
Sparse Attention
X
O(z)
O(km)
((mm)
MInference (Tiang al., 2024)
L (Xiao et 2024)
SnapKV (Li et al., 2024c)
KV Cache Dropping
x
/
O(k)
0(12)
O(km)
PyramidKV (Cai al., 2024)
KIVI (Liu et al., 2024e)
KV Cache Quantitation
x
/
O(n)
O(n²)
((rm)
CacheBlend (Yao et al., 2024a)
KV Cache Retrieval
x O(n)
O(n²)
0(um)
Quest (Tang et al., 2024)
Cache Loading
/ O(n)
O(n²)
O(km)
RetrievalAAtentioon (Liu et ul., 2024b)
Recently, series of works (Gu & Dao, 2024; Xiao et al., 2024; Jiang et al., 2024) have explored vari- ous strategies to reduce the inference cost long-context LLMs, enabling their application (Jimenez al., 2024; OpenAI, 2024a) to downstream tasks lower computational expense. In long-context
4

---

Published as a conference paper at ICLR 2025
LLM inference, the KV cache plays pivotal role by effectively reducing the computational overhead during the decoding phase. This importance has led to the development of numerous system- optimizations (Sheng et al., 2023; Qin et al., 2025) focused on KV cache management and scheduling.
In this work, we propose novel perspective: these long-context methods can viewed as optimiza- fions centered around the KV cache different stages. Specifically, introduce KV-cache-centric framework that systematically categorizes long-context methods into four stages: KV Cache Genera- tion, Compression, Retrieval, and Loading, as illustrated in Fig.
Specifically, the four stages of the KV-cache-centric framework are defined as follows:
1. KV Cache Generation: This stage optimizes the efficient generation of KV cache during inference. Techniques include sparse attention (e.g., A-shape. Tri-shape, MInference (Jiang et al., 2024), NSA (Yuan et al., 2025), MoBA (Lu et al., 2025)), SSM or hybrid approaches (e.g.. Mamba (Gu Dao, 2024; Dao & Gu, 2024; Lieber et al., 2024)), and prompt compression (e.g.. LLMLingua- 2 (Pan et al., 2024)).
KV Cache Compression: After generation, the KV cache is compressed before being stored. Methods include KV cache dropping (e.g.. StreamingLLM (Xiao al., 2024), SnapKV (Li et al., 2024c)) and KV cache quantization (e.g., KIVI (Liu et al., 2024e)).
KV Cache Retrieval: Relevant KV cache blocks are retrieved from a storage pool based on the request's prefix, reducing methods like CacheBlend (Yao et al., 2024a).
(TTFT), Approaches include semantic retrieval
This stage dynamically loads the KV cache and computes sparse attention, from KV cache storage (e.g.. VRAM, DRAM, SSD, or RDMA) to GPU on-chip SRAM, including Quest (Tang et al., 2024), RetrievalAttention (Liu et al., 2024b), and MagicPIG (Chen et al., 2025).
KV Cache Loading:
In our work, we evaluate four stages 13 long-conter methods, as shown in Table 1. Additionally, list the KV cache size, pre-filling stage complexity, decoding stage complexity, and whether efficient operations are performed during the pre-filling and decoding stages for each method.
Tri-shape Sparse Attention: We introduce Tri- Tri-shape, novel training- free sparse attention method that improves first-turn accuracy (Fig. 5). Unlike A-shape, which retains only the sink token and local window, Tri-shape also pre- serves the last window query region, forming a triangular sparse attention pattern for pre-filling. Motivated by our SCBench findings that A-shape with dense decoding im- proves after multiple requests, Tri-shape enhances both turn-0 and multi-request performance ($4) while main- taining LLM enstructionfoolloing ability ($G). Notably, recent concurrent work (Acharya et al., 2024) has explored similar patterns for accelerating long-context pre-filling
A-shape head
Tri-shape head
Figure 5: The sparse attention methods framework.
BENCHMARK BUILDING
SCBench features 12 tasks assessing four long-context abilities: string retrieval, semantic retrieval, global information processing, and multi-tasking, across two shared context modes-multi-tun and multi-request. These tasks span diverse domains, including code, retrieval, question answering, summarization, in-context learning, and multi-hop tracing (Fig. 2b). In total, SCBench includes 931 multi-turn sessions with 4,853 queries, averaging turns per session. Task statistics are provided in Table 2, with examples and configurations in Table 3. Below, we describe the benchmark construction.
3.1 LONG-CONTEXT TASK DETAILS
String Retrieval The core requirement for long-contexs LLMs retrieving relevant information from lengthy, potentially noisy inputs. String retrieval tasks commonly used for this evalua- tion (Hsich et al., 2024; Zhang et al., 2024a). Our benchmark applies complexity analysis, inspired by algorithmic problem-solving (e.g., LeetCode), design three tasks with varying difficulty levels.
5

---

Published as a conference paper at ICLR 2025
Table 2: Overview of SCBench tasks.
Task
Description
Capability
Avg. Input Length
Avg. Output
#Sessions
Length
/
Retr.KV
value retrieval from many key value pairs Find string with specific prefix and suffix diet
String Retrieval
125K
943
100X500
Retr. Suffix
String Retrieval
112K
914
100/500
Muli.Ho
Tracking variables ussignment in long input
String Retrieval
124K
410
90/450
Code: RepooA
Functions retrieval from GitHub repo
Semantic Retrieval
65K
6,058
88/440
En.QA
English Question Ans weriing
Semantic Retrieval
198K
272
69/351
Zh.QA
Chinese Question Answering
Semantic Retrieval
1.5M
322
35/189
En. MultiChoice
English Multi- Choich Questions
Semantic Retrieval
188K
215
58/299
Math Find
Math computation tasks within long sequence arrays
Global Information
120K
172
100/240
Hundreds-shot in-context learning
Global Information
22K
975
54/270
Sum
Summarize doe given mulliple does input
Global Information
104K
1.170
79/350
Mix. Sum+NIAH
Multi-tasking of Eu.Sum and Needle in A Haystack
Mulditasking
10SK
3,441
70/560
Mix. RepoQA+KV
Multi-tasking of RepoQA and KV retrieval
Multi-tasking
68K
5,318
88/704
Total
227K
1,684
931/44853
Additionally, by shifting the target string's position, we assess how well models utilize their full context window (Kamradt, 2023).
(i) Retrieve. Given a large JSON object with numerous key-value pairs, models must accurately retrieve the value for a specified key (Liu et al., 2024d). The random KVs pose challenges for long-context LLMs, as the input is often incompressible. demanding striet O(n) space for storage. This task is particularly useful for evaluating memory fuzziness in long-context methods. Each session retrieves five KV pairs, with target KVs evenly distributed across the input.
(ii) Retrieve. Prefix-PPffiif Given large list of variable-length strings, models must retrieve string matching specified prefix and suffix. This task is particularly challenging, requiring complex functions akin to prefix tree, with computational cost of OEΣ w:;",, where ut; is the length of the i-th string' The presence of distractors sharing only the prefix or suffix prevents models from relying on simple lookups or induction heads (Olsson et al., 2022) for effective retrieval.
(iii) Retrieve. Multiflop: First introduced in RULER (Hsieh et al., 2024), this task evaluates LLMs' multi-hop tracing capabilities within long input prompts. Models must track and recall key informa- tion changes, making it ideal for testing long-context methods in KV cache reuse. Five multi-hop variable assignment chains are embedded across the context, and each test turn requires retrieving the exact multi-hop chain, i.c., all variables assigned to a specific value.
Semantic Retrieval Beyond string retrieval, many real-world long-context applications require semantic understanding, such retrieving a function from textual descriptions or answering questions from long documents. These tasks are essential in SCBench, as lossy long-context methods often struggle to abstract or comprehend information in multi-request scenarios.
(i) Code. RepoQA: This task requires the model to retrieve a specific function (including its name, input parameters, and full implementation) from long source code chunk based on precise natural language description. Unlike the original RepoQA benchmark (Liu et al., 2024c), our inputs extend 64K tokens, with target functions evenly distributed by position within the codebase. The function descriptions were generated using GPT-4 based on the functions themselves. We also expanded the range of repositories and programming languages to include Python, C++, Java, PHP, Rust, Go, and TypeScript. Each test session involves GitHub repository, with the model required to retrieve one function per turn across turns.
(ūi) QA, Zh. QA, Fn.luutiChoice: These tasks are extended from InfiniteBench (Zhang et al., 2024a), which provides high-quality, human-annotated QA tests based on fictional novels to eliminate external knowledge influence. The tasks require models to locate and process information from lengthy inputs, performing reasoning through aggregation or filtering. The two primary question types are: 1) Aggregation, compiling scattered information across the input (e.g.. "How much money in total did A spend on food?"), 2) Filtering, identifying specific details from a larger set (e.g., "What color dress did A wear when A met B for the second time?" lime"") In SCBench, QA pairs sharing the same input context are combined to create shared-context test sessions.
Phttps://leetcode httppp///eetcccee..cm///oohh ems http://effiiiiiii
auffix xearrcch/

---

Published as a conference paper at ICLR 2025
Table 3: Task examples and configurations in SCBench. We use different colors highlight the questions, answers, and distractors in our examples.
Task
Source
Configuration
Example
RC
in Middle (Liu et al., 20244)
num kv pairs 2500
Input: (<key <valuc on daey #100> <valuc #100>)
len of key & value 36
Tum The value of the <key fls is? Answer 1: evvalue #l2...
metric Accuracy
Tum The value of the <key 420> is? Answer 2: <value #20b...
Tum value
ckey
Answer 3: ...lll K40l...
Retr:Prefix Prefix--Sfiiis
Ours
size of dict = 6000
Input: Dictionary = |ssstr H1>, <str 42>. <str #100=1
len of string [65, 123)
Tum Prefix: spx filp Suffix: esx tls. The word with both prefix
metric Accuracy
and suftix from the diet is? Answer: <30>>
Tum 2: Prefix: <px #2>; Suffis: <ax A2> Answer: <itr>
Rete RulcHHol
RULER
num chains 2
Input: VAR
12345 -....
54321 to..sc
(Hsiih et al., 2024)
mum hops 2
VAR X2 XI ..... VAR Y2 Y1 Choines
metric Accurncy
VAR X3 X2 VAR Y3 = Y2 Choissss
Tum Variables that assigned to 12345? Answer X2
Tum 2:
Variables that are assigned to 54321? Answer I: Y1 Y2 Y3
Code ReporQA
RepoQQA
fune deseription from GPT- metric= Pass11
Input: clune <func
<fune 100>
(Liu et al., 2024c)
Tum <deseription of func 1>, Answer I: <func Tum 2: sdescription of func 20> Answer 2: <func 20p
En.QA Zh.QA
InfiniteBench
ground truth human
Input: Read the book helow and unswer a question. <context>
(Zhing et al., 2024a)
metric Accumey
Tim <questions Be very comcise. Answer I: ...canss...
Tum squestion Be vary concise. Answer 2: .a.cans....
En.MultiChoice
InlinitBennch Zhang et al., 2024a)
ground truth from human metric Accuracy
Input: Read the book and answer the guestion. coontett>
Tum I: <question> <Option A,B.C.Ds, Answer I: canso......
Tum squestions <Option A,B.C.Do. Answer canso...
Math Find
Ours
len urray-300000 num, digitsm3
Inpui: <a large urray of numbers
Tur I: The max number in the array is? Answer I: amax number>... max number
metric Accuracy
Tum
array Answer 2: m.. number....
ICL. Shot
SantSto
num ctam_essspes = -150
Input: ICL Demo. + Demo. 2+.... + Demo. 1000
(Srivastava al., 2023)
Tasks date, salient, tracking?
Tum squestion>s Answer 1: .a..c.......
metric Accuracy
Tum 2: <question>> Answer 2: ...Kans....
FumSum
Ours
Crecatenated arXiv papers
Input: Doc + Doc 2 Doc 3 1 + Doc 10.
ground Iruth GPT-4 num document -- -8
Turn Please summarize Doc Answer
<summary
Iz...
Tur 2: Please summarize Doc 3. Answer 2: csummary of Doc io...
metric ROUGE
Tum 3:
Please summariz: Doc 5. Answer 2: <summary of Doc So...
Ours
num needle= =5
Input: Doe <<Passkys>> + Doc 2- ... + <Passkeys> + Doc 10.
sum document= = -8
Pleuse summarize Doc 1. Answer I: ...ummmary of Doc I>... Tum What the needle? Answer ..nneelles...
Tum 1:
ROUGE Ace
Mex.ReAoQA+ Ours
mum KV pains - 1000
Input: efune Is KV pains <func 2> t... KV pains cfunc 100p>
metric Pass 1 Ace
Tum edescription of fune 1>, Answer I: <fune 1>
Tum value of <key is? Answer 2: <valuu A1>..
Global Information Processing In addition to retrieval, some long-context tasks require leveraging and aggregating global context information, such as summarization, statistical tasks, and in-context learning (ICL) (Yu et al., 2020; Srivastava et al., 2023; Hao et al., 2022). Our benchmark includes three tasks to evaluate how well different long-contee context methods handle global information in multi-request.
(i) Many-shot ICI.: We use datasets from Big-Bench Hard (Srivastava et al., 2023) to evaluate many- shot ICL capabilities. This includes three sub-tasks: date understanding, salient error translation detection, and tracking seven shuffled objects. Many-shot ICL contexts are shared across turns within a test session, and all sub-tasks are presented as multiple-choice questions with four options.
(ūi) Math. Find: We extended the math find task from InfiniteBench (Zhang et al., 2024a), expanding it from finding only the maximum value to multiple statistical values. Given a large array, LLMs must find the minimum or median values, requiring effective comprehension of global context, comparisons, and statistical operations.
This task uses concatenated academic papers from arXiv as input, with document lengths ranging from 8K to 20K tokens. Ground truth summaries, averaging 654 tokens, were generated using GPT-4, which was prompted to produce concise one-sentence summaries for each document. The target documents for each turn are evenly distributed across the full context length.
(iii) En.Sum:
Multi-Tasking In real-world applications, LLMs often handle multiple tasks within single session using a shared input context. For example, users may request both summarization and content retrieval simultaneously. To reflect this, SCBench includes two multi-tasking tasks:
(i) NIAAH: This task combines document summarization with the Needle in a Haystack (Kamradt, 2023) task using a shared input prompt. A random "needle" is evenly in-
7

---

Published as a conference paper at ICLR 2025
serted into the input of the En.Sum task (concatenated academic papers). The model alternates between summarization and NIAH retrieval in each test session.
(ii) M This task combines the RepoQA task with KV retrieval using a shared input prompt. Multiple KV pairs are evenly inserted into the RepoQA input (a long chunk of source code), including 100 KV pairs with four target Kss and the rest as distractors. The model alternates between RepoQA and KV retrieval in each test session.
3.2 LONG-CONTEXT SHARED CONTEXT MODES DETAILS
In addition to the carefully designed long-context tasks, we include two shared context modes- turn and multi-request- better reflect real-world applications, as shown in Fig. 2b.
(i) Multi-turn Mode: A typical scenario in long-context applications includes long-context chat, multi-step reasoning (e.g.. Tree-of-Thoughh (Yao et al., 2024b)), and long-generation CoT. This mode is relevant for long-context methods with KV cache reuse, as focus shifts across turns can lead to information loss in the KV cache. Following Zheng al. (2023a); Wang al. (2024), we use ground-trune -ruth answers instead of model-generaee content as the context for follow-up turns.
(ii) Multi-request Mode: Context sharing spans sessions or users, such as collaborators on shared code repository. Models can encode shared context and reuse the KV cache across requests. Evaluat- ing long-context methods is crucial, as some depend on the query for sparse encoding/decoding. For instance, MInference and SnapKV use the input's final segment (typically the query) to estimate the sparse pattern, assessing their generalization without query access.
EXPERIMENTS & RESULTS
Models & Implementation Details We selected six open-source long-context LLMs: Llama- -BBB8//0B (Dubey et al., 2024), Qwen2.5-72BB/22B (Team, 2024), Llama-3-8B-262K (Gradient, 2024), and GLM-4-9B-IMM IM (GLM al., 2024), along with two gated linear models: Codestal Mamba 7B (team, 2024) and Jamba- 1. 1.5-Mini (Lieber et al., 2024). This selection covers Transformer, SSMs, and SSM-Attention Hybrid models, representing leading open-source long-context LLMs. For stability, all experiments used greedy decoding in BFloat16 on four NVIDIA A100 GPUs. We evaluated models via HuggingFace or vLLM with FlashAttention-2 (Dao, 2024) and leveraged MInference (Jiang et al., 2024) to reduce GPU memory overhead. More details on these models and infrastructure are $D.1.
Long-Context Method Details We evaluated eight long-context solution categories on our bench- mark: Gated Linear RNNs (e.g., Codeestrll-Mmmaa), SSM-Attention hybrids (e.g., Jamba), sparse attention, KV cache dropping, prompt compression, KV cache quantization, retrieval, and loading, as detailed in Table I. All methods were tested on Transformer-based long-context LLMs, except cadestral-Mambr and Jamba. We also report KV cache size, pre-filling and decoding complexity, and whether efficient operations are applied (details in $2). More details are shown in $D.2.
Main Results Tables 10, and Fig. 9 present the performance of various long-context methods across tasks and shared context modes in different base LLMs. Key findings include: 1) In retrieval tasks, most methods, except MInference, perform poorly, especially in exact retrieval tasks like string matching. 2) Sparse attention outperforms sparse decoding as request rounds increase, with A-shape showing the greatest improvement. Tri-shape, which adds dense bottom query tokens to A-shape, enhances first-round performance but has little effect on later rounds. It generalizes well across tasks, ranking second only to MInference. Our analysis indicates that Tri-shape improves first-turn instruction-followe following. boosting overall performance, while A-shape disrupts instruction information, causing random outputs (Table 17). 3) KV cache compression methods generally underperform in shared scenarios, offering only minor benefits in the first round. 4) Prompt compression improves global information tasks like many-shot ICL but significantly degrades garieval-relater performance. 5) Att-etttnnoon hybrids perform well in single-turn interactions but suffer accuracy drops in multi-ture turn tasks, especially in RepoQA and Math. Gated Linear RNN models struggle in shared context modes.
8

---

Published as a conference paper at ICLR 2025
Table 4: Average performance of various long-context methods across different LLMs in two shared context modes SCBench. For additional results base models such Llama-3.1-70f Qwen2.5- 32B, and Llama-3-8P B-2262,, see Table 10 in $E. Here, denotes the target compression rate.
Multi-turn Mode
Multi-request Mode
Methods
ReirnString
Retr.Semantic
Global
Multi task
AVG.
Retr-Siring Reir-Semantic Global Mulli-task
AVG.
HaMA-331-68
48.7
29.5
39.2
A-shape
1/32
14,0
28.9
31.7
33.7
27.1
3.2
33.2
46.3
27.8
27.6
Tri-shape
1/32
18.1
31.5
33.5
37.9
30.3
7.8
25.7
45.6
24.6
25.9
Minference
1/32
39.1
38.5
34.4
57.8
42.5
28.9
35.6
50.1
30.9
36.4
LLMLingua-2
1/3
5.7
25.3
32.3
49.6
28.2
3.9
24.4
41.2
22.8
23.1
LL
1/32
0.4
13.5
33.6
14.3
15.5
0.0
11.3
30.3
16.4
14.5
SnapKV
1/32
1844
37.9
21.1
20.9
0.3
14.2
35.7
10.7
15.2
PyramidKV
37.1
21.0
0.3
15.1
34.7
11.0
15.3
KIVI
1/8
12.0
34.3
31.0
50.7
32,0
7.6
30.1
33.1
28.4
24.8
CacheBlend
Quest
1/32
8.2
27.3
33.0
20.1
22.1
6.7
25.6
31.8
14.2
19.6
RetrivvalAAttetionn
GLM-4-98-IM
44.8
A-shape
1/32
27.2
31.2
30.7
58.5
36.9
20.2
24.1
40.5
42.6
31.8
Triishappe
Minierence
1/32
38.2
37.2
31,8
70.8
44.5
34.1
29.0
43.4
48.3
38.7
14.8
38.5
LLreamingLLMM
1/32
0.7
14.6
28.1
12.7
14,0
0.2
10.2
32.7
17.1
15,1
V
1/32
18.1
18.7
33.1
26.0
0.9
10.1
37.5
24.0
18.1
PymmiidV
1/32
18.1
23.9
30.8
34,9
26.9
0.6
4.9
39.8
21.1
16.6
1/8
26.5
33.7
23,8
51.2
33.8
2.5
20.3
39.5
43.5
26.5
Quest
1/32
20.1
30.3
25.8
36.4
28.2
0.0
15.6
33.9
15.2
16.2
Owen2.5-728
51.5
44.4
38.9
77.0
$2.9
31.1
46.8
$3.0
$2.4
45.8
A-shape
36.7
15.2
35.5
47.7
43.1
35,4
Tri-shape
1/32
25.7
36.6
37.7
63.8
40.9
18.6
38.3
48.5
44.9
37.6
Minference
1/32
45.6
43.5
38.4
72.8
50.1
28.6
44.7
52.2
$2.0
44.4
L.MI Ligunn22
1/3
4.2
28.7
46.2
26.6
2.7
31.2
49.0
25.8
27.2
StreamingLLM
1/32
0.7
16.2
40.2
18.7
18.9
0.0
4.2
4,4
0.0
2.2
KV
26.5
PyramidKV
1/32
4,9
23.3
42.0
34.3
26.1
0.0
17.5
44.5
11.2
18.3
1/8
12.9
3.0
46.6
35.8
Quest
1/32
5.2
28.3
41.9
31.5
26.7
1.2
24.8
42.1
15.2
20.8
Jomba-1.5. M.5.Mini
67.4
28.6
37.5
47.5
32.8
21.7
61.8
5.6
38.9
48.0
Mamba- Codeba-Codestal
3.9
25.8
54.8
1.0
wtth wwwttutoooo
"ol22"
Itm
PMB
0.6
Itn
UFaMo
0.2
T5
1K
2K
5<
(a) Critical KVs Vary Across Queries
(b) Attention Map of Retr.KV Across Turns
Figure 6: Attention visualization Retr.KV for the shared context across multiple turns.
ANALYSIS
Sub-O(n) Memory is Almost Infeasible in Multi Decoding. Fig. 6b visualizes the attention map for the Retr.KV task across significantly between queries. This explains why O(k) KV cache compression methods perform well in single-query tests but fail in follow-ups. However, the SSM-attention hybrid model Jamba shows promise in reducing memory costs by using SSM layers while maintaining O(n) memory in select attention layers for future lookups (Waleffe et al., 2024). Another promising approach is CPU-GPU collaboration, where full (n)) memory is stored in CPU RAM, dynamically loading relevant K to the GPU for sub-O(n) decoding (Liu et al., 2024b). Haris Onak (2025) theoretically proved that linear memory is required to preserve attention representational capacity, which aligns with it.
While important KV's remain stable within a turn, they vary
The Sparsity in Encoding and Decoding. We examined how sub-O(n) sparse decoding struggles maintain accuracy across multiple requests in shared context scenarios. Interestingly, sparse

---

Published as a conference paper at ICLR 2025
methods perform well in encoding if decoding remains dense. As shown in Fig. 3a, with dense decoding (O(n) memory), Tri-Shape and A-Shape achieve strong multi-request performance. While this phenomenon has been noted in single-turn tests (Sun et al., 2024c; Jiang et al., 2024), we are the first to demonstrate its potential in shared context settings. Conversely, extending sparse patterns to decoding severely degrades performance (e.g., StreamingI.LMM) Even with dense encoding, sparse decoding methods, particularly KV cache compression, perform poorly in shared context scenarios. This may stem from redundancy in encoding outputs, whereas decoding plays a crucial role in generation (Deng et al., 2024). Sparse encoding can still capture key information due to redundant input prompts, but sparse decoding weakens per-layer connectivity, limiting focus on critical tokens. Since sparse decoding relies on proxy tokens for global access. it constrains the formation of complex attention functions (Yun et al., 2020). We highlight the need for more advanced sparse patterns in sparse attention. Dynamic sparse attention can improve connectivity and accelerate information propagation (Jiang et al., 2024), better approximating full attention performance compared to static sparse patterns (Fig. 9).
Compressible and Incompressible Tasks. While (()) memory is crucial for multi-request sce- narios with shared context, it can be relaxed for highly compressible inputs in simpler tasks. For example, the "needle") within repetitive noise (the "haystack"), allowing sub-O(n) methods to achieve reasonable accuracy due to the high compressibility of noise. Similarly, summarization tasks involve com- pressible contexts, enabling sub-O(n) methods to balance efficiency and performance. However, for dynamic and complex inputs, sub-O(n) methods often fail to retain all necessary information, leading to poor performance in challenging retrieval tasks. Tasks like Retr.KV and Retr.Prefix Prefix-Suffix, which involve random and incompressible key-value pairs and strings, require models to fully utilize their context window. In summary, while compressible tasks may overestimate a model's capabilities, sub-O(n) methods remain viable for simpler tasks due to their efficiency.
sealeeii-tte-Hyssteee
benchmark (Kamradt, 2023) embeds key information (the
Sparse Methods without Query Aware- ness. A key concern with long-context methods in KV cache reuse scenarios is their reliance on the query for compression to enable efficient encoding or decoding. However, in real-world applications, a sin- gle context often shared across multiple
Table 5: Results of query-awarenes long-context meth- ods. w/ (first) and w/o (later) query.
1J-MA-.1-88
Retr.String
Retr Semantic
Global
Multi-rask
SnapKV
0.0
19.0 9.7
17.9/ 14.6
5.1/0.0
Tri-shape
12.1/78
314/ 25.7
/
28.0/246
MInference
28.1 28.9
40.4/ 35.6
50.1550.1
28.3 /3.99
queries, requiring these methods to operate without query access. This raises the question: Can query- dependent non-context methods generalize effectively without it? Table 5 compares the performance of three query-aware long context methods with and without the query, highlighting performance degradation in its absence (underlined). We found that both the KV cache compression method SnapKV V and the static sparse attention method Tri-Shape struggled to maintain accuracy without the query. contrast, the dynamic sparse attention method MInference exhibited stronger generalization, likely due to its adaptive sparse patterns, particularly its diagonal connections in the attention map.
6 CONCLUSION
This paper addresses key gap in evaluating long-context methods, which have traditionally focused on single-turn interactions while overlooking shared long-context scenarios-common in real-world LLM applications. To bridge this, we introduce SCBench, comprehensive benchmark assessing long-context methods with KV cache reuse across 12 tasks, covering string retrieval, semantic retrieval, global information processing, and multi-tasking, evaluated in two shared-context modes. Using our benchmark, we categorize long- congcconext methods into four KV cache-centric stages: generation, compression, retrieval, and loading. We evaluate eight method categories (e.g., gated linear RNNs, hybrid models, sparse attention, KV cache dropping, quantization, retrieval, loading, and prompt compression) on eight state-of-the-art LLMs, including 1...a----18//700. Qwen2.5- 72B/32B, (aaama---BB-222-- GLM-4-9B, Codestal Mamba, and Jamba- Our results reveal a clear KV cache management trade-off: (n)) methods excel in multi-request scenarios, while sub-O O (n) methods perform well in single-turm but struggle with complex interactions. These findings highlight the need for evaluating long-con long-context methods in shared-context multi-ture -urn scenarios, offering a more realistic benchmark and valuable insights for improving future long-context models and architectures.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shantanu Acharya, Fei Jia, and Boris Ginsburg. Star attention: Efficient Ilm inference over long sequences. ArXiv preprint, abs/2411.17116, 2024. URL 17116.
/2411
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanic C.Y. Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024, URL https: //openreview. net www.eewwwweeeotmmmieieiee wtw.:ummmddneeeeeee
Joshua Ainslie, James Lee- Lhorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4895-490) Singapore, December 2023. Association for Computational Linguisties. doi: https: hatpass///aalaaholoog/.. org/ 2023. emnlp ma i nn. 298.
1tt5333/1/223.mm///p/ 298.. URL
Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher Ré. Just read twice: closing the recall gap for recurrent language models. ArXiv preprint, 0oos/2407.0548., 2024. URL https: /tttpps://rxii.../ org abs/ 2407.05 05483.
Microsoft Azure. Prompt caching, https:
2024.
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. MT-bench-101: fine- nne-grained benchmark for evaluating large language models in multi-turn dialogues. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7421-7454, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10. htt/://111224..cl-1olo https: / ac anlahhathooyyy org/ 2024 acl acl-lcnlann 401.
URL
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3119-3137, Bangkok, Thailand, 2024b. Association for Computational Linguistics. doi: 10. 1 18653/11/20244 a.l-022...lloooogg. 172. URL https: / aacllll antho logy org/ 2024 acl- 172.
Ramakrishna Bairi, Athary Sonwane, Aditya Kanade, Arun Iyer. Suresh Parthasarathy, Sriram Rajamani, B Ashok, and Shashank Shet. Codeplan: Repository-Ievel coding using 1lms and planning. Proceedings of the ACM on Software Engineering, 1(FSE):675-69) 2024.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. ArXiv preprint, abs/2004.05) 150500, 2020. URL https: www.xiii.oorgoo abs /2004. 05150.
Avi Caciularu, Matthew Peters, Jacob Goldberger, Ido Dagan, and Arman Cohan. Peek across: Improving multi-document modeling via cross-document duestion-answente In Anna Rogers, Jordan Boyd-Grabed and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1970-1989, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1e.1186//1/2223... acl.oo.......ogg.......m 110. URL https: acl anthoooohhyoyyy org/ / 22022222222222 ..l.accl-loo 110.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, al. Pyramidkv: Dynamic ky cache compression based on pyramidal information funneling. ArXiv preprint, abs/2406.02069, 2024. URL https: //arxivvorr//ae /206. 02069.
11

---

Published as a conference paper at ICLR 2025
Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, and Beidi Chen. MagicPIG: LSH sampling for efficient LLM generation. In The Thirteenth International Conference Learning Representations. 2025. URL htwps::////tt. oww.oopeevvvooooo ew net/ forum? d= TQUgAaa....
Rewon Child, Scott Gray, Alec Radford. and lly Sutskever. Generating long sequences with sparse transformers. ArXiv preprint, abs/ 1994.994.110 10509, 2019. URL https://arxiv... wwwpariwii..iraaaaaaaaaaa 1904. 10509.
Claude. Prompt caching with claude. 2024.
Jincheng Dai, Zhuowei Huang, Haiyun Jiang, Chen Chen, Deng Cai, Wei Bi, and Shuming Shi. Sequence can secretly tell you what to discard. ArXiv preprint, abs/2404.15999, 2024. URL https://arxix: org/ abs/2404 15949.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In
The Twelfth International Conference on Learning Reprexentations, 2024. URL https:
// //openreview.o net //
Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference on Machine Learning, 2024.
URL https::pp/.............tt //ppnreeeeeeeeeeeeeeeeeeeeeeeeee eww.. net forum? wwt.idddttttntnt.i
Yichuan Deng, Zhao Song, and Chiwun Yang. Attention is naturally sparse with gaussian dis- tributed input. ArXiv preprint, abs/2404.02690, 2024. URL https: //tp:i//axivvoia org/ 2404. 02690.
Aditya Desai, Shuo Yang, Alejandro Cuadron, Ana Klimovic, Matei Zaharia, Joseph E Gonzalez, and Ion Stoica. Hashattention: Semantic sparsity for faster inference. arXiv preprint arXiv:2412.14468, 2024.
Domeccleston. Domeccleston//harcgpt:: Easily share permanent links to chatgpt conversations with your friends. https://githtnn wwt. http://ccttcco.o//cooooooo sharregt...app...... 2023. Accessed: 2024-09-15.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. ArXiv preprint, abs/2407.21783. 2024. URL https: arxivv org/ abs / / 2407 1477..187........ 21783.
Sarah E. Finch, James D. Finch, and Jinho D. Choi. Don't forget your ABC's: Evaluating the state-of-the-art in chat-oriented dialogue systems. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15044-15071, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v 1/2021.22aaaa aocoo/ 8399 URL https: / /alaaclanthhtt aclanthhooogoyyyoooo org/ 202...2.1c22.....
839.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang. Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 14909, 2024.
Yao Fu. Challenges in deploying long-context transformers: theoretical peak performance analysis. ArXiv preprint, abs/2405, .00944, 2024. URL https: /ttp//axivvvv////////////////.av org/abs 2405. 08 9.44
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive ky cache compression for 1lms. In The Twelfth International Conference on Learning Representations, 2024. URL https:t/////t httttp:://opeeevieee...o nww.:/rrttff? forum?
id=uNrFpDPMyo.
Gemini. Context httppp//lal.ooog...eee///eaaaaap 2024.
12

---

Published as a conference paper at ICLR 2025
In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. Prompt cache: Modular attention reuse for low-latency inference. In P. Gibbons, G. Pekhi- menko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6, pp. 325- 338, 2024. URL https: http://ooocooddoooooooooooooo..o ngs, ml sys. wtwppppapr_aaaeee fi wel.ffillsppaee http:/ppperrr/22/2 aate//666aaali wttee//aecaaaaaffff455333333 701777733333443 3 Paaeaaaeeaeeeeeaaaaeaaoaaaeeeeee
Team GLM, Aohan Zeng. Bin Xu, Bowen Wang, Chenhui Zhang. Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, al. Chatglm: family of large language models from glm- to glm-4 all tools. ArXiv preprint, abs/2406. 12793, 2024. URL https: http///xvvvv./////////// org.
2406. 12793.
Gradient, Llama-3 8b instruct gradient 4194k (v0.1), 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference Language Modeling, 2024. URL https: //ttteee///peeevieew net forum?id= tEYskw1VY2.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: / openreview. n forum? widiiid=FFFFF. MalFoozvlll vC..
Chi Han, Qifan Wang, Hao Peng. Wenhan Xiong, Chen, Heng Ji, and Sinong Wang. LM- infinite: Zero-shot extreme length generalization for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers). pp. 3991-4008, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.1866.. 118835/v11200/ 1 222. URL https: / /aw.aaaaaanooooooo.aooooooooooaa org/ 2024. nwacc waaaacl-lone long. 222.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in-context learning to 1,000 examples. ArXiv preprint, abs/2212.06713. 2022. URL https:///axxiiiiii /ttp//axivv///////////eee/e/.ai. wwg/oogg//aoo.. 22121167.....
Themistoklis Haris and Krzysztof Onak. Compression barriers for autoregressive transformers. arXiv preprint arXiv:2502.15555 2025.
Ramin Hasani, Mathias Lechner, Tsun-Hsual Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///ppeeeeeeeeeeeeeeee n forum wiw.mmmmrdddd4-0-555
Namgyu Ho, Sangmin Bae, Tachyeon Kim. hyunjik.jo, Yireun Kim, Tal Schuster, Adam Fisch, James Thorne, and Se- Young Yun. Block transformer: Global-to-local language modeling for fast inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL htw.t: http///pperrvieeeeee
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Monishwaran Maheswaran, June Paik, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Squeezed attention: Accelerating long context length IIm inference. arXiv preprint arXiv:2411.09688 2024.
Cheng-Ping Hsich, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's s the real context size of your long-context language models? In First Conference on Language Modeling. 2024. URL https: http::::///pperevviiiee net/forum? id=kIo3bc76Sw.
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and eran malach. Repeat after me: Trans- formers are better than state space models at copying. Forty-firsl International Conference on Machine Learning, 2024. URL https: /oppnreviewwwe.ceooooooee htte//ooommmmddddRGoooooo
Huiqiang Jiang, Qianhui Wu, Chin- Yew Lin, Yuqing Yang. and Lili Qiu. LLMLingua: Com- pressing prompts for accelerated inference of large language models. In Houda Bouamor,
13

---

Published as a conference paper at ICLR 2025
Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 13358-13376, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v1l 1/20e3.2222eaa www.ee-mainn...... URL https:
//aclanthol ogy. org/2023. emnlp-ma 825..
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin- Lin, Yuqing Yang. and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https: / //openreveww.. www.eeewwnn//oummiieeeeeeee (eimmidd=PPAAAbaSS
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL /ttpp::/openreteeeee
net/ Eorum?id-VIE 8yNQM66.
Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. Ragcache: Efficient knowledge caching for retrieval-auggmented generation. ArXiv preprint, abs/2404.122577, 2024. URL https: httpaaxxiv.... /taer///////// /2w0./////22.... 12457.
Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughi Ilm inference with shared prefixes. ArXiv preprint, abs/2402.05 2024. URL https: / arxii. org/ /abs/a////// 2200242220220 05099.
Greg Kamradt. Needle in a haystack pressure testing 1lms, 2023.
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT-eval: multi-turn capabilities evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 20153-20177, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653/v emnlp-main. 1124.
http5////1/222444 1/2024 wa0.:244.mllp-mmaoee main. 1124. URL https: httpp//s///aaaaaaaaaaaa aaw.aaaaaaanooooooooooooooooooaa org/
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23. pp. 61 611-226, New York. NY, USA, 2023. Association for Computing Machinery, ISBN 9798400702297. doi: 10.1145/3600000636611666 URL https:: /ooiiii.iio org/ 10. 00114/3360000
3613165.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //tp////pp///eee.//////e/ net forum?
id=0Ej[IbelrT.
Jaqq Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE: Can long-contex. language models understand long contexts? In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 16304-16333, Bangkok, Thailand, August URL
2024a. Association for Computational Linguistics. doi: 10. https: httpss///allaathhooogoo
202//2222.222m aabo24acclloong 859.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning, ArXiv preprint, abs/2404.02060, 2024b. URL / abs/2404, 02060.
Xuechen Li, Tianyi Zhang, Yann Dubois. Rohan Taori, CG Ishaan Gulrajani, P Liang, and TB Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloving models. 2023. URL http::/github. com/hatsu-lablalpacca eval, 2023a.
14

---

Published as a conference paper at ICLR 2025
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Pro- ceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 6342- 6353, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp-main.99 URL https:ttpp:..mmm a/cl///clll/a anthology wttg//222 emn 1- i 391.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c. URL http:: opwpreveeeeewweeeee net wttp::/wwww.eeettt.....eeeeeee
4G0q21.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer- mamba language model. ArXiv preprint, abs/2403. 198877, 2024. URL htttps://aaxii...t// abs/2403. 112440...8888 19887.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of- experts language model. ArXiv preprint, /0o//2405.04444, 2024a. URL https: http///aavvvv..////e/eee/ea.a. org/ abs/2405 04434.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattention: Accelerating long- context Ilm inference via vector retrieval. ArXiv preprint, abs/2409.10516, 2024b. URL pittps:/aariie http://xxxxv...../aa//.
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. Repoqa: Evaluating long context code understanding. ArXiv preprint, abs/2406.06025, 2024c. https://:/ http://axxvvv.ieeeee oww. abs / 220/ 06025.
abs
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024d. doi: 10. 10.1162/tacl a 00638. URL
wrg. 2otg//024.tacl11...
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL https:/ /ttps/////eeeeeew..// net / forum idd= J2fg6wGi6g.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Forty-first International Conference on Machine Learning, 2024e. URL peattp:://pppnreeee
net/
Cerum?id=LOn 2= 22q80.
Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, al. Moba: Mixture of block attention for long-context 1lms. arXiv preprint arXiv:2502.13188 2025.
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. ArXiv preprini, abs/2404.0714 007143, 2024. URL htttss://arxt wwtgg/////////ooo .4114411444.......
Piotr Nawrot, Adrian Lańcucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. Dynamic memory compression: Retrofitting LLMs for accelerated inference. In Forty-firl International Conference on Machine Learning, 2024. URL https: /ttps:///pppeaee opwnrevieeeeeeeeeeoe net forum?id= EDRYrAKOB7.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. ArXiv preprint, abs/2209.1 1895, 2022. URL https: //ax/vivvvv//////////a/////////a org/ abs /2009. 1 11895.
15

---

Published as a conference paper at ICLR 2025
OpenAL. Learning reason with llms, 2024a. URL https:://ppeaai.coom//meee
OpenAl. Prompt caching in the api.
https://openai.com//
api peal-poopt-cachiee
2024b.
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang. Chin- Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. L.MLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics ACL 2024, pp. 963-981, 1, Bangkok, Thailand and virtual meeting, 2024. Association for Computational Linguistics. doi: (0.18653/v112022 fei.i...iiiing-acao acI.57. URL https: www.annnlnnooyyyyeoooooooooollll org/ 2024. findings-anll.....
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra human behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, URL https:/ /at::://pp...
www.:orgg/oooooooooo : 258040990.
Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung. Leon Derczynski, Xingjann Du Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Ji Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:/ / wttp//peeeewweeeee/e/ee net forum? inmum?idis
Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation - a KVCache- centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), pp. 155-170, Santa Clara, CA, February 2025. USENIX Association. ISBN 778-1-939133-45-88 URL https::///// /wwww/www... usw.ix..n............. org wttc///eecceeaeeeeeeeeeeeeeeccee fast22/ presentation/ qin.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, Abs/2403.055304 2024. URL https: //axxii org/ http://.rrrgaba///////////// 0553C.
Liliang Ren, Yang Liu, Yadong Lu, yelong shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https: pestts::/opeee /pppnreeee.e.eeeeeee net peet/fose forum? wddbbInnnbbbbbbbbb....
Luka Ribar, Ivan Chelombiey, Luke Hudlass-Galley Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidttheeffcient LLM inference. In Forty-first International Conference Machine Learning, 2024. URL https: / opwpeeveew.eeeeee////pe net forum? fimimidd-ss. 055-55ddmmmtt1...
Noam Shazeer. Fast transformer decoding: One write-head is all you need. ArXiv preprint, abs/1911.00 1,02150, 2019. URL htt.:h:sssp/....oootottt
/191 02150.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with single GPU. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094-31116. PMLR, 23-29 Jul 2023. URL https: httts:://prceeeeeeeee proceeoceeeeeo ngs. mlr. press/v200 sheng23a.....mm html.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. ArXiv preprint, abs/2408.03314, 2024. URL https: //wxx/v////v/////////ev///eeaa/a vv. org/abs 2408 03314.
16

---

Published as a conference paper at ICLR 2025
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garrga-Alona et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long contecontext Ilm inference. ArXiv preprint, abs/2410.21465, 2024a. URL https: /tttppss//aaiii...// 21465.
/arxivixvv//////////////////aav/ org...oo........oooo...ooo...o.. /2410.
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Con- ference on Language Modeling, 2024b. URL https: http////ppe/eeiieeeeee/eeeeeeeee net forum?id HVK6n13197.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor transformer for large language models. ArXiv preprint, abs/2307.08621, 2023, URL https: arxiv. org/ abs 2307. 08621.
Yutao Sun, Dong, Yi Zhu, Shaohan Huang. Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. In The eighth Annual Conference on Neural Information Processing Systems, 2024c. URL https: wpw.review.eeeeeeeeeeeeee net/ forum?idie 1dd.mmioo22225566
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query- aware sparsity for efficient long-context LLM inference. In Forty-first International Conference Machine Learning, 2024. URL https: / /openrevieww net. forum? d=KzACYOOTVV
Mistral AI team.
Codestral mamba, 2024.
URL https://mistrallaa/nee
codestral -aaaba/.
Qwen Team. Qwen2.5: A party of foundation models, 2024. URL WtLLtttss////ntt github. io/blog/qwen2.00
Philippe Tillet, H. T. Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10-19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196 doi: 10.1145/3315508.332222
URL httppss//dooo.rrr///1111////1513 3329973..
vLLM. Automatic prefix caching. https: /wwc/oocsooooooooooooo wtt.://wmmmmmm/a/aaaaaaa/aaaaaaa /ttp://www....aaa/aaaatt///tttta
htm.ppp....mmmmmmmmmmmmmmm..mmmm htm1, 2024.
Kiran Vodrahalli, Santiago Ontanon, Nilesh Tripuraneni, Kelvin Xu, Sanil Jain, Rakesh Shivanna, Jeffrey Hui, Nishanth Dikkala, Mehran Kazemi, Bahare Fatemi, et al. Michelangelo: Long context evaluations beyond haystacks via latent structure queries. ArXiv preprint, abs/2409.12640,, 2024.
URL https:ttp//// /ttp///axvvvv///a///a//eee//aw./ org/ abs 220.//499..00 12640.
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Alhert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba- based language models. ArXiv preprint, abs/2406.07887, 2024. URL https: http:xxxxxv.o///////.../// abs/2406. 07887.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024. URL hwtph::t///oto http::::///peeevieeee net/ fott/formm?ie id 3o= jp3gWrMuIZ.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL
net
17

---

Published as a conference paper at ICLR 2025
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-contex. LLM inference with retrieval and streaming heads. In The Thirteenth International Conference Learning Representations, 2025. URL httpss: / /ttp:///ppemnn www.eeeeeweeeeeeeeeeeeeeeeeewwee net forum? d= xUU.
Shang Yang, Junxian Guo, Haotian Tang, Qinghao Hu, Guangxuan Xiao, Jiaming Tang, Yujun Lin, Zhijian Liu, Yao Lu, and Song Han. Lserve: Efficient long-sequence Ilm serving with unified sparse attention. arXiv preprint orXiv22502114800 2025.
Jiayi Yao, Hanchen Li. Yuhan Liu, Siddhant Ray. Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. Cacheblend: Fast large language model serving with cached knowledge fusion. org/ abs / 2405 1444.
ArXiv preprint, abs/2405. 164444 2024a. URL
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024b.
Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien- Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, and Luis Ceze. Cascade inference: Memory bandwidth efficient shared prefix batch decoding, 2024. URL https: / //fff//asnnfff...eeeeeeee a 222224 02/ 02/ cascade inference html.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context contet language models effectively and thoroughly. ArXiv preprint, abs/2410.02694, 2024. URL hwtph:://///occ /ttttss//arxivy org/ abs 2410. 02694.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. Dialogue- based relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4927-4940, Online, 2020. Association for Computational Linguistics. doi: 10.
URL
https: / acl antlaanthhhhogoo org/ 2020 acl acl
444.
Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 4623 0623-4648, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18651/663/111 1/2024. wwe.2224.ininn--emmaee findings-emnlp.266 URL org/ 2024. findings emn 1p. 266
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint axXiv:2502.1088. 2025.
Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli,, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. O(n) connections are expressive enough: Universal approximability of sparse transformers. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpps::/poooeeeingggggoo nwwrips..p...m wt/pc///pppp ptcppaper/200/ 2020/hashh 8 www.aeed0a2a223335ee3e33 -aa.dd55-ee42233333a---aaaaaaa .tml.
Bed27554c$93b5baaae
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, and Maosong Sun. Infinitebenc:: Extending long context evaluation beyond 100K tokens. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, Bangkok, Thailand, 2024a, Association for Computational Linguistics. URL https:tpp://mm. pttttts://acaaattiit haa/aaaaaannntooeo .rg/ worgg/22e2000 2024 acl - long. 814.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024b.
18

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a -a--a---jjudge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4666 2023a.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024. URL https: /opnneeeeeeeeeeeeee www.eiwweeee.. net/ forum? i d=Vqkkxqi .pq.
Ningxin Zheng, Huiqiang Jiang, Quanlu Zhang, Zhenhua Han, Lingxiao Ma, Yuqing Yang. Fan Yang, Chengruidong Zhang, Lili Qiu, Mao Yang, and Lidong Zhou. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, pp. 331-347, New York, NY. USA, 2023b. Association for Computing Machinery, ISBN 9798400702297. doi: 10.1145/3600006.661333 URL https:://///..ccc.pm / /tw.i/o...o/// otg/10gg///o 10. 00444//3600000 3613139.
Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, et al. Sampleattention: Near-lossless acceleration of long context Ilm inference with adaptive structured sparse attention. arXiv preprint arXiv:2406. 15466, 2024.
A RELATED WORKS
Prefix Caching (also known as KV cache reuse) optimizes time-to-first-tokn in LLM inference frameworks, particularly for shared contexts like multi-turn conversations or chatbot sessions (Zheng et al., 2024; Kwon et al., 2023; Gim et al., 2024). This technique is widely adopted by LLM providers (Gemini, 2024; Claude, 2024; OpenAl, 2024b; Azure, 2024). Recent optimizations focus on enhancing KV cache efficiency. PagedAttention (Kwon et al., 2023) reduces memory costs by partitioning the KV cache into blocks with a lookup table. HydraGen (Juravsky et al., 2024) and Cascade Inference (Ye et al., 2024) decouple attention computation for shared prefixes and unique suffixes, supporting batched multi-query kernels. Radix Attention (Zheng et al., 2024) accelerates KV lookups using radix tree with (kk complexity and is integrated into the vL.LM framework (vLLM, 2024). RAGCache (Jin et al., 2024) caches KV tensors for retrieved documents in
generation, while CacheBlend (Yao et al., 2024a) improves cache utilization via partial recomputation. Despite these advancements, no existing long- ong-context benchmarks evaluate KV cache reuse scenarios.
Conversational and Multi-Turn Benchmarks While multi-turn benchmarks better reflect real- world applications, many evaluations still focus single-turn (Li et al., 2023a; Finch et al., 2023). Benchmarks like MT-Bench (Zheng et al., 2023a), ShareGPT (Domeccleston, 2023), MINT (Wang al., 2024), MT-Bench-100 101 (Bai et al., 2024a), and MT-Eval (Kwan al., 2024) assess conversational abilities, instruction-followw and complex task-solving across turns. However, they primarily focus on model consistency and information extraction rather than evaluating long-context inputs.
Long-Context Methods of LLMs Long-context inference faces two key bottlenecks: computa- tional cost during pre-filling and memory cost during decoding (Fu, 2024). Pre-filling optimizations include state space models (Gu& & Dao, 2024; Gu al., 2022), linear attention methods (Peng et al., 2023; Sun et al., 2023), memory-based approaches (Munkhdalai et al., 2024), sparse attention (Jiang al., 2024: Zhu et al.. 2024; Lai et al., 2025; Yuan et al., 2025; Lu al., 2025), hybrid techniques (Lieber et al., 2024; Ho et al., 2024; Ren et al., 2025; Fu et al., 2024; Xiao et al., 2025; Yang et al., 2025), and prompt compression (Li et al., 2023b; Jiang et al., 2023; Pan et al., 2024). Decoding optimizations focus on: 1) Attention KV reuse to reduce storage (Shazeer, 2019; Ainslie et al., 2023; Sun et al., 2024c: Liu et al., 2024a; Nawrot et al., 2024); 2) Static KV compression (Xiao et al., 2024; Han et al., 2024); 3) Dynamic KV compression, including cache discarding (Zhang et al., 2024b; Ge et al., 2024; Liu et al., 2023; Li et al., 2024c) and offloading (Ribar et al., 2024; Tang et al., 2024; Dai et al., 2024; Liu et al., 2024b; Chen al., 2025; Sun et al., 2024a; Hooper et al., 2024; Desai et al., 2024); 4) Hierarchical speculative decoding (Sun et al., 2024b). Most methods tested single-turn benchmarks and employ query- conditoond lossy techniques, which may
19

---

Published as a conference paper at ICLR 2025
degrade performance in multi-turn scenarios with prefix caching. This limitation motivates the design of SCBench, benchmark that evaluates long-context solutions in shared context settings.
B COMPARED TO PRIOR LONG-CONTEXT BENCHMARK
We have compared SCBench against existing long-context benchmarks across long-context capability assessed, request types considered, and implementation they adopted, as shown in Table 6.
Table 6: Comparison of Long- Context Benchmarks.
Long-Context Capability
Roquest Type
Implementation
Precise Semantic
Global
Multi-
Single
Multi- Multi-
KV Cache
Retrieval Retrieval Information
Tasking
Question
Turn
Roquest
Reuse
LongBench (Bai ct al., 2024b)
InfiniteBench (Zhang ul. 2024a)
RULER (Hsich
2024)
LongCTXBench (Yuan et al., 2024)
HELMET (Yen et al., 2024)
Michelangelo (Vodrahalli al., 2024)
SCBench
We also directly compare the testing results of long-context methods on prior benchmarks and SCBench show the unique insights our benchmark provides. We mainly compare com- mon long-context capability: summarization (as shown in Table 7), and retrieval (as shown in 8). The summarization sub-tasks we used En.Sum for InfiniteRench (Zhang et al., 2024a), and gov-report for LongBench (Bai et al., 2024b). The retrieval sub-tasks we used is Retr.KV for InfiniteBench (Zhang al., 2024a), and Passage-retrieval for LongBench (Bai et al., 2024b). addi- tion, LongCTXBench (Yuan et al., 2024) also analyzes the performance boundaries of long-context efficient methods from a EV-eache-centre perspective. However, it does not consider multi-request scenarios and only focuses the KV Cache Generation and Compression stages.
Table 7: Comparing the summarization capability of efficient long-context methods on prior bench- marks and our SCBench.
Prior Benchmarks
SCBench
Multi Request
Model
InfiniteRench
LongBench
Turn1
Turn-2
Turn-3
Turn-4
Turn-5
Alama-3.1-8P
28.5
36.6
38.3
44.2
42.1
35.8
37,6
42.3
A-Shape
24.5
33.5
28.8
26.1
30.8
33.8
40.8
40.4
Tri-Shape
33.9
32.1
30.0
34.0
41.0
40.3
Minference
28.9
33.9
40.6
36.1
39.7
43.5
43.7
StreamingLLM
27.3
32.0
30.2
29.4
26.1
27.7
27.3
26.9
SnapKV
28.3
33.2
29.9
36,2
29.4
28.6
28.1
31.0
LLMLingua
23.1
32.0
30.1
32.5
22.5
26.6
25.7
26.6
Table 8: Comparing the retrieval capability of efficient long-context methods on prior benchmarks and our SCBench.
Prior Benchmarks
SCBench
Multi Request
Model
InfiniteBench
LongBench
Turn-1
Turn-2
Turn-3
Turn-4
Turn-5
Llama-3.1 1.1ma-3.11-8 --ssStt
100
62
59
68
66
70
A-Shape
42
22
28
33
Tri-Shape
100
25
32
38
Minference
100
14
31
35
46
56
50
StreamingLLM
84
2
1
0
0
0
SnapKV
100
0
0
0
0
0
I. Lingua
90
0
2
0
0
We found SCBench can better identify the weakness of long-context methods under the KV cache reuse scenarios, such the general incapability of KV cache compression methods on multi-request
20

---

Published as a conference paper at ICLR 2025
Minference
A-shape
Budget
203 Perforrnncc
40-
Aug. Performa
32K
20
64K
10
Ium
lum
In-shhpe
StreamingLLM
Avg. Porfomance
Aug, Purformincc
40
20
Tur SnapkV
Tumn
LLMLingua-2
50
Agg Neferaree
Aug Nerformarca
30
Figure 7: Hyper-parameters analysis: averaged performance of efficient long-context methods with different computing budgets under the multi-turn mode of SCBench. The input length 128K, meaning that 4K, 8K, 16K, 32K, and 64K correspond to sparsity budgets of 1/32, 1/16, 1/8, 1/4, and 1/2, respectively.
SCBench: Multi-Request
32.5
Method
Hinforonce
Anshupe
AvR. Performanc
Iisahipe
Streamingt
Snapky
II Inguee?
175
15.0
4K
8K
16K Bucget
33K
G4K
Figure 8: different computing budgets under the multi-request mode of SCBench. The input length is 128K, meaning that 4K, 8K, 16K, 32K, and 64K correspond to sparsity budgets of 1/32, 1/16, 1/8, 1/4, and 1/2, respectively.
analysis: averaged performance efficient long
long-context context methods with
mode and follow-up queries in the multi-turn mode, as well as the increasing accuracy of sparse attention under multi-turn mode.
C HYPER-PARAMETERS OF EFFICIENT LONG-CONTERS METHODS
We conduct extensive experiments with various computing budgets for the efficient long-context methods covered. The results shown in Figure and Figure 8 for the multi-turn mode and multi-request mode respectively.
From the results, mance degradation at a 1/2 budget (c.g., A-shape and Tri-shape drop by 5-6 points, SnapKV drops by 11 points). However, sparsity increases, performance declines significantly. For example,
derive the following insights: 1) Most methods show minimal perfor-
21

---

Published as a conference paper at ICLR 2025
StreamingLLM LLM and SnapKV drop by 26 and 19 points, respectively, under 1/4 budget. 2) More accurate sparse methods can maintain performance even under higher spansity. For instance, MInfer- 1/4 budget. 3)
ence achieves performance at 1/32 budget comparable to A-shape and Tri-shape While some methods exhibit similar performance in single-turn scenarios, they diverge significantly in multi-turn and multi-request scenarios. For example, SnapKV outperforms StreamingLLMM in turn-l but performs significantly worse in turn-2. In some tasks, changing the budget has little impact on turn-l performance but substantially affects turn-2 and subsequent turns, such as Long Document QA tasks and summarization.
D EXPERIMENT DETAILS
D.1 LONG-CONTEXT METHODS DETAILS
This section will introduce the long-context methods (as shown in Table 1) that involved in our paper.
State Space Models (SSMs) are powerful models often used for modeling dynamic systems, particularly in time series analysis, control theory, and machine learning. As language are naturally time series data, recent advancements have integrated SSMs into language modeling architectures, showcasing their potential as alternatives to traditional models like RNNs and Transformers. Due to their linear complexity, they are especially suitable for long sequence tasks. For instance, models such as S4 (Hasani et al., 2023) and Mamba (Gu & Dao, 2024) have demonstrated superior efficiency handling sequential data with reduced computational complexity compared to their predecessors and comparable accuracy in tasks such as language modeling. However, SSMs were also criticized for their reduced memorization capability and their limited capability in copy- sopy--asting (Jelassi et al., 2024).
Mamba-Attention Hybrid Architecture interleaves blocks of Transformers and Mamba layers, aiming to obtain the benefits of both architecture, i.e., the expressive power of Transformer and the linear complexity of Mamba layers. Jamba (Lieber al., 2024) and Samba (Ren et al., 2025) are representative efforts on this direction. Waleffe et al. (2024) also highlights the potential of such hybrid architectures and found only a few number of attention layers can lead significant performance increase compared to pure SSMs models.
Sparse Attention is extensively studied for long sequence processing, including image synthesis and multi documents question answering. We test three sparse attention approach in our paper: A-shape, Tri-shape, and MInference. In A-shape attention, each token is only allowed in to attend to initial tokens and local tokens, resulting A-shape on its attention map (Xiao et al., 2024). Tri-shape attention is a variant of A-shape method, we introduced in our paper, where we add dense attention space at the bottom of the triangle A-shape attention matrix. This is based on the promising results of sparse encoding with dense decoding, where the dense space dense decoding idea. MInference (Jiang et al., 2024) is the state-of-the-an dynamic sparse attention approach where the exact sparse pattern are dynamically built on-the-fly better approximate full attention operation.
added is natural extrapolate of the
KV Cache Compression is a series of studies that attempt to solve the linearly growing memory (often referred as KV Cache) cost LLMs inference. For example, StreamingLLM (Xiao al.. 2024) use a constant size of KV Cache in their decoding steps, where only the state of initial and local tokens are preserved, and the rest part of KV Caches are evicted from the memory. SnapKV (Li et al., 2024c) introduces the concept of the observation window. It selects the top-K KVs that are extensively attended to in the observation window, and removes other KVs from the Cache, This method was reported to performance well in simple Neeld-in-A-Haysstee tasks and many other natural language tasks.
KV Cache Quantization aims to reduce the memory footprint of KV cache via quantization. KIVI (Liu et al., 2024e) employed per-channel quantization for the Key tensor, and por-token quantization for Value tensor. In our evaluation, we use a 2 bit algorithms with group size of 32 an residual length of 32.
22

---

Published as a conference paper at ICLR 2025
KV Cache Retrieval indicates the operation to retrieve pre-cached KV cache and reuse them for incoming requests. Most of the frameworks employ an exact match algorithm, i.c. only retrieve and reuse the KV cache is the the shared context match exactly. However, there also exists approach such as CacheBlend (Yao et al., 2024a) that retrieve KV cache once the input is similar enough semantically with one former request from the cache base
KV Cache Loading Due the huge memory result from the long-context KV cache, researchers have proposed novel approaches that make use of the extensive CPU RAM and load only partial of the KV cache GPU per-token for more efficient decoding. For example, Quest (Tang al., 2024) estimates the importance of Keys in a page granularity, and only the topk important Keys and corresponding Values are loaded CUDA HBM for the attention computation. Retrieval Attention (Liu et al., 2024b) constructs vector database on CPU RAM to find the topK critical Keys efficiently. In additional, it tailored a pipeline for decoding to hide the memory movement from CPUs to GPUs.
Prompt Compression aims to compress the prompt to obtain a more compact representation of the input before send the LLMs (Li et al., 2023b; Jiang al., 2023). LLMLingua-2 (Pan et al., 2024) is supervised model that assess the importance of individual token as a token classification task. It was shown in provide up to 20x compression on many tasks, with only minimal performance sacrifice.
D.2 ADDITIONAL IMPLEMENTATION DETAILS
Table 9: Configurations of long-context methods in SCBench.
Methods
Configurations
SSMs
Mamba-Codesttul-7BB..
chunk size: 256, conv kernel: 4, expand: 2, head dim: 64, hidden size: 4096, intermediate size: 8192, groups: 8, norm before gate: true, num heads: 128, num hidden layers: 64, state size: 128
Hybrid Models
A121-Jambu-l,5-Lummm L-5gee
num hidden layens: 72, hidden size: 8192, intermediate size: 24576,
num attention heads: 64, num key value heads: mamba d state: 16.
manba conv: 4, mamba expand: 2, mamba conv bias: true.
num experts: 16, num experts per lok: 2, attention:mamba 1:7,
number layers per block: 8
Tri-Shape
num local: 4096, num initial: 128, num dense rows: 128
Sparse Attention
A-Shape
num local: 4096, num initial: 128
Minference
Pattem search data: KV retrieval
a-shape: 1024/4096
vertical-slash: 30/2048, 100/1800, 500/1500, 3000/200
block-sparse: 100 blocks
KV Cache
StreamingLLM
num local: 4096, num initial: 128
Compression
PyramidKV
window size: 32, max capacity prompt: 4096, kemel size: 5, pooling: avgpool
SnapKV
window size: 32, max capacity prompt: 4096, kernel size: 5, pooling: avgpool
KV Cache Quantization
KIVI
bit size: group size: 32
residual length:
KV Cache
CacheBlend
recompute ratio: 15%,
Retrieval
chunk size: 512
KV Cache
Quest
chunk size: 16; token budget: 4096
Loading
RetrievalAttention
lopk: 2000; index type: index
Prompt Compression
I.I M.IMuaigga22
compression rate: 01,33
In Table 9, we report the configuration we used for the long-context methods we involved in our experiments. For the Mamba-Codestral Codestral- 7B-00. model and A121 1121-Jamba-l L.rge, we report the architecture details of other models. For SSMs models, the state size and number of layers are crucial properties, as all previous information are compressed and saved in this fixed size of states. Moreover, the number of groups and number heads are also important they implement channel mixing which
23

---

Published as a conference paper at ICLR 2025
Table 10: The average results of various long-context methods on Llama-3.1-70B, -ww.-22200...255-200055 Qwen2.5-32B, and Llama-3-8B-262K with shared context modes on SCBench.
Multi-turn Mode
Multi-request Mode
Methods
Retr.String Semantiiitic Global Multi-task AVG.
RetrnString
RetaSemmanti Global Multi-task AVG.
Llama-3-8B-262X
33.3
26.7
17.1
30,0
25.5
34.1
26.7
A-shape
27.2
25.6
55.6
29.6
7.8
27.3
22.0
35.2
23.1
Tri-shape
11.1
29.6
26.3
60.6
31.9
8.2
22.4
22.5
35.9
22.3
MInfereace
33,5
26.7
36.2
8.3
32.1
25.6
40.0
26.5
StraaminLLM
12.6
22.6
114
22.6
0.1
SnapKV
0.5
4.2
21.9
0.5
6.7
0.0
1.1
24.5
0.1
6,4
LLMI iingua-2
21.0
24.5
23.0
18.0
3.9
24.4
42.4
22.6
23.3
L1ama-3.1-708
20.9
45.4
45.7
70.3
45.6
3.1
47.9
48.1
47.8
36.7
A-shape
34.7
40.5
26,9
26.7
3.2
35.7
46.3
33.8
29.7
Tri-shupe
37.1
42.0
29.2
3.8
40.5
46.5
34.2
Minference
19.5
42.5
43.1
65.6
42.4
7.3
43.7
48.2
46.1
36.3
StreamingL.MM
0.2
6,4
22.8
3.7
8.3
0.0
10.9
31.2
0.0
10.5
SnapKV
3.7
14.0
LLMLingua-2
6.7
38.8
38.7
31.0
28.8
4.5
32.0
38.6
26.7
25.5
Qwen2.5-328
46.8
42.6
40.6
73.4
50.9
25.0
44.5
55.3
49,9
43.7
A-shape
15.0
33.8
38.7
59.5
36.7
9.6
34,1
53.7
38.6
34.0
Tri-shape
18.5
34.6
40.4
64.0
39.4
11.7
37.4
56.4
41.1
36.7
Minference
35.4
39,9
40.8
46.5
17.7
42.7
56.4
48.6
41.4
StreamingLLM
0.2
4.3
8.4
6.3
4.8
0.0
1.8
7.4
0.0
2.3
SnapKV
3.3
3.9
27.1
1.5
9.0
0.0
4,9
9.8
0.0
3.7
M.ngMang
28.2
shown to be critical for the expressive power. For Mamba-Attention hybrid architecture, the present the ratio of attention layers and mamba layers, As the Jamba model is also MoE, also represent the number of experts and the number of experts activated per token.
In Sparse Attention, we report the the local size and initial size of tokens that Tri-shape and A-shape can attend to. For Tri-shape, we add dense space of size 64 at the bottom of the attention matrix. MInference is dynamic sparse attention, where the exact sparse patterns are built conditioned on the inputs. According to Jiang et al. (2024), we search the sparse patterns for attention heads with the task of KV retrieval, and also report the search space (i.e.. the distribution of sparse index) for the exact pattern. In KV Cache compression, we report the composition of KV used in Streamingl LLM. The observation window and max capacity of KV Cache size, the kernel size used identify top-k KVs are reported in the Table. For KV cache quantization, retrieval and loading, hyper-parameter in their original implementation and reported them in Table 9.
the default
We use tensor parallel when testing models larger than 7B parameters, with 8*A100 40GB machines or 4*H0 80GB machines. Specifically, we use our customized A-shape, Tri-shape, and MInference kernels in sparse attention testing. utilizing PIT (Zheng al., 2023b) with FlashAttention (Dao, 2024) implemented on Triton (Tillet et al., 2019). vLLM-0. overwritten with
is used as the inference framework in our
testing, and the flash_at wwssshttttt22_em kernels compression, our implementation is based on the huggingface implementation of SinkCache for StreamingLLM' and official implementation of 4 For SSMs and Mamba-Atteninne Hybrid models, we use the triton version of mamba² kernels together with causal cconvdd11 46, For prompt compression, we use the official implementation of LLMMLinuuua 2 compressed the prompt first then use vLLM for further inference.
kernels. For KV Cache
E ADDITIONAL EXPERIMENT RESULTS
The results for Llama-3.1-70B, Qwen2.5-32B, and Llama-3-8B-262K are shown in Table 10.
Phttps: httppp/:/gtthubb.coo//////pppppp
"https: http://////ttttoooooooooooo/ttoo
24

---

Published as a conference paper at ICLR 2025
RullAttention
Trishapee
LLMUinguaa22
FullAttention
Trishape StreamingLLM
1 LLMiingua-2 Quest
Minference
StreamingL SnapkV
Quest
Minference
A-shape
A-shape
Snap KV
Tum 2
Tum-2
Turn-3
Turn3
13.9
25
5.1
ummu-
A
Turn-
Tum-5
Tum-5
(a) String Retrieval
(b) Semantic Retrieval
Figure 9: Performance of different long-context methods across various tasks and turns. The results for multi multi-tasking tasks are shown in Fig. 10, and the results are averaged across all tested base LLMs.
RullAttention
Tri-shape
LLMLingua
FullAttention
Tri-shape
L1MLingua-2
Minference
StreamingLLM
Quest
Minference
StreamingLLM
Quest
A-shape
SnapKV
A-chape
SnapKV
Tum-2
Tum-2
Tum
28
16.3
02.
uu--
Turn-4
Tum
Tum-5
Tur-5
(a) Global Information
(b) Multi-task
Figure 10: Performance of different long-context methods across various turns in Global Information and Multi-tasking tasks on SCBench. The results are averaged across all tested base LLMs.
We can found similar the following key insights from Table 10. MInference consistently outperforms other approaches across tasks, particularly in multi-turn mode, demonstrating strong results in both retrieval and multi-task scenarios. Sparse attention methods like A-shape and Tri-shape show promise, with Tri-shape excelling in multi-request mode due its integration of bottom query tokens, which boosts first-turn performance and improves instruction-following However, Tri-shape's advantage decreases slightly multi-task settings, although still ranks second overall. KV cache compression methods underperform in shared contexts, offering minimal gains, especially in retrieval and global information tasks, with SnapKV showing particularly poor results. Prompt compression methods perform well in tasks requiring global context, such many-shot ICI., but struggle significantly in retrieval tasks, leading to performance degradation. Meanwhile, StreamingLLM and SnapKV consistently deliver the weakest results, particularly in multi-turn mode, indicating they are not well- suited for long-context tasks with repeated requests. Overall, methods like Tri-shape and MInference. which combine sparse attention and efficient token management, demonstrate the most consistent improvements, while compression-foot -occuuee approaches show limited effectiveness in more dynamic retrieval-heavy tasks.
25

---

Published as conference paper at ICLR 2025
Table 11: The results breakdown of SCBench for all sub-tasks in multi-turn mode.
Mix.Sum Mix. RepoQA
Methods
Retr.V
Retr.P Math RepoQA En.QA
Zh.QA
En.MC
ICL EN
Math
+NIAH
+KV
GLM-4-IM
49.0
39.2
58.6
60.5
33.6
15.2
50.2
47.0
37.8
14.4
67.4
78.2
Minference
51.2
28.6
34.8
53.4
33.3
15.0
49.3
47.0
37.7
10.6
68.3
73.4
Ahshape
25.2
42.4
14.0
42.3
28.1
14.1
42.4
49.6
32.9
9.6
65.3
51.8
shi-shape
14.4
28.1
50.0
63.4
StreamingLLM
0.0
0.0
0.0
0.2
5.2
2.0
32.2
70.0
5.8
3.0
12.7
0.0
SnapKV
0.0
26.0
0.5
13.9
2.4
34.2
70.0
6.5
7.2
41.6
LLMLingua-2
1.6
15.8
2.0
5.2
3.5
45.6
48.2
Lloma-3.1-8B
80.8
42.8
47.6
29.3
21.1
$7.0
42.6
40.7
22.0
60.7
70.7
MInference
70.8
15.6
30.8
48.2
30.1
22.5
49.3 57.9
39.8
14.2
56.3
59.3
A-shape
17.8
5.6
18.5
17.1
48.1
13.4
50.8
16.6
Tri-shape
24.2
7.0
23.2
34.5
24.6
20.6
50.5
48.1
35.2
17.2
50.8
25.0
StreamingLLM
0.2
0.0
0.1
0.5
8.7
10.5
39.1
68.9
27.2
9.6
28.9
0.5
SnapKV
0.0
0.0
1.7
1.9
42.6 61.5
3.4
4.2
LLMLingua-2
0.0
1.6
15.4
2.0
23.5
23.0
50,4
35.4
11.2
49.6
49.6
1.mma-3-8
24.0
15.8
47.8
41.8
29.0
13.8
53.1 30.7
37.5
11.8
67.0
60.0
MInference
16.0
3.6
32.8
42.5
30.1
12.3
32.6
37.0
10.4
68.6
63.4
A-shape
2.2
2.0
25.4
32.3
21.7
12.7
46.6
32.2
32.8
11.8
64.8
46.4
Tri-shape
3,4
3.8
26.2
33.4
24.1
12.8
48.3
33.3
32.7
12.8
65.3
55.9
StreamingL.IMM
58.9 17.7
SnapKV
0.0
0.0
1.4
0.0
1.2
1.3
62.2 57.2
2.1
1.4
0.9
0.0
1.MMLnngaa22
0.4
9.8
1.1
21.2
34.8
31.6
7.2
45.9
1ammu-3.1-700
27.2
1.6
33.8
67.0
35.4
20.7
62.2
58.5
41.2
37.4
62.1
78.4
Minference
28.0
29.4
23.3
54.4
35.2
52.1
77.0
A-shape
1.2
0.0
13.2
50.0
27.0
18.0
46.7
52.2
36.5
32.8
35,3
18.6
Tri-shape
0.2
17.1
50.5
55.6
33.4
38.3
23,9
StreamingLLM
0.0
0.0
0.5
0.4
6.0
0.8
23.0
61.1
3.6
3.8
7.0
0.3
SnapKV
0.0
0.0
2.2
0.0
1.3
1.5
14,9
64.5
19
8.8
2.2
0.7
LLMLingua-2
4.2
16.0
22.5
56.7
22.2
60.6
Qwenn25-72B
40.8
62.2
51.5
66.7 67.0
12.2
71.9
82.0
Minference
43.4
46.4
47.0
59.3
41.2
11.4
64.1
38.2
12.8
72.0
73.6
A-shape
17.4
22.7
45.9
64.4
12.0
69.4
46.6
Tri shape
21.0
31.4
24.8
48.0
32.8
12.6
$7.5
64.8
35.8
12.4
70.0
57.5
StreamingLLM
0.0
0.0
1.2
0.2
3.8
0.5
63.9
19.3
3.9
0.0
14.5
SnapKV
0.0
L.LMLingua-2
0.0
3.2
9.3
4.3
32.5
14.7
73.8
72.2
33.1
33.2
53.8
0.9
Qwen2.5.322
56.4
39.4
44.7
64.5
37.1
6.0
68.3
75.9
35.5
10.4
69.8
77.0
MInference
27.8
50.6
34.5
8.0
76.3
10.4
70.7
69.1
shape
14.4
13.6
16.9
46.4
30.1
4.6
54.1
76.7
30.6
8.8
67.2
51.8
Tri-shape
18.2
16.6
20.8
47.3
30.1
6.8
59.6
76.3
33.8
11.2
68.2
59.8
StreamingLLM
0.0
0.4
3.3
0.0
3.6
12.5
SnapKV
0.0
0.0
10.0
0.0
0.3
0.8
37.4 66.7
18.1
2.3
41.6
2.7
0.4
LLMLingua-2
4.0
6.2
6.7
31.4
15.9
66.7
29.5
20.4
52.7
Jambu-1.5-Min
67.4
28.6
37.5
47.5
32.8
21.7
61.8
38.9
48.0
5.6
71.0
71.6
Codestrai-Momes
0.0
0.0
5.7
5.1
21.8
33.3
4.0
12.4
In Table 11 showcases the performance of various methods across a range tasks, including retrieval (Retr.KV, RS), QA (En. QA, Zh. QA), summarization (En.Sum), code understanding and function retrieval (RepoQA), math, and in-context learning (ICL). Each method demonstrates varying strengths and weaknesses across these domains.
Retrieval tasks (Retr.KV, Retr.PS), which test exact information retrieval ability, are dominated by methods such GLM-4-IM and MInference. GLM-4-IM consistently performs well these tasks, with KV at 49.0 and Retr.PS at 39.2. MInference also demonstrates strong performance in retrieval, particularly with score of 51.2 2 in Retr.KV, However, methods like StreamingL.MM and SnapKV show almost retrieval capability, with near-zero scores, indicating poor handling of exact information recall.
For natural language tasks like QA (En.QA, Zh.QA) and summarization (EN.Sum), we see different pattern. GLM-4-IM and Qwen2 models excel in these areas, particularly English and Chinese QA tasks. For example, Qwen2-72B achieves scores of 40.0 in En. En.QA and 66.7 7 in EN.Sum, indicating strong natural language processing abilities. MInference also performs well but is slightly behind GLM-4- IM and Qwen2, with comparable scores. Interestingly, methods like Tri-shape and A- A-shape show moderate performance QA but underperform in summarization tasks compared to the top performers.
26

---

Published as a conference paper at ICLR 2025
In code understanding tasks (RepoQA), GLM-4- 1M leads with score of 60.5, followed by Qwen2-72B at 65.5, demonstrating strong capabilities in handling structured language and retrieving functional information. Methods like MInference (53.4) and Tri-shape (44.8) perform moderately well, while Streamingl L.M and SnapK' are almost ineffective, scoring near zero. This suggests that StreamingL.LMM and SnapKV struggle with code-related tasks requiring structured reasoning.
In math tasks, MInference and GLM-4-INM are the top performers, with scores of 34.8 and 58.6, respectively, showing proficiency in handling mathematical reasoning. However, methods like Tri- shape and A-shape struggle in math tasks, indicating that these sparse attention mechanisms may not generalize well numerical reasoning. StreamingLLLMM and SnapKV again show little to no ability in math, with minimal scores across the board.
Finally, in in-context learning tasks, where the model's ability to generalize and adapt is tested, GLM-4-IM IM and Qwen2 models stand out. Qwen2-72B achieves a high score of 66.7, while GLM-4- 1M also scores well 47.0, indicating strong adaptability. MInference, Tri-shape, and A-shape show moderate ICL performance, but methods like SnapKV and LLMLingua-2 lag significantly, reflecting their limited generalization capabilities in ICL.
Overall, GLM-4-1M 1M and MInference consistently perform well across most tasks, especially in retrieval, QA, and ICI., with the Qwen2 models also excelling in natural language processing and in-context learning. Sparse attention methods like A-shape and Tri-shape show moderate performance in specific areas, while methods like StreamingLLM and SnapKV consistently underperform across the board, particularly in tasks requiring retrieval and code understanding.
In Table 12, we present the results breakdown for the multi-request mode. Comparing the performance across multi-tura and multi-request modes, we found the following key differences, particularly in retrieval tasks. In multi-turn mode, methods like GL.---1M and MInference demonstrate strong retrieval capabilities, with high scores in Ret.KV (49.0 and 51.2, respectively). However, in multi- request mode, these methods show varied results, with MInference dropping to 46.8 in Ret.KV and GLM-4-IM slightly improving 50.6. Sparse attention methods like A-shape and Tri-shape perform relatively poorly in both modes but exhibit more stable results across multiple requests. Notably, the performance of MInference in math tasks significantly improves in multi-request mode (from 34.8 to 51.0), indicating its ability to adapt better over repeated queries. In contrast, methods such as StreamingI.LM and SnapKV V remain consistently weak across both modes, particularly retrieval and math tasks, showing near-zero scores, reflecting their inability to handle dynamic multi-request contexts effectively. Overall, methods like MInference and GLM-4- 1M maintain their dominance across both modes, but their adaptability in multi-reques mode is crucial for retrieval-be hhavy and computational tasks. Note that we did not run
F ERROR PROPAGATION USING GENERATION AS CONTEXT.
Following Zheng et al. (2023a); Wang al. (2024), in our multi-turn testing, we use the golden answer instead of the model generation as the context for the next query. This prevents po- tential interference from misleading generations in subsequent turns. How- ever, this approach naturally provides an in-context learning environment where the model can learn from pre- vious turns in answering later queries.
Table 13: Results when disabling golden answer as con- text. The later number indicate the gap compared to golden- answer-as-conter
Tum
Turn 2
Turn 3
Turn 4
Turn
Llama-3.1-SB
32.4/-2
47.7
36.8 /-13
41.6/-6
29.8
A-shupe
16.5
29.8/42
23.1
1588/11
22.0
Tri-shape
24.5
34.7
24.7
17.1
1-33
StreamingLLM
14.8/66
/-10/-12
5-60
2-80/-11
5-60
Minference
34.5/40
31.7/88
26.2/-19
25.2
25.4
Here we analyze the effect of disabling golden answer as context, to observe whether our findings and observations long-context methods can be maintained in this setting.
As shown in Table 13, we have found similar results on multi-turn setting when model generation is used context compared to main results at Table 4: dense decoding methods perform generally better than sparse decoding. And more robust and dynamic sparse patterns achieve better metrics to static sparse methods. But using model generation context does demonstrate lower overall accuracy
27

---

Published as a conference paper at ICLR 2025
Table 12: The results breakdown of SCBench for all sub-tasks in multi-requests mode,
Mix. Sum Mix.ReppQA +NIAH
Methods
RetKV
Ret.PS
Ret.MHI RepoQA En.QA
Zh.QA
EN.MC
ICI.
EN Sum Math.Find
+KV
GLM-4-IM
50.6
44.6
39.2
54.3
32.8
5.0
32.3
70.4
38.5
21.2
66.2
29.8
MInference
46.8
40.2
15.4
45.0
30.5
5.0
35.4
67.8
38.9
23.5
66.9
29.8
A-shape
26.2
25.8
8.6
39.5
24.5
69.3
31.3
20.7
63.1
22.0
Tri-shape
34.0
30.4
12.0
40.5
25.1
5.3
30.1
68.1
34.7
21.4
63.0
23.0
StreamingLLMM
SnapKV
0.0
0.0
0.0
0.0
8.9
0.9
4.0 63.3
0,1
5.9
0.0
0.0
LLMMingua-2
3.0
1.8
24.2
4.7
70.7
33.1
11.6
48.6
IJamo-3.1
56.2
16.8
25.1
38.4
55.4
23.0
MInference
48.6
15.6
22.5
43.2
23.6
12.5
62.9 62.6
36,6
51.0
45.9
15.9
A-shape
0.n
9.3
33.9
25.6
13.7
59.8
59.6
30.1
49.2
43.6
11.9
Tri-shape
19.2
20.3
17.9
10.1
60.4
29.2
47.2
38.2
10.9
StreamigLLMM
0.2
0.4
0.4
0.0
7.6
5,9
16.4
45.2
6.9
2.7
0.0
0.0
SnapKV
14.3
32.3
I. ingua-2
0.0
1.6
10.1
1.6
19.9
14.5
61.6
73.0
33.7
17.0
42.9
2.8
Llami-3-88 &B
11.8
4.0
35.6
22.7
28.2
8.1
61.1
33.0
36.8
6.9
33.5
14.8
26.5
A-shape
0.6
0.2
22.5
25.5
22.2
8.5
52.8
28.9
31.1
6.2
55.4
15.0
Tri-shape
1.2
23.2
26.1
23.6
30.7
30.7
31.7
56.8
15.0
Streaming.LMM
0.0
0.0
0.0
0.0
3.8
0.1
0.0
67.8
0.1
0.0
SnapKV
0.0
0.0
0.0
0.0
4.3
0.1
0.0
73.3
0.2
0.0
0.0
0.2
LLMLingua-2
42.9
I.ama-3.1 1.mma--.11700 3-1-
2.4
7.0
62.5
32.2
18.3
78.6
67.4
38.4
38.4
62.2
33.4
MInference
3.4
0.0
18.5
57.3
30.4
16.5
70.5 59.4
34.3
$1.0
61.1
31.2
A-shape
25.6
59.6
Tri-shape
0.2
0.0
11.2
44.5
28.5
20.1
69.0
58.9
33.3
47.2
44.7
23.6
StreamingLLMM
0.0
0.0
25.3
66.3
18.7
SnapKV
0.2
0.0
0.0
0.0
11.7
7.0
37.4
76.7
19.9
14.2
0.0
0.0
1LMMingua-
2.8
10.7
6.7
32.2
17.1
72.1
50.0
35.0
30.8
50.7
Qwen2.5-722
37.8
45.2
10.2
64.3
37.0
82.1
74.1
41.6
43.2
71.1
33.6
MInference
40.4
28.6
16.9
56.4
38.5
4.1
79.9
68.5
42.2
45.8
71.3
32.7
A-shape
13.2
22.0
10.4
42.7
29.3
3.7
6644
67.8
38.
37.3
68.0
18.2
Tri-shape
17.2
25.4
44.1
31.6
68.1
39,5
37,9
69.2
20.7
Streaming!.IMM
0.0
0.0
0.0
0.5
5.4
1.6
9.4
8.2
5.1
0.0
0.0
0.0
SnapKV
0.0
2.7
11,0
13.7 79.2
LLMLingua-2
0.0
2.8
5.3
6.7
35.1
3.8
76.7
36.2
34.2
48.9
288
Qwen2.5-328
27.2
23.0
24.9
60.2
35.6
3.0
79.0
84.1
37.3
44.4
68.7
31.1
Minference
27.8
12.8
12.6
$5.0
34.2
85.2
37.8
46.2
60.0
37.2
shape
11.0
7.0
10.7
43.6
26.5
63.3
81.9
31.9
47.2
44.5
32.7
Tri-shape
14.2
9.2
11.8
45.7
28.5
3.0
72.5
83.0
34.2
$2.0
47.7
34.5
StreamingLLM
0.0
3.0
12.8
SnapKV
0.0
0.0
0.0
0.0
12.1
1.7
5.9
13.3
13.7
2.5
0.0
0.0
LLMLingua
29.7
Jamba-1,5-Mint
64.4
13.2
31.9
19.6
37.0
25.2
68.5
27.7
Mamba Codestral
0.0
0.0
8.4
0.2
8.5
2.9
24.5
42.6
6.4
2.6
9.6
0.5
which indicates the error propagation where the follow-up turns will be impacted by misleading answer from previous queries.
G CASE STUDY
In this section, provide more detailed analysis for individual cases. We first present case study of the task Su.Sum across various language models and long- congcct approaches in Table 14. The quality of summarization appears correlate positively with model scale. For example, Llama-3.11700 and Qwen2.5-721 provide more comprehensive and fine-grained summaries compared to others. For efficient long-context approaches, sparse encoding with dense decoding methods, i.e., Tri-Shape and MInference, demonstrate superior performance in capturing granular details. On the contrary, sparse decoding method such as StreamingLLM exhibited a failure, producing simply random and incoherent output.
We then present the results of hybrid architecture Jamba achieve the most accuracy performance. This is non-trivial as Retr.Prefixx Suffix task require an rather large space and time complex and Mamba layers are reported to perform poorly such dimensions. On the contrary, full attention LLMSs such as Llama and Qwen series
Suffix task Table 15. Interestingly, Mmaba- Atteb--tttention
28

---

Published as a conference paper at ICLR 2025
Table 14: Case Study of En.Sum. We use blue to indicate mising informaiton, and orange mark potential hallucination.
Model
Response
Ground Truth
The key contribution of the paper "Accelerating Clinical Evidence Synthesis with Large Language Models" is that it introduces an LLM Lii-driven pipeline called MetaSyns to stream- line the entire clinical evidence synthesis process, including study search, screening. data extraction, and evidence synthesis, while incorporating human oversight ensure accuracy and trunsparency.
Jamba- 1.5-Mini
The paper introduces a generative Al-bused pipeline conducting medicul systematic reviews, utilizing large language models improve the efficiency and accuracy of clinical evidence synthesis.
Llama-3.1-80
The paper proposes large language model-based pipeline for accelerating clinical evidence synthesis, which cun efficiently search, screen, and extruct relevant studies, and synthesize high quality clinical evidence, outperforming traditional methods and achieving high recall rites 0.897-1 1000.
Llama-3.1-700
The paper introduces clinical evidence synthesis pipeline enhanced by large language models, named [blunk], which decomposes the clinical evidence synthesis process into four main tasks and incorporates human expert oversight to minimize errors, enabling reliable and high-quality clinical evidence synthesis to improve clinical research efficiency.
Qwen2.5-32B
The paper introduces a method that leverages large language models (LLMs) enhance the efficiency and quality of clinical evidence synthesis across domains, demonstrating improved performance in study searching, screening, and approcches.
phases compared traditional
Qwen2.5-72B
The key contrihution of the paper "Accelerating Clinical Evidence Synthesis with I Large Language Models" is the development that significantly improves the efficiency and accuracy of clinical evidence synthesis by automating the literature search, study screcning, data extraction. and evidence synthesis processes, while maintaining human oversight ensure reliability.
large language model (LLMM)drrvven pipeline
Llama-3.11700 w/ A-Shape
paper proposes novel approach accelerate clinical evidence syathesis leveraging large language models (LLMMs) to streamline the clinical evidence synthesis process, enahling the creation more representative and accurate clinical evidence.
Llama-3.1 w/ TripSShape
paper introduces clinical evidence synthesis pipeline enhanced by large language models, named [blank]. which accelerate the literature review process, achieving high recall rates and outperforming traditional language model embeddings-busg methods in screening. and surpassing direct GPT-4 performance result extraction, with accuracy ranging from 0.65 to 0.84.
Llama-3.1-70B w/ MInference
The paper introduces a generative Al-based pipeline called [blank] that leverages large language models (LLMs) to improve the efficiency and accuracy systematic medical literature reviews optimizing study scarching. screening, data extraction, and evidence symthesis, while muintaining high performance even when dealing with novel classes from different domains.
Lluma-3.1-70B w/ StreamingLLMM
rucooniicunesghuations OfCUUssrboouciiiveeelleeemnnndo measemeicherle menti-
WithOpiionsiianeeyyotttm litarus [just random content)
Table 15: Case Study of Retr. Prefix-Suffix. Orange used mark the difference of model response compared to the ground truth.
Model
Response
Ground Truth
RviB- TTaAliie__SCCaaamss
Jamba- Mini
RviB- FaAAggcnSFCgaanstt
Llama-3.1-88
RviBmszHq
Llama-3.1-700
RviB--KejaaszHHg
Qwen2.5-32B
RviB-YiaAL YiAAtgeeNrQdayyMTT...
Qwen2.5-72B
RviB-wY sele-xxyymmkaalleemsswww mszHq
Llama-3.1 1-70B w/ A-Shape
RviB-mszHq
Llama-3.1-700 w/ ShippShape
RviB-mszHq
Llama-3,1-70B w/ Minference
WiB-YinAUge.. nSFscaumszHq
Llama-3.1-70B w/ StreamingLLM
incerirketystic WetzinggS0ellperobbeennkiin [just random content]
models all failed in this task. Although many models can still remember variable length of prefix, but they often fail reproduce the entire string. For example, Llama-70B with MInference can almost retrieve the entire string, but misspell several characters in the middle. This can be attribute to the weakness of induction head (Olsson et al., 2022) in the Transformer attention heads, it can also result from the sparse input for these efficient long-context methods.
29

---

Published as a conference paper at ICLR 2025
Table 16: Case Study Mix.RepoQA + KV. Orange indicate the potential model hallucination.
Model
Response
Ground Truth
daf _ot.__ooit_ollsnngg___o paren (line: Line, last: Leaf,
line_ int) -> bool;
*** 'can_omitt_
length we..mit_iiviii iinit___iisilllleprrrssss
line.denth
geen_other_brackets - Palse
for _index, 1eaf, leaf_lergth in line. wtteeeemmmratt_etthlloooooo
length leaf_lenath
if leaf is last.opaning_brackes
if hee__other_brackts or 1ength <= line_lergth:
return True
elff lesf.type OPENING CPENING_ARACKETS: BRACKETS:
There brackets
further split
been_other_bracett. True
return False
Alama-3.1-700
260ebd59-2068 2668-48b7-8241 32a9-6bdfad08bel
def can pin_oom=__ceinn parer. (1ine: Line, *, last: Leal,
line_ Line_lenghh: int) -> bool:
***ee rea_oomt_invissilee_pee
remainder = False
length 4 line.depth
_index
leaf length Aine.enmeeatee with Lenggg if leaf.type in CLOSING_BRACKETS and leaf opa..opning_.rrccet bracket is last:
remainder Tzue
if zemainder:
length +- Lear_length
11 Length > line_lengle
break
Leaf.type in OPENING BRACKETS: -PENING_BRACKETS:
+ There brackels
Durlher splil
remainder - False
eise:
$ checked the entire string and line length waan/t exceeded
11 Len (Line. leaves) (Line. return True
-- Lndex 1:
Llama-3.1 3-1-0BBBBB w/ MInference
200ebd59-2668-48h 111-44bb77-22--hddad0hhheee
python"> def can_omi tamit_closiin celossngg_aaeeee (appparraainn (1ine: line, last:
Leaf, Line_JengLh: InL) bool: *** 'can_omit_ htt_oomi__iniiiilleeerrrre
remainder = Palse
lengLh
1ine.depth
Lindex -1
for _index, leaf, Leaf_lergth in ni enumerate, with withh_eete ()ngth() ir Leef.Lype in BLOSING_BRACKENT and leaf, leaf, opening brackel. is last:
remainder
if zemainder:
length +- leaf_lengt
if length > line_length: break
if leaf. styeetype in
$ There are brackets we can further split or.
rema immainder Palse
5e:
if len (line. leaves) Linex 1:
return True
In addition, we present result for some long-context methods in the multi-tasking test, i.e., ReV in Table 16. The ground truth provides an answer from KV retrieval and one answer from reporqa. Both Llama-3.1-700 and its variant with MInference accurately retrieved the value, demonstrating a good performance on the key-value retrieval. However, their reproduction of the Python function reveals interesting differences. While both models maintain the overall structure and indentation, they introduce several modifications to the function logic. Llama-3. 3.1-70B reproduced the wrong function name and implements brand new algorithm, yet preserves only limited original
30

---

Published as a conference paper at ICLR 2025
Table 17: Case Study of Retr.KV to compare A-shape and Tri-shape.
Model
Respouse
Ground Truth
Turn 1.255002--133--9999-----llaafff
Turn 2: 222-:e66655255444444222233222003
Llama-3.1-708
Turn -99-230--5aa---233888--5555---
1:945bbbbbb.-99422259777.77 (8 n18d844d
4a0d0b3 02:72100220077222445555000 b7b2u4c775e0000 b7h224cc775e0000071222a e715eM00772 u4c775e0d0b7h224e 7f5c0d0b7b 2u4c715eeM007762d44777
Turn 2: cow.0777.7776--8-888444444444400
Llama-3.1 3,1-708 A- Shape
Turn I: assistant<lend header jdbsasiistantellnd header idb> assistant<lend header idis ussistant-dend header idisassistant idiassistant
header idbbassisantelled header idl
assis-
tant<lend header idbeassistauttleed header_ Turn 2: 0at.----5---40b----------8--33--
Llama-3.1-708 w/ Tri-Shape
Turn I:
3ehmMS:3-35al -ibere----------b----e----e 1hc Saechu0d--ccf-4222aaa1 (77
elements. The MInference variant closely mirrors the base model's 's output, with minor differences such as the addition of a Python code block identifier. Notably, neither model exactly replicates the ground truth function, suggesting challenges in precise function reproduction. But believe the results of MInference is more due to the limited long-context capability of the base Llama model instead of the sparse nature of the encoding approach.
In Table 17, we also highlights the performance of A-shape and Tri-shape models in Retr.KV. Notably, Tri-shape demonstrates strong performance even first turn, effectively maintaining the instruction-followinn capabilities of the model. In contrast, A-shape significantly disrupts the model's s ability to follow instructions, leading to incomplete and erroneous outputs. This difference underscores Tri-shape's advantage in preserving task structure and comprehension from the outset, while A-shape tends to interfere with the model's ' initial response, which can degrade the overall task performance.
31


Reviews:
[{"review_id": "s4nSfoB1Y9", "reviewer": "Reviewer_5eBA", "summary": "This paper proposed a new (set of) dataset focusing on the multi-round capability of LLMs under a long context setting.", "questions": "Recently, there have been many newly proposed long context evaluations (HELMET, Michelangelo, etc.) I wonder if the author can provide a discussion regarding such works so that future users will have some guidance on what to adopt.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The proposed dataset fills a major gap in long context evaluation — where there is often only a single question after a long input, with the question being answerable by simply retaining a few pieces of information within the long context — this makes a lot of eviction-based methods seemingly able to achieve strong performance on many long context datasets, although they are not actually long context capable.\n\n2. Aside of the contribution of collecting this new dataset, the author also conducted a decent benchmark of different methods on their dataset.", "weaknesses": "1. The author should consider highlight the source and origin of each dataset a bit better. Many of the featured datasets looks similar from InfiniteBench. I am not sure if they are direct adoptions or modified; in either case, this deserve a better noting.\n\n2. While the proposed dataset focused on multi-round QA under a \"shared context\" scenario. It looks like few or none of its question demand information from a previous QA. The author should consider adding some datasets (or new setting of existing datasets) covering this gap, or at least highlight it in its limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FJPIu6MXd", "reviewer": "Reviewer_WW1Y", "summary": "This paper introduces SharedContextBench, a benchmark specifically designed to evaluate the efficacy of long-context methods in scenarios that utilize KV cache reuse. It features an array of 12 tasks that test four long-context abilities across two modes of shared context. Evaluations are performed on five different categories of long-context methods using eight leading LLMs, with a detailed analysis of the results to uncover insights into the management of KV caches.", "questions": "- How were the real-world use cases that influenced the design of the benchmark tasks selected, and what impact did they have on the weighting of the evaluation criteria? \n- What improvements could be implemented in sub-linear memory methods to enhance their applicability and efficiency in multi-turn interactions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- SharedContextBench represents a significant innovation in benchmarking by addressing the evaluation gap in shared long-context scenarios, providing a robust assessment tool for practical applications. \n- The benchmark comprises a wide array of tasks across various capabilities and domains, ensuring a comprehensive evaluation of long-context methods. \n- A diverse range of these methods are tested against multiple large language models, offering an extensive overview of their performance. The findings bring to light considerable variations in KV cache management, emphasizing the critical importance of multi-turn and shared-context scenarios in method development. \n- The paper also delves into detailed analyses concerning memory usage and the sparsity observed in encoding and decoding processes, yielding valuable insights for future research endeavors.", "weaknesses": "- Certain methods, such as KV cache compression, demonstrate limited effectiveness in shared contexts, which may restrict their practical deployment. \n- The paper points out that many of the long-context methods tested are essentially extensions of existing models adapted for multi-turn dialogues, such as StreamingLLM. \n- The tasks within the benchmark may not capture all possible real-world application scenarios, possibly overlooking specific needs within certain domains. \n- Moreover, while the performance of these methods is thoroughly evaluated, the paper does not explore the deeper internal mechanisms of the models, such as the dynamics of information propagation in sparse attention mechanisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GIUjaF0nNu", "reviewer": "Reviewer_fTBZ", "summary": "This paper introduces a new benchmark designed to evaluate model performance on tasks at the intersection of long-context and shared-context settings. The authors argue that existing benchmarks inadequately address this domain, which is critical because long-context methodologies facilitate applications with substantial context reuse. Using this benchmark, the authors assess state-of-the-art open-source transformer models, their adaptations for long-context processing—such as sparse attention, KV caching, and prompt compression techniques—as well as gated linear RNN models. Additionally, the paper contributes a new static sparse attention mechanism that performs better than existing sparse attention methods. Finally, they analyze the effectiveness of various long-context strategies.", "questions": "1. Why do the experiments exclude prefix caching methods?\n2. What unique insights did the proposed benchmark offer that guided the design of the proposed Tri-shape sparse attention mechanism?\n3. Are there inconsistencies between the results obtained with the proposed benchmark and existing long-context or multi-turn benchmarks that can demonstrate the unique insights that this benchmark can provide?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*Originality*\nThis benchmark measures the performance of long context language models in settings that are currently unaddressed by existing benchmarks.\n\n*Quality*\nThe benchmark covers a wide range of tasks that vary in reasoning complexity and memorization difficulty. All tasks are performed in two shared context modes that are commonly used in real applications. Additionally, the paper demonstrates that the benchmark can be used to draw novel insights about the effectiveness of different long context approaches by evaluating state-of-the-art language models with a wide range of long context methods.\n\n*Clarity*\nThe goal and contribution of the paper are very clear. Experimental results are clearly visualized.\n\n*Significance*\nMany applications enabled by long context models involve context sharing. Thus, this benchmark enables other researchers to make more informed decisions when building long context language models that are effective in real use cases.", "weaknesses": "1. The authors listed prefix caching methods in the related works section but they were missing from the experiments.\n2. The paper does not explicitly highlight the unique insights that they were able to obtain with this benchmark that do not surface with other benchmarks. For example, it does not specify whether the relative performances of these methods consistent when they are measured with other benchmarks.\n3. There does not seem to be a discussion on how well the synthesized data reflects real use cases. In particular, the retrieval use cases artificially vary the positions of the target strings to ensure long context utilization. There is no discussion on its importance in real life use cases, where queries may obey a power law distribution.\n4. It is unclear how the proposed benchmark offered insights that led to the creation of the proposed Tri-shape sparse attention mechanism or how the benchmark uniquely highlights the benefits of this new mechanism, especially since it only significantly outperforms A-shape sparse attention in the first turn.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RBPOy2sRuK", "reviewer": "Reviewer_JF9i", "summary": "This paper introduces a long-context benchmark SharedContextBench, to reveal how lossy are long-context methods in KV cache reuse scenarios. SharedContextBench includes 12 tasks with two shared context modes, covering four categories of long-context abilities: string retrieval, semantic retrieval, global information processing, and multi-task capabilities. The proposed benchmark is evaluated with several long-context methods and six LLMs.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces the long-context benchmark SharedContextBench, to reveal how lossy are long-context methods in KV cache reuse scenarios. The proposed benchmark includes 12 tasks with two shared context modes.\n2.\tThe proposed benchmark is evaluated with several long-context methods and six LLMs. The experimental evaluation is comprehensive.\n3.\tThe paper is easy to follow.", "weaknesses": "The paper did not analyze other long-context benchmarks. It would be benefical, if related long-context benchmarks are summarized and compared in a table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZMa7PbluV", "reviewer": "Reviewer_WDDV", "summary": "The paper presents **SharedContextBench**, a benchmark for evaluating long-context Large Language Models (LLMs) in realistic, multi-turn, shared-context scenarios requiring KV cache reuse. It includes 12 tasks across four abilities—string retrieval, semantic retrieval, global information processing, and multi-tasking—tested in multi-turn and multi-request modes. Evaluated across five categories of open-source LLMs, results show sub-O(n) memory methods struggle in multi-run scenario while sparse and full attention methods perform well.", "questions": "1. The benchmark is primarily designed to evaluate the accuracy, but with some adjustment people may also use it for performance evaluation on multi-run scenario?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SharedContextBench fills an important gap by providing tasks that reflect the performance of LLM in realistic, multi-turn interactions.\n2. The benchmark includes diverse tasks that thoroughly test LLM's long-context processing abilities, such as retrieval and global information aggregation, which matches the needs in research community.\n3. By testing five different long-context methods on a range of LLMs, the benchmark offers valuable comparative insights, making it a robust tool for LLM evaluation in shared-context scenarios. The visualization in the paper is also very intuitive and helpful.", "weaknesses": "1. It would be more convincing to include a table listing the source of the dataset of each task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "76RS2iEwGe", "reviewer": "Reviewer_tui2", "summary": "The paper introduces SharedContextBench, a comprehensive benchmark designed to evaluate long-context Large Language Models (LLMs) in scenarios where multiple requests share the same input context. The benchmark covers 12 tasks across four categories of long-context abilities: string retrieval, semantic retrieval, global information processing, and multi-tasking. It includes two shared context modes: multi-turn and multi-request. The paper evaluates five categories of long-context methods on six transformer-based long-context LLMs, highlighting the impact of sparsity in encoding and decoding, task complexity, and more.", "questions": "Could you provide the source codes for the community to verify other methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The benchmark includes tasks that reflect real-world applications, such as multi-turn conversations, self-play Chain-of-Thought reasoning, and repository-level code debugging.\n2. The extended retrieval tasks of key-value retrieval, prefix-suffix retrieval, and multi-hop retrieval are interesting to benchmark the LLMs.\n3. The paper provides a detailed analysis of various long-context methods, highlighting their strengths and weaknesses in different scenarios, which offers valuable insights for future research and practical applications.\n4. The taxonomy of different long-context methods in Table 1 is comprehensive and help to illustrate their advantages and disadvantages.\n5. The paper highlights the challenges faced by sub O(n) memory methods in maintaining accuracy in multi-turn scenarios, providing an important insight into the limitations of these methods.\n6. The study shows that sparse encoding methods with O(n) memory and sub $(O(n^{2})$ computation in prefilling generally perform well, offering a practical solution for efficient long-context processing.\n7. The paper finds that dynamic sparse patterns in prefilling often produce more expressive memory compared to static methods, suggesting a direction for future research in optimizing long-context methods.\n8. The breakdown results (Table 8 and 9) help to show more fine-grained insights.", "weaknesses": "1. The paper provides configuration details for the long-context methods, but it lacks a detailed analysis of how hyperparameters were tuned or their impact on performance.\n2. A more thorough analysis of hyperparameter sensitivity could provide insights into the robustness of the proposed methods.\n3. Table 4 is a little confusing. Why some highest scores are not bold? And what is the meaning of the underscore?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)