Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$  Pruning
Authors: Chaofan Lin
Jiaming Tang
Shuo Yang
Hanshuo Wang
Tian Tang
Boyu Tian
Ion Stoica
Song Han
Mingyu Gao
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Model
Sparse Attention
Decode
KV Cache
Primary area: deep_learning
TL;DR: We propose a method which exploit KV cache sparsity efficiently and dynamically through Top-P sampling.

Abstract: Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been of great importance recently. However, most existing sparse attention algorithms use a fixed budget of how many tokens to use in their computations. This simple static decision raises critical issues in real-world deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. 
In this paper, we reveal a key insight that leveraging the idea of top-$p$ sampling (a.k.a., nucleus sampling) in sparse attention could enable efficient and adaptive budget decisions. Based on this, we propose Twilight, a framework that enhances any existing sparse attention algorithm with adaptive budget decision capabilities without sacrificing accuracy. 
Empirical results show that Twilight can adaptively prune up to 98% tokens with nearly no accuracy loss in both mid- and long-context scenarios, leading to a $1.4\times$ speedup over state-of-the-art sparse attention mechanisms.

Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning
Chaofan Lin Jiaming Tang Shuo Yang Hanshuo Wang
Tian Tang Boyu Tian Ion Stoica Song Han Mingyu Gao
Tsinghua University Massachusetts Institute of Technology
University of California, Berkeley
ccf240mails .atismmaalsssttinguuuaaa edu.cn gaomy@tsinghua. edu. cn
https: https://github. http///ttwb..om/rssnnaaaaaaaaaaa htw.siingguu-ddallllllllle..
Abstract
Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been great importance recently. However, most existing sparse atten- tion algorithms use fixed budget of how many tokens This simple static decision raises critical issues real-world deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we reveal key insight that leveraging the idea of top-p sampling (a.k.a., nucleus sampling) in sparse attention could enable efficient and adaptive budget decisions. Based on this, we propose Twilight, framework that enhances any existing sparse attention algorithm with adaptive budget decision capabilities without sacrificing accuracy. Empirical results show that Twilight can adaptively prune up to 98% tokens with nearly no accuracy loss in both long- and medium-context scenarios, leading to a 1.4x speedup over state-of-the -fttt-rrrt sparse attention mechanisms.
in their computations.
Introduction
Large language models (LLMs) with long-context capabilities have revolutionized wide array of natural language processing applications, such as retrieval-basee tasks, document summarization [1], and code generation [16]. The increasing availability of models supporting context windows up to 1M to 10M tokens [47, 37] highlights the growing potential of these advancements. For instance, video language models (VLMs) [41] often require of thousands of tokens for video processing. Similarly, large reasoning models [8, 38), which are rapidly growing in popularity, frequently demand substantial token lengths to enable chain-of- of-thought (CoT) reasoning. Consequently, the importance of long-contens LLMs is increasing rapidly to meet the needs of these sophisticated applications.
Despite the substantial potential of long-context LLMs, they come with excessive computational and memory costs [58, 36, 45], primarily from the attention mechanism. Particularly, in the decod- ing stage of LLMs, the key-value (KV) cache size grows rapidly the token sequence becomes longer. These data need to repeatedly loaded from the memory, leading to significant latency overheads. Furthermore, the substantial size of the KV cache significantly increases the GPU memory consumption, compounding the challenges of continuously scaling long-context LLMs.
Previous research has extensively investigated the use of attention sparsity (a.k..,, KV cache sparsity) to accelerate long-conter inference, both during the prefilling and decoding stages. The core idea is to compute approximate attention subset of tokens, often referred to "critical tokens" or "heavy hitters" [58]. The number of selected tokens, denoted as B, is commonly referred to as the KV cache budget. A top-k operation is required to identify the indices of the critical tokens that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Approximate Attention
Σ Ma> a9
E
(a) Top-k Sparsity (H20, Quest.... budget results either over-selection or under-selection.
Focused Altention
Diffiee Atlentio
(b) Topy Sparsity (Ours) dynamically adjusts the budget accumulate sufficient attention weights.
(pooling, pruning, quantization...) an-
Σ A
Comparison of top-k and top-p for attention sparsity. Approximate attention typically employs techniques such as pooling, channel pruning, and quantization to approximate the query (Q) and key (K) and estimate the attention weights. These weights are then used to select important tokens for sparse attention. (a) Top-k sparsity, utilized by most existing designs, relies on fixed k-token budget and often results in over-selection (E P > 0.9) or under-selection ( Pr < 0.9). (b) Our proposed top-p sparsity dynamically adjusts the budget to accumulate just sufficient attention weights (Σ p. = 0.9), enabling more efficient and adaptive sparse attention.
Figure/ 1:
correspond to the B highest estimated scores. However, a key tradeoff exists for the selection of the budget. smaller B value significantly reduces the memory accesses and computations, while larger B value retains more contextual information and thereby minimizes the accuracy loss.
Identifying the optimal value of B to balance both accuracy and efficiency is inherently chal- lenging due to major reasons: (a) The best budget choices vary dynamically at runtime.
Perplexity (lower is better)
Oracle TopK
10
Quest
DS
Previous works [44, 45J have demonstrated that some heads, referred to as "retrieval heads", are
trained to extract important information from
long contexts, while others focus only on local
information. From Figure 1 we see that the dis- tribution of attention weights may vary across different attention heads. Some attention distri- butions concentrate on small subset of tokens, which we refer to as focused attention. Other attention distributions may be flatter, where many tokens have similar attention weights; we call this diffuse attention. For focused attention, using fixed token budget for top-k attention often leads to over-selection, as only a few tokens are needed to accumulate sufficient attention weights. In contrast, for diffuse attention, a fixed budget result in under- selection, as a larger number of tokens are necessary to ensure accurate attention modeling. (b) Existing algorithms need different degrees of over-selection to compensate the estimation inaccuracy. As shown in Figure 2, the optimized budgets highly depend on the specific algorithms, necessitating offline calibration to determine the appropriate budget for each algorithm individually. Certain methods, like Quest [36] or DS [50]. have to over-select some tokens to compensate for the inevitable inaccuracy in estimating the importance of tokens compared the oracle.
128
256
384 512 640 768 896 1024
Budget (# Takens)
Relationship between the KV cache budget and the perplexity the PG-19 dataset in different top-k sparse attention methods.
Figure 2:
In this work, we reveal that the top-k methods exhibit issues similar to those previously encountered in LLM sampling. Drawing this analogy, we introduce top-p sampling into sparse attention to address the budget selection dilemma. Our study demonstrates that top-p can determine the KV cache budget in more intrinsic and dynamic way compared top-k. Based on these observations, build Twilight, a hierarchical KV cache pruning framework that enhances existing sparse attention algorithms with adaptive budget selection capabilities. Specifically, Twilight first lets the base algorithm select relatively large subset tokens using a conservative budget, and then further refines this subset by retaining only the top-p tokens.
Our evaluations for Twilight are conducted in two aspects: accuracy and efficiency. First, we demonstrate that Twilight optimizes the base algorithms with nearly no accuracy loss on both medium-context benchmarks (GSM8K [4], COQA [33], PG-19 [32]) and long-context benchmarks (Longbench [1], RULER [14]). Next, show that Twilight achieves up to 15.8 x performance

---

improvement over the full attention operation. Compared to prior sparse attention methods, Twilight enables 1.4x speedup for the self-attention operator itself, and 1.35x speedup for the end-to-end decoding. Our contributions are summarized as follows:
We conduct an in-depth investigation into critical challenge in top-k sparse attention: the difficulty in identifying the optimal budget (i.e., the number of selected tokens). We propose to use top-p sampling instead to dynamically determine this budget at runtime.
We introduce Twilight, a framework that can endow any existing sparse attention method with adaptive budget selection capabilities, thereby further improving their efficiency.
We evaluate Twilight in terms of both accuracy and efficiency, demonstrating a speedup of 1.xx over existing sparse attention methods with nearly no accuracy loss.
2 Related Work
Top-k Sparse Attention. H2O [58], StreamingLLM [46], and SnapKV [23] evict non-critical tokens in static, query-agnostic manners, which are often referred to KV cache compression. In contrast, SparQ [34], Quest [36], Double Sparsity (DS) [50], and HShare [43] retain tokens in the GPU memory but select critical tokens to save data accesses. Recent works like Retrieval/Attention [26] and PQCache [55] adopt advanced algorithms to better estimate the token criticality. NSA [54] and MoBA [29] explore opportunities in trainable sparse attention. However, these methods are all based on top-k which requires proper budget selection and configuration beforehand, and thus suffer from the over/under--electionn issues.
Dynamic Budget. More recent studies have extensively demonstrated that the optimal budgets vary significantly across different layers [2, 48), attention heads [9, 45, 35], and prompts (tasks) [61]. Please see Appendix A for details, These works tend to focus on only one aspect of the dynamism. In this paper, we point out that it is the different distributions of attention weights that are the root cause of such dynamism.
Non-top-k Sparse Attention. Some recently emerged designs also go beyond top-k methods. MagicPIG [3] uses locality hash (LSH) sampling instead of dropping tokens to estimate attention weights, but requires complicated algorithm-system co-design. SampleAttention [64] also features adaptive sparsity but focuses on the prefill stage. A concurrent work with ours, Tactic [62], also dives into top-p sparsity but it uses function fitting to estimate the weight distributions. Although it potentially has lower estimation cost, it usually overestimates the budget.
Other KV Cache Optimizations. Several alternative approaches focus on optimizing the KV cache beyond sparsification, including quantization [13, 27, 17,30], 30], linear attention [42, 18], and memory- efficient attention mechanisms such as FlashAttention [5] and SageAttention [57, 56). Our approach orthogonal these methods, and can be combined with them for enhanced performance.
3 Bringing Top-p Sampling to Sparse Attention
In this section, we formulate the current sparse attention methods and re-examine the root cause of their inefficiencies. We argue that to mathematically approximate the attention, the goal is to select a minimum set of indices such that the sum of their attention scores meets a certain threshold. Therefore, we propose to use top-p sampling instead of top-k to efficiently identify the critical tokens.
3.1 Problem Formulation
We start by formulating the sparse attention mechanism. Consider the attention computation during the decoding phase, where we have the query vector q E gg and the KV cache K, V E gaxd Here, d denotes the head dimension, and represents the context length. Definition 3.1 (Sparse Attention).
Let I be the set of selected indices. Sparse attention calculates
q KT
ô = softmax
A V e Rlxd
Vd
{
1
if = j and i E otherwise
where Az RRan Az Az
3

---

Umarmalizzd Attertion Weight
Unnormalized Attontion Vinlgh
2.5
2.5
0.0
Unnarmalizzed Atterttion
0.0
-2.5
-5.0
-5.0
7.5
b L
-10.0
-10.0
-12.5
-12.5
0
500 1000 1500 2000
2500
500
1000 1500 2000 2500 Taken Position
500
1500 2000 25000 Taken Position
1000
Token Position
Figure 3: Diverse distributions observed in attention weights. The leftmost image illustrates "flaat" distribution (diffuse attention), where the weights are close to uniformly distributed. The middle image depicts "peaked" distribution (focused attention), where the weights concentrated on the tokens at the two sides. When overlaid as in the rightmost image, the differences between these distributions become readily apparent.
Let the accurate attention output be softmax 9.KT I I To minimize the error ||o ō||- va
we need to carefully select the subset of tokens used in However, directly optimizing this objective function without loading the full KV cache is challenging. According to the sub-multiplicative property of the Frobenius norm, we can bound the error as in Equation 2. Earlier research has shown that the distribution of V is relatively smooth [59], which implies
[[ be viewed a constant.
f 1o 6| [(Az 1"X") V
W [(A 1xx")] |
Therefore, the objective becomes minimizing [W(z 1.x))l = 1- Wii]. which means
selecting a subset of tokens that maximize the sum of their attention weights. If we fix the size of this subset, i.e. [Z], then we have the oracle top-k attention:
Definition 3.2 (Oracle Top-k Sparse Attention). Given the budget B,
I = arg max
Ew(
s.t. I B
(3)
I
REI
This serves as theoretical upper bound of the current top-k sparse attention methods.
3.2 Rethinking the Problem of Top-k
The Achilles' heel of top-k sparse attention, as described earlier, the dilemma in determining universally applicable budget B to all scenarios. We find that this predicament is quite similar to a previous problem encountered the sampling phase of LLMs, during which the model samples the final output token from the predicted probability distribution. Nucleus sampling [12), a.k.a., top-p sampling, was proposed to address the problem that top-k sampling cannot adapt to different next-word distributions.
Motivated by this insight, examine the distributions of attention weights more closely. As indicated by Equation 2, the output error is bounded by the sum of the selected attention weights. Therefore, the objective should become selecting the minimum number of tokens to satisfy given requirement for the output error. Figure 3 displays two different types of attention weight distributions in several real-world LLMs mentioned Figure 1. is easy observe that, compared to the peaked distribution, a greater number of tokens must be selected in the flat distribution to reach the same cumulative threshold.
Therefore, we argue that the core reason for budget dynamism is the dynamic nature of attention weight distributions at runtime. We thus introduce top-p sparse attention by directly applying threshold to the sum of attention weights.
Definition 3.3 (Oracle Top-p Sparse Attention). Given the threshold p.
I arg min [
s.t.
w
(4)

---

Compared to top-k, top-p is more advanta- geous because it provides theoretical upper bound of error as
Cumulative Attn. Score
1.0
197.0.80
p) | from Equa- tion 2. Under this circumstance, top-p reduces the budget as low as possible, making it both efficient and adaptive to different distributions. To demonstrate how top-p reduces the budget,
0.8
1024:6:07
0.6
1.6.0.77
0.4
0.2
we investigate real distribution of attention
scores shown in Figure 4. Compared to two fixed budget strategies B = 16 and B = 1024 that respectively result in mnder-selection and over-selection, the p = 0.8 point selects a very small budget B 97 to reach similar error requirement to that of B = 1024.
0
500
1000
1500
2000
2500
Taken Position
Figure 4: Cumulative attention scores of different budget selections in one example attention head.
Twilight
With the efficient and adaptive top-p sparse attention, our primary goal is to use it to endow existing algorithms with adaptive budget selection capabilities, rather than simply inventing yet another sparse attention design. We are mainly motivated by two reasons. On one hand, despite the challenge of budget selection, existing sparse attention algorithms have achieved significant success in LLM serving systems [20, 60], thanks to their effective token selection strategies. These strategies can be readily reused and enhanced with our adaptive sparsity. On the other hand, we anticipate that future sparse attention methods may still employ top-k selection. By developing general solution, we aim automatically equip these future methods with adaptive attention sparsity, while avoiding extensive redesign. Consequently, we position our system, Twilight, as an optimizer for existing algorithms.
Nevertheless, applying top-p var- ious existing sparse attention algo- rithms faces three key challenges both the algorithm and system per- spectives. (C1) Not all algorithms are suitable for top-p. Top-p im- poses strict constraints on the layout of attention weights. For example, simply replacing top-k with top-p Quest [36] would not work, as Quest performs max pooling on weights with per-page layout (16 tokens per page). Additionally, some other methods [49, 26] do not use attention weights to select critical tokens at all. (C2) It is harder to estimate weights for top-p than top-k. In order to find critical tokens without loading full K data, low-precision representation of the cache is usually used [50, 55]. However, the precision requirement of top-p is higher than that of top-k, because the former requires a certain degree of numerical accuracy while the latter only demands ordinality. Table I compares top-k, top-p, and full attention. The precision requirement of top-p attention lies between the other two, necessitating reconsideration of appropriate precision choices for the cache. (C3) System-level optimizations are needed. Since our work is the first to introduce top-p to attention weights, the relevant algorithms need to be efficiently implemented on the GPU, including efforts on both parallel algorithm designs and kernel optimizations.
Table I: Comparison of different pruning methods on attention weights. "Normalization" means sof tmax.
Precision Requirement
Output
Need Normalization?
Method
Efficiency
Accuracy
Top-k TToppp Full Attn.
High
Median
High
Median
High
Low
High
High
In Section 4.1, we address C1 by proposing a unified hierarchical pruning framework for top-p sparse attention. In Section 4.2, we mitigate the runtime overheads with efficient kernel implementations (Top-p, SpGEMV, Attention) and 4-bit quantization of the K cache, addressing C2 and C3. Lastly, in Section 4.3, we analyze the overheads of Twilight and discuss some additional issues.
4.1 Hierarchical Pruning with Sriecttthe-Prune Architecture
To uniformly support various sparse attention mechanisms, we propose a two-step, hierarchical pruning process. We first capture the base algorithm into a black-box Token Selector as long as it has the common semantics of selecting subset of critical tokens, while the exact algorithm details of how to select do not matter. We let the Token Selector conservative, relatively large budget,
5

---

O.K.V
k.
Metadata for Base Algorithm
Token Selector
of Box Alpe (Qoad DS,
W soltmaxxSpGEMVQC KD)
Token
0.5
0,03
0.02
0.15
0.04
0.25
Selocted Indices lg
INT4 Quantized Paged Cache
A
R
Twillght Pruner
IS
Step
p 0.9)
Selected
Indices 1
0.5
0.25
Paged Key-Value Cache
=0
Sparse (Paged) Attention Kernel
0.1
Token
Output
Figure 5: Twilight architecture. Twilight incorporates a certain existing base sparse attention algorithm and further optimizes it. It computes self-attention in three steps. First, the Token Selector selects critical tokens using the base algorithm under conservative budget. Then, the Twilight Pruner prunes the selected token subset via top-p thresholding. Finally, the pruned token indices are passed to the Sparse Attention Kernel to perform the attention computation.
e.g. 1/4 sparsity. Then, we have a Twilight Pruner after it to further optimize the selected indices by only retaining the top-p tokens, i.e., the minimum subset whose attention weight sum exceeds the threshold We call this design as the Select-then- then-Prume architecture, as illustrated in the middle of Figure 5. The final sparse attention kernel thus only computes on the top-p tokens, achieving the benefits of efficiency and adaptivity as proved in Section 3.
4.2 Efficient Kernel Implementations
Now we briefly describe the details of the Twilight architecture, particularly for the Pruner step. For more details of the kernel implementations, please refer to Appendix B.
Efficient SpGEMV with 4-bit Quantization of Key Cache. The beginning part of the Pruner is similar to other sparse attention algorithms, which to esti- mate the importance of tokens. As we formulated in Section 3.1, this can be done by estimating the similar- ity between q and K, i.e., q K. Since loading K is known be memory bound, we reduce the memory access cost by quantizing K into lower precision. But what precision shall we choose? Table shows the precision requirement of top-p lies in between top-k and full attention. Some existing top-k designs [50, 55] have pushed the compression of the K cache to tremely low precisions
Sum of Attention Weights
0.8
0.6
0.4
2-bit
4-bit
8-bit
0.0
Layer ld
Figure 6: Sums of normalized attention weights for the selected tokens under differ- quantization precisions, with p 0.85.
to 2 bits. For full attention, SageAttention [57] has demonstrated 8-bit precision with smoothing K and per-block quantization. In this work, we empirically find that 4-bit precision strikes balance between accuracy and efficiency for top-p, illustrated in Figure 6. Here the sum of attention weights with 2-bit quantization drops significantly, while 4-bit and 8-bit methods both maintain enough stability.
Hence we implement an efficient sparse GEMV (SpGEMV) kernel based on FlashInfer [52], a regh-performance kernel library for LLM serving. Here "sparse" means the quantized K cache data are stored/loaded in a paged manner [20] to align with the original KV cache layout. We maintain this extra INT4 asymmetrically quantized K cache the GPU as shown at the right of Figure 5. The INT4 K vectors are unpacked and dequantized the shared memory, reducing data accesses from the global memory at most 1 1/4, resulting in considerable end-to-end speedup.

---

Efficient Top-p via Binary Search. A brute-force way to do top-p sampling is to sort the elements by descending order and accumulate them until the sum meets the threshold. This is quite inefficient in parallel hardware like modern GPUs. As our top-p method is motivated by the top-p sampling, we also implement this kernel by modifying the top-p sampling kernel from FlashInfer [52]. Specifically, our kerne adopts a parallel-friendly binary search method as in Algorithm 1. Note that glement-wise operations like max, where, and sum can be fused into a single loop. which is tensorized on the GPU. Thus we do not need to materialize the intermediate variables like Wo.
Algorithm 1 Top-p via Binary Search.
Input: normalized attention weights W É R top-p threshold p. hhppeh--paaae syper-parameter c.
Output: indices I, mask M E {0, 1) 1
= 0, r = max (W), m = (1 (1+ r)
repeat
Wo where (W < m, 0.0, W);
Wi = where IW - 1, INF, W);
W2 = where (W >T; -INF, W);
if sum (W)) 2 p then
[
else
r = m; end if until max(W2) min 2
Select indices and set mask M where W 2 L; return I. MM:
Load Balancing with Awareness of Head Dynamism. The top-p Pruner enables head-wise dynamic budgets, but also raises load imbalance issues in the attention kernel. Traditional implementations allocate uniform computation resources to all heads. FlashInfer [52] deeply investigates this load imbalance problem, but only for requests with dynamic lengths. Twilight further reuses the load balancing algorithm in FlashInfer to address head-wise dynamism, by flattening the head dimension.
4.3 Overhead Analysis and Discussion
Execution Time, The execution time of Twilight consists of three parts according to the pipeline in Figure 5: Trkennset + Trrner SparseAAtm Compared to the baseline sparse attention without Twilight, our method introduces an extra latency term Traner but reduces TppaseAumn Our hierarchical architecture naturally matches the hierarchical sparsity, where the number of tokens gradually decreases as the precision increases. Suppose the base algorithm in the Token Selector estimates token importance with a 1/16 sparsity and/or precision reduction. Then the theoretical speedup can N/16+B, Bo be formulated as 1/16+BB//4BB; where Bo = [To] is the budget of the base Token Selector, and Bi [Z|| is the budget after pruned by Twilight with INT4. Assuming Bo = N/4 and B1 N/64, the speedup would be approximately x. Here we omit the overheads of the top-p kernel since SpGEMV dominates the latency when Bu is around N/8 to N/4.
Memory Overheads. Twilight introduces an extra INT4 quantized cache, which brings a 1/2 x 1/4=1/8 extra KV cache memory overhead. However, this additional cost does not appear in all cases. First, some base algorithms, like DS [50], already maintain an INT4 K cache. Second, some recent efforts have explored INT4 full attention [56]. This allows us to directly reuse the estimated attention weights calculated by the INT4 K cache in the attention computation, without maintaining the original FP16 K cache. Moreover, offloading and selective quantization (e.g., keeping the extra INT4 K cache only for hot tokens) can be leveraged the GPU memory becomes a bottleneck, which we leave future work.
Integration with LLM Serving Systems. Our system design naturally aligns with PagedAtten- tion [20], so Twilight can be seamlessly integrated into popular serving systems like vLLM [20] and SGLang [60]. Other common techniques, such as prefix sharing and multi-phase atten- tion [24, 60, 63, 51, 53], are also compatible with Twilight since we use page-level or token-level sparse operations, and can achieve flexible computation flow.
5 Evaluation
In this section, we perform quantitative experiments to demonstrate that equipping state-of-the-art (SOTA) sparse attention algorithms with Twilight could improve efficiency while preserving accuracy. We present the accuracy and efficiency results Section 5.1 and Section 5.2, respectively. At last, we perform ablation studies in Section 5.3.
7

---

5.1 Accuracy Evaluation
Benchmarks and Models. We evaluate Twilight on two types of benchmarks: long-context, which includes Longbench [1] and RULER [14], and medium-context (500 to 2k tokens), which includes GSM8K [4], COQA [33], and the perplexity on the PG-19 dataset [32]. We select three widely used models, Longchat- 7B- [22], L.L.MA2-7B-Cha [40], and LLaMA-3.1 1---3.1-8B-Inn 1-8B-Insttruct [28] (128k context length), with two of them having long context ability 2 32k. They cover two mainstream attention implementations of multi-head attention (MHA) and group query attention (GQA) [28].
Baselines. We use two SOTA top-k sparse attention methods, Quest [36] and DS [50], and one SOTA non-top-k method, MagicPIG [3], as our baselines. Following the baselines, we do not apply any sparse methods to the first two layers to ensure fair comparison. For DS, we use the optimized configurations tuned for each model provided by its official repository. The hyperparameter p of Twilight is set 0.95 for LLaMA-2/3 and 0.85 for Longchat, which will be explored in Section 5.3. Note that MagicPIG does not employ the budget mechanism but instead relies on two configurable parameters, K and I,, which directly influence its accuracy. In our experiments, we adopt two standard configurations from the original MagicPIG paper. Due to the lack of official MagicPIG support for LLaMA-2, we exclude these experiments from our evaluation.
Results on Longbench. We comprehen- sively evaluate Twilight's long context abil- ity on 12 different tasks chosen from Long- bench, covering all task types, using two long-context models. For each top-k base- line, we vary the budget from 256 to 8192, and then apply Twilight to dynamically de- termine the budget. We also equip "Full" with Twilight, in which the Token Selector a trivial one that keeps all tokens.
Table 2: Average scores on 12 different tasks from Longbench. We report relative error changes (improve- ment or degradation) when integrating Twilight with each base algorithm. Detailed results are in Table 5 in Appendix C.
Longchat-7B -v1.5-32k
LLaMA-3.1-8B
Budget
-Instruct
32k Twillght
36,78
52.01
Full
38.52 (44.7%)
51.64 (-0.7%)
The results are shown in Table 2. In Longchat, the Twilight framework able to outperform its original version by up to 5.7% in the score, while successfully prun- ing up to 98% % of the redundant tokens over- selected by the base algorithm. In L.LaMA- 3.1 1--B--Instruct Twilight achieves nearly zero accuracy loss ( (<1%) with a slight in- crease in budget usage. We hypothesize that this slight increase due to the knowledge being more compressed in LLaMA-3.1.
MagicPIG
K=8, 1=75 K=10, L=150
51.70
51.32
256
31.26
38.20
36.85
47.79
Quest
4096
37.33
50.79
8192
37.10
51.44
Twilight
38.04 (+2.5%)
51.57 (+0.3%)
256
35.32
45.74
1024
35.96
49.43
DS
4096
36.31
$0.98
8192
36.62
51.14
Twilight
38.71 (+5.7%)
51.73 (+1.2%)
Results on RULER. 8B-Instruct model, which incorporates specialized tests including CWE/FWE for comprehensive non-retrieval accuracy evaluation. As presented Table 3, while the standard Quest implemen- tation underperforms the non-top-k: approaches, DS demonstrates surprisingly competitive results. When enhanced with Twilight, both variants show significant improvements: Quest-Twi achieves performance comparable to the SOTA non-top-k method MagicPIG, while OS-Twi establishes new record-breaking performance, surpassing all existing methods.
We further evaluate Twilight on the RULER benchmark using the LLaMA-3.1
Results on Medium-Contedd Tasks. We then demonstrate that the Twilight Pruner itself does not negatively impact performance on two zero-shot generation tasks, GSM8K and COQA using the Im Im-harness framework [10], as well as one perplexity test on the PG-19 dataset. Since we are specifically evaluating the Pruner, we do not integrate Twilight into the baseline models. All the basclines use budget of 128, which is comparable to the budget after Twilight's 's pruning. The results in Table 4 show that Twilight outperforms Quest and DS by significant margins, with nearly zero loss compared to full attention.
5.2 Efficiency Evaluation
Datasets. We evaluate the efficiency of Twilight on both the self-attention operator and the end-to-end decoding stage single A100 GPU. We use Longbench, from which select three different types
8

---

Table 3: Average scores on RULER.
Table Results medium-contex benchmarks.
Budget
16k
32k
64k
96k
Avg.
GSMSK(hexchleesmiccrf JOQAem/f1 Perplexity.
Full
100%
92.88
42, 85.17 85.23 88.18
LLaMA-2-78 7B Char
Twilight
93.13
89.10 84.64
83.10
87.49
0.229000 2282
0.5035/0 7511
7.503
MagicPIG
K=8, L=75
92.22
89,37 84.07 82.58 87.06
Quest DS Twilight
0,2191/0.1190
0.0523000508
0.577100
0.585510.7400
7.622
K=10, L=150
91.38
88.20 83.34
86.23
0.21530.2115
1.6088N0.7642
7,600
4%
79.35 79.8 78.64 73.22 77.75
(Tiviligtt Arg. Budget)
91.86
102.58
Quest
8% Twilight
87.31
83.06 80.82 75.28 81.62
IJ 1-88--11---nnmaaa
91.53 87.97
84.12 82,96 86.65
Full Quest Twilight (Tivilght Arg. Budget)
0.7726/0.7475
0,636310 7882
7.490
0.36390.3533
7554
4%
92.04 88. 84.43 82.56 86.79
0.6194/0.6027
0.6455M0.966
7,967
DS
8%
92.89 88.70 84.39 82.72 87.18
0.7771/1,7604
(.632550 7869
7.529
Twilight
93.54 89.24 85.91
82.81 87.88
112.40
86.85
110.98
12500
25000
6000
20000
Time (UR)
2000
Time (px)
4000
Time (us)
7500
Time (us)
15000
5000
2000
2500
5000
10k
20k
30k
10k
20k
30k
10k
20k
30k
0
10k
20k
30k
Sequence Length
Sequence Length
Sequence Length
Sequence Leng
Methods
FA2
Flashinfer
Quest
FlashInfer-Tiv
Quest-Twi
Figure 7: Latencies and speedups of self-attention at different sequence lengths and batch sizes.
of tasks: Qasper [6] for QA, GovReport [15] for summarization, and LCC [1I] for coding. We use prompts ranging from 10k to 30k tokens for evaluation. Given that Twilight designed for deploying sparse attention in LLM serving systems, use batch inference in experiments.
Baselines and Implementation Details. We compare our methods with the following baselines: scaled-dot-produut--teeeeee Efficient Attention [21] as the backends; FlashInfer [52], herhh-performance kernel library for LLM serving: Quest, which achieves SOTA runtime performance among sparse attention methods. We integrate Twilight with both FlashInfer and Quest, resulting in FlashInfer-T Twi and Quest-Twi. We modify the Quest kernels to support batch inference. We implement Twilight using both CUDA and OpenAl Triton [39], following the technical details described in Section 4.2.
PyTorch's scaled-dot- product-attention (SDPA), with FlashAttention2 (FA2) [5] and Memory-
Self-Attention Speedup. We first evaluate the speedups the self-attention operator across different batch sizes and sequence lengths. As Figure shows, FlashInfer- Twi and Quest- Twi achieve speedups up to 6.5x and 15.8x compared with FlashAttention2. Moreoever, they accelerate the respective base algorithms FlashInfer and Quest by 2.4x and 1.4x.
End-to-End Decoding Speedup. We evaluate end-to-end decoding with batch sizes ranging from 32 to 256 for various serving scenarios. Figure 8 illustrates that Quest- Twi achieves up 3.9x speedup compared with FlashInfer, and a 1.35x x speedup compared to Quest without Twilight.
Batch Size
32
Batch Size 64
Batch
128
Batch Size 256
125
400-
800
Time (me)
100
Time (ms)
2.50
300
600
75
100
Time
200
25
100
200-
0
10k
20k
30k
0
10k
20k
30k
10k
20k
30k
10k
20k
30k
Sequence ence Length
Sequence Length
Sequence Length
Sequence Length
Methods
NSN
Flashinfer
TB
Quest
BB
Quest-Tw
Figure 8: Time Time-Per-Output-Teee Token (TPOT) improvements in end-to-end serving scenarios.

---

Perplexity flawrr is better)
Quest-Tai
Components
12
120
16
Quest
SpGEMY Tap
11
Batch Size
100
Quast-Twi
Sparse
32
8
64
Quest-Tei
Quest
7
0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00
500
1000
1500
2000
2500
Time (μS)
Figure 9: PG-19 perplexity (accuracy) and sparse attention latency (efficiency) under different threshold p values.
Figure 10: Time breakdown of self-attention. batch size 64, Quest- -wis outperforms Quest by about 2x.
5.3 Ablation Study
Sensitivity to Threshold p. Notably, although we introduce the threshold p in order to get rid of the budget k, argue that p more reasonable and tunable hyperparameter. This is because represents the accumulated probability, which is less influenced by the different distributions that may occur for different heads/layers/queries In contrast, & is highly sensitive different distributions, as illustrated in Figure This allows us to simply tune for a fixed model, a way such as calibrating with small dataset.
For the impact p on model accuracy, we test the perplexity on the PG-19 dataset when using different thresholds p. For the impact runtime efficiency, the p value directly controls the pruning aggressiveness and affects the attention time via the pruned token number. We evaluate the sparse attention kernel speed after pruned on the TrívialQA dataset. As Figure 9 shows, the accuracy and efficiency strike balance at R 0.85, making us choose = 0.85 for Longchat-7B-v1.5-32.
Time Breakdown for Twilight. Given Twilight's hierarchical architecture, which comprises three distinct components, it is insightful analyze the exccution time breakdown further understand the benefit and cost. Figure 10 illustrates the time breakdown for different batch sizes in 32k retrieval task. In this scenario, Quest employs budget of 8192 (1/4 sparsity), while Twilight further prunes this budget down to 256. The breakdown aligns closely with the theoretical cost model presented in Section 4.3, demonstrating that Twilight significantly reduces the time required for the sparse attention kernel while introducing minor overheads.
Conclusion
In this paper, we first highlight that existing top-k sparse attention methods struggle to find optimal budgets due to the dynamic nature of attention weight distributions. We then introduce Twilight, framework with a hierarchical select-then- prumn architecture that leverages top-p sampling to address this issue. Twilight can adaptively prune up 98% tokens, resulting in 15.4x speedup for the self-attention operator and 3.9x reduction the end-to-end per-token latency. Comparing to the base sparse attention algorithm it is applied to, Twilight offers additional 1.4x speedup. Our work underscores the importance of adaptive attention sparsity, and paves a promising way for future research on sparse attention mechanisms.
Acknowledgment
The authors thank the anonymous reviewers for their valuable suggestions, Yilong Zhao for helping us on kernel optimization, and the Tsinghua IDEAL group members for constructive discussion. Mingyu Gao is the corresponding author.
10

---

References
[1] Y. Bai, X. Lv, J. Zhang. H. Lyu, J. Tang, Z. Huang. Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong. J. Tang, and J. Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3137, Bangkok, Thailand, Aug. 2024, Association for Computational Linguistics.
[2] Z. Cai, Y. Zhang, B. Gao, Y. Liu, T. Liu, K. Lu, W. Xiong. Y. Dong, B. Chang, J. Hu, and X. Wen. PyramidKV: V: Dynamic KV cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406. 02069, 2024.
[3] Z. Chen, R. Sadhukhan, Z. Ye, Y. Zhou, J. Zhang, N. Nolte, Y. Tian, M. Douze, L. Bottou,
Z. Jia, and B. Chen. MagicPIG: LSH sampling for efficient LLM generation. arXiv preprint
2XXi:2410.16117 2024.
[4] K. Cobbe. V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word
problems. arXiv preprint arXi::2110.14166 2021.
[5] T. Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[6] Dasigi, K. Lo, Beltagy, A. Cohan, N. A. Smith, and M. Gardner. A dataset of information- seeking questions and answers anchored in research papers. arXiv preprint 100.:0200 2021.
[7] DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.0444 04434, 2024.
[8] DeepSeek-AI, D. Guo, D. Yang. H. Zhang. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, Wang. X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji. E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang. H. Bao, H. Xu, H. Wang. H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, Wang. J. Chen, J. Yuan, Qiu, J. Li,
L. Cai, J. Ni, J. Liang. J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang. L. Zhang, L. Xu, L. Xia, M. Zhang. M. Zhang. M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang. Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang. R. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang. S. Yu,
S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang. X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. XU, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhann, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong,
Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li.
Y. Zheng, Y. Zhu, Y. Ma. Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu,
Z. Xie, Z. Zhang, Z. Hao, Z. Ma. Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song,
Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang. DeepSeek-RI: Incentivizing reasoning capability in LLMS via reinforcement learning. arXiv preprint 2501.12948, 2025.
[9] Y. Feng, J. Lv, Y. Cao, X. Xie, and S. K. Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. arXiv preprint 2407.
2025.
[10] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell,
N. Muennighoff, J. Phang, I.. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang. and A. Zou. A framework for fhw- language model evaluation. Version vO. O. Sept, 10:8-9, 9, 2021. [11] D. Guo, C. Xu, N. Duan, J. Yin, and J. McAuley. Longeoder: A long-range pre-trained language model for code completion. In International Conference on Machine Learning (ICML), pages 12098-12107. PMLR, 2023.
[12] A. Holtzman, J. Buys, I.. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv: 190..0994, 2019.
[13] C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, and
A. Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. arXiv preprint
2024.
[14] C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg. RULER: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0663 2024.
11

---

[15] L. Huang. S. Cao, N. Parulian, H. Ji, and L. Wang. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.022 02.112 2021.
[16] N. Jain, K. Han, A. Gu. W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and
I. Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2403.07974 2024.
[17] H. Kang, Q. Zhang, S. Kundu, G. Jeong, Z. Liu, T. Krishna, and T. Zhao. GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv preprint 2403.05527. 2024.
[18] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregres-
sive transformers with linear attention. arXiv preprint 2006.16238 16236, 2020.
[19] Y. J. Kim, R. Henry, R. Fahim, and H. H. Awadalla. Who says elephants can't 't run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211.10017 2022. [20] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, page 61 1-666, New York, NY, USA, 2023. Association for Computing Machinery.
[21] B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu. J. Hu, M. Tintore. S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, Reizenstein, and G. Sizov. xFormers: A modular and hackable transformer modelling library. https://gi
com/
facebookresearcch. htt//forrmi ormers, 2022.
[22] D. Li, R. Shao, A. Xie, Y. Sheng, L. Zheng, J. E. Gonzalez, I. Stoica, X. Ma, and H. Zhang. How long can open- soercs LLMs truly promise on context length?, June 2023.
[23] Y. Li. Y. Huang, B. Yang, B. Venkitesh, A. Locatelli, H. Ye. T. Cai, P. Lewis, and
D. Chen. SnapKV: LLM knows what you are looking for before generation. arXiv preprint wwXiv:2404.144669 2024.
[24) C. Lin, Z. Han, C. Zhang, Y. Yang. F. Yang, C. Chen, and L. Qiu. Parrot: Efficient serving of LLM-based applications with semantic variable. In 18th USENIX Symposium on Operal- ing Systems Design and Implementation (OSDI 24), Santa Clara, CA, July 2024. USENIX Association.
[25] Y. Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving, arXiv preprint arXiv:2405.04532 2024.
(26) D. Liu, M. Chen, B. Lu, H. Jiang, 7. Han, Q. Zhang, Q. Chen, C. Zhang, B. Ding, K. Zhang, C. Chen, F. Yang, Y. Yang, and L. Qiu. RetrievalAAttetiion: Accelerating long-context LLM inference via vector retrieval. arXiv preprint arXiv:2409. 10516. 2024.
[27] Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, and X. Hu. KIVI: tuning-free asymmetric 2bit quantization for KV cache. arXiv preprint arXiv:2402.0277
2024.
[28] Llama Team, Al @ Meta. The Llama 3 herd of models. arXiv preprint 1.Xii:2407.217880 2024. [29] E. Lu, Z. Jiang, J. Liu, Y. Du, T. Jiang, C. Hong, S. Liu, W. He, E. Yuan, Y. Wang, Z. Huang, H. Yuan, S. Xu, X. Xu, G. Lai, Y. Chen, H. Zheng, J. Yan, J. Su, Y. Wu, N. Y. Zhang, Z. Yang, X. Zhou, M. Zhang, and J. Qiu. MoBA: Mixture of block attention for long-context LLMs. arXiv preprini arXiv:2502.11 13189, 2025.
[30] P. Nawrot, A. Lancucki, M. Chochowski, D. Tarjan, and R. M. Ponti. Dynamic memory compression: Retrofitting LLMS for accelerated inference. arXiv preprint 2403.09636 2024.
[31] NVIDIA. Frsterrransformer: Providing a script and recipe to run the highly opti- mized transformer-based encoder and decoder component. https:
FasterTransf ormer, 2023.
[32] J. W. Rac, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: /.005507 2019.
[33] S. Reddy, D. Chen, and C. D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266., 2019.
[34] L. Ribar, I. Chelombiev, L. Hudlass-Galley, C. Blake, C. Luschi, and D. Orr. SparQ attention: Bandwidtt-efffiiient LLM inference. arXiv preprint arXiv:2312.0498 2023.
[35] H. Tang, Y. Lin, J. Lin, Q. Han, S. Hong. Y. Yao, and G. Wang. RazorAttention: Efficient KV cache compression through retrieval heads. arXiv preprint arXiv:2407.1589/ 2024.
[36] J. Tang, Y. Zhao, K. Zhu, G. Xiao, B. Kasikei, and S. Han. Quest: Query-aware sparsity for efficient long- long-context LLM inference. arXiv preprint 10774, 2024.
12

---

[37] G. Team, P. Georgiev. V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang. S. Mariooryad, Y. Ding. X. Geng, F. Alcober, R. Frostig. M. Omernick, L. Walker, C. Paduraru, C. Sorokin, A. Tacchetti, C. Gaffnny, S. Daruki, O Sercinoglu, 7. Gleicher, Love, Voigtlaender, R. Jain, G. Surita, K. Mohamed, R. Blevins, J. Ahn, T. Zhu, K. Kawintiranon, O. Firat, Y. Gu, Y. Zhang, M. Rahtz, M. Faruqui, N. Clay, J. Gilmer, J. Co-Reyes, I. Penchev, R. Zhu, N. Morioka, K. Hui, K. Haridasan, V. Campos, M. Mahdich, M. Guo, S. Hassan, K. Kilgour, A. Vezer, H.-T. Cheng. R. de Liedekerke, S. Goyal, P. Barham, D. Strouse, S. Noury, J. Adler, M. Sundararajan, S. Vikram, D. Lepikhin, M. Paganini, X. Garcia, F. Yang, D. Valter, M. Trebacz, K. Vodrahalli, C. Asawaroengchai, R. Ring, N. Kalb, L. B. Soares. S. Brahma, D. Steiner. T. Yu, F. Mentzer, A. He. L. Gonzalez, B. Xu, R. L. Kaufman, L. E. Shafey, J. Oh,
T. Hennigan. G. van den Driessche, S. Odoom, M. Lucic, B. Roelofs, S. Lall, A. Marathe, B. Chan, S. Ontanon, I.. He, D. Teplyashin, J. Lai, P. Crone, B. Damoc, I.. Ho, S. Riedel, K. Lenc, C.-K. Yeh, A. Chowdhery, Y. Xu, M. Kazemi, E. Amid, A. Petrushkina, K. Swersky, A. Khodaei, G. Chen, C. Larkin, M. Pinto, G. Yan, A. P. Badia, P. Patil, S. Hansen, D. Orr, S. M. R. Arood, J. Grimstad, A. Dai, S. Douglas, R. Sinha, V. Yadav, X. Chen, E. Gribovskaya, J. Austin, J. Zhao, K. Patel, P. Komarek, S. Austin, S. Borgeaud, L. Friso, A. Goyal, B. Caine, K. Cao, D.-W. Chung, M. Lamm, G. Barth-Maron, T. Kagohara, K. Olszewska, M. Chen, K. Shivakumar, R. Agarwal, H. Godhia, R. Rajwar, J. Snaider, X. Dotiwalla, Y. Liu, A. Barua, V. Ungureanu, Y. Zhang, B.-O. Batsaikhan, M. Wirth, J. Qin, Danihelka, T. Doshi, M. Chad- wick, J. Chen, S. Jain, Q. Le, A. Kar, M. Gurumurthy, C. Li, R. Sang. F. Liu, L. Lamprou, R. Munoz, N. Lintz, H. Mehta, H. Howard, M. Reynolds, L. Aroyo, Q. Wang, L. Blanco, A. Cassirer, J. Griffith, D). Das, S. Lee, J. Sygnowski, Z. Fisher, J. Besley, R. Powell, 7. Ahmed, D. Paulus, D. Reitter, Z. Borsos, R. Joshi, A. Pope. S. Hand, V. Selo, V. Jain, N. Sethi, M. Goel, T. Makino, R. May. Z. Yang. J. Schalkwyk, C. Butterfield, A. Hauth, A. Goldin, W. Hawkins, E. Senter, S. Brin, O. Woodman, M. Ritter, E. Noland, M. Giang, V. Bolina, L. Lee, Blyth. I. Mackinnon, M. Reid, O. Sarvana, D. Silver, A. Chen, L. Wang, L. Maggiore, O. Chang, N. Attaluri, G. Thornton, C.-C. Chiu, O. Bunyan, N. Levine, T. Chung, E. Eltyshev, X. Si, T. Lillicrap. D. Brady, V. Aggarwal, B. Wu, Y. Xu, R. Mcllroy, K. Badola, P. Sandhu, E. Moreira, W. Stokowiec, R. Hemsley, D. Li, A. Tudor, P. Shyam, E. Rahimtoroghi, S. Haykal, P. Sprech- mann, X. Zhou, D. Mineu, Y. Li, R. Addanki, K. Krishna. X. Wu, A. Frechette, M. Eyal, A. Dafoe, D. Lacey, J. Whang, T. Avrahami,, Y. Zhang, E. Taropa, H. Lin, D. Toyama, E. Ruther- ford, Sano, H. Choe, A. Tomala, C. Safranek-Shrader, N. Kassner, M. Pajarskas, M. Harvey, S. Sechrist, M. Fortunato, C. Lyu, G. Elsayed, C. Kuang, J. Lottes, E. Chu, C. Jia, C.WW. Chen, P. Humphreys, K. Baumli, C. Tao, R. Samuel, C. N. dos Santos, A. Andreassen, N. Rakićević, D. Grewe, A. Kumar, S. Winkler, J. Caton, A. Brock, S. Dalmia, H. Sheahan, I. Barr, Y. Miao, Natsev, J. Devlin, F. Behbahani, F. Prost, Y. Sun, A. Myaskovsky, T. S. Pillai, D. Hurt, A. Lazaridou, X. Xiong, C. Zheng, F. Pardo, X. Li, D. Horgan, J. Stanton, M. Ambar, F. Xia, A. Lince, M. Wang, B. Mustafa, A. Webson, H. Lee, R. Anil, M. Wicke, T. Dozat, A. Sinha, E. Piqueras, E. Dabir, S. Upadhyay, A. Boral, L. A. Hendricks, C. Fry, J. Djolonga, Y. Su, J. Walker, J. Labanowski, R. Huang. V. Misra, J. Chen, R. Skerry-Ryan, A. Singh, S. Rijhwani, D. Yu, A. Castro- Ros, B. Changpinyo, R. Datta, S. Bagri, A. M. Hrafnkelsson, M. Maggioni, D. Zheng, Y. Sulsky, S. Hou, T. L. Paine, A. Yang. J. Riesa, D. Rogozinska, D. Mareus, D. E. Badawy, Q. Zhang. L. Wang, H. Miller, Greer, L. L. Sjos, A. Nova, H. Zen, R. Chaabouni, M. Rosca, J. Jiang, C. Chen, R. Liu, T. Sainath, M. Krikun, A. Polozov, J. Lespiau, Newlan, Z. Cankara, S. Kwak, Y. Xu, P. Chen, A. Coenen, C. Meyer, K. Tsihlas A. Ma, J. Gottweis, J. Xing. C. Gu, J. Miao, C. Frank, Z. Cankara, S. Ganapathy, I. Dasgupta, S. Hughes-Fitt, H. Chen, D. Reid, K. Rong. H. Fan, J. van Amersfoort, V. Zhuang, A. Cohen, S. S. Gu, A. Mo- hananey, A. Ilie, T. Tobin, J. Wieting, A. Bortsova, P. Thacker, E. Wang, E. Caveness, J. Chiu, E. Sezener, A. Kaskasoli, S. Baker, K. Millican, M. Elhawaty, K. Aisopos. C. Lebsack, N. Byrd, H. Dai, W. Jia, M. Wiethoff, E. Davoodi, A. Weston, L. Yagati, A. Ahuja, I. Gao, G. Pundak, S. Zhang, M. Azzam, K. C. Sim, S. Caelles, J. Keeling, A. Sharma, A. Swing. Y. Li, C. Liu, C. G. Bostock, Y. Bansal, Z. Nado, A. Anand, J. Lipschultz, A. Karmarkar, L. Proleev, A. Itty- cheriah, S. H. Yeganeh, G. Polovets, A. Faust, J. Sun, A. Rrustemi, P. Li, R. Shivanna, J. Liu, C. Welty, F. Lebron, A. Baddepudi, S. Krause, E. Parisotto, R. Soricut, Z. Xu, D. Bloxwich, M. Johnson, B. Neyshabur, J. Mao-Jones, R. Wang, V. Ramasesh, Z. Abbas, A. Guez, C. Segal, D. D. Nguyen, J. Svensson, L. Hou, S. York, K. Milan, S. Bridgers, W. Gworek, M. Tagliasac- chi, J. Lee-Thorp. M. Chang, A. Guseynov, A. J. Hartman, M. Kwong, R. Zhao, S. Kashem, E. Cole, A. Miech, R. Tanburn, M. Phuong, F. Pavetic, S. Cevey, R. Comanescu, R. Ives, S. Yang, C. Du, B. Li, Z. Zhang, M. linuma, C. H. Hu, A. Roy, S. Bijwadia, Z. Zhu, D. Martins,
13

---

R. Suputro, A. Gergely, S. Zheng, D. Jia, I. Antonoglou, A. Sadovsky, S. Gu, Y. Bi, A. Andreev,
S. Samangooei. M. Khan, T. Kocisky, A. Filos, C. Kumar, C. Bishop, A. Yu, S. Hodkinson, S. Mittal, P. Shah, A. Moufarek, Y. Cheng, A. Bloniarz, Lee, P. Pejman, P. Michel, S. Spencer, V. Feinberg, X. Xiong, N. Savinov, C. Smith, S. Shakeri, D. Tran, M. Chesus, B. Bohnet,
G. Tucker, T. von Glehn, C. Muir, Y. Mao, H. Kazawa, A. Slone, K. Soparkar, D. Shrivastava, J. Cobon-Kerr, M. Sharman, J. Pavagadhi, C. Araya, K. Misiunas, N. Ghelani, M. Laskin,
D. Barker, Q. Li, A. Briukhov, N. Houlsby, M. Glaese, B. Lakshminarayanan, N. Schucher, Y. Tang, E. Collins, H. Lim, F. Feng, A. Recasens, G. Lai, A. Magni, N. D. Cao, A. Siddhant, Z. Ashwood, J. Orbay, M. Dehghani, J. Bremnan, Y. He, K. Xu, Y. Gao. C. Saroufim, J. Molloy, X. Wu, S. Amold, S. Chang, J. Schrittwieser, E. Buchatskaya, S. Radpour, M. Polacek, S. Gior- dano, A. Bapna, S. Tokumine, V. Hellendoom,, T. Sottiaux, S. Cogan, A. Severyn, M. Saleh, S. Thakoor, I.. Shefey, S. Qiao, M. Gaba, S. yiin Chang, C. Swanson, B. Zhang. B. Lee, P. K. Rubenstein, G. Song. T. Kwiatkowski, A. Koop, A. Kannan, D. Kao, P. Schuh, A. Stjerngren, G. Ghiasi G. Gibson, L. Vilnis, Y. Yuan, F. T. Ferreira, A. Kamath, T. Klimenko, K. Franko, K. Xiao, I. Bhattacharya, M. Patel, R. Wang. A. Morris, R. Strudel, V. Sharma, P. Choy, S. H. Hashemi, J. Landon, M. Finkelstein, P. Jhakra, J. Frye, M. Barnes, M. Mauger, D. Daun, K. Baatarsukh, M. Tung, W. Farhan, H. Michalewski, F. Viola, F. de Chaumont Quitry, C. L. Lan, T. Hudson, Q. Wang. F. Fischer, I. Zheng, E. White, A. Dragan, J. baptiste Alayrac, E. Ni, A. Pritzel, A. Iwanicki, M. Isard, A. Bulanova, L. Zilka,
Dyer, D. Sachan, S. Srinivasan, H. Muckenhirn. H. Cai, A. Mandhane, M. Tariq, J. W. Rae, G. Wang, K. Ayoub, N. FitzGer- ald, Y. Zhao, W. Han, C. Alberti,
Garrette, K. Krishnakumar, M. Gimenez, A. Levskaya, D. Sohn, J. Matak, I. Iturrate, M. B. Chang, J. Xiang, Y. Cao. N. Ranka, G. Brown, A. Hut- ter, V. Mirrokni, N. Chen, K. Yao, Z. Egyed, F. Galilee, T. Liechty, P. Kallakuri, S. Ghemawat, J. Liu, D. Tao, C. Thornton, T. Green, M. Jasarevic, S. Lin, V. Cotruta, Y.-X. Tan, N. Fiedel, H. Yu, E. Chi, A. Neitz, J. Heitkaemper, A. Sinha, D. Zhou, Y. Sun, C. Kaed, B. Hulse, S. Mishra, M. Georgaki, S. Kudugunta, C. Farabet, I. Shafran, D. Vlasic, A. Tsit- sulin, R. Ananthanarayanan, A. Carin, G. Su, P. Sun, S. V, G. Carvajal, J. Broder, I. Comsa, A. Repina, W. Wong, W. W. Chen, P. Hawkins, E. Filonov, L. Loher, C. Hirnschall, W. Wang, J. Ye, A. Burns, H. Cate, D. G. Wright, F. Piccinini. L. Zhang, C.-C. Lin, I. Gog, Y. Kulizh- skaya, A. Sreevatsa, S. Song, L. C. Cobo. A. Iyer, C. Tekur, G. Garrido, Z. Xiao, R. Kemp, H. S. Zheng, H. Li, A. Agarwal, C. Ngani, K. Goshvadi, R. Sittmmaria-Fernadderr W. Fica, X. Chen, C. Gorgolewski, S. Sun, R. Garg, X. Ye, S. M. A. Eslami, N. Hua, J. Simon, P. Joshi, Y. Kim, I. Tenenny S. Potluri, L. N. Thiet, Q. Yuan, F. Luisier, A. Chronopoulou, S. Scellato, P. Srinivasan, M. Chen, V. Koverkathu, V. Dalibard, Y. Xu, B. Saeta, K. Anderson, T. Sellam, N. Fernando, F. Huot, J. Jung. M. Varadarajan, M. Quinn, A. Raul, M. Le, R. Habalov, J. Clark, K. Jalan, K. Bullard, A. Singhal, T. Luong, B. Wang, S. Rajayogam, J. Eisenschlos, J. Jia, D. Finchelstein, A. Yakubovich, D. Balle, M. Fink, S. Agarwal, J. Li, D. Dvijotham, S. Pal, K. Kang, J. Konzelmann, J. Beattie, O. Dousse, D. Wu, R. Crocker, C. Elkind, S. R. Jonnala- gadda. J. Lee, D. Holtmann Biltm, K. Kallarackal, R. Liu, D. Vnukov, N. Vats, L. Invernizzi. M. Jafari, H. Zhou, L. Taylor, J. Prendki, M. Wu, T. Eccles, T. Liu, K. Kopparapu, F. Beaufays. C. Angermueller, A. Marzoca, S. Sarcar, H. Dib, J. Stanway, F. Perbet, N. Trdin, R. Sterneck, A. Khorlin, D. Li, X. Wu, S. Goenka, D. Madras, S. Goldshtein, W. Gierke, T. Zhou, Y. Liu, Y. Liang, A. White, Y. Li, S. Singh, S. Bahargam, M. Epstein, S. Basu, L. Lao, A. Ozturel, C. Crous, A. Zhai, H. Lu, Z. Tung, N. Gaur, A. Walton, L. Dixon, M. Zhang. A. Globerson, G. Uy. A. Bolt O. Wiles, M. Nasr, I. Shumailov, M. Selvi, F. Piccinno. R. Aguilar. S. McCarthy, M. Khalman, M. Shukla, V. Galic, J. Carpenter, K. Villela, H. Zhang. H. Richardson, J. Martens, M. Bosnjak, S. R. Belle, J. Seibert, M. Alnahlawi, B. McWilliams, S. Singh, A. Louis, W. Ding, D. Popovici, L. Simicich, L. Knight, P. Mehta, N. Gupta, C. Shi, S. Fatehi, Mitrovic, A. Grills, J. Pagadora, T. Munkhdalai, D. Petrova, D. Eisenbud, Z. Zhang, D. Yates, B. Mittal, N. Tripura- neni, Y. Assael, T. Brovelli, P. Jain, M. Velimirovic, C. Akbulut, J. Mu, W. Macherey, R. Kumar, J. Xu, H. Qureshi, G. Comanici, J. Wiesner, Z. Gong. A. Ruddock, M. Bauer, N. Felt, A. GP, A. Arnab, D. Zelle, J. Rothfuss, B. Rosgen, A. Shenoy, B. Seybold, X. Li, J. Mudigonda, G. Erdogan, J. Xia, J. Simsa, A. Michi, Y. Yao, C. Yew, S. Kan, I. Caswell, C. Radebaugh, A. Elisseeff, P. Valenzuela, K. McKinney, K. Paterson, A. Cui, E. Latorre-Chimotee S. Kim, W. Zeng, K. Durden, P. Ponnapalli, T. Sosea, C. A. Choquette-Choo, J. Manyika, B. Robenek, H. Vashisht, S. Pereira, H. Lam, M. Velic, D. Owusu-Afriyie, K. Lee, T. Bolukbasi, A. Parrish, S. Lu, J. Park, B. Venkatraman, A. Talbert, L. Rosique, Y. Cheng, A. Sozanschi, A. Paszke, P. Kumar, J. Austin, L. Li, K. Salama, B. Perz, W. Kim, N. Dukkipati, A. Baryshnikov, C. Ka- planis, X. Sheng, Y. Chervonyi, C. Unlu, D. de Las Casas, H. Askham, K. Tunyasuvunakool,
Palmer,
14

---

F. Gimeno, S. Poder, C. Kwak, M. Miecnikowski, V. Mirrokni, A. Dimitriev, A. Parisi, D. Liu,
T. Tsai, T. Shevlane, C. Kouridi, D. Garmon, A. Goedeckemeyer, A. R. Brown, A. Vijayakumar,
A. Elqursh, S. Jazayeri, J. Huang, S. M. Carthy, J. Hoover, I.. Kim, S. Kumar, W. Chen, C. Biles, G. Bingham, F. Rosen, L.. Wang. Q. Tan, D. Engel, F. Pongetti, D. de Cesare, D. Hwang. I.. Yu,
J. Pullman, S. Narayanan, K. Levin, S. Gopal, M. Li, A. Aharoni, T. Trinh, J. Lo, N. Casagrande,
R. Vij. L. Matthey, B. Ramadhana, A. Matthews, C. Carey, M. Johnson, K. Goranova, R. Shah, S. Ashraf, K. Dasgupta, R. Larsen, Y. Wang, M. R. Vuyyuru, C. Jiang. J. Ijazi, K. Osawa, C. Smith, R. S. Boppana, T. Bilal, Y. Koizumi, Y. Xu, Y. Altun, N. Shabat, B. Bariach, A. Ko- rehemniy, K. Choo. O. Ronneberger, C. Iwuanyanwu, S. Zhao, D. Soergel, C.-J. Hsich, I. Cai, S. Iqbal, M. Sundermeyer, Z. Chen, E. Bursztein, C. Malaviya, F. Biadsy, P. Shroff, I. Dhillon, T. Latkar, C. Dyer, H. Forbes, M. Nicosia, V. Nikolaev, S. Greene, M. Georgiev, P. Wang, N. Martin, H. Sedghi, J. Zhang, P. Banzal, D. Fritz, V. Rao, X. Wang. J. Zhang. V. Patraucean, D. Du, I. Mordatch, I. Jurin, I.. Liu, A. Dubey, A. Mohan, J. Nowakowski, V.-D. Ion, N. Wei, R. Tojo, M. A. Raad, D. A. Hudson, V. Keshava, S. Agrawal, K. Ramirez, Z. Wu, H. Nguyen, J. Liu, M. Sewak, B. Petrini, D. Choi, I. Philips,, Z. Wang. I. Bica, A. Garg, J. Wilkiewicz, P. Agrawal, X. Li, D. Guo, E. Xue, N. Shaik, A. Leach, S. M. Khan, J. Wiesinger, S. Jerome, A. Chakladar, A. W. Wang. T. Ornduff, F. Abu, A. Ghaffarkhah, M. Wainwright, M. Cortes, F. Liu, J. Maynez, A. Terzis, P. Samangouei, R. Mansour, T. Kepa, F.-X. Aubet, A. Algymr, D. Banica, A. Weisz, A. Orban, A. Senges, E. Andrejezuk, M. Geller, N. D. Santo, V. Anklin, M. A. Merey, M. Baeuml, T. Strohman, J. Bai. S. Petrov, Y. Wu, D. Hassabis, K. Kavukcuoglu, J. Dean, and O. Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 05530, 2024.
[38] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, C. Tang,
C. Wang, D. Zhang, E. Yuan, E. Lu, F. Tang, F. Sung, G. Wei, G. Lai, H. Guo, H. Zhu, H. Ding, H. Hu, H. Yang. H. Zhang, H. Yao, H. Zhao, H. Lu, H. Li, H. Yu, H. Gao, H. Zheng, H. Yuan, J. Chen, J. Guo, J. Su, J. Wang. J. Zhao, J. Zhang, J. Liu, J. Yan, J. Wu, L. Shi, L. Ye, L. Yu, M. Dong, N. Zhang, N. Ma, Q. Pan. Q. Gong, S. Liu, S. Ma, S. Wei, S. Cao, S. Huang. T. Jiang, W. Gao, W. Xiong. W. He, W. Huang, W. Wu, W. He, X. Wei, X. Jia, X. Wu, X. Xu, X. Zu,
X. Zhou, X. Pan, Y. Charles, Y. Li, Y. Hu, Y. Liu, Y. Chen, Y. Wang, Y. Liu, Y. Qin, Y. Liu, Y. Yang, Y. Bao, Y. Du, Y. Wu, Y. Wang, Z. Zhou, Z. Wang. Z. Li, Z. Zhu, Z. Zhang, Z. Wang.
Z. Yang. Z. Huang. Z. Huang, Z. Xu, and Z. Yang. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint 2501. 2025.
[39] P. Tillet, H.-T. Kung, and D. Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Langugges, pages 10-19, 2019.
[40] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[41] P. Wang, S. Bai, S. Tan, S. Wang. Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and Lin. Qwen2-VL: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint 2409.1219, 2024.
[42] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint 2006.04768, 2020.
[43] H. Wu, L. Li, H. Huang, Y. Tu, J. Zhang, M. Yu, and J. Yan. HShare: Fast LLM decoding by hierarchical key-value sharing. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[44] W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu. Retrieval head mechanistically explains long-context factuality. arXiv preprint 2404. 15574, 2024.
[45] G. Xiao, J. Tang, J. Zuo, J. Guo, S. Yang, H. Tang, Y. Fu, and S. Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. arXiv preprint arXiv:2410. 10819, 2024.
[46] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
[47] A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, J. Jiang, J. Tu, J. Zhang, J. Zhou, J. Lin, K. Dang, K. Yang, L. Yu, M. Li, M. Sun, Q. Zhu, R. Men, T. He, W. Xu, W. Yin, W. Yu, X. Qiu,
X. Ren, X. Yang, Y. Li, Z. Xu, and Zhang. Qwen2.5-1M technical report. arXiv preprint
2501.1538.. 2025.
15

---

[48] D. Yang, X. Han, Y. Gao, Y. Hu, S. Zhang, and H. Zhao. PyramidInfer: Pyramid KV cache compression for nightthroughuu LLM inference. arXiv preprint arXiv:2405. 12532. 2024. [49] L. Yang, Zhang, Z. Chen, Z. Li, and Z. Jia. TedalDecode: Fast and accurate LLM decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076, 2024.
[50] S. Yang, Y. Sheng. J. E. Gonzalez, Stoica, and L. Zheng. Post-training sparse attention with double sparsity, arXiv preprint ar arXiv:2408. 07099, 2024.
[51] I.. Ye, Z. Tao, Y. Huang, and Y. Li. Chunk Efficient self-attention with prefix-aware
KV cache and two-phase partition. arXiv preprint arXiv:2402. 15220, 2024.
[52] Z. Ye, L. Chen, R. Lai, W. Lin, Y. Zhang, S. Wang. Chen, B. Kasikci, V. Grover, A. Krishna-
murthy, and L. Ceze. FlashInfer: Efficient and customizable attention engine for LLM inference
serving. arXiv preprint arXlv:2501.01005, 2025.
[53] Z. Ye, R. Lai, B.-R. Lu, C.-Y. Lin, S. Zheng, L.. Chen, T. Chen, and L.. Ceze. Cascade inference:
Memory bandwidth efficient shared prefix batch decoding, February 2024.
[54] J. Yuan, H. Gao, D. Dai, J. Luo, L.. Zhao, Z. Zhang, Z. Xie, Y. Wei, L. Wang, Z. Xiao, Y. Wang,
C. Ruan, M. Zhang, W. Liang, and W. Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502./1088
2025.
[55] H. Zhang, X. Ji, Y. Chen, F. Fu, X. Miao, X. Nie, W. Chen, and B. Cui. PQCache: Product quantization- based KVcache for long context LLM inference. Proceedings of the ACM on Management of Data, 3-3):-330, 2025.
[56] Zhang. H. Huang, P. Zhang. J. Wei, Zhu, and Chen. SageAttention2:
Efficient attention with thorough outlier smoothing and per-thread INT4 quantization. arXiv preprint 24/1.10958, 2024.
[57] Zhang, Wei, Zhang. J. Chen, and J. Zhu. SageAttention: Accurate 8-bit attention for mug-and-play inference acceleration. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[58] Z. Zhang, Y. Sheng, T. Zhou, T. Chen, L. Zheng, R. Cai, Z. Song, Y. Tian, C. Re, C. Barrett,
Z. Wang, and B. Chen. H2O: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems (NeurIPS), 36:34661-
34710, 2023.
[59] Y. Zhao, C.-Y. Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and
B. Kasikci. Atom: Low-bit quantization for efficient and accurate LLM serving. In Proceedings
of Machine Learning and Systems (MLSys), volume 6, pages 196-209, 2024.
[60] L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, C. Barrett, and Y. Sheng. Efficiently programming large language models using SGLang. arXiv preprint 2312.07104, 2023.
[61] X. Zhou, W. Wang, M. Zeng, J. Guo, X. Liu, Shen, M. Zhang, and I.. Ding. DynamicKV: Task- aware adaptive KV cache compression for long context LLMs. arXiv preprint 4rXiv:2412.14888 2024.
[62] K. Zhu, T. Tang, Q. Xu, Y. Gu, Z. Zeng, R. Kadekodi, L. Zhao, A. Li, A. Krishnamurthy, and B. Kasikci. Tactic: Adaptive sparse attention with clustering and distribution fitting for long-context LLMs. arXiv preprint arXiv:2502. 12216, 2025.
[63] I.. Zhu, X. Wang, W. Zhang, and R. W. H. Lau. AtlayAAtention for efficient large language model serving with long system prompts. arviv preprint 2402. 14808, 2024.
[64] Q Zhu, J. Duan, C. Chen, S. Liu, G. Feng, X. Lv, X. Chuanfu, D. Lin, and C. Yang. Sam- pleAttention: Near-lossless acceleration of long context LLM inference with adaptive structured sparse attention, arXiv preprint arXiv:2406. 15486, 2024.
16

---

A Budget Dynamism at Different Levels
As introduced in Section 1, various levels of budget dynamism exist. We propose and analyze four distinct levels of KV cache budget dynamism, as illustrated Figure 11. They are prompt-wise, query-wise, layer-wise, and head-wise dynamism.
Prompt 158/10739
Prompt 1 408/17501 =2.33%
Prompt 244/5441
Prompt 3 195/2023 =9.64%
Dynamism Across Prompts (Budget/Pronpt Len)
=1.4%%
=4.48%
Query 0 120
Query 152
Query 2 174
Query 185
Dynamism Across Queries (Budget)
Layer 27
Layer 10 356
Layer 15 105
Layer 26 228
Dynamism Across Layers (Budget)
Head 0
Head 2707
Head 15
Head 23
Dynamism Across Heads (Budget)
37
886
144
Figure 11: Budget dynamism observed in oracle top-p attention. We observe the dynamism across four dimensions: different prompts (tasks), different queries within the same prompt, different layers in the same query, and different heads in the same layer.
Table 5: Full results on Longbench. The highest score in each task (except for "Full") is marked in bold. The average budget after Twilight's pruning shown following the method name. We also report the relative error changes (improvement or degradation) when integrating Twilight with each base algorithm.
Budget
Single-Due. Q4
Multi-Dec: Q4
Summarization
Synthetic
Avg,
Quger
HoquetAA
Musique
Trmmag
Lugglui-76-r1.530
Full
Twillght 146)
32%
31,74
33.59
25.65
13.93
23.15
26.30
34.50
54.98
38.52
256
26.00
23.23
22.64
20.98
25.05
33,60
48.70
31,26
1024
31.63
02.36
30.47
29.96
22.70
26.39
$4.21
34.5
51.52
53.95
36,85
Quest
4056
29.77
62.71
32.94
23.96
1324
31.56
22,86
26.45
$4.37
31.50
$3.17
55.52
37.33
$192
20.34
41.70
33.27
23.46
13.51
31.18
23.02
26.48
$4.70
30,00
$3.00
55.57
37.30
Twillight (Aag. 131)
31.95
43.25
31.02
26.57
13.48
32.21
22.79
26.33
$4.93
33.50
$4.56
56.70
38.04 (62.5%)
256
25.25
39.75
27.10
20.75
0.34
29.68
21.79
25.59
$3.97
32.00
$3.01
$3.44
35.32
1034
50.55
61.27
50.85
21:87
7.37
26.82
22.05
36.51
83.22
31 1.550
$3.23
$5,50
06
DX
4006
25.95
41.90
$0.53
23.65
8.00
20
33.75
26.58
$3.34
30.00
50.77
$5.48
36.
#102
20105
41.43
51.79
22.05
12.50
30.44
22.50
36:43
$3.63
30.50
$3.67
$5.31
36.62
Twilight (Aug. 126)
32.34
43.80
34.67
25.43
13354
31.88
23.01
26.32
$5.29
35.50
55.03
57,27
38.71 (45.7%)
LLMMA 3.1 SB Instrcct
Pull
Twilight (Ang. 478)
125k
66.17
53.33
55.36
43.95
27.08
35.01
25.24
27.37
91.18
54,50
64.06
62.17
$7.76
$2,01
(40266)
MagiaPECI
K=8,
45.03
91
50.07
1024
Quest
Twilight
256
43.58
10244
26.71
DS
4056
43.50
53.17
$4.21
46.70
23.14
34.75
25.40
26.91
92.78
99.50
62.59
50,96
$192
41.82
53.71
$4.10
45.15
23.72
34.27
24.98
26. 69
91.61
100.00
62.40
$2.87
$1.14
Twilight 446)
43.08
$3.89
54.68
44.56
24.88
34.09
75.20
27.00
91.30
100.00
63.95
$8.93
$1.73 (+1.286)

---

B Kernel Implementation Details
B.1 Implementation of Mixed-Precision SpGEMV
Calculation Process. As outlined in Section 4.2, our implementation requires GEMV kernel that computes the product of an FP16 query vector and an INT4 quantized key matrix (qfp16 1 6iKint4) with paged indexing. We adapt the attention de-
Batch Size
1000
INT4
INT8
Latency
FP16
600
coding kernel from FlashInfer [52] for this pur- pose. The kernel execution follows two main steps: (1) asynchronously loading and dequantiz- ing the quantized K cache from the global mem- ory into the shared memory using cp async; and (2) computing the dot product. To mitigate long memory latency, we employ two-stage pipeline that overlaps the data loading of a subsequent block with the computation of the current block. We use FP16 to store the dequantized K cache instead of FP32 to optimize the computation given that such accuracy tradeoff is tolerable as a score estimator.
400
200
256
512
1024
2048
4096 8192 16384 32768
Sequence Length
Figure 12: SpGEMV operator lateney with differ- ent quantization bits.
Dequantization. Following the design of QServe [25], we employ per-head, dynamic KV quantiza- tion and store the FP16 scale and zero for each head using the same paged memory layout as the K cache. The K matrix dequantized on-the-fly using per-head quantization parameters (scale and zero). Our dequantization routine builds upon the fast algorithm from [19] (as implemented in NVIDIA's FasterTransfonnn [3I]), which utilizes custom PTX assembly instructions for efficient type conversion between INT4 and FP16.
Bit-packing. INT4 K elements are packed into an uint8_t t buffer, with two 4-bit elements stored within each 8-bit byte - this aligns with the byte-addressable nature of C++. To simplify the dequantization logic, we first add an offset of +128 to each INT4 element, converting value, before packing them in interleaved manner. Address calculation for this packed buffer is remapped to stride a 4-bit granularity; this is achieved by halving the effective byte offset [59].
an unsigned
We conduct an ablation study on the impact of quantization bits on the efficiency as Figure 12 shows. Our optimizations on the dequantization and dot product make the operator memory-bound and thus benefit from quantization.
B.2 Tackling Group Query Attention
Spli Query (aum bradeegnupp sinc eg. 523mm)
400
hends-32
Split Query (num, beads-1)
heade-32
300
Flatened Cache beads-1)
Hattened KV Cache beads-1)
200
K"
100
64
Batchh
Hoce Attuntion wi Haadowisc Dynamiim
Query Ommpouise Dysomiem
Methods
Figure 13: Left: Head- hasnddwws/ggruuwwiii varlen attention with flattend paged KV cache in Twilight. Right: Comparison among the three attention methods on a real budget distribution of a LLaMA-3, 7B layer on 16k retrieval task. Here "Padded" means padding all heads to the maximum budget length; "Head Varlen" loads KV at the head granularity which causes repeated loading; and "Group Varlen" strikes balance between the methods.
Group Query Attention (GQA) [28], a technique widely adopted recent model architectures like LLaMA 3, maps group of query heads to single key-value head. This structure, however, inherently incompatible with query-aware sparse attention. The incompatibility arises because query-award sparse attention relies on individual query vectors to identify important tokens, but
18

---

GQA creates a mismatch at the granularity of attention heads. A brute-force solution would be to load tokens independently for each query head, but this leads inefficient, repeated memory reads. Twilight addresses this issue by operating at the granularity of query groups. Specifically, the set of tokens selected for given query group is the union of tokens identified by all query heads within that group [54].
As discussed in Section 4.2, our top-p attention mechanism natively supports head-wise dynamism. However, when integrated with GQA, this head-wise dynamism inherently transitions to group-wise dynamism, meaning that all heads within the same group share common token budget. Figure 13 shows our attention design with flattened paged KV cache, which supports head-wise varlen attention for MHA and group-wise varlen attention for GQA. This design represents a deliberate trade-off, balancing implementation efficiency with compatibility for modern attention algorithms. We also compare the efficiency of the three different attention implementations Figure 13.
C Full Results on Longbench
Please refer to Table 5.
D Accuracy Comparison with Token Dropping Methods
As discussed in Section 2, top-k sparse attention methods can be broadly categorized into two types: token dropping and token selecting. Prior research [36] has established that token selecting generally outperforms token dropping, the latter inevitably incurs irreversible information loss. To further validate this observation, we conduct comparative experiments between Twilight and two representative token-dropping methods: StreamingLLM [46] and SnapKV [23]. As demonstrated in Table 6, Twillghttt achieves notably better performance over both bascline methods.
Table 6: Comparison of StreamingLLMM SnapK and Twilight on the Longbench benchmark with the Longchat-7B-v1.5-322 model.
Dataset StreamingLLM (Budget=4096)
SnapKV (Budget=4096)
DS-Twilight
Qasper
26.39
29.44
32.34
MulQA-en
33.2
40.03
43.89
HotpotQA
24.29
33.67
34.67
2WikiMQA
20.1
24.13
25.43
Musique
10,87
13.45
13.84
GovReport
26.92
26.09
31.88
QMSum
20.8
22.53
23.01
MultiNews
26.46
25.61
26.32
TrivialQA
75.6
80.82
85.29
PR-en
24.17
30,25
35.50
LCC
52,47
52.62
55,03
Repobench-P
51.02
55.99
57.27
Avg.
32.69
36.22
38.71
E
Efficiency Evaluation in Offloading Scenarios
Notably, in memory offloodiin scenarios where the per-token loading cost dominates, Twilight could achieve more significant gains. This because Twilight reduces the number of loaded tokens with fixed estimation cost. Table 7 shows Twilight could achieve up to 16x speedups compared to Quest.
Table 7: Latency (in microseconds) of single attention operator in offloading scenarios, where corresponding tokens in the KV cache are loaded from the CPU memory.
10k
20k
30k
Quest
3038.98
5990.75
8490.95
Quest-Twi
415.89
480.61
527.77
19

---

Limitations and Future Work
While Twilight effectively accelerates existing top-k sparse attention methods, our analysis in Figure 10 reveals non-negligible estimation overheads. This makes Twilight particularly advantageous scenarios like serving with large batch sizes or offloading, where the cost of loading tokens from the KV cache dominates. Section B.2 shows head-wise dynamism is unfriendly with GQA, which leads to some challenges to integrate Twilight with new model architectures. Future research could focus on optimizing the estimation method to further improve the end-to-end latency and throughput, and how to integrate Twilight with other model architectures like multi-head latent attention (MLA) [7].
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstraction and introduction are empirically verified by Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix F. We also discuss the extra overhead (both time overhead and memory overhead) introduced by our method Section 4.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: The only theoretical result in this paper is Equation 2, which is proven by a textbook conclusion (the sub- sub-multiplicative property of the Frobenius norm). This part can be found in Section 3.1.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5 includes all experiments setup and configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the algorithms and methods are introduced in Section 4. The experiment setup is described in Section 5. Our code and scripts are released at https: / titb.. fom/tsinghue ideal/Twilghtt.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ Seriides/CodeSubiise
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. cctpps.ccppubbccccuucucc gww.///w/eee/eeee//eeeccee..e Ctte//Cooeeuiiiiic wwwssssiiccpiyiyo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments measure the benchmark scores and execution latencies, which are all highly deterministic and reproducible. Thus we do not include error bars in the figures. We report the improvements in the figures (both accuracy improvements and speedups), which statistically significant in problems. See Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The experiments are conducted on common NVIDIA GPUs, which are de-
scribed in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurlPS Code of Ethics https ://p:////////r/ http///eeurrp..eeee cc cc/ppubicc/thiicssuuelllll..
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This paper introduces method to accelerate LLMs, so it does not have direct societal impacts. However, the use LLMSs may have some social impacts, which can be amplified by this technique whether it is positive negative.

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release new models, datasets, or applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper uses open-source datasets and models in its evaluation (Section 5). The usage respects all the original licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddataees has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core part (method, algorithm) of this paper does not involve L.M usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettta://neuriann wwt.c//oeeeeenne//22222222.2222 for what should or should not be described.
27


Reviews:
[{"review_id": "PzvaXRDlPw", "reviewer": "Reviewer_sLWS", "summary": "The key idea of this paper is crystal clear: top-p is more effective than top-k for sparsity token selection. Although the key idea is quite simple, implementing top-p in state-of-the-art inference engines is not easy. The paper introduces a new method to select and prune tokens during the inference process and provides an efficient kernel implementation. The paper is well written and easy to follow.", "questions": "I expect more detailed in-depth analysis of the selected token distribution between top-p and top-k in experimental section.  \n\nIt would be better to have more discussion on how to integrate it to existed frameworks other than Quest and FlashInfer.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The idea of this paper is simple, but unexpectedly effective. It also seems to be easily adopted by mainstream inference engines as well as most sparsity approaches.\n\nThe new method introduces a small amount of extra overhead during the selection and pruning phase. However, it is unclear whether this trade-off is worthwhile compared to the gains it provides (though 1.4x is significant in some cases).", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and insightful advice. Below, we address each concern and outline planned revisions to strengthen the paper.\n\n**1. Benefit-Overhead Tradeoff during Selection and Pruning**\n\nWe appreciate the reviewer for mentioning the tradeoff between the extra overhead of Twilight and its benefits it brings. In Section 4.3, we discuss the overhead of Twilight and empirically analyze the time breakdown of Twilight’s attention operator in Figure 7, where the visualization makes this tradeoff particularly clear.\n\n**2. Analysis of Selected Token Distribution between Top-p and Top-k**\n\nWe thank the reviewer for their insightful suggestion. In response, we will augment the experiment section with an in-depth analysis of token selection distributions (like Figure 3 in our paper, but more detailed) under both top-p and top-k sampling. This analysis will elucidate why top-p consistently achieves superior performance despite selecting fewer tokens, providing readers with a clearer understanding of its efficiency advantages.\n\n**3. Integration on Other Frameworks**\n\nWe fully acknowledge the importance of system integration, especially given the practical focus of our method and its substantial system-level contributions. To clarify, Twilight's integration operates at multiple levels, which we categorize as follows:\n\n- **Kernel Library (FlashInfer)**. Currently Twilight leverages FlashInfer for high-performance implementations of key LLM operators (e.g., attention, FFN). While FlashInfer was chosen for its superior performance and flexibility, integration with alternative kernel libraries (e.g., Triton) is also possible. We plan to explore compiler-based approaches (e.g., Triton) in future work.\n- **Existing Sparse Algorithms (e.g., Quest)**. Here, \"integration\" refers to applying Twilight's optimizer to existing sparse algorithms. As detailed in Section 4, we demonstrate this through experiments with Quest, DS, and others.\n- **LLM Serving Frameworks (vLLM, SGLang)**. Serving frameworks typically build upon kernel libraries (e.g., FlashInfer is used by both vLLM and SGLang). Section 4.3 briefly discusses this integration layer, but we agree with the reviewer that concrete deployment examples would strengthen the work. We will add real-world implementation details in the camera-ready version."}, {"review_id": "pk0zOluCwO", "reviewer": "Reviewer_mTkt", "summary": "The paper proposes Twilight, a framework that enhances existing sparse attention algorithms by replacing top-k token selection with top-p sampling to dynamically adjust the KV cache budget. The authors argue that this approach addresses the limitations of fixed budgets in sparse attention mechanisms, leading to improved efficiency and minimal accuracy loss. The framework is evaluated on both short- and long-context benchmarks, demonstrating speedups over existing methods while maintaining competitive accuracy. However, the technical contribution is incremental, and several methodological and experimental aspects raise concerns.", "questions": "See my above comments.", "limitations": "See my above comments.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Positive points**\n\n1.\tThe paper clearly identifies the limitations of static budget allocation in sparse attention mechanisms and provides a well-motivated rationale for adopting top-p sampling.\n2.\tThe authors conduct extensive experiments across multiple benchmarks and models, demonstrating the generalizability of their approach.\n3.\tThe paper is well-structured, with clear figures and tables that aid understanding. The hierarchical pruning architecture is described in sufficient detail.\n\n**Negative points**\n\n1.\tThe paper’s core contribution is incremental—replacing top-k selection with top-p in existing sparse attention methods. This adaptation lacks significant novelty, as it does not introduce a fundamentally new mechanism.\n2.\tWhen integrated with Quest/DS, Twilight prunes more tokens yet improves performance. If this is due to better removal of irrelevant tokens, why does Twilight not outperform full attention (which retains all tokens)? The authors must clarify this contradiction, as it suggests unresolved trade-offs in the pruning strategy.\n3.\tThe performance improvements on long-context tasks are limited. While Twilight shows improvements over Quest/DS, its performance on long-context benchmarks (e.g., Longbench, RULER) remains inferior to MagicPIG.\n4.\tIn short task evaluations, the authors state that baselines use a fixed budget of 64, which is claimed to be comparable to Twilight’s pruned budget. However, the average post-pruning budget for Twilight is not reported, making it impossible to verify this claim.\n5.\tRecent advances in sparse attention (e.g., [A-E]) are neither compared nor discussed. Position the method against current state-of-the-art techniques.\n\n**Reference**\n\n[A] MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention, NeurIPS 2024.\n\n[B] Core Context Aware Transformers for Long Context Language Modeling. ICML 2025\n\n[C] Flexprefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference. ICLR 2025\n\n[D] MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention. ICML 2025\n\n[E] Curse of High Dimensionality Issue in Transformer for Long Context Modeling. ICML 2025.", "was_revised": true, "final_justification": "The authors address all of my concerns.", "rebuttal": "Thanks for your detailed review and insightful advice. Below, we address each concern and outline planned revisions to strengthen the paper.\n\n**1. Comparing with Recent Sparse Attention Works**\n\nWe sincerely appreciate the reviewer's valuable suggestions regarding prior work comparisons. We have already compared Twilight with two SOTA **top-k decoding methods** Quest/DoubleSparse and one SOTA **non-top-k decoding method** MagicPIG. But we acknowledge that our current presentation may have caused some confusion about our methodological focus. To clarify: Twilight specifically optimizes **the decoding stage**, which dominates the computations in emerging workloads like reasoning and conversation, while works [A-E] primarily target **the prefill stage**, addressing different optimization challenges. Our approach is fundamentally orthogonal to such prefill-stage optimizations. We will enhance the Related Work section to clearly delineate these two optimization domains and better position Twilight within the decode-stage optimization landscape.\n\n**2. Novelty**\n\nWe respectfully but firmly disagree with the characterization of our work as \"incremental.\" While our solution appears simple, this simplicity stems from our key novel insight (as listed below), while prior approaches to dynamic budget allocation in sparse attention (e.g., PyramidKV, AdaKV, DynamicKV) have overcomplicated the problem. Our core contributions are:\n\n- Algorithmic Insight: We demonstrate that the essential dynamism in sparse attention actually lies in the “distribution”, and can be achieved straightforwardly through top-p sampling, bypassing the need for complex, over-parameterized controllers prevalent in prior work.\n- Novel Hierarchical Architecture: We firstly introduce a hierarchical “selector-pruner” architecture, which successfully deploys top-p on existing top-k methods elegantly.\nSystem-Level Practicality: We show that top-p attention is not just theoretically sound but also hardware-friendly.\n\nAs evidenced by our experiments, this approach outperforms existing dynamic methods while being more deployable. We contend that the simplicity achieved through fundamental understanding should not be conflated with incremental progress.\n\n**3. The “Contradiction” of the Performance**\n\nWe appreciate that the reviewer raises an insightful question: why top-p is better than top-k with less tokens while also being worse than full attention? We are glad to provide a clarification. \n\n**Why top-p is worse than full attention?** The reason is that most *training-free* sparse attention designs are just numerical approximations to the full attention output, instead of actual *knowledge distillation/retrieve*. Training-free sparse attention is more like low-bit quantization, which inevitably causes accuracy loss. Trainable sparse attention (e.g. Native Sparse Attention from DeepSeek) has potential to surpass the full attention with better “focus”.\n\n**Why Twilight’s top-p is better than top-k with less tokens selected?** Twilight employs a hierarchical optimization process with two distinct budget parameters:\n\n- Initial Selection Budget (B0, follows the notation in Section 4.3): Our Token Selector (top-k) uses a conservative, relatively large B0 to maximize token coverage (Section 4.1). Crucially, this stage's computational cost is largely independent of B0's size so that we can use a conservative value of B0.\n\n- Pruned Budget (B1): The subsequent pruning stage aggressively reduces the token count to B1 using INT4-approximated importance scores.\n\nThis two-stage approach provides key advantages over conventional top-k:\n\n- Broader Context Awareness: B0 > typical top-k budgets. Therefore Twilight sees more candidate tokens.\n- Precise Selection: INT4 scoring offers better importance estimation than typical top-k (which often reaches a sparsity of 1/16, as it is discussed in Section 4.\n\nThis \"look more, prune better\" strategy consistently outperforms standard top-k attention in both accuracy and efficiency metrics. Note that there is also a trade-off in this two-stage method, which is calculated in Section 4.3.\n\n**4. Limited Long-Context Performance**\n\nWe respectfully disagree with the characterization of Twilight as \"inferior\" to MagicPIG on long-context benchmarks. Our analysis of Tables 2 and 3 suggests the two methods are statistically comparable, with performance differences falling within expected experimental variance (<1%). Notably, Twilight with DS slightly outperforms MagicPIG in both Tables.\n\nWe acknowledge MagicPIG as an excellent state-of-the-art sparse attention method that, like Twilight, demonstrates the advantages of moving beyond traditional top-k approaches. Both works provide independent validation of top-p sampling's superiority. Twilight has an advantage in respect of integration with existing algorithms due to its hierarchical architecture, while MagicPIG is a totally new algorithm-hardware co-design, with a unique sampling mechanism and attention computation workflow.\n\n**5. Average Post-Pruning Budget in Short-Context Evaluations**\n\nWe appreciate the insightful advice from the reviewers. Below are the average post-pruning budget statistics in three short-context benchmarks. We will include them in our camera-ready version and we appreciate this opportunity to strengthen our experimental validation.\n\n| Benchmarks | Average Pruned Budget (Llama2) | Average Pruned Budget (Llama3) |\n|------|-------|-------|\n|GSM8K | 65.68 | 46.05 |\n| COQA | 71.01 | 41.09 |\n| PG-19 Perplexity | 70.18 | 72.91 |"}, {"review_id": "9kvizf8sww", "reviewer": "Reviewer_HTDe", "summary": "This work proposes one idea, that is to replace the top-k strategy used in current sparse attention methods with top-p. The idea is based on the observation that different attention heads can produce very different probability distributions, with some attending to a few tokens and others having more flat attending behaviors. In this case, static top-k strategy can either over-select or under-select in varying scenarios. In contrast, top-p selection has a more normalized behavior that is self-adaptive to different distributions on attention heads. This is similar to the top-p sampling in LLMs. While the idea is simple and directly applicable to existing sparse attention methods, there are certain architecture designs that need improved for efficient implementation, which is the other major contributions of this work. These include efficient sparse GEMV with 4-bit quantization of key cache and efficient, parallel top-p selection with binary search.\n\nThe empirical experiments consist of three parts, effectiveness, efficiency and ablation study. The first part shows improved accuracies over the base methods on different types of tasks. The second part breaks the efficiency benefits of the proposed method. The ablation study investigates the influence of the threshold p and the cost breakdown.", "questions": "- I wonder why the time spent for sparse attention in Quest-Twi is so much shorter than that in Quest? Is that because of the average number of tokens selected by Quest-Twi is much lower than that by Quest?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Simple yet effective idea, which successfully borrows the idea of top-p sampling.\n+ The authors successfully identify the difficulties of apply top-p sampling to existing sparse attention methods and address them well.\n+ Impressive performance with further reduced computation cost.\n\nWeaknesses:\n- Since replacing top-k with top-p is a straightforward idea, the technical contributions of this work mostly lie at the how the implementation issues are solved with most efficiency. However, the ablation study fails to do so. We are not clear how important the \"efficient\" kernel implementation described in Section 4.2 is for the efficiency shown in the experiments.\n- It is not clear how sensitive the efficiency is to the value of p. Figure 6 only shows the influence on the model performance. Also, Figure 6 has the wrong label on x-axis.", "was_revised": true, "final_justification": "The authors' responses addressed many of my concerns. I remain positive towards this work.", "rebuttal": "Thanks for your detailed review and insightful advice. Below, we address each concern and outline planned revisions to strengthen the paper.\n\n**1. Detailed Description of Efficient Kernel implementation**\n\nWe fully agree with the reviewer's assessment regarding the importance of our kernel implementations to the overall contribution. While space constraints limited our presentation of these technical details in the current version, we will significantly expand the kernel implementation documentation in the camera-ready version, including warp-level reductions, better utilization of the memory bandwidth, fusion of dequantization and SpGEMV, details on head-wise load balancing, etc.\n\n**2. The Impact of p on Efficiency**\n\nWe sincerely thank the reviewer for catching the incorrect x-axis label in Figure 6, which has now been corrected. Regarding the impact of the `p` value on computational efficiency, the following shows a basic statistics evaluated on the TrivialQA dataset:\n\n| p | 0.65  | 0.70  | 0.75  | 0.80  | 0.85  | 0.90  | 0.95  |\n|------|-------|-------|-------|-------|-------|-------|-------|\n| PG-19 Perplexity (Lower is better)  | 13.021| 11.050| 8.857 | 7.777 | 7.332 | 7.324 | 7.228 |\n| Pruned Tokens Number  | 72 | 93 | 117 | 160 | 237 | 381 | 807 |\n| Sparse Attn Time bs=32 (us)  | 45.63 | 45.01 | 45.62 | 45.06 | 44.93 | 66.38 | 135.46 |\n| Sparse Attn Time bs=64 (us)  | 46.60 | 45.90 | 54.93 | 64.88 | 84.32 | 124.68 | 243.84 |\n\n\nIn summary, the `p` value directly controls pruning aggressiveness and affects the time of the Sparse Attention stage by the pruned tokens number. We will expand this analysis in the ablation study part of the camera-ready version.\n\n\nNotably, although we introduce the hyperparameter `p` in order to get rid of `k`, `p` is a more reasonable and tunable hyperparameter. This is because `p` represents the accumulated probability, which is less impacted by the different distributions that may occur for different heads/layers/queries. In contrast, `k` is highly sensitive to different distributions, as illustrated in Figure 1 in the paper. This allows us to just tune `p` for a fixed model. We can use a small dataset to calibrate it.\n\n\n**3. Question on Time Breakdown**\n\nThanks for your interesting question! Your understanding is fully correct. And as Table 5 shows, Twilight can prune the original budget (~8192) to extremely low (100 to 400), which largely reduces the time of the Sparse Attention part."}, {"review_id": "L2EH8Fhkbh", "reviewer": "Reviewer_QNJM", "summary": "The paper introduces Twilight, a select-then-prune framework that upgrades any existing top-k sparse-attention scheme with a top-p pruning stage. A base token-selector first picks a conservative subset of KV positions; a lightweight pruner then keeps the smallest subset whose estimated attention weights sum to a user-set threshold p. Across LongBench, RULER, GSM8K, COQA and PG-19, Twilight reportedly prunes up to 98 % of tokens with almost no accuracy loss and delivers 15.4x operator-level and 3.9x E2E speed-ups, or ~ 1.4 times beyond the best sparse baselines.", "questions": "1. The INT4 cache adds ~12 % KV footprint. Is it possible to quantify the wall-clock impact compared to a top-k based method which does not incur additional memory footprint? \n2. How sensitive is the method to different `p` thresholds? The paper mentions usage of different values for different models. \n3. Binary search still needs repeated sum() reductions. Could reservoir sampling or approximate CDF estimates cut this further?", "limitations": "No limitations section - would be good to include some of the ones mentioned in \"Weaknesses\" section - boundedness by the base token selector and KV overhead.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* The idea and motivation of using top-p instead of commonly used top-k is natural and intuitive.\n* Significant latency gains compared to the baselines.\n\nWeaknesses:\n* The method efficacy is bounded by the base token-selector. \n* Extra INT4 copy increases KV memory by up to 12.5 % and is always resident on-device", "was_revised": true, "final_justification": "The authors addressed my concerns about sensitivity to parameter `p` with an ablation study, which makes the submission more sound. Unaddressed concerns is that limitations (KV cache overhead) could be stated more clearly in the main part of the paper, instead of appendix. As the rebuttal addresses some of my technical doubts about the approach, I will raise the score.", "rebuttal": "Thanks for your detailed review and insightful advice. Below, we address each concern and outline planned revisions to strengthen the paper.\n\n**1. Sensitivity to Hyperparameter p**\n\nWe appreciate the reviewer's valuable suggestion regarding the efficiency analysis of parameter p. In our current Figure 6, we have shown the impact of p on model performance (accuracy). The following table shows the impact on efficiency evaluated on the TrivialQA dataset. We will augment this analysis in the camera-ready version.\n\n| p | 0.65  | 0.70  | 0.75  | 0.80  | 0.85  | 0.90  | 0.95  |\n|------|-------|-------|-------|-------|-------|-------|-------|\n| PG-19 Perplexity (Lower is better)  | 13.021| 11.050| 8.857 | 7.777 | 7.332 | 7.324 | 7.228 |\n| Pruned Tokens Number  | 72 | 93 | 117 | 160 | 237 | 381 | 807 |\n| Sparse Attn Time bs=32 (us)  | 45.63 | 45.01 | 45.62 | 45.06 | 44.93 | 66.38 | 135.46 |\n| Sparse Attn Time bs=64 (us)  | 46.60 | 45.90 | 54.93 | 64.88 | 84.32 | 124.68 | 243.84 |\n\nIn summary, the `p` value directly controls pruning aggressiveness and affects the time of the Sparse Attention stage by the pruned tokens number. We will expand this analysis in the ablation study part of the camera-ready version.\n\nNotably, although we introduce the hyperparameter `p` in order to get rid of `k`, `p` is a more reasonable and tunable hyperparameter. This is because `p` represents the accumulated probability, which is less impacted by the different distributions that may occur for different heads/layers/queries. In contrast, `k` is highly sensitive to different distributions, as illustrated in Figure 1 in the paper. This allows us to just tune `p` for a fixed model. We can use a small dataset to calibrate it.\n\n\n**2. Overhead of INT4 KV Cache**\n\nWe fully acknowledge the overhead introduced by INT4 KV Cache in Twilight. As the reviewer correctly identified, this manifests in two key dimensions: (1) Storage overhead, as the reviewer mentioned, 12.5% extra overhead to GPU memory; (2) I/O overhead, which occurs during the attention forward process.\n\nThe wall-clock impact, as far as we understand, refers to the I/O overhead where the SpGEMV needs to read corresponding INT4 KV blocks to estimate the attention score. This can be found in the breakdown analysis of Figure 7, where “SpGEMV” part reflects this overhead. Note that SpGEMV is a memory-bound kernel like the Attention computation. Hence the total kernel time is roughly equal to the read time.\n\nTo deal with the Storage Overhead (on-device INT4 KV Cache), offloading may be a possible solution. But we find it is more suitable to offload the full precision KV instead of INT4 KV as the pruned tokens number (B1, follows notations in Section 4.3) is far less than the initial budget (B0). We discuss this potential scenario in Appendix D.\n\n**3. Improve Binary Search during Sampling**\n\nWe sincerely appreciate the reviewer's valuable suggestion regarding alternative sampling approaches. After careful experimentation on the approach of CDF estimation, we find it is unsuitable for Twilight's architecture because our selector-pruner architecture requires higher precision. A concurrent work that also highlights the potential of top-p attention, Tactic (cited in our paper), makes this method work upon full attention.\n\nFor random methods like reservoir sampling, it is worth mentioning that MagicPIG, a work which is evaluated and compared with Twilight, uses another method called LSH sampling. Twilight tends to adopt non-randomized algorithms to maintain stability as an optimizer. \n\nHowever, we fully acknowledge the optimization potential in our current binary search implementation and are actively exploring improvements. Notably, the FlashInfer community's recent breakthrough in sorting-free GPU sampling kernels presents a promising solution, as documented in their technical blog. Their novel Dual Pivot Rejection Sampling algorithm achieves an approximately 2x speedup over traditional binary search. We are trying to integrate it with Twilight and will incorporate this collaborative advancement in our camera-ready version.\n\n**4. Section of Limitations**\n\nWe discuss the limitations in Appendix E, as required by the submission guidelines."}, {"review_id": "N6BId1kjRw", "reviewer": "Reviewer_Pft7", "summary": "Twilight proposes an adaptive framework for long-context large language models (LLMs) to address the limitations of static budget settings in traditional sparse attention methods. By leveraging top-p sampling (nucleus sampling), Twilight dynamically prunes tokens to maintain a cumulative attention weight threshold (e.g., 90%), overcoming over/under-selection issues in fixed top-k approaches. The hierarchical \"select-then-prune\" architecture first uses base algorithms (e.g., Quest/DS) to select tokens conservatively, then refines via top-p with 4-bit quantization and GPU kernel optimizations. Experiments show Twilight achieves up to 1.4× speedup over state-of-the-art sparse attention, prunes 98% redundant tokens, and maintains near-zero accuracy loss across mid/long-context benchmarks (Longbench, RULER)", "questions": "Mention in Weakness", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Adaptive Budgeting with Top-p Sampling Replaces static top-k with dynamic top-p sampling, which adjusts the token budget based on the cumulative attention weight sum.\n2. Significant Efficiency Gains. Achieves up to 1.4× speedup over state-of-the-art sparse attention methods (e.g., Quest, DS) for self-attention, and end-to-end decoding speedup. Reduces memory access and computation by dynamically pruning redundant tokens, especially in long-context scenarios.\n3. System-Level Optimizations. Implements efficient kernels (e.g., SpGEMV with 4-bit quantization) and load balancing for head-wise dynamism, reducing latency and memory overhead.\n\nWeakness:\n1. The top-p pruning introduces an additional computation step, which may offset speed gains in scenarios with small batch sizes or minimal token pruning.", "was_revised": true, "final_justification": "I acknowledge the limitation and look forward to a future fix. I will maintain my score.", "rebuttal": "Thanks for your detailed review and insightful advice. Below, we address each concern and outline planned revisions to strengthen the paper.\n\n**1. Additional Computation Cost of Top-p Pruning**\n\nWe sincerely appreciate the reviewer's insightful observation regarding the Pruner's computational overhead. As noted in Appendix E, we acknowledge that the current estimation process introduces non-trivial latency that can diminish speed benefits in certain edge cases, like the reviewer mentions, small batch sizes or scenarios with minimal token pruning. While we acknowledge the current computational overhead in the Pruner as noted by the reviewer, we maintain that Twilight's core contributions remain significant for the top-p sparse attention and hierarchical selector-pruner framework."}]

Author rebuttal:

Meta-review: Twilight presents a novel framework for adaptive sparse attention in large language models (LLMs), leveraging top-p sampling to address static budget limitations in existing sparse attention methods. Its core innovation lies in a hierarchical "selector-pruner" architecture: a conservative base selector first identifies candidate tokens, followed by top-p pruning with INT4 quantization to dynamically adjust token budgets based on cumulative attention weights. This design enables aggressive pruning (up to 98% tokens) with minimal accuracy loss and 1.4x speedups over state-of-the-art methods, making it practical for long-context LLM inference.

Reviewers highlighted strengths in its simplicity, effectiveness, and real-world applicability. While some initially noted concerns—including overhead-benefit tradeoffs, incremental novelty perceptions, and INT4 cache overhead—the authors’ rebuttal comprehensively addressed these. They clarified pruning-stage costs are offset by reduced computation in long contexts, added analyses of top-p vs. top-k token distributions, detailed integration with frameworks like FlashInfer and VLLM, and distinguished Twilight from prefill-stage optimizations by focusing on decode-stage efficiency.

Most reviewers maintained or raised scores post-rebuttal, acknowledging the work’s significance. The framework’s compatibility with existing systems and empirical robustness across benchmarks (LongBench, GSM8K) further solidify its value.
Decision: Accept (spotlight)