Kinetics: Rethinking Test-Time Scaling Law
Authors: Ranajoy Sadhukhan
Zhuoming Chen
Haizhong Zheng
Beidi Chen
Venue: NeurIPS 2025 poster ()
Keywords: scaling law
test-time scaling
sparsity
Primary area: deep_learning

Abstract: We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. The Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold (14B) than on smaller ones. A key reason is that in test-time scaling, attention—rather than parameter count—emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60-point gains in low-cost regimes and over 5-point gains in high-cost regimes for problem-solving accuracy on AIME and LiveCodeBench. These results suggest that sparse attention is essential for realizing the full potential of test-time scaling because, unlike training where parameter scaling saturates, test-time accuracy continues to improve through increased generation.

Kinetics: Rethinking Test-Time Scaling Laws
Ranajoy Sadhukhan" Zhuoming Chen" Haizhong Zheng Beidi Chen
Yang Zhou
Emma Strubell
Camegii Mellon University, Pittsburgh, PA
(rsadhukh, zhuominc, haizhonz, haizhon,, yangzhoo http:/heeessstsuellll. estrubel, beidi Handrew. cmu edu
Abstract
We rethink test-time scaling laws from practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded compute-optimalle
overlooks critical memory access bottlenecks introduced by inference- tentenece-meeeeeeee strategies (e.g., Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above threshold (14B) than smaller ones. A key reason that TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 point gains in low-cost regimes and over point gains in high-cost regimes for problem-solving accuracy on AIME and BiveCodeBench. These results suggest that sparse attention is essential for realizing the full potential of test-time scaling because, unlike training, where parameter scaling saturates, test-time accuracy continues improve through increased generation.
Iinetits Scaling Law Previous Scal Law
(Sparse) Frontier (Densa)
B200 seconds
3.0K
Accuracy
0.66
0.4
Accuracy
Kinetics
Previous
0.2
0.4
0.6
3.8
1.0
2
101
Pareto Frontier Model Selection
(a) Kinetics Scaling Law
(b) Kineties Sparse Scaling
Figure 1: (a): Pareto Frontier for Qwen3 series on AIME24. Previous test-time scaling laws 4 74 focus solely compute optimality, neglecting the significant bottleneck of memory access long-sequence generation. This leads suboptimal resource utilization. By incorporating memory access, the Kinetics Scaling Law reduces resource demands by up to to achieve the same accuracy. (b): Inspired by Kinetics Scaling Law, we show that sparse attention models scale significantly better than dense models, achieving over 50-point improvements in AIME24 in the low-cost regime and consistently outperforming dense models in the high-cost regime, in addition substantial efficiency gains. B200 second represents the amount of work performed by single B200 full utilization for one second.
Equal contribution.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

10a
10
2wan3-45
Gward 325 Sparis
GAmbl-1 Sparee
3wmb--12
CoTe
100000
D0000
Goan3- 1a Spone
Qwen3-15
Q4en3-128 Spane
Gward-312
20.6
Parera (henter
Paresa donteerdBicck Tapp))
60000
10
40000
0.4
45
10
20000
0.2
10a
10a
Generated Tokens
10a
10
10
10
8200 seconds
1000
10
102
0.0
10
10-1
10
E200 seconds
(a) Attention Cost Dominates
(b) Tokens Costs
(c) Block Sparse Attention
Figure 2: (a) Inference cost is dominated by attention, which is 100 1000 more than model parameter computation, sparse attention fundamentally mitigates this bottleneck. (b) Under the same resource constraint, sparse attention generate massive tokens out of the reach of dense models, which is proven to enhance the effectiveness of test-time scaling. (e) Simple block sparse attention yields substantial improvinn accuracy by points in the low-cost regime and achieving equivalent accuracy while using 8.58x fewer resources.
Introduction
Test-time scaling (TTS) has recently emerged as a powerful strategy (e.g.. Best- -,N, Long-CoT [66]
for enhancing the reasoning capabilities of large language models (LLMs) 30
38 81 particularly in
scenarios where agents interact with complex environments, e.g.. writing code, browsing the web [66
92 or reinforcement learning (RL) with LLMs-in-thhe-ooe [36 20 8 These capabilities, however, in- troduce substantial inference-time costs, making it critical understand performance scaling in this new paradigm. Existing scaling law studies 4 74 3] primarily focus on floating- point operations (FLOPs) while ignoring memory access costs, which are often the dominant factor in determining wall-clock latency in TTS regimes. As shown in Figure this gap can lead to sub-optimal deployment decisions.
In Section 3. we introduce the Kinetics Scaling Law for TTS, derived from a novel cost model that explicitly incorporates memory access costs. This new perspective reveals markedly different conclusions about Pareto-optimal strategies for allocating test-time compute (Figure Tap. Specifically, we find that: (1) prior scaling laws consistently overestimate the effectiveness of small models enhanced with inference-time strategies; and (2) computational resources are best spent first on increasing model size- up a critical threshold (empirically around 14B parameters) before investing in test-time strategies, such as Best-of-N sampling or long CoTs Guided by the Kinetics Scaling Law, approach yields up to a 3 throughput improvement on B200 hardware.
Our roofline analysis across suite of state-of of-the-art reasoning models reveals that the shift in optimal test-time compute strategies arises because test-time strategies (e.g.. long CoT', Best-of-N dispropor- tionately increase attention costs rather than parameter costs (Figure 2ap. Our Iso-cost analysis shows that the quadratic growth of attention with generation length, combined with the disproportionate scaling of KV memory relative to model parameters, drives a preference for scaling up model size over generations. This imbalance is further exacerbated by MoE architectures (72 22 which reduce active parameter count without alleviating attention overhead.
40)
Building on this analysis, in Section we introduce a new scaling paradigm, centered on sparse attention, which fundamentally reshapes the scaling law and significantly enhances the scalability TTS (Figure lb). According to our Kinetics Sparse Scaling Law, computational resources are best allocated test-time strategies rather than reducing sparsity. As more computing invested at test time, higher sparsity becomes increasingly critical to fully leveraging the benefits of these strategies. Guided by this principle, it increases problem-sol solving rates by up to 60 points in the low-cost regime and over 5 points in the high-cost regime on AIME24 and LiveCodeBench, through massive generated tokens, which is unaffordable for dense counterparts.
In Section we demonstrate the practicality of the Kinetics Sparse Scaling Law using simple block- sparse attention mechanism built on top of paged attention. This approach achieves up to 25x wall- clock speedup on H200 GPUs. While sparsity has traditionally been employed either for regularization in small models [2] 64 or to reduce computation in over-varamenn parameterized networks (62 62 15| 24 (55) our work introduces a fundamentally different perspective: sparsity as a central enabler of efficient and scalable test-time inference. contrast to pretraining - where scaling laws often exhibit dimin- ishing returns (37 - TTS continues to benefit from increased token generation and more optimized inference paths. We hope this study can guide and encourage future co-design of model architectures, inference- time strategies, and hardware systems to fully unlock the next wave of scaling deployment.

---

2 Related Work and Problem Settings
In this section, first review several lines of related work relevant to Kinetics Scaling Law. Then we introduce a cost model accounting for computation and memory access, followed by aroofline analysis uncovering key departure from traditional scaling laws. Finally, we outline the experimental setup used in the subsequent analysis. Notation is summarized in Table
Attention
800
Parameter
Computation
600
Parameter Memory
Latency
has extensively examined the scaling laws of pretraining, exploring the trade-off between model size and the number of training tokens under fixed FLOPs budget. More recently, studies such as [74, 87 have extended this analysis test-time scaling (TTS), with focus compute-opiimaliy. While these works offer strong theoretical foundation, they largely overlook the critical bottleneck posed by memory access in current TTS systems.
Scaling Laws. Prior work 42 33 45
400
Model and Context Length
Figure Cost Breakdown (bsz 4K).
Test-Time Scaling, Recent LLMs such DeepSeck-RR R1 [30]] OpenAl-o 38 and QwQ 81 generate extended CoT rea-
soning 86 solve complex problems, including those from
AIME
59 60] Techniques such as parallel search through repeated sampling 44, majority voting
(self-consistenyy) [85], and reward-model-guide
inference (e.g., REBASE [87). MCTS [3] 74]) aim
to improve reasoning accuracy. Strategies such as [26] 26.2.0 80) and hybrid models 449 69 have been proposed reduce the cost of test-time scaling.
Sparse Attention. A significant line of prior work has focused overcoming the quadratic computational bottleneck of attention mecha- nisms during LLM training by leveraging the natural sparsity of attention matrices [1] 44
Table 1: Notation Used throughout the Paper.
Symbol
Description
Syumbol
Description
T.T
Task (set)
Loue
Gen tokens
17 98 3 [66. More recently, sparse atten- tion has experienced a resurgence in the con- text of LLM inference, where methods such as 101 88 79 (54 10, 35 restrict the memory access of the key-value (KV) cache during gener- ation while maintaining strong performance. These advances form a strong and steady foundation for our exploration of new TTS paradigm.
Model
N.Nv
Reasoning
C.Cri())
Algorilhm
B. B.B
KV badget
Lim
Prompe length
P
Parameters
size token
GQA mio
2.1
Cost Model
We first calculate the inference cost for the cases where the batch size is 1, and then extend to a more general case in TTS. Finally, we propose our cost model using equivalent FLOPs.
Computation. As discussed in 4 the computation consists of two parts: linear modules and self- attention, which is (we assume the model is served in BFloat16.)
Coomp
P
+r(2Lin + Loust Lora
model parameterrs computation
self-atiention
Memory Access.
Memory access also consists of two parts: model parameters and KV cache.
Crmm
2PLout
+ +2L Lout D
I D
model parameter access
prompt KV cache decoding KV cacbe
In real serving scenarios, a large batch size will be used [8] with growing GPU VRAM 83 and model parallelism [70]. The access to the model parameter will be amortized across requests in batch (Figure3Jshows shows parameter access time is negligible when the batch size is large). Thus, only consider the second term (i.e., KV cache loading) in our cost function. Furthermore, in the cases that we have N reasoning trials, the prompt cache access [4] 102 is also shared across these N trials. Thus,
Coomp [) 2PN Lout + 2r Loua D +r D Lu D
(I)
Cmem (N)= = 2L out D N NN2
(2)
3

---

eFLOPs. We propose eFLOPs (equivalent FLOPs) to capture both compute and memory access cost,
coLoPs=Camm mem
(3)
where / is the arithmetic intensity of hardware, which reflects that modern accelerators usually have much larger computation capacity over memory bandwidth, and the gap is growing over the years [7] In this work, we use 562.5 from NVIDIA B200 [33.
With Equations to 3 we obtain the final cost model.
CTTS
L.
2r Lin Dont + N DL o 2in Dlout + INDI oott
computation
self-attention computation
access
where P are syype--paaamee determined by model M
Roofline Analysis. Our key insight is attention-related cost dominates in long CoT's We show this by estimating the ratio of attention-related cost parameter-related cost Φ.
2r Lin D (r++
2P
As shown in Figure 2a in the regime of long CoTs, where the generation length exceeds 4096 tokens, the cost of attention surpasses that of model parameters by a factor 100~ 1000.
MLA 5 reduces KV memory access by a constant factor (similar to in GQA), it insufficient for achieving true scalability due to several limitations: (1) MLA does not reduce attention computation; (2) the gap between FLOPPs and memory bandwidth is expected to widen in the future; and (3) emerging fine-grained MoEs [
40 drastically reduce FLOPs in linear layers by a factor of 10 ~ 20x,
further increasing the relative cost of attention.
Under the context of long-CoTTs being widely adopted, can safely assume generated length Lout Lin or at least proportional to Lin Hence, the bottleneck of inference is shifted from linear term Low? to the quadratic term 2 D
Experimental Setup. In our analysis, we focus on three challenging reasoning benchmarks: AIME24 [59], AIME25 (60), math datasets spanning algebra, combinatorics, and geometry, and LiveCodeBench (3)) which includes complex programming problems from recent coding competi- tions. We evaluate performance across various model sizes of the Qwen3 [91] series. More results shown in the Appendix. We utilize the specs from the latest and most powerful Nvidia B200 as the basis of our theoretical studies.
3 Rethinking Test-time Scaling Law
In Section we first introduce the Kinetics Scaling Law, derived from empirical investigations across the Qwen3 model series. Then, we explore the underlying reasons for the divergence between Kinetics and prior scaling laws through an Iso-Cos analysis in Section 3.2
3.1 Kinetics Scaling Law
In this section, We study the scaling behavior of the Qwen3 |0
90 considering the following problem:
For each fixed maximum inference budget, eFLOPs per question, what is the Pareto frontier of achievable accuracy across different L.MM configurations?
we first formally formulate the objective of the test-time scaling law, focusing on the tradeoff between model size and the number of generated tokens.
With the refined cost model in Section 2.
Dense Scaling Law. Given a problem instance 7 and a total inference budget C, our goal is to explore the optimal tradeoff between two key factors: the choice of language model M, and the number of reasoning trials N and the maximum generation length n. More precisely,
(N,)),, M. arg max Acc MT;) s.t. CrTs CN M (N.)).M
(5)
Rooffline model max(Coomp. Cnem 1) also works here and favor our claims more since most of time Cren x / dominates the cost. We choose to use an additional model because Ceamp mainly comes from linear layers while Cmem mainly comes from the self attentoon layer. The parallelization of these components during decoding remains active area of research [103.. We discuss this roofline cost model in the Appendix. 3 Since Toout might differ across reasoning trials, we take the expectation for [.[Tour]] and E. (.
'This is why we call ours Kinetics Scaling Law-simile to Ess = mmv.

---

A-
Owen3-325
Optinal Canvealion Langet
20005
Owe13-328
Owon3:
Owo13
0mur3
15000
Dwo
Owen3-46 48
Owe
Owen3..18
10000
10-13-1.78
Qward
Owe 13 0.68
0.2
2000
30
(+) Thr-
(b)
Tere- 100h
3-
Optirea Chenereeao Langth
20000
Owe 328
13-13-148
Owen3
15000
Owe13 30
Owen3-44 48
Owe 13-8
Owen3.../8
10030
Qie 15-1.7
Qward 6B
Qiee 15-0.6
3000
10
30
10
30*
(c)
SFLOPS
oFLOPs
Figure& 4: AIME Pareto Frontier (Long-CoTs), We first launch evaluations for Qwen3 series models. By controlling the maximum allowed generation lengths, we control the incurred inference cost in eFLOPs (ab for our scaling law) or FLOPs (ed for previous scaling law) and measure the accuracy (Pass@ 1) in AIME24. The optimal model is marked with different colors in (ac). The optimal generation length is presented in (bd).
Let Ace(N N ",,, AT) denote the gobblem-solving rate of model M on task T, using N reasoning trials, each with a maximum reasoning length of n. We investigate two inference strategies: Best-of-N (with fixed n) and Long-Col (with fixed N).
In the Long CoT's scenarios, where N1 1, we vary Br to evaluate the model performance under different costs. We present our results in Figure 4 Our Kinetics Scaling Law highlights two important findings compared to the previous one, which focused on merely FLOPs.
Efficiency of small models overestimated. As shown in Figures [b) (ac), smaller models, despite having fewer parameters, are not as efficient as commonly assumed. For example, the 14B model outperforms both the 4B and 8B models even at low accuracy levels (e.g.. below 40%), and the 0.6B model only lies on the Pareto frontier in regions where accuracy is negligible. In contrast, under previous scaling laws, models of all sizes span a meaningful portion of the Pareto frontier.
Extending CoTss are more effective than enlarging parameters only for models beyond a critical scale (empirically, 14B). The Kinetics Scaling Law reveals that under constrained compute budgets, allocating resources to model scaling yields greater returns than increasing CoT length. As illustrated in Figure 4 (bd), only the 14B and 32B models benefit from generating CoTs longer than 10K tokens; for smaller models (e.g., 1.7B and 4B), switching to a larger model is more advantageous when Lout <5K. This suggests that, in practice, most of the available compute should be devoted to increasing model size rather than lengthening generations (Figure (c)))) In contrast, previous scaling laws assumed that longer CoTs provided consistent benefits across all model sizes, recommending model scaling only after CoT performance gains had plateaued.
In the Best-of-N - setting, we fix the maximum number of generated tokens nr,, and vary the number of reasoning trials N to evaluate the problem-solving rate (i.e., the probability that at least one trial produces a correct answer). We have similar observations in Figures 66 6c Under the previous scaling laws (Figure 6b), the most cost-effective strategy to achieve high accuracy is to apply repeated sampling using smaller models. Kinetics Scaling Law Figure/6a]revee with fewer reasoning trials is more efficient. We also observe a critical size of 14B. For models smaller than 14B, increasing compute is best allocated toward model scaling rather than additional trials. For models at or above 14B, however, further computation is more effectively spent on increasing the number of reasoning trials, up to diminishing returns.
that deploying a 14B model
3.2 Iso-Cost Study
We attribute the above divergence between Kinetics and previous scaling laws to reasons.
Disproportionation between KV memory size D and model parameters P. Smaller models tend to require significantly more KV cache relative to their parameter size. For example, Qwen3-0.6B demands 3.5GB of KV cache to store 32K tokens, despite the model itself occupying only 1.2GB. In
contrast, Qwen3-32B uses just 8GB of KV cache for the same sequence length. Empirically, doubling model parameters results in only a 1.18x increase in KV cache size. As shown in Figure 5a this
phenomenon consistently observed across model families such as OPT 100 (1.55x), Qwen2.5 990
(1.46x), and LLaMA3 27 (1.27x).
5

---

25000
3.0
25000
2.0
10 memory boen (tyees)
Gam3
Optimal Genreretion Length
Optimal Generation Length
10f
Cwen22
OPT
LiiM 2
20000
15000
2.0
2.5
150-
20000
15000
1.5
FLOPs
1.5
1.0
104
10000
10000
1.0
0.5
0.5
10'
10
10
10
1.0 2.0 4.0 8.0 16.0
1.0 2.0 4.0. 0.0 16.0.22.0
Sizw (tylus)
Model Sizes (6)
Model Sizes (B)
(a) KV v.s. Parameters
(b) Iso-eFLOPs
(c) Iso-FLOPs
Figure Explanation the New Scaling Law. Left: Analysis across four LLM families reveals consistent trend of disproportionately slower KV memory growth relative to model size. For the Qwen3 series in particular, doubling model parameters results in only a 1.18x increase in KV cache size. Middle and Right: We compare the Iso-Cost landscapes under the proposed cost model (b) and the traditional model (c).
0.8
0.8
30
Opelie Seatin Use Previou Sta Law
25
0.6
0.6
Sahing
Gwenh-14
Owess
0.4
Owmi1-48
Ownii
Qwen2 48
Owend-4 45
0.2
Qwin2-2.7 2.78
Owend 1.78
Qwen2-5.c0 c0
Gwe11-0.c c
0.0
10
10
10
30a
10
10
10
10
Tera-4FLOPs
Tera-FLDPs
Solving Rate
(a) Accuracy (eFLOPs)
(b) Accuracy (FLOPs)
(c) Optimal Models
Figure 6: AIME Score Curve Envelope (Best-of-N), We control the incurred inference cost in eFLOPs (a) or
FLOPs (b) and measure the solving rate (Coverage) in AIME24 for various models by varying the maximum allowed number of reasoning trials. By taking the curve envelopes, we can project the optimal models in (c).
Shift from linear to quadratic cost model. Under this revised model, increasing generation length incurs a substantially higher cost than scaling model size; consequently, the tradcoff between model capacity and token budget shifts meaningfully. For instance, under the linear LP model, the cost of generating 8K tokens with a 14B model (which is usually insufficient to solve complex tasks) is treated as equivalent to generating 24K tokens with 4B model (sufficient to complete most tasks). However, under the D model, the same 14B @ 8K generation is only comparable in cost to a 4B @9K generation. This tighter bound makes much harder for smaller models to compensate for their limited capacity through extended generation alone. Thus, only if the gap in model capacities is small enough (e.g., 32B only improves the accuracy by 3% on AIME24 compared to 14B), the benefits of extending generation length might be more effective than directly enlarging model parameters.
Figures 5b 5c show an Iso-Cost analysis comparing two cost models. Under Kinetics Scaling Law, the cost grows quadratically with Lout, while the KV cache scales sublinearly with model parameters P. As result, when total budget is low, the Iso-eFLOPss contours tend stretch horizontally, favoring larger model sizes over longer generation lengths. This implies that increasing model size is more efficient use resources than generating longer outputs. In contrast, the traditional FLOPs-based model leads to steeply vertical contours, encouraging longer generation before increasing model size.
4 Scale Test-time Scaling with Sparse Attention
Based on our findings in Section 3 we propose a new scaling paradigm centered on sparse attention. We begin by presenting a simple greedy algorithm for optimal resource allocation in sparse attention models, which we use to identify the Pareto frontier in Section 4.1 We then analyze the resulting changes in the scaling law and show that sparse attention models with massive TTS strategies lead to significantly higher mroblemm-sllving rates Section 4.2

---

4.1 Optimal Resource Allocation with Sparse Attention Models
Problem statement. Let A denote the corresponding sparsity patterns (e.g., top-k, block sparse and local. Our goal is to explore the optimal tradeoff among three factors: model M, KV budget B, and number of trials, and the maximum generation length ()) Specifically,
(N. B. = arg max M,, M,,
s.t. (Nrs(N,n, n. (..sNN..B.A.M:TTT A.A.MTT)))
(6)
The cost function CrTs differs from the one in Equation 4 as it incorporates sparse attention mecha- nisms (which makes the quadratic T,2 D term back to linear term D). This modified cost model is discussed in detail in the Appendix.
Greedy algorithm for optimal resource allocation: We present method to optimally schedule generation parameters (N,n) and the KV budget B for each task, establishing an upper bound on achievable performance and enabling analysis of the core tradeoff between TTS strategies and sparsity. We begin by solving the subproblem for each individual task 7
max Acc( Aco(Nr, ,r,,,,,,,BBr (r,,,,, s.t. (Nr. AT.,A.M
Empirically, we discretize the searching space. For instance, in Best-of-N, we discretize the space of N and B by producing a search grid:
G= GNo...-......NN.e (B.,
For each pair (Na,,Bb) EG, we compute the corresponding cost (a. and accuracy (c.T. We use (Nr, (Nr,Br) Br) E G which maximizes the accuracy under the cost constraint C' as an approximation for Equation 7 By combining the optimal configurations (Nr, Br) for all tasks T, we obtain a solution to the overall problem Equation (6). Similar discretizations also applies for Long-Cole -oos. Thus we find the optimal resource allocation.
4.2 Kinetics Sparse Scaling Law
Sparse attention fundamentally reshapes the Kinetics Scaling Law in Section 3 and significantly enhances the scalability TTS. We present three important findings below.
performance. As shown in Figures Sa to 8i compared to dense baselines, for both of the inference strategies and models of various sizes, sparse attention models improve problem-solving rates by up to 60 points in the low-cost regime and over 5 points in the high-cost regime. From an efficiency perspective, dense models require over 10x more eFLOPs to match the same solving rate. These findings underscore sparse attention as a key enabler for unlocking the full benefits of test-time scaling.
Sparse attention significantly enhances
Sparse attention becomes increasingly valuable in high-cost scenarios. We investigate the tradeoff between KV budget B and generation tokens (N.).. For Long- CoT, we model the trade-offs between KV cache size, generation length, and total cost. For of we analyze how the optimal KV budget and the number of generated tokens scale with cost across N reasoning trials. Our analysis reveals a consistent trend: allocating additional compute toward generating more tokens is generally more effective than expanding the KV cache. In Best-of-N N frontier, doubling the cost leads to only 1.18x increase in KV budget, compared to 1.74x increase total generated tokens. Similarly, for Long-CoT, the KV budget grows by 1.23x, while the number of generated tokens increases by 1.52x. These scaling patterns imply that as the total resource budget grows, the sparsity ratio, defined as the ratio of KV budget to total generated tokens, should decrease.
Sparse attention reshapes the Kinetics Scaling Law. As shown in Section 4.2 applying sparse attention significantly improves the efficiency of smaller models (0.6B, 1.7B, 4B), allowing them to re-emerge on the Pareto frontier across broader range. Sparse attention reduces atten- tion memory access from a quadratic cost term (L² D) to linear one (LBD), making it negligi- ble or comparable when compared to the cost of computing with model parameters (LP).
0,7520
2,030
028
14.338
32.768
Regimes (Single Ber)
Long (Sparse)
Bast-o6-N
(Spanse)
Long- COT
(Dense)
10:
10a eFLOPs
10
Figure 7: Compared the scaling law for the dense mod- cls (a), small models (0.6B, 1.7B, 4B) are more effective with sparse attention. In other words, they occupy more space in the Pareto Frontier (a).
$ For fairness, we do not schedule resources across tasks, but consider resource upper bound for all the tasks.
7

---

Tameto (Sparen) Tamato (Dense)
Scorse: 0.53
Dence
Saase 0.93
Lere 0.75
Saurac 0.75
Dense:
0.6
500409 Rate
0.05
Sabing Rate
2.05
0.2
0.2
02
0.0
0.2
Perse 0.03
0.0
Dense: 0.03
10
10
10
204
20
204
20f
8.00 scconds
(a) Best-of-N Scaling
(b) Best-of-N 8B
(c) Best-of-N N 32B
Parcto Fronder Sparse)
Dense
Sparse
Sparse
Spines 0.
0.5
Scroe:
2.4
0.2
02
2rnee 0.15
Deagh: 0.08
0.0
0.0
10-1
B200 seconds
10c
Tora-eFUOPs
(d) Long-CoT's Scaling
(e) (LLon-Cools 8B
(f) Long CoTs 32B
Figure 8: Sparse Attention Boosts Test-Time Scaling. In (a) and (d), we show that sparse attention models significantly improve the cost-accuracy trade-off under both inference strategies, ultimately achieving higher problem-solving rates at lower computational budgets. In (b)e) and (e)(f,, we analyze individual model per- formance (SB and 32B) and observe that sparse attention provides notable gains. In low-cost regimes, it can enhance problem-solving rates by 50-60 percentage points. Even in high-cost regimes, sparse models maintain an adyantage of around 5 points, while reaching these performance levels much earlier. For reference, workload of 10 Vera-eFLOP- corresponds to approximately 22 seconds of full utilization on single B200.
OptimallCas Caneratirn Langtt
101
Tara L00s
10.7
Tita 10
(a) Best-of-N Gen.
(b) ofst-of-N Budget
(c) Long- Cooeg-CoTsCCo Gen.
(d) Long-CoTl Conss Budget
Figure 9: Tradeoff Between Generated Tokens and KV Budget. We empirically investigate how to balance the tradeoff between generating more tokens and allocating larger KV cache budget, which may yield more accurate but potentially shorter outputs. Using Qwen3-8B as a representative model, we fit curves to characterize this tradeoff. For Best-of-N, we find that for every doubling of the total compute cost, the optimal KV budget increases by factor of 1.18x, while the total number of generated tokens increases by 1.74x. For Long- CoT, the corresponding factors are 1.23x and 1.52x, cost is dominated by model parameter-relate computation rather than token generation or KV cache usage. We incorporate a model-specific constant into the cost model to account for this effect.
L.52x,repeectinn respectivelly Notably, when the KV budget small, the computational
Experimental Validation
In this section, we demonstrate the practicality of our sparse scaling law through block top-k attention. We report empirical improvements in task throughput (number of tasks performed per unit time) using our block-sparse implementation and conduct ablation studies with alternative sparsification strategies, such as local attention, to highlight the importance of the KV selection mechanism.
5.1 Block Top-k Attention
While t attention offers attractive theoretical scaling, it is computationally intractable in practice. Instead, we adopt block top-k attention for two key reasons. First, it exploits temporal locality in attention patterns 75 to retrieve semantically related key-value (KV) blocks. Second, its localized retrieval is hardware-efficient and integrates seamlessly with paged attention [46), enabling high- throughput decoding. In practice, we compute a representative vector for each KV block by averaging
8

---

0.8
0.8
Fanto Furkicr (Slcck Tope?
Parezo Fronier (Pemsl
0.8
0.7
Natio horba Slace bosk)
Na Hostia Merse
5313
0.6
Solving
0.6
lock
Aecurage
50.6
7005
0.4
0.4
0.
0.0
0.2
02
0.2
0.1
0.0
10'
10"
Tera-ef LOPS
10
10a
0.0
10a1
10-1
8200 seconds
100
0.0
10-2
0200 seconds
10-1
10
(a) Sparse Algorithm Comparison
(b) Best-of-N of-N Scaling
(c) Long CoT' Scaling
Figure 10: Sparse Attention Algorithms. In (a), we contrast the effectiveness block top-k attention against oracle top-k attention and local attention. In (b) and (c), we illustrate the optimality of block top-k sparse attention in terms of TTS on AIME24 dataset. Although upper bounded by the oracle top-k attention performance, block top-k achieves goed trade-off between effectiveness and tractability. Whereas, although easy implement, the performance of local attention substantially poor.
its key vectors, and use these to score the relevance of blocks to each query. Importance scores are shared across query heads within group, following the Grouped Query Attention (GQA) scheme.
Although local attention is more efficient due to its static sparsity pattern, performs significantly worse. Its poor test-time scaling prevents it from outperforming dense attention except in very low-accuracy regimes.
We compare block top-k with local attention in Figure 10a
Implementation. We build our inference backend on Flashinfer [94), incorporating support for paged attention [46] and continuous batching [95]. Alongside the paged KV cache, we introduce an auxiliary data structure store block-level average key vectors. The KV block size is chosen such that the memory load from the block-average vectors and the selected top-k KV blocks remains balanced. This design enables sub sub-quadratic KV loading cost as the number of reasoning tokens increases.
5.2 Empirical Results
We quantify TTS efficiency using lask throughpul, defined as the number of tasks completed per unit time. This metric is particularly relevant for reasoning tasks, where the utility of generation hinges entirely the correctness of the final output-unlike tasks such as summarization content creation, where partial outputs may still be useful. We illustrate the benefit of block top-k attention across different model sizes on 8 H200 machines with an extremely large batch size of 4096. As shown in Figure block top-k attention substan- tially improves task throughput, particularly for smaller models. For instance, the Qwen3-0.6B model achieves 12.6x to x increase in throughput as the generation length extends from 16k to 32k tokens. This improvement reflects the growing inefficiency of dense attention at longer contexts, which disproportionately affects smaller models. Thus, the use of sparse attention not only alleviates this bottleneck but also restores much of the practical utility of smaller models in resource-constraing settings by abling them to use more test-time compute more cost-effectively.
15.0
12.5
Throughout
10.0
5.0
2.5
2.38x
0.0
+
1.78
0.66
Model
Figure 11: Task throughput improvement with block top-I attention.
Conclusion and Discussion
This work introduces the Kinetics Scaling Law based on the insight that attention costs rather than parameter counts are the dominant factor in TTS. We demonstrate that sparse attention fundamentally reshapes the scaling landscape, enabling longer generations and significantly higher accuracy. We envision the Kinetics Scaling Law as a foundational tool for guiding end-to-end design across LLM serving, agent frameworks, and reinforcement learning environments. Kinetics Sparse Scaling may signal a new paradigm, enabling continued progress even beyond pretraining plateaus. While our analysis centers NVIDIA GPUs, the underlying principle that scaling memory bandwidth is more challenging and costly than scaling FLOPs applies broadly across hardware platforms. Ultimately, this study highlights the need for co-designing model architectures, test-time inference techniques, and hardware infrastructure as a critical step toward enabling the next wave of scalable L.M deployment.
We include discussions of limitations and broader impact in the Appendix of supplementary material.

---

Acknowledgements
We would like to thank Yong Wu, Xinyu Yang and Harry Dong for providing us constructive feedback on our paper and computing resources of NVIDIA. This work was partially supported by Google Rescarch Award, Amazon Research Award, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant No. 2326610 10. Any opinions, findings, and conclusions recommendations expressed those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
Llama 4 model card. 2025. URL https::/gittuu.. http://ttuub.ccmmmmetttlllaaaa [1] hoaiee//s/boob/mmimmomoeeeeee
CARD md
[2] Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025, URL https://arxiv. org/abs/2502.04443..
[3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long documdecument transformer. arXiv preprint arXiv:2004.05/50 2020.
[4] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le. Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint 2rXiv:2407.21777, 2024.
[5] Ruisi Cai, Yuandong Tian, Zhangyang Wang. and Beidi Chen. Lococo: Dropping in convolutions for long context compression. arXiv preprint arXiv:2406.05317, 2024.
[6] Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances in Neural Information Processing Systems, 34:17413-17424 2021.
[7] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Bean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013118 2023.
[8] Kevin Chen, Marco Cusumano- Lusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long- interactive Ilm agents. arXiv preprint arXiv:2502.0006 2025.
[9] Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models. arXiv preprint vrXiv:2505.10475 2025. URL httpl:ttt///// //tpvv/rxvvvv//////////e///a.vv. wwwpxiii..rg//bbba org/abs/ ab25 10475,
[10] Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient Ilm generation. arXiv preprint arXiv:2410.16179) 2024.
[11] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 1904. 10509, 2019.
[12] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 1et000099.4499. 14794, 2020.
[13] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expprrt language models. arXiv preprint arXiv:2401 1,6066, 2024.
[14] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR, abs/2307. 00022307.0861, 2023. doi: 10.48550/ARXIV.22000006 URL https:::s: :tltps://dof http://ioi..o/////////
48550/ar arxiv. 2307 08691
10

---

[15] Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song. Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations (ICLR), 2021. RR.. httpss///s//xii org/ bog/abs/2131 2212..211.... 00029
[16] Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficien exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference Neural Information P Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022.
2022.
[17] Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf: Efficient attention using asymmetric clustering. arXiv aeeprinn.axxh::2200055333.
[18] DeepSeek-AL. Dew.apeeeeeecccccoonnnniniaaaai 2025, http://aaw....o.o......otottteaa orge::/ize.ieee200mmmm//200/00e0
thing._.
[19] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 00318-30332, 2022.
[20] Danny Driess, Minh Nguyen. Fei Xia, et al. Palm-e: An embodied multimodal language model. arXiv preprint 2rXiv:2303.03378, 2023.
[21] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Dehao Chen, Yonghui Wu, and Jeff Dean. Glam: Efficient scaling of language models with mixture-of-experts. arXiv preprint
2021.
[22] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (1):5232-5270, 2022.
[23] Xidong Feng, Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309. 17179, 2023.
[24] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR, 2023.
[25] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.173235 2022.
[26] Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving Ilm reasoning programs with certaindex. arXiv preprint arXiv:2412.20993, 2024.
[27] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
[28] Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces. CoRR, abs/2312.007552 2023. doi: 10.48550/ARXXI.231220000 URL.Bttps://di..oo org/
10. 448500/aXXivv 2312.00752 00752
[29] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representationss ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
net/
[30] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseek-rl: Incentivizing reasoning capability in llms 2025.
via reinforcement learning, arXiv preprint
11

---

[31] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models reason in continuous latent space. arXiv preprint UrXiv:2412.06769, 2024.
[32] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241): 1):11444,,144 2021.
[33] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv: 2022.
[34] Coleman Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural In- formation Processing Systems, volume 37, pages 1270-1303. Curran Associates, Inc.. neur neur cc/ htcc/pppr__fft firefffiees///ee httpe/iisssaappp///////aeeee pappr//024/fflll 2024. URL oag.apaaeeeeccoooooooeeeeoeaorra Confer pdf 0B550cBB455500004611029--------2
[35] Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Efficient long-decoding inference with reasoning- aware attention sparsity. arXiv preprint arXiv:2502.111477 2025.
[36] Wenlong Huang, Fei Fei, and Chelsea Finn. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXXi:2201.07000 2022.
[37] Sutskever Ilya. llya sutskever: "sequence sequence learning leamning with neural networks: what decade". ottpaaaaaaataaaaaa////////////oa http://wwttttcccccaaaaaaaaaaaaaa
[38] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 2rXi::2412.667277 2024.
[39] Naman King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974. 2024.
[40] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401. 04088, 2024.
[41] Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Fu, Christopher Ré, and Azalia Mirhoseini. Hydragen: High-throughput Ilm inference with shared prefixes. arXiv preprint arXiv:2402.05099 2024.
[42] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2xXiv:2001.083611 2020.
[43] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5156-5163 PMLR, 2020. URL BRLLhtttp://rrreediiig mlr httpss///l1//saaarrooooooo/00000 html
[44] Nikita Kitaev, łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Machine Learning (ICML), 2020.
[45] Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, arXiv preprint arXiv:2411.04330, 2024.
12

---

[46] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. 06180
[47] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding, In International Conference on Machine Learning, pages 19274-19286.
PMLR, 2023.
[48] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis. and Deming Chen. Snapkv: Llm knows what you are looking for before generation, 2024. UtL.attps://axxi org/ aa///4/b//////// 14469
[49] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shward Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mann language model, 2024. https:::: http/v////vv///aa////a//aaaaaaaa org/ abs 2403. 19887
[50] Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang. Boyu Tian, Ion Stoica. Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning. arXiv preprint Xrxii22502.000 02770, 2025.
[51] Ji Lin, Jiaming Tang. Haotian Tang. Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[52] Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient IIm serving, arXiv preprint arXiv:2405.04555
2024.
[53] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint 1xXx::2440504434 2024.
[54] Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chengruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalattennioo: Accelerating long- context Ilm inference via vector retrieval. arXiv preprint arXiv:2409. 10516, 2024.
[55] Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22176 PMLR, 2023.
[56] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning- frre asymmetric 2bit quantization for kv cache. arXiv preprint 200i::2402.07750 2024.
[57] Wenjie Ma, Jingxuan He, Charlie Snell, Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.0988 2025.
[58] Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length-compressible chain-of-thought tuning. arXiv preprint arXiv:2502.09611, 2025.
[59]
MAA. 2024.
American invitational URL
mathematics examination 2024.
httpgt//aartofprobbemmmmllligggg com/ www.///ww//i.ioe...oo.e.eeeeeeee
php/ American Invitationale httpa///ooooottttaaaaaaaaaaaaaaa
[60]
MAA.
American URL
invitational
mathematics
examination
2025,
2025.
wttp:/g...ooooo///t.e....oeeeaet index
13

---

[61] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint or..i:23005009999 09781, 2023.
[62] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius, Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.083: 2021.
[63] Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: A chain-of- thought driven architecture with budget-adaptive computation cost at inference. arXiv preprint arXiv:2310.1083 2023.
[64] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings the 34th International Conference Machine Learning. pages 2498-2507. PMLR, 2017.
[65] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint (XXi::2001.19993, 2025.
[66] Reiichiro Nakano, Jacob Hilton, Jeffrey Wu, al. Webgpt: Bnwwerasiiteelpueiionnsseeeerers with human feedback. arXiv preprint arXiv:21 2021.
[67] Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M Ponti. The sparse frontier: Sparse attention trade-offs in transformer llms. arXiv preprint arXiv:2504. 17768, 2025.
[68] Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton- of of-thought: Prompting lims for efficient parallel generation. arXiv preprint wxXv:2237.15337 2023.
[69] Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y. Li, Aviv Bick, J. Zico Kolter, Albert Gu, François Fleuret, and Tri Dao. Thinking slow, fast: Scaling inference compute with distilled reasoners, 2025. URL htttltttt///rxtttto http://rrg//bbb/////eeeee 2502. 20339
[70] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606 624, 2023.
[71] Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, lan En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408. 11049, 2024.
[72] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- layer. arXiv preprint arXiv: orXiv:1701.063: 2017.
[73] Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pages 31094 31116. 16. PMLR, 2023.
[74] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint UrXiv:2408.03314 2024.
[75] Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding, 2024. URL.
https: /ttpiv///xvvvvvv org/ abs/ abs/////////2 11912
14

---

[76] Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang. Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2440.202290 2024.
[77] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec:
Massively parallel speculative decoding for interactive IIm inference consumer devices. Advances in Neural Information Processing Systems, 17:16342-16668. 2024.
[78] Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Lim pretraining with continuous concepts. arXiv preprint arXiv:2502.088 08524, 2025.
[79] Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. arXiv preprint arXiv:2406.1077 2024.
[80] NovaSky Team. Sky-tl: Train your own ol preview model within $450. https:/nnovasky-
ai iighhubbbooopsss//ttyy-l 2025. Accessed: 2025-01-09.
[81] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https:///wenlna: github. io/blog/quq-320
[82] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological,, (81)))667-288 1996.
[83] Ajay Tirumala and Raymond Wong. Nvidia blackwell platform: Advancing generative ai and accelerated computing. In 2024 IEEE Hot Chips 36 Symposium (HCS), pages 1-33. IEEE Computer Society. 2024.
[84] Junxiong Wang. Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M. Rush, and Tri Dao. MI: Towards scalable test-time compute with mamba reasoning models, 2025. URL https://arxivy org/ abs htlp..///2400.0
[85] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11
[86] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of- of-huuhf- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
[87] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.000 007244, 2024.
[88] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL http::ttts///rrttt. http:///aaxiiv..o/a//////aa
17453
[89] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and 2024.
Zhihao Fan. Qwen2 technical report. arXiv preprint 240.
[90]
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 205.::2412.11115,,0 2024.
15

---

[91]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Liannaao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. httptttss:/aaaa org/ abs/ 2505. 09388
[92] Shinn Yao, Jiaming Zhao, Dian Yu, et al. React: Synergizing reasoning and acting in language models. Advances in Neural Information Processing Systems (NeurlPS), 2023.
[93] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, Ad- vances in neural information processing systems, 36:1 809-11822, 2023.
[94] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang. Tianqi Chen, Baris Kasikei, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint arXiv:2501.01005 2025.
URL httph:tt////t //axxiv... wwwg/og//////.ooooo abtaa/aa//201 01005
[95]
Gyeong-In Yu, Joo Seong Jeong, Geon Woo Kim, Soojeong Kim, and Byung-Gov Gon Chun. Orca: A distributed serving system for Saansforme-Baase generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-9391.3-228- URL https //www.usenix. usenix. org/ www.oorggg/ooeeeeeeeeee
[96] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei. Lean Wang. Zhiping Xiao, et al. Native sparse attention: Hardware-aligma and natively trainable sparse attention. arXiv preprint arXiv:2502.1 1089, 2025.
[97] Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, et Llm inference unveiled: Survey and roofline model insights. arXiv preprint ar A40i 16363, 2024.
[98] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33, 2020.
[99] Amir Zandieh, Insu Han, Majid Daliri, and Amin Karbasi. Kdeformer: Accelerating transform- ers via kernel density estimation. In International Conference on Machine Learning, pages 40605 40603. PMLR, 2023.
[100] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. urXiv preprint arXiv:2205.0106 2022.
[101] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy- oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661 111111331101111 10, 2023.
[102] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph Gonzalez, al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-62555 2024.
[103] Kan Zhu, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Yufei Gao, Qinyu Xu, Tian Tang, Zihao Ye, et al. Nanoflow: Towards optimal large language model serving throughput. arXiv preprint arXiv:2408. 12757, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper diseuss the limitations of the work performed by the authors?
Answer: [Yss]
Justification:
We discuss the limitations of the work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to -to-txtt system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If apppiicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each thcoretical result, does the paper provide the full set assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
17

---

Justification:
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce We include complete assumptions and proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes].
Justification:
The paper has disclosed all the information in the method and experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soue source models, it may that access to the model is limited some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[Yes].
Justification: We include our code in the supplementary material and plan to publicize the code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Soblliccguiddes/Cdeettbmm..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
sewpiidded/ooSeSbmii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]].
Justification:
The experimental setting clearly described in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance the experiments?
Question:
Answer: [Yes]
Justification: All evaluation is repeated four times to calculate the average accuracy. We also evaluate our method on various settings to verify our conclusion.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative eπхоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes].
Justification:
The compute resource information is included in the experimental setting section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of
Answer: [Yes].
Justification: The research conducted the paper fully conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there a special considera- tion due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We discuss the broader impacts of our work in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Answer: [NA].
Justification: Our paper does not introduce any assets that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: We have explicitly mentioned the citations for the datasets and have ensured that all conditions are fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtpsestts.pppsswwwii.... thcode. www.ooo///cooo dndasatasett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is anonymized and provided the supplementary materials. Docu- mentation includes basic usage and setup instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/coddemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
We do not include any experiments with human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve human subjects or crowdworkers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration not required.
Answer: [NA]
Justification: We only use LLM to polish paper writing.
Guidelines:
The answer NA means that the core method development this research does not involve LLMs any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
23

---

Appendix
Table of Contents
Cost Model
Max Cost Model v.s. Additive Cost Model
Details about Sparse Attention Cost Model
26
Dense Scaling Law
26
Additional Benchmarks
26
Additional Reasoning Models
28
Sparse Scaling Law
28
Additional Benchmarks
Additional Analysis
Experimental Details
Estimate Cost, Accuracy and Solving Rate
30
Greedy Algorithm for Optimal Resource Allocation
Top-K Attention and Block Top-K Attention
Extended Related Work
Limitations, Future Scope, and Broader Impact
A
Cost Model
In this section, delve into the cost models used in the Kinetics Scaling Law. We show empirically that adopting max cost model does not alter the scaling behavior and outline methods for calculating the cost of sparse attention models.
A.1 Max Cost Model v.s. Additive Cost Model
Max cost model is widely used in performance modeling 197. It assumes that computation and memory operations can be fully overlapped with each other and only considers the bottleneck operation for cost measurement.
C maax-coos max Coompp Ciem I)
where Ceomp denotes the compute cost, Cmem the memory cost per access, and / the memory intensity.
In this section, analyze the Kinetics Scaling Law using the max cost model, For clarity, we refer to the cost model Ceomp + Ciemm x I, which used in the main paper, as the additive cost model.
We draw two conclusions from empirical results under the max cost model:
Kinetics scaling law for dense models still holds. We re-plot Figure (b) and Figure under the measurement of max cost models in Figures and 13 We find except that in Long-Co'l scenarios, large models become slightly more effective low-cost regime (with accuracy-00.3) the overall trends are very close to the plots with additive cost models.
Sparse attention solves problems more cost-effectively. re- Figures Saland 8d Figures 14a and 14b Under the max cost models, in Long- CooTs the accuracy and efficiency gaps increase gaps widen from 65 points and 10.67 x to 69.4 points and 19.64x. These results indicate that under the max cost model, our claim that sparse attention can enhance problem-solving performance is strengthen. Compared to dense attention models, sparse attention models tend to have more balanced memory and compute costs. Thus omitting one of them via a max cost model will favor sparse attention models.
from 47.5 points and 11.21x 52.8 points and 15.71 respectively. In Best-of-N,

---

DAe03-220
Qwen>-148
15000
Qwen1-80
Owen3 48
0.4
0w
10000
Owen3-1.77
3-0.8-0.88
Qwen3-0.8 0:68
5000
]0] Bra--eLOPP
P)) Trr--FLDD
0.8
20000
0wer3-248
Gereration
Qwen1-65
0.4
Qwen3-1 1 7B
Owen3-1.
Qwend 0.88
20000
Ower3-0.86
5000
(6) krr-
(1) Tara
Figure 12: ATME Pareto Frontier (Long-CoTss) with Max Cost Models. (a)(b) is the original plot with the additive cost model. (c)(d) is the corresponding plot using cost models. Compared to the original plots, the overall trend is similar except that larger models span a slightly broader region on the Pareto frontier. For example,
model now consistently outperforms the 4B model with noticeable gap around accuracy and maintains dominance thereafter. In contrast, under the additive cost model in Figure 4 [)), the two models alternate in performance until accuracy exceeds 0.4. This suggests that, when evaluated using max cost model, larger models appear slightly more efficient relative to their performance under additive cost models.
0.8
Solving Rate
0.6
Qwen3-328
Qwen3-148
0.4
Qwen3-88
Qwen3-48
0.2
Qwen3-
Qwen3-0
10a
10
10a
10
Tera-eFLOPs
Figure 13: AIME Pareto Frontier (Best-of-N) with Max Cost Models. We re-plot max cost models, The Pareto Frontier is very similar under different cost models.
0.8
Frontier (Sparse)
Frontler (Sparse)
1964
Pareto Frontier (Dense)
Pareto Frontier (Dense)
0.6
Accuracy
Accuracy
10.6
69/4
0.4
52.8
0.4
0.2
0.0
0.0
10-3
10-2
10-1
10a
10-2
B200 seconds
10-1
10a
101
8200 seconds
(a) Long-CoTs
(b) Best-of-N
Figure 14: Sparse attention scales significantly better under max cost models. re-poo Figures Salarde using max cost models. Compared the original plots, the performance and efficiency gaps between sparse attention models and dense models hecome more pronounced. In Long-CoT's, the accuracy and efficiency gaps increase from 47.5 points and 11.21 x to 52.8 points and 15.71 x, respectively. In Best-of-N, N, the gaps widen from 65 points and 10.67x 69.4 points and 19.64x.
25

---

Qwen3-328
Optima Gereell-rrreeeee UP
25000
Owent-145
Owen3-328
Owns3:88 8B
Gwend-4
Owen3-48
Owend 1.70
15000
Ower3-1.70
Qwend-0aa 0.40
10000
3-058
5000
10
127
10'
TaraFFLPPs
Owen3-228
Queen--125
Owe
25000
Qwen3-148
Ower3 85
Qwen1-80
Owen3-45 3-5
20000
Gwen3 48
Owe:3-1.70
Qman2-1.778
Qwen3-00 0.58
10000
5000
10
10
Figure 15: AIME25 Pareto Frontier (Long-CoTss) We conduct the same experiments as Figure
A.2 Details about Sparse Attention Cost Model
Sparse attention models follow different cost functions due to the sparsification of KV memory access. In this paper, we focus on algorithms that impose uniform KV budget (denoted as B) per attention head for each decoded token. We consider Lin 2 B for the sake of simplicity. Under this setting, the cost model for sparse attention is given by:
Sppase 2N 2N +2r NIN
(8)
compute
momory
In practical implementations, must also account for the overhead associated with retrieving searching KV memory, denoted as Cwearch, which depends on the specific sparse attention algorithm A. For example, in block top-k selection, the search cost is:
2N Lin DI.m +DDD 2Block-Size
21 DI om + +NDD2
Ceaarhh
(9)
2Block-Size
compute
memory
In our work, we choose the Block-Sin Size in such a way that Csparse and Ceerch are roughly balanced, so that the sparse attention cost increases sub-linearly with generation length.
For local attention and oracle top-k attention, we assume no search overhead, i.e., Cwearch =0
Many sparse attention algorithms skip the first layer [79 TO 101, resulting in only minor increase in total cost. For the Qwen3 series, this additional overhead bounded by 3.57% for the 0.6B model and by 1.56% for the 32B model.
B Dense Scaling Law
In this section, we further verify Kinetics Scaling Law for dense models proposed in Section 3 with extended experimental results of different benchmarks and model series.
B.1 Additional Benchmarks
We evaluate on AIME25 in Figures 15 and to 16eland LiveCodeBenct in Figures 17 and
8 18c
(excluding the 0.6B model), following the setting described in Section 3 The empirical results support Kinetics Scaling Law: across both benchmarks, the 0.6B and ..7B models are consistently less effective, and the Pareto frontier is almost always dominated by the 14B models.
26

---

Cietic Scalin Law
0.80
0.8
Nevous Scalin Law
0.6
dwmn3 228
0.6
Owent 325
Salving
0.4
Qwen2-10
0wm1-340
0.4
Geen12-0 -3
Gwe11-110
15
Gwen3 48
Owe11 45
10
0.2
@wen-- 18
Gwens-1 78
9wen2-2.68
Cwant
10a
10a
10a
10'
10'
10"
10
10*
0.0
0.2
0.4
0.6
0.8
Tera-eFLOPs
Selving Rate
(a) Accuracy (eFLOPs)
(b) Accuracy (FLOPs)
(c) Optimal Models
Figure AIME25 Score Curve (Best-of-N)), We conduct the same experiments Figures Salressaalo to
Ower3 328
25000
Owen3 32B
O-w:3-14B
Owwn33148
Ower3
20000
2.44
u.a
0.2
10000
5000
kra-FLOPs
20000
Gwen3-140
Cown3-318
Mar
0.44
Ower3
9oar3-485
Optinal Generadon
15000
0.3
Ower3-1..5 70
Qwen3-1.77
0.2
5000
Tete-
Figure 17: LiveCodeBench Pareto Frontier (Long-CoTs). We conduct the same experiments as Figure
0.8
0.7
20.6
0.7
@ 25
Sia
0.5
0..5
20
0.4
20.4
Model
Owml1-4
00.2
Own'1
Swml1-1
Qwn11-43
3.78
Owor2-1. L.78
0.0
10
100
10a
10
10
0.0
10
10a
10a
10
LOPs
Tera-FLDPs
Solving Rate
(a) Accuracy (eFLOPs)
Accuracy (FLOPs)
(c) Optimal Models
Figure 18: LiveCodeBench Score Curve (Best-of-N), We conduct the same experiments as Figures 6a to 6c
DeepSenk-RR Diell--wes11
Billl Lnepiiee AL dr Ovee
204
OceoScak-AL RL Qweeli-wween322 328
Geep -i--ee----o--114
Generaiinn
Lompiiee' Uniil dhee
5000
10a
10a
101
FLOPs
FLOPs
20000
204
Seek Owen
Seok- Disll-Owen-33 328
35000
DespSess-83 43 Dutil OwEn-7
Decebeek
Deepheak-Kt #3 Duail 1.0
Bo
30000
0.2
5000
10
10
10"
20
10
10-
Figure 19: AIME24 Pareto Frontier (Long-CoTss. We conduct the same experiments Figuro DeepSeck Distilled Qwen series.
27

---

0.8
Knolks Scaling Low
Pevioug Sia Law
Ranc
20
0.4
05-338
325
Model
0.3
04:
148
0.2
05-18
D5:15
0.1
05-158
D5-158
10'
10'
10a
10
10
10'
10"
10a
0.2
0.4
0.8
Tera-eFLOP
Tera-F LOPs
Solving Rate
(a) Accuracy (eFLOPs)
(b) Accuracy (FLOPs)
(c) Optimal Models
Figure 20: AIME24 Score Curve Envelope (Best-of-NN, We conduct the same experiments as Figures 6a on DeepScck Distilled Qwen series.
6c
B.2 Additional Reasoning Models
In Figures 19 and 20a to 20c we evaluate DeepSeek-RI Distilled Qwen models (abbreviated as DS
The DeepSeek series models further demonstrate that previous scaling laws-thoss those based on PLLOP--sinnfficntie overestimate the effectiveness of the 1.5B model. As predicted by the Kinetics Scaling Law, increasing the number of generated tokens for the 1.5B model is less effective than scaling up the model size, such as using the 7B or larger variants.
models) (3)] AIME24.
Interestingly, observe shift in the emerging model size: unlike Qwen3, where the 14B model dominates, the 7B model becomes the dominant choice the DeepSeek series. In Figures 19 200 20c the 7B model spans most of the Pareto frontier, and Figure s9]ss that 7B models with long CoTs are more efficient and effective than 14B models with short generations. We attribute this an architectural outlier in the DeepSeek-RI (Qwen2.5) model series. As shown in Table 2 the DeepSeek-R1 7B model is significantly more KV memory-efficien than the Qwen3-8B model. Unlike most model series illustrated in Figure 5a where KV cache size typically grows sublinearly with respect to model parameters, DeepSeek-BR Ri shows a deviation from this trend: the 14B model has approximately 3.4x more KV memory than the 7B model, while having only 2x more parameters.
Table 2: KV memory Size for Qwen3 and DeepSeck-RI Distilled models (per 32K tokens, unit: GB).
Qwen3
Qwen3-1.7B
Qwen3-8B
Qwen3-
14B
Qwen3-32B
3.5
4.5
8
DeepSeek DS-1.5B
DS-7B 1.75
DS-14B
DS-32B
0.875
8
This finding highlights the importance of concrete model architecture design, rather than focusing solely the number of model parameters. Whether KV memory size is directly related to reasoning performance remains an open question, which we leave for future investigation.
C Sparse Scaling Law
We present additional results supporting the kinetics sparse scaling law across multiple tasks and demonstrate how these insights enable scalable test-time scaling with sparse attention.
C.1 Additional Benchmarks
Live- CodeBench features complex programming problems from recent coding contests, while AIME25 consists of challenging math problems. In both cases, sparse attention ceanttionppatiiullart oracle top- s--consitentlly outperforms dense attention. Block top-k attention, tractable alternative, closely oracle.
Beyond AIME24, we evaluate our approach on LiveCodeBench [39 and AIME25 60
matches the performance
'For Livc dataset, we have sampled 50 examples from the v5 subset consisting 167 examples. Our subset comprises 24 hard, 16 medium and 10 easy examples respectively.
28

---

0.8
0.7
topk:
dence
Fares Forterr Chpd
20.0X
0.8
Panea Forttrr Docc Toga
2 0.5
g 0.6
black
0.6
20.5
20.4
1.22
0.3
8
0.1
0.0
10"
Tera-oFLOPs
10.
10a
10
10-3
10-1
0200
0200
207
(a) Best-of-N Scaling Comparison
(b) Best-of-N Top-K Sparse Scaling (c) Best-of-N Block Top- K Scaling
0.8
0.7
0.5
Fares Froeier (eeree
0.5
Panda Foonter Derool
0.6
0.5
0.4
0.8
0.3
0.3
8 0.2
dense
0.2
0.1
block
0.0
10a
10
0.0
0.0
Tera-eflops
8:00 seconds
B200 seconds
(d) Long-CoT's CoI's Scaling Comparison (e) Long-Cool CoI's Top-K K Sparse Scaling (f) Long- CoTss Block Top-K Scaling
1.0
0.6
0.5
0.3
0.2
0.0
0.0
0.0
20
10a
101
10"0
207
201
20
10T
101
(g) Best-of-N Scaling (Easy)
(h) Best-of-N Scaling (Medium)
(i) Best-of-N Scaling (Hard)
Figure 21: LiveCodeBench Sparse Scaling, We evaluate sparse scaling laws for Qwen3-14D 14B model using oracle top-k and block-top-k attention the LiveCodeBench dataset. (a)(d) compare block-top-k and oracle top-k with dense scaling under Best-of-N and long-CoT TTS settings. (b)(e) show cost-accuracy tradc-offs for top-k attention. (c)(f) show tradc-offs for block-top-k attention. (i) compare the oracle top-k scaling for casy, medium and hard difficulty questions.
For LiveCodeBench, we sample 50 problems from the v5 subset (24 hard, 16 medium, 10 easy). As shown in Figure 21] oracle top-k attention can achieve 10x speedup in high-accuracy regimes and improves coverage by 10-30% in low-cost regimes. Conversely, the tractable alternative, Block top-k yields 5-6x x speedup and 30-40% coverage gains. We further show how the benefits of sparse attention scale with problem difficulty (Figures 21 212
Figure 22 confirms similar trends for AIME25, with substantial gains in both accuracy and efficiency under sparse attention.
C.2 Additional Analysis
Fixing a model (e.g., Qwen3-88) we investigate the tradeoff between generating more tokens through Best-of-N and increasing the KV budget in Figures 22 to 23d As the figures suggest, on AIME25, each doubling of total compute cost increases the optimal KV budget by 1.13x, while generated tokens grow by 1.67 x; on LiveCodeBench, these factors are 1.14x and 1.89x, respectively. We find that although the concrete numbers depend on the types of tasks, the overall results confirm our suggestions in the main paper that allocating compute toward generating more responses is generally more effective than expanding KV budget, highlighting the scalability of sparse attention.
D Experimental Details
In this section, explain the details about experiments.
29

---

0.8
Tine frenter Fihhh
0.8
Fares Torrer Bick Topo
2.5
50ran
10
201
10-1
15
10t1
12
D200
5250
(a) Best-of-N Scaling Comparison
(b) Best-of-N Top-K Sparse Scaling (c) Best-of-N Block Top-K K Scaling
0.7
0.7
Panao Frontee (Denee
0.7
Faneo Porter Dareet
0.5
2.5
6.2
0.2
0.2
6.
0.1
0.3
Tera-flops
6209 secands
5250 secends
(d) Long- CoTTs Scaling Comparison (e) Long-Coti CoTT Top-K K Sparse Scaling (f) Long-CoT's Block Top-K Scaling
Figure 22: AIME25 Sparse Scaling, We evaluate sparse scaling laws for Qwen3- 14B model using oracle top-k and block-top-k attention on the AIME25 datasct. (a)(d) compare block-top-k and oracle top-k with dense scaling under of--- and long-CoT CoT settings. (b)(e) show cost-accuracy trade-off's for oracle top-k attention. (c) show trade-offs for block-top-k attention.
10
Ik 2-2 BE 161
Oginall CeREe on Tooa
Optin Cerersce Toare
Opaaa to Taarpets
Oytema Fo turponts
10
327
(a) AIME25 Gen.
(b) AIME25 Budget
(c) LiveCodeBecc Gen. (d) LiveCodeBencc Budget
Figure 23: Tradeoff Between Generated Tokens and KV Budget. We empirically characterize the tradcoff between increasing generation length and allocating a larger KV cache budget using Qwen3-8B. For AIME25 (()) and LiveCodeBench (())d), we identify the optimal KV budget and generated tokens (defined as number of reasoning trials times the average generated tokens per trial) achieve the highest problem-solving rate under every cost constraint C.
D.1 Estimate Cost, Accuracy and Solving Rate
When empirically measuring cost, one major challenge is the difficulty of controlling the actual generation length. Although it is possible to set an upper bound on the number of generated tokens, there is no guarantee that the model will utilize the full budget. For instance, in our Best-of-N experiments, we set the maximum number of generated tokens to 32,768, yet the average generation length was only 14K-16K tokens.
Furthermore, it important to model the relationship between actual inference cost and performance metrics, such as accuracy in Long-CoTTs allowed generation length to estimate cost can substantially underestimate the efficiency of models that solve problems with much shorter responses-aa ability that may reflect higher capability.
solving rate in Best-of-N. Relying solely the maximum
To address this challenge, we first sample S independent reasoning traces r1,"2....,'' from model M on task T,, with the maximum allowed number of tokens set to n. We slightly generalize Equation
30

---

as:
CrTs=2N PN Din + DEIL
+ Lou INDE
= +bB +c,
(10)
where a, b, and c are constants determined by the model architecture and test-time strategies (e.g.. .. the value of n). The expectations are estimated from the sampled traces, whose distribution is influenced by the model M, the token limit ni, and the task T.
For Long- CoTss we fix N = in Equation (IO) and vary n. From the sampled traces, we estimate the accuracy (Pass @ 1), and compute the corresponding cost by substituting the empirical values of Loud and E 2 measured under each n.
For Best-of-N, fix 302, 768, and estimate the solving (Pass@ K) following the methodology of Brown et al. The corresponding cost is then computed by substituting N K into Equation (TO).
Similarly, can estimate the cost for sparse attention models using Equations and 9
Advanced control of generation lengths is active area of research 91 65 57
but it is beyond the
scope of this paper.
D.2 Greedy Algorithm for Optimal Resource A Allocation
We describe the procedure for identifying optimal resource allocations and establishing the Pareto frontier for sparse attention models in Algorithms [ as supplement to Section Given a fixed cost constraint C, we perform grid search over key parameters: KV budgets and either reasoning trials or maximum generation lengths.
Empirically, we sweep over KV budgets (32, 64, 128, 256, 512, 1024]; reasoning trials {1, 11,2.4 4, 8, 16, 32) (with a reduced upper limit for the 14B and 32B models to save computation time); and generation lengths (2k, 4k, 6k, 8k, 10k, 12k, 14k, 16k, 18k, 20k, 22k, 24k, 26k, 28k, 30k, 32k).
By varying the cost constraint C' in Algorithms] 1 and 2 obtain the performance of sparse attention models under optimal resource allocation, as shown in Figures Sa to 8f and toa 10c
It is important to note that we do not consider inter-request resource scheduling strategies, such as early stopping or dynamic reallocation across requests [26), since we aim to ensure fairness across all inputs. Instead, the cost constraint C' is interpreted the maximum allowable cost per request (not the average), even some requests achieve saturated accuracy below that threshold.
D.3 Top-K Attention and Block Top-K Attention
In this section, we explain the sparse attention algorithms discussed the main paper, namely Top-K Attention and Block Top-K Attention.
During the decoding phase of large language model (LLM), the self-attention mechanism computes weighted average of past values follows:
qk'
o= SoStmax
V V= qeRixd, K,Ve
WERIXn
(11)
where d is the head dimension and n is the context length. The key and value matrices are given by K A...........kk V= [U1 (W,, where each k, E RRldd are cached from previous decoding steps.
Top-K Attention. Top- K Attention is sparsification method where only the K most relevant tokens (i.e., those with the highest attention scores) are selected to compute the output. Formally, instead of computing the full softmax, we define a sparse attention weight vector:
exp(x1)
if EZK,
gk?
W1
JETg exp(e;)
where 8; Va
IK= Ixp
(12)
0
otherwise,
Here, Ix denotes the indices of the top K attention scores 8;. By masking out the less important positions, this approach reduces the computational and memory cost attention from ((n) to O(K),
where K <.
31

---

Algorithm 1: Best-of-N optimal resource allocation under cost C
Data: Tasks T, KV budgets (....., B,.. trial counts (N......NN.) cost limit C Result: Average of maximum accuracy task under cost C AccumBestAce+- Counte0;
for task T in T do
for KV budget B₁, do Generate S max ] responses using B3 for task T; for trial count No
(T).
compute cost Cba" (T)
if b.a SC then
8
Compute accuracy Acc b,e
(T)
= -aass @ Na;
if Acc (T) BestAcc then
10
BestAcc + Ace),a
11
end if
12
end if end for end for AccumBestAco + BestAcc; Count 1;
13
14
15
16 end for
AvgBestAcc=Aae AccuBestAcc//Contt
18 return AvgBestAcc;
Block Top-K. K. Block Top-K K Attention is a block-level sparse attention mechanism. Instead of selecting individual tokens based on attention scores, this method selects entire blocks of tokens, thereby reducing the number of attention computations.
Specifically, assume the full sequence of n keys is divided into BLOCK_SIZE of size BLOCK_SIZE:
consecutive blocks, each
SIZE:
K = kk] + K
K, E R SIIEEx
For each block K,, we first compute the average key vector:
BLOCK SIZE
1
Σ
kj.
BLOCK_SIZE
Next, compute the attention score between the query q and each block's 's average key;
qk
for for
We then select the top K' Tor=TToKKgr Attention is computed only over the tokens within the selected blocks. The sparse attention weights are defined as:
SIZE blocks based on the scores 8,, denoted by the index set
exp(sz)
if if tokens in selected blocks,
Le expley)
Wi
otherwise
For both algorithms, K is the KV budget. For GQA, we conduct an average pooling across all the query heads in group, ensuring that the total number of retrieved key-value vectors does not exceed the allocated KV budget.
E Extended Related Work
has been comprehen- sively studied reduce the attention cost when processing long sequeces. In parallel, approaches like FlashAttention [16 [4] accelerate attention by maximizing hardware efficiency. To address the
Efficient Attention. Sparse attention
44
10
101
88
96
67
48
32

---

Algorithm 2: Long-CoTs optimal resource allocation under cost C
Data: Tasks T, KV budgets ( Bj gen. lengths (M.,....,kk,, samples S, cost limit C Result: Average of maximum accuracy task under cost C
AccumBestAce+- <- Count
for task I' in T do
BestAcc<-0;
for gen. length nl do
for KV budget B, do
Generate S responses using (B,, compute cost (T), h,a"
if 0,a SC then
(T)
8
Compute accuracy Ace b,e = -Pass@1
if Acc (T) BestAcc then
10
BestAcc + Ace),a
11
end if
end if end for end for EccumBestAco+-Bert Count 16 end for
12
13
14
15
AvgBestAce= AccuBeetAcc//Connt
18 return AvgBestAcc;
quadratic complexity of standard attention, researchers have also explored linear attention architec- tures [28] (2) 43 12]. Additionally, quantization and low-precision methods [56] 34 52 have been
broadly applicd for improving inference efficiency.
Efficient Inference. Orca [95]. vLLM [46). and SGLang 102 are widely adopted to enhance the efficiency of LLM serving. Our analysis builds on the practical designs and implementations of these systems. In parallel, speculative decoding 47 7
71 has been proposed to mitigate the ceanory-banwwicee bottleneck during LLM decoding. Additionally, model compression and techniques are playing a crucial role in democratizing LLM deployment.
61
offloading 19 51,
Efficient Test-time Strategies. Optimizing reasoning models to generate fewer tokens has been shown to directly reduce inference-time cost [80] 2 5
Recent work such as CoCoNut [31] and CoCoMix (78) explores conducting reasoning latent space. thereby reducing decoding time. Methods like ParScale [9]. Tree-of- Thoughts [93]. and Skeleton-of -houhhhss [68] aim to improve efficiency by enabling parallel reasoning. Architectural innovations such CoTFormer (63 further enhance efficiency by adaptively allocating computational resources across tokens. Efficient reward-
model-based 87 76 test-time scaling algorithms are also comprehensively studied.
F Limitations, Future Scope, and Broader Impact
Limitations. Our experiments primarily focus on Qwen3 9 and DeepSeek- CeepSeek-RI-Distillee
Qwen [30]. two state-of-the-arr pretrained reasoning model series, evaluated from the inference perspective. However, the effects of training and post-training strategies are not fully explored and may influence the performance gaps and robustness to sparse attention mechanisms. In addition, our cost analysis assumes cloud-based serving environment, where computational resources are typically sufficient and large batch sizes are feasible. In contrast, local deployment scenarios, such as those using O often face limited VRAM where access to model parameters can dominate inference costs. Smaller models may be more appropriate in such settings, and our findings may not fully extend to these use cases.
Future Scope. Our sparse scaling law offers valuable insights for enriching the applications of sparse attention algorithms and the design space of test-time scaling strategies. On one hand, except for top-k,
https::/giithbb.comm
33

---

median
median generation langth
(a) gen length vs Nopt correlation (top-k attention)
(b) gen length vs Nopt correlation (block top-k attention)
Correlation between Generation Length and Number Trials. Longer generations correlate strongly with the optimal number trials (Nopt). serving as a proxy for problem difficulty. (a) shows this trend for top-k and block top-k attention on the AIME24 dataset using the Qwen3-8B model.
Figure 24:
currently we only discuss simple variant, i.e., block top-k, and have already demonstrated strong scalability. More advanced sparse attention algorithms [79] 10 96 [5] are emerging these days. We do believe they can eventually push the scalability test-time scaling to much higher boundary. On the other hand, test-time scaling algorithms are proposed to adaptively allocate computation to tasks, or
even to tokens [2 63 58 57). Extending them towards new resource allocation problems in sparse attention is critical to reach the limit of Kinetics sparse scaling law. For instance, since generation length strongly correlates with the optimal number of trials under sparse attention shown Figure 24). it can be used as dynamic signal to adjust the number of trials and KV budget. Moreover, sparse attention drastically reduces inference cost, enabling more reasoning trials and longer generations. This unlocks greater flexibility in configuring TTS strategies within fixed resource budget.
Broader Impact. This work aims to contribute to the understanding of efficiency and scalability challenges in the test-time scaling era, spanning model architecture, system-level implementation, and hardware design. We highlight the central role of sparsity in addressing these challenges. Our study is algorithmic nature and does target specific applications. While large language models can be misused in harmful ways, this work does not introduce new capabilities or risks beyond those already present in existing systems. Test-time scaling can consume a substantial amount of energy. raising concerns about the environmental sustainability of widespread deployment. By promoting sparse attention, our work hopes to help reduce the carbon footprint and energy consumption of inference systems and support the broader goal of sustainable AI.
34


Reviews:
[{"review_id": "237koLyf0t", "reviewer": "Reviewer_G6EY", "summary": "The paper challenges conventional test-time scaling laws by accounting for memory access costs, not just FLOPs, and shows how attention dominates inference costs. \n\nThe proposed “Kinetics Scaling Law” and the emphasis on sparse attention provide a fresh, impactful perspective.\n\nMany quantitative studies show a clear analysis flow to readers.", "questions": "See above weakness\n\nand \n\n1. TTS has multiple meanings in this paper, and is hard to follow sometimes. Test-time scaling, total token sampling\n\n2. Real-world latency benchmarks? \n\n3. Add a brief intuitive explanation or visualization of how eFLOPs integrates compute and memory\n\n4. The paper shows block Top-k sparse attention provides the best trade-off, but the mechanism relies on heuristics such as block averaging and grouped query attention. The KV block size and selection granularity seem critical to the performance but are not deeply analyzed.\n\nHow sensitive is the performance of your sparse models to KV block size and grouping strategy? Could this become a tuning bottleneck when transferring across tasks (e.g., from code generation to math reasoning)? Have you considered learning the sparsity patterns or integrating them with routing mechanisms like in MoEs?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Clear motivation backed by tons of quantitative studies\n\n2. tremendous performance gains\n\n3. Block Top-k sparse attention (Section 5) is shown to improve throughput significantly—up to 25× for small models, enabling high-efficiency inference.\n\nWeakness\n\n1. The scaling law threshold of 14B is specific to Qwen3 models; it’s unclear how well this generalizes to other architectures (e.g., Mistral, LLaMA3), limiting generality.\n\n2. Want to see some real-world case studies\n\n3. In Equation (2), the memory cost assumes prompt KV cache is amortized across reasoning trials, i.e., shared across N samples. However, in realistic settings (e.g., retrieval-augmented generation or API-style usage), prompts can vary significantly between samples.\n\nHow does the Kinetics Scaling Law behave under non-shared prompts or in settings with dynamic or partially overlapping prompts? Can your model generalize to multi-user or heterogeneous prompt settings common in LLM deployment?", "was_revised": true, "final_justification": "The authors have addressed all of my concerns.\n\nAlso, I have carefully read other reviewers' comments.\n\nBased on both of them, I decided to increase my score.", "rebuttal": "Thank you very much for your thoughtful review and constructive suggestions. We are glad that the reviewer found our work to offer a **fresh** and **impactful** perspective, with **clear motivation** and sufficient quantitative support, along with **significant performance gains**. We have tried to address your questions carefully. We hope the reviewer will consider raising your score in light of our response.\n\n## W1: Scaling law threshold generalization\n\nWe thank the reviewer for this insightful question. In the main paper, we use Qwen3 series as an example which has 14B threshold, but the threshold varies for other model families depending on their architectural details. As shown in Appendix B.2 (Page 14), our analysis of the **DeepSeek-Distilled-Qwen2.5 model** yields a different threshold of **7B**. However, this still supports our general conclusions,\n1. Small models are often overestimated in effectiveness.\n2. Scaling model size up to a certain threshold can be more beneficial than using more samples or longer CoTs for smaller models. Although the threshold might vary across model families depending on model architecture, the existence of such a threshold generalizes.\n\n**Analysis.** We further analyze the differences between the two model series and identify model architecture design as the key contributing factor. In particular, architectures that achieve ***slower KV memory growth across model sizes*** can shift the threshold to larger models. The table below shows the KV memory sizes for Qwen3 and DeepSeek-R1 Distilled models.\n> KV Memory size (GB) per 32K tokens\n\n| Qwen3 | 1.7B | 8B  | 14B | 32B |\n|---|----|---|---|---|\n| KV memory size | 3.5  | 4.5 | 6   | 8   |\n\n| DeepSeek Distilled | 1.5B  | 7B   | 14B | 32B |\n|---|---|---|---|---|\n| KV memory size | 0.875 | 1.75 | 6   | 8   |\n\nAs shown in the table, the DeepSeek 7B model is significantly more KV memory-efficient than the DeepSeek 14B model: the 14B model has  approximately $3.4\\times$ more KV memory, while having only $2\\times$ more parameters.  While in other model series (LLaMa3, Qwen3, OPT), the KV cache size grows sublinearly, with respect to model parameters (Figure 5a, Page 6). \n\n> As Mistral and LLaMA3 series do not have sufficient reasoning models to conduct the qualitative analysis, we do not include them. We welcome any suggestions from the reviewer for future evaluation.\n\n\n## W2 & Q2: Real-world latency benchmarks\n\nWe implement block-topk attention with Flashinfer and torch compile.\n\n**Key Results.** We report Qwen3 decoding latency using 8xH100 GPUs. The KV Cache length is 32K tokens. The following table compares dense vs. Block Top-K attention across **different KV budgets**:\n\n> **Note**: When OOMs, we estimate total model latency using per-layer measurements.\n> *milliseconds (ms) per decoding step.*\n\nBatch Size: 1024\n| Model | Dense | KV budget=1024 | 512   | 256   | 128   |\n|---|---|---|---|---|---|\n| 0.6B  | 152.76     | 14.47        | 9.23  | 6.24  | 4.54  |\n| 1.7B | 153.33     | 15.15        | 9.89  | 6.84  | 5.21  |\n| 4B | 198.48     | 20.59        |13.85  |10.06  | 7.92  |\n| 8B  | 199.74     | 21.73        |15.30  |11.35  | 9.20  |\n| 14B | 225.13     | 27.77        |20.22  |15.83  |13.63  |\n| 32B | 363.46     | 47.92        |38.75  |29.15  |25.02  |\n\nBatch Size: 128\n| Model | Dense | KV budget=1024 | 512   | 256   | 128   |\n|---|---|---|---|---|---|\n| 0.6B | 21.2       | 4.14         | 3.39  | 3.04  | 2.16  |\n| 1.7B | 21.4       | 4.31         | 3.60  | 3.15  | 2.43  |\n| 4B | 27.8       | 5.87         | 4.94  | 4.33  | 3.82  |\n| 8B | 28.4       | 6.47         | 5.53  | 4.94  | 4.43  |\n| 14B | 32.3       | 7.85         | 6.77  | 6.24  | 5.62  |\n| 32B | 52.3       | 13.45        |11.79  |10.71  | 9.84  |\n\nBatch Size: 64\n| Model | Dense | KV budget=1024 | 512   | 256   | 128   |\n|---|---|---|---|---|---|\n| 0.6B | 11.6       | 3.46         | 3.02  | 2.97  | 2.04  |\n| 1.7B | 11.8       | 3.59         | 3.51  | 3.13  | 2.23  |\n| 4B | 15.3   | 4.91         | 4.37  | 3.95  | 3.79  |\n| 8B | 15.9 | 5.40         | 4.86  | 4.47  | 4.09  |\n| 14B | 18.2 | 6.58         | 5.90  | 5.55  | 5.08  |\n| 32B | 29.9 |11.25         |10.37  | 9.63  | 8.95  |\n\n\n## W3: Non-shared and heterogeneous prompts generalization\n\nYes. In summary, the Kinetics Scaling Law remains applicable in settings with non-shared/heterogeneous prompts as long as decoding remains the dominant inference cost. Moreover, our cost model naturally extends to realistic deployment scenarios, including multi-user or heterogeneous prompt settings.\n\n**Key Results.** We compare the scaling trend with both non-shared and shared prompt setting. The two resulting curves are similar. \n**Experiment Setups.** The following Pareto frontiers are provided for the Livecodebench using Best-of-N. The numbers are the minimum accuracy when a model appears in the Pareto frontiers. The maximum accuracy within our cost budget is $84\\%$. e.g., the minimum accuracy in the Pareto Frontier is 16.59 for 14B in prompt-sharing setting and 77.05 for 32B, this suggests to achieve any accuracy in [16.59, 77.05), using the 14B model is most efficient.\n\n| Best-of-N | 0.6B | 1.7B | 4B   | 8B    | 14B   | 32B   |\n|---|---|---|---|---|---|---|\n| W. Sharing | 0    | 1.88 | 2.75 | 13.01 | 16.59 | 77.05  |\n| W/O. Sharing | 0    | 1.88 | 2.75 | 12.54 | 16.79 | 76.92  |\n\n**Analysis.**\n\n**Generalization to non-shared & heterogeneous prompts:** When the cost model processes non-shared prompts, we treat every single prompt as an individual request (N=1 in Equation 2). For partially overlapping prompts, we assume the cost sharing of  the “shared” part in prompts in Equation 2, for the “unshared” part, we calculate them individually for each prompt and add them up to obtain the total cost. The same logic applies to heterogeneous prompts. Thus, as long as the decoding phase dominates for each batched request, the same scaling behavior is observed.\n\n**Non-shared prompts reinforce our main insight:**  Lastly, prompt sharing is actually able to reduce the KV memory access by enabling different requests to access the same piece of KV memory at decoding time [1][2]. Without prompt sharing, the bottleneck of KV memory access will become even worse.\n\n[1] Ye, Zihao, et al. \"Flashinfer: Efficient and customizable attention engine for llm inference serving.\"\n\n[2] vllm: [V1] [Performance] Optimize Cascade Kernel\n\n## Q1: TTS has multiple meanings\n\nWe thank the reviewer for pointing out this. Throughout the paper, we refer TTS to \"Test-time-scaling\" only. The “total token sampling” in Line 71 is a typo, we will fix this in the revised version.\n\n\n## Q3: Visualization of eFLOPs\n\nWe thank the reviewer for this suggestion. eFLOPs is a resource consumption metric, which reflects how much time it consumes for hardware to complete a workload at full utilization.  \nIf a workload consist of $A$ FLOPs for computation and $M$ bytes memory access, the time to complete the workload is modeled as $T = A/C + M/B$, where $C$ is the peak FLOPs and $B$ is the memory bandwidth. Assuming $C/B = I$,  then $T = (A + IM) / C$. Then we define ($A + IM$) as the eFLOPs. \n\nWe will visualize this in the revised version. \n\n## Q4: KV block size and grouping strategy\n\n**1. The performance is robust to KV block size and grouping strategy.**\n\n**Key Results.** We evaluate a fixed budget (1024) with different KV block sizes and grouping strategy (block-topk attention and Quest[1]). We observe that Pass@32 for different (block size, topk blocks)={(16,64), (32,32), (64,16)} are all 90% for Qwen3-8B using both block-topk and Quest.  \n\n**Analysis.** With the same reasonable KV budget, the performance of Block-TopK attention and Quest (can be viewed as two grouping strategies)  is usually robust. We briefly discuss our choice of KV block size in Appendix A.2 (Page 12). We will present more ablations in the revised version.\n\n**2. The configuration of block Topk Attention is able to transfer from code generation to math reasoning.** \n\n**Key Results.** We show that a certain KV budget reaches similar relative performance on the two benchmarks.  We evaluate Pass@32 of Qwen3-8B on LiveCodeBench and AIME24. \n\n| Algo | Dense (%) | (4, 64) (budget 256) | (32, 16) (budget 512) | (16, 64) (budget 1024)|\n|---|---|---|---|---|\n| AIME24 | 90 | 76.7 | 83.3 | 90 |\n| LCB | 74 | 48 | 66 | 74 |\n\nBesides, more results on LCB are presented in Appendix C.1 (Page 14). Block TopK attention also shows impressive scaling performance as in AIME24.\n\n**3. Sparse Attention Training.** \nIt is important to work on sparse attention training. Kinetics demonstrates that even inference-only algorithms (Block-TopK) can scale well. A well-trained sparse attention model is promising. However, there are some challenges: \n1. **Optimization**: Similar to MoEs, sparse attention might be difficult to optimize. For example, sparse attention may cause signal loss in gradients, resulting in higher training loss. \n2. **Implementation**: Efficient implementation of sparse attention kernels for training is non-trivial. FlashAttention and FlexAttention only support several constrained variants. \n\nRecent work like NSA[2] proposes a variant to solve these problems on 27B model. SeerAttention[3], MoBA[4], infLLM-v2[5] also demonstrate promising performance and will enhance the role of sparse attention in TTS. We also consider training sparse attention for test-time scaling as our important future work and are willing to discuss and receive suggestions.\n\n[1] Tang, Jiaming, et al. \"Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference\"\n\n[2] Yuan, Jingyang, et al. \"Native sparse attention: Hardware-aligned and natively trainable sparse attention.\" \n\n[3] Gao, Yizhao, et al. \"SeerAttention-R: Sparse Attention Adaptation for Long Reasoning.\" \n\n[4] Lu, Enzhe, et al. \"Moba: Mixture of block attention for long-context llms.\" \n\n[5] Team, MiniCPM, et al. \"MiniCPM4: Ultra-Efficient LLMs on End Devices.\""}, {"review_id": "o7D6VsyPan", "reviewer": "Reviewer_cPsH", "summary": "The authors find that small models effectiveness is overestimated in terms of inference-time scaling. Their study spans from 0.6B to 32B params, with a ‘kinetics’ scaling law, which can guide test-time compute strategies across model sizes. The key observation is that TTS requires longer context, making the FFN less of a problem and attention the major player. They propose scaling paradigms centered on sparse attn.", "questions": "None, well motivated paper, clean results, followed up by one potential way to alleviate the issue (Sparsity)", "limitations": "None, in-scope experiments are sufficient, will motivate future work well.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Exposes a often overlooked observation, key is to get pareto-frontier, not the best AIME score from the smallest possible model which may take O(1k) tokens more. \n- Clean results, easy to understand the core issue. \n\nWeaknesses\n- Sparse attention is good, but study on low-rank attention (KV-Cache compression) etc might be interesting. However, I treat it as orthogonal/out of scope for this paper. \n- Graph formatting can be made a bit more crispy, text is way too small or light.", "was_revised": true, "final_justification": "I have engaged with the authors, read the other reviewers notes and keep my vote to accept the paper.", "rebuttal": "Thank you very much for your thoughtful review and constructive suggestions. We are glad that you found our work to highlight an often overlooked observation, supported by clean and easily interpretable results. We have carefully addressed your questions and hope you will consider raising your score in light of our response.\n\n## Q1: Sparse attention is good, but study on low-rank attention (KV-Cache compression) etc might be interesting. However, I treat it as orthogonal/out of scope for this paper.\n\nWe thank the reviewer for this suggestion. Low rank KV compression can also effectively address the bottleneck of KV memory access and is usually **complementary** with sparse attention to further push the boundary of test-time scaling. We plan to investigate several lines of work on low rank attention into our scaling law framework.\n\n1. Saxena, Utkarsh, et al. \"Eigen attention: Attention in low-rank space for kv cache compression.\" arXiv preprint arXiv:2408.05646 (2024).\n\n2. Chang, Chi-Chih, et al. \"Palu: Compressing kv-cache with low-rank projection.\" arXiv preprint arXiv:2407.21118 (2024).\n\n3. Zhang, Rongzhi, et al. \"Lorc: Low-rank compression for llms kv cache with a progressive compression strategy.\" arXiv preprint arXiv:2410.03111 (2024).\n\nAlso, there are some works combining sparse attention and low rank compression, for example,\n\n4. Singhania, Prajwal, et al. \"Loki: Low-rank keys for efficient sparse attention.\" Advances in Neural Information Processing Systems 37 (2024): 16692-16723.\n\n5. Sun, Hanshi, et al. \"Shadowkv: Kv cache in shadows for high-throughput long-context llm inference.\" arXiv preprint arXiv:2410.21465 (2024).\n\n6. Yang, Shuo, et al. \"Post-training sparse attention with double sparsity.\" arXiv preprint arXiv:2408.07092 (2024).\n\nWe would be happy to hear about other suggestions.\n\nIn addition, we add evaluations on SnapKV[1], a KV compression method that maintains a fixed-size KV cache by retaining only the key-value pairs identified as important by the recent tokens. This algorithm is proposed for long-context problems but can also be used in long-generation. \n\n\n**Best-of-N**\n| Method  | B200 secs for acc=40% | acc=50% | acc=60% | acc=70% |\n|---------|------------------------|---------|---------|---------|\n| Dense   | 1.30                   | 1.52    | 2.23    | 3.54    |\n| SnapKV  | 0.41                   | 0.71    | 1.03    | 1.91    |\n\n**Long CoT**\n| Method  | B200 secs for acc=40% | acc=50% | acc=60% | acc=70% |\n|---------|------------------------|---------|---------|---------|\n| Dense   | 0.37                   | 0.82    | 1.31    | 2.29    |\n| SnapKV  | 0.51                   | 0.97    | 1.28    | 1.89    |\n\n**Analysis.** SnapKV requires a larger KV budget to approach the same accuracy as Block-TopK attention as they only keep a constant memory of KV cache. However, they have an important advantage of saving GPU memory, which is essential when there is limited VRAM.\n\n [1] Li, Yuhong, et al. \"SnapKV: LLM knows what you are looking for before generation.\" arXiv preprint arXiv:2404.14469 (2024).\n\n\n## Q2: Graph formatting can be made a bit more crispy, text is way too small or light.\n\nWe thank the reviewer for this important suggestion. We will make the text more visible in the revised version."}, {"review_id": "4ssOfVvOu9", "reviewer": "Reviewer_ms9j", "summary": "This paper focus on the current test-time scaling law, and first analyze how do the performance scale in terms of eFLOPs, a new metrics that  take both computation and memory into consideration. The authors then discuss about the potential of using sparse attention to improve the current scaling law. Experiments using block top-k attention is then provided to support the advantages empirically.", "questions": "1. How is the results in Fig. 8 generated? Are you using the same model but with the block top-K attention introduced in Sec. 5? If so, the order of introducing the method should be changed.\n2. Now that the current conclusion will be affected by the computation v.s. memory ratio, how does the results change over different GPUs?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The idea is relatively novel. The point of taking both compuation and memory into consideration is very interesting.\n- The conclusion are supported by both theoretically and empirically.\n\nWeakness:\n- I personally encourage the authors to revise their abstract and keywords to highlight what the paper is about. I personally was thinking that certain part of training is needed for the new sparse-attention paradigm, which does not seem to be the case in here. \n- While the overall paper looks well-written, certain details related to experiments are confusing. Please refer to my questions.\n- The main paper seems to be over-crowded with many experimental results. I will personally recommend the authors to move some part of it to the appendix, and bring back more preliminaries, e.g., top-k attention, to make the paper more self-contained.\n- While in the main paper, the authors is claiming to use three datasets, only AIME24 results are provided in the main paper, and the others are provided only in the appendix.", "was_revised": true, "final_justification": "My original concerns are mostly regarding clarity of the paper. The authors have promised and showed how they plan to change them. Because of the format of Neurips does not support directly change on the paper to verify the promise the author make, I cannot provide a higher rating.", "rebuttal": "Thank you very much for your thoughtful review and constructive suggestions. We are glad that the reviewer found our work to be novel and interesting, with both theoretical and empirical support. We have tried to address your questions carefully. We hope the reviewer will consider raising your score in light of our response.\n\n## Q1. Revise the abstract and keywords to highlight what the paper is about. \n\nThank you for raising this question. Our work studies test-time scaling law at **inference time**, focusing on the efficiency perspective. We investigate the tradeoff between models and generated tokens to reach a higher performance with a certain amount of **inference** resources. For example, to conduct a reasoning task, we can use the 32B model to generate 4K tokens, while we can also use 4B model to generate 32K tokens, but their actual cost and quality are different.\nWe will emphasize that we focus on **inference** and efficiency aspects for test-time scaling in both abstracts and keywords in the revised version.\n\n**To clarify the content of our paper**, we have revised the ordering and restructured the paper. We move the definitions and preliminaries (like TopK attention) and other benchmark results (AIME25/LivecodeBench) into the main paper.\n\nFor your reference, we provide a summary. \n**Summary of our paper**\n- **Objective:** We study the test time scaling law theoretically and empirically, which is about how to optimally tradeoff task performance and inference cost, *considering both computation and memory access costs*.\n- **Key contributions:** \n  - **Cost Model.** our cost model finds that attention cost dominates test-time scaling, especially for the KV memory access cost, which cannot be discovered under the previous scaling law, which only considers computation cost.  Therefore,  previously scaling law can lead to sub-optimal choices, such as overestimating small models (like 0.6B, 1.7B) with more generated tokens at test-time. \n  - **Kinetics Scaling Law.** By jointly considering memory access cost and computation cost (eFLOPs), Kinetics scaling law suggests that we need to first scale models to a certain size (14B for Qwen3) and then increase the generated tokens. \n  - **IsoCost Analysis.** In Figure 5 (Page 4), we use IsoCost curves to analyze how Kinetics scaling law and previous scaling laws are different. In Kinetics scaling law, the IsoCost curves are relatively flat along the dimension of model size and steep along the dimension of generated tokens. While in the previous scaling laws, there is an opposite trend. This suggests that, when properly considering the memory access cost, increasing CoTs is costlier than increasing model size before a certain threshold model size is reached. \n  - **Sparse Kinetics.** Based on the analysis of test-time scaling law, we propose a **new scaling paradigm with sparse attention**, which directly reduces the KV memory access, thus enabling a significant performance improvement with same inference cost (40-60 points), or reduction on inference cost  (8-10 times) to achieve the same performance. \n  - **Analysis of Sparse Kinetics.** Our analysis on Sparse Scaling Law (Fig 9., Page 8) finds that, on doubling of the total compute cost, the optimal KV budget increases by a factor of **$1.18\\times$**, while the total number of generated tokens increases by **$1.74\\times$** for Best of N, or **$1.23\\times$** and **$1.52\\times$** for Long CoTs.  This means Sparse attention becomes increasingly valuable in high-cost scenarios.\n  - **These results suggest that sparse attention is essential for realizing the full potential of test-time scaling because, unlike pretraining, where parameter scaling saturates, test-time accuracy continues to improve through increased generation.**  \n\n## Q2. How are the results in Fig. 8 generated? Are you using the same model but with the block top-K attention introduced in Sec. 5? If so, the order of introducing the method should be changed.\n\nWe thank the reviewer for raising this question.  Fig. 8 (Page 8) is generated with TopK attention defined in Appendix  D.3.  In Fig. 10 (Page 9), 21 (Page 15, Appendix C.1), 22 (Page 16, Appendix C.1), we used Block-TopK attention and made a comparison between the two variants.  We have restructured the way of introducing the two variants and moved the definition of TopK attention to the main paper in our revised version. \n\n\n## Q3. Now that the current conclusion will be affected by the computation v.s. memory ratio, how does the results change over different GPUs?\n\nWe show that our conclusion holds for different GPUs for both Kinetics Scaling Law and Sparse Kinetics.\n\n\n**Kinetics Scaling Law (for dense attention models)**\n\n**Experiment Setups.** The following device-specific Pareto frontiers are provided for the Livecodebench dataset. The numbers are the ***minimum accuracy when a model appear in the Pareto frontiers***. The maximum accuracy is 66%. For example, if the minimal accuracy in the Pareto Frontier is 17% for 14B and 59% for 32B, this suggests to achieve any accuracy in [17%, 59%), using the 14B model is the most cost-effective. \n\n**Key Results.** We found that 14B models still dominate the Pareto frontiers while smaller models, like 0.6B, 1.7B only have a limited scope. We confirmed that (1)  small models' test-time scaling effectiveness is over estimated. (2) we need to scale the model first then scale the generated tokens for other GPUs as well. \n\n| Long-CoT                     | 0.6B (%) | 1.7B (%) | 4B (%)   | 8B (%)   | 14B (%)  | 32B (%) |\n|-----------------------------|------|------|------|------|------|------|\n| Previous (compute-optimal) | 0    | 0    | 4 | 55 | 57 | 59 |\n| B200                        | 0    | 0    | 2 | 14 | 17 | 59 |\n| H100                        | 0    | 0    | 2 | 14 | 18 | 59 |\n| A100                        | 0    | 0    | 2 | 20 | 20 | 59 |\n| L40                         | 0    | 0    | 2 | 14 | 19 | 59 |\n\nOur scaling laws are centered on the observation that attention cost dominates the test-time scaling cost. Therefore, if there is a large gap between FLOPs and Memory I/O in hardware (which is usually the case for modern GPUs. Despite the gaps are different for different GPUs, almost every GPU has a FLOPs/IO ratio of several hundred.), the memory access to KV cache (in attention) will become the key bottleneck,  so the scaling law curve will be very similar to Figure 1a (Page 1) in our main paper.\n\n**Sparse Kinetics (for sparse attention models)**\n\nWe show that sparse attention still scales much better than dense counterparts in H100, A100, and L40.  We report the following table, which conduct the same experiments as Figure 10b (Page 9) but on configurations of different GPUs. The sparse scaling law curve remains very similar. \t\n\nWe report the following table to show the sparse scaling law curve remains very similar to Figure 8ad (Page 8) and Figure 10 bc (Page 9).  \n\nWe use best-of-N scaling for the following tables.\n\n**H100**\n| Best-of-N     | H100 secs @acc=40% ↓ | acc=50% ↓ | acc=60% ↓ | acc=70% ↓ |\n|---------------|--------------------------|-----------|-----------|-----------|\n| Dense         | 3.46                     | 4.71      | 6.40      | 8.70      |\n| Block-TopK    | 0.59                     | 0.69      | 0.87      | 1.38      |\n\n**A100**\n| Best-of-N     | A100 secs @acc=40% ↓ | acc=50% ↓ | acc=60% ↓ | acc=70% ↓ |\n|---------------|--------------------------|-----------|-----------|-----------|\n| Dense         | 5.51                     | 8.08      | 10.18     | 14.94     |\n| Block-TopK    | 1.19                     | 1.38      | 1.49      | 2.55      |\n\n**L40**\n| Best-of-N     | L40 secs @acc=40% ↓ | acc=50% ↓ | acc=60% ↓ | acc=70% ↓ |\n|---------------|-------------------------|-----------|-----------|-----------|\n| Dense         | 12.97                   | 17.64     | 23.98     | 32.60     |\n| Block-TopK    | 2.40                    | 2.79      | 3.52      | 5.16      |\n\n\n ## Q4&Q5. Over-crowded with many experimental results. Recommend to move some part of it to the appendix, and bring back more preliminaries, e.g., top-k attention, to make the paper more self-contained. Only AIME24 results are provided in the main paper, and the others are provided only in the appendix. \n\nThank you for the suggestions. We  have (1) brought back more preliminaries in the main paper. , (2) bring other benchmarks results in the main paper,  and will (3) remove part of AIME24 results to the appendix. We hope these changes will make the paper clearer."}, {"review_id": "96299MxEsq", "reviewer": "Reviewer_EZZU", "summary": "This paper challenges the prevailing test-time scaling laws that prioritize FLOPs as the sole cost metric and argues for a revised framework incorporating memory access costs. The authors introduce the Kinetics Scaling Law, showing that in inference-heavy regimes—such as Best-of-N sampling or long chain-of-thought (CoT) generation—attention costs dominate due to their quadratic complexity and KV cache overheads. Empirical studies with Qwen3 models reveal that, contrary to prior assumptions, compute is better spent on scaling model size (up to a ~14B threshold) than on increasing sample count or generation length. To improve efficiency, the authors propose a sparse attention paradigm, validated through block top-k mechanisms that significantly improve both task throughput and problem-solving accuracy, especially in compute-constrained settings.", "questions": "- Could the authors report results for non-Qwen models to show generality (e.g., LLaMA 3, DeepSeek, or Mixtral)?\n- Does the greedy search over (N, n, B) space generalize well when real-time inference constraints (e.g., latency budgets) are added?\n- How well does block top-k attention perform in settings with diverse prompt lengths or non-homogeneous input distributions?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The paper introduces a compelling revised cost model (Eq. 4) that includes memory access, a long-ignored but increasingly dominant factor in real-world inference.\n- Empirical results are thorough, spanning multiple model sizes and datasets (AIME24, LiveCodeBench), and clearly show the dominance of attention costs (Figures 2a, 4, 6, 8).\n- The proposed Kinetics Scaling Law provides actionable insight: scale model size first (up to ~14B) before investing in more sampling or longer CoTs.\n- Sparse attention is not just proposed but instantiated and benchmarked through block top-k attention, showing practical gains in throughput and accuracy (Figures 10, 11).\n- The iso-eFLOPs analysis is thoughtful and shows how resource allocation decisions shift once memory is included in the budget (Figure 5b vs. 5c).\n- The study is grounded in realistic deployment assumptions (e.g., B200 GPUs, paged KV cache), making it highly relevant to practitioners.\n\nWeaknesses\n- The generalizability of the 14B \"critical size\" threshold is not fully explored across different architectures, hardware (e.g., H100, CPUs), or tasks.\n- The paper evaluates sparsity patterns primarily using Qwen3 models and limited attention variants; inclusion of more baselines (e.g., LSH, SMYRF, Reformer) would strengthen the claims.\n- The greedy optimization approach in Section 4.1, while reasonable, is somewhat heuristic and lacks a formal analysis or guarantees.\n- Sparse attention evaluation focuses on throughput and solving rate, but ablation on latency variance or response consistency is missing.\n- Some important details—like KV cache reuse in Best-of-N and Long-CoT—are deferred to the Appendix and could be brought into the main paper for clarity.", "was_revised": true, "final_justification": "The paper argues that memory access, especially KV cache traffic, dominates test-time scaling and gives a cost law for compute allocation. My initial concerns were the generality of the ~14B “critical size,” behavior under heterogeneous prompts and latency limits, coverage of sparse-attention evidence, and the search procedure. The response adds cross-GPU Pareto analyses where 14B remains on the frontier and clarifies the CPU case, which addresses generality. It also reports runs with non-shared prompts and decoding-latency tables on 8×H100 with variance, covering prompt heterogeneity and latency. Added sparsity ablations (Quest, SnapKV) support the claim that sparse attention can realize the scaling gains. The grid/greedy search with an optimality bound and notes on mapping GPU-time cost to latency reduce concern about the heuristic, though formal treatment would help. Remaining gaps are validation beyond the Qwen and DeepSeek families and missing comparisons with other sparse baselines in an end-to-end serving setup. On balance, I recommend accept.", "rebuttal": "Thank you for your thoughtful review and constructive suggestions. We are glad you found our work to offer a compelling cost model, strong empirical results, and practical insights. We have addressed your questions carefully and hope you will consider raising your score based on our response.\n\n## W1&Q1. The generalizability of the 14B \"critical size\" threshold\n\nWe thank the reviewer for this insightful question. In the main paper, we use Qwen3 series as an example which has 14B threshold, but the threshold varies for other model families. This threshold generalizes across GPUs (but not CPUs) and different tasks. \n\n**Architectures**\n\nAs shown in Appendix B.2, **DeepSeek-Distilled-Qwen2.5 model** yields a different threshold of **7B**. However, this still supports our general conclusions,\n1. Small models are overestimated in effectiveness.\n2. Scaling model size up to a certain threshold can be more beneficial than increasing generated tokens. Although the threshold might vary across model families, the existence of such a threshold generalizes.\n\n**Analysis.** We further analyze the difference between the Qwen3 series and the DeepSeek-Distilled-Qwen2.5 series. The model architecture design is the key reason. The table below shows the KV memory Size for Qwen3 and DeepSeek models (per 32K tokens, unit: GB)\n\n| Qwen3 | 1.7B | 8B | 14B | 32B |\n|---|---|---|---|---|\n| KV mem | 3.5  | 4.5 | 6   | 8  |\n\n| DeepSeek | 1.5B | 7B | 14B | 32B |\n|---|---|---|---|---|\n| KV mem | 0.875 | 1.75 | 6   | 8   |\n\nAs shown in the table, the DeepSeek 7B model is significantly more KV memory-efficient than the DeepSeek 14B model: the latter has  ~$3.4\\times$ more KV memory, while having only $2\\times$ more parameters.  While in other model series, the KV cache size typically grows sublinearly, with respect to model parameters (Figure 5a, Page 6).\n\n> As Mistral and LLaMA3 series do not have sufficient reasoning models to conduct the qualitative analysis, we do not include them. We welcome any suggestions from the reviewer for future evaluation.\n\n**Hardware**\n\nWe show that 14B \"critical size\" generalizes to other GPUs. \n\n**Experiment Setups.** The following device-specific pareto frontiers are provided for Livecodebench. The number reported in the table is the minimum accuracy that the model appears in the Pareto Frontier. For example, the minimum accuracy in the Pareto Frontier is 17% for 14B and 59% for 32B, this suggests to achieve any accuracy in [17%, 59%), using the 14B model is most efficient. The maximum accuracy is 66%.\n\n| Long-CoT | 0.6B | 1.7B | 4B | 8B | 14B  | 32B |\n|---|---|---|---|---|---|---|\n| Prev (compute-optimal) | 0 | 0 | 4 | 55 | 57 | 59 |\n| B200 | 0 | 0 | 2 | 14 | 17 | 59 |\n| H100 | 0 | 0 | 2 | 14 | 18 | 59 |\n| A100 | 0 | 0 | 2 | 20 | 20 | 59 |\n| L40 | 0 | 0 | 2 | 14 | 19 | 59 |\n\n**Analysis.** Our scaling laws are centered on a dominant attention cost. A large gap between FLOPs and Memory I/O causes KV bottleneck to dominate, so the scaling behavior as well as the critical size remain the same. \nCPUs do not have such large FLOPs-to-I/O gap, so the 14B critical size does not apply (Appendix F, Page 19). \n\n**Task**\n\nIn Appendix B.1(Page 12), we show that the 14B “critical size” can generalize to other tasks.  The number reported in the table is the minimum accuracy that the model appears in the Pareto Frontier.\n\n| Long-CoT | 0.6B | 1.7B | 4B | 8B   | 14B  | 32B  | Max Acc. |\n|---|---|---|---|---|---|---|---|\n| AIME25 (%) | 0 | 2 | 21 | 40 | 40 | 67 | 67 |\n| LiveCodeBench (%) | 0 | 0 | 2 | 14 | 17 | 59 | 66 |\n\n## W2. Alternative sparse attention variants and models.\n\n**Sparse Attention Variants**\n\nWe ablate on two sparse variants Quest[1] and SnapKV[2].\n\n> Quest requires storing the full KV cache and performs retrieval during inference. In contrast, SnapKV maintains a fixed-size KV cache. Although proposed for long-context problems, these can also be used in long-generation.\n\nBest-of-N\n| Method | B200 secs @acc=40% ↓| acc=50% ↓| acc=60% ↓| acc=70% ↓|\n|---|---|---|---|---|\n| Dense | 1.30 | 1.52 | 2.23 | 3.54 |\n| Quest | 0.21 | 0.28 | 0.38 | 0.48 |\n| SnapKV | 0.41 | 0.71 | 1.03 | 1.91 |\n\nLong-CoT\n| Method | B200 secs @acc=40% ↓| acc=50% ↓| acc=60% ↓| acc=70% ↓|\n|---|---|---|---|---|\n| Dense | 0.37 | 0.82 | 1.31 | 2.29 |\n| Quest | 0.06 | 0.14 | 0.20 | 0.31 |\n| SnapKV | 0.51 | 0.97 | 1.28 | 1.89 |\n\n**Analysis.** Quest performs better than static methods like SnapKV which requires a larger KV budget to approach the same accuracy.\n\n**Other Sparse Attention.** Sparse algorithms with better theoretical guarantees, like SMYRF[3], Reformer[4] might perform better. However, these algorithms do not have sufficient CUDA kernel support to integrate into inference systems directly. We are working on designing the kernels for these algorithms (e.g., ANNS-based algorithms) to inference efficiently. These algorithms are important to fulfill the potential of test-time computation.\n\nWe will include this discussion and more experiments in the revised version.\n\n**Model Variants**\n\n**Key Results.** Similar to the Qwen3 model family, we see improved test-time scaling with sparse attention for **Deepseek-distilled models**.\n\n| Method | B200 secs @acc=40% | acc=50% | acc=60% | acc=70% |\n|---|---|---|---|---|\n| Dense | 0.45 | 0.72 | 1.46 | 3.17 |\n| Block-TopK | 0.13 | 0.23 | 0.42 | 0.57 |\n\n[1] Tang, Jiaming, et al. \"Quest: Query-aware sparsity for efficient long-context LLM inference.\"\n\n[2] Li, Yuhong, et al. \"SnapKV: LLM knows what you are looking for before generation.\"\n\n[3] Daras, Giannis, et al. \"Smyrf-efficient attention using asymmetric clustering.\"\n\n[4] Kitaev, Nikita, et al. \"Reformer: The efficient transformer.\"\n\n## W3&Q2. Optimality of greedy search\n\nWe show that the point found by our grid search consumes at most $2(1 + \\epsilon)$ times more resources than the true optimal point required to reach the same accuracy, where $0 < \\epsilon < 1$ and $\\epsilon \\to 0$ as the optimal generation length $n^*$ increases.\n\nLet the optimal configuration be $(n*, B*)$, achieving accuracy $m^*$. We perform grid search over points $(x \\cdot \\delta, 2^y)$, where $x, y$ are non-negative integers and $\\delta$ is a positive integer. Here, $n$ is the generation length and $B$ is the KV budget.\n\nAssuming accuracy is monotonic in both $n$ and $B$, the discovered point must satisfy $(k\\delta, 2^t)$ such that:\n- $(k - 1)\\delta < n^* \\leq k\\delta$  \n- $2^{t - 1} < B^* \\leq 2^t$\n\nThe worst-case resource overhead is:\n$$\n\\frac{k\\delta \\cdot 2^t}{n^* B^*} < \\frac{2k}{k - 1} = 2 \\left(1 + \\frac{1}{k - 1}\\right)\n$$\n\nIn practice, with $\\delta = 2048$ and typically $k \\gtrsim 6$, the overhead is generally bounded by $2.2 \\times$ compared to the optimal point on the Pareto frontier.\n\nThe greedy search over $(N, n, B)$ produces a configuration under cost $C$, which reflects the GPU time (not wall-clock time) spent on the task. To transfer to real-time latency, we can add a coefficient on $C$, to handle the case where hardware is not fully utilized or shared by other requests.  \n\n## W4. latency variance and response consistency are missing.\n\n**Latency:**\n\n**Key Results.** We report Qwen3 decoding latency using 8xH100 GPUs. The KV length is 32K tokens. The following table compares dense vs. Block Top-K attention:\n> For OOMs, we estimate full model latency using per-layer measurements. Std shown in parentheses.\n\n| Model | Dense (ms) | KV budget=1024 | 512 | 256 | 128 |\n|---|---|---|---|---|---|\n| 0.6B | 152.76 (0.03) | 14.47 (0.02) | 9.23 (0.02) | 6.24 (0.03) | 4.54 (0.02) |\n| 1.7B | 153.33 (0.03) | 15.15 (0.05) | 9.89 (0.05) | 6.84 (0.03) | 5.21 (0.03) |\n| 4B | 198.48 (0.07) | 20.59 (0.05) | 13.85 (0.04) | 10.06 (0.03) | 7.92 (0.04) |\n| 8B | 199.74 (0.04) | 21.73 (0.01) | 15.30 (0.09) | 11.35 (0.05) | 9.20 (0.04) |\n| 14B | 225.13 (0.04) | 27.77 (0.06) | 20.22 (0.01) | 15.83 (0.08) | 13.63 (0.04) |\n| 32B | 363.46 (0.17) | 47.92 (0.03) | 38.75 (0.08) | 29.15 (0.12) | 25.02 (0.08) |\n\n**Response consistency**\n\nWe show that the consistency of responses is good. Maj@32 means the accuracy obtained through self-consistency[1].\n\n**Qwen3-8B**\n| Budget | Pass@32 | Maj@32 |\n|---|---|---|\n| Full | 90.0 | 83.3 |\n| 256 | 76.7 | 70.0 |\n| 512 | 83.3 | 76.7 |\n| 1024 | 90.0 | 83.3 |\n\n**Qwen3-14B**\n| Budget | Pass@32 | Maj@32 |\n|---|---|---|\n| Full | 93.3 | 83.3 |\n| 256 | 80.0 | 80.0 |\n| 512 | 86.7 | 80.0 |\n| 1024 | 93.3 | 83.3 |\n\n[1] Wang, Xuezhi, et al. \"Self-consistency improves chain of thought reasoning in language models.\"\n\n## W5. Important details in appendix.\n\nThank you for the suggestion. We will bring the important details back in the main paper in the revised version.\n\n## Q3. Diverse prompt lengths or non-homogeneous inputs?\n\nWe address both efficiency and accuracy under such diverse settings.\n\n1. Efficiency under varying prompt lengths:\nBlock Top-K attention maintains strong speedups.\n\n**Key Results.** We report Qwen3 decoding latency for diverse input lengths using 8xH100 GPUs. The avg KV length is 32K with a std of 2K. The following table compares dense vs. Block Top-K attention:\n\n| Model | Dense (ms) | Block TopK budget=1024 | 512 | 256 | 128 |\n|---|---|---|---|---|---|\n| 0.6B | 153.34 (0.36) | 15.12 (0.12) | 9.58 (0.14) | 6.35 (0.19) | 4.60 (0.22) |\n| 1.7B | 154.67 (0.40) | 15.78 (0.13) | 10.20 (0.09) | 6.96 (0.11) | 5.23 (0.15) |\n| 4B | 199.91 (0.35) | 21.18 (0.15) | 14.46 (0.17) | 10.20 (0.14) | 8.00 (0.12) |\n| 8B | 200.55 (0.32) | 22.56 (0.14) | 15.69 (0.16) | 11.45 (0.17) | 9.30 (0.21) |\n| 14B | 226.88 (0.45) | 28.42 (0.18) | 20.88 (0.16) | 16.24 (0.20) | 13.47 (0.14) |\n| 32B | 365.21 (1.20) | 49.26 (0.23) | 36.37 (0.19) | 29.34 (0.24) | 24.98 (0.28) |\n\n**Explanation.** Our Flashinfer-based backend automatically partitions the workload of diverse input lengths into different pages. \n\n**2. Accuracy:** In our evaluations, different requests have different prompt lengths. Block-topk exhibits strong performance confirming its robustness across non-uniform input distributions."}]

Author rebuttal:

Meta-review: The paper received 1 borderline accept and 3 accept final ratings. Reviewers acknowledge the novelty and motivation of the paper: taking memory I/O into the scaling law consideration and improve attentions with sparse attention. Reviewers raised concerns about the writing and generalization of the method, etc. Those questions were largely addressed during the rebuttal session, one reviewer raised the score to accept (from borderline accept). AC recommends to accept the paper.
Decision: Accept (poster)