Title: Kinetics: Rethinking Test-Time Scaling Law
Venue: NeurIPS 2025
Accepted: yes
Keywords: scaling law

test-time scaling

sparsity
Abstract: We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-N, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. The Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold (14B) than on smaller ones. A key reason is that in test-time scaling, attention—rather than parameter count—emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60-point gains in low-cost regimes and over 5-point gains in high-cost regimes for problem-solving accuracy on AIME and LiveCodeBench. These results suggest that sparse attention is essential for realizing the full potential of test-time scaling because, unlike training where parameter scaling saturates, test-time accuracy continues to improve through increased generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges the prevailing test-time scaling laws that prioritize FLOPs as the sole cost metric and argues for a revised framework incorporating memory access costs. The authors introduce the Kinetics Scaling Law, showing that in inference-heavy regimes—such as Best-of-N sampling or long chain-of-thought (CoT) generation—attention costs dominate due to their quadratic complexity and KV cache overheads. Empirical studies with Qwen3 models reveal that, contrary to prior assumptions, compute is better spent on scaling model size (up to a ~14B threshold) than on increasing sample count or generation length. To improve efficiency, the authors propose a sparse attention paradigm, validated through block top-k mechanisms that significantly improve both task throughput and problem-solving accuracy, especially in compute-constrained settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper introduces a compelling revised cost model (Eq. 4) that includes memory access, a long-ignored but increasingly dominant factor in real-world inference.\n- Empirical results are thorough, spanning multiple model sizes and datasets (AIME24, LiveCodeBench), and clearly show the dominance of attention costs (Figures 2a, 4, 6, 8).\n- The proposed Kinetics Scaling Law provides actionable insight: scale model size first (up to ~14B) before investing in more sampling or longer CoTs.\n- Sparse attention is not just proposed but instantiated and benchmarked through block top-k attention, showing practical gains in throughput and accuracy (Figures 10, 11).\n- The iso-eFLOPs analysis is thoughtful and shows how resource allocation decisions shift once memory is included in the budget (Figure 5b vs. 5c).\n- The study is grounded in realistic deployment assumptions (e.g., B200 GPUs, paged KV cache), making it highly relevant to practitioners.\n\nWeaknesses\n- The generalizability of the 14B \"critical size\" threshold is not fully explored across different architectures, hardware (e.g., H100, CPUs), or tasks.\n- The paper evaluates sparsity patterns primarily using Qwen3 models and limited attention variants; inclusion of more baselines (e.g., LSH, SMYRF, Reformer) would strengthen the claims.\n- The greedy optimization approach in Section 4.1, while reasonable, is somewhat heuristic and lacks a formal analysis or guarantees.\n- Sparse attention evaluation focuses on throughput and solving rate, but ablation on latency variance or response consistency is missing.\n- Some important details—like KV cache reuse in Best-of-N and Long-CoT—are deferred to the Appendix and could be brought into the main paper for clarity.\n\nQUESTIONS:\n- Could the authors report results for non-Qwen models to show generality (e.g., LLaMA 3, DeepSeek, or Mixtral)?\n- Does the greedy search over (N, n, B) space generalize well when real-time inference constraints (e.g., latency budgets) are added?\n- How well does block top-k attention perform in settings with diverse prompt lengths or non-homogeneous input distributions?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on the current test-time scaling law, and first analyze how do the performance scale in terms of eFLOPs, a new metrics that  take both computation and memory into consideration. The authors then discuss about the potential of using sparse attention to improve the current scaling law. Experiments using block top-k attention is then provided to support the advantages empirically.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The idea is relatively novel. The point of taking both compuation and memory into consideration is very interesting.\n- The conclusion are supported by both theoretically and empirically.\n\nWeakness:\n- I personally encourage the authors to revise their abstract and keywords to highlight what the paper is about. I personally was thinking that certain part of training is needed for the new sparse-attention paradigm, which does not seem to be the case in here. \n- While the overall paper looks well-written, certain details related to experiments are confusing. Please refer to my questions.\n- The main paper seems to be over-crowded with many experimental results. I will personally recommend the authors to move some part of it to the appendix, and bring back more preliminaries, e.g., top-k attention, to make the paper more self-contained.\n- While in the main paper, the authors is claiming to use three datasets, only AIME24 results are provided in the main paper, and the others are provided only in the appendix.\n\nQUESTIONS:\n1. How is the results in Fig. 8 generated? Are you using the same model but with the block top-K attention introduced in Sec. 5? If so, the order of introducing the method should be changed.\n2. Now that the current conclusion will be affected by the computation v.s. memory ratio, how does the results change over different GPUs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The authors find that small models effectiveness is overestimated in terms of inference-time scaling. Their study spans from 0.6B to 32B params, with a ‘kinetics’ scaling law, which can guide test-time compute strategies across model sizes. The key observation is that TTS requires longer context, making the FFN less of a problem and attention the major player. They propose scaling paradigms centered on sparse attn.

STRENGTHS AND WEAKNESSES:
Strengths
- Exposes a often overlooked observation, key is to get pareto-frontier, not the best AIME score from the smallest possible model which may take O(1k) tokens more. 
- Clean results, easy to understand the core issue. 

Weaknesses
- Sparse attention is good, but study on low-rank attention (KV-Cache compression) etc might be interesting. However, I treat it as orthogonal/out of scope for this paper. 
- Graph formatting can be made a bit more crispy, text is way too small or light.

QUESTIONS:
None, well motivated paper, clean results, followed up by one potential way to alleviate the issue (Sparsity)

LIMITATIONS:
None, in-scope experiments are sufficient, will motivate future work well.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper challenges conventional test-time scaling laws by accounting for memory access costs, not just FLOPs, and shows how attention dominates inference costs. \n\nThe proposed “Kinetics Scaling Law” and the emphasis on sparse attention provide a fresh, impactful perspective.\n\nMany quantitative studies show a clear analysis flow to readers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Clear motivation backed by tons of quantitative studies\n\n2. tremendous performance gains\n\n3. Block Top-k sparse attention (Section 5) is shown to improve throughput significantly—up to 25× for small models, enabling high-efficiency inference.\n\nWeakness\n\n1. The scaling law threshold of 14B is specific to Qwen3 models; it’s unclear how well this generalizes to other architectures (e.g., Mistral, LLaMA3), limiting generality.\n\n2. Want to see some real-world case studies\n\n3. In Equation (2), the memory cost assumes prompt KV cache is amortized across reasoning trials, i.e., shared across N samples. However, in realistic settings (e.g., retrieval-augmented generation or API-style usage), prompts can vary significantly between samples.\n\nHow does the Kinetics Scaling Law behave under non-shared prompts or in settings with dynamic or partially overlapping prompts? Can your model generalize to multi-user or heterogeneous prompt settings common in LLM deployment?\n\nQUESTIONS:\nSee above weakness\n\nand \n\n1. TTS has multiple meanings in this paper, and is hard to follow sometimes. Test-time scaling, total token sampling\n\n2. Real-world latency benchmarks? \n\n3. Add a brief intuitive explanation or visualization of how eFLOPs integrates compute and memory\n\n4. The paper shows block Top-k sparse attention provides the best trade-off, but the mechanism relies on heuristics such as block averaging and grouped query attention. The KV block size and selection granularity seem critical to the performance but are not deeply analyzed.\n\nHow sensitive is the performance of your sparse models to KV block size and grouping strategy? Could this become a tuning bottleneck when transferring across tasks (e.g., from code generation to math reasoning)? Have you considered learning the sparsity patterns or integrating them with routing mechanisms like in MoEs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}