Inference-Time Hyper-Scaling with KV Cache Compression
Authors: Adrian Łańcucki
Konrad Staniszewski
Piotr Nawrot
Edoardo Ponti
Venue: NeurIPS 2025 poster ()
Keywords: reasoning
inference scaling
KV cache compression
token eviction
large language model
Primary area: deep_learning
TL;DR: Inference-time hyper-scaling uses key–value cache compression with Delayed Memory Sparsification (DMS) to boost Transformer LLM reasoning accuracy for equivalent compute or memory costs.

Abstract: Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlenecked by the size of the key–value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios. To make hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a novel method for sparsifying KV caches that only requires 1K training steps to achieve 8× compression, while maintaining better accuracy than training-free sparse attention. Instead of prematurely discarding cached tokens, DMS delays token eviction, implicitly merging representations and preserving critical information. We demonstrate the effectiveness of inference-time hyper-scaling with DMS on multiple families of LLMs, showing that it boosts accuracy for comparable inference latency and memory load. For instance, we enhance Qwen-R1 32B by 9.1 points on AIME 24, 7.6 on GPQA, and 9.6 on LiveCodeBench on average for an equivalent number of memory reads.

Inference-Time Hyper-Scaling with KV Cache Compression
Adrian Lańcuckit INVIDIA University of Warsaw
Konrad Staniszewskit
Piotr N Edoardo M. Ponti
University of Edinburgh
Abstract
Inference-time scaling trades efficiency for increased reasoning accuracy by gener- ating longer or more parallel sequences. However, in Transformer LLMs, genera- tion cost is bottlenecked by the size of the key-value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios. To make hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), novel method for sparsifying KV caches that only requires IK training steps to achieve 8x compression, while maintaining better accuracy than training-free sparse attention. Instead of prema- turely discarding cached tokens, DMS delays token eviction, implicitly merging representations and preserving critical information. We demonstrate the effective- ness of inference-tin timeee hyper-scaling with DMS multiple families of LLMs, showing that it boosts accuracy for comparable inference latency and memory load. For instance, we enhance Qwen-R1 32B by 12.0 points AIME 24, 8.6 GPQA, and 9.7 on LiveCodeBench on average for an equivalent number of memory reads.
1 Introduction
Scaling inference-time compyte- in mod- els such as OpenAT's ol (OpenAl al. 2024) DeepSeek's RI (Guo et al.) 2 trades off in- creased inference time and memory for higher rea- soning accuracy in large language models (LLMs). Models reason by generating intermediate steps that explore the problem before reaching an answer. Ad- justing the depth and breadth of this exploration- known as sequential and parallel scaling, respectively (Muennighoff et al.] 2025) controls the inference-
LixchuBBnch
RI-Qwar 2.5 7B
AIME24
GPQA
time compute budget ( Yao al. [0233] Uesato et al. 2022
Wang et al. 2023 Lightman et al. 2024).
MATH 500
Despite its success, scaling inference- setrennce-imeeeeeeee compute fundamentally bottlenecked in Transformer LLMs by the number of tokens from the key-value (or KV) cache that are attended during auto-regressive gen- eration. This cache grows linearly with respect to the length and number of reasoning chains, as the new key-value representations are appended to
Figure/ 1: Average absolute gains of DMS over the original LLMs during inference-time scaling on reasoning tasks for the same KV cache memory reads, (a proxy for latency).
Work done as an intern at NVIDIA.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hence, it can easily exhaust the memory of the accelerator and slow down each generation step. as attention is memory-bounde its cost dominated by the time needed to retrieve the cache from memory. Fortunately, several methods can mitigate these issues during post-training or inference.
These rely on training-free heuristics to sparsify the KV cache (Oren et al., 2024] [L et al., (024). selectively retrieve subsets of tokens (Tang et al.. 2024), or retrofit LLMs with the ability to choose whether to merge or append items to the cache (Nawrot et al. 2024).
In this work, we investigate for the first time whether efficient attention methods enhance inference- time scaling. In principle, by exploring more concurrent reasoning threads or longer threads for the same memory or runtime budget, efficient models can achieve higher-quality predictions than their original counterparts. However, this hinges upon the crucial assumption that efficient attention does not degrade the model's 's reasoning abilities, which unfortunately is often the side effect of training- tarenng--reeeee sparsification methods (Zhang et al. 2023a; Oren et al., 2024). On the other hand, KV cache compression during post-training usually better preserves the model's 's quality, but also demands costly retrofitting procedures (Nawrot et al. 2024).
In order to overcome these limitations, as second main contribution, we propose Dynamic Memory Sparsification (DMS), new method that combines the best of both worlds by retrofitting LLMs to sparsify the KV cache through an inexpensive procedure. We thus demonstrate that sparsification- rather than more complex token merging proposed in Dynamic Memory Compression (DMC: Nawrot it al. 2024) is sufficient to retain accuracy at high compression ratios. This, in turn, allows us to retrofit LLMs with KV cache compression in a much more sample-efficient way than DMC, achieving 8x compression with only 1K training steps. On the other hand, the superior performance of DMS highlights the benefits of retrofitting L.LMs over training- free heuristics.
We evaluate inference-time scaling capabilities of efficient attention (including DMS) on reasoning datasets such as MATH-500 (Hendrycks et al. 2021b) and AIME 2024 for math, GPQA Diamond (Rein et al., 2024) for hard sciences, and LiveCodeBench (Jain et al., 2025) for coding. We find that DMS significantly improves the Pareto frontiers across various model sizes and datasets, outper- forming vamilla LLMs in both memory reads (which is a proxy for runtime) and peak memory use. Notably, DMS consistently dominates other baselines for efficient attention, which we also verify on a broader set of tasks outside of inference- anference-time scaling. DMS variants even surpass the corresponding vanilla L.MSs on long-context tasks, such as needle-in-a- --aas---ak and variable state tracking (Hsieh) et al. 2024), while achieving higher efficiency. Overall, this validates the effectiveness of efficient attention unlocked by DMS for inference-time scaling, which improves the reasoning capabilities of models under any given inference-time budget.
2 Background
2.1 Inference-time Scaling
Inference-ting time scaling allows a model to 'think longer or more broadly' about a problem to enhance
the quality of its prediction. by leveraging extra compute during generation (Du al. 2024 Madaan
ct al. 2023 Yao et al., 2203)). In practice, when presented with prompt x, Large Language Model
FULM can explore n chains of reasoning [z,,..., 2, to generate corresponding answers .y...... "......y.. Yn
While some strategies involve guiding this exploration through Process Reward Model (PRM; Li
ct al.
2023 Feng et al. 2023) [ighhman et al.] 2024: Uesato et al.] [022] Wang et al.) 2024a] [nnel et al.
2024) by scoring each reasoning step, recent systematic comparisons established that simpler PRM-free strategies such as majority voting (Wang et al. 2025b) remain the most competitive.
Hence, scaling can be easily achieved in two ways: increasing the maximum length for chains of reasoning (known as sequential scaling) or increasing their number (known as parallel scaling). These two quantities can be controlled to set a 'token budget' for inference-time computation (Muennighoff al. 2023), which determines the memory load and latency. In fact, in Transformer LLMs, the KV cache grows linearly with the number of generated tokens. Crucially, it is stored on VRAM in GPU accelerators, contributing significantly to the overall memory load. At the same time, the larger the token budget, the more retrieving the KV cache through biggh-annwidth memory access dominates latency during generation. As a consequence, the KV cache constitutes a bottleneck for inference-time scaling. This leads to the natural question: by making the KV cache leaner, could we scale the length and number of reasoning threads and enhance the accuracy of existing LLMMS for an equivalent compute budget?
2

---

2.2 Training-free KV Cache Eviction
An intuitive strategy to reduce the size of the KV cache is to evict tokens, i.e., dynamically remove the key-value pairs of the least relevant tokens during inference. Recent methods have addressed this challenge by selectively managing tokens within a sliding window of context of size w. For instance, for each time step I. TOVA (Oren et al.] 2024) evicts the token with the lowest attention weight such that iTOVA = min; ΣLCH (t); where an (4); denotes the attention weight assigned to token by attention head h at time step f. Similarly, Heavy-Hitter Oracle (H20; Zhang et al. (023a) evicts the token with the lowest cumulative attention, additionally keeping a sliding window of recent tokens. This family of approaches (more are surveyed in Appendix B) incur minimal computational overhead due to their efficient heuristics for eviction scores, while retaining a maximum KV cache size of w.
A different strategy is adopted by Quest (Tag et al., 2024), which fully retains the entirety of the KV cache but only retrieves the most relevant pages (i.e.. fixed-size blocks of contiguous KV items) from memory. Relevant pages are determined through heuristic that approximates attention scores from the
dimensions of each page's ' KV items. While this approach accelerates generation by reducing memory transfers without permanently evicting tokens, it does not reduce memory load. In fact, to efficiently perform page selection, the method requires storing additional page representations, resulting in slight memory overhead rather than savings.
2.3 Learned KV Cache Compression
While mitigating latency and/or memory load, training-free free KV cache eviction methods often incur performance degradation at high eviction rates. To overcome this limitation, Dynamic Memory Compression (DMC; Nawrot et al. 2024) reduces KV cache size by dynamically compressing representations, potentially extracting and retaining vital information. At each timestep i, for every attention head separately, DMC models decide to either append the new key-value pair to KV cache as in standard Transformers, or merge it into the most recent cache entry using weighted averaging. As consequence, every attention head produces a uniquely compressed KV sequence with possibly distinct compression ratio (CR). This flexibility allows the model to preserve critical information while aggressively compressing redundant representations, unlike KV cache eviction and sparse attention methods that mostly impose uniform compression budgets (Nawrot et al. 2025).
Applying DMC requires continued training of existing LLMs (a.k.a. 'retrofitting'), ) during which discrete decisions (append versus merge) are relaxed into continuous variables via stochastic repa- rameterization (Louizos et al. 2018), enabling pradeent-based optimization. Although it requires a fraction of the pre-training budget, the computational cost is still significant: however, this helps retaining the original LLM quality as the model is trained to operate with compressed KV cache.
3 Dynamic Memory Sparsification
Token eviction strategies effectively reduce KV cache size, but degrade downstream performance at higher eviction rates. Conversely, DMC offers robust compression at the cost of expensive continued training. To scale inference-time efficiency even further, it is therefore essential to develop a KV cache compression method that is inexpensive, easy to integrate, and maintains accuracy at high compression ratios. To this end, we propose Dynamic Memory Sparsification (DMS), a method of teaching pre-trained models simple, adaptive token eviction policy. As such, it combines the advantages of eviction and trained compression, with significantly higher data efficiency than DMC.
3.1 Retrofitting Models with DMS
To develop DMS, we follow Nawrot et al. 2024)'s pipeline to retrofit pre-trained LLMS rather than training them from scratch. We introduce crucial modifications: (i) whereas DMC merges (weighted-averages)) tokens, DMS simply evicts them; and (ii) we separate the time of eviction decisions from the time of their execution-wher a token is flagged for eviction, the model is given a number of generation steps to integrate it's information. Below, we describe the procedure for a single attention head, though the same process is applied across all KV heads independently.
Eviction Decisions Given an input hidden vector to an attention layer h, at inference time step I, DMS predicts binary eviction decision a which controls the eviction of the key-value pair (ke, vt).
3

---

sliding window
kg
k1
12
scheduile A2 for eviction
bin
911
92
hin
9a
<A
log(1 oxs)
0
1
schedull Ag for eviction
log(1 ox)
log(1 (g)
0
oviet řg upon exiling the sliding window
Loo((1 oz)
log[I- og)
log(1 log(1-os)
(a) DMS key cache management during inference.
(b) Attention mask Ma during training.
Figure 2: During each inference step (left) the incoming key-value pair (k,, vz) might be selected for later eviction, based on predicted binary decisions a bin E {0, 1} (we show only a sequence of keys for clarity). The eviction takes place as soon as the pair falls out of the sliding window. During training (right), this behavior is induced with an additive attention mask. Eviction decisions are relaxed from binary to continuous E [0, 1].
To maintain differentiability during training, a₁ is learned through stochastic reparametrization with a Gumbel-sigmoid distribution as a gradient estimator:
Ox2 Gumbel-sigmii((h.' T + +, og E [0.
where w E Ted a vector of trainable weights initialized as w = [O, OTT In addition, we set a low temperature to encourage discrete eviction decisions and b= - in order to offset the logits and initiate training with a₂ s 0, preventing eviction early in training. Empirically, this configuration prevents initial loss spikes, which might cause catastrophic forgetting (Nawrot et al.) 2024).
During training, a sequence of eviction decisions 01:r is used to construct a mask Ma E (-00, 0J part of
(Figure 2bj, which is added to unnormalized attention scores QK The elements that the causal mask (set to -0o) are set to log(1 - oz).. The mask selectively modulates token visibility: 0g fully masks token, og indicates no masking, and values in between make token only partly accessible. It follows that evicting a particular k; entails evicting the corresponding v..
Delayed Eviction via Sliding Window Immediate eviction can harm the model's 's abilities by prematurely discarding useful context. To mitigate this, we propose delaying the execution of eviction decisions. Specifically, the eviction decision o₂ is made at timestep f, but the token selected for eviction remains available until a future timestep + This delay creates sliding window of size w and is implemented by setting positions within the window to when constructing Ma.
Previous work indicates that decoder-only models heavily attend to recent tokens (iio et al. 2024 Jiang et al. 2024). Consequently, delayed eviction enables the model to extract relevant information from such tokens before their removal. Foreshadowing Section 5.3 we find that immediate eviction leads to rapid accuracy degradation, whereas delayed eviction maintains stable training, dramatically reducing the number of training tokens needed to achieve a given CR.
Training Objective During training we follow DMC and apply a one-sided & loss term which forces the model to match the average value of predicted for given input to the target compression at, i.e., Laxx = max (a" L HT Eie Σnen Σter auh, 0), where L, H, denote the number of layers, KV attention heads, and sequence length, respectively. Over the course of training, the target compression a' is linearly annealed from 0 to (1 )R) ) We train the model using a logit distillation loss LD loss (Hinton al.] (2055), described in detail in Section 4 The distillation loss and
auxiliary loss are then combined into single objective: -= LD + Laux Since we do not enforce any constraints on compression for individual attention heads, they adopt possibly different compression ratios and produce KV sequences of possibly different lengths.
Performance Considerations The overhead of DMS on the attention mechanism comes solely from constructing and applying the additive attention mask, which never needs to be materialized.

---

For each attention head, it can be compactly passed as a vector of eviction decisions a1.r, and is implementable with existing tools (Wang et al. 2025a Dong al. 2024). Implementation-wine neuron is re-purposed from q: or k, to predict or instead of adding a parameter vector w for every attention head (Nawrot et al. 2024). Hence, no extra parameters are added.
3.2 Inference
Figure 2a
shows the inference time operation of DMS. The decision variables are rounded to the nearest integer at bin = [iggoid + +]] E {0, 13. a a then the (kr, vz) pair needs to be evicted at time t The sparsity introduced by DMS is also leveraged during the prefilling phase to climinate unnecessary computation (Wang et al. 2025a).
Performance-wise. DMS does not introduce any new read/write operations on the KV cache, since the evicted tokens could be simply overwritten by incoming ones, under the assumption that the keys are stored in the KV cache with positional information. Paged Attentnntnnion (Kwon et al. (003) facilitates storing the sparsified KV cache in memory, where pages are allocated to individual attention heads. This formulation enables our reuse of existing, efficient kernels that support PagedAttention.
4 Experimental Setup
Models and Baselines To evaluate inference-time scaling through KV cache compression, we primarily focus on reasoning models of different sizes, including Qwen 2.5 1.5B, 7B, and 32B distilled from DeepSeek RI (Guo et al., (225)) and Qwen3-8B distilled from Dwwnn-2235BB2
(Yang et al. 2025). In addition, as sanity check on other families of models and for initial ablations on method design, we test the accuracy of efficient attention methods also on Llama 3.2 1B Instruct (Grattafioni al. (024).. We retrofit all these models with DMS and compare them against the original models, DMC, and training-free free KV cache sparsification methods described in Section 2.2] Token Omission via Attention (TOVA; Oren et al. 2024). Heavy-Hitter Oracle (H20; Zhang et al., 2023a),, and Quest (Tang et al. 2 - Crucially, all the LLMs included in the experiments use Grouped Query Attention (GQA:[Ainslaa al. 2023), hence KV tokens are shared among multiple query heads. This exacerbates the destructive effects of training-free token eviction.
Logit Distillation and Retrofitting In contrast to conventional retrofitting methods employing standard language modeling loss (Nawrot et al. 2024), retrofit all models through logit distilla- tion (Hinton et al. 2015). In particular, the original LLM acts as the teacher and the DMS retrofitted one as the student. As previously observed in other settings (Sreenivas et al. 2024 Minixhofer et al.) 2025), we found that logit distillation provides greater robustness to shifts in training data, since the original data mixtures rarely public, and is especially beneficial for fragile LLMSs with lower parameter counts. We provide information on the training data for distillation in Appendix B
The retrofitting process is inspired by DMC (Nawrot et al. 2024). The amount of required data depends directly on the context length retrofitted models and the target compression ratio: higher ratios necessitate larger datasets. We employ a linear schedule that runs for 100 training steps for each unit of compression ratio: CR(t) = + 1. Crucially, annealing the CR generates a family of models with different compression ratios from single retrofitting run. Unless otherwise stated, train DMS models with a sliding window-and equivalently an eviction delay-of 256 tokens.
5 Results
5.1 Inference Time Hyper-Scaling
Goal and Metrics We aim to determine whether KV cache compression increases downstream performance by effectively leveraging a larger 'token budget' for equivalent latency and memory load compared with vanilla Transformers. We run experiments for inference-ting hyper-scaling on datasets that require advanced reasoning capabilities, following (Snel et al. (2024) and Guo et al. (2025). Specifically, we evaluate AIME 24 and MATH-500 (Hendryeks et al. 2021a) for math prob- lems, GPQA Diamond (Rein et al. 2024) for physics, chemistry, and biology, and LiveCodeBench
"Our baseline implementations closely follow the publicly available reference implementations.
5

---

Quest
Vanilla
DMS CRS
Quest CR8
TOVA CR8
Qwen-R1
Qwen-Bi
Qwen-RI
4-16K
40
12K
70
AIME 24
Lok
4-32K
1-32K
30
3220
4-16K
4.32K
60
XBK
B
F.loK
1-15K
F3CK d83K
98
LiveCodeBench
25
166
8-16K
K-KK
8- 16K
90
4-8K
16K
B-T5K
416k
4-16K
8-16K
4-16K
1-165
T-1SK
16K
10K
16K
30
15
Nr.
1-K
KVV cache memory reads
K-IKK
3-16K
32K
40
AIME
30
1-30K
14-1KK
60
32K
2.3365
1-11KK
glack
60
30
50
21
20
2
4-18K
50
R-15K
2.160
LiveCodeBence
25
4.JAK
1.16 4-160
20
1-16K
1-16K
1-1sK
1
90
15
40
L.K
20
4.8
2'
Peak tokens in memory
4-TK
ISK
80
ALLC 7-ISK
700
F-H5K
AIME 24
S-1nk
1.10K
5-16K
70
1-10K
LAK
1-15K
C-RK
1-32K
60
50
LiveCodeBente
25
1-1KK
3-48
40-KK
44
4-16K
3-16K
4-15K
3-16K
60
--6K
5:3%
1:16K
1-16K
35
S0
1-4K
-KK
30
1-KK
40
Throughpul (examples/min)
Figure 3: Inference-time scaling results comparing exact-match accuracy (y-axis) against perfor- mance metrics (æ-axis). Point colors indicate the compression algorithm used, shapes the compression ratio, and W-L labels denote the scaling strategy (W: number of sampled reasoning threads; L: se- quence length). Colored lines indicate the respective Pareto frontiers. The horizontal black lines mark the accuracy reported by Guo et al. (202)) for the 1-32K vanilla model. Top: A comparison implementatioonaanossit proxy for attention compute. Middle: A comparison in terms the peak number of tokens in memory, reflecting memory load. Bottom: Throughput calculated the maximum batch size that accommodates the corresponding W-L configuration. Across plots, DMS attains the best Pareto frontiers, indicating that K KV-aache compression an effective strategy for improving inference-time scaling.
in terms of K VV-acche token reads, used

---

Qwen-R1 1.38
Qwen-R1 7B
90
Qwen-Rt R1 32B
butch sizee 32
batch size
80
butch size 32
19
22
20
60
Vanilla
Latency
16
50
DMS CR4
35
30
batch size
Qwen3-SB
Out of memory
35
batch memory.
Qwen3 SB
35
40
batch
OWBB33SB
Out of gumbiey
mumory
DMS CX8
30
25
20
0K
4K
8K
12K
16K
0K
4K
8K
12K
16K
0K
4K
8K
12K
16K
Context length
Figure 4: Latency of models (y-axis) at different context lengths (æ-axis). Top: We compare the effect of different model sizes (Qwen-R1 R1 1.5B, 7B, 32B) for the same batch size (32). Bottom: We compare the effect of different batch sizes (32, 64, 128) for the same model (Qwen3-8B). Batch size reflects both the number of parallel reasoning threads and the number of queries the model is serving. These plots show that inference becomes memory-bound at different context lengths depending on model scale and batch size, yielding distinct accuracy-efficinne trade-offs.
(Jain et al. 2025) for coding, As the performance metric, we use exact match after mapping outputs to unified math representation (for MATH-500 and AIME 24) or to one of the four available choices (for GPQA Diamond). For LiveCodeBench, we report pass @all, i.e., we count a success if any generated sequence passes the tests.
As metrics for the effective budget in time and memory, we focus first on two implementaaion-aaaoosii quantities: (i) V-cachee token reads, the total number of items in the KV cache attended each generation step, summed across steps. This reflects runtime efficiency, as loading the KV cache from memory quickly becomes main bottleneck during generation, contributing a share of inference latency that increases with sequence length (Tang et al. 2024 Nawrot et al. 2025). Second. (ii) peak tokens in memory. which represents the maximum K KV-ccche size, critical for memory-constraime hardware such as GPUs or edge devices. Finally, (iii) throughput at the maximum batch size that
accommodates each tested configuration.
To establish a range of trade-offs between downstream performance and compute, we run experiments with varying budget configurations in terms of maximum the number of parallel reasoning chains or width (W), sequence length (L), and compression ratio (CR), defining each setting as a tuple W-L-CR, where CR 1x denotes vanilla models. By identifying the Pareto frontier for each method, we determine which ones offer superior performance for the same budget. We report results for AIME 24 and LiveCodeBench in Figure 3 and additional results on MATH-500 and GPQA Diamond in Appendix D For simplicity, as basclines we report only the state-of-the-as training-free free methods: Quest for lacuracy-latency and TOVA for accuracy- memory load.
Accuracy vs. Memory Reads and Peak Memory From Figure 3 and additionally Appendix D and Figure 7 we observe that KV cache compression methods generally yield superior Pareto frontiers compared to vanilla LLMs across model sizes and datasets. Specifically, the best-performing method in each dataset-size combination substantially improves the scores at comparable memory transfer budgets (which drive latency). Averaging Pareto frontier margins across budgets, as detailed in Appendix G and Table X and summarized Figure [ we find average gains for DMS of 12.5 for AIME 24, 2.3 for MATTI-500, 5.8 for GPQA Diamond, and 8.3 for LiveCodeBench. Variability across datasets primarily reflects their saturation levels; for instance, models achieve very high performance MATH-500 even with limited budgets. Notably, performance gains from DMS decrease with increasing model scale on MATH-500, yet increase with scale on GPQA Diamond and LiveCodeBench. Overall, these findings indicate that KV cache compression exhibits more favorable scaling properties than full KV retention in vanilla LLMs, highlighting its potential for advancing their reasoning capabilities.
Moreover, comparing DMS with other KV cache compression methods, we find that its Pareto frontier clearly dominates the best baselines for both efficiency metrics: Quest for KV cache memory reads and TOVA for peak tokens in memory (Figure This even more remarkable considering that Quest sacrifices memory efficiency by fully preserving the KV cache mitigate accuracy
7

---

Table 1: Evaluation of Llama 3.2 1B Instruct on broader array of tasks, across different methods and compression ratios (CR). We note that due to full-dense attention prefill, Quest is equivalent to vanilla on MMLU and HellaSwag. The DMS model used in this comparison window of just 16 tokens. As datasets, we include GSM8K (Cobbe et al. (021) for grade-school
trained with sliding
math, MMLU (Hendryeks et al. 2021a) for factuality, HellaS wag (Zellers et al. 2019) for zero-shot
common-sense question answering, and Needle in a Haystack (NIAH: Kamradi 2023 and Variable
Tracking (VT: Hsieh et al. 2024 for long context processing.
CR
1x
2x
3x
4x
win-16
Method
Vanilla
H2O
TOVA
Quest
DMC
DMS
H2O
TOVA
Quest
DMC
DMS
H2O
TOVA
Quest
DMC
DMS
GSM8K
47.0
44.0
45,0
45.1
31.9
46.9
32.9
40.1
44.7
6,4
46,5
14.7
20.2
39.9
3.6
42.3
MMLU
47.9
45.7
43.4
47.9
34.9
48.0
37.6
38.1
47.9
26.3
45.2
32.7
35.2
47.9
25.6
40.3
HellaS
43.4
42.9
42.8
43.4
42.2
43.3
42.1
42.5
43,4
40.0
43.3
41.3
41.8
43.4
39.4
43.4
NIAH
96.4
34.0
65.2
95.8
0.0
97.8
17.2
40.2
95.6
1.8
93.6
13.4
28.0
95.8
0.0
96.8
VT
55.8
27.4
56.2
53.0
0.0
63.2
17.6
45.2
50.4
0.2
69.2
12.6
33.8
49.6
4.0
67.6
degradation- and yet DMS still offers a better latency-accuracy trade-off. Datasets like MATH-500, where Quest's Pareto frontier mostly overlaps with vanilla at all scales, illustrate that gains from larger token budgets can be eaten away, unless performance is retained even at high CRs. DMS meets this desideratum in a data-efficient way, thus offering inexpensive hyper-scaling with existing LLMs.
Zooming in on specific results, we can assess which W-L-CR configurations tend to lie on the Pareto frontier for DMS. For most tasks, these consist of combination of sequential and parallel scaling, hinting at the necessity of using both for inference-ting scaling. Moreover, most DMS points (for CRs of 4x and 8x) lie on the Pareto frontier. indicating that even higher compression retains sufficient quality to afford superior trade-offs.
Latency and Throughput Measurements Next, we investigate how the reduced memory reads and peak memory of DMS translate into latency and throughput, measured on an NVIDIA H100 SXM GPU in 16-bit precision, using a simple implementation based on the Hugging Face Transformers library and of DMS and vanilla Qwen-RT models for different context lengths in Figure Initially, latency roughly constant, but it rises early due to the increasing cost of reading the KV cache. The exact context length at which this occurs depends primarily on token batch size, which reflects both the number of queries served and the number of reasoning traces per query in parallel scaling. In addition, the vanilla LLM can exhaust VRAM quickly at high batch sizes. For more details, consult Appendix !!
(Dao 2024). First, we illustrate the latency of a single generation step
In real-world scenarios, LLM systems should serve many queries as possible while retaining high quality. Hence, we compare the throughput of DMS and vanilla LLMs. As Figure 3 (bottom) illustrates, at equivalent accuracy AIME 24 and LiveCodeBench (determined by the specific inference-time hyper-scaling configuration), DMS models can serve significantly more queries in parallel when using the maximum batch size that fits in memory. This demonstrates that the gains observed in Figure translate into effective speedups when deploying LLM systems with DMS.
5.2
DMS for General-purpose LLMs
Moreover, we aim to establish whether DMS is effec- tive beyond inference-time scaling settings, so that a model retrofitted with DMS can be reliably deployed as a general-purpos LLM. To this end, we first com- pare DMS with respect to vanilla models for equiva- lent generated token lengths (rather than actual com- pute budget). We focus again the same models and datasets as Section 5.1 From Tables To 12 in Appendix G it emerges that DMS mostly preserves the original accuracy at CR 4x and yields minimal degradations at CR 8x.
Table 2: Evaluation of DMS 8x Qwen3-8B with a sliding window of 512 tokens.
win=512
Benchmark
Think
Vamilla
DMS 8x
GPQA Diamond
58.8
$7.6
MMI.U-rr
V
74.2
73.5
AIME 2024
V
75.0
73.0
MATH-500
V
95.1
95.5
HumanEval
87.8
89.6
IFEval
90.3
88.8
ArenaHard v0.1
88.4
89.7
We also benchmark Qwen3-8B on broader set of tasks in Table The tasks include math and science
(GPQA Diamond, AIME 2024, MATH-500), factuality (MMLU-Pro Wang al. 2024b), coding
8

---

xx
DMS win-O (immediate)
DMC
40
20
1x
4x
4x
0-
4x
DMS win-16
40
5x
GSMSK Score
20
DMS w6
54
DMS win- (immodiate)
6x
6x
Ax
48
2x
48
40
6x
5x
6x
win-64
6x
win 64 (immediste)
1x
2x
3x
6x
5x
6x
40
20
DMS 256
DMS win=256 (immediate)
DMS win
0
20
Training tokens (R)
Training tokens
Figure 5: GSM8K 0-shot scores of Llama 3.2 1B Instruct across different compression variants. Left: delayed eviction (default) with 16-token window consistently preserves reasoning abilities of the model, while immediate eviction causes rapid degradation. The quality gap only widens as the compression gets stronger. Right: DMS requires an order of magnitude less data to train than DMC. This was also observed for Qwen 2.5 RI models with 1.5B, 7B, and 32B parameter scales.
(HumanEval; Chen et al. 2021) conversation (ArenaHard v0.1; Li et al. 2025 and instruction following (IFElal: Zhou al.) 2023). Technical details for the experimental setup provided in Appendices G and H From Table 2 it emerges that DMS is within close range of the accuracy of vanilla Qwen3-8B. Moreover, in Figure we compare the vanilla and DMS Qwen3-8B models in terms of the accuracy-thin thrruacyyhhrouuhputt trade-off during inference scaling on LiveCodeBench and find that DMS consistently matches the accuracy of vanilla, while allowing up 5x higher throughput.
Finally, as way to ensure that DMS's sparse prefilling does not affect performance in short generation settings, we evaluate Llama 3.2 1B Instruct, small, non-reasoning model, a broad set of tasks (Table T). In this setting DMS also stands out as the most robust method for accelerated inference. Overall, DMS's accuracy retention at high compression ratios makes it suitable not only for inference- time hyper-scaling but also for
use, independent of the context or generation length.
5.3 Ablations
The design choices in DMS were informed by results of small-scale experiments. We present ablations on eviction policy and data efficiency during retrofitting of the Llama 3.2 1B Instruct models. To evaluate the impact of delayed eviction, we trained additional models with immediate eviction, which aligns more closely with existing token eviction methods:
Delayed eviction: determines the eviction of (kg, vz) at future time step t + Immediate eviction: at+w determines the eviction past (k,, vt) at time step t + w
Both policies were tested with different sliding window sizes. Remarkably, DMS retains reasoning capabilities with a window of only tokens up to a compression ratio of 4x, as shown in Figure Larger sliding windows better preserve reasoning capabilities, which is expected in zero-shot setting. In contrast, immediate eviction drastically deteriorates scores for every tested sliding window length.
Regarding data efficiency, the right panel of Figure shows how scores vary when retrofitting with different training token budgets. Crucially, DMS achieves higher scores than DMC while using 8x fewer training tokens. In practice, the reasoning models described in Section [.1 were trained with x0x less training data achieving CR 4x within 300 training steps and CR 8x within 700 steps.
'DCC was reported to require 44K training steps reach CR8, with performance deteriorating when the amount of data halved (Nawrot et al. 2024).

---

Compressee 220
CR2
06
KVHooal
CRA
04
10000
20000
30000
Conted length
Contect length (logs
12
Laysr
24
Figure 6: Left: The measured compression ratio for Qwen-R1 7B, trained with DMS CR 4x, while processing AIME 24, MATH-500, and GPQA Diamond problem instances. Right: Average per-head compression learned by the model, as a percentage of retained tokens sorted for every layer.
Finally, we measured how the CR varies for different lengths of the sequences generated through DMS FFigur/@]eftt, The resulting pattern closely matches that reported in Nawrot al. (2024). The model sparsifies less than the target CR early parts of a sequence, but even more aggressively than specified beyond 10K tokens. This behavior stems from the training objective and from the tendency of the conditional entropy rate of natural-language text to decrease as the context length grows. In Figure O (right), also observe that early layers are compressed smaller degree than later layers.
5.4 Discussion
Research inference-tine scaling has far mostly assumed an equivalence between compute budget and generated tokens, in terms of sequence length or parallel samples (Brown et al. 2024 Zhang al. 2023b; Wang et al.; (023). This budget can be allocated adaptively to the complexity of the task (Snell et al. 2024) or forced to meet an amount pre-defined by the user (Muennighoff et al.) 2025). To the best of our knowledge, the effective compute (latency and peak memory load) when reasoning in the discrete language space. In fact, we show how KV cache compression methods same compute budget. A separate family of strategies are based on latent space reasoning (Geiping al. 2025), which add recurrent block on top of Transformer LLMs; however, this effectively requires separate architecture rather than boosting existing LLMs, and it remains unclear whether these scale similarly reasoning in the discrete token space.
work is the first fully disentangle generated tokens from
effectively expand the token budget for the
While in this work we opt for verifier-free scaling strategies, adopting the recommendations of Wang et al. (2025b), inference-time scaling can rely on process reward models (PRMs) verify intermediate reasoning steps. This allows effective self-critique loops and re-ranking candidate solutions (Uesato al., 2022) [Lihhtman al., 2024) [iang al., 2024). Nonetheless, remark that hyper-scaling can be extended to PRM strategies, too. In particular, the verifier's complexity quadratic in the sequence length; complement the benefits of KV cache compression of the LLM, the PRM would need to be accelerated by prefilling-tin sparse attention methods, such as MInference (Jiang et al. 2024). We leave this possible direction to future work.
Conclusions
We introduce inference-time hyper-scaling: by compressing the key-value cache of Transformer LLMs via sparse attention, we improve downstream reasoning accuracy by enabling longer token sequences more parallel sequences at the same compute budget-in terms of latency or memory- compared to the original LLM. A fundamental requirement of inference-tie sitrence--imee hyper-scaling is to increase efficiency without sacrificing accuracy. Sparsification (DMS), novel, trainable KV cache reduction method that delays eviction decisions, while remaining highly data-efficient. Empirically, we observe large gains on benchmarks involving advanced math, scientific problems, and coding, demonstrating the effectiveness of hyper-scaling. Overall, our approach provides an inexpensive strategy for converting L.LMss into more effective reasoners, pushing inference-tine time scaling to new frontiers.
achieve this, we propose Dynamic Memory
10

---

Acknowledgments
This work is supported by the ERC Starting Grant AToM-FM (101222956) awarded Edoardo M. Ponti. The authors would like to thank Marcin Chochowski, David Tarjan, and Andrzej Suiecki for helpful discussions, Szymon Migacz for his assistance with the computing infrastructure, as well as Przemyslaw Strzelczyk, Krzysztof Pawelec, Daniel Korzekwa, Alex Fit-Florea, and Michael Lightstone for support in releasing this paper.
References
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. GQA: Training generalized multi-query transformer models from multi-head checkpoints In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing.
Edward Beeching, Lewis Tunstall, and Sasha Rush. 2024. Sccaling test-time compute with open models
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and
Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated
sampling Preprint, arXiv:2407.21787.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji. Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. 2021. Evaluating large language models trained on code. ArXiv, abs/2107.03374.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the A12 reasoning challenge Preprint, arXiv:1803.05457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems Preprint, arXiv:21 14..4
Tri Dao. 2024. FlashAttention-2; Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR).
DeepSeck-AL. 2024. DeepSeck V2: A strong, economical, and efficient mixture-of-experts language model Preprint, ar XXXv:2405.004334
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. 2024. Flex Attention: A programming model for generating optimized attention kernels, Preprint, arX.w:2412.0549%
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving factuality and reasoning in language models through multiagent debate In Forty-first International Conference Machine Learning.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. 2023. AlphaZero like tree-search can guide large language model decoding and training NeuriPS 2023 Foundation Models for Decision Making Workshop.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. The language model
evaluation harness
11

---

Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. 2025. [Scaling up test-time compute with latent reasoning: A recurrent depth approachi Preprint, mrXiv:2502.05171.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. 2024. The Llama herd of models Preprint, arXiv:2407.2
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Depssek-RI: Incentivizing reasoning capability in LLMs via reinforcement learning. Preprint, arXiv:2501.12948.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. 2021a. Measuring massive multitask language understanding In International Conference on Learning Representations.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Mcasuring mathematical problem solving with the MATH dataset In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in neural network Preprint,
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization, In Advances in Neural Information Processing Systems, volume 37,
pages 1270-1303.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What's 's the real context size of your long-context language models? In First Conference on Language Modeling.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. [ivecodebench: Holistic and contamination free evaluation of large language models for code In The Thirteenth International Conference on Learning Representations.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. 2024. Minference 1.0: Accelerating pre-filling for long-context LLMS via dynamic sparse attention In Advances in Neural Information Processing Systems, volume 37, pages 52481-52515.
G. Kamradt, 2023.
https://githis com/ wwt.:kaaaaa..t..../////.... gkamradt/
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention In Proceedings of the 29th Symposium on Operating Systems Principles.
Hynek Kydlíček and Greg Gandenberger. 2025. Math-verify
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap. Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez. and Ion Stoica. 2025. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline In Forty-second International Conference on Machine Learning.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023. Making large language models better reasoners with step-aware verifier Preprint, arXiv:2206.02336
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. SnapKV LLM knows what you are looking for before generation] In Advances in Neural Information Processing Systems, volume 37, pages 22947-22970.
12

---

Zhenwen Liang. Ye Liu, Tong Niu, Xiangliang Zhang. Yingbo Zhou, and Semih Yavuz. 2024. Improving LLM reasoning through scaling inference computation with collaborative verification Preprint, ar arXiv:2410.05318.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In The Twelfth International Conference on Learning Representations.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A tuning-free asymmetric 2bit quantization for KV cache In Proceedings of the 41st International Conference on Machine Learning volume 235 of Proceedings of Machine Learning Research, pages 32332-32344.
Christos Louizos, Max Welling, and Diederik Kingma. 2018. Learning sparse neural networks through Lo regularization In International Conference on Learning Representations.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Seef-refinne Iterative refinement with self-feedback In Advances in Neural Information Processing Systems, volume 36, pages 46534-46599
Benjamin Minixhofer, Ivan Vulié, and Edoardo Maria Ponti. 2025. Cross-tokenizer distillation via approximate likelihood matching Preprint, arXiv:2503.20083.
Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark Attention: Random-access infinite
context length for transformers, In Advances in neural information processing systems.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer. Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025, sl: Simple test-time scaling Preprint, arXiv:2501.19393.
Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. 2024. Dynamic memory compression: Retrofitting LLMs for accelerated inference. In Forty-first International Conference on Machine Learning.
Piotr Nawrot, Robert Li, Renjie Huang, Sebastian Ruder, Kelly Marchisio, and Edoardo M Ponti. 2025. The sparse frontier: Sparse attention trade-offs in transformer LLMs Preprint, arXiv:2504. 17768.
NVIDIA. 2024.
Megatron-L.M: Ongoing research training transformer models at scale
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, et al. 2024. OpenAl ol system card Preprint, arXiv:2412.16720.
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. 2024. Transformers are multi-state RNNs In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18724-18741, Miami, Florida, USA.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level Google-proof Q&A benchmark In First Conference on Language Modeling.
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. 2024. Eigen attention: Attention in low-rank space for KV cache compression, In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13332-15344, Miami, Florida, USA.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models In International Conference on Learning Representations.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM test-time compute optimally can be more effective than scaling model parameters Preprint, arXiv:2408.03314
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, et al. 2024. LLM pruning and distillation in practice: The Minitron approach, Preprins, arXiv:2408. 11796.
13

---

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. 2024. QUEST Query-aware sparsity for efficient long- long-context LLM inference In Proceedings of the International Conference on Machine Learning (ICML).
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic. Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models Preprint, htt.::237.009288.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process and outcome- based feedback Preprint, arXiv:2211..4222
Guoxia Wang. Jinle Zeng, Xiyuan Xiao, Siming Wu, Jiabin Yang, Lujing Zheng, Zeyu Chen, Jiang Bian, Dianhai Yu, and Haifeng Wang- 2025a. Flashmask: Efficient and rich mask extension of flashattention In The Thirteenth International Conference on Learning Representations.
Junlin Wang, Shang Zhu, Jon Saad-Falcon, Ben Athiwaratkun, Qingyang Wu, Jue Wang, Shuai- wen Leon Song, Ce Zhang, Bhuwan Dhingra, and James Zou. 2025b. Think deep, think fast: In- vestigating efficiency of verifier-free inference-time-scall methods, Preprint, arXiv:2504. 14047.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024a. Math-shepherd: Verify and reinforce LLMS by-p-by-ttep without human annotations) In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426-9439, Bangkok, Thailand.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models In The Eleventh International Conference on Learning Representations.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024b. Mmmlu-pro A more robust and challenging multi-task language understanding benchmark] In Advances in Neural Information Processing Systems, volume 37, pages 95266-95290. Curran Associates, Inc.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025. Qwen3 technical report Preprint, arXiv:2505.099 00988
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models Advances in Neural Information Processing Systems, volume 36, pages -1822
14

---

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang. Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang. and Wangding Zeng. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention Preprint, arXiv:2502.11033
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. 2024. LoRC: Low-rank compression for LLMs KV cache with a progressive compression Strategy In Workshop on Machine Learning and Compression, NeurIPS 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen, 2023a. H20: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems 36.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompting in large language models In The Eleventh International Conference Learning Representations.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. cssructioo-following evaluation for large language models Preprint, LtXiv:2311.07911
Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa. Gaowen Liu, Dawn Song, and Xin Eric Wang. 2025. Thh hidden risks of large reasoning models: safety assessment of RI Preprint, arXiv:2502. 12659.
15

---

A Limitations, Future Work and Impact
Larger Model Sizes, Longer Contexts, and Higher Compression Ratios In this work, we focus on models ranging from 1B to 32B parameters, context lengths to 32K tokens, and compression ratios up to 8x. Exploring even larger models, longer contexts, and higher compression ratios remains an exciting avenue for future research.
Integration with Other Efficient Attention Mechanisms We demonstrated DMS with the stan- dard multi-head attention mechanism used in Transformer-basee models such Llama and Qwen. Extending DMS to alternative attention variants, such as Multi-head Latent Attention (DeepSeek-Al 2024) represents a promising direction for future investigation. Moreover, DMS compresses the KV cache, whereas Quest (Tang et al., 2024) selectively retrieves cache items. Hence, the two are orthogonal and could be combined to further push the Pareto frontier for inference time scaling.
Our approach does not introduce novel risks. However, it may amplify existing concerns associated with large-scale reasoning models. For detailed analysis of these risks, we refer readers to Zhou et al. 2025).
Broader Impact
B Related Work for KV Cache Size Reduction
The challenge of KV cache reduction has garnered significant interest in recent years, with approaches falling into three main categories: attention sparsification, quantization, and decomposition. In addition to the sparse attention baselines considered in Section 2.2 Landmark Attention (Mohtashami
and Jaggi (202)) and Native Sparse Attention (Yuan et al. 2025) create representations for each KV cache chunk and retrieve only the most important chunks for attention computation, effectively reducing the amount of data transferred from the device HBM memory. Compared to these methods, DMS not only accelerates inference but also reduces memory load and allows for dynamically selecting different compression ratios across layers and heads based on the input. Morcover, DMS improves on other retrofitting methods, such as DMC (Nawrot al. 2024). both in terms of data efficiency and downstream accuracy.
Another strategy for KV cache size reduction is quantization, exemplified by methods such KIVI (Liu et al. (2024) and KVQuant (Hooper et al. 2024). which quantize keys per channel and values per token. Finally, KV cache reduction can be achieved via SVD-based decomposition. LoRC ZZhang et al. 2024) directly reduces the ranks of key and value matrices, whereas Eigen Attention (Saxena
al. 2024) moves the attention computation into truncated space induced by SVD. While being
less expressive than DMS as they assume uniform compression, both quantization and decomposition orthogonal DMS and can be potentially combined with it to further improve efficiency.
16

---

C
Additional Details for Retrofitting
DMS Implementation Unlike (Nawrot al. 2024), which extracts a, from key representations affecting all query heads in group, we 'borrow' the first neuron from the first query head in each query group and use it extract og eliminating the need for additional parameters while minimizing the impact on attention computation. This requires a short continued training, during which we gradually zero out the first dimension the first query head in each group: Qt,frr: [0) <-
qs,ūrs: [0] (1-1)) where denotes the current training step and n, 2000. After this initial stage, the models are ready for the main DMS retrofitting phase, where they learn to dynamically evict tokens. After we extract ar from the first query head, we set fr. = to avoid d influence on the result of attention calculation, while leaving other query heads in the group unaffected. We note that instead borrowing the neuron from a query head, one could use a separate, trainable, initialiiziiz projection from the hidden state extract at, effectively eliminating the need for continued training.
Training Configuration We use batch size of 1024 following the original Llama recipe (Touvron et al. 2023). Context lengths are set to 4096 tokens for Llama 3.2 1B Instruct and Llama 2 7B models, and 8 192 tokens for Llama 3.1 8B and RI R1-distilled models to accommodate the longer sequences required by AIME and MATH-500 benchmarks. For Qwen3-8B we use 256 batch size and 32K context length.
Default DMS Configuration Unless otherwise specified, all DMS models use delayed eviction with sliding window of 256 tokens and increment the compression ratio by 1 every 100 training steps. We denote different DMS variants using the notation DMSwin where y represents the sliding window size. Unlike DMC (Nawrot al., 2024), omit the third fine-tuning phase (training with fixed compression ratio) as it provided negligible benefits for DMS.
Infrastructure and Computational Requirements All models trained NVIDIA H100 GPUs using Megatron-LMM (NVIDIA) 2024) in 6 precision, with optimizer states stored in FP32. We provide details regarding the computational costs in Table 3
Table 3: Cost retrofitting the model from CR to CR i We note that over the course of single retrofitting run, the target CR is linearly increased from CR I to the target CR. As result, single run produces a family of models with different compression ratios.
Model Family
#Params
CR i + CR itt 1
BS
Context
Window
Steps
LR
GPU Hours
1B
4096
256
100
1e-5
10
Llama 3
1024
8B
8192
16
100
3e-5
100
1.5B
1e-5
30
Qwen-R1
7B
1024
8192
256
100
3e-5
32B
3e-5
345
Qwen3
8B
256
32768
512
100
3e-5
270
17

---

D Additional Inference- Time Scaling Results
DMS CR4
Quest CR4
TOVA CR4
Vanilla CR]
DMS CR&
Quest CR8
TOVA CR8
Qwen-RI 1.5B
Qwen-RI 7B
Qwen-RI RR 32B
8-15K
4-16K
96
4-16K
4-1
MATH
3.5
1-52K
1:32K
J.EKK
94
1-22
JAK
92
-KK
92
2
GPQA Diamond
R-SK
AKC
-32K
55
4-168
70
C.SK- AK
3.5
65
50
1-32K2
1122K
322
2
KV cache memory reads
9-52K
8-10k
4-16K
16K
MATH 500
15K
4-16k
95
1-32K-4
1: 1-325
1-15K
35
1:328
L.32
GB.
SKK
94
80
4-8K
92
92
1-8K
2
2
GPQA Diamond
4-K
4-12K
8-3K
55
4-10K
++
1-32K
1-10
1-323
21:5K
1-22K
122
45
2
2
2
2
Peak tokens in memory
A-TAK
E-16K
4:16K
90.0
MATH--500
8.
87.5
4.3%
4-8K
2-16K
1-8K
1.8%
85.0
116KK
ALLSK
$1
82.5
92
1.5K
2:8K
1-8K
80.0
92
GPQA Diamond
40.0
A-K
6AK
55.0
4-K
70
AAK
37.5
JaLSK
8K
52.5
35.0
J-16K
65
50.0
1-1684
47.5
1.8K
2
2"
Throughput (examples/min)
Figure 7: Inference-ting time scaling results calculated on MATH-500 and GPQA Diamond. The methods are compared in terms of accuracy on y-axis and efficiency metrics on r-axis: (top) KV cache memory reads, which serve as a proxy for attention compute; (middle) the maximum number of used tokens, as a proxy for memory load, and (bottom) throughput measured on NVIDIA H100 SXM GPU. For details, please refer to Figure and Section 5.1
18

---

Qwen3-8B
70
X-38152K
65
3-32K
32K
LiveCodeBenci
60
7-16K
5. 16K
3-32K
3-32K
Vanilla
3-16K
1-32K
2- 1-16K
5- 1-IKK
DMS CR4
DMS CR8
3-16K
50
--66
DMS CRI2
M-32K
32K
45
J-16K
J-16K
40
2-
2
2
2
Throughpul (exumples/min)
Figure Throughput measured with maximum batch size that accommodates the particular inference- time scaling configuration. The plot compares DMS-enabled Qwen3-8B models at compression rations 4x, 8x and 12x the vamilla model. emerges from the plot that, through the combination of inference- time scaling and KV cache compression, DMS enables matching the vanilla results with up 5x higher throughput, effectively lowering the cost of serving the model in multi-user environment.
Training Data
For the Qwen-RI R1 models, utilize logit distillation leveraging the OpenRI-Math 220k dataset. This dataset contains high-quality reasoning traces sampled from DeepSeek R1. To further enhance data quality, apply filtering step using Math-Vor Verify (Kydliček and Gandenberger 2025), retaining only traces resulting in correct mathematical solutions.
For the Llama 3.2 IB Instruct model, the training corpus comprises two main components: (1) carefully curated set programming language examples covering languages such as Python, C, and C++, and (2) synthetic data generated by prompting the model. In particular, we utilize the Llama 3.2 IB Instruct model itself to produce completions for the one-dimensional linear algebra subset DeepMind mathematics dataset (Saxton et al. 2019), which follows the structured format:
Task format in one-dimensional linear algebra
cX d for X. Solve aX b
Llama 3.2 prompt for generating responses
_eadee__dddd_dd>> <lstart_header saaaaerr._dlysstttsteeedeeeeeedd
Cutting Knowledge Date: December
Today Date: 23 July 2024
You helpful assistant. assistant.<leot.u.. <leot_idlll ltarl//sstttt_hheeeaeeee httpeeeeeeeddddeeeeeeeeeddddddd end_haeerrdde
Given the following problem, reason and give a final answer to the problem.
Problem: Solve 5+b 2355 -50+b 2740 for b. Your response should end with "The final answer is [answer] " where [answer] is the response to the
In contrast with the data mixture for Qwen-RI models, we do not perform correctness filtering on this synthetic, model-generated dataset.
19

---

Additional Downstream Evaluations for DMC and DMS
In Table we show that DMS can extrapolate beyond the retrofitting context length of 4K, whereas DMC may fail to so. In T we show comparison between Vanilla model, DMS, Quest, and DMC on Llama 7B.
Table 4: Needle in the Haystack and Variable Tracking results for IB parameter Llama 3.2 Instruct model. We note that in contrast to DMC, DMS can extrapolate beyond the retrofitting context length. Note that on the heavily compressible Variable Tracking task, DMS achieves significantly higher scores than the vanilla model.
Method/Task
NIAH
VT
Context
3K
4K
8K
3K
4K
8K
Vanilla
99.4
96.4
97.2
61.4
55.8
41.2
CR2
TOVA
62.8
65.2
75.0
56.0
56.2
49.8
H2O
29.0
34.0
37.0
25.6
27.4
21.4
Quest
99.2
95.8
97.4
60.0
53.0
36.4
DMC
99.0
0.0
62.4
0.0
DMS 16
99.0
97.8
99.4
72.0
63.2 63.2
56.0
CR3
TOVA
25.8
40.2
41.6
38.4
45.2
40.6
H2O
16.6
17.2
19.8
15.6
17.6
13.4
Quest
99.0
95.6
97.0
60.4
50.4
31.8
DMC
99.0
1.8
0.0
46.4
0.2
1.2
DMS win 16
99.2
93.6
24.2
76.2
69.2
58.8
CR4
TOVA
16.8
28.0
26.4
31.4
33.8
30.2
H2O
9.4
13.4
12.8
11.8
12.6
11.0
Quest
98.4
95.8
97.6
57.4
49.6
32.4
DMC
97.0
0.0
0.0
48.6
4.0
0.8
DMSwin=1e win=11
99.4
96.8
12.2
74.8
67.6
57.2
Table 5: Results for base Llama 2 7B parameter models. Both DMS and DMC were trained using LM-loss without logit distillation. Since these models are not instruction-tuned, we evaluate with 8-shot prompting GSM8K, 5-shot MMLU, 1-shot Needle in Haystack, and zero-shot ARC-Challenge and HellaSwag. (Nwrrot et al. 2024).
Method
ARC-C
GSMSK
HS
MMLU
NIAH
Vanilla
45.6
14.9
75.5
45.4
100.00
CR4
DMSwin=16
45.8
14.2
76.0
43.7
100.0
Quest
45.6
14.5
75.5
45.4
100.0
DMC
46.2
12.2
76.3
43.9
100.0
CR8
DMS win=16
46.2
10.5
76.4
40.2
60.0
Quest
45.6
11.6
75.5
45.4
100.0
DMC
44.7
10.0
75.4
41.7
100.0
20

---

Table I compares downstream performance at 2 3x, and 4x compression ratios DMS stands out as the most robust method, achieving higher scores than both training- free and retrofitted baselines in most combinations of tasks and CRs, with Quest as second-best contender. While in short-context tasks, DMS performance is close the original L.M,, long-context tasks (such as NIAH and VT) DMS even surpasses it. Moreover, long-com long-context performance provides evidence that-compare with DMC-DMS is more successful at extrapolating compression lengths beyond those observed during retrofitting, albeit only up certain limit (see Appendix F Among learned compression methods, DMC collapses quickly, likely due to its more challenging training objective amplified by the limited IB model capacity
G Downstream Results Significance
We provide further analysis regarding the statistical significance and robustness of our experimental results. Specifically, we report standard errors for the Llama 3.2 1B Instruct models in Table 6 and quantify the average Pareto improvement in Tables 8 and 9 To precisely measure the Pareto improvement, we extract Pareto frontiers for DMS, the best KV cache reduction baseline, and the vanilla baseline from Figures 3 and (top and middle). Then, for each task and model size, we identify the largest common budget interval shared by each pair of methods A and B, and compute the average improvement as:
Jaer (A(x) B(x)) dr [1]
where Axx) and (xx) denote the best accuracy achieved by method A and B, respectively, at budget J. For budget values not explicitly measured, we employ linear interpolation.
Table 6: Results from Table expanded with standard error computed by Language Model
Evaluation Haresss (Gao et al. 2024).
Method
ARC-C
GPQA
GSM8K
HS 44.941.4
Vanilla
31.2+1.4
25.042.
43.4
CR2
DMS DMSwin=1e win=16
31.3
25.7
46.6
43.3
TOVA
29.6
25.2121
45.0
42.8 -0.5
H2O
31.1
26.812.1
44.0
42.9
Quest
31.2+14
25.0+2.0
45.1+1.
43.4
CR3
DMSwin=1e
DMS
win=16
31.1+1.4
24.642.
45.541.4
43.3
TOVA
30.0+1.3
23.742.
40.1+1.
42.5
H2O
31.2+1.4
24.342.
32.941.3
42.1
Quest
31.2
25.042.
44.711.4
43.4
CR4
DMS
win=16
31.111.4
24.3
41.0
43.4
TOVA
29.011.
23.712.
20.2
41.8
H2O
27.5
23.7
14.7
41.3
Quest
31.2+14
25.0420
39.941.
43.4
'While H2O and TOVA designed for long context tasks with large sliding windows, consciously evaluate them with short sliding windows meet target CRs. "Nevertheless, in Appendix F we show that, while still lagging behind, this collapse does not occur for shorter contexts and larger non-GQA model.
21

---

Table 7: Throughput-Accuracy Pareto frontier difference. We use linear interpolation for the unknown values of the frontier.
Method
AIME
MATH 500
GPQA Diamond
LiveCodeBench
1.5B
7B
32B
1.58
7B
1,58
7B
32B
1.5B
7B
32B
DMS Vanilla
-1.5
10.1
7.4
5.0
2.2
3.3
5.6
6.0
8.4
8.1
7.3
13.2
Table 8: KV Reads-Accune Accuracyy Pareto frontier difference. We linear interpolation for the unknown values of the frontier. NA denotes that the projections of the Pareto frontiers on the budget axis are disjoint.
Method
AIME 24
MATH 500
GPQA Diamond
BiveCodeBench
1.5B
32B
1.5B
7B
32B
1.5B
7B
32B
1.5B
7B
32B
DMS vs Vanilla
10.6
15.0
12.0
4.2
1.0
1.6
4.8
4.1
8.6
7.3
7.9
9.7
Quest vs Vanilla
-6.8
3.1
2.0
1.8
0.6
NA
NA
NA
2.3
2.5
5.0
3.8
DMS Quest
18.8
13.5
5.8
6.2
2.1
1.4
NA
NA
NA
4.9
3.4
5.6
Table KV Memory Usage-Accuracy Pareto frontier difference. We use linear interpolation for the unknown values of the frontier. NA denotes that the projections of the Pareto frontiers on the budget axis are disjoint.
Method
AIME 24
MATH-500
GPQA Diamond
LiveCodeBench 7B
1,5B
7B
32B
1.5B
7B
32B
1,5B
7B
32B
1.5B
32B
DMS vs Vanilla
15.7
0.5
8.5
16.7
TOVA Vanilla
-0.2
2.6
1.3
11.2
1.8
-1.1
-2.1
2.1
3.3
4.9
DMS vs TOVA
9.6
15.6
8.1
2.3
1.8
5.6
6.5
6.3
4.0
6.0
11.1
Table 10: Results from Figure 3 and Figure 7 (top) specified max Length and Width=1 configurations. Those points allow for direct comparison with Vanilla model.
Task
Model
Size
CTX
Vanilla
DMS CR
Quest CR4
1.5B
32k
30.0
30.0
26.7
AIME 24
Qwen-R1
7B
32k
53.3
53.3
55.5
32B
32k
70.0
73.3
73.3
1.5B
32k
84.8
84.0
84.1
MATH 500
Qwen-R1
32k
94.0
92.8
93.2
32B
32k
94.8
94.6
94.6
1.5B
32k
36.5
36.9
37.0
GPQA Diamond
Qwen-R1
7B
32k
51.5
48.5
50.2
32B
32k
63.1
63.6
65.4
1.5B
16k
17.3
17.0
15.8
L LiveCodeBench
Qwnn-R1
7B
16k
35.9
34.4
33.1
32B
16k
57.0
54.8
54.5
22

---

Table 11: Results from Figure 3 and Figure 7 (middle) specified max Length and Width=1 configura- tions. Those points allow for a direct comparison with Vanilla model.
Task
Model
Size
CTX
Vanilla
DMS CR4
TOVA CR4
1.5B
32k
30.0
30.0
30.0
AIME 24
Qwen-R1
7B
32k
53.3
53.3
46.7
32B
32k
70.0
73.3
70.0
1.5B
32k
84.8
84.0
84.3
MATH 500
Qwen-RI
7B
32k
94.0
92.8
91.8
32B
32k
94.8
94.6
95.2
1.5B
32k
36.5
36.9
34.3
GPQA Diamond
Qwen-RI
32k
51.5
48.5
47.5
32B
32k
63.1
63.6
63.1
1.5B
16k
17.3
17.0
14.9
LiveCodeBench
Qwen-R1
7B
16k
35.9
34.4
30.7
32B
16k
57.0
54.8
51.1
Table 12: Results from Figure 3 and Figure comparing DMS wit CR8 to Vanilla (CRI) on specified max Length and Width=1 configurations.
Task
Model
Size
CTX
Vanilla
DMS CR8
1.5B
32k
30.0
23.3
AIME 24
Qwen-RI
7B
32k
53.3
50.0
32B
32k
70.0
73.3
1,5B
32k
84.8
80.0
MATH 500
Qwen-R1
7B
32k
94.0
93.0
32B
32k
94.8
94.6
1.5B
32k
36.5
31.3
GPQA Diamond
Qwen-R1
7B
32k
51.5
46.5
32B
32k
63.1
64.6
1.5B
16k
17.3
16.1
LiveCodeBench
Qwen-RI
7B
16k
35,9
33.4
32B
16k
57.0
51.7
23

---

H Evaluation Details
H.1 Implementation of TOVA, H2O, Quest, and DMC
For TOVA (Oren et al., 2024). H2O Zhang et al. 2023a and Quest (Tang et al., 2024), we calculate the K KV-budget by summing the input length and the maximum generation length, then dividing by the compression ratio. For H2O, the KV-budget is evenly split between the recent cache and the heavy-hitter cache. During evaluation, memory-optinisinn methods such as TOVA and H2O first perform a standard prefill phase until the KV-budget is reached and subsequently switch to their respective memory-efficiint mechanisms.
Quest (Tang et al. 2024), unlike TOVA, H2O, DMC, and DMS, does not reduce the KV cache memory footprint. Thus, following the authors' recommendations, full dense attention and set the block size to (16 2cr). This configuration provides Quest with an advantage over the other methods. Additionally, which can result in an increased number of memory transfers for Quest compared to DMS, DMC, TOVA, and H2O. However, the computational cost remains similar. We use this approach Quest was originally designed for models without GQA, and we wanted to avoid a custom modification that could potentially degrade the performance. In plots regarding kv-cache memory reads we calculate the total number of different blocks retrieved from a single key-head. That is we assume optimal implementation that makes use of topk intersections between query heads and retrieves each block only once.
permit Quest to perform prefilling using
employ separate top-k for each query head,
For DMC, follow the implementation described in the original paper (Nawrot et al. 2024).
H.2 Downstream Tasks
We evaluate all downstream tasks in zero-shot setting, unless stated otherwise.
For Qwen3 results in Table 2 we evaluate the model using temperature= 0.6 and top-p = 0.95 with sequence length limit of 131072 tokens. AIME 2024 results were averaged over 10 runs (different seeds) and MATH-500 over 3; MMLU-Pro uses micro-averaging.
For GSM8K (Cobbe et al., 2021). MMLU (Hendrycks et al. 2021a). ARC-Challenge (Clark et al. 2018 and HellaSwag (Zellers et al. 2019 we use the Language Model Evaluation Harness (Gaa
et al.
2024 version 0.4.3.
For Needle in Haystack (NIAH) (Kamradt 2023) and Variable Tracking (VT), we adopt the evaluation implementation provided by RULER (Hsieh et al.) (022)) and use the context length defined in the retrofitting procedure. For NIAH, utilize the essay version with a single needle, whereas for VT, we utilize the version with 40 variable chains and O hops, filled with repeating text.
For AIME24 6 GPQA Diamond (Rein et al. 2024 and MATH-500 (Lightman et al. 2024). evaluate models using the Search and Learn framework (version 0.1.0) (Snell et al. [2024] Beeching al. 2024), with math-parsing functionality derived from Verify (version 1.0.0) (Kydličck and
Gandenberger, 2025). For LiveCodeBench we utilize the tasks from 2024-08-01 till 2025-01-31.
For few-shot tasks from Language Model Evaluation Harness we directly utilize the framework to provide the few-shot examples. For RULER (Hsieh et al. 2024), we use the example generator to sample few-shot examples. Below we present remaining prompts that were used during the evaluation, except the prompts to base models, which were set to unaltered task input, and prompts for zero-shot evaluation of instruction tuned models, which were set to task inputs wrapped with HuggingFace tokenizer chat template
For GSM8K zero-shot evaluation, we adopt the prompt from Meta.
24

---

GSM8K zero-shot prompt
<|start_headee _ncdeer__d__a >system< end _
Cutting Knowledge Date: December 2023
Today Date: 23 July 2024
You are a helpful assistant. <leot_iid>><tte ioot_id///sstttt...mm hetarstttaateeeeeedddllueaeeee hnd_hederridd
Given the following problem, reason and give final answer problem.
Problem: p...poblem_tee. text__e...........
Your response should end with "The final answer is [answer] where [answer] is the response to the headerr id >assistant< leadeedddheae_idd problem. probllmm<<leot_. <leot_
For Qwen-RI AIME 24, MATH-500 and GPQA O we adopt the prompts from Open-RI repository
AIME 24 and MATH-500 prompts
< Sollee the following math problem efficiently and clearly.
The last line of your response should be of the following format: hope it is correct' (without quotes) where ANSWER is just the final number or expression that solves the problem. Think step by step before answering.
'Therefore, the final answer is: (ANOEEd(A
problem_text__. / Assistant <etbisttant><think> >><thin>>
GPQA Diamond prompt
User Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer:
$LETTER' (without quotes) where LETTER is one ABCD. Think step by step before answering.
) /AAssistant/>>thiii
problem_text..c.mmee
For coding tasks we utilize the following prompt adopted from LiveCodeBench((ain et al. 2025 DeepSeek-RI setting:
25

---

LivcCodeBench prompt
conversation between User and Assistant. The asks question, and the Assistant solves it. The assistant first thinks about the reasoning process the mind and then provides the user with the
answer. The reasoning proceas and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e.,
<think> reasoning process here </think> <answer> answer here
</answer>. . </User >You will be given question (problem
specification) and will generate a correct Python program that matches the specification and passes all tests.
Question: ....-rollemmtext. Assistant ><think>>
I Influence of KV Cache on Inference Latency
In this section, we provide a simplified analysis estimating the proportion of inference latency introduced by reading from the key-value cache to the entire latency of the step, during single auto-regressive inference step of an LLM on a GPU. Our calculations are based on the architecture of the Llama 3 model family. Specifically, derive sample equations for Llama 3.1 8B, parameters of which are listed below.
Parameter
Value
Description
n
32
Number of layers
d 14336
4096
Hidden dimension
dy
Internal dimension of the MLP layers
d 128256
1024
Dimension of the Key/Value sequences
V
Vocabulary size
The estimates can be expressed terms batch size B and sequence length I,, which determine the number of tokens in the KV cache. The number of floating pointt operations (FI.OP)) can be approximated as
FLOPS L) 2 nB (6ddg 4d + 4dd + 4dL) + 2BdV.
(2)
This calculation considers only major matrix-vector multiplications (assuming two FLOPs per operation), omitting minor operations such as normalization and pointwise non-linearities.
Similarly, we estimate the number of reads from the High Bandwidth Memory as:
Reads( [, L) Rs n (6ddy + 4d2 + Addg + 4BLds)) + 2dV,
assuming bytes per parameter (16-bit precision). Note that only the KV cache (An,Ldd,,) scales with batch size and sequence length. As a sanity check, we confirm that Reads(1, (),0)/2 Rs 7.5B approximate the model's parameter count (without 0.5B for the input embedding table, which does not have to be entirely read during an inference step). Finally, the approximations for Llama 3.1 8B have the following form:
BL FLOPS (L, 1.45 10 5.24 100 BL
(4)
Reads L) 1.50 10 10
+ 1.31 10a BL.
(5)
For the remaining calculations, we use the peak performance values for NVIDIA H100 SXM (https://www.nv)) wwwp:/.wviiia..mmmm http://ww.a..aooeaaaaaaaaaaaaaaa for 16-bit calculations without 2:4 sparsity:
BFLOAT16 Tensor Core performance GPU Memory bandwith
989.5 TFLOPS
3.35 TB/s
26

---

Since memory reads are significantly slower than computations, the latency contribution from KV cache reads (1.31 x 10y BL term) dominates at larger sequence lengths and batch sizes. Thus, KV cache size is a critical factor in inference latency for long sequences.
The inference latency per step can be approximated as
FLOPS L) Reads (B,L) L) 989.5 TFLOPS'
max
(6)
3.35 TB/s
assuming ideal overlap between computation and memory operations. Approximating KV cache reads as 4n Ldg, and following the same calculations for other Llama and Qwen models, we visualize the fraction of latency contributed by KV cache reads to the latency of entire inference steps (Figure
2:11
XX
0.K
41
1164
0.75
A/3
O/E
0011
E-H1
ILT
8.45
OZ
CRT
ES1
IEES
OZS
41
1812
8.5
6A6
0.57
902
CA5
8.01
11
Llana 5.2
064
078
0.66
0.19
6.44
6221
8.07
EA1
8.0
05S
32k
C3
C45
C.4
0.75
0.7
0250
0.56
0.8
0C2
91
9.18
6.5
1.4
8:54
C.78
0.87
8.53
E32
8.45
6.1
628
0.5
046
0T3
0.5T
84k
6.65
6778
097
6.41
1.95
6.9
0.18
19.66
8:64
6.91
6.41
1.49
E45
031
6118
0885
0644
8.78
Qwenn 2.5 1,53
E.32
9.T
0:5
025
0.55
0UR
053
904
900
8.13
0.55
1.71
8X
8:1
012 0.4
0.22
A25
0.2
16k
6.07
8:13
623
638
035
671
0455
0.11
0.05
062
0.6
9.0
0.13
8.23
8.58
4:55
6.21
8.55
501
8.42
034
637 0.5
023 AJ8
0.55
0.71
32k
0.12
2:23
031
0.0
0.88
0.13
022
6711
8.51
8.54
047
028
64k
Context leigth
8.34
0.55
041
045
0.94
0.56
1855
CA5
1.43
8.15
031
4.55
8.71
C.33
6.11
045
GA3
0.99
0.64
1.4
0.23
0.08
0.55
(97)
6.33
8:11
0.45
6.1
6.99
K15
8:3
6.88
6.55 0.11
053
AM 0.55
0.96
1/0
8.25
E47
027
0.36
1..0
0/
0/0
9.0
916
8:01
011
6.20
8.95
0.51
02
681 632 965 023 9.12
1922
0.
Liams 3.1 SD
68
E.M
8.T
012
022
035
0.5
V.7
002
8.5
023
0/0
904
000
1112
922
0.00
0.55
0.7
8.5
831
032 0.4
007 0.22 A22
002
0.5
166
012
9.2
022
4.22
9.26
032
0U9
057
0.12
022
924
L.53
6.22
1.55
8.T7
032
C22
022
125k
4.55
6.53
6.82 6/9
6.45
4.07
0.64
0.96
0.12
0.22
0.66
055
8.
8-22
6.9
6A5
6.47
647
8.2
022
6.5 055
69 0.95
6.82
0.5
0.7
0.22
0.56
0.53
6.12
030
6.99
8.22
036
092
82
0.1
0.2 2.8%
035
0.12
1921
1035
11.22
93
1901
9111
C.06
0.22
0.21
86
02
6.0
900 OEt 0Z2 km
1.12
4k
2.41
8.27
6.55
0.17
021
1.55
1912
11/0
0/9
0.1
110
0.21
1.11
86
02
600 F31 A00 029
1912
Queen 2.5 7B
6.42
8:53
021
0.29
0.5
402
900
021
0.5
8.52
08
031
OZ6
A12
8.95
9:32
2:21
8.35
0.94
052
0/0
9.6
1212 8:21
8.55
2.64
E91 9X2 083 E.5 0.12 021 JU55 1912
648
412
8:1
C35
E.52
045
088
0.9
0454
OUT
053
0U6
9.12
021 8.2
8.52
0.11
02
532
8.53 036
0.2
021
025 AS2 (468 0.51
128k
4.2
9.25
031
021
1.52
0.06
032
022
RMM
16
12
216
16
178
255
64
128
Batch size
Figure 9: Percentage of total latency attributed to KV cache reads. Those reads clearly dominate latency as batch size and sequence length increase. When the KV cache is compressed (CR 4x and 8 8x), more tokens can be accommodated before the latency of reading the KV cache becomes an issue.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Figures 3
7 and Table
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Question:
Justification: Limitations in Section Section/AJalona Alalong with details about evaluation in Appendix H and
G Performance considerations in Section 3.1 and Section 3.2
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofss
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper provides only experimental results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
28

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix C we provide the details about the retrofitting procedure. Ap- pendix E describes the dataset used for training. While it does not disclose details, believe that public datasets should be enough for verification of claims, even more so that the models were trained through logit distillation. Finally, Appendix HH provides the details about evaluation. Section 3 describes the introduced method. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While we do not provide code with sufficient instructions for reproduction, we attach to the submission code showing how implement DMS using the public codebase for Dynamic Memory Compression. The dataset used for retrofitting is proprietary and we provide an overview of its contents in Appendix [ The data used for retrofitting Qwen-R1 models is publicly available. Language Model Evaluation Harness, RULER, Search and
Learn and Math Verify are publicly available.
29

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https Codeeuumisssonnn
/ //nips htttpssccc/uulcc//eeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc. The authors should provide scripts to reproduce all experimental results for the new reproducible, they proposed method and baselines. If only subset of experiments
should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendices C
D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Inference-sealing reasoning tasks are evaluated across different budgets and seeds. In Appendix G we provide results with error bars, which have been omitted in the main paper for the sake of clarity. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative emror rates).
30

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: While we do not provide the exact numbers, reference numbers in Appendix C
allow to form estimates of the required resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: No human subjects involved. As mentioned, the work does not introduce new risks but potentially exacerbates the existing ones regarding use of LLMs.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussion provided in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal
impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
31

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not provide any new data nor models as it describes generic optimization applicable to Transformer models. Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide citations and links used assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the www.rsswtthhheeeeee package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or nta-standard components. Please refer to our LLM /ttp:////////errps..o///oooooeee http:/pp/p.p//oooeeeee////////// for what should or should not be described.
33


Reviews:
[{"review_id": "e0bCCQ1GsI", "reviewer": "Reviewer_1K9q", "summary": "This paper introduce Dynamic Memory Sparsification (DMS), a method for sparsifying KV cache by evicting tokens towards a target compression ratio. The core idea is that even if a token is decided to be evicted at some time t, the actual eviction is delayed for a later generation step t + w. Eviction decision probability is sampled from a Gumbel-sigmoid distribution as a function of constants and the first component of the query vector at time t. During inference this probability is binarized. During training it enters a summation over layers, attention heads and the sequence head so that it that the one-sided L1 loss from the target compression ration over them is minimized; also the model is retrofitted through logit distillation. \n\nExperiments over AIME24, MATH500 and GPQA Diamond problems are then conducted for vanilla Deepseek-R1 distilled models of various sizes, their DMS-retrofitted models and for different training-free KV-cache sparsification methods (TOVA, H2O, Quest) under varying token budget configurations (in terms of maximum sequence length (L), number of parallel reasoning chains (W) and compression ratios (CR)). The accuracy as a function of runtime efficiency (KV cache token reads) and out-of-memory risk (peak tokens in memory) for different L-W-CR configurations can identify Pareto frontiers for each method. DMS achieves superior frontiers over benchmarked approaches with absolute gains of up to 10 points over vanilla methods, i.e. much improved accuracy of scaled inference as more tokens for the same compute budget can be generated thanks to the presented KV-cache compression scheme (inference-time hyper-scaling). Ablation studies over a small instruct model further elaborate the role of the delay window w (down to the immediate eviction scenario). Additionally DMS is applied on more tasks with particularly good performance on long-context ones.", "questions": "1. Can different retrofitted models (different context lengths and different compression rations) share checkpoints during their training process?\n\n2. Can you identify scenaria where one of the training-free KV-cache compression methods would be more attractive? Are such considerations included in limitation sections?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ The idea is simple (delaying token eviction), yet particularly effective.\n\n+ Integrating the core idea of delayed token eviction can be conveniently implemented with an additive attention mask.\n\n\n- Models need to be retrofitted, which involves training.\n\n- Training requirements seem to be a function of target context lengths and compression ratio (CR), so multiple model checkpoints need to be fine-tuned.", "was_revised": true, "final_justification": "Overall, I found the idea of \"delaying token eviction\" simple to present and integrate/implement and empirically effective; also the authors provided useful clarifications during rebuttal. Therefore, I maintain my initial positive score (4: borderline accept).", "rebuttal": "We appreciate Reviewer 1K9q’s taking the time to comment on our paper and would like to thank them for their questions and remarks.\n\n**Q1: Can different retrofitted setups (context lengths and CRs) share checkpoints during retrofitting?**\n\nYes indeed! A single DMS retrofitting run produces a sequence of checkpoints with increasing compression ratios. The target CR increases linearly from 1x to the preset maximum, and intermediate checkpoints (e.g., 2x, 3x) are saved along the way. The DMS models in Figures 2 and 3 were generated from single retrofitting runs per model size.\n\nThe answer for context length has an additional nuance. For Figures 2 and 3, we trained R1-Qwen models with a fixed 8k context and evaluated them by generating sequences beyond this length, such as 16k and 32k. Hence, the DMS retrofitting procedure did not affect length extrapolation (possibly due to taking only a few steps and teaching to evict recent tokens).\n\n**Q2: In which scenarios training-free KV cache compression methods would be more attractive?**\n\nApart from the main advantage, which is the lack of need for extra training, training-free methods might offer advantages in particular edge cases. Sliding-window methods like TOVA maintain full KV cache fidelity until the window is exceeded, preserving vanilla accuracy on short inputs. This stands in contrast with DMS, which compresses all sequences to some extent, albeit it compresses shorter sequences to a ratio inferior to the target (and longer ones to a ratio superior to the target). For the same reason, methods like TOVA allow for pre-specifying the memory cap, whereas the KV cache in DMS grows as a function of the sequence length. \n\nNonetheless, for most intents and purposes, retrofitting is superior to training-free alternatives, as these do not offer a statistical guarantee of preserving performance across tasks and often lead to significant degradations [1]. We will include all these considerations in the Limitations section.\n\n[1] P. Nawrot, R. Li, R. Huang, S. Ruder, K. Marchisio, E.M. Ponti, The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs, 2025."}, {"review_id": "gqWYGexidg", "reviewer": "Reviewer_BS8X", "summary": "This paper investigate how KV cache compression facilitates inference-time scaling by allowing more reasoning samples or longer reasoning length. Moreover, it introduces Dynamic Memory Sparsification (DMS), a KV cache eviction method through post-training retrofitting, utilizing logits distillation during training and delayed eviction during inference. Experiments show that DMS improves performance-efficiency Parato frontiers for KV compression methods on reasoning tasks, and maintains near-lossless performance on other tasks.", "questions": "The writing of this paper requires major improvements. The “Logit Distillation and Retrofitting” and zeroing out of $q\\[0\\]$ are only introduced in the experimental setup section, making the Sec. 3 hard to understand (why $q\\[0\\]$ can predict the eviction decision). \n\nThe notation in Line 149 is also ambiguous: L, H, T denotes both the numbers and the sets. And is the total loss simply summation of distillation loss and auxiliary loss, with no coefficients needed? \n\nIn Line 156, it is not clear that why the additive attention is never materialized.\n\nIn Line 192, it seems that only one $q\\[0\\]$ is retrofitted for a GQA group, so are the eviction decisions consistent for all heads within a group? This requires more specific demonstration and notation. \n\nIn Figure 4 right, for win=16 (the preferred configuration), reducing the number of training tokens incurs significant drops in GSM8K scores, why the authors conclude that the model is robust (Line 259).", "limitations": "For training or retrofitting, the essential issue of DMS is independent retrofitting for different compression rates. Moreover, the amount of retrofitting data increases for higher compression rates. These can result in high retrofitting costs, hindering its practical deployment. \n\nFor inference, DMS uses KV cache token reads as the metric for speed-efficiency. While it is reasonable to some extent, it would be more convincing to show practical latency speedups, which can cover the overhead of DMS eviction. The authors mentioned their efficiency considerations such as overwriting and PageAttention (Line 166, 168), while simply using KV cache token reads cannot demonstrate such considerations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Important problem: inference-time scaling is an emerging trend, while the memory and efficiency issues caused by large KV cache of generated tokens are rarely studied.\n- The sparsification by retrofitting through logits distillation is novel and effective.\n- Extensive experiments demonstrate that DMS improves the Pareto frontiers of KV cache compression on reasoning tasks.\n\nWeaknesses:\n- The writing of this paper is confusing, with some critical details unclear. See more in the Questions part.\n- The costs of retrofitting are not mentioned. Moreover, DMS requires independent retrofitting for different compression rates, and the amount of retrofitting data increases for higher compression rates, which might hinder its practical deployment.\n- Practical evaluation on efficiency is not provided.", "was_revised": true, "final_justification": "The authors have addressed my concerns and I will keep my original score.", "rebuttal": "We appreciate Reviewer BS8X’s careful reading and helpful comments, which we address below. \n\n**W1: Improvements in writing**\n\nWe will revise the manuscript to correct the title casing, resolve issues with the notation, and reorder the introduction of $q[0]$ (see below). We will also double-check the format instructions to rule out any other inconsistencies.\n\n**W2: Does DMS require independent retrofitting for different compression ratios?**\n\nFortunately, this is not the case! A single run with a gradually increasing CR yields multiple checkpoints for various compression levels. All our experiments are based on this setup. This enables a more efficient use of compute and flexible continuation for higher compression ratios. We will clarify this better if the paper is accepted.\n\nBelow we specify resource estimates for retrofitting the models used in the manuscript at 100 steps/CR to CR 8x. Please see the response to Q5 for details.\n\n| Model         | GPUh  |\n|---------------|------:|\n| Llama 3.2 1B  |    75 |\n| Llama 3.1 8B  |   700 |\n| R1-Qwen 1.5B  |   220 |\n| R1-Qwen 7B    |   530 |\n| R1-Qwen 32B   |  2400 |\n\n**W3: Practical evaluation on efficiency**\n\n**We provide latency measurements during generation using the Megatron-LM framework**. We run our measurements on H100 GPUs in bfloat16 precision using FlashAttention2 and the flash_attn_with_kvcache routine for both vanilla and DMS. We note that we use research code for DMS that has not been optimised (e.g., through fusion of multiple operations), resulting in a visible latency overhead. Also, for small to moderate token batch sizes, KV cache size does not drive the majority of latency (see Appendix H and Figure 6 for details). However, a practical inference scaling scenario of serving multiple users at a time and multiple rollouts per query (parallel scaling) requires substantial batch sizes. In addition, Qwen 2.5 models have smaller KV cache sizes than the Llama 3 family or the newer Qwen 3 family, when measured in relation to the number of model parameters. Finally, we remark that our Megatron-LM framework setup is designed for fast inference, albeit we could not optimise it to the point of matching state-of-the-art inference engines due to time constraints of the response period. \n\nFor latency benchmarks, we measure the generation of reasoning chains for AIME24. The notation “Lat @ X” denotes the latency when generating a token at position X. From the results below, it emerges that even for moderate batch sizes (such as 64 and 128) and sequence lengths (>-8k), CR4 and CR8 decrease latency considerably. The reason why there is not a direct correspondence between the reduction in memory reads (due to higher CR) and reductions in latency is that their dependency emerges only at large token batch and sequence sizes. We refer to Appendix H for an explanation with heatmaps. Overall, **these findings suggest that DMS results in effective latency reductions due to decreased KV cache traffic in regimes such as inference-time scaling (parallel or sequential)**. In turn, this enhances reasoning quality by allowing for a larger effective token budget for the same wall-clock runtime (while also reducing memory load).\n\n| Model | Batch size | Lat @ 4k | Lat @ 8k | Lat @ 12k | Lat @ 16k |\n|-|-|-:|-:|-:|-:|\n| R1-Qwen 1.5B Vanilla | 16 | 19.1 ms | 19.3 m | 18.9 ms | 18.7 ms |\n| R1-Qwen 1.5B DMS 4x | 16 | 27.5 ms | 26.1 ms | 26.2 ms | 26.3 ms |\n| R1-Qwen 1.5B DMS 8x | 16 | 26.6 ms | 26.3 ms | 26.4 ms | 25.5 ms |\n| R1-Qwen 1.5B Vanilla | 64 | 21.9 ms | 39.0 ms | 40.7 ms | 41.1 ms | \n| R1-Qwen 1.5B DMS 4x | 64 | 26.2 ms | 26.3 ms | 25.9 ms | 26.4 ms |\n| R1-Qwen 1.5B DMS 8x | 64 | 25.5 ms | 25.2 ms | 24.9 ms | 24.9 ms |\n| R1-Qwen 1.5B Vanilla | 128 | 23.1 ms | 41.0 ms | 58.4 ms | 76.0 ms |\n| R1-Qwen 1.5B DMS 4x | 128 | 26.0 ms | 25.8 ms | 26.2 ms | 26.1 ms | \n| R1-Qwen 1.5B DMS 8x | 128 | 26.7 ms | 26.1 ms | 26.1 ms | 25.9 ms | \n\n\n| Model | Batch size | Lat @ 4k | Lat @ 8k | Lat @ 12k | Lat @ 16k | \n|-|-:|-:|-:|-:|-:|\n| R1-Qwen 7B Vanilla | 16 | 24.9 ms | 25.7 ms | 26.4 ms | 27.1 ms |\n| R1-Qwen 7B DMS 4× | 16 | 25.9 ms | 25.8 ms | 25.8 ms | 26.3 ms |\n| R1-Qwen 7B DMS 8× | 16 | 26.7 ms | 27.2 ms | 27.4 ms | 26.5 ms |\n| R1-Qwen 7B Vanilla | 32 | 25.8 ms | 42.8 ms | 44.4 ms | 45.0 ms |\n| R1-Qwen 7B DMS 4× | 32 | 25.7 ms | 25.9 ms | 25.9 ms | 25.6 ms |\n| R1-Qwen 7B DMS 8× | 32 | 26.0 ms | 26.9 ms | 26.9 ms | 27.1 ms |\n| R1-Qwen 7B Vanilla | 64 | 26.8 ms | 44.5 ms | 62.2 ms | 79.7 ms |\n| R1-Qwen 7B DMS 4× | 64 | 26.1 ms | 26.8 ms | 26.8 ms | 26.4 ms |\n| R1-Qwen 7B DMS 8× | 64 | 26.2 ms | 27.0 ms | 26.8 ms | 26.0 ms |\n\n\nWe will include latency plots for all model sizes in the final version of our paper.\n\n**Q1: Extraction of the decision variable from $q[0]$**\n\nWe agree with the Reviewer that the role of $q[0]$ in predicting the eviction decision should be better clarified in Section 3, we will adopt this suggestion in the next version of our paper. The reason why it was only introduced in the experimental setup is that this is not part of the “core” design of our method. In principle, alternative approaches exist, such as introducing a trainable vector $w$ and computing the decision variable as $x^Tw$. Reusing q[0] or k[0] is primarily a form of performance optimisation.\n\n**Q2: Ambiguous L, H, T symbols on L149**\n\nThank you for bringing this to our attention. We will revise the formula using $|L|$, $|H|$, and $|T|$ to denote the number of layers, attention heads, and sequence length, respectively.\n\n**Q3: Avoiding materialisation of the attention mask**\n\nAs shown in Figure 1(b), the attention mask can be reconstructed from a vector $\\alpha = [log(1 - \\alpha_1), \\log(1 - \\alpha_2), …, \\log(1 - \\alpha_n)]^T$ , as these values are replicated as columns of the mask, with the diagonal and sub-diagonals zeroed to account for the sliding window. This regular structure can be leveraged for a more efficient implementation of self-attention. A detailed discussion of a similarly compressed representation for a family of structured masks—which our mask belongs to—is given in [1], along with source code based on FlashAttention 2. We will convey this better in the text and reference this work in the final version of our manuscript.\n\n[1] G. Wang, J. Zeng, X. Xiao, S. Wu, J. Yang, L. Zheng, Z. Chen, J. Bian, D. Yu, H. Wang, FlashMask: Efficient and Rich Mask Extension of FlashAttention, 2025.\n\n**Q4: Only one q[0] is retrofitted for a GQA group, so are the eviction decisions consistent for all heads within a group?**\n\nOur models use GQA, where keys are shared among queries. Extracting the decision variable from a single query for each KV group impacts only one query-key pair, whereas extracting the variable from k[0] would impact all query-key pairs in which this key is used. We emphasise that all queries within the GQA group share the same KV cache after eviction.\n\n**Q5: Method not being robust in Figure 4 right for win=16**\n\nThe win=16 result highlights that DMS can evict even very recent tokens effectively—contrary to prior assumptions—when delayed eviction is used. That said, a larger window (e.g., win=256) yields better data efficiency and makes the model operate more like vanilla, which results in overall better accuracy (as seen in Figure 4, right). Importantly, since CR is enforced over the entire context (e.g., 4k or 8k), increasing the window size should not increase memory use. We ran a series of post-submission experiments confirming that enlarging the window to 256 tokens does improve the results in reasoning models, and allows for reducing the number of retrofitting steps by an additional factor of 30x from 3000/CR to 100/CR. We will clarify this new practical insight and update the results in Figures 2 and 3. Below we show an excerpt from the results for R1-Qwen 7B in these two setups: (win=16, 3000 steps/CR) and (win=256, 100 steps/CR). AIME24 scores and KV budgets have been averaged over 3 runs with different random seeds due to the small dataset size.\n\n| Model | L-W-CR | AIME24 Score | AIME 24 KV Reads | MATH500 Score | Math 500 KV Reads | GPQA ◇ Score | GPQA ◇ KV Reads | \n|-:|-:|-:|-:|-:|-:|-:|-:|                                                      \n| Vanilla | 8-4-1 | 50.00 | 9.27E+07 | 93.20 | 2.21E+07 | 50.33 | 8.98E+07 |   \n| DMS (win=256) | 8-4-4 | 60.00 | 3.02E+07 | 93.40 | 8.54E+06 | 54.00 | 3.16E+07 |\n| DMS (win‑16) | 8-4-4 | 46.67 | 2.63E+07 | 93.00 | 6.69E+06 | 53.87 | 2.65E+07 |\n| Vanilla | 32-1-1 | 53.33 | 8.84E+07 | 94.00 | 1.24E+07 | 51.50 | 4.54E+07 |  \n| DMS (win=256) | 32-1-4 | 53.30 | 2.52E+07 | 92.80 | 3.66E+06 | 48.50 | 1.39E+07 |\n| DMS (win=16) | 32-1-4 | 44.43 | 2.07E+07 | 93.00 | 3.44E+06 | 49.50 | 1.22E+07 |\n| DMS (win=256) | 16-8-8 | 73.30 | 8.03E+07 | 96.60 | 1.62E+07 | 56.10 | 6.33E+07 |\n| DMS (win=16) | 16-8-8 | 62.23 | 6.87E+07 | 95.80 | 1.16E+07 | 52.17 | 5.10E+07 |\n\nThe table shows that win=256, even with 30x shorter retrofitting, yields much better models.\n\nAs for the results in Figure 4 right for win=16, Llama 3.2 1B Instruct model proved brittle during retrofitting, with GSM8K scores degrading quickly (e.g., DMC at CR 3x and 4x in Table 1). We use \"robust\" to mean the model still performs reasonably well and does not collapse the score under difficult settings (e.g., win=16 and low token budgets), but also maintains strong results with larger windows like win=256.\n\n**L1: Multiple runs per CR and data efficiency**\n\nPlease see our response to W2 and Q5, which hopefully assuages the Reviewer’s concerns, since only a single run is needed for all CRs. In addition, we pushed data efficiency to 60x the previous SOTA trainable KV cache compression (DMC)\n\n**L2: Showing latency measurements gains instead of proxy metrics**\n\nWe refer to our latency measurements reported in our response for W3. In a nutshell, DMS translates into effective latency reduction in regimes with large batch sizes and sequence lengths, which are common in inference-time scaling.."}, {"review_id": "yanYoCYfd7", "reviewer": "Reviewer_2bQ1", "summary": "This paper addresses the practical bottleneck of inference-time scaling in Transformer-based LLMs caused by the rapid growth of the key-value (KV) cache, particularly during reasoning-heavy tasks requiring long or multiple generation threads. The authors propose Dynamic Memory Sparsification (DMS), a method for compressing the KV cache that delays token eviction rather than immediately dropping tokens, thereby allowing models to preserve more critical information with less destructive context loss. DMS is positioned as a lightweight and sample-efficient retrofitting approach compared to prior methods, notably Dynamic Memory Compression (DMC) and existing training-free sparsification strategies. Through experiments on various LLMs (including DeepSeek R1-distilled and Llama 3.2 Instruct models) and multiple reasoning benchmarks (AIME 24, MATH 500, GPQA Diamond), the method demonstrates measurable improvements on the Pareto frontier of accuracy vs. efficiency (runtime and memory), outperforming existing KV cache compression techniques at similar or higher compression ratios.", "questions": "Hardware Efficiency Details: Can the authors provide more quantitative results for end-to-end inference metrics (e.g., wall-clock latency, throughput on mainstream GPU/TPU hardware) alongside the proxy measures (KV reads/peaks)? This will help practitioners assess real-world impact.\n\nGeneralization Beyond Reasoning Tasks: Have the authors considered, or do preliminary experiments support, the efficacy of DMS on common NLP tasks such as summarization, translation, dialogue, retrieval, or code completion? Would the benefits persist under very different prompting or sequence structures?\n\nPotential Failure Cases: Are there identified or observed types of prompts or inputs where DMS harms or degrades model reasoning in an unexpected manner? E.g., extremely long-range dependencies, highly nonlocal attention patterns, or adversarial token input.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe paper addresses a pressing challenge for deploying large language models (LLMs) at scale—specifically, the memory and runtime bottlenecks associated with attention computation via the key-value (KV) cache. This problem is increasingly relevant as LLM adoption grows, and existing methods struggle to balance efficiency and quality. The proposed method, DMS, is well-articulated and clearly distinguished from prior work. Its central idea—delayed eviction through a sliding window—represents a simple yet well-substantiated improvement over both immediate-dropping methods (e.g., TOVA, H2O) and merging strategies like DMC.\n\nThe empirical evaluation is thorough, covering a diverse set of LLM architectures including Qwen-R1-distilled (1.5B, 7B, 32B) and Llama 3.2 1B Instruct. Benchmarks span several challenging, reasoning-intensive datasets such as MATH 500, AIME 24, and GPQA Diamond. Results are well-presented in figures and tables: Figures 2 and 3 clearly illustrate trade-off curves (Pareto frontiers) between KV cache reads, peak tokens, and performance. DMS consistently outperforms alternatives in the accuracy-memory/runtime trade-off, particularly for larger models. Table 1 shows that DMS maintains high accuracy across datasets like GSM8K, MMLU, HellaSwag, NIAH, and VT even at 4× compression, where baseline methods degrade sharply.\n\nSection 5.2 and Figure 4 offer a solid ablation and design analysis, exploring the impact of sliding window size, delayed versus immediate eviction, and the training token budget. These experiments yield practical guidance for tuning hyperparameters and understanding the cost of retrofitting. The methodology is presented clearly, with readable equations and sufficient implementation details to support reproducibility for expert audiences. The claims are backed by extensive empirical results. Section 6 effectively contextualizes DMS among contemporary techniques such as sparse attention, quantization, and low-rank approximations, and emphasizes its orthogonality or complementarity to those approaches.\n\nWeaknesses\n\nDespite its strengths, the paper’s evaluation lacks breadth in terms of task coverage. It focuses almost exclusively on complex reasoning tasks and does not explore standard NLP applications such as summarization, dialogue, or code generation, making the generality of DMS uncertain. Furthermore, although the work is motivated by memory and runtime limitations, it provides little hardware-level evidence. There are no direct reports on wall-clock speedups, VRAM savings, or inference latency and throughput on real devices, which would better contextualize the practical benefits of DMS.\n\nThe paper also omits analysis of failure modes. There is no exploration of how the method behaves on pathological prompts, out-of-distribution inputs, or inputs with irregular attention patterns—factors that could significantly affect robustness. Lastly, the reliance on logit distillation with sampled data for retrofitting raises concerns about generalization. It remains unclear how well the approach transfers to domains with significant distribution shifts or task variations.", "was_revised": true, "final_justification": "The authors have addressed the main concerns effectively. First, on task coverage, they added experiments on LiveCodeBench (code generation) and QASPER (long-context QA), showing that DMS maintains performance beyond long reasoning tasks. Second, on hardware-level speed-ups, they provided H100 latency benchmarks and clarified that gains appear primarily at large batch and sequence sizes; while the results are based on unoptimized research code, the trends are credible. For pathological inputs, they demonstrated strong results on NIAH and VT tasks involving long dependencies and non-local attention, though more systematic adversarial testing remains for future work. Regarding generalization, their explanation—supported by literature—on how small learning rates and short runs of logit distillation mitigate overfitting risks is convincing. Remaining limitations include broader NLP coverage, absence of adversarial stress tests, and latency validation on optimized inference engines. Overall, the additional experiments and clarifications substantiate the core contribution, and the remaining issues are natural extensions rather than fundamental flaws; I therefore maintain a positive recommendation.", "rebuttal": "We thank Reviewer 2bQ1 for taking the time to read our paper and for providing thoughtful and constructive feedback.\n\n**W1 (a) and Q2: Broader representation of NLP tasks in the evaluation**\n\nWe appreciate your suggestion to include a broader set of tasks! We note that DMS is primarily designed for compressing the KV cache during the auto-regressive decoding phase, rather than the parallel prefilling phase. Hence, the focus of our evaluation is primarily on long-generation/reasoning tasks rather than long-context tasks. Among the tasks you suggested, a prime example of the former is code generation: hence, we ran additional experiments for R1-Qwen 2.5 models on LiveCodeBench, trained with win=256 and 100 steps per CR. For parallel scaling, we report pass@all, namely whether any of the generated threads passed the tests for a given problem.\n\n| Model             | L-W-CR  | 1.5B Score | 1.5B KV Reads | 7B Score | 7B KV Reads | 32B Score | 32B KV Reads |\n|------------------|--------:|-----------:|--------------:|---------:|------------:|----------:|-------------:|\n| Vanilla          | 16-1-1  | 17.30      | 8.50E+07      | 35.90    | 7.21E+07    | 57.00     | 6.04E+07     |\n| DMS (win=256)    | 16-1-4  | 16.42      | 2.01E+07      | 34.62    | 2.04E+07    | 53.60     | 1.82E+07     |\n| DMS (win=256)    | 16-1-8  | 15.04      | 1.18E+07      | 32.61    | 1.20E+07    | 54.46     | 1.20E+07     |\n| Quest            | 16-1-4  | 16.96      | 5.95E+07      | 33.74    | 5.05E+07    | 54.42     | 4.54E+07     |\n| Quest            | 16-1-8  | 16.68      | 3.98E+07      | 32.02    | 3.50E+07    | 54.20     | 3.25E+07     |\n| TOVA             | 16-1-4  | 16.04      | 4.14E+07      | 32.22    | 3.71E+07    | 49.54     | 3.35E+07     |\n| DMS (win=256)    | 16-4-4  | 24.80      | 8.07E+07      | 44.90    | 8.17E+07    | 63.80     | 7.29E+07     |\n| Quest            | 16-4-4  | 25.40      | 2.37E+08      | 48.00    | 2.02E+08    | 64.10     | 1.81E+08     |\n| TOVA             | 16-4-4  | 25.10      | 1.66E+08      | 43.00    | 1.48E+08    | 58.50     | 1.34E+08     |\n| DMS (win=256)    | 16-8-8  | 26.60      | 9.44E+07      | 47.70    | 9.58E+07    | 70.00     | 7.49E+07     |\n\nSimilar to other long-generation tasks, we find that hyper-scaling with DMS yields vastly improved performance for the same compute budget, measured as cumulative memory reads.\n\nIn addition, we attach the results of Llama 3.1 8B on the QASPER benchmark (win=256, 200 steps per CR) for QA on scientific research papers. We find that DMS mostly maintains performance on this long-context NLP task.\n\n| Model    | Llama 3.1 8B |\n|----------|--------------|\n| Vanilla  |         33.1 |\n| DMS 8x   |         32.9 |\n\nWhile our focus is on long-sequence reasoning, these results indicate DMS maintains performance on other NLP tasks. We agree that broader evaluations would be valuable and are a natural extension of this work.\n\n**W1 (b) and Q1: Little hardware-level evidence for speed-ups, no reports of measurements**\n\n**We provide latency measurements during generation using the Megatron-LM framework**. We run our measurements on H100 GPUs in bfloat16 precision using FlashAttention2 and the flash_attn_with_kvcache routine for both vanilla and DMS. We note that we use research code for DMS that has not been optimised (e.g., through fusion of multiple operations), resulting in a visible latency overhead. In addition, we would like to note that for small to moderate token batch sizes, KV cache size does not drive the majority of latency (please refer to Appendix H and Figure 6 for details). However, a practical inference scaling scenario of serving multiple users at a time, requires substantial token batch sizes. In addition, Qwen 2.5 models have smaller KV cache sizes than the Llama 3 family or the newer Qwen 3 family, when measured in relation to the number of model parameters. Finally, we remark that our Megatron-LM framework setup is designed for fast inference, albeit we could not optimise it to the point of matching state-of-the-art inference engines due to time constraints of the response period. \n\nFor latency benchmarks, we measure the generation of reasoning chains for AIME24. The notation “Lat @ X” denotes the latency when generating a token at position X. From the results below, it emerges that even for moderate batch sizes (such as 64 and 128) and sequence lengths (>-8k), CR4 and CR8 decrease latency considerably. The reason why there is not a direct correspondence between the reduction in memory reads (due to higher CR) and reductions in latency is that their dependency emerges only at large token batch and sequence sizes. We refer to Appendix H for an explanation with heatmaps. Overall, **these findings suggest that DMS results in effective latency reductions due to decreased KV cache traffic in regimes such as inference-time scaling (parallel or sequential)**. In turn, this enhances reasoning quality by allowing for a larger effective token budget for the same wall-clock runtime (while also reducing memory load).\n\n| Model | Batch size | Lat @ 4k | Lat @ 8k | Lat @ 12k | Lat @ 16k |\n|-|-|-:|-:|-:|-:|\n| R1-Qwen 1.5B Vanilla | 16 | 19.1 ms | 19.3 m | 18.9 ms | 18.7 ms |\n| R1-Qwen 1.5B DMS 4x | 16 | 27.5 ms | 26.1 ms | 26.2 ms | 26.3 ms |\n| R1-Qwen 1.5B DMS 8x | 16 | 26.6 ms | 26.3 ms | 26.4 ms | 25.5 ms |\n| R1-Qwen 1.5B Vanilla | 64 | 21.9 ms | 39.0 ms | 40.7 ms | 41.1 ms | \n| R1-Qwen 1.5B DMS 4x | 64 | 26.2 ms | 26.3 ms | 25.9 ms | 26.4 ms |\n| R1-Qwen 1.5B DMS 8x | 64 | 25.5 ms | 25.2 ms | 24.9 ms | 24.9 ms |\n| R1-Qwen 1.5B Vanilla | 128 | 23.1 ms | 41.0 ms | 58.4 ms | 76.0 ms |\n| R1-Qwen 1.5B DMS 4x | 128 | 26.0 ms | 25.8 ms | 26.2 ms | 26.1 ms | \n| R1-Qwen 1.5B DMS 8x | 128 | 26.7 ms | 26.1 ms | 26.1 ms | 25.9 ms | \n\n| Model | Batch size | Lat @ 4k | Lat @ 8k | Lat @ 12k | Lat @ 16k | \n|-|-:|-:|-:|-:|-:|\n| R1-Qwen 7B Vanilla | 16 | 24.9 ms | 25.7 ms | 26.4 ms | 27.1 ms |\n| R1-Qwen 7B DMS 4× | 16 | 25.9 ms | 25.8 ms | 25.8 ms | 26.3 ms |\n| R1-Qwen 7B DMS 8× | 16 | 26.7 ms | 27.2 ms | 27.4 ms | 26.5 ms |\n| R1-Qwen 7B Vanilla | 32 | 25.8 ms | 42.8 ms | 44.4 ms | 45.0 ms |\n| R1-Qwen 7B DMS 4× | 32 | 25.7 ms | 25.9 ms | 25.9 ms | 25.6 ms |\n| R1-Qwen 7B DMS 8× | 32 | 26.0 ms | 26.9 ms | 26.9 ms | 27.1 ms |\n| R1-Qwen 7B Vanilla | 64 | 26.8 ms | 44.5 ms | 62.2 ms | 79.7 ms |\n| R1-Qwen 7B DMS 4× | 64 | 26.1 ms | 26.8 ms | 26.8 ms | 26.4 ms |\n| R1-Qwen 7B DMS 8× | 64 | 26.2 ms | 27.0 ms | 26.8 ms | 26.0 ms |\n\nWe will include latency plots for all model sizes in the final version of our paper.\n\n**W2 (a) and Q3: Exploration of how the method behaves on pathological prompts**\n\nThe Needle-in-a-Haystack and Variable Tracking (VT) tasks (Table 1 in the manuscript; Tables 5 and 7 in Appendix D) consist of synthetically constructed inputs, and DMS models perform well on both (often exceeding the original model’s performance). In particular, the VT task measures the ability to track long-range dependencies, and NIAH attention patterns are highly non-local. As our models are retrofitted without access to private pre-training and post-training data, we cannot confirm whether the inputs for these two tasks are OOD, although there is a high chance that this is the case. Searching for adversarial inputs is interesting, but beyond our core focus on compressing KV caches for long reasoning chains, where we assume that the prompt is well-formatted.\n\n**W2 (b) Logit distillation with sampled data raises concerns about generalization. How well the approach transfers to domains with significant distribution shifts or task variations?**\n\nLogit distillation (model matching) mitigates domain mismatch more effectively than language modelling loss (data matching) when retrofitting LLMs [1, 2]. In addition, DMS performs well under short retrofitting runs (e.g., 100 steps per CR in the LiveCodeBench experiments above), which involve considerably less data than used for pre-training. Combined with small learning rates, this suggests minimal risk of overfitting to the distillation data and altering the model behaviour.\n\n[1] S.T. Sreenivas, S. Muralidharan, R. Joshi, M. Chochowski, A.S. Mahabaleshwarkar, G. Shen, J. Zeng, Z. Chen, Y. Suhara, S. Diao, C. Yu, W.-C. Chen, H. Ross, O. Olabiyi, A. Aithal, O. Kuchaiev, D. Korzekwa, P. Molchanov, M. Patwary, M. Shoeybi, J. Kautz, B. Catanzaro. LLM Pruning and Distillation in Practice: The Minitron Approach, 2024.\n[2] B. Minixhofer, I. Vulić, E. M. Ponti. Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching. arXiv preprint arXiv:2503.20083, 2025."}, {"review_id": "bjStjujjXZ", "reviewer": "Reviewer_eA55", "summary": "This paper introduces Dynamic Memory Sparsification (DMS), which closely mirrors Dynamic Memory Compression of Nawrot et al. 2024, except that it decides on evicting tokens from the KV cache instead of accumulating tokens on top of existing tokens. The approach yields improvements for the Pareto-frontier between efficiency and quality on a broad set of reasoning-intensive tasks.", "questions": "- Figures 2 and 3 are a bit crowded with the W-L-CR representation. I would encourage the authors to tidy up those plot for easier readability.\n\n- How would the right subfigure of Figure 4 look like with a 1B training token budget (as in the DMC paper?)", "limitations": "The authors don't have a devoted discussion of their limitations. I would encourage them to contextualize the limitations of DMS in the limitations of DMC (e.g. alternatives to the DMC training procedure adopted here).", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The paper builds in an intuitive way on the prior success of DMC to create a simpler method of DMS. The proposal is an intuitive one, mirroring the merits of DMC.\n\n- Delayed eviction is a simple and creative technique that is shown to effectively reduce KV cache size without quality degradation.\n\n- The evaluation is strong, as DMS is shown to be on the Pareto frontier of inference-time scaling quality with respect to efficiency.\n\n### Weaknesses\n\n- The inference-time scaling configurations of the form W-L-CR are presented without much motivation or hypotheses as to which ones might be expected to yield the best results and why. That would be an useful study/ablation to include.\n\n- The fact that the number of training tokens is 1B is not insignificant — does it fail with fewer training tokens? If so, this method is a relatively expensive one.", "was_revised": true, "final_justification": "This paper builds naturally on top of Dynamic Memory Compression, except for Sparsification, without any significant weaknesses in my opinion, although my familiarity with this line of work is limited. I maintain a positive assessment of this paper, as a valuable continuation of DMC.", "rebuttal": "We appreciate Reviewer eA55’s time and effort in reviewing our work and for the helpful suggestions they offered.\n\n**W1: Which L-W-CR setups yield the best results and why?**\n\nIn our manuscript, we hint at the fact that sequential scaling rather than parallel scaling brings more gains (L240); however, we also remark that often hybrid strategies employing parallel *and* sequential scaling lie on the Pareto frontier. For CR, the key trade-off is whether the accuracy gains from scale outweigh accuracy losses due to sparsification. In our main experiments, CR 4x consistently achieves the best balance. Post-submission, we improved performance further using win=256, shifting the optimal CR to 8x - 10x. Hence, the optimal CR could be pushed even further by developing better retrofitting strategies and model configurations that do not degrade the model’s quality.\n\n**W2: Does the method fail with fewer than 1B training tokens?**\n\nDMS works even on small numbers of training tokens, as we demonstrate below in the answer to Question 2. However, longer retrofitting may reduce the variability of the results. In our experiments after the submission, we found that enlarging the sliding window also helps to stabilise the training and reduce variability, which can be seen in Figure 4 (right). With win=256, we were able to improve the results of R1-Qwen models shown in Figure 5 with only 100 steps for every CR increment instead of 3000 steps. With 100 steps for every CR increment at 8k context and batch size 1024, retrofitting requires 839M tokens per CR, and we have not attempted to optimise these numbers further. Please note that DMC uses 24B–168B tokens depending on CR; their 1B number applies to the neuron decay phase, not full adaptation.\n\n**Q1: Figures 2 and 3 are a bit crowded with the L-W-CR labels**\n\nWe agree and will revise the plots to improve clarity: adjusting label positions, plot layout, font size, and potentially removing less important labels from non-Pareto points.\n\n**Q2: How would the right subfigure of Figure 4 look like with a 1B training token budget (as in the DMC paper?)**\n\nReducing the amount of training data increases the variability of results, especially at small sliding window and model sizes. We have run additional experiments retrofitting Llama 3.2 1B Instruct with only 50 steps per CR, achieving CR 4x with ~630M training tokens. We ran 3 seeds for different window sizes (16, 64, 256). Considering that the vanilla model achieves 47.0 on GSM8K, the additional data-efficient GSM8K scores are:\n\n| Model                    | CR 2x | CR 3x | CR 4x |\n|--------------------------|------:|------:|------:|\n| DMS (win=16, seed 1)     | 43.4  | 44.5  | 28.7  |\n| DMS (win=16, seed 2)     | 41.8  | 41.9  | 39.3  |\n| DMS (win=16, seed 3)     | 41.8  | 33.1  | 35.6  |\n| DMS (win=64, seed 1)     | 41.7  | 43.1  | 40.3  |\n| DMS (win=64, seed 2)     | 43.9  | 41.8  | 41.8  |\n| DMS (win=64, seed 3)     | 42.4  | 35.9  | 42.1  |\n| DMS (win=256, seed 1)    | 43.9  | 45.6  | 43.9  |\n| DMS (win=256, seed 2)    | 45.1  | 43.9  | 44.9  |\n| DMS (win=256, seed 3)    | 43.9  | 44.7  | 43.3  |\n\nWhile DMS incurs some degradation due to the small model size and the extremely fast training regime, we note that a larger window size significantly stabilises training and mostly preserves the original performance.\n\n**A discussion of limitations is missing**\n\nWe discuss the limitations of our work in Appendix A. We will also expand this in the Related Work section to highlight which limitations are inherited from DMC and which choices in the DMS training procedure were able to overcome some of the limitations of DMC, such as being data-hungry, more cumbersome to implement, being more prone to lead to performance degradation etc."}]

Author rebuttal:

Meta-review: (a) Summary of claims and findings:
The paper tackles the KV-cache bottleneck in inference-time scaling for LLMs and proposes Dynamic Memory Sparsification (DMS), a retrofittable KV compression method that delays token eviction by a sliding window before enforcing sparsity. The delayed eviction aims to preserve salient information while achieving a target compression ratio (CR), enabling “hyper-scaling”: generating more tokens or parallel chains under the same compute/memory budget to improve reasoning accuracy. Training uses a sparsity objective plus logit distillation; at inference a structured additive mask enforces eviction. Across several reasoning benchmarks (AIME24, MATH500, GPQA Diamond) and model families (DeepSeek R1-distilled/Qwen, Llama 3.2 1B), DMS improves accuracy–efficiency Pareto frontiers compared to training-free methods (TOVA, H2O, Quest) and prior trainable compression (DMC). During discussion, authors clarified that a single retrofitting run produces multiple CR checkpoints, introduced a larger window (win=256) that greatly reduces steps per CR (~30×) while improving stability, provided H100 latency measurements showing gains in large-batch/long-sequence regimes, and added broader task results (LiveCodeBench code generation, QASPER long-context QA).

(b) Strengths:

- Relevance and impact: KV cache is a primary limiter for inference-time scaling; improving the accuracy–efficiency frontier is practically valuable.
- Simple, implementable idea: delayed eviction via a structured mask that integrates with standard kernels; compatible with GQA.
- Strong empirical evidence on reasoning-heavy tasks and multiple model sizes; consistent Pareto improvements in KV reads/peak tokens versus accuracy.
- Practicality improvements substantiated in rebuttal: single-run, multi-CR retrofitting; win=256 dramatically improves sample efficiency and stability; latency results on H100 show real gains in realistic scaling regimes.
- Clear positioning relative to training-free and prior trainable approaches; ablations on window size and data budget provide actionable guidance.

(c) Limitations and remaining gaps:

- Novelty is evolutionary: DMS is a pragmatic variant of prior trainable KV compression (e.g., DMC), exchanging “merge” for “delayed evict.” Nonetheless, the delayed strategy is simple, effective, and easier to implement.
- Practical efficiency results rely on research code (non-optimized); latency wins are most visible at large batch/long sequence lengths. Production-engine validation (e.g., vLLM/TensorRT-LLM) would strengthen the case.
- Broader task coverage is still limited beyond reasoning; added results on code gen and long-context QA are helpful but brief.
Some methodological details (decision signal design, mask materialization, per-head policy) were clarified in rebuttal and should be surfaced in the main text; figure labeling (W–L–CR) can be tightened.

(d) Reasons for decision:

The paper addresses an important and timely bottleneck with a conceptually simple mechanism that consistently improves the accuracy–efficiency trade-off, enabling more effective inference-time scaling. The method is practical to integrate, the evaluation on reasoning tasks is thorough, and the rebuttal substantially strengthens concerns on retrofitting cost (single run, multi-CR; win=256 for ~30× fewer steps), latency (H100 measurements), and generalization (code, long-context QA). While the contribution is more systems/pragmatics than theoretical novelty, the demonstrated gains and practicality justify acceptance as a poster.

(e) Suggestions for camera-ready:

- Move key clarifications into the main text: single-run multi-CR procedure, win=256 data-efficiency/stability improvements, decision-variable options (q[0] vs k[0] vs learnable probe) with brief ablations, GQA implications, and mask implementation (e.g., relate to FlashMask).
- Add a concise end-to-end retrofit cost table (tokens, GPU-hours) per model/CR/window; include variance across seeds and practical heuristics for choosing window size and CR ramps.
- Summarize H100 latency results in the main paper and, if feasible, add a small study on a production inference stack (throughput/VRAM vs batch/length).
- Improve figure readability for the W–L–CR trade-off plots; add an explicit limitations section noting regimes where training-free methods may be preferable (short inputs, strict memory caps).
- Keep the new broader-task results (LiveCodeBench, QASPER) and briefly outline future stress tests (e.g., adversarial/irregular attention patterns).
Decision: Accept (poster)