ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
Authors: Akshat Ramachandran
Marina Neseem
Charbel Sakr
Rangharajan Venkatesan
Brucek Khailany
Tushar Krishna
Venue: ICLR 2026 Oral (2026)
Keywords: Large Reasoning Models
KV Cache Compression
Quantization
Eviction
Sparsity
Thought-Aware Compression
Primary area: generative models

Abstract: The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key–value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization–eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over SoTA baselines.

Published as a conference paper at ICLR 2026
THINKV: THOUGHT-ADAPTIVE KV CACHE COM- PRESSION FOR EFFICIENT REASONING MODELS
Akshat Ramachandran' ', Marina N Charbel Sakr", Rangharajan Venkatesan", Brucek Khailany?, and Tushar Krishna' !Georgia Institute of Technology. Atlanta, USA
2NVIDIA Research, Santa Clara, USA
akssatt r@gatech. edu, tushar@ece wwt.eh..eece....
2 (mneseem, csakr, rangharajany a
Work done during internship at NVIDIA
ABSTRACT
The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key-value (KV cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV M thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the COT. It applies hybrid quantization eviction strat- egy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts reasoning trajectories evolve. Furthermore, implement ThinK V, we design a kernel that extends PagedAtteentin to enable ef- ficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeck-Rl-Dissee son across mathematics and coding benchmarks show that ThinKV achieves near- lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8 x higher inference throughput over SoTA baselines.
iitiiaattioneiictiiinnn
GPT-OSS, and NVIDIA AceRea-
Desedo
(a)
Dxtened
Token
Resladan
Ohaervaion window
Canen
Takon
Stapk V. RocketKV
1120, RanS
LaoyEvictoon
R-KY
Thoughts
BarehodaBaichh Accuracy (Sa)
Snank R-KV
FulKK
Exsection
(Ours)
Lazy Eviction
(b)
Quanization
ReaS
Thooughs
H2O ScissorRands
20
ThinKY (Ours)
Output Token
Figure 1: Illustrative comparison of KV cache compression methods as tokens are generated: (a) Existing techniques: SnapKV, RocketK H2O, RaaS, LazyEviction and R-KV, and (b) ThinKV (Ours). (c) Accuracy vs. TPOT comparison for GPT-OSS-20N evaluated on LiveCodeBench.
INTRODUCTION
Long-context modeling (Yuan et al. 2025 is core capability for next-generation LLMs. While early work focused on long-inpur contexts, the advent of Large Reasoning Models (LRMs) eeg., OpenAl's O-series (OpenAl 2024) and DeepSeek-Ri (Guo al. 2025 has shifted 'attention' to
The may be interpreted either as Thin KV Think KV

---

Published as a conference paper at ICLR 2026
long-output contexts, involving generation of thousands of tokens (Liu et al. 2025). This capability facilitates extended reasoning (Zhu et al.) 2025) and long-horizon code generation (Seo et al. 2025).
LRMs attain state-of-the-art reasoning accuracy by generating long chains of thought (CoT) (Wei ct al. 2022), producing extended rationales to explore and verify solutions. However, long CoT generation incurs substantial memory overheads due to rapid growth of the dey-value (KV) cache during decoding (Li et al. 2025). In code generation (Jain et al., 2024), for instance, a GPT-OSS- 20B (Agarwal et al.) 2025) producing ~322 tokens with batch size 32 requires 50 GB for the KV cache and 40 GB for
ceanils-xxeeedin the 80 GB of an NVIDIA A100. Since the decode stage is memory- bound (Recasens et al. [aaa]]2222) (2055), the KV cache becomes the central bottleneck for long-outpur context generation. KV cache compression thus offers a promising solution.
1.1 RELATED WORK AND LIMITATIONS OF EXISTING COMPRESSION TECHNIQUES
Existing compression approaches span quantization, eviction, low-rank approximation, and hybrids thereof. Most, however, focus on the prefill phase of long-input tasks (Li et al. 2024) (Figure 1 a)) and are ill-suited for LRMs and long- generation. A few works study decode-stage compres- sion (Zhang et al., 2023 Shi et al. 2025 Liu et al., (024bb for LLMs, but typically use greedy recency-based eviction (Figure
[a) or uniform quantization, both of which overlook reasoning dynamics, leading to degraded LRM accuracy (Figure IJ)).. For additional details refer $B
LRM KV Cache Compression. Recent work has moved beyond simple recency-based eviction
towards methods that partially capture reasoning dynamics (Figure I[))). RaaS (Hu al.) 2025)
preserves tokens with re-emergent importance to avoid premature eviction; LazyEviction (Zhang
et al. 2025a) delays eviction retain tokens likely recur by tracking attention activity: R-KV CCai et al. (0255) combines attention-based importance with redundancy; and PM-KVQ (Liu et al., 2025) progressively reduces token precision during decoding. However, they operate at the token level, making compression decisions that overlook the broader semantic structure of reasoming. This can cause removal of reasoning- crittcal tokens or limit compression by overweighting less important ones, yielding suboptimal securacy-efficiene trade-offs (Figure I c)), under high compression.
System. Dynamic token eviction creates memory holes, causing internal fragmentation (Kwon et al.) 2023). H2O (Zhang et al. (203) mitigates this with circular buffers, but these support only contigu- ous eviction, whereas LRM policies conduct non-contiguous token removal. While other methods (Cai et al.) 2025) explore gather-based compaction; it requires irregular, index-based memory ac- cesses that contend heavily for HBM bandwidth. Our analysis ($5.T) reveals that gather, sharply increases time per output token (TPOT) (Figure I()),, consistent with Kwon al. (2023).
1.2 CONTRIBUTIONS
Motivated by these limitations, we ask: Can a KV cache compression framework go beyond loken- level heuristics preserve reasoning-critical information under high compression while maximizing efficiency? We present ThinKV (Figure Ib)), thought-adaptive hybrid quantization-evictin framework
2 with four key components:
Thought Decomposition ($3. $4.1 We show the CoT in LRMs can be decomposed into distinct
thought types, with their differentiation enabled by degree of sparsity in attention weights.
Think Before you Quantize (TBQ) (33.2) (4.2): We propose a KV cache quantization scheme
that assigns precision tokens based on the importance of their associated thought type.
Think Before You Evict (TBE) ($3.3) $4.3): We introduce TBE, a thought-adaptive eviction
scheme that leverages inter-thought dynamics to progressively evict tokens.
Continuous Thinking (55): We design kernel extending PagedAttention that efficiently reuses evicted memory slots for subsequent tokens without relying on expensive compactions.
Through algorithm-system co-design, ThinKV delivers aggressive KV cache compression while preserving accuracy and improving inference efficiency ($6). On mathematics and coding bench- marks with DeepSeek-RI! -1-t---i-Di-LL-aaaaae
GPT-OSS, and several other LRMs, ThinKV achieves near-lossless accuracy with under 5% of the original KV cache, outperforming state-of-thh-ar baselines with up to 1.68x lower TPOT (Figure (()) and up 5.80x higher throughput.
2

---

Published as a conference paper at ICLR 2026
2 WHY QUANTIZATION+EVICCIN ?
The memory footprint of the KV cache can be expressed as Mem(KV) x (1 + bLgen) x aB, where I is the prompt length, Lgon the total number of generated tokens, denotes the bytes per parameter. The factors b E [0, 1] capture memory reductions from quantization and eviction, respectively. Uncompressed KV cache corresponds
(full precision) and 1 (no eviction).
For quantization (Ramachandran et al. 2024b), we adopt KIVI (Liu jet al. 2024b) as the representative. As shown in Figure 2 GGPT-OSS 20B on LiveCodeßenchh, reducing fails to proportionally increase compres- sion ratio, since in LRMs we find that aggressive quantization inflates Lignn eroding memory savings and simultancously degrading accuracy. Under eviction-using R-KV (Cai et al.
Ouavitwvics + Furtton -
INhis
pod
8 R
bet 01?
initially in- creases compression ratio while preserving accuracy. Unlike quantiza- tion, eviction does not cause an increase in generation length; however, as b accuracy degrades with higher compression.
2
Comoression Racio
Figure 2: Accuracy- compression tradeoff of quantization, eviction and hybrid approaches.
Hybrid compression (ThinkV,$44 traces a Pareto-optimal frontier, sus- taining high accuracy much higher compression ratios. We believe that by combining quantization and eviction, it partially regularizes quantization-induced length inflation and maintains accuracy extreme compression by flexibly trading off token count and precision.
MOTIVATING ANALYSES
In this section, we present three key observations that motivate the design of ThinKV's algorithm.
3.1 ATTENTION SPARSITY FOR DYNAMIC THOUGHT DECOMPOSITION
denote the set of Definition 1 (LRM Thought Decomposition). Let T (c..
thought categories. During generation, an I. layer L.R produces a sequence (yo.-... 9n-1)), where each yi is a discrete token. At decoding step i, the cache of layer is denoted by St, representing the set of stored KV pairs up to that point. Thought decomposition is defined as.
For each step i E [n] and generated token yi. categorization function associates category label 0j for [[T]] as, é {yo,,... -3n-13 T ((y:) Cj-
Each token generates one KV entry per layer, which is assigned a category as identified above. Formally, S 5 [(K) V/,c,) c,)), where the KV entry associated with its thought type cj.
An exact realization of á is nontrivial. Prior works approxi- mate é by maintaining a keyword list for each category; Ven- hoff et al. (0025) found [ categories, while Chen et al. (2025b) identified [T|=3. However, keyword-basee methods fail when models generate lexical variations and tokens out- side keyword lists (Agarwal et al. 2025).
We present an empirical observation that enables a general- izable approximation of o, based on the sparsity pattern Ra- machandran et al. (2025b) of the normalized attention score Figure 3 reportt layer- wise sparsity ratios for two LRMs (R1-
(43 R-1-L.amm--BB Layer 10 AIME
Llama-8B and -70B) on AIME and LiveCodeBench prompts respectively. We draw the following key observations:
Observation la: The attention sparsity pattern across decode steps exhibits tri-modal distribution.
(6) RI Rl-lama-70,, Layer 4 LiveCh Figure 3: Layer-wise attention sparsity across decode steps for R1-Llama-8BB on AIME and R1- Llama-70B on LiveCodeBench.
To only interpret the sparsity regions, we follow Chen et al. (202bb and assign representative keywords ($D.2) as illus- trative labels. This categorization yields three thought types
The normalized attention scores defined as softmax and sparsity is measured by setting thresh-
old at 1% of the row-wise maximum, following 7hang 2023
3

---

Published as a conference paper at ICLR 2026
(7] 3): reasoning (R), involving systematic thinking; execution (E), encompassing calcula- tions, or code generation; and transition (T), capturing uncertainty and backtracking behavior.
Observation 1b: T (R, T, E), with T thoughts exhibiting the highest sparsity, followed by R thoughts, while E thoughts have the lowest sparsity.
Some layers exhibit more than three sparsity regimes ambiguous boundaries E.4 As shown in fixing [T] = 3 and choosing the optimal layer subset L achieves the best accuracy.
6
3.2 LRM THOUGHT IMPORTANCE
We examine relative thought importance the basis for thought-adaptive quantization scheme. Consider an LRM CoT output consisting of N thought segments (Y;), followed by final answer A. Inspired by [ogddn et al. (2025), we measure the counterfactual importance of each segment Y, by comput- ing the KL divergence between A's distributions obtained with and without Yi, averaged over 50 rollouts. Figure 4 presents thought importance for GPT-OSS-22 on AIME and Live- CodeBench.
1.55
Figure 4: Counterfactual impor- tance of thought categories for GPT-OSS-20B on AIME and Live- CodeBench.
Observation 2. We observe clear hierarchy of thought im- >> T. Interestingly, we find outlier T thoughts with unusually high importance which correspond to backtracking behavior and removing them causes the model to loop endlessly (see example in (E.17).
portance:
3.3 LRM THOUGHT ASSOCIATION
We analyze inter- thought dynamics by measuring pairwise sociations (Bogdan et al. 2025). For (Y,, Y,).j > i, we sup- press attention to Y; (all layers and heads) and compute the KL divergence of Yj's logits, averaging over its tokens to obtain a
directed association score, indicating the extent Yy depends on Y. FFigure 5) illustrates the influence of thought Y, (X-axis) on subsequent thoughts Y, (Y-axis) during generation for an AIME prompt (additional visualizations in E..5
Observation With every T thought, all prior thought seg- ments become progressively less influential (fewer tokens need to retained), underscoring role in altering the reasoning trajectory. Note R and E segments highlighted with and respectively. Additionally, T thoughts are weakly influenced by prior context (high sparsity) (L), while thoughts depend heavily on context bounded between consecutive transitions (low sparsity), bolstering Observation lb.
Figure 5: Pairwise thought sociations for GPT-OSS- 2OB on AIME. c5ar denotes thought seg- ment type and its position in COT.
4 THINKV METHODOLOGY
In this section, we present ThinKV's hybrid scheme, which first decomposes tokens into distinct thought types ($4.1) and then applies thought quantization (44.2) and eviction ($4.3)
4.1 ATTENTION SPARSITY GUIDED CONSTRUCTION OF 6
Building on the observations in $3.1 we now detail how ThinKV leverages attention sparsity to dynamically identify thought types, forming the basis of it's adaptive compression strategy.
Offline Calibration. We use kernel density estimation (KDE) (Pazenn 1962) to derive the |T|
sparsity thresholds θ .O.,..., .., ) that separate thoughts. From calibration set of P prompts, estimate KDE per prompt and select the layer subset L that exhibits [T] modes. We extract [T| - 1 thresholds by identifying local minima between modes (statistical term). and compute final thresholds by averaging across all prompts and layers in L Refer $D. algorithm.
4

---

Published as a conference paper at ICLR 2026
Phyncall By hieeoa
for GPU DRAM)
ZB
Togled KV Mocks
Block
G
0
Black
Rock 3
M
B
Blat4
Doasston
Joiest Talen
Cottenen Thikkng Back Take
Torgh Typ
Payee Blck
Sar ken
Seywr Madds
Fretie Mink
Dayeiigg Step
Davding Step
10)
4
1111
(0e)) So 11M
[10101101]
6000 fo0o
- 1000
Segtker Mini
Eriasse
Figure 6: Walkthrough of ThinKV with T -= 9 block size = 4, R (2).
Decode-Time Behavior. During generation, 6 is upproximated by averaging sparsity comparing with thresholds e to determine the thought type. From FFiuree 3) and consistent with Chen et al. (2025b), thought segment in the CoT typically span (00-300 tokens. We therefore set a refresh interval o 7= = 128 steps, updating categories only at these intervals to minimize overhead.
L and
4.2 THINK BEFORE YOU QUANTIZE (TBQ)
Problem Formulation (Thought-Adapinn Adaptive Quantization). Let B = {bo, (b0,b,,...,,,,77-1 denote the set of available quantization bit-precisions, ordered such that by b1 < < b We define a KV cache quantization policy that allocates precision to tokens according to thought importance:
Define an importance function p : T - N that assigns each thought type Cj T a score plez)) We then construct a mapping vb : T + B such that higher importance implies higher precision,
i.e. p(c,,) > p(Cz2) = ((c,,) (((,))-
quantized with bit- bit-precision Keee), yielding (K,, V.,eg) .)), where K,, V. denote the quantized KV representations.
Each new KV entry (Kt, V(,c,) E
Building on the observed thought importance in 3.2 p(R) = 2, P(E p(T) = We con- struet B {2, 4, 8} with ternary for 2-bit, NVFP4 (Alvarez al. 2025) for 4-bit, and FP8 for 8-bit. Ternary and NVFP use group quantization with 16 and a shared FP8 (E4M3) scale factor, whereas FP8 employs a per-tensor FP32 scale factor (see $D.3p. We assign R. E, and thought tokens to 8-, 4- and 2-bit precision, respectively. Notably, as shown in $6 R tokens maintain comparable accuracy even 4-bit, allowing adoption of 4-bit for R in practice without loss of per- formance. Following Liq et al. (2024b), keys are quantized per-channel while values are quantized per-token, buffer Brut of size g stores tokens in full precision until the group size is reached, after which they are group quantized. Figure 6 (TBQ) presents an example with g = 4.
4.3 THINK BEFORE YOU EVICT (TBE)
Problem Formulation 2 (Thought-Adaptive Eviction). Let k be the cache budget, ((e)) S St be the KV entries of a thought segment of type cj and R [Ro, Ri...., Rm-1) denote the set of m retention rates, in descending order, where Rn, specifies the number of tokens to be preserved when a segment is selected for eviction the n-th time. Eviction policy π (c;)) + S. (c;) is defined as,
Case 1: If a reasoning trajectory-changing thought a is generated, progressively evicts preced- ing thoughts such that [S (c,)) = min (IS((c)), (c₂)], Rn), where n identifies number of times preceding thought сj has been selected for eviction (i.e., the number of trajectory changes in reasoning).
Case 2: If no at thoughts are generated, but [ k. we find the oldest and least important thought segment to apply until |
Following from the observation in $3.3 transition thoughts are the reasoning trajectory-change thoughts e, Since we employ a refresh period of 128, every thought segment contains 128 tokens. Therefore, following Problem Formulation 2 we define the retention schedule as R (64,32 16, 8, 4) for all thought types, with a minimum retention of 4 tokens per segment
'A contiguous span of tokens assigned to the same thought type.
5

---

Published as a conference paper at ICLR 2026
(see Figure 11 a)). At each transition thought ct, the eviction policy anneals preceding segments (including previous transitions) by reducing them to the next lowest retention level in (see (see ure 6). With successive transitions, all previous thought segments are progressively shrunk until the minimum retention value reached. If no Gr occurs or all segments before the current e are already at their minimum, evicts from the oldest and least important segment to its next lowest retention level in R. TBE is proactive eviction scheme that operates at the granularity of thought segments, evicting large sets of fow-importance tokens as opportunities arise rather than waiting for cache saturation and stepwise per-token removal. This strategy reduces eviction frequency and, as shown in 6 minimizes overhead.
Eviction Policy (T). We apply K-means clustering to post-RoPE key embeddings (He et al. 2025 with k= min[] (c:)], R( R(m,c;)) Cz)). The cluster centroids correspond to keys that are retained, and the corresponding value tokens are preserved. An illustration is provided in $ 4.4
THINKV SYSTEM IMPLEMENTATION
We introduce Continuous Thinking (CT), an extension of PagedAttention (Kwon et al. 2023 to enable in-place memory reuse of evicted KV tokens, without expensive gather-based compactions.
5.1 THE COST OF GATHER- WATHER-BASSD COMPACTION
Time Outout Taken (ms)
200-
Perilaeer Gather
Existing LRM eviction methods drop non-contiguous tokens from arbitrary positions within the COT, causing internal frag- mentation that requires gather-based compaction. To quantify its overhead, we study R-KV Cai et al. (202)) with a 1024- token budget. We implement two Triton gather kernels: (a) a sequential variant and (b) an overlapped variant employing separate CUDA streams to run concurrently. Figure 7 reports kernel performance on DeepSeek-RI- RI-Distiil--.amm------
150-
100-
Not Deeeltr
128
Botch
(a) Sequential Gather
Time Per Cutput Taken
Time Per Output Token
Pen-aaer Attention on Seodown
Observation 4a (Sequential). Per-layer gather overhead grows sharply with batch size Figure aa)), causing up to 37x TPOT slowdown.
60
40-
Observation (Overlapped). At small batch sizes, the gather cost is effectively hidden, yielding lower TPOT relative to the sequential case. As batch size grows, however, over- lapped gather begins interfere with subsequent-layer's atten- tion, as shown in infiie b). Specifically, contention arises on HBM bandwidth, where the gather kernel's KV writes conflict with the attention kernel's KV reads. This contention inflates down), and thus causes higher TPOT.
Overlapped (Parallel) Gather Figure 7: Performance of sequen- tial and overlapped gather kernel on RI-Llama-&B. attention time (up to ~35% slow-
5.2 CONTINUOUS THINKING (CT)
Block Table. PagedAttentton maintains a block table for each request and each layer. Figure 6 (see
for detailed walkthrough) shows the modified block table, recording the following information (new fields in green),
Physical block and Filled: KV block index in GPU memory and its token count.
Thought type: Thought type of tokens in block; CT implements thought-aware paging.
Start indices: Records the start position of the thought segment of tokens in the physical block.
Segment masks: If there are multiple start indices, the segment mask is bit vector (length=block size) that marks the locations corresponding to each start index with a 1.
Eviction mask: A bit vector marking positions of tokens evicted by TBE with 1s.
TBE with CT. The CT kernel collaborates with TBE to perform eviction. As shown in Figure 6 TBE selects segments for progressive eviction using the thought type and start index fields. Tokens marked for eviction are not immediately removed; instead, they are sork- in the eviction mask, with actual removal deferred until new tokens arrive overwrite into the evicted slots.

---

Published as a conference paper at ICLR 2026
FullkV (baseline)
Rad5
R-KV ThinKV (Ours)
H2O
LazyEviction
R1-Liama-88
R1 -1--mmma-708
GPT-O8S-20E
Ace 148
00.30
Accuracy
Taandy xtx Avip
TheieV Ang
5003
Thcckt PHI
00.00
2003
4000
017000
2,075
Thinkt ArD
200
512
1864
3040
4901
2040
0.
790
2000
$1.39%
31
(b) LiveCodeBench
Theakh Aung
106
2001
300
2.32%
8,14
10.00
Token Budget
Token Budger
Token Buddees
en Eudoe Fullky Memory
Memory
Fulkk Memory
FUlkY Memory
ThinKV Others
ThinKV Others
ThinKV Othars
ThinkV o/ Others
(c) MATH-500
Figure ThinKV compared with SoTA eviction baselines, reported as pass @ accuracy.
Efficient Memory Management. When new tokens of a thought type are generated, the CT kernel the eviction mask to identify reclaimable slots in existing blocks of the same type. The start index of the new thought segment is appended to the existing block table entry, and the segment mask updated mark its token positions. By reusing slots place, CT avoids compaction and eliminates fragmentation. Moreover, tokens need not be reordered during attention computation, since attention permutation-invarian ($ CC.). Therefore, our modifications leave the PagedAttention kernel for attention computation unchanged enabling seamless integration with serving frameworks.
EVALUATION
6.1
EXPERIMENTAL SETUP
Models and Datasets. We evaluate DeepSSck-R1--issillLllm (8B and 70B), DeepSeek-
RI-Distill-Qwen-145 14B, GPT-OSS (20B and 120B), QwQ-32B, AccReason-Nemotron: 14B, and MobileLLM-R1-9500 Evaluations span mathematies (MATH-500 (Lightman al. 2023 AIME
MMA 2024). GSM8K (Cobbe et al., 202Tg) and code generation (LiveCodeBench (Jain et al. For calibration, we randomly sample 100 prompts from s1K (Muennighofi et al. 2025).
2024
Hyperparameters. We set number of thoughts |T| = 3, optimal calibration layers [C] = 4. refresh rate 7 128, group size 16, retention rates R {64, 32, 16, 8,43 and E thoughts are quantized to 4-bits and T thoughts to 2-bits.
4] and CT block size R
Baselines. We compare accuracy against eviction baselines, H20 ZZanng al. 2023) (LLMs) and RaaS (Hu al. 2025), R-KV (Cai et al. 2025). LazyEviction (Zhang et al. (225a) (LRMs), as well as quantization baselines, KIVI (Liu et al., (024b) (LLMs) and PM-KVQ (Liu et al. 2025) (LRMs).
System Optimizations. We implement ThinKV in a hardware-friendly manner for GPUs. We following Liu et al.) (2024b), we fuse
design optimized CUDA kernels for group quantization dequantization with matrix multiplication to reduce overhead. Two T tokens at 2-bits packed into a 4-bit format, consistent with R/E tokens, ensuring aligned memory. TBE's K-means-based eviction is accelerated GPUs with CUDA, following Kruliš & Kratochvil (002). CT is fully implemented in Triton, extending the PagedAttention kernel OpenAl (2025).
Evaluation Setup. All experiments conducted 1xNVIDIA A100 80GB GPU and 1x NVIDIA GH200 Superchip. Following Cai et al. (2025), we constrain maximum generation length to 32K tokens. For accuracy evaluation, for each question, generate 8 independent responses

---

Published as a conference paper at ICLR 2026
and compute pass accuracy as pass @ C Σ Ps. where pt denotes whether the i-th sampled response is correct Ramachandran et al. (2024a). All throughput and latency numbers are obtained by averaging across 3 independent runs. Importantly, in experiments we treat prefill-tokens as R type (see Figure [b)). Refer Appendixx for additional details.
6.2 MAIN RESULTS
Accuracy Comparison with Eviction Baselines.
Table 1: Comparison of ThinKV with KV
In Figure 8, we evaluate diverse LRM families on reasoning datasets with KV cache budgets rang- ing from 64 to 4096 tokens. The average gener- ation lengths are 9,020 tokens on AIME, 16,166 on LiveCodeBench, and 2,468 on MATH-500. On challenging reasoning benchmarks such AIME and LiveCodeBench, ThinKV achieves competi- tive accuracy with cache budget of 1024 to- kens, accounting for < 3.67% of VullKV mem- ory, whereas other methods require: 12% reach similar accuracy. For RI-Llama-81 and AceReason- 14B on AIME, ThinKV sustains < 4% drop using only ~ 1. 3% of the KV cache. ThinKV's 's hybrid quantization-evinn
quantization baselines.
Model
Method
Bit-Withh
JAME
LiveCndeeeechh
Baseline
16:16
53.33
47.90
R1-Qwen-14B
PM-KVO
KIVI
3.2.322
2.2
201000
43.33
41.97
14.36
ThinKV (k-1024)
3.5:3.5
Raseline
16-16
73.33
55.45
QwQ-32B
PM-KVV
KIVI
3.5-3.5
2.2
67.86
60.56
40.75
46.68
ThinKV (k-1024)
3.4-3.1
Table 2:
Throughput (tokens/s) comparison
on GPUs. "Mem. ftprnt: Memory footprint (%) normalized to FullKV.
Methed
Tol. Budaet Mem. 3prnt (5)
Batch
A100
[Bachh
CIRZI00
Toks
Fally
100
201.2
19
433
R-KY seg
1034
5.48
268
1450.5
350
2415
R-VV lowi)
1084
5-48
268
2130:9
540
4911.3
TMRKY
1824
2.51
711
8412.2
938
10578.5
and thought-adaptive scheme, enables superior accuracy while sustaining higher compression. ThinKV operates at an average pre- cision of 3.4 bits, with harder problems achieving lower precision due to more frequent transition thoughts.
No butch As compecsiios compuotton
KKV GeaT
1024
3:48
1768
256
R-KV ThisKY TBQ
$298.4
Accuracy Comparison with Quantization Baselines. We sum- marize our findings in Table I using & = 1024 for ThinKV. KIVI applies uniform INT quantization across all tokens, while PM-KVQ progressively reduces precision to a final 2-bit repre- sentation. Both approaches treat all tokens equally important, leading to substantial accuracy degradation on LRMss. In contrast, ThinKV's thought-adaptive quantization (TBQ) assigns precision based on thought-type importance, achieving minimal accuracy loss with an average precision of
Table 3: ThinKV throughput on RI-Llama-SB (A100 (A100-80GB, 32K generation) with 2048 to- ken budget.
Batch
Method
(max)
Budget
Throughopu
PallKY
50
13
297.5
ThinKY
50
290
2048
4688.4
3.4 bits.
reports end-to-end throughput on two GPUs for R1-Llama-88 performing continuous generation of 32K tokens. As baselines, we include two R-KV variants: one performing sequential gather (seq) and the other overlapped gather (ovl). FullKV and R-KV use FlashAttention (Dao 2023), while ThinKV employs the CT kernel. For each method, we report the maximum batch size achievable on different GPUs. At batch size 1, all techniques achieve comparable performance with only marginal improvements over FullKV (Cai et al. 2025). The main throughput gains come from ThinKV's ability to sustain larger and more efficient batch inference. Specifically, ThinKV's ' hybrid scheme attains a higher compression ratio, supporting up to 3x larger batch sizes than R-KV and yielding throughput gains of up to 5.8x over R-KV (seq) and 3.6x over R-KV (ovl). To isolate CT kernel's impact on ThinKV throughput at larger batch sizes, we conduct an iso-batch, iso-compression (ThinKV w/o TBQ) comparison with a batch size=256. ThinKV achieves up to 3.2x and 1.6x higher throughput than R-KV (seq) and R-KV (ovl), respectively due to the elimination of gather-based compaction.
Throughput Analysis. Table 2
we report results using a 1024-token budget with R4E4T2 precision, which limits ac- curacy loss to < 1% for most LRMs and datasets. For more sensitive models, we also evaluate a conservative 2048-token budget that preserves accuracy across all settings. As shown in Table 3] ThinKV at 2048 tokens increases the maximum batch size from 13 to 290 and achieves a 15.8x throughput gain over V demonstrating substantial acceleration under preservgi con- straints.
In Table 2
E2E System Throughput versus User Latency Analysis. Motivated by the Synammcc-srving anal- yses in Kwon et al. (2023): Yu et al. (2022), we evaluate ThinKV under multi-user concurrency.
8

---

Published as a conference paper at ICLR 2026
DHAW
D200
1,00
11
500
3000
+30 500
640
ONtrnde 2119
())
(b)
(6)
4270
APME
LlanOuttBeenh
Dospeet
3:50
(d)
(0]
03
Figure 10: ThinKV ablation experiments: (a) recall rate of tokens with Top-10 10 attention scores for RI-Llama-8B on AIME, (b) ThinKV eviction curve. Impact of (c) refresh rate (7) for GPT-OSS- 20B model on LiveCodeBench, (d) Impact of compression on generation length for RI-Llama-8B, (e) impact of block-size throughput, (f) % breakdown of thoughts for R1 R1-Llama-8B.
For batch size of B, we issue B parallel requests to emulate B active users and measure the achieved system throughput (request- s/s) together with the average end-to-end latency experienced by each user. The goal of this experiment is to evaluate performance when B concurrent requests are actively being served. We report our findings in Figure 9 for R1-Llama-8B on A100-80GB GPU. We randomly sample B AIME prompts and employ cache budget of 1024 tokens. FullKV cannot sustain batch sizes beyond = Under iso-batch comparison at B 8, ThinKV achieves up to 58% lower latency while sustaining higher request loads. At B 256, again under iso-batch conditions, ThinKV achieves 38 % higher reqs/s and 27% % lower latency compared to R-KV.
NAW
DAY
User Ustkky
Figure 9: throughput
vLLM
system
versus latency comparison.
user-
6.3 DISCUSSIONS AND ABLATIONS
Impact of ThinKY Components. In Table we ablate the accuracy, throughput, and latency con- tributions of ThinKV's components on GPT-OSS 20B using LiveCodeBench. For a fair comparison, we employ an iso-batch comparison with batch size of 8. TBQ, operating at an average precision of 3.5 bits, maintains accuracy comparable to FullK However, as shown in Figure 10 10)d), its substan- tial generation-length inflation negates most of the compression gains, yielding only a modest 1.1x improvement in throughput.
TBE at smaller eviction budgets (e.g., 512) achieves large performance gains-up to 1.78x higher through- put and 0.36x lower latency- at the cost of notice- able accuracy loss. At larger eviction budgets, TBE approaches near-lossless accuracy while still provid- ing throughput improvements of up 1.48x. ThinKV (TBQ+TBE) combines both mechanisms, delivering strong compression with only marginal accuracy reduction. We would like to note that TBQ's
Table 4: Impact of ThinKV components accuracy, performance (iso-batch) for GPT-OSS-201 on LiveCodeBench.
Agg Roceo
BBsch
Num
Neem
Badget
Accuracy
Thouglps
Laccs
TSQ
155
775
11%
0:6x
TRE
512
62.5
1.76x
0.56
THE
2545
1.27*
(EQ-
3.8 1004
76.4
1.51%
645
average precision is lower than ThinKV's because its inflated generation length introduces more transition tokens. Importantly, ThinKV achieves up to 1.51x higher throughput and 0.42x lower latency by avoiding the severe generation-length inflation exhibited by TBQ (see Figure d)).
Thought-Adaptive vs. Token-Level Heuristics. To understand why ThinKV outperforms base- lines, we analyze average recall rate of tokens with Top-10 10 attention scores (Tang et al. 2024) on Rm---a Recall rate is the fraction of important tokens (Top-10) 10) preserved by compres- sion method relative to those under full attention at each decoding step. As shown in Figure 10(a), ThinKV sustains recall rates close to FullKV across token budgets compared to R-KV and LazyE- viction that rely token-level heuristics that overlook reasoning structure.
Compression Increases Generation Length. In Figure d0),), our RI R1-Llama- -BB results show that pure quantization can inflate generation length by up to 5.1x. In contrast, eviction- based approaches do not induce such drastic inflation. ThinKV (TBQ+TBE) inherits this desirable stabilizing behavior and avoids the severe length expansion seen in quantization-only baselines.

---

Published as a conference paper at ICLR 2026
TBQ Precision. In Figure [i),,, study the effect of quantizing R, T. and E thoughts different precisions for RI-Llama- 8B on AIME and RI-Llama-70B on LiveCodeBench, using the notation RxEyT2 with y/ 2 E B {2, 4, 8). We adopt R4E4T2 in all experiments due to its high accuracy and higher compression (also see D.3
ThinKV's eviction strategy enforces proactive eviction (coarse-grained, in con- trast to the fine-grained, stepwise eviction of H2O, R-KV. [igure b)]b) shows ThinKV's eviction behavior. As shown in Table 5 with ThinKV, the number of times layer performs eviction across decode steps is minimal, 4.59% compared to R-KV's 82.93% because R-KV waits for the budget be exceeded evict one token per decode step.
Eviction Behavior:
Overhead Analysis. In Table 5, we report operation-level breakdowns for RI-Llama-SB. The dequantization overhead of TBQ is included as part of the attention time. While TBE and thought re- fresh comprise ~ 14% of per-layer execution, their infrequent invocation ensures layers run overhead- free 95% of the time. Evidently, for R-KV, eviction and gather emerges as a major bottleneck (32.91%) since it is invoked in nearly every decoding step.
Table 5: Per-layer time breakdown (%) and call rates across decode steps.
Time
duf Calle (%)
Time Breakdow (%)
Pidf Calle (%)
Breakdown (%)
Thouulh Rainch
3.50
0.7
R&& Eviciun
$2.93 22. 65
10:46
Giatee Timee
0
THE Bstetionn
4.59
Attention
38:65
100
Refresh Rate. In Figure 10 c). we ablate different choices of refresh rate (T) for a GPT-OSS-20E model on LiveCodeBench. 128 offers the best trade-off between accuracy and over- head. Accuracy drops with larger 7 as it skips thought changes and reduces opportunities correct mispredictions.
& of
a Theugiss
22.2..
44S
10.0
76.4
Optimal of Layers. In Figure 11 a). we ablate different |C* for R1-Llama-8BB LiveCodeBenchh We select | as it best balances accuracy and efficiency. Using all layers ([-- 32) degrades accuracy ($3.1).
# of Thought Types. In Figure aa),, we show that [T| 3 yields the best accuracy on RI-Llama-SB evaluated on Live- CodeBench. For each (T), we select layers exhibiting [T] sparsity modes (can be less than [C' and quantize according to thought importance. When T < 3, there is no notion of trajectory-changing thoughts. Therefore, eviction occurs only upon exceeding the KV budget (case 2 in Problem Formulation 2). See $E.10 for generalization to LLMs with [7] =
AMM
(b)
UnsDbedtendh
Figure 11: (a) Impact of |C* [T] and min R on LiveCodeBench ac- curacy for RI-Llama-8B, (b) anal- ysis of precision assignment for RI-Llama-B 8B on AIME and RI- Llama-70B on LivchodeBench.
Minimum Token Retention. In Figure aa), we show why the minimum retention (R) per thought segment is set to 4. Complete eviction (min R 0) severely degrades accuracy, as the model loses track of reasoning trajectories and results in an endless reasoning loop. Retaining minimal subset min R- preserves semantic structure of reasoning and offers the best trade-off.
% Breakdown of Thoughts. Figure 10 shows the distribution of R, T, and E thoughts for R1- Llama-8B. Complex datasets (AIME) exhibit more transitions, than simpler ones (MATH-500).
Block Size. In Figure e0), we evaluate the effect of different block sizes on throughpur. Block sizes of 8-16 deliver the best performance. Larger blocks, however, may pack more thought seg- ments per block, incurring substantial metadata overhead in block table and increase eviction time.
CONCLUSION
We introduced ThinK thought-adaptive KV cache compression framework for LRMs. Exploiting attention sparsity, ThinKV decomposes chains of thought into reasoning, execution, and transition segments, enabling joint thought-aware quantization and eviction that sustains accuracy under high compression. On the system side, our Continuous Thinking kernel manages memory efficiently under dynamic decode-time eviction without costly compactions. This ceoorithmmsstten co-design delivers near-lossless accuracy with <5% of the original KV cache, while enabling up to 5.8x throughput gains and substantially larger batch sizes across diverse reasoning benchmarks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Reena Elangovan, Steve Dai, Yaosheng Fu, Zoey Song, Yingyan (Celine) Lin, Ben Keller, Andre Green, Souvik Kundu and Mingyu Lee for insightful discussions and valuable feedback that helped improve this work.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul Arora, Yu Bai, Bowen Baker. Haiming Bao, et al. gpt-oss- 120b & gpt-oss-20b model card. arXiv preprint Xrvi222008. 10925, 2025.
Eduardo Alvarez, Omri Almog, Eric Chung, Simon Layton, Dusan Stosic, Ronny Krashinsky, and Kyle Aubrey. Introducing NVFP4 for efficient and accurate low-precision inference. NVIDIA Technical Blog, June 2025.
Simon Aytes, Jinheon Baek, and Sung Hwang. Sketch-of-thought Efficient IIm reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05177 2025.
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context 1lms. arXiv preprint 0e:::2408000555 2024.
Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which Ilm reason- ing steps matter? arXiv preprint arXiv:2506. 191443336.111,,,.. 2025.
Zefan Cai, Yichi Zhang, Bofci Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyra- midal information funneling. arXiv preprint arXiv:2406.02069 2024.
Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li- Wen Chang, Jiuxiang Gu, et al. R-ky: Redundancy-aware kv cache compression for training-free reasoning models acceleration. arXiv preprint arXiv:2505.24113, 2025.
Chi-Chih Chang, Wei-Cheng Lin, Ch Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. arXiv preprint arXiv:2407.21118. 2024.
Kaiwen Chen, Xin Tan, Minchen Yu, and Hong Xu. Memshare: Memory efficient inference for large reasoning models through kv cache reuse. arXiv preprint arXiv:2507.21433, 2025a.
Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free. arXiv preprint arXiv:2504 07986, 2025b.
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, and Hua Wu. Nacl: A general and effective kv cache eviction frame- work for Ilms at inference time. arXiv preprint arXiv:2408.. 240.:555..03775 2024.
Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint ArXiv:2412.12: 13171,, 2024.
Wen Cheng, Shichen Dong, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for I1m kv cache. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 3542-2550, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ 10. 14168, 2021.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 2023.
11

---

Published as a conference paper at ICLR 2026
Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S Kevin Zhou. Ada-kv: Optimizing kv cache evie- tion by adaptive budget allocation for efficient Ilm inference. arXiv preprint 1rX5:2240..15550, 2024.
Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter:
A head-level kv cache compression method with integrated retrieval and reasoning. arXiv preprint arXiv:2410.
19258, 1922988. 2024.
Ravi Ghadia, Avinash Kumar, Gaurav Jain, Prashant Nair, and Poulami Das. Dialogue without lim- its: Constant-sized kv caches for extended responses in llms. arXiv preprint S.div:2503.00979, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501.12945 2025.
Insu Han, Zeliang Zhang, Zhiyuan Wang, Yifan Zhu, Susan Liang, Jiani Liu, Haiting Lin, Mingjie Zhao, Chenliang Xu, Kun Wan, et al. Calibquant: 1-bit kv cache quantization for multimodal Ilms. arXiv preprint arXiv:2502.1488% 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprini arXiv:24/2. 18547, 2024.
Jitai Hao, Yuke Zhu, Tian Wang, Jun Yu, Xin Xin, Bo Zheng, Zhaochun Ren, and Sheng Guo. Om- nikv: Dynamic context selection for efficient long-context 1lms. In The Thirteenth International Conference on Learning Representations, 2025.
Junhui He, Junna Xing, Nan Wang. Rui Xu, Shangyu Wu, Peng Zhou, Qiang Liu, Chun Jason Xue, and Qingan Li. A2ats: Retrieval-based kv cache reduction via windowed rotary position embedding and query-aware vector quantization. arXiv preprint arXiv:2502.12663 2025.
Coleman Hooper. Schoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun S Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length I1m inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270
2024.
Coleman Hooper, Sebastian Zhao, Luca Manolache, Sehoon Kim, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Multipole attention for efficient long context reasoning. arXiv preprint arXiv:2506. 13059, 2025.
Junhao Hu, Wenrui Huang, Weidong Wang, Zhenwen Li, Tiancheng Hu, Zhixia Liu, Xusheng Chen, Tao Xie, and Yizhou Shan. Raas: Reasoning-awar aware attention sparsity for efficient Ilm reasoning. arXiv preprint arXiv: 147, 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of 1lm. arXiv preprint arXiv:2403.05522 2024.
Lang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W Lee, Sangdoo Yun, and Hyun Oh Song. Kvzip: Query-agnostic kv cache compression with context reconstruction. arXiv preprint arXiv:2505.23416, 2025.
Martin Kruliš and Miroslav Kratochvíl. Detailed analysis and optimization of cuda k-means al- gorithm. In 49th International Conference on Parallel Processing ICPP, ICPP '20, New York, NY, USA. 2020. Association for Computing Machinery. ISBN 9781450388160. 160. doi:
10.11457300499.34444266 URL httpt:ttt//tt...t. /ttts:/:://doii /tttps://doi..or/1100/ 11tp://..1114334497.7.33333 3404337433777 3404444
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611 111662, 2023.
12

---

Published as a conference paper at ICLR 2026
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-229977 2024.
Zhong-Zhil Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang. Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. /7419, 2025.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2.Xiv:2305.20050 2023.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Infor- mation Processing Systems, 37:139997-140031, 2024a.
Tengxuan Liu, Shiyao Li, Jiayi Yang, Tianchen Zhao, Feng Zhou, Xiaohui Song, Guohao Dai, Shengen Yan, Huazhong Yang, and Yu Wang. Pm-kvq: Progressive mixed-precision kv cache quantization for long-cot Ilms. arXiv preprint arXiv:2505.1 18610, 2025.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang. Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for 11m kv cache compression at test time. Advances in Neural Information Processing Systems, 36:52342 -23664, 2023.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024b.
MAA. Aime 2024 problems. ottes://arroforoooeemaal////////
php/2024 AIME Problems 2024. Accessed: 2025-08-30.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
Piotr Nawrot, Adrian Lańcucki, Marcin Chochowski, David Tarjan, and Edoardo M Ponti. Dy- namic memory compression: Retrofitting 1lms for accelerated inference. arXiv preprint arXiv:2403.00666, 2024.
Nsight. Nvidia nsight systems. 2025. Accessed: 2025-09-12.
OpenAl. Openai ol. https::/openai.comt/ 2024.
OpenAl. Issue #2522: 2025. Accessed: 2025-09-21.
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathe- matical statistics, 165)-166 1962.
Akshat Ramachandran, Souvik Kundu, and Tushar Krishna. Clamp-vit: Contrastive data-free learn- ing for adaptive post-training quantization of vits. European Conference on Computer Vision, pp. 307-325. Springer, 2024a.
Akshat Ramachandran, Zishen Wan, Geonhwa Jeong, John Gustafson, and Tushar Krishna. Algorithm-hardwe co-design of distribution-awwree logarithmic-poss encodings for efficient dnn inference. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pp. 1 1-6, 2024b.
Akshat Ramachandran, Souvik Kundu, and Tushar Krishna. Microscopiq: Accelerating founda- tional models through outlier-aware microscaling quantization. In Proceedings of the 52nd An- nual International Symposium on Computer Architecture, pp. 199--1209, 2025a.
13

---

Published as a conference paper at ICLR 2026
Akshat Ramachandran, Souvik Kundu, Amab Raha, Shamik Kundu, Deepak K Mathaikutty, and Tushar Krishna. Accelerating Ilm inference with flexible n: M sparsity via a fully digital compute- in-memory accelerator. arXiv preprint arXiv:2504. 14365, 2025b.
Akshat Ramachandran, Mingyu Lee, Huan Xu, Souvik Kundu, and Tushar Krishna. Ouromamba: A data-free quantization framework for vision mamba. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 21 21177-21188 186 2025c.
Pol G Recasens, Ferran Agullo, Yue Zhu, Chen Wang, Eun Kyung Lee, Olivier Tardieu, Jordi Tor- res, and Josep Ll Berral. Mind the memory gap: Unveiling gpu bottlenecks in large-batch Ilm inference. arXiv preprint arXiv:2503.0831/ 2025.
Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rehecca Joseph, Shlok Natarajan, Etash Kumar Guha, et al. Intelligence per watt: Measuring intelligence efficiency of local ai. arXiv preprint arXiv:2511.07885, 2025.
Minju Seo, Jinheon Back, Seongyun Lee, and Sung Ju Hwang. Paper2code: Automating code generation from scientific papers in machine learning. arXiv preprint vrXiv:250..1712, 2025.
Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, and Minjia Zhang. Minikv: Pushing the limits of 2-bit kv cache via compression and system co-design for efficient long context inference. In Findings of the Association for Computational Linguisties: ACL 2025, pp. 18506-18523, 2025.
Dachuan Shi, Yonggan Fu, Xiangchi Yuan, Zhongzhi Yu, Haoran You, Sixu Li, Xin Dong, Jan Kautz, Pavlo Molchanov, et al. Lacache: Ladder-shaped kv caching for efficient long-context modeling of large language models. arXiv preprint 21V::2507..1204, 2025.
Jiwon Song, Dongwon Jo, Yulhwa Kim, and Jae-Joon Kim. Reasoning path compression: Com- pressing generation trajectories for efficient Ilm reasoning. arXiv preprint arXiv:2505. 2025.
Hanshi Sun, Li-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. Shadowkv: Kv cache in shadows for high-throughput long-context 1lm inference. arXiv preprint arXiv:2410.21466 2024.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference, arXiv preprint urXiv:2406. 10774, 2024.
Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. arXiv preprint a7X1112506.18167, 2025.
LLMM PR 16160. vilm pull request #16160. httpss///gtthubb.omm///immmmmp v vilm/pplui 111m/puuu/161600 2025. Accessed: 2025-02-18.
Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adap- tive kv cache merging for 1lms on long-contex: tasks. arXiv preprint arXiv:2407.08454 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of of-thuh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yonggi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502. 12067, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft thain-of-thhoglee for efficient reasoning with llms. arXiv preprint arXiv:2502.12 1214 2025.
14

---

Published as a conference paper at ICLR 2026
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Inftythink: Breaking the length limits of long-contex. reasoning in large language models. arXiv preprint arXiv:2503.06692 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Gyeong-In Yu, Joo Seong Jeong, Geo Kim, Soojeong Kim, and Byung-Gon Chun. Orea: A distributed serving system for (Transformer-Bassee generative models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502. 1089, 2025.
Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, and Song Guo. Lazyeviction: Lagged kv eviction with attention pattern observation for efficient long reasoning. arXiv preprint arXiv:2506.15969 2025a.
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502. 15589, 2025b.
Rongzhi Zhang. Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Yelong Shen. Lore: Low-rank compression for Ilms kv cache with a progressive compression strategy. arXiv preprint 11 2024a.
Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, and Anshumali Shrivastava. Kv cache is bit per chan- nel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37:3304-3331, 2024b.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances Neural Information Processing Systems, 36:34661-34710 2023.
Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Zhangyang Wang. Q- hitter: A better token oracle for efficient I1m inference via quantie- kv cache. Proceedings of Machine Learning and Systems, 6:381-394, 2024c.
Dawei Zhu, Xiyu Wei, Guangxiang Zhao, Wenhao Wu, Haosheng Zou, Junfeng Ran, Xun Wang, Lin Sun, Xiangzheng Zhang, and Sujian Li. Chain-of-thought matters: improving long-context language models with reasoning path supervision. arXiv preprint 2022:2250.. 2025.
APPENDIX
Overview of Mathematic Notation
17
B Extended Related Works
17
Supplementary Background
18
LRM Inference Stages
18
Attention Mechanisms
19
KV Permutation Invariance of Attention
19
Group Quantization
20
Paged Attention
21
15

---

Published as a conference paper at ICLR 2026
D
Supplementary Details on ThinKV
21
D.I
Thought Decomposition Calibration Process
D.2
Thought Keyword List
21
D.3
Quantization Data Formats
22
D.4
TBE Eviction Policy
22
D.5
ThinKV Pseudocode
22
D.6
V Walkthrough Example
23
Extended Eva Avaluations
24
Dataset Details: AIME
24
Evaluation Sctup Details
24
E.3
Visualization of Attention Maps
25
Attention Sparsity Plots
26
E.S
Pairwise Thought Association Maps
26
Results on MobileLI.MRRI 950M (GSM8K)
26
E,7
Results on GPT-OSS 120B (LiveCodeBench)
26
Ablation on Data Formats
26
E.9
Quantization Sensitivity Analysis
27
10
Generalization LLMs
27
E.
Pareto-front Analysis
27
12
Throughput Evaluation of ThinKV vLLM
27
13
Experiments on Qwen3 Models
28
E.
14
Latency Breakdown Across Batch Sizes
28
15
Time-per-Request Analysis
28
E.
16
Integration with SnapKV
28
17
LRM Example Reasoning Trace
29
L Limitations
34
Impact Statement
34
LLM Usage Statement
35
16

---

Published as a conference paper at ICLR 2026
Table 6: Summary of notation used in the paper.
Symbol
Description
A
Final answer produced after reasoning
L
Number of layers in the LRM
gi
Token generated at step
Y;
thought segment consisting of multiple discrete tokens
Yo....
Yx-1
Sequence of thought segments a CoT output
8
KV cache of layer after decoding step with associated thought type Retained KV cache of layer after eviction Key and value vectors token ys at layer & Quantized key and value representations Set of thought categories Sparsity thresholds separating thought categories Optimal subset of layers
S
(K) V
K. V
T
{co.... eT-1)
01. 0r-1
C
Refresh interval for thought categorization Set of available quantization precisions Importance score function for thought categories
B
(br
by
Mapping from thought types quantization precisions KV cache budget Eviction policy
π
R
Annealing schedule
A OVERVIEW OF MATHEMATIC NOTATION
Table 6 summarizes the key notations used throughout the paper.
B EXTENDED RELATED WORKS
Pre-LRM KV Cache Compression. As LLMs began to support increasingly long contexts, the KV cache emerged as a primary target for optimization. Early work primarily addressed long put put-context tasks by compressing the prefill KV cache. SnapKV (Li et al. (2024)) AdaKV (Feng) al. (2024)) and HeadKV (Fu al. (2024) prune tokens using attention statistics via feature clustering per-head budget allocation-wiis
PyramidKV (Cai et al. (2024) applies pyramidal strategy, preserving more tokens in lower layers and compressing higher ones. These methods
fectively reduce prompt memory but ill-suited for LRMs, where the challenge lies in managing long outputs. To manage cache growth during decoding, methods such as StreamingLLM (Xiao al. 2023 ScissorHands (Liu al. (2023) ), H2O (Zhang et al. (2023)) MorphKV (Ghadia al. (2025) ), and KIVI (Liu et al. (2024b)) reduce memory through attention sinks, probabilistic reten- tion, heavy-hitter selection, sliding windows, and uniform quantization, respectively. More recent works, including Q-Hitter (Zhang et al. (2024c)) and MiniKV (Sharma et al.) (2055)) demonstrate that eviction and quantization can be co-designed, pointing toward hybrid strategies that maximize compression and throughput. While effective for extending traditional LLM outputs, these decode- time approaches often degrade accuracy on LRMs. as strategies driven by token recency or uniform compression fail to capture the reasoning progression and token importance characteristic of LRMs.
Compression approaches generally fall into four (2025): (Zhng et al., (2023); Liu et al. all](22222 (2023)) quantization (Liu et al. (2024b); Hooper et al. (2024)), merging (Nawrot al. (2024); Wang et al. (2024): Liu et al. (2024a)) ) and low-rank decomposition
eviction (Li et (2024); Ghadia al.
KKan al. (2024); Sun et al. (2024)).
Eviction: StreamingI.LMM (Xiao al. (2023) retains fixed-size sliding window together with a few attention sink tokens. MorphKV (Ghadia et al. (2025)) maintains small set of recent tokens and selectively preserves older ones most correlated with the current context, providing constant-sine -ized caches suitable for extended responses. LaCache (Shi et al. (2025) introduces ladder-shaped KV cache that preserves early tokens in shallow layers and later tokens in deeper layers, combined with iterative compaction of older caches, thereby supporting continuous long-context generation.
Quantization: Several works reduce KV cache memory by lowering precision while keeping all tokens. KVQuant (Hooper al. 2024)) explores ultra-low precision by quantizing keys Pre-RoPF.
17

---

Published as a conference paper at ICLR 2026
applying sensitivity-awarc non-uniform formats, and mixing dense/sparse quantization. More ag- gressive approaches investigate 1-bit quantization: methods such as Coupled Quantization (CQ) (Zhang et al. (2024b)) exploit inter-channel correlations to encode KV states with just bit per channel, while calibration-based schemes (Han et al. 20255) introduce scaling and correction fac- tors preserve accuracy.
Merging: Several works compress by consolidating semantically similar tokens. MiniCache (Liu al. (2024b)) merges redundant prompt tokens into compact representations, while NACL (Chen et al. 2024)) prunes and merges tokens in a one-shot prefill step. These strategies reduce redundancy without per-step eviction but can blur token-level distinctions reasoning tasks.
Low-rank Decomposition: Several works compress KV caches by factorizing them into low-rank representations to reduce memory and transfer costs. GEAR (Kang al. (2024)) couples low-rank approximation with sparse correction to mitigate quantization errors. ShadowK (Sun et al. (2024) stores low-rank keys on the GPU while offloading values CPU, reconstructing minimal sparse KV blocks the fly. Other approaches such as LoRC (Zhang et al.] (2024a)) and Palu (Chang) jet al. (2024)) apply progressive or layer-sensitive low-rank factorization of KV matrices, often in combination with quantization, to cut cache size and accelerate attention.
Long Reasoning Compression. A complementary line of work focuses on compressing the reason- ing path rather than only the KV cache. Several approaches shorten
(CoT) the output level: TALE (Han et al. (2024) and SoT (Aytes et al. (2025)) guide models through prompt engineering generate more concise explanations, while TokenSkip (Xia al. (2025)) fine-tunes condensed CoT datasets to reduce redundancy multi-step reasoning. Other methods equip models with summarization capabilities, such InftyThink (Yan et al. (2025)) and LightThinker (Zhang eet al. (2025b)) which compress intermediate reasoning into summaries to save tokens. A differ- ent direction operates in latent space, with approaches like CCoT (Cheng & Van Durme (2024)) and SoftCoT (Xu et al. (2025) enabling reasoning directly on compressed interna representations rather than verbose token sequences. Most recently, RPC (Song et al. (2025)) adaptively prunes, merges, or reorders reasoning trajectories while preserving correctness.
System-Level Optimizations. System-level methods complement algorithmic compression by managing KV storage at runtime. Quest (Tang et al. (2024) loads only query-relevant KV pages, while OmniKV (Hao et al. (2025)) streams KV from CPU in small chunks to reduce GPU mem- ory pressure-thouge both retain O(N) complexity in sequence length N. MiniKV (Sharma et al. (2025) introduces FlashAttention conttiion-ommtbbleee kernels for compressed KV, and Q-Hitter (Zhang et al. 2024c)) unifies eviction and quantization reduce GPU I/O overhead. H2O (Zhang et (202)) and KVZip (Kim et al. (2025) avoid costly gather operations with ring-buffered caches, while MemShare (Chen al. )025a)) enables block-level KV reuse across reasoning segments.
C SUPPLEMENTARY BACKGROUND
C.I LRM INFERENCE STAGES
The inference process of an L-layer LRM proceeds two distinct phases: the prefill stage, which processes the input prompt, and the decode stage, which generates the output autoregressively. These phases differ fundamentally in their parallelism and computational bottlenecks.
Prefill. Given a prompt of length (prmpt, the model embeds the input into hidden representations X p xd where b is the batch size and d the hidden dimension. For each layer e, keys and values are computed as
=x=XW K: Xv =W
with We We E Rddd denoting the projection matrices. The resulting KV tensors K.
{(K5, V2)
1pomn -1 stored in Sye.. for subsequent use. Since all prompt tokens are processed in j=
parallel, the prefill stage is dominated by quadratic attention cost in (prompt and is typically latency- bound.
Decode. Once the cache has been initialized, generation proceeds autoregressively. At decode step i, the current token embedding yi produces
ys W Vi =WWW =
18

---

Published as a conference paper at ICLR 2026
which are appended to the existing cache:
S + S Uf(K v()).
Attention is then computed against all cached keys:
(g((Kb))))
A = softmax
Of = AVV
Vd
This process repeats for Igen output tokens. Unlike preAll, decoding reuses the cache and extends one token time, making the stage inherently throughput-bound due to repeated KV cache lookups and memory traffic.
In summary, prefill amortizes computation across the entire prompt to initialize the cache, while decode iteratively expands the cache to produce the final output sequence.
C.2 ATTENTION MECHANISMS
We briefly summarize two widely adopted attention variants: Multi-Head Attention (MHIA) and Grouped-Query Attention (GQA). ThinKV is applicable to both attention variants.
Multi-Head Attention (MHA). In the autoregressive setting, each decode step produces a single query vector 9b, R Ixd for head h, which attends over the stored key vectors Kn E gg from the n past tokens. The attention matrix given by,
4h Kr Vd
ah = softmax
E Rllx
(1)
The attention weights are then applied to the value states V/, E Raxdd and the outputs from all heads are concatenated and projected back to the hidden dimension. For sparsity analysis, attention scores are averaged across all heads.
Grouped-Query Attention (GQA). In GQA, several query heads share a common set of key and value states. For head group indexed by h. the cached keys and values are (Ka, V%.) E Raxd G--1 produced within the group. The attention score for while G distinct query vectors g query head g is given by
Kh.ghh R
0h.9
Vd
These per-query matrices are aggregated element-wise across the group using max pooling:
group
maxpool (at.o,.((i..,....... Gn.G-1) R1
Finally, the consolidated scores are renormalized along the key dimension to obtain the final atten- tion weight as, for the group,
an = softmax (a group Ixn.
(4)
For sparsity analysis, attention scores are averaged across groups.
C.3 KV PERMUTATION INVARIANCE OF ATTENTION
Theorem (KV Permutation Invariance of Attention). Given Rixd K E Raxd V E Raxd define
softmax
9 Vd
V E Iglxd
For any permutation matrix IT E
R
softmax
g(T) Jd
(IVV softmax
ak Va
V.
Proof. Let = 1
va T E R Rxn, Since II is a permutation matrix, II II = I. and for any
[2lxn we have
softmax(ull") softmax(u) (Equivariance Property)
19

---

Published as a conference paper at ICLR 2026
Algorithm 1: Calibration Process for Thought Decomposition
Input:
Pre-trained LRM M with L layers, calibration dataset of prompts, number of thought types T, optimal number of layers !
2: Output: Optimal layer subset L* sparsity threshold set + -.,, 1TII-
3: Initialize Ue for each layer
4: for each prompt E D do
5: Run M p and generate sequence of length Mp
6: for each decoding step t E [Mp] do
7: for each layer & [L] do
8:
Compute sparsity u from attention scores
9: Append Mi [[]
10: end for
end for
12: end for
Initialize L
14: for each prompt do
Initialize L" [)] <-
16:
for each layer 8 do
17:
Apply KDE (x))
Mh _ K ((T sm [p
18:
Estimate modes 52(") 19: [94g] = T then
{z (z) 0, fM (x) < 0}
20: Add & L"[p]
21: end if
22: end for
23: end for
24: L - [x]]
25: for each layer E L do
26: for each prompt E [P] do
27: Identify local minima of the KDE and record thresholds (0(e.) ....., ((p)))
end for
29: end for
30: Compute final thresholds 0, = TE-T EiEc Epsi ((()) [] 1]
31: return L" (0........ 1
Applying this with w yields
softmax La ((ILK))) (uv) (ui)) softmax (sll (uV)
(softmax( 8) (s) (IIV)
softmax( $ (II HIV
softmax(s)
Remark. The same invariance holds for GQA: for any group with shared (K,, Vh), a joint permu- tation of their rows leaves the group attention output unchanged. Remark. This permutation invariance explains why ThinKV avoid reordering the KV cache during attention computation.
C.4 GROUP QUANTIZATION
Group quantization Ramachandran et al. (205cc) reduces precision by partitioning tensors into fixed- size groups and sharing scale (and optionally zero-point) within each group. Given a tensor e R and group size g, the entries are divided into groups XG, of length g. Each group is quantized
20

---

Published as a conference paper at ICLR 2026
V cuche
0
1
2
3
4
5
8
9
10
12
13
IS
K cache
2
3
4
10
12
13
14
15
Perform K-means clustering keys
10*14
12
15
7 10
10%
Retain key centroids and conrepooning values
Figure 12: Illustration of eviction policy n's k-means-based eviction mechanism.
Table Keyword list interpret different thought types.
Reasoning
Think, Approach, Remember, Find, Okay, Supposc, Verify Wait, Hmm, Wait no, Alternativcly, But wait, Earlicr I said that
Transition
Execution
Now, The steps are, Mathematical equations, Code syntax
XG.
max (X.)) 1
G round
A
where b is the target bit-width and A; is the group-specific scale.
Smaller group sizes yield tighter ranges and lower error, while larger groups reduce metadata over- head. Group quantization thus provides flexible trade-off between accuracy and efficiency, and serves the default scheme for low-bit KV cache quantization LRMss.
C.5 PAGED ATTENTION
PagedAttention is attention algorithm introduced in vLLM to address the inefficiencies of manag- ing key-value (KV) cache memory during large language model serving. Traditional systems store each request's KV cache in contiguous memory, leading tion as output lengths vary, and preventing memory sharing across sequences. Inspired by virtual memory paging. PagedAttention partitions the KV cache into fixed-size blocks that can be stored non-contiguously in GPU memory. Logical blocks are dynamically mapped to physical blocks through block tables.
internal and external fragmenta-
D SUPPLEMENTARY DETAILS ON THINKV
D.1 THOUGHT DECOMPOSITION CALIBRATION PROCESS
Algorithm depiecs the algorithm for the offline calibration stage. This process estimates the spar- sity thresholds that separate different thought categories by analyzing layer-wise attention sparsity distributions over calibration dataset.
D.2 THOUGHT KEYWORD LIST
To aid interpretation of sparsity regions, provide representative keywords for the three thought types in Table 7 These keywords are illustrative and only serve to map sparsity regions to reasoning, execution, and transition thoughts. They are not used for thought identification during inference.
21

---

Published as a conference paper at ICLR 2026
D.3 QUANTIZATION DATA FORMATS
We employ three element formats of different precision levels:
FP8 (E4M3). This is 8-bit floating-point format with 1 sign bit, exponent bits, and mantissa bits. It provides a balance between dynamic range and accuracy and serves as the highest-precision option for thought-adaptive quantization, used primarily for reasoning tokens. This format only uses a per-tensor FP32 scale factor.
NVFP4. NVIDIA's recently introduced 4-bit floating-point format, NVFP4 (Alvarez et al. 2025). combines sign bit, 2 exponent bits, and mantissa bit optimized for inference workloads. NVFP4 employs a group-wist wise scale factor (Ramachandran et al. 2025a) with FP8 (E4M3) representation and a group size of 16. Execution and reasoning tokens are stored in NVFP4 to reduce memory footprint while retaining sufficient accuracy.
Ternary (2-bit). This format encodes each element with two bits, covering three distinct values (-1,0,+1). Of the four possible codes, one corresponds -0, which is redundant and simply mapped to O.Similar to above, ternary also employs group-wise scale factor with FP8 (E4M3) representation and a group size 16. In our design, ternary quantization is applied exclusively to transition thoughts, where lower precision can be tolerated with minimal impact overall accuracy.
Together, these formats enable a precision hierarchy (FP8 NVFP4 > Ternary) aligned with the observed importance of reasoning, execution, and transition thoughts.
D.4 TBE EVICTION POLICY
Figure Ijillsstrates the K-means eviction process. When thought segment is selected for eviction, target number of groups, determined by the anneal- ing schedule R. Each cluster is replaced by its centroid key, and the corresponding value entry is retained. As shown, color-coded blocks indicate tokens that are close in the embedding space; cen- troids (marked with star) are selected from each cluster, and only these representative key-value pairs are preserved in the cache.
we cluster the post- key embeddings
While prior work (Hooper et al. 2025) has highlighted that RoPE can induce token drift, thereby complicating the clustering of keys. we observe that this effect negligible when clustering is restricted tokens within single thought segment. Each thought segment spans only 128 tokens, and the limited span ensures that RoPE-induced drift remains minimal, in contrast to clustering over the entire chain of thought (CoT) as done in (Hooper et al.. (2025), where the drift accumulates more substantially. Furthermore, if future evidence suggests that drift becomes noticeable even within thought segment, the Windowed RoPE strategy (He al.) 2025) can be readily employed complementary technique to mitigate this issue.
D.5 THINKV PSEUDOCODE
def generation_lon _oopptoinn_ooppoooeoo (prompt, max_gen_len, I, params) :
# Prolegue
knit_block_tables ()
thresholds (theta_low, theta_high
refresh perriid_pprriiddee harams.refres
group_size params. larramms.roupssiie
budget params. budget
Generate
for i in range (max_gen________mm gen_len) for 1 in range (I) Forward attention
q, k_fp, v_fp project_gkv((hID)
# Thought refresh: O=transition, 1=execution, 2=reasoning if i & refresh_pericn -- 0:
spars measure_sparsity())
22

---

Published as a conference paper at ICLR 2026
prev thought [1] thought [1]
thought classify (spars, thresholds)
A TBQ: group quantizatien
buffer_add k_fp, v_fp)
if buffer_size (1) grcup_size:
k_grp, v_grp buffer_take (1, group_size)
it thought [1] == 2:
kg, vq Q4 (Kgg v_grp)
f NVF24
elif thought == 1:
kg, vq
(k_grp, v_grp)
NVFP4
else:
kg, vq Q2 (k_grp, v_grp) ternary kv cacce wy_cache_update update (1, kq, vq)
# TEE: anneal at end each transition segrent if $ cefresh_period = and prev_ prov_thought [1] == 0:
prev_sogments
Lind__sements__effrr (1, step=i) for seg in prev_segments:
t seg.type
keep anneal_size(t)
ids = mark evictevic, seg, ids)
(1, seg, keep)
TBE: budget- eviction if kv_size (1) budget:
candidates active_thoughtttyes (1)
t argmin_importance (candidates) oldest keep snneal_size(ts
wwt__ccldeessseleeeeeee (1, t)
ids = Emeans_select (1, oldest, keep) mark_evictedd,, oldest, ids)
Attention computation h [1+1] attend K(1], V[1))
Epi logue
decode_tokens ()
Listing 1: ThinKV generation loop.
D.6 THINKV WALKTHROUGH EXAMPLE
We provide a detailed walkthrough of ThinKV using the illustration in Figure 6
TBQ Quantization. During decoding, tokens are first appended to B in full precision. Once the group size is reached, they undergo group quantization. In the illustration, we highlight reasoning important to note that the block table
(R) tokens, which quantized into the NVFP4 format. indexes only quantized tokens i.e., the block table updates at group-size granularity.
Step a. Following quantization, CT kernel queries the block table to determine whether a physical block of type-2 (reasoning) tokens has available capacity. Since the table is initially empty, new entry created with thought type 2, and a physical block is allocated. The start index of this reasoning segment is recorded as O. Because the block currently stores only single segment, the segment mask is initialized to all 1s, while the eviction mask remains all Os.
Step b. When token 'D' is generated, refresh occurs, switching to type-1 (execution) thought. Execution tokens are likewise group quantized to NVFP4. CT then allocates a new entry for the execution thought type. Importantly, CT enforces thought-aware paging: execution tokens never placed into partially filled blocks of other thoughts, even if capacity remains.
Step Beginning with token ', the decode refreshes to type-O (transition) tokens. As defined in $4.3) the end of this transition segment (the 'L' token) triggers the TBE kerne.. The kernel scans the block table, identifies all prior segments via their start indices, and applies the eviction pol-
23

---

Published as a conference paper at ICLR 2026
Podduces
Reasoning Thoughts
Den Notte
Execution Thoughts
can Poter
Transition Thoughts
Figure 13:
Visualization of attention maps across different thought types. At decode time only single query is present; maps are broadcasted for clarity of visualization.
icy. Instead of physically removing tokens, the eviction mask is updated to mark evicted positions, deferring eviction.
Step d. After the next refresh, decoding returns to reasoning. CT inspects the eviction mask to identify available slots in existing reasoning blocks. For tokens 'M' and 'N', it locates two free slots in physical block 4, places the tokens there, and resets the eviction mask to all Os once the slots are filled. In parallel, it appends the start index of the new reasoning segment and updates the segment mask to indicate the token positions for each segment. By reusing evicted slots in this way, ThinKV achieves efficient memory utilization without introducing additional HBM bandwidth pressure or stalling the inference critical path. For tokens 'O' and 'p' since there no empty slots available, new block is allocated.
E EXTENDED EVALUATIONS
E.1 DATASET DETAILS: AIME
Following Cai et al. (202); Liu et al.) (2025), we construct AIME benchmark of 30 prompts, comprising prompts sampled from AIME 2024 and 15 from AIME 2025.
E.2 EVALUATION SETUP DETAILS
We use the latest model checkpoints available on Hugging Face for all evaluations. We build on the Hugging Face Transformers codebase and implement the ThinKV algorithm modifying it. The Hugging Face Transformers codebase employs the FlashAttention-22 kernel as its default at- tention backend, which leverage for all baseline comparisons. In addition, modify Triton
24

---

Published as a conference paper at ICLR 2026
RI-L.lamn-de en AIME
GP GPT-0SS-2088 on AIME
RI I--Inmo--8 on Live RedeRench
Layer 12
Lager
Tayer
BenCachBBench
RI RL-Q LAB on AIME
RI-Ques- 148 on EiveClideeeencc
Laver 1.5
Layer
Deeode Ser
2003
GIT SETT-OSS-AO 20B (Lland to distinguish cases) un AIME
Layer
Layar
LACK
1R0 22,00
10V00
Figure 14: Layer-wise attention sparsity across decode steps for different models and datasets.
Figure 15: Additional visualization of pairwise thought associations for different input prompts from different datasets (AIME and LiveCodeBench).
implementation of Paged/Attention and integrate it into the Hugging Face Transformers stack; this baseline PagedAttention supports all features present vLLM's implementation. This integration was carried out as proof of concept to quickly evaluate ThinKV's performance. This proof-of- concept serves as a stepping stone toward full integration with optimized inference engines. Al- still expect commensurate improvements when
though this stack is not the most optimized, running on frameworks such as vLLM, as ThinKV's modifications are orthogonal to specific ker- nel implementations. To validate this, integrate ThinKV inspired by this PR in vLLM vLLM PR 16160 (205).. Our integration targets only the vLLM version. Specifically, our major modi- fications centered around "block_table.pp flags in 'envs.py', we can seamlessly toggle between R-KV, ThinKV, and a no-compression (Full- KV) baseline, enabling comparisons within the same vLLM framework.
'flash attn. attn.py' and 'esrc/attention'. By adjusting the
For measuring gather overhead, we profile this behavior on A100 and H200 GPUs using NVIDIA Nsight (Nsight 2025)
E.3 VISUALIZATION OF ATTENTION MAPS
Figure 13 shows the attention weight matrices different decoding steps, each corresponding to single query. The visualization reveals that transition thoughts exhibit the highest sparsity, followed by reasoning, and then execution.
25

---

Published as a conference paper at ICLR 2026
Quart pasonn Seroiwity Nessoning Thoughts
Ouarttation Seralivity Execction Thoughts
Viseo
Goartrattirr 003
Quandbarinn b8s
Ouerttattonn
(a)
(b)
Figure 16: Quantization sensitivity analysis of KV cache for (a) reasoning, (b) execution and (c)
transition thoughts.
E.4 ATTENTION SPARSITY PLOTS
we present attention sparsity across decode steps for several model families. For GPT- OSS-20B in particular, we highlight layers where the sparsity structure is difficult to distinguish, leading to ambiguous or poorly defined boundaries between thought categories.
In Figure 14
E.5 PAIRWISE THOUGITT ASSOCIATION MAPS
In Figure 13 we show the inter-thought dynamics for additional prompts drawn from AIME and LiveCodeBench.
E.6 RESULTS ON MOBILELLM-RI 950M (GSM8K)
Table 8: Comparison of ThinKV
For GSM8K, we set the KV cache budget 256 tokens for an average generation length of 1500. Under this set- ting. ThinKV operates at an average precision of 3.9 bits and achieves a 24x compression ratio while maintaining accuracy comparable to R-KV, which compresses at only 6x. This demonstrates ThinKV's effectiveness in sustaining reasoning quality under high compression even for lightweight models such MobileLLM-RI 950M.
and R-KV on GSM8K using
Mohilel.LMRR1-9500 Method Compression
GSMSK
FullKV
1
67.5
R-KV
6
60.8
ThinKV
24
60.1
E.7 RESULTS ON GPT-OSS 120B (LIVECODEBENCI))
Table 9: Accuracy of ThinKV
We evaluate ThinKV on GPT-OSS 120B using Live- CodeBench under a fixed KV budget of & 1024 tokens. GPT-OSS exposes reasoning effort knob (low/meiiumhhigh) that controls the model's ss reasoning budget; we sweep medium and high settings our study. Across both effort levels, ThinKV tracks FullKV closely: at high effort, ThinKV attains 67.5 vs. 69.4 for FullKV (-1.9 points); at medium, 59.3 vs. 61.8 (-2.5 points). Higher effort predictably yields better ac- curacy but longer generations, increasing KV stress; ThinKV sustains accuracy under this regime despite the the 1024-token cache. Across both reasoning efforts ThinKV maintains average pre- cision of 3.6-bits.
vs FullKV across reasoning effort levels for GPT-OSS- 120B on Live- CodeBench.
Method Reasoning Effort Accuracy
FullKV
High
69.4
ThinKV
High
67.5
FullKV
Medium
61.8
ThinkV
Medium
59.3
E.8 ABLATION ON DATA FORMATS
We further investigate the impact of different data formats on ThinKV. Specifically, we ablate the use of conventional integer quantization, where we employ INT4 and INT2 rep-
Table 10: Impact of data format
choices accuracy for R1-Llama- 8B.
resentations with same scaling as described in $D.3 This
Method
AIME
LiveCodeBench
allows to isolate the effect of the number representation from the scaling strategy. As shown in (Table 10 ThinKV
Baseline
32.14
ThinK w/ INT
46.7
28.5
with INT4/IN22 suffers notable accuracy degradation on both
ThinKV
32.14
AIME and LiveCodeBench. This demonstrates the combination of NVFP4 and ternary data format the better choice.
26

---

Published as a conference paper at ICLR 2026
E.9 QUANTIZATION SENSITIVITY ANALYSIS
Following Cheng et al. 2025), we analyze the quantization sensitivity of the KV cache across rea- soning, execution, and transition thoughts in Figure 16 Using INT quantization on RI-Llama-70B (LiveCodeBench), we sweep the precision of either K or V within a single thought type while fixing all remaining KV entries 8-bit. The results show that transition thoughts are highly bobust-both K and V tolerate aggressive similarly remain stable down to 4 bits. In contrast, the K cache of Reasoning thoughts is significantly more sensitive, consistent with the K/V asymmetry observed in Cheng et al. (2025), while the cor- responding V cache remains resilient. These findings directly validate the precision assignments adopted in ThinKV.
Supporrrin our use of 2-bit precision. Execution thoughts
E.10 GENERALIZATION TO LLMS
To evaluate ThinKV's generalizability beyond LRMs, we test it on the long-response benchmark LongWriter (Bai et al.) 2024). which includes 60 prompts across domains such as emails, blogs, essays, and novels, with response lengths rang-
Table 11: LLM accuracy compari-
on Long NongWriter task.
Method
Llama-8B
Phi-14B
FullKV
66.5
62.9
ing from 100 to 12K words. Following Zhang et al. (2023). constrain the KV cache budget to 5% of decode tokens.
H20 (5%)
68.1
61.5
ThinKV (3.75%)
67.9
63,8
with B = 4, treating all tokens as a single category. In this setting, eviction occurs only when the cache budget is reached, after which prior tokens are annealed to the nearest power of two. For evaluation, we follow Ghadia et al. (2025) and use an LLLMbbaed judge (Mistral-Large- 123B) to score responses across multiple criteria. As shown in Table 11] ThinKV generalizes effectively to LLMs, matching or even surpassing H2O while delivering higher compression through its hybrid scheme.
Unlike LRMs. LLMs do not exhibit distinct thought types; hence, we set [T|
E.11 PARETO-FRONT ANALYSIS
Figure 17 illustrates the relationship between KV-cache size and accuracy across several SoTA compression and eviction baselines for R1-Llama-70B on LiveCodeBench. For this analysis, inspired by (Sharma al., 2025), we sweep differ- ent configurations (token budget, quantization precision) for each of the evaluated methods. Methods such as Laittiin,oon, PM-KVQ, and R-KV achieve moderate compression but suffer significant accuracy degradation, while high-accuracy config- urations require substantially larger KV budgets. In contrast, ThinKV consistently delivers F accuracy at dra- matically smaller KV-cache sizes, tracing dominant curve that establishes the new Pareto frontier. Specifically, most ThinKV configurations lie strictly above competing methods at equivalent or smaller memory footprints. This frontier shift highlights ThinKV's ability to achieve the best possible trade- off between accuracy and memory, outperforming both quantization-onie and eviction-only ap- proaches and confirming its strong scalability across compression regimes.
Jecurny (50)
Cat fucto
10
Figure 17: Accuracy vs KV cache size comparison of ThinKV against SoTA baselines for RI- Llama-70B on LiveCodeBench.
E. THROUGHPUT EVALUATION OF THINKV IN VLLMM
As shown in Table 12 we report throughput under two iso-batch comparisons: (i) batch size = 8 against Ful-
Table 12: Throughput comparison un-
der different batch sizes implemented in
IKV and R-KV (ovl), and (ii) batch size 256 against R-
vLLM. Method
KV (ovl). All methods have been implemented vLLM
Batch Size
Budget Throughput
for fair comparison and measurements conducted on A100-80GB GPU. At a batch size of 8, ThinKY delivers higher throughput than both FullKV and R-KV (ovl), im-
FulKKV
228.3
R-KV (ovl)
1024
331.9
ThinKy
8
1024
346.9
R-KV
256
1024
4883.3
proving over FullKV by more than 50%. At larger batch
ThinkY
256
1024
6622.4
size of 256, ThinKV's advantage becomes more pronounced: it achieves substantial throughput
27

---

Published as a conference paper at ICLR 2026
increase over R-KV (ovl) of up to 1.35x. ThinKV demonstrates superior scalability by eliminat- ing gather- sather-based compaction and achieving higher KV-cache compression, both which translate directly into faster model execution.
E. EXPERIMENTS ON QWEN3 MODELS
The Qwen3 model family YYang et al. 2025 enables seam- less switching between thinking and non-thinking modes via flags. Using representative Qwen3-8B model, we compare its non-thinking mode against ThinKV-enabled thinking mode, ThinKV achieves 2.2% accuracy drop across eviction budgets while using 6.87% of FullKV memory. In contrast, the non-thinking mode exhibits a drastic > 33% accuracy degradation. This highlights that aassoing--augmette decoding is essential for correctness.
Table 13: Accuracy comparison
between thinking, non-thinking, and ThinkV-enabled thinking modes
on Qwen3-8B evaluated Live- CodeBench.
Avg. Prodision
Method
Mode
Eviction Budget Accuracy (%)
FulKyY
Non Nainkiin
21.3
FullKY
Thinking
55.6
ThinKV
Thinking
36/ 1024
53.4
ThinKV
Thinking
E.14 LATRNCY BREAKDOWN ACROSS BATCH SIZES
This experiment is conducted to better understand how the per- formance of ThinKV's components scale across batch sizes. For this analysis, focus cludes mechanisms in action. Figure 18 measurements show that ThinKV's overhead (TBE eviction + thought refresh) remains minimal across batch sizes, consistently accounting for only ~ 14% of the total latency, while Attention and MLP operations dominate with more than 80-85% of the runtime. As batch size increases, the proportion of time spent in core model execution (attention, MLP) grows, confirming that ThinKV scales efficiently with increasing batch size.
Thaight lattch
representative decode step that in-
12.3
413
Sae
Figure 18: Latency break- down across different hatch sizes.
E.15 TIME-PER- REQUEST ANALYSIS
MIN-PERRRREQEST
Table 14 reports the average end-to-end request la- TPR), accuracy, and Intel- ligence/Watt (Saad-Falcon et al. 2025) for various K KV-compression strategies evaluated the AIME benchmark using R R1-Llama-8BB. ThinKV at a to- ken budget of 1024, while simultaneously achieving lossless compression, is able to achieve up to 6% lower latency on average per request as compared to the FullKV baseline. These gains extend beyond what highly optimized framework like vLLM al- ready provides, and ThinKV's benefits become es- pecially pronounced at larger batch sizes. Recent works have demonstrated that Intelligence/Want (Saad-Falcon et al. 2025) offers unified view of both capability and efficiency, making it prin- cipled metric for comparing compression strategies. As shown in Table 14] these latency improve- ments materially increase ThinK V's Intelligence/Watt over FullK and R-KV.
Table 14: Comparison of per-eeuue-RRttt
tency
(TPR), Accuracy, and Intelligence/ /aatiggenc//Wttttt (In-
tel./Wtt).
Method
Token Bodest
TPR (+)
Accuracy (%)
Intel/Waitt
FollKY
259.6
50.0
0.20
(seq)
R-KV (ovl)
512
240.8
40.0
0.17
ThinkV
512
237.5
46.7
0.21
(sog)
0.20
R-KV (ov))
1024
246.0
46.7
0.20
ThinKV
1024
243.6
50.0
0.22
(seg)
R-KV (ov))
2048
253.7
50.0
0.20
ThinkV
2048
251.0
50.0
0.21
E. INTEGRATION WITH SNAPKV
ThinKV is orthogonal to other prefill-optimization tech- niques, and can be combined with them seamlessly. To il- lustrate this, we evaluate ThinKV integrated with SnapKV on LongBench v2 ?. LongBench v2 is suite of long- context evaluation tasks characterized by both large input contexts and substantial output lengths. For this study, evaluate on RI-LLama-8B and select the Code Repo QA task from Bengh v2, which features 16K-32K average input lengths and output sequences up
Table 15: Ablation of Prefill and De- code Settings for Hybrid (SnapKV ThinKV) compression.
Priill
Madall
Eviction Budges
Eviction Buslest Ascuracy
Panky
10I
ThinkV
4-bi 4 2048
311513
1k
SnapKV Thinky
377/51
Snapky Thinky
has 4096
344/512
28
28

---

Published as a conference paper at ICLR 2026
to 8K tokens, making it representative long-input/lon--ouppe setting. Prefill tokens are quan- tized to bits using TBQ and evicted according to SnapKV's eviction policy under varying budgets, while the decode stage employs ThinKV with fixed KV-cache budget of 512 tokens. Our results are summarized in (Table 15) ThinKV incurs only a small accuracy drop, despite achieving 4x pre- fill compression and up to 66x decode compression. When combined with SnapK 4096 prefill token budget results in only 2% accuracy degradation.
E.17 LRM EXAMPLE REASONING TRACE
Let ABCD be a rhombus whose vertices all lie on the hyperbola and are in that order. If its diagonals intersect at the origin, find the largest number less than BD for all rhombuses ABCD.
Input Prompt 1:
a² 20
2 24
Annotated GPT-OSS-20B Model Output: (Reasoning Thought) "We have rhombus ABCD with vertices on hyperbola x 2/20 y 2/244 = 1. Diagonals intersect at origin. So origin is intersection of diagonals, meaning the diagonals are perpendicular? In rhombus, diagonals are perpendicular only if it's a square? Wait rhombus: diagonals perpendicular if it's a kite? Actually in rhombus diagonals are perpendicular only if rhombus is square? Let's recall: In rhombus, diagonals are perpendicular if rhombus is a kite? So a rhombus is a kite with equal sides. For a rhombus, diagonals are perpendicular if it's a square? No, square diagonals are perpendicular? In square, diagonals are perpendicular? Actually square diagonals are perpendicular? Yes, square diagonals are perpendicular? Are square diagonals are perpendicular? In square, diagonals are perpendic- ular? Yes they are perpendicular? Let's 's check:
In square, diagonals are perpendicular? Actually in square, diagonals intersect right angles? I think yes they are perpendicular? Wait think: In a square, the diagonals are perpendicular? For square, the diagonals are perpendicular? I recall that in a square, diagonals are perpendicular? Let's compute: square of side s, diagonals length 2x/2. The angle between diagonals is 90 degrees? In a square, the diagonals are perpendicular? Let's 's draw; coordinates (s,)),s,O)sssss), s). Di- agonals: (0,0)-(s,s) line slope 1; other diagonal (s,0)-(0,s) slope -I. They are perpendicular, So yes. X An InBut rhombus general: diagonals are perpendicular only if rhombus is a square? Ac- tually rhombus with all sides equal but not necessarily right angles. In general rhombus, diagonals are perpendicular if rhombus is a kite? There is also a rhombus where diagonals are perpendicular but not equal sides?
Transsiioon Thought) Wait property: In a rhombus, diagonals are perpendicular iff rhombus is square? Let's confirm: In rhombus, diagonals are perpendicular if rhombus is a kite? Actu- ally a kite has perpendicular diagonals. But rhombus is special kite with equal sides. For rhombus, diagonals are perpendicular only if rhombus is a square? Let's test: Consider rhom- bus with sides length 1, angles 60 and 120: diagonals lengths: d1 = 2- (=]]cos(30)=² Wait for- mula: In rhombus with side a, angle \ (u003b8 between adjacent sides. Diagonals lengths: d1 a /2{22ccos') { (u03b8)= 22 cos(', ( (j0338822). d2 aa((2-cos'), (2cosss(u0030 (u03b88) = 2a sin( { (u03b8/22. They are perpendicular if product of slopes? Actually if dd2d For perpendicular, product of slopes? Wait slope of dl relative to d2? In rhombus,
(Execution Thought)diagonal are perpendicular if d1'2 + d22 4a Wait maybe property: In rhombus, diagonals are perpendicular iff rhombus is a square? Let's check: For rhombus with d2=2a sin(30)=a. Are they perpendicular? In a square, M0J00b8=990, di=d2=a,f2 Are they perpendicular? Let's compute slopes: For VD 1u033b==00r, coordinates: place A at (0,0), B at (a,0). Next side AB to BC with angle 60: C= B (a cos60, a sin60) (a + a/2, at (3/2). D A (a cos60, a sin60) (a/2, at (3/2). Diagonals: AC from (0,0) to (a (a++o a/2, a* (3/2) slope = (a* /3/2)/a+ a/2)= (332V332)==333 BD from (a,0) to (a/2, a* (3/2) slope = (a* 432- OV/a/aaa222-2 a) (33/2M(-a22== 33. Product slopes -3? Not -1. So not perpendicular So
Nu03b8=60r, dl=2a cos(30)=2a f3/2)=a..3 =0x
rhombus with M0u03b8=60 not perpendicular: So only square? Let's test rhombus with AD MDu03b8=90; square. Diagonals slopes 1 and -1, product -I. So yes. (on)) (nThus rhombus with diagonals perpendicular is a square. But we don't 't require perpendicular. We just need inter- section at origin. So origin is intersection of diagonals; rhombus centered at origin. So vertices
29

---

Published as a conference paper at ICLR 2026
are opposite pairs symmetrical about origin. So if A at (xl.yl), then C (-x1,-y). Similarly B at (x2,y2), D at (-x2,-y2). All four points lie on hyperbola. (0n) OnThus we have points (x1,yI), (x2.y2) such that both satisfy hyperbola equation: x'2/20 22224 = LM00) need to find maximum possible BD?2?
Wait we need largest number less than BD22 for all rhombus? Wait question: VD "find the largest number less than BD2 for all rhombus ABCD Means we need minimal possible BD"2 Wait ND" "largest number less than BD2 for all rhombus {1 " ammiguuus.. Let's parse: We want the largest number less than BD 2 for all rhombus? Means we consider all possible rhombuses, compute BD2,, then find the maximun possible BD"2? But they ask V0" "largest number less than BD2 for all rhombus { maybe they want supremam of BD Actually VD) "largest number less than BD 2 for all rhombus' {)" " could mean find maximum possible BD"2? Wait V "largest number less than BD22 for all rhombus {}" " could mean find the greatest lower bound? Let's 's's read: '0 "find the largest number less than BD22 for all rhombus ABCD), ()"" Might mean find the maximum possible value of BD'2? But phrase VJ" "less than BD 22)) " maybe they want the largest value that is less than any
BD??? That would be infinum? Wait MD" "largest number less than BD2 for all rhombus) " maybe they want the maximum of BD 22 Let's ss's analyze. X (1n' InWe need to interpret: VD "largest number less than BD 22 for all rhombus ABCD' Suppose we want to find a number L such that for all rhombus, BD22 2 I. And I. is as large as possible. That is essentially the infimum of BD22 over all rhombus? Wait we want largest lower bound? Actually MD "less than BD2"),) meaning L i BD22 for all rhombus. So L is a lower bound of the set of BD'2 values. Largest such L is the infimum (greatest lower bound). So they ask for infimum of BD'2? Wait \O "largest number less than BD BD2XXL) XX " might mean the greatest number that is less than
BD'2 for all rhombus? That is indeed infimum. But they might ask for maximum possible BD"2? Let's check: Suppose BD'2 can vary. They want largest number less than BD?2? If BD"2 can be arbitrarily large, then largest number less than BD"2 is infinite. But maybe BD'2 has a max- imum? Let's compute BD22 formula maybe depends on angles; maybe BD22 has maximum. But they ask "V" larrgsst number less than BD'2 for all rhombus ()) That seems weird. Let's compute BD2 as a function of something. Perhaps BD 2 can be arbitrarily large? Let's check: Points on hyperbola; as x large, y small. But y must satisfy hyperbola. For given x, y22 = 24%(x -I). So " sqn1(24"xx22000 2/20 -1). For large- -x-, y grows roughly sqr1(24/20) - sqri(6/5)
So as -x- large, y " -1.055-x-- So coordinates far away produce large distances. BD2 between points B and D? B at (x2,y2). D at (-x2,-y2). So BD vector = (-x2 - x2, -y2 y2) (-2x2, -2y2). So "D22 4(x2'2 + 222 That's 4"(x2"2 y2"2). Since x2.y2 satisfy hyperbola, we can compute x2'2 + y2'2 variable. As -x2- large, y2 also large, so BD'2 large. So no maximun; so maybe they want minimal BD?2? Let's read again: \O" "ind the largest number less than BD'2 for all rhombus ABCD. O Maybe they mean find the largest number that is less than every BD 22 That the greatest lower bound? That would be minimal BD?2?
The largest number less than all BD'2 would be the maximum lower bound = infimum. So they ask for infimum of BD' across all rhombus. That is the minimal possible BD'2. So we need to find minimal BD'2. So they want largest number less than BD'2 for all rhombus, i.e., the greatest lower bound. Equivalent to minimal BD'2? If BD"2 has minimum, then that minimum is greater than all smaller numbers; but largest number less than BD2 for all rhombus would be the minimum BD'2 itself? Actually if BD'2 has a minimum value m, then for all rhombus, BD22 2= m. Then m is less than equal to all BD2,, But largest number less than BD2 for all rhombus would be m? But m is not less than BD 2, for rhombus that achieve m; it's 's equal. But phrase
MD "Iylees than BD'2([)" maybe they mean VD "less than equal to'(0 " ambigu- ous. (JnAA)nnet analyze: If we want minimal BD we can compute BD22 = 4(x"2 + y2). For point (x.y) on hyperbola. So
+ y22) We need to minimize this subject to hyperbola
equation. So we need to find minimal value xf2 + y22 given x22/20 x22/20-y222244 = 1. That is a con-
strained optimization." ), (. (nSett x²2 = 20(1 4 + 222/44) Hyperbola: x22/20- 22/20-y22224- = 1 =2 = 20 + (20/24) = 20 + (5/6) 22. Sox2 y = 20 + (5/6)y + y2 = 20 20 (11/6)y So BD = 4(20 + (11/6)y²2) = 80 (44/6) y2 = 80 + (22/3) y 2. (1) OnThus BD22 depends on y Since
y2
30

---

Published as a conference paper at ICLR 2026
6= 0, BD'2 80
80. So minimal BD'2 is 80 when y=0. But point with y=0 lies on hyperbola? Plug
y=0: x'2/20 =1 = x22=00 =7 x= x=x[][0011, /20. So vertices at (000001122000) Then A and C.at (220,0) and (-v20,0). B and D at (x2.y2) and (-x2,-y2). But y=0, then B and D at (x2,0) and (-x2,0). But then all four points are collinear? y=0 for B and D, then B and D on same horizontal line as A and C? Actually A at (220,0), C at (-,20,0), So all four vertices would be on same line, degenerating rhombus? That would not be rhombus. But we must have non-zero y2 to have rhombus shape? Let's think: For a rhombus, we need four distinct points not collinear.
If B and D share same y=0, then B and D are on same horizontal line as A and C; but then the rhombus would have zero area? Actually if all points lie on same line, then shape degenerates. So we need y2 + O. So minimal BD'2 might be 2 180. MDn) OnWe need to ensure that A,B,C,D form a rhombus: all sides equal. But we only used that diagonals intersect at origin; we need to ensure rhombus condition: AB=BC=CD=DA, We used only that A and C are opposite, and D opposite. But need side lengths equal. So we need to impose side equality. V (]ja(]neee coordinates: A=(xl.y),, C=(-xx,-yI). (=xx2,y2), D=(-x2,-y2), Then side AB: vector B-A = (x2 x1, y2 - yI). Length squared LI (x2-
xly2 + ()2 -012-yy)2222 (10))000000ae
C-B = (-X1 x2, x2. -2). Length squared L2 (-xl - x2) (-yl -y2y2= = (xl + x2) + (yl + 22) (MCCD (anCD: D.C (-x2 4 xl, x22 + +yl)= = (xl x22, yl - -2)). Length squared 13 = (xl- x2) (yl y2) = same as L1? L3 equals L1? Actually AB and CD are opposite sides; they should be equal. Similarly BC and DA equal.) \(nx) (nnut for rhombus, all
four side lengths equal: L1 = L2 = L3 = L4. But due to symmetry, L1 = L3 automatically? Let's
check: L1 = (x2- -x1 + (y2 y2) L3 = (x1 x2)2 + (yl y2)2 same
LI.
So L1 = L3. Similarly, L2 = (x] + x2)2 + +yl + y2)2. L4 = (x2 + (12 (y2 = same
as L2. So we need LI = (n))))))))))))
condition: (x2 (222 (xl x2)2 +
(yl + =2)2))) (2)2))))') (nnn)EnEpadd: LHS = xx2xx1j2-0 + =22
+ x12 x/22 y22 2yly2 + y.122. (0RMHS +x]++xxx + x2) + +yl 2
+2yyy2 2222. (nn)) (nSel equal: LHS = RHS. (0.00 (nSimplify: LHS minus = (x222 2x/x2 + xx22
y222 2yly2 y1 (x122 + 2x/x2 + x2 x222 yl2 (22) OnCncee x2'2,
yl22, y22,,222, y122 Left
with (-2x1x2 (elxx-2yyy2)) 2yyyy))- (2x1x2 2yly2)?? Compute: (00. -y:22222222222222222222 12xx22+2y11)
-2x1x2 - 2yly2 - -2x1x2 - 22ly2 = 4x1x2 (ylly.Xmmm (aaam)(()m) (JnSet equal to 0: -4x1x2 - 4y1y2 = = we also have that all points on hyperbola: x1 20122200-0 -y1 = 1; 2/20 -00--y2 2222224 = JADm)fWWWe need to find
xlx2 + yly2 = 0.700)) ((]n)(nnTann QnThus rhombus condition: xlx2 + yly2 0. 100000nB
solutions (x1.y1),(x2,52) satisfying x1x2 + 0.3 (ann).nnn Onee also want BD = 4(x222 + y2'2). We need to minimize BD2 subject to constraints:) (at.sss)))..a.(Oe.........t (JnLet's denote variables: Let coordinates for A: (xl.yl), For B: (x2.y2). Condition: x1x2 yly2 = O. MMNN)OnW can param using angles? Since hyperbola: paramization: x= = (20 cosh v24 sinh t? Hyperbola
standard: -2200-y2/244 1. Use param: == -20 cosh u, y= sinh u. Because cosh'2 2 indeed x/2/20 = cosh'2 u, y22/4 = sinh u. So coordinates: ( (.220 cosh u, ,24 sinh
11. 21.
u). Similarly for second point: (2/20 cosh v, -24 sinh v). But we also need x2 2/20- y222/2244 =1, So both points satisfy.), ), (n))()nTTusss x1 220 cosh u, yl -24 sinh u. x2= = 220 cosh v, y2 /24 sinh v. v.y Aaa ([n)))))e (yNNow rhombus condition: x1x2 + yly2 = 0. 0.A0m)) (Bmn((nCCmputee xlx2 + yly2 = 20 sinh
cosh u /20 cosh v + /24 sinh u * f24 sinh v = 20 cosh u cosh + 24 sinh u sinh v.O ( equal to 0: 20 cosh u cosh
v.+ 24 sinh u sinh = 0.M0a) (a.00a((nDDDa OnDivide by 4: 5 cosh cosh 4 6 sinh sinh v= 0. Om) On can use identity: cosh(u+v) = cosh u cosh v sinh u sinh v. But coefficients differ. Maybe use hyperbolic identiry: cosh(u+v) cosh cosh + sinh sinh v. Similarly for v. Then compute:" \ J0 ( Oncosh u cosh V = 1/ (I,(((-a'2)(1bbb)))) sinh u sinh v = ab / (((1-a22x1-b"2)) So equation: A0a) 0n5
11 (((((-a2))11b)))
+6 * ab/ / ((11((()((()bb = O. O0000O0Mulippppp both sides by 6ab 00b=0=aam ab = -5/6. 130.)0m) (nnD)nnTus tanh u # tanh v= BnnNhaTT we have relation between u and v. -Sine tanh u in (-1,1). So product negative implies u and v of opposite signs. So we can pick
+
0z0. vi0 etc. (on) (yTThus we have relation: tanh u * tanh -5/6. (0)) ( (nNoww we need BD 2 = 4(x22 +222) = 4(20 cosh 2 v + 24 sinh 2 v) = 4(20 cosh 2 + 24 sinh 2 v). But cosh - sinhh 2
31

---

Published as a conference paper at ICLR 2026
=1. So cosh = 1 + sinh So 20 cosh 22 v sinh v 20(1 + sinh v) + 24 sinh 2 20 (20+24) sinh '2 = 20 + 44 sinh'2
MA0 BD22 = 4(20 44 sinh v) = 80 + 176 sinh'2 n. v\0n' Maa0a0nnniiinn OnSimilarly we could express in terms of tanh v. Since sinh v = tanh "2 V $ cooh'2 v. But maybe easter: Use identity:
sinh'2 y = (cosh 2 -1). But we can express cosh v in terms of tanh cosh'2 = 1/([I-annh>] "2). So sinh = cosh '2 -1 (1/(1- - -1) (1
12/11 - -22). Where I = tanh BD'2 = 4(20 + 44 * (1/2(111122) So BD2 = 80 + 176 a (2A1-12)M(a) (MVV)N()nnnaaa (nNoww
w WMDn')nnThnn
we need to minimize BD22 over possible |satisfying relation with u: tanh u $ tanh V = -5/6, But 1 =
tanh V.
Let a = tanh u. Then a = = where b v = So -5((6)). Since a in (-1.1). So t must 10:--;==---000 ======= 1 -0- 2 5/6. So a must be between 5/6 and I or and -5/6. But a is tanh u; so u large magnitude ?? tanh 25/6 implies -1t- 2 artanh( 5/6). So a in (5/6,1) or (-1,-5/6). But since and have opposite signs? Because product negative, if az then 1,0, etc. So we can consider az, 5/6 I negattiee." }.n)), (arr)..nnja (JnNow we need to compute BD2 in terms of of Because I expressed in a. But BD22 depends only on (since BD 2 formula). So we need
satisfy ;I. So -eed--5/eee
to minimize BD22 over 1 satisfying relation? Wait BD22 depends only on 1. But 1 = tanh V. But v's relation to ensures 1 == -5/(6a). So we can vary a in (5/6, ). For each a, we have 1 == -5((a). Then Let's compute expression in a. \. (on) (arr).nnle (nLett I = -5/(6a). Then 1'2 = 25/(36 a22). Compute 1-22 25/ a = (36 25) a 2). Then 1 (22(1-1' = (25/(66 -25) = 25/(36 a²2 -25) ( InThus + (176425) a²2 - 25). Simplify: 176*25 = 4400. So
BD'2 = 80 + 176* *
a²2)) ((3)) a22 (
(36
BD'2 = 80 + 176 25/(36 a22 - 25) = 80 + BD²
80 + 4400/(36 a2 - 25)AAaa) (aNOOnNOWwwwWWa a in (5/ We need to minimize BD2? Actually BD'2 increases as denominator decreases. Denominator 36 a²2 -25. Since a²2 11, denominator ranges?
For a close to 1, denominator = =3*1 =1. For a close to 5/6, a²2 = 25/36, denominator = 36*(25/36)-25= = 25-25=0. So denominator goes to 0 from positive side? So a"2 25/36 =t 36 a²2 =25 =| denominator = 0. So as a approaches 5/6, denominator tends to 0+, so BD'2 tends to infinity. So BD"2 minimal occurs at largest denominator, i.e., a=1. So minimal BD'2 occurs when
a=l (tanh u =1). But tanh u cannot be exactly 1? As u. (fu2192) fu22le, tanh u²) (uu1121. So a can approach 1 from below, Then denominutor=36*1 -25=11 So BD'2
=80 + 4400/11 = 80 + 400 = 480. 4400/11 = 400. So BD'2 tends to 480 as a (Qu21121. So minimal BD = 4802 But we need to confirm that a=1 corresponds to u infinite, but we still need to satisfy rhombus condition with finite coordinates? As u u/,0u22922 (00pu22922222222222 fu22le, point A tends to ( (v20 cosh u, .24 sinh u). As u3,0u2192,(02221 cosh u "(e'u)2 large. So coordinates huge. But we also have such that tanh -5/(6a) --5/6. So v finite negative such that tanh -5/6. So B coordinates finite, So rhombus with one vertex at infinity? A at infinity, B finite, But rhombus with infinite vertex? Not allowed. So we cannot let a Ju2111 exactly? Actually a=1 means
a (NJu2292) (12212222222222 (u22le,, so
A's coordinates infinite. So not allowed. But we can approach arbitrarily close to 1, giving arbitrar- ily large coordinates for A. But we need finite coordinates for all vertices. So minimal BD may not be achievable but we can approach 480 from above. So largest number less than BD for all rhombus is 480? Wait we need largest number less than BD'2 for all rhombus. Since BD2 480 for all finite rhombus? Let's test: For any finite u, a i1, BD 2 2 480? Since BD'2 = 80 + 4400/(36 a 2 - 25). Denominator à O. As increases, denominator increases, BD2 decreases. So BD2 minimal when denominator maximum. Denominator increases with a a2,. So maximum a 22 il. So minimal BD22 & 80 + 4400/(11)?? Wait compute at a close
1: 0.99 Let's 's compute approximate: a d02=0.99800, denom = 0,99009 =
-25=10.928036. BD'2 80 4400/1109220366 = 80 + 402.9 = 482.9. So BD 2 480, So minimal BD22 2 480. So largest number less than BD22 for all rhombus is 480? But BD always i 4800 Actually at close to 1, BD'2 442.9 2 480. So BD22 2 480 for all finite rhombus. So the greatest lower bound is 480? But BD'2 2 480 for all rhombus. So 480 is less than BD22 for all rhombus. But is there any rhombus with BD arbitrarily close to 480? As a) d'uu21921, BD"2 tends to 480 from above. So 480 is the greatest lower bound. So answer: 480. A80ALD N00) () AnBut they ask VD "largest number less than
32

---

Published as a conference paper at ICLR 2026
BD22 for all rhombus (}" meaning the supremun of numbers less than all BD'2. That is 480.
So answer: 480.)[m'' (nLet's double-check with example: choose u large but finite, say u=5,
tanh u - 0.9999? Actually tanh 5 - 0.999909, a²2 - 0.9998. Denominator
3640.9999....... -25 35,996 -2.-20 996.. BD2 80 + 4400/10.996 = 80 + 400.6 = 480.6. So BDD2"480.6 2 480. So indeed.) (0n),O0n OuThus answer: 480.) Onne should confirm rhombus condition: x/x2 + yly2 = 0. Let's test with u large, V finite such that tanh = -5/66 tanh 1). For u=5, 0,999, I=
5/(640.9999) zs -0.8334. Then V artann(f-088336 R8 -1.0? Actually tanh(-1) = -0.761. Not -0.833, So need v such that tanh = -0.833. V v. -1.2? Let's 's compute tanh(-1.2)
-0.833. Yes. So coordinates: A at (x1,y) huge, B at finite. So rhombus vertices: A large, B finite. C opposite of A large negative, D opposite of B. So rhombus large, but
still finite? Actually A coordinates huge, but still finite. So rhombus defined. So BD22
4*(x222 + y222). x2 funite, y2 finite. So BD22 finite. SSo rhombus exists. So BD
2 480. So answer 480. (pn (uTThus final answer: M0A (oxxed(480)).... .ssissaastta the vertices of the rrhombus be M0000 DADA 10000A--=---0000 L.
O Iquad
four points lie C= D), MONO Dquad D=(-x 2 100 OA In A UnAll the hyperbola's & (onddOn. AD DBANAAD On On' AndHH
B= 2), 20000)0000 MOMQuua
fx (2)) (20)- Parameterisation MO MrAOfracty hyperbolay, Oni OnWrite On OnN0 OA Oat 0000sm ON Ocosh of the AM000guuan y= DENOMOs Os0rg((4))) (24) COMMOOOOO (iink 10 OF OAOAGA OaX Onfor a real param-
I.
eter BMOOGOO ONOO.
Manhhss OnNOn'd ORNDNON COA0A-=- IAA= Aggg 200OOM Ooosh w) WMMMtgquue CONOAAO (44gg(2441000 M sinh
EMOM CAMMAAE OA Asrr 24 siinh
Mhn= AABB-000 AOM Osgrt (20 MON F D DAD O COOOOA On InHHH
2.
Rhombus condition's X OnA
OnThhe sides of a rhombus
are
equal,
so On (AA00) On 0M000000 On-(x (x-(x22 2-x 2-y.12yy22222 I+x 2 14y 10 10 OO MANDOAA 100000 Jnx 2+y 2=0. hyperbolic
10 OA0 MorAAOn OnThis
simplifies
parametrin tnrametrisation gives On On COAAAOOOOO COMO On O0Dsgg (20) 20A00 a ADMDAD (00ggrr(20 (20) (20) MON Acosh
1000 Olag DA COO.OOOOOO
Oa000nnnnnnnnn OnSubsittaing
the
O 249 MAAMA
OO sinh
a O COMOMN AAAOOO Osgrt MOA0 Usinh
A0AM O COAAAA Onor O0DD ns
MOMcoohh a A O'0cohh
4+6 DOMOOO (PDDsin sinh
Aiini v=0.
Onx O OnUsing
O taan tanh
WE CAMM Odfracl OM Osinh
of OAOA
and
MAAONAD ( becomes Jn Onl OF 000005 ON
== 3000
MEA FAO sishh
o WHOM feosh
o AAAAN
nnquuttonn
(2)
O COO ONO Hann
++6=0
MO OF
N Equad &O% OO00 Longrightarron
On ONO
Hanh
A MOMDAMM 0 Itanh
v=- -AN Afracc6
OO 10000 Orag (ogg33))00 3y00 CO CAOAOAAOOO
ON
3.
Diagonal
length (000BADD)
XO BADYBDD KAMDDAA M OnThe
diagonal
NOXO (XO)
joins
and
so s0) (00 ORA00 OOON MANDD 2
2 2 22
=A -A0 O Foohs 2
v+24 O'isinh 2
A bign
=N O MONDE (FA taanh
M0ihh 2
M OnLet
WiThen
V= 10000 Offrac adffaaerr22iaaa (22H)-22 (AND.
and
2= A0000OOO0 Oigiiigi20 441200440000000 OfOrfrat ( 2 (100BigrA
=80+ VOA Ofract 176 176 ONO. H
(0A0rag(3a)
AO OA AntHH 4.
Express N0A00 0000000 ONO) through
ih O0MM00 OAA00000nannn OnFrom (3) we have
M0X0= A0A Orffrac (5) (6V0NOAD
COA00O000aa Otanh a 4M (DD) Mater A0%0(a= N.
" AO (a ONO).
Then's On Onl C CAONNN One ND Mfrrra (25) 2}
MAVO Equad Aaaad-OA ONA Longrighttrroow OOF OM Iquad UA M.A (rac
[2]= ON Ofaac 25) (36a 2-25)
N On OOOAO A Innubbstituiin
into
(3a).
A Oankd In COMOOOOOO 1 2 80+
MM Airrcc (4400) 4400)(6a" (36a
1000aag PAOOMMOOA 10000O0 OMO An#tH S. Minimising
On 0 OnFor VorThe denominator (336 2-
a rhombus we need NON000 MaagiDD (OO)).
25 25ND)
increases
with
so M0N0 is as large as possible, L.e the limit MAMMA On & InTaknng
is On 9 minimal ** when M0A0AAONO
as
KOND
COAAAN CANAAAAAAN (ylm ta OD Otol
BBD BD 2080+) OP (4400) (11) 1480
AQ SOAMAAO TADAAA (AA0 OnFor every finite rhombus MMDDaa DYNOOA AAAAA0:BB 2840 (0800 (MM))
hence O (0A0000000000 (4880 (AD)
is the N000 2 CAOAAAAAAAAAAA DAA0ON0 (boxed(400)
greatest lower bound
of
2
33

---

Published as a conference paper at ICLR 2026
Input Prompt 2: Every morning Aya goes for a 9-kilometer-long walk and stops at coffee shop afterwards. When she walks at a constant speed of 8 kilometers per hour, the walk takes her hours, including l minutes spent in the coffee shop. When she walks + 2 kilometers per hour, the walk takes her 2 hours and 24 minutes, including minutes spent in the coffee shop. Suppose Aya walks at + 2/2 kilometers per hour. Find the number of minutes the walk takes her, including the t minutes spent in the coffee shop.
GPT-OSS-20B Model Output:
We have a problem: Aya walks 9 km, stops at coffee shop for L
minutes. When walking at speed km/h, time walking 9/s hours. Total time including coffee
shop = 9/s + t/60 hours 4 hours. So equation (1): 9/s - t/60-
4. When walking at speed
s + 2 km/h, walking time- 9/(s + 2). Total time = 9/(s + + 2) + f/60 = 2 hours 24 minutes
=2+ 24/60
2 + 0.4 = 2.4 hours. So equation (2): 9/(s + + t/60 2.4. We need t
and Solve. Let's denote mmin = t (minutes). Convert t/60 to hours. Use equations. Equation
(1): 9/8 = 4 - 1/60. Equation (2): 9/(s +
2.4 1/60. Let's denote = 1/60. Then:
9/3 = 4- A 3
9/(4- A). (Equation A) 9/(# + 2) 2.4 A 7 2
9/(2.4-A) = 3-9/(24-AA)
(Equation B) Set them equal: 9/(4- A) = A)
Solve for A. Let's compute. Multiply both sides by denominators. Let's let A. Equation:
9/(4 - z) = 9/(2.4 - x) - -2 Bring terms: 9/(4 x) + 2 = 9/(2.4 - 2). Compute LHS:
9/(4- - +2 - +9 2(4 x))/(44- x) (9
2r//(4- 1144-3 x) (17 2x)/(44- x). Thus
RHS: 9/(24 r).
-r).. So (17- 24)/(4- - x) = 9/(2.4 - x). Cross-multiply: (17 (2.4 -x)
9(4-x). Compute left side: Expand. Let's compute (17- (2))22(222422
Multiply: 17. 2.4 40.8.
(-- -17x. (-2x) 2.4 -4.8r. (-2x) = 2x Sum: 40.8 - 17x - 4.8x + 2x² =
17. =
40.8 21.8x + 2x.. Right side: 9(4 x) 36 9r. Set equal: 40.8 21.8 +
36 9r.
Bring all to left: 40.8 21.8x + 2x² 36 + 9x = 0 = (40.8 - 36) = 4.8. Combine terms:
-21.8x + 9x = 12.8x. So equation: 2x 12.8x + 4.8 =0=0 0. Divide by 2: x² 6.4x + 2.4 = O.
Let's just solve: x² 6.4? Let's just solve 9/(4? I I'm stuck. Let's just solve: x² - 6.4? I'm stuck.
Let's ' just solve the equation: x²2 6.47 Eh. Ok, let's solve: x² 6.47 I'm stuck. Let's ' just compute: x² 6.47 I'm messing. Ok, just do step by direct solving. Equation: x² - 6.47 I'm messing. Let's just solve the quadratic: x² - 6.42 Let's solve: x² -6? I'm stuck.... I'm failing. Ok I'm messing. I'm stuck. I I'm failing. I realize I I'm messing. I'm stuck. I'm failing. Ok, I I'm failing. I'm stuck. I'm stuck. Ok. just compute: x² 62 I I'm failing. I'm stuck. I I'm stuck. This is failing. Ok, I I'm going to solve: x² -6 I'm stuck. This failing. I'm sluck. I'm stuck. I'm failing.
Note: The example above illustrates that if a transition thought is fully evicted or quantized to 0 bits, the model enters an endless loop, underscoring that the notional presence of transition thoughts is critical for stable generation.
F LIMITATIONS
While ThinKV demonstrates strong performance for long-output reasoning tasks, it is not directly applicable settings dominated by long input contexts. Should future LRMs place greater emphasis on long-input contexts, additional exploration will be required.
G IMPACT STATEMENT
This work improves the generation efficiency of large reasoning models (LRMs) by compressing the KV cache, substantially reducing memory overhead while preserving reasoning accuracy. This en- ables continuous long-output generation without out of memory (OOM) failures and supports larger batch sizes, yielding higher throughput. Beyond reducing memory, our method maximizes effi- ciency, contributing to more sustainable Al deployment and expanding accessibility to commodity hardware. As LRMs scale to produce longer outputs, KV cache compression remains an underex- plored yet critical direction; our framework offers generalizable solution that may inspire future algorithm-sye -errtthm--ystemm co-design. Importantly, while enhancing efficiency, our method introduces no additional societal risks beyond those inherent to LRMss.
34

---

Published as a conference paper at ICLR 2026
H LLM USAGE STATEMENT
Portions of this paper were refined with the assistance of a large language model (LLM), specifi- cally ChatGPT 5, used exclusively polish writing and help reduce verbosity to meet page limit. All technical content, methods, and results were conceived and developed entirely by the authors, without influence from any Al tool.
35


Reviews:
[{"review_id": "jix3ScqDmt", "reviewer": "Reviewer_P5bA", "summary": "ThinKV is a KV-cache compression framework for large reasoning models (LRMs) that exploits attention sparsity to decompose chain-of-thought (CoT) into three semantic segments: reasoning (R), execution (E), and transition (T). It applies hybrid compression, including quantization, eviction and system co-design. Across DeepSeek-R1, GPT-OSS, AceReason (math/code benchmarks), ThinKV achieves <5% KV-cache with near-lossless accuracy and 5.8× throughput against previous SOTA methods.", "questions": "1. The experimental results show that the proposed method achieves a significant speedup. However, according to Appendix E.2, the implementation is based on **Transformers**. It is suggested that the authors further evaluate the throughput performance on industrial deployment frameworks such as **vLLM** or **SGLang**, to verify the acceleration effect in practical scenarios and compare it with other methods (e.g., the **Double Sparse** method in **SGLang**).  \n\n2. It is suggested to include additional KV cache compression methods beyond **H2O** and **RKV** in the experiments, so as to better demonstrate the superiority of the proposed approach.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The overall quality of this paper is pretty good. The authors demonstrate deep insights in their observation and analysis of the problem and propose an innovative algorithmic design and operator implementation. \n\n- The overall presentation and expression are excellent.\n\n- The paper demonstrates strong empirical results. It sustains >95% accuracy with 20–40× compression on AIME/LiveCodeBench, while 3× batch size and 5.8× throughput gains over R-KV/H2O.", "weaknesses": "1. The current experimental setup shows a relatively large accuracy drop. It is recommended that the authors investigate how much acceleration can still be achieved when the accuracy drop is controlled within 1%, to better demonstrate the stability and practical value of the proposed method.  \n\n2. The paper appears to lack **ablation studies** related to the operator design. The authors are encouraged to add relevant experiments to analyze the impact of different operator designs and usages on overall throughput and accuracy, thereby enhancing the completeness and persuasiveness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zE5k8qr265", "reviewer": "Reviewer_ESb6", "summary": "The paper proposes a hybrid quantization+eviction methods for reasoning model, achieve high compression ratio with minimum accuracy degradation.", "questions": "1. How many times of averaged score do you use for one reasoning question?\n2. Can you explain more on block_size in flg 9e?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper combines eviction with quantization, and shows that different tokens should quantized differently.\n2. Experiments are run on latest reasoning models and challenging benchmarks.", "weaknesses": "1. A very unfortunate thing for post-training efficiency technique of reasoning models is the inflation of generation length. Quantization is servere as mentioned in the paper. Sparsity attention can also lead to inflation but tends to be smaller. So it is meaningless to show metrics like **throughputs** or **compression ratio** in the paper as they hide this effect. Similar to max-batch size. You will never get these high batch size as 1. larger batch size comes from larger compression ratio-> 2. larger compression ratio means more inflation of kv cache-> 3. larger kv cache per seq means lower batch size. **So from my side, the only reasonable metric for reasoning efficiency evaluation is how long you take to solve an benchmark e2e such as AIME and LCB using same GPU setting.** And this should be normalized to standard serving framework with dynamic batching like vllm or sglang so that it reflects the ablility of your kv compression method. \n2. The paper writing is not clear. It's difficult to follow. I suggest the methodology just focuses on describing what the algorithm looks like without so many definition, observation and notation. Even with so many paragraphs, I have not been convinced and motivated. For example, what if future reasoning model reasons in hidden states instead of explicit outputing tokens? Is there still going to be R, T, E? I am not sure, but I believe attention sparsity and quantization still exisit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "au07DMc7rs", "reviewer": "Reviewer_pvM3", "summary": "This paper presents a combination of strategies to reduce the KV cache size during inference with a decoder transformer.\n\nThe method combines quantization and deletion of blocks of tokens in the KV cache based on a category which is inferred from the attention sparsity. The authors also propose an efficient implementation derived from PageAttention.\n\nThe resulting gain in speed for a given accuracy level beats existing methods.", "questions": "Some questions, in no specific order:\n\n- The clear separation of the three classes is very impressive in Fig 3, and Fig 13 provides other examples, for different models, but how does it hold on other datasets? Are all this graphs from AIME?\n\n- There are many hyperparameters in 3.2 and 3.3, how were they chosen?\n\n- The method is compared to other KV cache quantization or eviction, but what are the sota methods overall? What about other types of methods (MLA, YOCO)? It would be great to give a clearer sense of the pareto front accuracy vs. memory footprint across methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem addressed by the authors is key in the deployment of large language models, and more generally reducing memory footprint is a central issue: Lower memory means possibly larger batches, larger model dimension, longer contexts, and longer / more c-o-t, which ultimately translate into smarter models.", "weaknesses": "This is an highly engineering paper. The method proposed does not derive from a new principle nor rely on formal results or quantitative analysis. There are some motivating experiments that are interesting, but the bulk of the method is a recipe with lot of hyperparameters.\n\nFor the presentation, I encourage the authors to change the scaling of fig 1 and have both axes starting from 0, the current framing is IMO misleading. \n\nI am doubtful the inclusions of figures respect ICLR's style, and captions could be more detailed (e.g. fig 3, what data set?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dk1A5VEosh", "reviewer": "Reviewer_86j3", "summary": "This paper introduces a novel approach to KV Cache management specifically designed for Large Reasoning Models (LRMs). Recognizing that these models generate extremely long sequences, leading to significant KV Cache demands, the authors address that existing methods often overlook the unique characteristics of reasoning models. The core contribution lies on observations about attention sparsity variations across different reasoning stages. Based on these findings, the proposed method, ThinKV, dynamically manages the KV Cache through a combination of quantization and eviction strategies. This approach aims to simultaneously reduce latency and improve peak throughput, enhancing the overall efficiency of LRM inference.", "questions": "* **Accuracy–Efficiency Trade-off with Non-Reasoning Modes.**\n   While ThinKV improves accuracy relative to other compression baselines, it still introduces slight degradation compared to Full Cache. Recent models such as the Qwen3 series support *optional reasoning (thinking vs. direct answer)* modes. It would be valuable to compare ThinKV-enabled reasoning against no-thinking (full attention) decoding, to quantify whether the added reasoning—and compression—yields net gains in both accuracy and efficiency.\n* **Sensitivity of different Thought Types to Quantization Precision.**\n   Section 3.2.2 discusses precision settings on different reasoning stages. Prior work such as QAQ [2] demonstrates that Key and Value Cache have different sensitivity on precision. A similar quantization-sensitivity study across thought types would clarify the boundary of compression on different reasoning stages.\n\n[2] Dong, Shichen, et al. \"Qaq: Quality adaptive quantization for llm kv cache.\" arXiv preprint arXiv:2403.04643 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **A Novel Design.** The paper introduces a thought-adaptive framework that maps attention sparsity to reasoning stages, enabling hybrid mixed-precision quantization and adaptive eviction. This design achieves fine-grained, semantically informed KV cache compression beyond traditional token-level compressions.\n* **A Comprehensive Implementation.** ThinKV is supported by a strong system-level realization, featuring gather-free memory reuse via an extension from vLLM mechanism and fused dequantization–matrix multiplication inspired by KiVi. The paper further provides detailed kernel-level breakdowns and overhead analyses, demonstrating solid engineering rigor.\n* **A Strong Result.** Across math and coding benchmarks, ThinKV consistently achieves higher accuracy under different memory budgets and higher throughput compared to R-KV, showcasing both algorithmic and system-level efficiency gains.", "weaknesses": "* **Lack of Latency Comparison.** Although Section 5 reports throughput under large-batch settings, the paper does not present a detailed latency analysis. For large reasoning models (LRMs), metrics especially TPOT (Time per Output Token) are critical, as generated sequence will be long and interactive responsiveness matters. A fine-grained latency breakdown across decoding steps or batch sizes would strengthen the system-level evaluation.\n* **Limited on long input tasks.** ThinKV is evaluated on datasets whose input prompts average only a few hundred tokens. However, some practical workloads (e.g., SWE-bench [1]) involve very long input contexts. Since ThinKV does not compress the prefill cache, its effectiveness and efficiency under long-input regimes remain unclear.\n\n\n\n[1] Jimenez, Carlos E., et al. \"SWE-bench: Can Language Models Resolve Real-world Github Issues?.\" ICLR. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)