KV Cache Transform Coding for Compact Storage in LLM Inference
Authors: Konrad Staniszewski
Adrian Łańcucki
Venue: ICLR 2026 Poster (2026)
Keywords: transformer
kv cache
compression
Primary area: foundation or frontier models, including LLMs
TL;DR: We present KVTC, a lightweight transform coder that allows for extended retention of transformer KV-cache via compression.

Abstract: Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20x compression while maintaining reasoning and long-context accuracy, and 40x or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks
including AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER.
It consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches.

Published as a conference paper at ICLR 2026
KV CACHE TRANSFORM CODING FOR COMPACT STORAGE IN LLM INFERENCE
Konrad Staniszewski & Adrian Lańcucki' NVIDIA', University of Warsaw? ks kstanisrewsttnvidia.cc
ABSTRACT
Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. How- ever, stale caches consume scarce GPU memory, require offloading, or force re- computation. We present kvIe, lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, kvtc combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, kvtc achieves up to 20x compression while maintaining reasoning and long- context accuracy, and 40x or higher for specific use cases. We test k with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, GSM8K, LiveCodeBennhh, LongBench, MATH-500, MMIJU, Qasper and RULER. It consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compres- sion ratios. These results support kvt c as practical building block for memory- efficient LLM serving with reusable KV caches.
INTRODUCTION
Key/ cache
Chat-based interfaces, commonly used for inter- acting with large language models (LLMs), enable users to iteratively refine answers across open- domain dialogues and specialized tasks, such as code generation (Chiang al.] 2024, Köpf et al.) 2023). Each conversational turn extends the key- value (KV) cache associated with a conversation, storing hidden activations for every previous to- ken. For modern Transformer models, this cache easily occupy multiple gigabytes. As mod- els scale up in size and reasoning capability, gen- erating increasingly long reasoning chains (Ope- nAI et al. 2024),, the KV cache footprint increases, posing significant bottleneck for throughput and latency. During user turns, stale KV caches left on- chip occupy memory, which is needed for serving other users, yet ensure the fastest responses in the future. Conversely, caches could be discarded, in- curring the cost of recomputation, CPU DRAM or local/network storage, leading to transfer overheads. This tension creates latency- throughput dilemma in production systems and ne- cessitates careful configuration.
Linear projection
Adaptive quantization
Entropy coding
Storage
Learned on Dynamic calibration
DEFLATE
programming
(a<COMP)
Figure 1:
The kvtc transform-coode
pipeline. Features are linearly decorrelated via PCA, and the resulting coefficients are quantized using variable bit widths. The PCA basis V is computed once on calibration dataset and reused for all caches.
Quantization
PCA
30
18-22
offloaded to
keffc 16x
Jaste
Jaste
Figure 2: KV cache compression ratios con- tributed by parts of the kvtc pipeline for Llama 3.1 8B. DEFLATE's variability is marked with black stripes.
Crucially, inference frameworks view the local KV caches as databases. Strategies such block-level

---

Published as a conference paper at ICLR 2026
Better Alganax
Algenc
Bistors Alghteer
Alianes
12-
20
Stenibe
Stnilahh
Real
126
Hoa
126
Hoad Lndox
Head todox
Head Index
HeadIndes
(a) Keys
(b) Values
Figure 3: Cosine similarity before and after alignment between key (a) and value (b) heads calculated using Llama 3.1 8B inputs from Qasper (Dasiii et al. 2021; Shaham et al., 2022). For each example, we calculate cosine similarity between all keys/values from the same position and then average the batch. Orthonormal alignment matrices produced using 20 samples from the RedPajama v2 (Weber et al. 2024).
paging and prefix sharing promote reuse of caches whenever prompt prefix matches (Kwon al, 2023). Scaling LLM serving increasingly hinges on KV cache management and reuse (Liu et al,
2024b Cheng al. 2024 Yao et al. 2025). but current systems struggle store, move, and refresh these caches efficiently. CacheGien (Liu et al. 2024b) compresses caches for transmission, offering most 8.6x KV cache reduction comparison a 16-bit baseline. SVDq Yankun
2025) and xKV (Chang et al. 2025) pursue low-rank compression during prefill, but both require calculation of per-prompt SVD. Long and frequently used prompts may justify investing more compute for offline training of corpus-specific caches (Eyuboglu et al. (025)..
Meanwhile, intensively studied KV cache compression methods, aimed improving the runtime efficiency of autoregressive generation, offer interim measures to the cache retention problem (Yuan 2024). Prior work hinges on observations that KV cache can be quantized (Frantar et al, 2023 sparsified (Liu et al. [224d]] Hooper et al., (024), average-poolett (Nawrot et al. (0024), or shared between layers (Brandon et al., (204)); the cache itself compressible (Yuan et al. 2024), and dimensions of keys and values for separate heads show high degree of correlation (Zhang et al., 2023). For long contexts, these methods offer substantial throughput and latency improvements, by lowering KV cache sizes and thus the memory traffic during next token prediction. However, due to tight latency constraints, often coupled with refraining from modifying weights of the model, these techniques tend be brittle (Tang al., 2024), and accuracy degradation prohibits combining meth- ods for compounded benefits. Finally, these methods seldom exploit the strong low-rank structure of KV tensors.
In this paper, we introduce kvtc: a simple yet powerful transform coding scheme, compressing KV caches for storage. Inspired by classical image codecs, it applies learned orthonormal transform followed by channel-wise scalar quantization, which dynamically allocates bits, and entropy coding. The resulting bitstream on average 20x smaller than the original 16-bit one, while maintaining comparable accuracy. The method also exposes smooth compression-accurac trade-off, with 40x higher compression attainable at modest accuracy decrease. Thus, kvtc largely mitigates the problem of KV cache management: lowering the cost of its on-chip retention and the bandwidth required for offloading, without compromising interactive latency.
Table 1: KV cache size in 16 bits for IK tokens of context.
PRELIMINARIES
KV Cache Structure During decoding in autoregressive Transformers with multi-head self-attention, the keys and val-
Model
Qwen 2.5 RI 1.5B
28 MiB
ues produced for each processed token are cached to avoid re- computation. The collection of these tensors is the KV cache. For layers, heads, head dimension dpoad and sequence length I, 16-bit KV cache occupies (4 dhead bytes.
Qwen 2.5 R1 7B
56 MiB
Llama 3.1
MiB
Llama 3.3 70B Instruct
MiB
Mistral NeMo 12B
160 MiB
MN. Minitin 8B
160 MiB
2

---

Published as a conference paper at ICLR 2026
100
compresiion
slidnng compeesion
kvw
50
GSMBIK
MMLU
LITM
BULER-VT
GSMSX
MMLU
LITM
(a) Attention sink compression ablation
(b) Sliding window compression ablation
Figure 4: Ablation of kvtc with compression ratio 64x x on Llama 3.1 8B: (a) compression disabled for attention sink tokens; (b) compression disabled for the final 128 tokens. All other settings are fixed. Additional ablations provided in Appendices B.3 and B.7
Motivated by work on cross-layer KV cache sharing and compression (Brandon et al. 2024 Chang al. 2025). we ask whether keys from different attention heads, and analogously values, lie in shared latent space. To be more precise, we examine if possible to align key or value caches, produced by different atlention heads, via linear transformations. Specifically, for each pair of atten- tion heads hi, hy in the model, we attempt to align their caches Ki, K, E RRt ReXdoor with an orthogonal map found by solving the Procrustes problem (Gower& & Dijksterhuis 2004):
Rt arg miną |K; - Ky
R. RTR--- =
We then compute token-wise cosine similarity between K; and Kj before alignment, and between K; and K₁Rt after alignment. We repeat the same procedure for values using V,, Vj. Before alignment, inter-head cosine similarity is typically below 0.2. After orthogonal alignment, similarity increases substantially for keys and moderately for values (Figure This pattern suggests that key heads largely inhabit common subspace up before alignment likely stems from random initialization of key and value projection matrices. We note that for data matrix A E Raxd if k directions suffice explain all of the variance of A, then k directions suffice explain all of the variance of B [A, AR] (d+d'' for R Rdxxa This motivates our choice of PCA as a dimensionality reduction method.
orthogonal transformation; their dissimilarity
Another motivation comes from the work efficient attention kernels (Jiang et al. 2024). To be more precise, from the observation that different attention heads can show similar attention patterns. In simplified setting without RoPE (Su al. 2024), where keys are equal queries and assume the exact equality of dot products that create the attention patterns, the key spaces are equal up to an orthogonal transform by the uniqueness of Gram realizations (Horn Johnson 2013).
Sliding Windows and Sink Tokens We avoid compressing both the w most recent tokens and oldest tokens (attention sinks) due to their disproportionately high contribution to typical attention patterns (Jiang et al. 2024). In transform coding for vision and audio, bits allocated to transform coefficients so that quantization induces minimal perceptual distortion. By loose analogy, the atten- tion mass allocated to token can be viewed as proxy for its importance. In addition, when PCA is used to reduce the dimensionality keys and values, the initial tokens yield higher reconstruction errors, shown in Figure 6
Foreshadowing the experiments described in Section we have chosen w = 128 and x = for evaluation. To illustrate their influence accuracy, ablate their compression by setting either w. = 0 or = 0, as shown in Figure These experiments show that compressing these tokens can significantly lower, or even entirely collapse the accuracy high compression ratios.
Multi-Ture Conversations
Let conversation be an ordered sequence
C = ((ro. yo), (I1, ..)....)
where 24 denotes the user (or system) input at turn and y the generated reply. Generation of y consists of a prefill pass, which produces the KV cache for all preceding tokens (Ios : followed by iterative decoding. which generates the tokens of ye. one time.
3

---

Published as a conference paper at ICLR 2026
Cross-domain PCA reconstruction error
PCA roconstruction error
DP bit allocation
crror (%)
60
Vilco F'melkeb HipeWe
Vilue Finelics Openlt
100
Keys Values
Average allocation
30
Keys CR 16x
Key FineWes Qpank1
Relative crror
Keys CR 32x
40
Key FineWes FineWih
20
Válues CR 16x
50
10
50%
100K
150K
5000
10000
Calibration tokens
Context position
Principal component index
Figure 6: Calibration of Llama 3.1 8B with kvtc. Left: Reconstruction error function of the size of calibration set. The arrow A + B denotes fitting PCA on dataset A and calculating the error on B. Middle: The reconstruction error as a function of position in the context. The error is higher for the initial tokens. Right: Bit assignment computed via dynamic programming, counting in the per-group scaling factors.
When a new user prompt 2; is received, the existing KV cache can be re-used and only newly added tokens have to be forwarded through the model, reducing computation and time-to-firsttooken (TTFT). However, if the cache has been deleted, the model must reprocess the entire conversation as prompt, resulting in quadratic recomputation of attention across the input.
KV Cache Management while Serving Efficient LLM deployments often split prefill and decode across separate nodes due to their distinct perfor- mance profiles (Zhong et al. 2024), as shown in Fig- ure The prefill node produces the KV cache and transmits to the decode node over high-speed fabric, typically RDMA-capable such as InfiniBand or RoCE, minimizing latency and CPU overhead on the host. Both nodes may maintain tiered KV caches: GPU HBM (hot), CPU DRAM (warm), and NVMe/SSD (cold), managed with retention policy. Long-term storage might require sending caches to a remote location. Crucially, the decision to select node for either prefill or decode of certain input can dictated by the already-held KV cache with matching prefix. In such setups, KV cache transfers typically the dominant cross-node traffic.
Prefill Node
Cache-war Roster
Deusie Neak:
KV Cache
HRM DRAM SSD
Mansgez
DRAM SSD
Object Sterage
Mileariity
ROMM
Figure 5: A high-level architecture of KV-
cache-aware LLM serving environment.
Why compress KV caches after the prefill or decode phase? Compression can: (i) extend the effec- tive capacity of KV cache database, and thus the lifetimes of caches, roughly in proportion to the compression ratio, and (ii) reduce network traffic. We discuss both angles below.
(i) Extending KV cache lifetimes increases cache hit rates in higher tiers (HBM/DRAM), often avoiding or substantially reducing prefill time for prompts with long, recently processed pre- fixes. For example, during a session with coding assistant, a single 1,000-line file tokenized at ~ 10 tokens per line and processed by Llama 3.3 70B yields about 1.6 GiB of 8-bit KV cache. Subsequent conversation turns, or few parallel conversations about this file. might reuse the corresponding cache. However, on a node which serves multiple clients, the volume of generated KV cache might shorten its hot/warm residency even at moderate batch sizes. A 20x lifetime extension via compression might determine whether a KV cache remains hot/warm until becomes useful, or needs be recomputed from scratch.
(ii) Prefill time scales O(r²) with prompt length and typically dominates transfer time. In addition, KV caches can be streamed by layer during prefill in order to further reduce TTFT (Qin et al. (025). However, KV cache compression reduces memory traffic proportionally to the compression ratio, which might be critical when the network bandwidth is saturated and becomes the bottleneck.
4

---

Published as a conference paper at ICLR 2026
3 METHOD
Our Key- Bey-Value Transform Coder (kvtc), shown in Figure 1 builds upon the transform-coding framework (Ahmed et al. 1974, Goyal 2001), widely adopted methodology for designing image and video compression algorithms such as JPEG (Joint Photographic Experts Group JPEG 1994). It applies feature decorrelation by projecting onto orthonormal basis matrix V obtained via the singular value decomposition (SVD) of centered calibration data (i.e., principal component analy- sis, PCA). Quantization parameters selected using dynamic programming algorithm, and the resulting symbols are entropy-coded with DEFLATE (Wu 2017). The three modes of operation of kvtc are:
Calibration This step is performed only once for every model and compression ratio. During calibration, we obtain the key and value caches for the calibration dataset, calculate SVD(C-p)
UEV where is the mean of the data, and store the projection matrices V (for key and value caches separately). Next, we run dynamic programming algorithm that produces the optimal bit allocation for each of the principal components. This step is fast; for instance, calibration for a 12B model can be completed within 10 minutes on an H100 GPU (Appendix [B.5]. Appendix B.14 we show that the additional amount of data stored per model is relatively small (2.4% of model parameters for Llama 3.3 70B). In Appendix [B.00] we perform an ablation study on the number of layers over which we concatenate the KV caches.
Compression Keys and values are compressed independently using the (V, (1) parameters and bit allocation obtained during calibration. Compression is performed between inference phases (e.g., after decoding or between prefill and decoding) and can be executed on either GPU (affecting TTFT) CPU (if the cache is already in storage). Importantly, during decoding, the model operates on decompressed KV caches; compression is used only for storage or transfer.
Decompression Decompression reverses the compression steps. The most computationally in- tensive operation-thh the inverse projection using V can can be performed layer-by-laye using sub- matrices of V allowing generation to begin early.
In the following sections we describe in detail the components of kvtc during calibration, com- pression and decompression.
3.1 FEATURE DECORRELATION
Unlike prior SVD-based methods that calculate separate decomposition for each prompt (Yankun jet al., [2025] Chang et al. 2025), we compute the KV cache projection matrices once using a calibra- tion dataset C, and reuse them across all requests inference time. Preparing single, generalizable rests on three observations. First, SVD must be computed on a large, representative sample; sam- pling token positions from diverse calibration set suffices for generalization and is computationally tractable. Second, excluding keys and values corresponding to the most recent tokens and attention sinks improves the achievable compression ratio (see Tables 6 and 13 in Appendix [B. Third, po- sitional embeddings distort the apparent low-rank structure of keys and should be removed before compression (Sun al. 2025).
During calibration, we forward all sequences from C through the model and collect their KV caches. For each sequence, cache entries are concatenated along the time dimension to form a global pool of positions. We then sample n token positions from this pool, excluding attention sinks. For each sampled position, we take the corresponding keys (and, equivalently, values) from layers and heads, undo positional rotations, and concatenate them along the hidden dimension dhead- This yields a data matrix C Raxp with p dreat, whose rows index sampled token positions. Let IR be the per-feature mean of C. We compute the SVD of the centered matrix,
(2) with singular values on diag(E) sorted in descending order (equivalently, PCA of C). For any X E mx the decorrelated representation and its inverse are
c- U.V
D= (X V.
X DV +
where the equality holds when all p components are used. When r < p and the basis is truncated randomized SVD (Halko et al. 2011) calculated on a GPU with target rank r < p. substantially reducing runtime and memory. Results for this variant are shown in Figure 6 with additional details and ablations in Appendix B.1
to V Ro then X x D + μ.. For scalability,
5

---

Published as a conference paper at ICLR 2026
3.2 QUANTIZATION
PCA orders principal components by explained variance. We exploit this ordering to allocate a fixed bit budget across PCA coordinates that high-variance components receive more bits. The allocation is computed once on a calibration set and reused at inference. We quantize D coordinate- wise obtain D9h--r where qi Zzo is the bit width assigned the i-th principal component. Under a global bit budget, we minimize the Frobenius reconstruction error
|DV Du V 1|
(4)
Because right-multiplication by an orthonormal matrix preserves the Frobenius norm, we have
|DV - V 1 || 19 yv D
D [
(5)
Thus, optimal bit allocation can be found directly in the decorrelated domain.
We solve the constrained allocation with simple dynamic programming (DP) algorithm that keeps two tables: (1) the minimum reconstruction error achievable using the first i principal components under payload of b bits; and (2) backpointer storing the optimal local decision. Pseudocode and a proof sketch optimality under these constraints are in Appendix [B.17] Furthermore, inspired by the Microscaling data formats (Rouhani al. 2023), coordinates together, each group with shared 16-bit shift and scaling factors. The DP optimizes both the per-group bit width and group size, restricted {1, 16, 64, 256, 1024) components per group. The total budget equals the sum of payload bits across all coordinates plus per-group shift and scaling factors.
quantize groups of subxequent PCA
An example allocation is shown in Figure 6 As expected, the learned bit widths decrease mono- tonically for subsequent principal components. Crucially, the DP assigns zero bits to a substantial number of trailing principal components. This observation motivates reducing the dimensionality early during the calculation of PCA, lowering the cost of calibration, and trimming V the dimen- sions with nonzero bit widths for faster compresion/decompresiii benefits of DP guided quantization over pure PCA in Appendix [B.9
during inference. We show the
3.3 ENTROPY CODING
Finally, the quantized values are packed into single byte arra and further compressed using the DEFLATE algorithm (Wu 2017). Crucially, we leverage nvCOMP (NVIDIA) 2020). which enables parallel operation directly GPU. This step is lossless, but the added compression ratio is content- dependent. We ablate the choice of the lossless compression algorithm and the influence on final compression in Appendix B.8
EXPERIMENTS
Models We models from three families: Llama (Grattafiori al.) 2024), Mistral NeMo (Mistral AI team 2024) Sreenivas et al.] 2024). and RI Ri-distilled Qwen 2.5 (DeepSeek-Al et al. 2025). The selection includes models ranging from 1.5B to 70B parameters of base, instruct, and reasoning kind. Table | lists the models along with their KV cache sizes. Notably, MN-Minitron 8B has been pruned from the Mistral NeMo 12B base, retaining the original KV cache size.
We compare our method against KIVI Liu et al. 2024d), GEAR (Kang et al. 2024) and an FP8 quantization of KV cache the E4M3 format (Micikevicius al., 2022). For eviction baselines, we compare with TOVA (Oren et al. 2024) and H,O (Zhang al. 2023 For SVD baselines, we compare our method with xKV (Chang et al. 2025). Finally, we compare kvtc with DMS (Lańcucki et al. 2025), trained token eviction method, on reasoning tasks.
Methods
For all methods, we follow their original intended protocols, performing prefill in the vanilla mode and compressing the KV cache only after the self-attention has been calculated. For every method except xKV, we simulate a sequence of short conversations by running compression/eviction on the cache every tokens, where depends the method's 's original sliding window policy. For kvt which is run with a sliding window w = 128, we compress/decompres every = 16 tokens, leaving the window in the 1 112-128 token range. In the case of xKV, we compress only the prefill tokens,

---

Published as a conference paper at ICLR 2026
Table Accuracy cache compression methods. Results within score point vanilla are bold. See Appendix B.12 for standard error analysis. kvECCRx denotes kvt set for CRx before DEFLATE.
Vanilla GFARgan
H₂O
TOVA
xKV
FP8
kvtcsx xVtC16x
kVt
kvt
CR
5
8
1-5
2
9-10
18-22
34-44
60-88
GSM8K
56.8
52.8
52.8
54.3
54.5
56.6
55.2
$7.0
56.9
57.8
S7.2
MMLU
60.5
60.1
QASPER
40.4
40.4
39.1
34.3
38.6
35.6
40.8
40.1
40.7
39.4
37.8
LITM
99.4
96.9
88.8
20.2
1.2
99.9
99.4
99.3
99.3
99.1
90.2
RULER-VE
99.8
99.8
98.9
50.4
99.7
99.8
99.9
99.1
99.1
98.9
95.9
MN-Minitron 8B
CR
5
8
1-5
2
10-11 10-11
17-21
32-46
53-95
GSM8K
59.1
57.9
58.0
55.3
59.2
59.3
60.1
60.6
60.3
39.1
57.8
MMLU
64.3
6.366
63.2
43.5
48.1
63.1
64.3
64.2
64.1
63.77
62.1
QASPER
38.2
38.2
38.2
30.0
33.9
38.3
37.7
38.1
LITM
99.8
96.0
86.3
16.6
0.3
99.6
99.8
99.4
99.3
86.9
59.5
RULER-VT
99.4
98.3
96.8
39.2
99.3
99.1
99.2
98.8
98.8
96.0
93.4
Mistral NeMo
12B
CR
5
8
2
10-11
17-20
31-43
51-87
GSM8K
61.9
59.8
59.7
57.0
60.3
61.9
61.7
62.5
62.0
62.2
61.9
64.0
64.3
45.4
49.0
64.5
64.4
61.4
QASPER
38.4
38.6
38.2
29.5
36.0
33.5
37,9
37.6
37.6
37.5
38.0
LITM
99.5
96.9
91.9
16.2
8.7
97.9
99.0
99.9
99.8
99.6
95.3
RULER-VT
99.8
99.4
98.3
35.2
99.6
99.4
99.8
99.5
98.9
98.0
providing it with an advantage, since xKV is specifically designed for prefill optimization, and re- computing SVD matrices for newly decoded tokens would be prohibitively time-consuming. For fair comparison, we only report the prefill compression ratios for non-Qwen models.
Notation-wist kvtccRe denotes kvtc in the default setting, where CR is the target compression for the DP. For all methods, we calculate CR only on the compressed tokens, not counting the sliding window tokens.
Tasks We evaluate compression effects Llama 3.1 8B, MN-Minitron 8B and Mistral NeMo 12B across the following task categories, with results presented in Table
Math & Knowledge:
8-shot Chain of Thought (CoT) GSM8K (Cobbe al. 2021 4-shot CoT MMLU (Hendrycks et al.] 2021aa
Long Context Performance: 0-shot key-value retrieval task from (Liu al. 2024a) (denoted
LITM), 1-shot RULER (Hsich et al. (2024) Variable Tracking (denoted RULER- VT), and 2-shot
Qasper (Shaham al. 2022). In Appendix B.13 we provide an extended long context evaluation
using 2WikiMulliHopON (Ho et al. 2020). MultFiellQA (Bai et al., 2024). MuSiQue (Trivedi
al. 2022). QMSum (Zhong et al. 2021). SAMSum (Gliwa 2019)) from LongBench (Bai
al. 2024) and Common/Frequen Words Extraction (CWE/FWE), Needle in Haystack (NIAH) (Kamradt] 2023), HotPotQA (Yang et al., 2018), SQuAD (Rajpurkar et al.) 2016)] 2018) from
RULER (Hsich al. 2024), showing that kvtc can still maintain performance comparable to vanilla with approximately 20x compression ratio.
We evaluate R1-distilled models challenging mathematical competitions AIME 2024-2025 (Art of Problem Solving 2025) and coding tasks from LiveCodeBench (Jain et al. 2025). with results presented in Table We additionally evaluate kvtc with Llama 3.3 70B Instruct
MATH-500 [Hendrycks et al. 202 Lightman et al. (2023), the key-value retrieval task from (Liu et al. 2024a) and Needle in Haystack (NIAH) (Kamradt 20233] Hsieh al. 2024), with results presented in Table Detailed evaluation protocols can be found in Appendix A and ablations and details about parameter choices for kvt e in Appendix
Calibration Data We sample a 1:1 mixture of short and long documents, with lengths in the 1-8K and 8-32K ranges, respectively. Rotary positional embeddings removed for calibration; further details are provided in Appendix [ Ablations showing kvtc stability and generalization across domains of the calibration data are provided in Appendices B.5 and B.6

---

Published as a conference paper at ICLR 2026
4.1 RESULTS
In all experiments, kvtc applies the same compression ratio to both the key and value caches, An ablation of their individual compressibility is presented in Table 7 (Appendix B suggesting that further adjustments could yield additional gains.
General-Purpose Base Models We evaluate kvt c on general pprppose models the 8-12B scale, featuring three GQA-enabled models (Table The compression ratio of kvtc varies due to the data-dependent nature of the DEFLATE algorithm, which, on average, achieves a compression ratio of approximately 1.23 on top of quantization. Crucially, kvtc maintains high accuracy across tested tasks, even at substantial compression ratios of 32x and 64x. Conversely, quantiza- tion SrtthodsGGEAR and KIVI-exhibi signs of performance degradation in the Middle tasks at 5x CR; cache eviction methods such as HzO and TOVA perform poorly as generic KV cache compressors. We also note that xKV performs well across most tasks, except for Qasper. Interestingly, in certain cases, kvtc at very high compression ratios even surpasses the performance of the vanilla models. A similar observation was made in (Eancucki al. 2025) for token eviction method; we provide additional insights in Appendix B.6 Table Crucially, kvtc 16x compression (approximately 20 after DEFLATE) consistently maintains results within score point (accuracy or F1, depending on the task) of the vanilla models. The standard errors for these results are reported Table 19 (Appendix B.12
GSM8K and Lost
Reasoning Models In order test kvtc under more challenging conditions where context plays critical role, we complex math and coding tasks (Table 3). Due to high variability, AIME results are averaged over eight independent runs with results reported as score On coding tasks, kvtcax shows minor accuracy drops of 0.3pp for the 1.5B model and 0.2pp for the 7B model. Notably, the KV cache size of the 1.5B model is already small at 29 KiB/token, compared 131 KiB/token for Llama 3.1 8B, and 9x compression shrinks it to only 3.2 KiB/token. We also compare our method against DMS, a state- of-the-art autoregressive KV cache token eviction method. DMS achieves competitive results, and since employs token eviction, it could potentially be combined with kvte for even lower KV cache footprint.
Table 3: Reasoning quality (sampling temp 0.6, top-p 95%) of DeepSeek-R1-dit RI-disillled Qwen2.5.
DMS results reported by Lancuuci et al. 2025
AIME24
AIME25
LCB
Method
CR
Competition Math
Coding
Qwen 2,5 R1 1.5B
Vanilla
26.2+4.8
21.7+2.9
16.4
kvt Gax
25.4
24.2+1.
16.1
kvtelex DMSsx
18
27.9
22.5+6.2
13.3
23.3
N/A
16.1
Qwen 2.5 RI 7B
Vanilla kvtcsx
50.944.9
40.8+4..
36.7
9-11 18-21
52.5+3.6
40.8+6.2
36.5
kvt DMSsx
50.916.8
38.3
31.6
50.0
N/A
33.4
Multi-GPU Inference To investigate attainable compression ratios for models distributed across multiple GPUs, we evaluate kvtc using Llama 3.3 70B (Table 4). The model runs in pipeline- parallel setting (Hu et al., (02T)) on four GPUs, each handling 20 layers. We maintain a local KV cache on each GPU, applying kvtc separately. Accuracy drops on the MATH-500 task are within 1.5 stderr, with accuracy decreasing by 1.2pp at 10x compression and 3.0pp at x0x.
Latency We calculate the latency of elements of the compression pipeline and provide the results in Table 5 In contrast to full re-computation of KV cache for 8K context length, kvtc16x can reduce time-to-fisst-okken (TTFT) up to 8x.
LIMITATIONS AND FUTURE WORK
Online Compression and Composability with Other Methods kvtc was designed for efficient storage and reducing time-to-first-toke However, the advantage of having single, generalizable PCA matrix for initial compression of cache renders it suitable for further exploration of inference directly in the principal component space. We leave this future work. Notably, kvt c does not alter the structure of KV cache and does not change how the attention is calculated. Consequently, it is directly compatible with token eviction methods, including but not limited to these used in the
8

---

Published as a conference paper at ICLR 2026
Table 4: Compression applied a model which is split across four GPUs (pipeline par- allel), with KV cache chunks being com- pressed separately with kvtc. In certain sce- narios, like offloading to CPU RAM, these chunks could be compressed joinlyy for higher accuracy.
Table Latency of simple implementation based on the Transformers library (Hugging Face 2025b), measured on an NVIDIA HT00 GPU us- ing Mistral NeMo 12B in bfloat16. In addition, we compare TTFT during recomputation of KV caches with decompression of compressed caches. BS denotes batch size, CTX context length.
Method
MATH-500
NIAH
LITM
Module
BS=8 CTX=8K
BS=2 CTX=16K
Comp
Decomp
Comp
Decomp
Llama 3.3 70B Instruct
Project
153 ms
156 ms
78 ms
75 ms
Vanilla
75.61.92
100.0
100.0
Quantize
67 ms
37
39
27
kvtcax
73.2
100.0
100.0
Deflate
137 ms
64 ms
66 ms
36 ms
kvtc1ox
74.41.05
100.0
100.0
Total
379 ms
267
194 ms
143
kvtcléx
73.21.98
100.0
100.0
kvtC20x
72.61.99
100.0
100.0
Vanilla recompute TTFT
3098 ms
1780 ms
kvt decomp TTFT
380 ms
208 ms
experimental section, such as TOVA. Finally, kvtc could be used to compress the latent state in Multi-head Latent Attention (DeepSeck-ATe et 2024).
Scalability and Generalization Limits We approximate deployment by evaluating kvtc on benchmark tasks in simulated multi-turn settings, which may not fully reflect real content distri- butions or interaction patterns. Our experiments cover dense decoder-only models from 1.5B to 70B parameters; evaluating larger models under conditions that more accurately mirror production is left for future work. For calibration, we process approximately 200K tokens on a single NVIDIA H100 SXM 80GB GPU. Under these settings, computing the PCA basis with the randomized algorithm of Halko et al. (201T) completes within minutes. As shown in Figures 6 10 and [ increasing the calibration set size consistently reduces the Frobenius norm reconstruction error. Scaling kvtc beyond 200K tokens, primarily a matter of scaling the computation
PCA, is left for future work. Finally, report Frobenius- norm reconstruction error as proxy for downstream task accuracy. While convenient, this metric does not guarantee task-level gains and its predictive power may be task-dependent. We provide initial correlation analysis in Appendix B.5 and defer systematic study of alternative proxies to future work. Finally, the compression and decompression times of kvte can be substantially reduced via kernel fusion and hierarchical PCA: first at the level of in- dividual layers, then across groups of layers. Nevertheless, even a simple implementation yields substantial benefits and, in many cases, incurs only marginal overhead.
6 RELATED WORK
Tuning-Fres Free Quantization Quantization-basee based methods that avoid model fine-tuning offer a straightforward path to KV cache compression (Zhao al.] 2024 Sheng et al.) 2023). Works such as KIVI (Liu et al., 2024d) and KVQuant (Hooper et al.) 2024 have advanced this direction by developing separate quantization strategies for key and value embeddings. These methods leverage the observation that keys benefit from per-channel quantization, while values are better suited to per-token quantization. Our approach diverges from these methods by first projecting concatenated embeddings from attention layers using SVD matrices derived from calibration set. Quantization then applied in this transformed space, with the precision dynamically optimized via dynamic programming bit allocation. While we adopt KIVT's uniform quantization scheme, our application occurs in the WVD-transformed domain. Similar to KVQuant, we apply compression before RoPE (Su et al. 2024) to preserve model quality.
Tuning- Tuneng--Base Quantization A complementary approach to post-training quantization involves fine-tuning models to adapt to quantized activations. LLM-QAT (Liu et al.) (024c) leverages gen- erations from the pre-quantized model for fine-tuning, while BitDistiller (Du al. 2024) merges Quantization Aware Training (Jacob et al. 2018) with Knowledge Distillation (Hinton et al. 2015) In contrast, our method eliminates the need for parameter modifications.

---

Published as a conference paper at ICLR 2026
Singular Value Decomposition Approaches SVD has emerged straightforward method for removing the redundancy in KV caches and exploiting its low-rank structure. GEAR (Kang et al.
2024
improves quantization through low-rank correction mechanisms, whereas LoRC (Zhagg al.
2024
minimizes computational overhead by directly reducing the rank of key and value matri-
ces. Eigen Attention (Saxena al. (024) restructures attention computation by projecting into truncated subspace defined by SVD, enabling efficient operations. A similar mechanism could be devised for value vectors kvtc, and key vectors for layers that do not employ positional embed- dings. GEAR (Kang et al. 2024) improves KIVI quantization through low-rank correction mecha- nisms. Building on ShadowKY KV (Sun et al. 2025), SVDq (Yankun et al.) 2025) integrates SVD with quantization, leveraging singular value magnitudes for a simple precision allocation; xKV (Chang) al. 2025) aggregates KV caches across multiple layers before decomposition. This work differs in three respects: (i) it models rotational relationships between non-adjacent layers to enable cross- layer concatenation before decomposition; (ii) it selects ranks and bitwidths via a dynamic program under compression budget; and (iii) it applies entropy coding to the quantized factors. Empirical comparisons and ablations (e.g., treatment of early sink tokens) are reported in Appendix B
Sparse Attention Strategies Sparse attention mechanisms provide a complementary paradigm for managing sequence length dimensions by selectively discarding non-essential keys/values dur- ing inference. Techniques such as H₂O (Zhang et al.) (003)) and TOVA (Oren al., 2024) employ prioritization strategies to dynamically prune less informative elements from the KV cache. In con- trast, chunk-based approaches like Quest (Tang et al.
2024 Landmark Attention (Mohtashami & Jaggi 2023), and Native Sparse Attention (Yuan et al.
2025 construct compressed representations
of the KV cache by partitioning sequences into chunks. These methods retrieve only the most critical chunks during attention computation, significantly reducing the number of memory transfers. Con- currently, dynamic compression techniques such as Dynamic Memory Compression (Nawrot et al.) 2024) and Dynamic Memory Sparsification (Lancucki et al.) (025) optimize KV cache memory age through pooling/eviction of keys/values. These strategies could be integrated with quantization and SVD methods to achieve further gains (Yankun et al. 2025).
Transform Coding Transform coding underpins media codecs because it decorrelates local struc- ture and compacts energy into few coefficients, enabling aggressive quantization and entropy cod- ing of the resulting sparse residuals (Ahmed et al., 1974] Wallace 1992, ISO & IEC| 1998 [VT] 2003 Sze et al. 2014). Similar low-frequency structure might appear neural models, motivating transform coding for activations and compression-avee awwee training (Baskin et al., 2021 Young et al.)
2021,, or repurposing hardware H.264/H.265 encoders as codecs for LLM weights and KV caches Xu et al. 2025 Our approach similarly builds on the transform coding paradigm. However, rather than rel relying on existing algorithms, we design a novel combination of decorrelation, quantization and entropy coding, which exploits lryss-layer dependencies to achieve higher compression ratios.
Cache Management Systems Finally, cache management systems address the operational chal- lenges of KV cache handling in production environments. Paged Attention (Kwon al., 2023) mitigates memory overhead by introducing chunked memory allocation for KV caches, Continuous batching techniques, as implemented in systems like vLLM (Kwon et al. 2023) and FasterTrans- former (VVDIAA 2021), optimize device utilization by enabling parallel processing of multiple sequences. CacheGien (Liu et al., 2024b) advanced the field with a distributed framework for long- term KV cache management, incorporating compression, streaming, and cross-node coordination. Our approach extends these systems by integrating fine -rained compression capabilities.
CONCLUSION
We introduce kvtc, a method for compressing KV cache up to 20x with negligible quality degra- dation, and higher compression ratios of 40x or more available for specific use cases. We empiri- cally show that key and value caches exhibit substantial redundancy, which kvtc exploits through a simple, transform coding pipeline. It is built around linear dimensionality reduction and dy- namic programming algorithm, which assigns variable numbers of bits to principal components. We demonstrate the effectiveness of kvtc across both regular and thinking model families, evaluat- ing models from 1.5B to 70B. We believe that kvtc paves the way towards more efficient LLM deployments, lowering the cost of M-a--iiiitted iterative workflows.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
To foster reproducibility of our results, we provide extensive details about the calibration of PCA matrices in Appendix [8.1 along with ablations regarding kvtc parameters in Appendices B.3 (sink tokens), B.4 (key vs value compressibility), B.5 (amount of calibration data), B.6 (effect of cali- bration data domain) and [B7 (sliding window). In Appendix [A] present the details about the evaluation setup: tasks, used prompts, and baseline configuration. We note that we utilize LM Eval- uation Harness (Gao al. (2224) and RULER (Hsieh al. 2024) for evaluation, which are publicly available. In Appendix [B17 we provide the pseudocode for the dynamic programming precision assignment algorithm along with the sketch of the optimality proof and complexity analysis.
ETHICAL STATEMENT
As method that aims to improve aspects regarding LLM usage, kvtc does not introduce new risks. However, we note that it can amplify existing ones. Therefore, we refer to the existing body of knowledge
the ethical risks of LLM development, such as Ethics Threats in LLM-Based Agents (Gan et al. 2024), potential reversal of safety alignment (Xu et al. 2024), and more general risks regarding LLMs (Li Fung 2025).
ACKNOWLEDGMENTS
The authors thank Mikotaj Blaž, Przemyslaw Podezasi, Piotr Tarasiewicz, and Przemyslaw Strzel- czyk for many helpful discussions; Kevin Shih and Dima Zhylko valuable comments on earlier versions of the manuscript; Szymon Migacz for assistance with computing infrastructure; and Alex Fit-Florea and Michael Lightstone for their support for the publication of this work.
REFERENCES
N. Ahmed, T. Natarajan, and K.R. Rao. Discrete cosine transform. IEEE Transactions on Comput- ers, (1), 1974.
Art of Problem Solving. American invitational mathematics examination, 2025. URL https:
//artoofprbbemmcllvii athematics athematicc__Exammnatiooo Examination Art of Problem Solving Wiki.
com/ htmm//wwkiindexx//// php/ Ammeriicn_Invitattttttaaa
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2024. doi: 10.18651 URL
/v1/2024.acl-long..
ac1211ccc111 acn.ooo.....nn......... 172/
Chaim Baskin, Brian Chmicl, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson. CAT: Compression-awant training for bandwidth reduction. Journal of Machine Learning Research, (269), 2021. URL hRRhhtt:: jmir.. wwtpppppps//////22222 m1
William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan- Kelley. Reducing transformer key-value cache size with cross-layer attention. In Advances in Neural Information Processing Systems, 2024.
Chi-Chih Chang. Chien-Yi Yu Lin, Yash Akhauri, Wei-Cheng Lin, Kai-Chiang Wu, Luis Ceze, and Mohamed S. Abdelfattah. xKV: Cross-layer SVD for KV-eache compression, 2025.
arXiv:2503.18891.
Yihua Cheng. Kuntai Du, Jiayi Yao, and Junchen Jiang, Do large language models need content delivery network?, 2024. arXiv:2409.
Yihua Cheng, Yuhan Liu. Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du and Junchen Jiang. LMCache: An efficient kv cache layer for enterprise-scale LLM inference. arXiv preprint arXiv:25 2025.

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. 1cXiv22100144666
Yann Collet. LZ4 - extremell fast compression. /ttps://1z4... 10-03.
2011. Accessed: 2025-
Pradeep Dasigi, Kyle Lo, Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. dataset of infornation-sseeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, 2021.
DeepSeck-AL, Aixin Liu, Bei Feng, et al. DeepSeek-V2. V2: A strong, economical, and efficient mixture-of-experts language model, 2024. MIXiv:245.004444
DeepSeek-AL, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. arXiv:2501.1299
DaYou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Jarek Duda. Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding, 2014. arXiv:1311.2555
Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re. Cartridges: Lightweight and general-purpose long context representations via self-study, 2025. arXiv:2506.06266.
Facebook. Zstandard fast real-time compression algorithm. https://accbook..iiit 1o/zstd/ 2015. Accessed: 2025-10-03.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers, 2023, arXiv:2210.17323.
Yuyou Gan, Yong Yang, Zhe Ma, Ping He, Rui Zeng, Yiming Wang, Qingming Li, Chunyi Zhou, Songze Li, Ting Wang, Yunjun Gao, Yingcai Wu, and Shouling Ji. Navigating the risks: A survey of security, privacy, and ethics threats in LLM- agents, 2024. Xxiv:2411.09522.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL
www.g//rccorssco 212608602
Bogdan Gliwa, Iwnna Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, 2019, doi: 10.18653/v URL httttst///cc/
Ttehology..rrg/11133
Google. Snappy a fast comppresoorddecooppesssoe https::/ bacpoglle..tuuuu github..hu..... io/ iomnabpp.///pppy/ 2011. Accessed: 2025-10-03.
J. C. Gower and G. B. Dijksterhuis. Procrustes Problems. Oxford University Press, 2004.
V.K. Goyal. Theoretical foundations transform coding. IEEE Signal Processing Magazine, (5), 2001.
12

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, et al. The Llama 3 herd of models, 2024. arXiv:2407.21783.
Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, (2), 2011.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. urXiv:1503002530
Xanh Ho. Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, 2020. doi: 10. 1t.116661//2220.comm col ing main. 580/
580.580. URLhttpps /ttttpss//cclaattooooo.ooooo
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, 2024.
Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 2 edition, 2013.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's 'ss the real context size of your long-context language models? arXiv preprint arXiv:2404.066 06654, 2024.
Yang Hu, Connor Imes, Xuanang Zhao, Souvik Kundu, Peter A. Beerel, Stephen P. Crago, and John Paul N. Walters. Pipeline parallelism for inference on heterogencous edge computing, 2021. utXiv:2110.14895
David A. Huffman. A method for the construction of minimum codes. Proceedings of the IRE, (9), 1952.
Hugging Face, Math- aath-Verify. htttps://githiii hww.tuug...oo....gggg..... watpgggiinnace//Maaaaaa Verify 2024. A tool for verifying mathematical answers and expressions with advanced parsing capabilities.
cww
Hugging Face. Open RI: A fully open reproduction of DeepSeek-R1, 2025a. URL https: /tg.hhub.coo//gggggggafee///////
r1
Hugging Face. Transformers documentation.
ormers en/ index 2025b. A framework for state-of-the the--tth-rtt machine learning models in text, computer vision, audio, video, and multimodal tasks.
International Organization for Standardization ISO and International Electrotechnical Commission IEC. Information technology generic coding of moving pictures and associated audio infor- mation - part 3: Audio. International Standard ISO/IEC 13818-3:1999 1998.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
13

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Joint Photographic Experts Group JPEG. JPEG standard (ISO/IEC 10918-1). International Stan- dard 10918, ISO/IE, 1994. Image compression standard consisting of multiple parts including core coding technology, compliance testing, extensions, and file interchange format.
JVT. Draft ITU-T recommendation and final draft international standard joint video specification (ITU-T rec. h.264/1SO/IEC 14496-10 10 AVC). Technical Report JVT-G050, Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, 2003.
Greg Kamradt. LLMTest. Www.nNeeelIIIAHaattteee Doing simple retrieval from LLM models at various com/ /tt.///aaaaaaaaeee......s.ss /LMMTes..... Ne
context lengths to measure accuracy. htttss://itthiii
ledle InAHaystack 2023, Accessed: 2025-09-24.
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient error reduction framework for KV cache compression in LLM in- ference. In Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri von Rútte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arny Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. OpenAssistant conversations democratizing large language model alignment. In Advances in Neural Information Processing Systems, 2023. Datasets and Benchmarks Track.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium Operating Systems Prin- ciples, 2023.
Miles Q. Li and Benjamin C. M. Fung. Security concerns for large language models: A survey, 2025. urXiv:2505.18889
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang. Olivier Dehaene, Joel Lamy-Poirier, Joao Montciro, Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, et al. StarCoder: may the source be with you! Transactions on Machine Learning Research, 2023. Reproducibility Certification.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. PRM800K: A process supervision dataset. arXiv preprint arXiv:2305. 2023. A dataset containing 800,000 step-level correctness la- bels for model-generrated solutions MATH problems.
Nelson F. Liu, Kevin Lin, John Hewitt Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2024a.
Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayaman, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. CacheGen: KV cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, 2024b.
14

---

Published as a conference paper at ICLR 2026
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACI. 2024, 2024c.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: tuning-free asymmetric 2bit quantization for KV cache. Proceed- ings of the st International Conference on Machine Learning, 2024d.
Adrian Lańcucki, Konrad Staniszewski, Piotr Nawrot, and Edoardo M. Ponti. Inference-time hyper- scaling with KV cache compression, 2025. arXiv:2506.05345.
Meta. Llama 3.3 evaluation details.
11ama aacpa//mmeaaaaaaaaaaaaaaaaaaaaaa mdd 2024. GitHub repository
els/blob/ mtt//maiiiiae////m wtwiiin/moom////eemmmmmmmmeeeeee awa.mamammmmmmmmmmmm 3/033_33evv__
containing evaluation details for Llama 3.3 models.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Comea,, Pradeep Dubey, Richard Grisen- thwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, and Hao Wu. FP8 formats for deep learning. 2022. Kkiv:2209005432.
Mistral Al team. Mistral NeMo: our new best small model, 2024. BRLpstss://mitttt a
/news /tm//mmstttt/ stral- nemo Announcement of 12B parameter model with 128k context length, built in collaboration with NVIDIA.
Amirkeivan Mohtashami and Martin Jaggi. Landmark Attention: Random-access infinite context length for transformers, 2023. arXiv:2305.16300.
Piotr Nawrot, Adrian Lańcucki, Marcin Chochowski, David Tarjan, and Edoardo Ponti. Dynamic memory compression: Retrofitting LLMs for accelerated inference. In Proceedings of the 41st International Conference Machine Learning, 2024.
NVIDIA. nyCOMP, 2020. URL https:://gtthub http:/ub.ccomm//DA////oocooo GPU-accelerated compressoon/eccompession library.
NVIDIA. Faster/Tranfformer: A fast and efficient transformer implementation.
https://gith
ub hab..c////wiDDaaaeeesrrrrrrerros
2021. Apache-2.0 License.
NVIDIA. Accelerating load times for DirectX games and apps with GDeflate for DtrectStorage.
directx- games- and htt.::/pp......a...........oaaaa
pemps://devee 1 oper. awviiviii
mes- for
2022.
Accessed: 2025-10-03,
NVIDIA. nvCOMP
arks. html 2024. Accessed: October 2025.
Open RI. Open 1- httts::/huugingggee http:s//hgggnngacc....o.aastttte
enR1-Mathh 2025. A large-scale dataset containing 220k math problems with verified
reasoning traces.
OpenAl, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar. Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, et al. OpenAl ol system card, 2024. 11Xiv:2412.16720.
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz. Transformers are multi- state RNNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The Web datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
15

---

Published as a conference paper at ICLR 2026
Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation a KVCache- centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25), 2025.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don know: Unanswerable for SQuAD. 2018. doi: 10.18653 Ut/Attpp:://caaattttooooy..yg
2124/
RASIP working group. The 1.777 algorithm, 1997.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Coreea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Ven- mugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gau- rav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Ro- driguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Mi- cikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Mi- croscaling data formats for deep learning. 2023. arXiv:2310.10537.
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen Attention: Attention in low-rank space for KV cache compression. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Perey Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng. Zijia Chen, Yoshi Suhara, Shizhe Diao, Chenhan Yu, Wei-Chun Chen, Hayley Ross, Oluwatobi Olabiyi, Ashwath Aithal, Olek- sii Kuchaiev, Daniel Korzekwa, Pavlo Molchanov, Mostofa Patwary, Mohammad Shoeybi, Jan Kautz, and Bryan Catanzaro. LLM pruning and distillation in practice: The Minitron approach, 2024. arXiv:2408.11799
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: En- hanced transformer with rotary position embedding. Neurocomput., (C), 2024.
Hanshi Sun, Li- Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen. ShadowKV: KV cache in shadows for thghthhroughput long-context LLM inference, 2025. arXiv:2410.21465.
Vivienne Sze, Madhukar Budagavi, and Gary J. Sullivan. High Efficiency Video Coding (HEVC); Algorithms and Architectures. 2014.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. QUEST: Query-aware sparsity for efficient long-context LLM inference. In Proceedings of the 41st Inter- national Conference on Machine Learning, 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multi- hop questions via single-hop question composition. Transactions of the Association for Compu- tational Linguistics, 2022. doi: 10. 10.116///acl...0445.. URL https::///// pettts::/aclannee aaaaaaaaannnooooooooooo org
/2C22.tac1 1.31/
G.K. Wallace. The JPEG still picture compression standard. JEEE Transactions on Consumer Electronics, (1), 1992. doi: 10. 109/33. 125072.
16

---

Published as a conference paper at ICLR 2026
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. RedPajama: an open dataset for training large language models. NeurlPS Datasets and Benchmarks Track, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Yingquan Wu. Deflate compression algorithm, 2017. URL [patent/US$57765582
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating lan- guage model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024.
Ceyu Xu, Yongji Wu, Xinyu Yang, Beidi Chen, Matthew Lentz, Danyang Zhuo, and Lisa Wu Wills. :.5: Video codees are secretly tensor codecs. Proceedings of the 58th JEEE/ACM In- ternational Symposium Microarchitecture, 2025. doi: 10.1 10.1145/372288333755677 URL
https: 1tt.:/...org11111133333358881111 3756078
Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. Uncovering safety risks of large lan- guage models through concept activation vector. In Advances in Neural Information Processing Systems, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, et al. Qwen3 technical report, 2025.
arXiv:2505.09388
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-ho hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v1/D18-.. 1 1259. URL
logy org/D13
12597
Hong Yankun, Li Xing, Zhen Hui-Ling, Yu Xianzhi, Liu Wulong, and Yuan Mingxuan. SVDq: 1.25-bit and 410x key cache compression for LLM attention, 2025. a5X1v:2502.15304.
Jiyy Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the Twentieth European Conference on Computer Systems, 2025.
Sean Young, Zhe Wang, David Taubman, and Bernd Girod. Transform quantization for CNN com- pression. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. ISSN 1939- 3539. doi: 10. 1099///am.2021.33888.0 .2021 3084839
URL http: /ttt:://xx.oo doi org 1109///AAAMMMII
Jiay Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang. Zhiping Xiao, Yuqing Wang. Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native Sparse Attention: Hardware-aligner and natively trainable sparse attention. In Proceedings of the 63d Annual Meeting of the Association for Computational Linguistics, 2025.
17

---

Published as a conference paper at ICLR 2026
Rongzhi Zhang, Kuang Wang, Liyuan Liu, Shuohang Wang, Hao Cheng, Chao Zhang, and Ye- long Shen. LoRC: Low-rank compression for LLMs KV cache with a progressive compression strategy, 2024. arXiv:2410.03111.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang "Atlas" Wang, and Beidi Chen. H20: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, 2023.
Yilong Zhao, Yh Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikei. Atom: Low-bit quantization for efficient and accurate LLM serving. In Proceedings of Machine Learning and Systems, 2024.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, 2021. doi: 10. 10.18863/vvln 1/202 sitacl-main.477 URL
https: accl /aaps:///caaaaa antho ogy .r..org/ggoo........ 2021 naacc ma i nn. 472/
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), 2024.
18

---

Published as a conference paper at ICLR 2026
APPENDIX
A EVALUATION DETAILS
A.1 TASKS
For evaluation, we utilize Language Model Evaluation Harness (Gao et al., (024) and RULER
(Hsieh et al., 2024) with the Transformers library (Hugging Face 2025b) serving as an inference backend.
A.1.1 GSM8K
GSM8K Cobbe et al. 2021 is an established task for the evaluation of the reasoning of non- reasoning models (models without thinking phase (sthink>... </think>> before answer) Yang et al. 2025). We evaluate it in an 8-shot CoT setting with few-shot examples from (Wei
ct al. 2022). Following (Meta 2024) we allow for the generation of up to 1024 tokens. Task name in LM Evaluation Harness is gsmö k_cot.
GSM8K 8-shot prompt example
There are 15 trees in the grove. Grove warkers will plant crees in the greve today. After they are today? there will originally. many did atter some planted. theze were
ase must have been 21 15 6, The answez is 6.
many ia parking A: Thore are ariginally 3 cars. 2 more care arcive. 2 5. The answer is 5.
It there lott
parking lot and
nany Originally, tesh had chocolates, Bez sister had 42, Bo in total they had 32 + 42 14, After + cating 35, they had 74 35 39. The answer 39.
chocolates
had
they have left
Jason 1ollipopa. He gave Denny some lollipops. Jason 12 lollipopa. Now many lellipaps did Jasca giva to Derny?
Jasor. started with 20 lollipops. Then he had 12 arter giving come Denny, So gave Derny 20
Q: Blave bas Live Loys. For Che Chrisima,, be got Lvo Loys each from his non and ded. Hlw many Loys does is
b be have now?
Shawn. started
toys. 5. ensver 9.
from his dad,
Thore vare nine computers in the server zoom. Five more computars vere installed each day, fzon How nany corpitesa vare oriainally computers.
the daya, noze computora ware added. computers were adied, 9. 20 is 25, The ansver is 29.
acnday
Hichael had 58 qoif balls. zeaaday, last have at the
balls. ca wodnesday, amdneaday?
many golr balla
A: Nichaul acarted with gold balla. Aftez losing 23 on Luesday, he had 23 35. Aftes Losing 2 noze, ba had 35 = 33 colf balls. The answer is 33.
has 523.
bought five bagels for $3 each. How much money dowa have left?
A: o1 ivia had 23 dol laa... 5 begels for dallars eech will be 5 15 dal lars. So she has 23 15
dollars left. 23 15 is 8. The anaver is 8.
(OURSTTONS
19

---

Published as a conference paper at ICLR 2026
A.1.2 MMLU
MMLU (Hendrycks al. 2021a) is collection of multiple-choice questions spanning 57 subjects. For MMLU evaluation, we use 4-shot
coo_ffewhoot from LM Evaluation Harness, allowing for generation of up to 256 tokens. We pick this setup instead of the perplexity- -based forward-pant -aass evaluation of MMLU, because our baselines perform a prefill using full precision KV cache. Therefore, for fair evaluation, we need make the model generate a non-zero number of tokens before providing the answer, otherwise the performance would be identical to vanilla.
MMLU 4-shot prompi example
The tollowing are mu-tiple choics questions 1with answere) szout abstract aigebre, Qt Stazement 1 group. Statement 2 eymetcic
Every element group
group
(A) true, Trus (BI Faise, Foine (C) true, Falcs (DI False, True
At let think step by step. A oyelic group is group that is generated by single elenenc, Hence
subgroup generated by single elenent group is eyelis and Statement ix Prue. The (C).
The symetzic group 48_nd has
actorialiniS eiements, hence Ptaracteristio
95_11019
ring
(2) & (B) 3 (2I 12 (D) 30 At let'e think step by step, charscteristic at a ring is in Sns it the statement 9ka 05 for all Sakin 285 les that 9k8 mulsiple of 9na, Assume that Sia as for all Salir 275 for In particular
answer (AI.
statoment 11 Every function fron finite set onto itself must be one to ene. Statement Every Teue,
sbelian False,
abelian.
True, Falcs let's think stop by step. Statemant i is true. let 938 be a finita set, iI 62:8
ichearrow 55 is cate functicn, then $181 1218) 18. It 9r0 was not one to ene, then for tirite than
donsin 959 atazoment truc.
elenents, subgroup of abelian group 6H5 is abolian. let 9a,5 Vin #6, ther 93,b Min 65 and Sab-naa, Since 968 is abeliar, bab-bag. Since $a8 is subarcup of 628, sab vin 86. Therefere, sab-bas and 588 abelian. The ansver is
contradiction.
show
Q: Statement 1 if all is an slenert of factor qreap, then [a8l divides (al. Statement 2 I2 and + K aze subgrcups of G then as is subgroup of G. True Let's think stop by stop. Statemant eycle (11,2) 9 and SK5 be a subçroup of 95_35 generated by the cycle 811,318, 31,, Both 685 and SK5 tave tиo elements, the generazors and the ideatity, Rovever 5885 ccntains cycles (1,2), (1,31 and is
False
True, Fales
Feise, 988 ba subezoup 98_35 generazed by
12,3,1), subgroup. ansxer (B)
(QUESTION) think
20

---

Published as a conference paper at ICLR 2026
A.1.3 LOST IN THE MIDDLE
Lost in the Middle (Luu al. 2024a) evaluated in O-shot 100-keys (300 keys for Llama 3.3 70B) setup, allowing generation of 64 tokens using the prompt presented below. We implement the evaluation using the LM Evaluation Harness framework and utilize the UUID strings and code from (Liu et al. 2024a). This benchmark allows for methodical testing of the model's s ability to access imput context. an evaluation metric, we utilize the exact match between the UUID returned by the model and the gold answer.
Lost the Middle question example (base models)
Extraet the value corsesponding to the specitied key in the JSON abject belox.
tod
1ee05---100----aee 3d3 16c64991,
18 1883322bb-51 571. 4w55-4934 #x8dF11b3349*:
*c54095ef -951 9951--99-666
848h 109 F72",
-oes 19904 34d55641
#=c2c510-2d00-1-e0 2:-22-20--00000022---- 21a0022222210111000001 (ba85clea'"'
Lost in the Middle question example (Llama 3.3 70B instruct)
<bberin_o__tert pcrtttct >oystoms haddderdd
Cutzing Knowledge Dases Decanber 2023
Today Date: 2024
aaaaaooo__ Rxtract corssaponding the specifled key in sbject below.
JSCN data:
("24650470- (eet--t-----ee 4e53-8364 18881188558837
"u4852667-4D4E 40ce C146553776791,
-62120833-2255-442222
99003300-77922-68 1------0-------- 42733-4400422Smms"
"65688427-9155 44233-5---2------11---11-33 400---222--add--3a4333333333
"c1736979-584d (-2-------------0000-00000 "d:b5asce-72b2 222-2222--22-2222 4228--bo2---faaaee------
What the value associated che key +24850474
4653
16836686647d" Answes us ling
following Cosmat:
"The
key 22850447 [w61 4c53 1485658d547d"
REBR
Where ANSMER HEAR is the va loe essec ated with the key
< 22880447 tw6: 4e53 8844 1d85S6BUE4Te* (eSS5BeeaT"..... vs id] start, Jheader___rrr_reeeeerrr Pessisaant Jend headesed
21

---

"here and back again, There and Published as a conference paper at ICLR 2026
A.1.4 VARIABLE TRACKING
Variable Tracking is evaluated in I-shot (with relatively short example shown below) using RULER (Hsich et al. 2024) with context length 8K, limiting the generation to 128 tokens. The benchmark the model's 's ability to track variable assignments across unrelated contexts.
Variable Tracking prompt and question example
Momorizo and track the chainis; variable assiganent hiddon rellcwing
The grasa is green. The sky in blue. The sun in yallow, Baze va eo.
There and back aqain.
The graza is green. The sky is blce. The sun is yellow. Beze
There and hack again. agaia.
graas yreen.
is ywllow.
gu.
grass green.
sky blce. The aun ia vellow. Beze go. Chere and back aqain.
The gress i5 green. The sky 15 bles. The cun 15 yellow. Here we 20.
Thore and back aqzia.
The
yreen. The sky La blue. The sun is ywllow. Bleee JUP 37438 The again.
There and back again. Thare and back
is
blue. The
The greas is green, The sky la blue. The sun is ywllow. Bere au go.
There and back agaia.
The grass green. The sky is blce. The sun is yellow. Beze
There and back aqain. again, grass green. sky blue. The aun is ywllow. Bere go. Thare and back again.
gress green.
sky
is ywllos.
gu.
The grass is green. The sky is blca. The sun 1s vollow. Heze we 20.
Thore and back aqzia. There and back agsin. and
The
The sky is blue. The xun is yellow. Heere
ADD
grass green.
yellos.
again.
The greas ix green, The sky is blue. The sun is ywllow. Bleee on gu. There and back again. The grass green. The sky ia blce. The aun in yellow. Bizee
There and back again. again,
gress green.
sky
is yellow.
go.
sky blue. The sun is The grass is green. The sky ia blce. The sun 1s vellow. Hoze we 20. There and back agaia.
graas green.
Beee gu. There and back again.
VAR KCB VAn aGn "he
greer. The sky blue. The is yellox. llere
There and
again.
The grass 18 green. The sky is blue. The eun is yellow. Hlere go. There and back again, The gress 18 green. The sky is blue. The xun is low. Hlere
"here and back again, There and batk again. again, go. Phere and back again,
The
.yrem.. The sky blue. The sun is ywl luw. gress green.
Bleee
sky
go.
gress green. sky blue. The aun is yelyyll low. leere
The greas is yreen. The sky ia blue. The sun is ywllow. Bere
gu. Chere and back agaia,
The
The
blue. The 15 yello.. llere
"here and agsin, go.
grwes green.
sky
yellmx.
back sgain.
The gress 18 green, The sky is blue. The sun is yellow. green, The sky is blue. The is yelluw. llere
low. ltere
VAR 19P VAn LJP The
again.
Quest ion:
Find all variables that are ass igned the value 97498 in the text above. Ansuers : According the chain(s) variable aseigrment they
5 varisbles assgined the value
Nomorize the choinis: or variable aesigament hidden rellewing text.
Quastion: Find all veriables assigned 79092 (h) variable assignment above, variables are assgined the value 19092, they are:
Answeri According
22

---

Published as a conference paper at ICLR 2026
A.1.5 NEEDLE IN HAYSTACK
Needle a Haystack (NIAH) (Kamradt 2023) is evaluated 0-shot using RULER (Hsieh al 2024) with context length 100K for Llama 3.3 70B and 8K for other models, limiting the generation to 128 tokens. The benchmark tests the model's ability retrieve information hidden in long text. The name of the task in the RULER is niah single.2.
NIAH prompt example
begin wwegin_of__eeee (ext]><]ssee Seader_ Payssene beedee_ idlp
Cutzing Knowladge Date: Decenber 2023vaToday
2024
eot_idd>>< (ad__idds//ttttt //
special negic you about nunbex aflerwards.
biddeen within following
menorize will
Lext prefixt soaedle>
deassy text suffiso
What ls the special nagic namber for abrasive peliulogy awat ioned in Lh provided text? The special nagto for abrasive-gsthol mentlened. provides diseee header__ddj>
isclent
A.1.6 QASPER
Qasper Dasigi et 2021 Shaham et al. 2022 is evaluated in 2-shot using LM Evaluation Harness task scrolls saeper with full autoregressive generation and Fl score (same reasons as with MMLU evaluation). We limit the generation to 128 tokens. This benchmark evaluates the model's ability to answer questions about paper presented as part of the input.
A.1.7
ADDITIONAL TASKS FROM LONGBENCH
We use the LM Evaluation Harness' implementation of LongBench's 2WikiMultHHoPQA (Ho al. MultiFielQQA (Bai et al. 2024). MuSiQue (Trivedi et al. 2022). QMSum (Zhong et al.
2020
2021
SAMSum (Gliwa et 2019). We fix the double question error using
Question:
instead of
Question:
Question:
and end the generation after new line for 2WikiMultiHopAA, MultiFieldQA, MuSiQue and SAM- Sum, whereas after double new line for QMSum. Evaluation is performed in single shot setup.
A.1.8 ADDITIONAL TASKS FROM RULER
We use the LM Evaluation Harness' implementation of RULER's CWE/FWE, HotPotQA YYagg al. 2018) and SQuAD (Rajpurkar et al. 2016 2018). We evaluate the models zero-shot.
23

---

Published as a conference paper at ICLR 2026
A.1.9 AIME 2024-2025
We implement AIME evaluation using LM Evaluation Harness. Following (Lańcucii et 2025) we utilize prompts adopted from the Open-RI repository (Hugging Face 2025a) and limit the gen- eration 30K tokens. AIME competitions are popular for the evaluation of reasoning models such as DeepSeek RI (DeepSeek-Al et al. 2025). To check the correctness of an answer, we utilize the following code with Math- Verify (Hugging Face 2024):
AIME 2024-2025 evaluation code
zxon nath-verify wer.eyy.strric inpert matkunstric
erom wor.tyyppacee import Latoxsstraaccooooolll. Expeaktrractooniontii
ceade_anewez (pzoblen, nodol_ancwez):
Le_die = Palse
veriEy_fune nach_natzie(
LelexBettract iontonff it elae Begr
aggrngal fuw (Expeestracciontoooiaar LacosBaaraccioonoooiiiaa..
pracision
gald prublen "andwwe"
try:
with Limsoul 1aeuerda=331; ouason class Lo Uleoww and except lon LI code does aud complete undez 20 socenda
grade, annowwess ver [ggoee) [modl [anwe:)) reluzn grade
excegEt
relurn Fallow
AIME 2024-2025 prompt
begin_of. Serginnn____eeennne Leeeteeceeeeeee t>x (><]Ueer|sddl the following math problen ee efficiently and clearly:
simple proaiems stepe
Provide consise solution with mininal explanation.
For complex problens Usa stop-by-stop tormat:
morel:
#4 Step 1s (Conciss description]
[Briot explanation calculatione]
#t Stop 2: [Canciss descriptisa]
[Brict explanation and calculatione]
Regardless at the approsch, always conclude with:
Tharetore, the
stboxedtanexeriii hope correct.
Whare [аnлжer] is just the final aumber or expression that solves the problem.
Problam: (FROBLEM;
AlAsiistant l><thinks
A.1.10 LIVECODEBENCH
For LiveCodeBench (Jain et 2025) evaluation, utilize the official repository and, following (Lancucii et al. 2025 limit the generation to 16K tokens and data range from 2024-08-01 to 2025- 01-31. The benchmark consists of problems from sites like leetcode.co and codeforces.com.
24

---

Published as a conference paper at ICLR 2026
A.1.11 MATH-500
For evaluation on MATH-500 (Lightman et al. 2023). subset of MATH (Hendrycks et al. 2021
introduced by Lightman et al. (202)), limit the generation 5120 tokens following (Meta) 2024). We also change the sliding window size to w 256. We utilize the following prompt adopted from MATH-500 evaluation, and the following code optimized for reproduction of Llama 3.3 70B results without the LLM judge:
MATH 550 eval code
from math_veriffy import parse, verify
def answer somaliieeanswerr str) -> str:
answer auswer.reelacce((""\ddt"''' "")
[-1] spp..ttpiit"l....... (")) .trrppppppp ()
answer answer. "aaoor..cgplacce(""""rghhh"" "")
answer answer. answor.rogplas011l [rourr_cccoaassterooooooaaaaaa, "")
answer answcr. (0w.c...cplccco(rocooccaggggggg "")
апawer answer answer. onseer..rgllaaccrcccoeaiiieeeeee
"")
answer anawer. anseee
"")
aпswer answer ( ""*
answer answer (F
if answer answer.start.wwa. sttarsswitt((("""eex"
answer
(1" "")
answer ("""err "")
if answer starrartttttssttt((xx1"1""
answer answer.replace (r"x)in") "")
if stweer.taarssitb(((y==)))) answer answer. reswerrreplace "")
return answer
def sampare_answera gold = parse (answer)
str, model_answer: str) bool:
model = parse moodel_answer
res (gold, model)
if not res:
answer answerrnoommllieennnsse
model_answer = answer_normal (model_answe)
print (answer, model_answer)
gold parse
model parse (model_answerz)
if not verify model): # sometimes fails for improper expressions
verify
gold answer model model_answe
if res:
return True
else:
return False
else:
return res
25

---

Published as a conference paper at ICLR 2026
MATH-500 prompt
Cotcing Knowiadge bace:
Today Date: 26 du.. 2024
</cot_id|>< sttrrt >users snf_hcad:r_idd>>
Bolve the following nath preblen efficiently and ciearly:
simple
consise solution with tewerl: explanation.
For complex problens 13 stepa or mozet:
Jea stap-by-stap tornat:
#t Step 1: [Cancise description]
[Brict explanation and calculatione]
atap 2: [Cancise description] [Brict explanation and calculations]
Regardleas of the approachy alwaya conclude with:
Theretore, the final
mncerss, hope
Where [answer] is just the tinal aurber or expression that selves the problom. Problen: (FROBLEM) < (dst_ Sttaatat__aadde____ pesslatants [
26

---

Published as a conference paper at ICLR 2026
A.2 BASELINES CONFIGURATION
A.2.1 KIVI
We follow (Liu et al. 2024d Section 4.1) and use group size = 32 and residual length= 128, with 2-bits per key and 2-bits per value. We utilize the official implementation.
A.2.2 GEAR
We follow (Kang al. 2024 github repository and underlying quantization to KIVI with group size 64, streaming gap 64, 2-bit keys and values. Additionally, we set the rank of key/value correction for prefill and for generation. We utilize the official implementation.
A.2.3 XKV
We follow (Chang al. 2025 For Llama 3.1 8B group layers by and set the pre-RoPE key rank to 512 (compression ratio 8) and value rank to 768 (compression rank 5z). For Mistral NeMo and MN-Minitron, and set the pre-RoPE key rank to 640 (compression ratio 8) and value rank to 960 (compression rank 5.).
group layers by (those models have 1.25 more layers than Llama 3.1 8B)
We utilize the official implementation.
A.2.4 H2O
We utilize the official implementation of Zhang al. 2023) and set the recent and heavy hitter fractions to Io of (input + max possible output size). This results up to an 8x compression ratio. Lower compression ratios occur when the model produces shorter outputs than the maximal specified value. We note that this can give an advantage to H2O over other methods.
A.2.5 TOVA
We utilize the official implementation of (Oren et al. 2024 and set the max cache size to of (input possible output size). This results in up 8x compression ratio. Lower compression ratios occur when the model produces shorter outputs than the maximal specified value. We note that this can give an advantage to TOVA over other methods.
A.2.6 DMS
We report the results for DMS published in Lancucki al. (2025
A.3 HARDWARE
Experiments were run on a node with 8 NVIDIA H100 GPUs (80 GB each). All main jobs finished within except MMLU and Llama-3.3-70B (< 8 h) and xKV (< 12 h). For baselines (KIVI, GEAR, xKV, TOVA, H20) we used the authors' public code. All runs used batch size (except Qwen and vLLM evaluations) prevent padding that could bias some of the baselines.
27

---

Published as a conference paper at ICLR 2026
B ABLATIONS AND ADDITIONAL DETAILS
We provide additional details and ablations related to kvtc the following appendices:
Appendix B.1
Additional details about the hyperparameters of kvtc, along with justifications and references to relevant ablation studies.
Appendix B.2
Properties of key and value channels.
Appendix B.3
Omitting sink tokens for KV cache compression result in significant gains
at higher compression ratios.
The difference compressibility between keys and values suggests that long- context retrieval tasks may benefit from using higher precision for keys.
Appendix [..4
Appendix [.5
Increasing the amount of calibration data helps preserve performance at higher compression ratios, while smaller amounts remain competitive for lower compression.
Appendix B.6
Influence of the calibration data domain on downstream performance.
Appendix
8.7
Influence of the sliding window size downstream performance.
Appendix
B.8
Ablation of lossless compression algorithms.
Appendix
B.9
Benefits of DP quantization over pure PCA.
Appendix
B.10
Benefits of cross layer PCA.
Appendix
B.11
Ablation per per-prompt vs one-time PCA
Appendix
B.12
Results from T with standard errors computed by LM Evaluation Harness
(Gao et al.
2024).
Appendix 2024).
B.13
Additional results on RULER (Hsieh et al. 2024 and LongBench (Bai et al.
Appendix B.14
Sizes of PCA projection matrices (stored per model), with an emphasis the fact that their size is only small fraction of the model parameters.
Appendix B.15
Influence of sliding window on cache size.
Appendix 1.16
Latency evaluation a simplified scenario with vLLM (woon et al. 2023 and
LMCache (Cheng et al. 2025).
Appendix B.17:
Pseudocode for the Dynamic Programming (DP) precision assignment algo- rithm, along with complexity analysis and a sketch of the optimality proof.
28

---

Published as a conference paper at ICLR 2026
B.1 PCA CALCULATION PARAMETERS
As mentioned in the main text, we utilize 8 iterations of the randomized algorithm from (Halko et al, 2011] for PCA. We utilize 160K calibration tokens for Llama 3.1 8B, Llama 3.3 70B instruct, Mistral NeMo 12B, and MN-Minitron 8B with a dimensionality cut-off of 10K. This choice is motivated by memory efficiency and the results presented in Figures 6 and IT where the initial boost from the increase in the number of calibration tokens from 10K to 1OOK is relatively large compared to the boost attained when increasing from 100K to 160K. We leave the exploration of larger calibration sets for future work. In Appendix [B.5 we ablate the influence of the amount of the calibration data on downstream results. Appendix B. 1.14 we provide the sizes of the projection matrices, noting that they are relatively small when compared to the model size (2.4% of model parameters for Llama 3.3 70B)
For Qwen models, due to their smaller number of KV heads, utilize 200K calibration tokens and dimensionality reduction of 8K. For all models (unless stated otherwise), we utilize a 50/50 mixture of Web (Penedo et al. 2024) and OpenR MMth traces (Open RI 2025 due to the duality of our benchmarks (reasoning and general purpose; see Appendix [B66] for an ablation on calibration data source). We take samples from both datasets with the only filters being minimum and maximum length (IK and 32K, respectively) for both datasets and quality score (2 0.95) for Wen (the quality score is attached to the dataset). Additionally, we ensure that the number of tokens from documents below 8000 and above 8000 tokens is roughly the same (except in the MN-Minitron case, as this model supports up to 8K context length). Token counts and cutoffs are chosen so that a single calculation of PCA fits on single H100 GPU with 80GB of memory and completes within 10 minutes (for details see Appendix B.5).
We emphasize that for a given model, we use the same PCA matrix for all compression ratios. The only change between compression ratios the precision assignment, which done automatically via a dynamic programming algorithm. Additionally, we note that the manual adjustment needed by a practitioner to adapt kvtc to a new model is limited to choosing the initial PCA dimensionality cutoff (if the practitioner decides to use the efficient algorithm by (Halko et al. 2011)), the number of samples that the chosen PCA implementation can handle (based on GPU/CPU memory) and the calibration sample choice if special scenarios are desired for increased compression. In this paper, we note that across a wide range of tasks, the choice of a 50/50 mixture of both short and long context Web (Penedo al., (024) (for generic text) and OpenR IMath (Open RI (2025 (for thinking traces) data is sufficient for Llama, Mistral, and R R1-distilled Qwen models for variety of applications. In particular, we note that our method is not harder adjust than xKV or SVDq, due to automatic precision assignment via dynamic programming. In particular, instead of aiming for the best reconstruction error for a specific compression ratio, the user can easily alter the algorithm to aim for the highest compression ratio within given reconstruction error constraint.
29

---

Published as a conference paper at ICLR 2026
B.2 ADDITIONAL DETAILS ABOUT K CACHE PROPERTIES
We study the relative channel activation patterns of Llama 3.1 8B, Mistral-Nemo 12B and Qwen 2.5 RI (Figures] and 8). The relative activation results are computed on 50/50 mixture of FineWeb (Penedo et al. 2024) and OpenR Math (Open R 2025), with document lengths between 1K and 8K tokens. We observe that keys and values of all models show potential for dimensionality reduction and quantization (low absolute activations and variance).
Liama 3.1 8B
Liama 3.1 8B
Keys
Values
Keys
Values
Head Index
Head Index
Head Index
Index
Channel Index
Channel Index
Channel Index
Channel Index
0.0
0.2
0.4
0.6
0.8
T.0
0.0
0.2
0.4
0.6
0.B
T.
(a) Relative (within head) Absolute Activations
(b) Relative (within head) Variance of Activations
Mistral-NeMo 12B
Mistral-NeMo 12B
Keys
Values
Keys
Values
Head Index
Lo0
Head Index
120
Head Index
Lo0
Head Index
120
104
100
100
Channel Index
Channel Index
Channel Index
Channel Index
0.0
0.2
0.4
0.6
0.8
T.0
0.0
0.2
0.4
0.6
0.8
T.0
(c) Relative (within head) Absolute Activations
(d) Relative (within head) Variance of Activations
Figure 7: For key/value head and channel j. we define per-channel mean absolute activation a. = 1 D /channele,cg and plot the relative activation heatmap,,, = az./ max, (av,o) - Panels where varo,,j is the variance of channely,,j over t.
(c). Panels (d). show maxe [varc,.e]
varei.j
30

---

Published as a conference paper at ICLR 2026
Qwen2.5 1.5B
Qwen2.5 1.5B
Keys
Values
Keys
Values
Head Index
Head Index
Head Index
Head Index
2000 90
D. an
Channel Index
Channel Index
Channel Index
Channel Index
0.2 0.4
0.2 0.4
(a) Relative (within head) Absolute Activations
(b) Relative (within head) Variance of Activations
Qwen2.5 R1 7B
Qwen2.5 R1 7B
Keys
Values
Keys
Values
Head Index
Head Index
Head Index
Channel Index
Channel
Channel Index
Channel
0.0
0.2
0.4
0.6
0.8
T.0
0.0
0.2
0.4
0.6
0.8
(c) Relative (within head) Absolute Activations
(d) Relative (within head) Variance of Activations
Figure 8: For key/value head and channel j. we define per-channel mean absolute activation a. En /channel,,cl and plot the relative activation heatmap:,; = az./ max, (au,o) - Panels (o). Panels (f), show vare;,;
max, [varc;,c] where vare,,, is the variance of channel,,, over t.
31

---

Published as a conference paper at ICLR 2026
B.3 EXCLUDING SINK TOKENS FROM COMPRESSION
We consider the first four tokens of key and value caches (i.e., tokens at positions 0, 1, 2, and 3) be sink tokens, motivated by the experimental results reported by (Xia et al. 2024). Reducing the dimensionality of key and value caches with PCA causes larger information loss of the initial tokens than the remaining ones, as shown in Figure two setups: one that excludes first four tokens from compression (denoted kvtc* and one that compresses them along with other tokens (denoted kvtc The results are shown in Table 6 High compression ratio of 64x drastically reduces downstream task scores for the Llama 3.1 8B the kvtc case; for MN-Minitron and Mistral NeMo 12B causes regression on the long context tasks (Lost in the Middle and Variable Tracking) when compared with kvtcl.
order assess the influence, we compare
Table 6: Ablation the skipping compression of the first four tokens. We note that the difference starts to be visible with larger compression ratios, and that it is in favor of skipping compression of potential attention sinks (Xiao et al.) 2024). Results are presented as scorestderr where stderr bootstraped by LM Evaluation Harness (where available) (Gao et al. 2024). 'he differences in compression ratios, are due to the fact that count the skipped tokens into compression ratio.
GSM8K
MMLU
QASPER
LITM 100
RULER-VT
Model
Method
CR
Math & Knowledge
Long Context
kvtc1ex A0
19-22
56.81.4
60.3o.4
40.2
99.2o.1
98.2p.4
kvtclex
18-22
56.91.4
60.
40.7
99.3p.1
99.1p.3
Llama 3.1 8B
kvtc64x
P0
76-90
1.60.3
9.90.2
27.6
0.0p.o
61.81.5
kvt.64x
60-88
57.21.4
60.7o.4
37.8
90.2p.4
95.90.8
MN-Minitron 8B
kvt C64x kvtos4x
79-97
59.51.4
61.9g.4
37.9
55.1g.6
91.5p.9
53-95
57.81.4
62.1o.4
38.1
59.5p.6
93.4p.a
kvtcolx kvtc64x
77-89
61.91.s
62.4g.4
37.5
84.8p.4
95.8p.s
Mistral NeMo 12B
51-87
61.91.3
61.40.1
38.0
95.30.3
98.0o.4
32

---

Published as a conference paper at ICLR 2026
B.4 SEPARATE ADJUSTMENT OF COMPRESSION RATIOS FOR KEY AND VALUE CACHES
All experimental results of kvtc (unless stated otherwise) have been obtained with 1:1 compres- sion of keys and values. We present additional results of manually adjusting the compression ratio separately for keys and values (Table 7 The results suggest that for long-context retrieval tasks, the value cache could compressed more than the key cache. We attribute this phenomenon the necessity to precisely attend to selected tokens in the cache, which hinges on the high accuracy of key vectors. On the other hand, stronger compression of values shows noticeable degradation on the GSM8K and MMLU tasks.
Table Ablation of key/value compressibility with kvtc. We the same kvt configuration as in Table 2 but independently change key and value lossy compression rates. To denote that value compression ratio was set 32 and key was set to 64 we use kvtc
vi32x
Method
CR
GSM8K MMLU QASPER LITM RULER-VT
Llama 3.1
kvt
k:256x
55-80
57.1
57.4
36,6
48.61.6
91.9g.9
kvt ke V/260
55-77
56.81.4
57.5
36.3
71.91.4
95.9g.6
Mistral NeMo
12B
kvtc 0
69-79
62.21.3
61.7
34.6
73.21.4
90.40.9
k:256x 0 ka22
kvtc
69-77
57.11.4
59.8
35.2
77.51.3
89.51.o
33

---

Published as a conference paper at ICLR 2026
B.5 CALIBRATION DATA TOKENS DOWNSTREAM PERFORMANCE
Mistral NeMo 12B
We test how the amount of calibration data affects cal- ibration times and the downstream performance. From Figures [ and [T we already know that increasing the amount of the calibration data can bring down the recon- struction error. The question that remains is how such an increase correlates with the downstream performance. In Table 8 we show that an increase in calibration data clearly benefits a high compression ratio 256x, with more moderate returns for smaller (64x, 32x) compres- sion ratios. This a positive result, as it allows for trading calibration stage complexity for improved down- stream performance, with 40K token budget bringing al- ready competitive results for 64x ratio. We additionally note that PCA calibration can be completed within 1.5 minutes for 160K tokens using (Halko et al. [201]] algo- rithm, and that the DP calculation time (performed once per model and compression ratio) can be finalized within minutes.
4.6
Perplexity
4.4
KVICCR
4.0
Calibration Tokens (K)
Figure 9: Model perplexity under differ- ent CRs and calibration token budgets.
Table Ablation of the number tokens used for kvt calibration, along with respective PCA and DP calibration times. PCA calibration was performed using single H100 80GB GPU, calculation of DP tables (except simulation of quantization) was offloaded to the node cpu. We additionally limit DP calibration data to first 32K tokens, therefore we do not see increase in dp time after 40k calibration tokens. All calibration datapicces come from a 50/50 mixture of FineWeb (Penedo et al. 2024) and OpenMathRI (Open RI 2025) traces between IK and 8K tokens. Reconstruction error and perplexity calculated on n a held-out 50/50 mixture of Fine and OpenR Math data.
Calibration
Error
Downstream Accuracy
Method
Data
CR
PCA DP
Key
Value
PPL
GSM MMLU
QASP LITM VT
AVG
Mistral NeMo
12B
kvtc
6.5m
0.184
0.365
3.864
31-42
63.0
63.9
34.8
97.3
98.9
71.6
kvtco4x
20K
41s
5.4m
0.222
0.440
3,993
63-87
63.9
61.7
31.8
88.8
96.9
68.6
kvtC2sex
3.9m
0.293
0.565
4.696
148-340
59.6
51.5
23.3
10.4
66.8
42.3
kvtes2x
7.2m
0.172
0.341
3.828
31-43
64.2
63.0
35.3
98.7
99.5
72.1
kvtco4x
40K
48s
6.1m
0.212
0.421
3,958
63-88
63.7
61.9
32.4
95.7
97.8
70.3
kvtC
4.6m
0,285
0.555
4.865
148-344
58.3
50.1
23.7
9.2
70.6
42.4
kvtes2x
7.2m
0.163
0.322
3.798
31-43
63.6
63.9
36.0
98.0
99.3
72.2
kvtco4x
80K
60s
6.1m
0.204 0.40S
3.934
63-88
63.3
62.0
32.2
94.5
97.8
70.0
kvtC256x
4.6m
0.279
0.543
4.626
148-339
60.5
51.7
23.3
13.6
83.7
46.6
kvtcs2x
7.2m
0.156
0.308
3.766
31-43
63.2
64.3
36.6
99.5
99.1
72.6
kvtc6ix
160K
86s
6.1m
0. 198 0.394
3.901
63.2-87
6-1.6
62.2
32.8
96.9
97.7
70.8
kvtC250x
4.6m
0.275 0.535 4.462
148-441
61.4
55.5
21.3
34.2
79.5
51.0
Correlation (Pearson)
PPL
AVG Score
Rec Error Key
0.959
-0,988
Rec Error Value
0.953
-0.939
PPL AVG Score
1.000
-0.992
-0.992
1.000
34

---

Published as a conference paper at ICLR 2026
Llama 3.1
Mistral-NeMo
Kny PinelWeb 0genRtl niter=0
Kry FineWebb - OpenRt niter=0
Belaliee Resonsttyccon Exto
0.16
OpenRl aiter=1
Reletive Regonstrrctoon Errve
0.21
Key FineWeb OpenRl
Kny PinelWeb Web niter
Kry FineWeb- - PineWeb niter
0.14
FinelWehh FineWeb 1
Kay FineWWih- niter-1
0.20
0.12
0.18
0.16
0.00
0.06
0.10
20
40
60
80
100
120
140
160
20
40
60
80
100
120
140
160
#Calibration Tokens (Thousands)
#Calibration Tokens (Thousands)
(a)
(b)
Liama 3.1 8B
Mistral-NeMo 12B
0.425
Value Web OpenRi niter=8
Value Web OpenRl niter=8
Rebetvve Rocor.trsetoon Eireor
----
Value Fine Web
Openfi nitersi
Rocon ctn.rtattiion Blrvor
0.425
Value Fine Web Opmnk1 niters
-+h Vallan FineWeb- Web
FineWihh nitre
Value WineWWb FineWish niter=
PineWeb
0.375
0.350
0.350
0.325
0.325
0.300
0.300
0.75
0.275
0.250
100 120
160
#Calibration Tokons (Thousands)
#Calibration Tokens (Thousands)
(c)
(d)
Figure 10: Relative reconstruction error when calibrating kvtc decorrelation step ablation of the number of algorithm (Halko et al.) 2011) iterations. Figures (a)-(b) show key reconstruction error, whereas (c)-(d) show value reconstruction error. Other parameters in Figure 6 We note that the larger number of iterations provides slight improvements.
Llama 3.1 8B
Mistral-NeMo 12B
0.6
0.6
Value OpenR OpenRl
OpeRt
Vallan OpmRI
FineWeb
Relative Recornstruciion Emorr
Value OpmRI FineWeb
0.5
Key Openk] OpenR]
0.5
OpeRt OpenR1
Key OpenRt FineWab
Key OpeRt FineWeb
Relative Reconng
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
20
40
60
80
100
120
130
160
0.0
20
40
60
00
100
120
140
180
#Callaration Tokens (Thousands)
#Calibration Tokons (Thousands)
(a)
(b)
Figure 11: Relative reconstruction error when calibrating kvtc decorrelation step on OpenR Math
(Open RI 2025) traces with the eтrоr calculation on FineWeb (Penedo al. 2024). Parameters as in Figure 6 We note that OpenR Math traces were published after the release of Llama 3.1 8B and Mistral NeMo 12B, and possibly due their specificity result in higher generalization error.
35

---

Published as a conference paper at ICLR 2026
B.6 CALIBRATION DATA DOMAIN VS DOWNSTREAM PERFORMANCE
To examine how the calibration data domain influences downstream performance, we prepare two additional versions of kvtc for Mistral NeMo 12B: one using only Fine data and the other using only OpenR Math data. Table 9 shows that using OpenR Math calibration data better maintains MMLU and key-value retrieval scores than Fene higher compression rates. This improvement is likely related to the question-thn teae-tthin--sssse with the evaluation tasks, compared to the more general web collection nature of Febe However, for 32x compression, both choices remain competitive. We note that for 256 compression, the 50/50 mixture of FineWeb and OpenR Math results in the best scores. For reconstruction errors, regarding calibrating on OpenR Math/Fine Web and testing on Fen//pe/O Math, see Figures 10 and
structure of OpenR Math, which may be more aligned
We additionally check the standard deviation of task scores when sampling different calibration sets (see Table IO). We observe that kvtc is relatively stable when sampling 160K tokens - a 50/50 mixture of Web and OpenR IMath data as a calibration set. We also briefly check the influence of calibration data length distribution on downstream performance, showing the results in Table 11
In Table 12 we test kvtc calibration under strong domain shifts. We observe that using Python, C, or Assembly from StarCoder (Li et al., 2023) instead of 50/50 mixture of Web and OpenR I Math data results in performance degradation on GSM8K. However, we note that despite a strong domain shift (code vs natural language). the model retains in-context retrieval abilities (LITM, RULER-VT) and for cr 16x obtains scores that are either on par or better than KIVIGEAR 2-bit (except the case when we calibrate on C). We note that files contained in StarCoder consist of both code and comments, which can potentially explain retention of lingual abilities.
Table 9: Ablation of the source of data. We consider kvtc calibrated fully on Web (Penedo et al., 2024), fully on OpenMathR1 (Open RT 2025) and 50/50 mixture. For all cases we utilize a 50/50 mix of documents between 1K and 8K tokens along with documents between 8K and 32K tokens. We calibrate using 160K tokens.
Method
Data
CR
GSM8K MMLU QASPER
LITM RULER-VT
Mistral
NeMo 12B
Web + OpenRIMatt
31-43
62.21.3
63.8a.4
37.5
99. 99.6g.1
98. 98.7a.4
kvt
Fine FineWeb Web
31-43
63.51.3
63.5u.4
37.5
98 5..2
98.9u.3
OpenR Math
31-43
63.51.3
64.80.4
37.8
99.7o..
99.3o.3
Fine Web + OpenR Math
51-87
61.91.3
61.4u.4
38.0
95.3g.3
98.0u.4
ca4x
Fine
63-87
63.21.3
59.9u.4
36.5
92. 92.0g.3
97.4a.c
IMath
63-86
62.21.3
63.7a.4
38.2
98 5..2
96.5a.6
Fine Web + OpenR Math
148-340
60.01.3
52.2a.4
31.6
40.0o.6
84.31.1
kvt C266x
Fin
148-343
57.91.4
51.50.4
31.0
14.1g.4
74.51.4
IMah
156-342
56.61.4
54.0g.4
29.4
21.0g.5
81.31.2
Table 10: Mean and standard deviation of downstream performance measured on 5 different cali- brations sets (50/50 mixture of OpenR Math (Open R1 2025) and Web (Penedo et al. 2024). 160K tokens, documents between 1K and 8K tokens). Results presented as mcan
Method
CR
GSMSK MMLU QASPER
LITM RULER-VT
Mistral NeMo 12B
Vanilla
61.91.3
64.50.4
38.4
99.5o.1
99.80.2
kvtc16x
17-20
63.52o.9
64.720.3
37.220.4
99.640.3
99.7
kvtcs2x
31-43
63.120.
64.440.6
36.1
98.112.0
99.2
kvtco4x
50-88
63.1
63.3
32.944.7
93.1
98.1
36

---

Published as a conference paper at ICLR 2026
Balanced 50/50 mixture of OpenR Math (same amount tokens from documents below 8K and above SK) vs calibration on documents only up to 8k tokens. We observe that calibrating the model on shorter data can result in slightly better than vanilla performance on short context tasks.
Table 11:
Method
Calibration Data Len
GSM8K MMLU
QASPER
LITM
RULER-VT
Mistral NeMo 12B
Vanilla
61.91.3
64.5p.4
38.4
99.5g.1
99.8n.2
XVIC16x
IK 8K
17-20
63.5
64.740.3
37.2
99.6.0.3
99.7
XVtC16x
IK 32K
17-20
62.01.3
64.40.4
37.6
99.8a.0
99.50.2
KVtCa2x
IK 8K
31-43
63.1
64.4
36.1+0.7
98.142.
99.2+0.3
XVtCazx
1K 32K
31-43
62.21.a
63.8u.4
37.5
99.6g.1
98.7u.4
XVtC64x
-K
50-88
63.1
63.3
32.920.7
93.1
98.1
kvtc64x
-K-33 -32
51-87
61.91.3
61.40.4
38.0
95.30.3
98.0o.4
Table 12: Ablation of out of distribution source of data. We consider kvtc calibrated on 50/50 mixture of Wib (Penedo al. 2024) and OpenMathR1 (Open 2025 along with calibration on Python/C/Assemmb] from the StarCoder dataset (Li et al. 2023). In all cases we utilize files between IK and 8K tokens and calibrate on 160K tokens. For configurations that were tested using several seeds we report score whereas for others report scorestderr
Method
Calibration Data
CR
GSM
MMLU QASPER
LITM RULER-VT
Mistral NeMo
12B
Vanilla GEAR 2bit
61.91.a
64.5u.4
38.4
99.5p.1
99.8u.2
59.81.4
64.0u.4
38.6
96.9o.z
99.4u.a
KIVI 2bit
59.71.4
64.30.4
38.2
91.96.3
98.3o.4
Web + OpenR Math Math
17-20
63.54
64.7
37.2=u.4
99. 09.640.3
99.7
kvtc16x
Python
17-20
59.41.4
65.20.4
37.4
99.00 99.9p.a
99. 79.77..
C
17-20
55.01.4
65.40.4
38.0
99.2o.
99.3o.3
Assembly
17-20
58.91.4
65.2p.4
37.0
99.9o.0
99.7o.2
Fine Web + OpenR
31-43
63. 1
64.4
36.120.7
98. 1
99 99.2+0.3
kvtcc32x
Python
35-43
59.01.4
64.30.4
36.3
99.7o.1
99. 99.2a.3
C
31-43
50.21.4
64.70.4
36.8
96 96.20.2
99.30.3
Assembly
31-43
55.21.4
64.2p.4
36.7
99.6p.1
99.40.3
Fine Web + OpenR
50-88
63.1
63.3
32.92n.7
93. 133.6
98. 98.1
kotC
Python
63-87
56.31.4
62.0o.4
33.9
98.0p.2
97. 97.40.5
C
63-87
54.41.4
63.30.4
34.5
95.40.3
97.80.5
Assembly
50-86
60.91.3
62.2o.4
33.0
99.1o.1
98.30.4
37

---

Published as a conference paper at ICLR 2026
B.7 SLIDING WINDOW SIZE VS DOWNSTREAM PERFORMANCE
We ablate the influence of sliding window of recent not-compressed tokens on downstream perfor- mance in Table [3 We observe that increasing the length of sliding window improves downstream performance of the model, with most noticeable difference between sliding windows v 16 and sliding windows 64.
Table 13: Ablation of the sliding window size of recent tokens that not compressed. For fair compression in this table we simulate sequence of short conversations by running compress- ing/eviction on the cache every s instead every s 16 tokens. Therefore, the w 16 token window is left int the range 15-16 instead of 0-16.
Method
Window Size
CR
GSM8K MMLU QASPER LITM RULER-VT
Llama 3.1
59-88
41.21.4
53.8a.4
36.5
68.3p.6
87.71.0
16
58-88
53.01.4
56.0o.4
38.1
80.0o.5
88.01.0
kvbco4x
64
57-88
55.81.4
59.20.4
37.7
89.8p.4
95.8g.6
128
60-88
56.81.4
60.5g.4
40.4
99.4o.1
99.8o.2
Mistral NeMo 12B
61-87
53.01.4
54.90.4
36.4
88. 88.60.4
90.70.9
16
60-87
60.11.3
56.40.4
37.7
88.90.4
95.80.6
kvt
64
59-87
62.21.3
59.0u.4
37.8
93.0b.a
98.00.4
128
51-87
61.91.3
61.40.4
38.0
95.3p.3
98.0g.4
38

---

Published as a conference paper at ICLR 2026
B.8 ABLATIONS LOSSLESS COMPRESSION
We ablate the differences between:
ANS (Duda 2014)
Bitcomp with default nyCOMP settings (NVIDIA 2020
DEFLATE (Wul 2017) (Huffman (Huffmman 1952 LZ77
(RASIP working group 1997
GDeflate NVIDIA|2022)
LZA (Collet 2011)
Snappy (Google 20111
Zstandard (Facebook] 2015)
Identity additional lossless compression
We present the results in Table 14 We note that DEFLATE can be easily substituted by faster GDeflate optimized for GPUs, the highest measured difference in compression ratio is < 0.1. We observe that the cache generated for RULER Variable Tracking is significantly more compressible than the cache generated for other tasks. We hypothesize that it may be effect of repeated noise used as context filler by the authors (see Appendix A). We observe that ANS, DEFLATE, GDe- flate, and Zstandard improve significantly over Identity in all studied cases. For detailed study of throughput vs compression ratio of tested algorithms, we refer to (NVIDIA 2024).
Table 14: Ablation of the lossless compression algorithms, performed using Mistral Nemo 12B with kvt C32x Identity stands for additional lossless compression (just PCA DP Quantization; note that we count the omitted sinks into compression ratio, what is notable for tasks with shorter contexts). We mark results that have compression ratio advantage over Identity smaller than in red.
Compression Ratio
Algorithm
Min-Max
GSM8K QASPER LITM RULER-VE
ANS
32.8-37.6
32.8
37.6
36.8
37.0
Bitcomp
29.6-32.3
29.6
32.3
32.2
32.2
DEFLATE
34.7-42.9
34.7
39.5
39.7
42.9
GDeflate
34.6-42.8
34.6
39.4
39.6
42.8
Identity
29.6-32.4
29.6
32.4
32.2
32,3
LZ4
29.5-35.5
29.5
32.4
32.7
35.5
Snappy
29.1-31.5
29.4
32.2
32.4
34.5
Standard
34.6-46.3
34.6
39.4
39.9
46.3
39

---

Published as a conference paper at ICLR 2026
B.9 BENEFITS OF DP QUANTIZATION
We compare k to a variant that does not use DP quantization, but instead removes a fraction of the least important principal components (denoted -DPQ). The results presented in Table [5] We observe that removing PCA components, rather than applying DP quantization, leads to significant performance degradation on long context tasks. Additionally, the performance DPQ on short context tasks deteriorates further as the length of the sliding window of uncompressed elements is reduced. We also note that DEFLATE can be much more efficient on quantized data. In general, our findings demonstrate that quantization is crucial component of kvtc, and omitting it hinders scaling to larger compression ratios.
Table 13: Ablation of the importance of DP quantization over pure PCA. For alteration of sliding window size we follow the protocol from Appendix B.7
Window Size
Method
Modification
CR
GSM8K MMLU QASPER
LITM RULER-VE VT
Llama 3.1 8B
9-10
57 57.51.4
59.40.4
40.1
19.1a.1
98.2o.
kvtcsx
-DQQ
8-9
56.91.4
47.50.4
40.1
98.60.1
93.0o.s
18-22
57.21.4
59.50.4
40.6
99.40.1
98.30.4
16
kvtcıex
-DPQ
16-17
52.01.4
53.80.4
36.9
69.2o.6
87.01.1
34-44
55.51.4
58.20.4
39.6
98.8u.1
94.8p.7
kvtcs2x
-DPQ
31-34
42.91.4
36.80.4
34.2
44.4o.6
83.01.2
58-88
53.01.4
56.0u.4
38.1
80.0u.s
88.01.o
kvtc64x
DDQQ
52-68
21.81.1
6.5u.2
26.7
5.10.3
55.51.6
9-10
56.71.4
59.9u.4
40.0
99 39.30.1
99.1o.a
kvtesx
DDQQ
8-9
57.01.4
60.7o.4
40.2
99.5u.1
98.5u.4
17-22
57. 17.4
60.4
40.7
99. 99.3o.1
99.0o.3
kvtc1ex
128
-DQQ
16-17
55.71.4
59.7o.4
37.1
85.0o.4
95.50.7
33-44
58.41.4
60.80.4
39.3
99.1o.1
98.90.3
kvtc32x
-DPQ
31-34
55.11.4
56.90.4
35.3
64.5o.s
89.61.0
60-88
56.81.4
60.5u.4
40.4
99.4o.1
99.8u.2
kvtc64x
-DPQ
47-68
57.21.4
49.3u.4
28.1
13.1u.4
58.71.5
40

---

Published as a conference paper at ICLR 2026
B.10 PCA FEATURE CONCAT SIZE VS PERFORMANCE
We study how the number of layers over which we concatenate key/value heads influences the per- formance of k We present the results in Tables T6 and 17 To better isolate the influence of the number of concatenated key/value heads, we run kvtc without dynamic programming quanti- zation (that is, we run kvt c-DPQ introduced in Appendix B.9) and with sliding window w 16. We note that results support our hypothesis about cross-layer similarity between key/value heads from Section 2 the more layers we concatenate for calibration (PCA), the better the downstream performance.
Table 16: Ablation of the number of layers used for PCA. To better isolate the influence of the number of concatenated key/value heads, we run kvtc without dynamic programming quantization (that is, we run kvtc introduced in Appendix B.9) and with a sliding window 16.
Method
PCA Layers|
GSM8K MMLU QASPER LITM RULER-VT
Llama 3.1
27.51.2
24.30.3
28.2
19.
70.81.4
2
44.81.4
41.10.4
34.2
77.60.5
84.91.1
53.31.4
51.9p.4
37.5
95.9g.2
93.0p.s
kvtcax-DPQ
55.91.4
55.3p.4
39.7
99.22 99.2n..
90.7p.9
56.01.4
57.1o.4
38.8
98.80.1
92.80.8
32
56.91.4
57.50.4
40.1
98. 98.6u.1
93.00.s
2.50.4
0.2g.0
18.4
0.0o.o
0.5p.2
2
13.91.0
2.30.1
22.1
0.70.1
12.91.0
4
33.31.3
19.30.3
29.1
32.0u.e
62.71.6
kvtC
49.61.4
43.4u.4
34.4
60.7a.6
85.81.1
51.61.4
49.1o.4
36.0
72.70.6
89.01.0
32
52.01.
53.80.4
36.9
69.2o.6
87.01.1
1.1o.3
0.
17.8
0.0o.p
0.1p.1
2
1.50.3
0.2o.0
18.1
0.0o.0
0.2g.1
4
3.90.5
0.90.1
20.0
0.0o.0
1.60.1
kvtcgax-DPQ
28.01.2
6.40.2
24.7
2.60.2
33.01.5
40.1
26.0o.4
31.2
24.6n.5
60.41.5
32
42.91.4
36.8p.4
34.2
44.40.6
83.01.2
0.90.3
0.3u.D
18.0
0.0g.0
0.1o.1
2
1.40.3
0.
17.7
0.0o.o
0.1o.1
4
1.50.3
0.1o.o
18.1
0.0o.0
0.1o.1
kvtC DDPQ
2.40.4
0.6o.1
19.4
0.0o.0
0.2o.1
13.91.0
1.9u.1
22.1
0.0g.D
10.0p.9
32
21.81.1
6.5u.2
26.7
5.1o.3
55.51.6
Table 17: Ablation of the number of layers used for PCA with 32x dimensionality reduction applied separately to keys and values (that is, either keys or values are compressed, not both). We follow the protocol from Table [6] We observe that keys benefit more from global concatenation, whereas values show significant boost when increasing the number of concatenated layers from 8 to 16.
Cache
PCA Layers GSM8K MMLU QASPER LITM RULER-VT
Ilama 3.1 8B with kvtc -PQ
3.20.5
4.4..2
21.7
0.0g.0
0.1p.1
4
25.91.2
27.50.4
25.9
0.2g.0
2.50.5
Keys
8
49.9
43.6n.4
35.2
44.50.6
74.91.4
52.51.4
52.3n.4
38.6
86.0n.4
90.
32
53.21.4
56.30.4
38.8
96.70.2
97.4n.s
2.0p.4
0.9p.1
18.0
0.0p.o
2.80.5
4
28.91.2
29.0o.4
30.0
15.60.5
68.21.5
Values
8
40.91.4
41.80.4
31.6
53.1u.
76.41.
16
49.21.4
45.70.4
37.1
82.60.a
88.61.0
48.71.4
48.60.4
38.7
81.40.5
89.01.0
41

---

Published as a conference paper at ICLR 2026
B.11 PCA DIFFERENT PER-PROMPT VS ONE-TIME
Conversation simulation proceeds as follows: the model maintains window of only 16 uncompressed recent tokens and performs compression each time a new token is generated. We note that per-prompt calibration results in significantly lower compression ratios, owing the overhead of storing the per-prompt projection matrix V Moreover, a per-prompt V can generalize poorly and fail to compress the continuation the conversation effectively.
Table 18: Ablation of PCA Calibration. We follow the protocol described Table 17
Calibration Per-Prompt Conversation
Simulate
GSM8K Math
LITM 100 Long Context
Method
CR
Llama 3.1 8B
False
False
7.9-8.0
56.81.4
98.8o.1
True
7.9-8.0
56.91.4
98.6p.1
kvtesx -DQQ
True
False
1.0-1.1
56.71.4
99.5o.1
True
True
1.0-1.1
31.41.3
99.5o.
False
False
15.4-15.9
54.61.4
73.7o.s
False
True
15.4-15.9
52.01.4
69.2a.a
kvtciex-DPQ
True
False
1.0-2.1
56.71.4
99.4o.1
True
True
1.0-2.1
31.41.3
99.5o.
False
False
29.4-31.9
48.71.4
49.5g.6
False
True
29.5-31.9
42.91.4
44.40.6
kvt DPQ
True
False
1.0-6.2
56.71.4
99.4o.1
True
True
1.0-6.2
31.41.3
99.3o.1
False
False
54.2-63.4
38.61.3
5.30.3
False
True
54.3-63.4
21.81.1
5.10.3
Brtc64x-DPQ
True
False
1.3-12.4
58.31.4
99.3g.1
True
True
1.3-12.4
28.41.2
98.8o.1
42

---

Published as a conference paper at ICLR 2026
B.12 STANDARD ERROR THE MAIN RESULTS
In Table 19 we attach the results from Table with their standard error, as bootstrapped by LM Evaluation Harness (where available) (Gao et al. 2024). We note that downstream evaluation runs, except the Qwen models, were performed using
seed and greedy evaluation.
Downstream task results, presented also in Table 2 here shown with standard error as reported by LM Evaluation Harness (where available) and CR computed both with and without sliding window (where applicable).
Table 19:
CR
GSM8K MMLU
QASPER
LITM 100 Long Context
RULER E--V
Method
Window
Window
Math & Knowledge
Llama 3.1 8B
vanilla GEAR 2bit
56.81.4
60.5o.4
40.4
99.4o.1
99.8g.2
3-5
52.81.4
59.6p.4
40.4
96.9u.2
99.8u.2
KIVI 2bit
3-5
52.81.4
59.6p.4
39.1
88.8u.4
98.9u.3
XKV2/16
2/16 4key
1-5
56.61.4
59.5p.4
35.6
99.9o.0
99.8a.2
3/16 4value
FP8
55.21.4
60. 60.1p.4
40.8
99.4n.1
99.9g.1
kvt
9-10
3-9
57.01.4
59.8p.4
40.1
99.3n.1
99.1o.s
kvtc16x
18-22
4-17
56.91.4
60.1o.4
40.7
99.30.1
99.1o.3
kVtC32x
34-44
5-29
57.81.4
60.6p.4
39.4
99.10.1
98.9g.3
kvtC64x
60-88
5-45
57.21.4
60.7p.4
37.8
90.2o.4
95.9u.6
H2O
/6 recent
54.31.4
44.30.4
34.3
20.2a.5
50.41.6
1/16
TOVA
54.51.4
44.80.4
38.6
1.2o.1
99.7o.2
MN- MN-Minitron
Vanilla
59.11.4
64.3p.4
38.2
99.8u.0
99.4q.3
GEAR 2bit
3-5
57.91.4
63.60.4
38.2
96.00.2
98.3o.4
KIVI 2hit
3-5
58.01.4
63.2p.4
38.2
86.3o
96.8g.6
2/16 Gkey
1-5
1-5
59.31.4
63.10.4
34.5
99.6o.1
99.1o.3
3/16 Svalue
FP8
60.11.3
64.3p.4
38.3
99.8u.1
99.2a.s
kvt Csx
10-11
3-9
60.61.3
64.2p.4
39.1
99.4u.1
98.8u.3
kvtC16x
17-21
3-16
60.31.3
64.1p.4
38.6
99.3u.1
98.8u.3
kvt C32X
32-46
3-27
59.11.4
63.70.4
37.7
86.9o.4
96.00.6
kvtC64x C84X
53-95
3-38
57.814
62.1o.4
59.50.6
93.4g.8
H2O 1
/16
55.31.4
43.5p.4
16.6u.s
39.21.6
pass
TOVA
59.21.4
48.10.4
33.9
0.3o.1
99.3a.3
Mistral NeMo 12B
Vanilla
61.91.3
64.5p.4
38.4
99.5u.1
99.8u.2
GEAR 2bit
3-5
59.81.4
64.0p.4
38.6
96.9u.2
99.4g.3
KIVI 2bit
3-5
59.71.4
64.30.4
38.2
91.9o.s
98.3o.4
XK 2/16 5key
3/1d Svalue
1-5
1-5
61.91.3
63.90.4
33.5
97 .7
99.4g.s
FP8
61.71.3
64.50.4
37.9
99.0o.1
99.8o.2
kvt.aa
10-11
3-10
62.51.3
64.60.4
37.6
99.9g.0
99.5g.2
kvtciix
17-20
3-16
62.01.3
64.4p.4
37.6
99.8u.0
99.5g.2
kvtC322
31-43
3-29
62.21.3
63.8p.4
37.5
99.6n.1
98.7g.4
kvtc644
51-87
3-47
61.91.3
61.4p.4
38.0
95.3n.3
98.0n.4
H2O !/16 recent
57.01.4
45.4p.4
29.5
16.2o.s
35.21.6
pass
TOVA
60.31.3
49.00.4
36.0
8.70.4
99.6g.2
g
43

---

Published as a conference paper at ICLR 2026
B.13 ADDITIONAL LONGBENCH AND RULER RESULTS
We additionally evaluate kvtc on 2WikiMultiHopQA (2WQA) Ho et al. (MFQA) (Bai et al. 2224), MuSiQue (MQUE) (Trivedi al. 2022). QMSum (QMS) (Zhong al.
2020), MultiFieldQA
202T), SAMSum (SAMS) (Gliwa et al. 2019) from LongBench (Bai et al. 2024). We also evaluate on Common/Frequent Words Extraction (CWE/FWE), Needle in Haystack (NIAH) (Kamradt, 2023), HotPotQA (HPQA) (Yang et al., 2018). SQuAD (SQA) (Rajpurkar et al. 2016 2018) from RULER (Hsieh et al. 2024). We present the results in Table showing that k can still maintain comparable performance to vanilla with z 20 compression ratio.
Table 20: Additional results (with stderr) LongBench (I host) and RULER (0 shot) tasks, methods configured as in Table 2 Given the RULER O-shot QA results we hypothesize that both Llama 3.1 8B and MN-Mimitron 8B base models were exposed to some amount of question answering data with format similar to the mentioned tasks, whereas it might have not been the case for Mistral NeMo 12B.
Task
Vanilla GEAR
KIVI
H₂O
TOVA
xKV
FP8
kvtc1ex kvtc22x kvt
Llama 3.1
8B
CR
5
5
8
8
4-6
2
18-20
35-39
62-78
2WQA
10.83.3
40.7a.3
12.3
38.5a.2
41.33.3
39.5a.2
40.63.3
10.3s.3
10.03.2
40.63.3
MFQA
50.32.6 49.62.
50.12.7
40.824
50.42.5
48.72.
50.92.s
51.12.6
49.52.6
50.22.6
MQUE
33.83.1
33.83.1
31.73.0
32.43.0
32.63.1
31.53.0
33.8s.1
33.73.1
33.9s.1
34.13.0
QMS
26.90.7
27.2o.7
25.70.8
25.40.6
25.40.6
24.40.7
26.1o.7
26.40.7
25.90.a
25.2a.6
SAMS
47.31.
47.21.
45.81.2
46.81.3
46.11.3
45.71.
47.11.3
47.01.3
46.61.3
45.61.3
CWE
94.71.0 94.01.1
91.01.3
64.92.1
76.51.9
68.92.1
94.51.0
92.41.2
90.71.3
88.01.5
FWE
92.11.2
91.91.2
89.91.4
75.71.9
69.52.0
88.61.4
92.31.2
89.01.4
88.11.5
83.31.7
NIAH
100g.0
100g.0
100n.0
6.2
99.6p.3
99.8g.2
100g.0
100g.0
99.8o.2
99.6g.3
HPQA
57 2 56.82.2
57.22.2
48.82.2
54.82.2
56.22.2
57.62.2
57.22.2
57.22.2
55.82.2
SQA
55.72.2
55.72.2
54.02.2
40.22.2
51.32.2
53.522
55.62.2
55.22.2
53.12.2
53.82.2
AVG
59.9
59.7
58.8
42.0
54.8
55.7
59.9
59.2
58.5
57.6
MN-Minitron 8B
CR
5
8
8
4-6
2
19
39-10
78-80
2WQA
15.83.3
45.33.3
15.4s.3
44.63.3
15.53.3
17.13.3
45.83.3
16.23.3
16.45.3
45.13.3
MFQA
12.62.9 12.22.8
13.02.9
34.82.5
10.32.7
11.62.9
13.22.9
12.62.9
13.42.9
13.22.9
MQUE
27.42. 26.82x
26.22.
25.62.7
26.02.s
26.82.
27.32.s
27.12.8
26.62.a
26.32.8
QMS
23.4n.6 23.0g.6
23.2p.a
21.90.5
22.5p.a
21.80.6
23.5p.s
23.1o.6
22.70.a
22.6g.6
SAMS
36.11.6 36.51.6
36.81.6
36.71.6
36.11.6
34.71.
36.21.6
35.91.s
35.91.
35.81.6
CWE
92.41. 90.51.
87.11.5
64.72.1
66.42.
69.72.
92.51.2
85.31.8
79.51.
75.21.9
FWE
86.215 86.51.
85.81.8
70.32.0
73.72.0
85.416
85.91.8
86.31.5
83.11.7
81.01.8
NIAH
100,.0
100g.0
99.8p.2
6.0
99.8p.2
97.6u.7
1000.0
1000.0
1000.0
100g.0
HPQA
62.02.2
63.22.2
58.42.2
49.02.2
57.82.2
56.62.2
62.82.2
62.22.2
55.82.2
54.62.2
SQA
64.92.1
65.52.1
62.42.2
48.02.2
62.72.2
63.92.2
64.62.1
64.72.1
62.52.2
62.22.2
AVG
58.1
58.0
56.8
40.2
53.1
54.5
58.2
57.3
55.6
54.6
Mistral NcMo 12B
CR
5
8
8
4-6
2
19
39-40
78-80
2WQA
43.3a.3
42.73.3
42.1s.3
39.2s.3
42.73.3
43.8a.3
43.3s.3
43.73.3
43.83.3
43.23.3
MFQA
51.52.
51.12.
50.62.7
40.82.
49.62.8
50.22.
50.72.7
51.02.7
50.92.a
51.32.6
MQUE
27.22.
27.22.8
27.22.8
24.92.7
27.12.8
26.72.
26.92.8
26.62.8
26.12.
27.92.8
QMS
25.70. 25.90.6
25.7o.6
21.90.6
24.2p.6
25.1a.
25.7n.s
25.2n.s
24.7o.7
25.
SAMS
45.61.4 45.21.
44.91.3
42.21.4
45.51.4
45.31.4
45.41.4
45.91.
45.11.4
45.61.3
CWE
93.21.1 92.61.2
93.31.1
65.52.1
69.92.1
75.21.9
93.71.1
90.91.3
86.51.a
85.91.
FWE
83.01.7 82.61.7
84.11.8
77.51.9
70.92.0
83.41.7
82.91.7
82.11.7
78.21.
78.91.
NIAH
100,.0 100g.0
99.80.2
6.0
1000.0
100,.a
100g.0
1000.0
1000.0
100g.0
HPQA
36.22.
35.821
35.42.1
28.82.0
31.82.
35.621
35.22.1
33.82.1
33.62.1
33.22.
SQA
22.41.9
25.01.9
23.81.9
18.31.7
21.71.8
21.11.
22.81.9
22.91.0
23.01.9
21.91.
AVG
52.8
52.8
52.7
36.5
48.3
50.6
52.7
52.2
51.2
51.3
44

---

Published as a conference paper at ICLR 2026
B.14 PCA MATRIX SIZES
In Table 21 we present the sizes of PCA projection matrices (V from UEVT) after being computed via (Halko 2011) algorithm. We note that the sizes are only relatively small fraction of the model parameters, and that they can be further reduced by the DP algorithm depending on the desired compression ratio.
Table 21: Number of parameters used by PCA matrices, before DP, for the tested models. For example, Llama 3.1 8B has 32 layers, each with 8 key/value heads, each head of size 128. Therefore, after cross-head concatenation, each key/value has projection V is cut to the first 10K principal components by (Halko al., 2011) algorithm for efficiency, resulting in 32768 x 10000 328M parameters. Further DP bit allocation can remove additional principal directions depending the desired compression ratio. Both models and PCA projection matrices are stored 16bit precision.
128 32768 features. The PCA
Model
#aramms
Key/Value Features
Key/Value PCA Cap
Key/Value PCA Params
TotalPCAParan Morda/arrams
Qwen 2.5 RI 1.5B
1.5B
x x 128 7168
51M
6.8%
8K
Qwen 2.5 RI 7B
7.1B
28 x 4 128
14336
115M
3.2%
Llama 3.1
7.5B
x 8 x 128 32768
328M
8.7%
Llama 3.3 70B
69.5B
80 x x 128
81920
10K
819M
2.4%
Mistral NeMo 12B
11.6B
40 x x 128
40960
410M
7.1%
B.15 COMPRESSION RATIO WITH SLIDING WINDOW
For the presented methods and sliding window of w uncompressed (high precision) keys/values corresponding to recent tokens, can the following formulas to calculate the compression ratio that includes the sliding window:
kvt e
etx
we keep the first tokens in FP8
kvt Cerx
we keep the first four tokens in FP8
+w+2
Other methods:
ctx
45

---

Published as a conference paper at ICLR 2026
B.16 LMCACHE vLLM
We provide additional end-to-end measurements in a simplified, multi-user scenario. To be more precise, have run vLLM (Kwon al. 2023) with LMCache (Cheng al. 2025) managing the KV cache on a workload which, for x 64K of initial input tokens, users ask questions that are between 16 and 100 tokens and receive 100-token answers token for TTFT measure), with no delays in between conversation turns. We have used Llama 3.3 70B in FP8 precision. split over 2x H100 80GB GPUs using tensor parallelism. LMCache has been configured to use 128GiB of host (CPU DRAM) memory GPU, which is equivalent to using ITiB in an 8-GPU server. We present the latency results Table 22
We observe that with 12 more clients, the dedicated amount of host memory becomes too little hold the KV caches and forces recomputation, which is reflected spiking latency. In a more realistic scenario, there would be more concurrent users taking pauses between conversation turns. We note that in this test, the KV cache is compressed with KVTC only for storage host memory; it does not take advantage of compressing the KV cache for storage in GPU HBM, which would bring additional latency benefits, as its implementation is more involved.
Table 22: Response latency for generation 100 tokens (left) and TTFT (right) vs. number of concurrent vLLM clients. In both cases, the initial input context is sampled to have between 62K and 66K tokens, and each user question sampled to have between 16 and 100 tokens. We use the same GPU for prefill, generation, compression, and decompression, which can result in increased latency for kvtc.
#Clients
Vanilla (s)
KVTC 16x (s)
#Clients
Vanilla (s)
KVTC 16x (s)
2.5
2.5
1
0.2
0.2
2.9
2.9
2
0.3
0.3
4
8.6
9.1
4
1.2
1.7
12.7
13.9
6
2.0
2.7
8
17.3
18.0
8
2.8
3.5
10
21.4
24.2
10
3.4
4.5
12
155.7
27.9
12
136.6
5.6
14
180.9
31.3
14
159.0
6,4
16
208.2
37.1
16
181.6
7.3
46

---

Published as a conference paper at ICLR 2026
B.17 DYNAMIC PROGRAMMING ALGORITHM
Below, present the pseudocode for the dynamic programming precision assignment along with proof sketch.
Dynamic Programming Precision Assignment Pseudocode
iibration ion data natrix shape (betich, nun (esturee)) oz each teature across the batch cinension
D.mean (dimno, Teepdin-Trust
svd
2 UB5
assume co columns sorted singul ar values 2.shape
batch, sun_considdeeeti Features
inftial reconstrustionn error corresponds to quantixing everything with zero bits hasctll_ccoostrruttto..crrs (BvE)
squared Frozenius carn
is initially quantized f and wa progressively considar nor-zero quantization ef more and nore teatures
initial
pest_error tensor (shape- coneideeedddr seasideres_eeatues reatures nox_bit_ 1h, valueseinitia reconst ion error]
best_orsor_type acray Ishage (pur_consssrree ceatures nax_bit_ 11, valuee-9)
aest cest_error_Bi arrorrbblook block tensorr (num considered teatures + nax budget + 11, values-0) _et__orzo__iit_caae tensor (shape-
box_bit_budget 1), valcas=0)
allowedttsoiioozze [i, 16, 64, 255, 1024]
the count usage ot typs as recaure it directly correspeads to the removal of principal comporants. types [None, int2, ints, sp8)
that quantizes
acray
rangetl, com_consieered..c restures
for block_size in allowet_Diock_sieen
12 block_size 1:
budget range (1, aax_bit_badre:
it best_arzor[1, budgst] > best_crrar(i, bedget 11:
best_ecrer 1, sudget] past_erroripg budget
best_erres_typall badget] best_ersor_eypess, bedest budget 1]
best_ecree_Biocs_dleet busget] best_ocror__bocct Metteerrer__cccooott budget) batt_ budget :]
tloct_to_quantiee P(:. block_size:t]
quantized_data, used_bits simiste_qanntiaatinn toto
used_bits budget:
zero auant (black Mockk_t Lu qun) Lann
quat--aacioneeroo: black_to quantized_data
quantizacio__croo: (quntiiarcoo_errre quantization .un
ozror_change -zaro_hlt_cuattioo.rrr quantization
block_sise, badget used_iitas best_orsoris, budces] arsor_change bast_erroz bleck_oize, budeas csod_bits] eer.rrrorrpppll budget] =t budget] biock_
Ii, badget]
best bww._eroor_blook_a. budget] used
best_arsor_it__ee
we can use best_ beat_ best blokk and beat zablea to b gaz tha quantizacion foz civen budget
The proof of the optimality follows by simple induction on and budget. To be more precise we to prove that best.error [j, ql is the smallest reconstruction error (squared Frobenius norm) one can achieve when considering first features of P (setting other features to 0 0-bit quan- tization) and quantization restricted types from types that can only be used to quantize blocks of contiguous features of sizes in a 1.loo lowed block s zes sizes, while utilizing no more than budget bits. For simplicity we assume that the smallest reconstruction error (squared Frobenius norm) for q-0 budget cases is iee.iaa..eoonstrutttoo...... the proof by induction can be conducted as follows:
(P+P) sum () Then
For i=0 or budget=0 0 have that if we consider quantization of the first 0 features and leave other features as zeros or budget of size 0, then the reconstruction error indeed (P xP) sum ().
Then to prove for i>0 and budget>0 we assume the optimality of best error i
q] for j<i, and for j=i and qıbudget. Then note that the algorithm enumerates all
47

---

Published as a conference paper at ICLR 2026
possible quantization blocks that the quantization of the first features end with within the budget budget.
Computational complexity can be directly inferred from the pscudo-code:
O'mum_considere ffnaturrrrd.fatturesxx
Jallowed block sizes]
max bit budget |types] X
Isim (aaamaallllo block sizes batch))
where
Asim (max (allowed blockk )lockssies) batch)
the time taken to simulate quantization. Assuming that alloo and |types]
constant and quantization simulation can be performed in
O (alowwlloood blocck blo..lloocssssss batch)
can write the asymptotic bound the algorithm runtime
@mnum_oonsidered_coo featurrssssssssee x max_bit max_bit_budgel x batch)
We additionally provide the runtime of the algorithm Table in in Appendix B.5
48


Reviews:
[{"review_id": "swskDkvLw6", "reviewer": "Reviewer_rALC", "summary": "This paper introduces kvtc, a lightweight transform coder designed to compress KV cache for efficient storage in LLM inference. Drawing on classical media compression techniques, kvtc integrates PCA-based feature decorrelation, adaptive quantization, and entropy coding, requiring only brief initial calibration without modifying model parameters . It achieves up to 20× compression ratio on KV cache of models like Llama 3.1, Mistral NeMo, and Qwen 2.5 R1, while maintaining reasoning and long-context accuracy across benchmarks such as GSM8K, MMLU, and LiveCodeBench.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The paper presents an interesting approach by using a transform-coding framework for KV cache compression. Experiments demonstrate that this method can maintain model performance even with a high compression rate.\n\n+ The ablation experiments are very thorough.", "weaknesses": "+ The paper uses a limited number of evaluation benchmarks. Recent related works typically conduct comprehensive experiments on benchmarks like LongBench and RULER, whereas this paper only uses one dataset from each (Qasper and VT).\n\n+ The \"Related Work\" section should also include content related to transform coding, as it is central to the proposed approach in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j2gKWLupAX", "reviewer": "Reviewer_B6EH", "summary": "The authors propose a method called KV Transform Coding (KVTC), which compresses the KV cache using a transform-coding pipeline inspired by classical image compression. The proposed approach does not modify model parameters or architectures and can be integrated seamlessly into existing inference systems. Experiments demonstrate compression up to 20$\\times$ with negligible accuracy loss across multiple models and benchmarks.", "questions": "- How exactly are RoPE rotations inverted and reapplied for keys during compression/decompression in mixed prefill/decode settings?  \n- Is the projection matrix V block-structured per layer/head to enable true layer-by-layer decompression, or is selective multiplication by submatrices used? Any accuracy/cost trade-offs?  \n- Can you report end-to-end CR including the uncompressed sliding window to better reflect practical storage savings?  \n- Which nvCOMP codec is used (Deflate vs. GDeflate/LZ4/Zstd)? How sensitive are results to codec choice and content?  \n- How robust is a single global V across domains? Any cross-domain generalization or per-domain calibration vs. global calibration ablation?  \n- What is the memory/latency overhead of storing and applying V, group scales/shifts, and metadata for large models (e.g., 70B) under pipeline/tensor parallelism?  \n- Can you provide perplexity/per-token CE vs. CR curves, and a stronger correlation study between reconstruction error and task accuracy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The KV cache is a key bottleneck for efficient LLM inference. The paper identifies and effectively tackles this real-world challenge.\n2. KVTC uses well-established techniques (PCA, quantization, entropy coding) in a novel context. It requires no retraining and can be plugged into existing frameworks.\n3. The authors test on diverse LLMs (Llama 3.1, Mistral-NeMo, R1-Distilled Qwen2.5) and datasets (MMLU, GSM8K, RULER, MATH500, etc.). The paper reports improvements in both latency and memory usage.", "weaknesses": "1. **Limited algorithmic novelty**:  Transform coding with PCA+quantization is classical, and several SVD/quant methods exist (e.g., SVDq, xKV). The core components are classical in signal processing. The main contribution is an effective adaptation of known techniques, rather than a fundamentally new algorithm.\n\n2. **Dependence on calibration data**: The PCA basis and bit allocation depend on a representative calibration dataset. When model structure changes, recalibration is required. There is limited end-to-end serving results (e.g., cross-node throughput, bandwidth savings, hit-rate improvements, multi-tenant scenarios).\n\n3. **Unfair latency comparison setup**: The reported \"8× faster TTFT\" compares KVTC against a full recomputation baseline, which is not representative of how production LLM systems operate. A fairer comparison should include: 1) recomputing only the KV cache for new tokens 2) other competitive caching strategies.\n\n4. The experimental protocol seems not align with the primary stated use case (compressing stale caches for storage between conversation turns). This mismatch, coupled with latency benchmarks from a \"simple implementation\" rather than a high-performance serving framework, makes it difficult to evaluate the true system-level overhead and viability of kvtc in its intended deployment.\n\n5. Some comparison choices could bias CR: CR computed only on compressed tokens (excluding sliding window), while Deflate gains are content-dependent.  The link between Frobenius reconstruction error and downstream accuracy is only partially validated; no perplexity/CE analysis.  More ablations on calibration set composition/size and GQA specifics would help.\n\n6. RoPE “undo” and reapplication details, and cross-layer/head concatenation vs. blockwise streaming claims, could use clearer algorithmic specification.  \n\n7. The assumption of a single, global low-rank subspace shared by all concatenated layers and heads is a strong and maybe potentially suboptimal design choice. The paper lacks a crucial ablation study to justify this global approach over more granular alternatives (e.g., per-layer or layer-block PCA), which might yield a better rate-distortion trade-off.\n\n8. SVD-related scheme and the bit allocation applied directly to KV Cache, inherently minimizes the **cache reconstruction loss**($\\|\\mathbf{K} - \\widehat{\\mathbf{K}}\\|_F^2$), which is Q-agnostic. However, the actual objective is minimizing the **attention output loss**, where Q acts as a dynamic projection (and maybe or maybe not can be approximated by the calibration process). Considering that the theoretical derivation and implementation may be more difficult after the introduction of Q, a brief discussion will make the paper more rigorous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uchpIkz1sI", "reviewer": "Reviewer_4EeH", "summary": "This paper proposes kvtc, a transform-coding pipeline for KV-cache compression that borrows from classic media codecs: PCA-based decorrelation, dynamic-programming bit-allocation for adaptive quantization, and Deflate entropy coding, with a one-time calibration and no weight changes. It targets the prefill↔decode boundary and multi-turn reuse, also handling practicalities like removing RoPE, skipping a recent-token window, and excluding early sink tokens. Empirically, kvtc attains ~16–20× (and up to ~40×) compression with accuracy within ~1 point of vanilla across GSM8K, MMLU, Qasper, RULER/VT, AIME/LCB, and MATH500, while reducing TTFT by up to 8× and outperforming or matching KIVI/GEAR (quantization), H2O/TOVA (eviction), and xKV/SVDq baselines in most settings.", "questions": "1. In Figure 1, the authors state that $T$ denotes the time dimension. Could the authors clarify what the “time dimension” specifically refers to in this context? Does it correspond to the token dimension or another notion of temporal ordering within the sequence? \n\n2. The authors propose using a calibration set to derive a generalizable transformation matrix $V$. However, an important question remains: how are the calibration samples selected to ensure representativeness across diverse input distributions? Moreover, is the calibration process sensitive to the choice or domain of these samples, and how does performance vary under distribution shift? In comparison to SVD-based methods that compute a separate decomposition for each prompt, how much performance degradation does this one-time calibration introduce? \n\n3. Some results in Table 2 are confusing. The proposed method occasionally outperforms the vanilla model, even under compression. Could the authors provide an explanation or analysis for this unexpected improvement?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written and easy to follow. The paper’s main strengths lie in its practicality, simplicity, and effectiveness. It introduces a lightweight, system-friendly transform-coding approach (kvtc) that achieves up to 20× compression of KV caches with minimal accuracy loss and no model modifications.", "weaknesses": "The proposed method leverages the traditional transform-coding framework, achieving high compression ratios with minimal performance degradation. However, its limitation appears to be decompression latency. How does the decompression time compare with other methods listed in Table 2? In Table 4, the authors report only the compression and decompression times of the proposed approach, which seems insufficient for a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EqFzwXpRAf", "reviewer": "Reviewer_jZ1g", "summary": "The paper introduces KVTC, a lightweight method for compressing KV caches in LLM inference. Inspired by classical transform coding, it combines PCA-based feature decorrelation, adaptive quantization via dynamic programming, and entropy coding. KVTC achieves up to 20× compression with negligible accuracy loss and scales across models such as Llama 3.1, Mistral-NeMo, and Qwen 2.5. Experiments on multiple benchmarks show that it outperforms prior quantization and SVD-based baselines while reducing inference latency.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. KVTC effectively reduces memory footprint by up to 20x with minimal degradation in model accuracy (<1 point), demonstrating strong practical value.\n\n2. The combination of PCA-based transform coding, adaptive quantization via dynamic programming, and entropy coding is well-grounded in classical signal compression theory, yet thoughtfully adapted to the KV-cache structure in modern Transformers.\n\n3. Evaluation spans multiple model families (Llama 3.1, Mistral-NeMo, Qwen 2.5) and a wide range of tasks (GSM8K, MMLU, Qasper, AIME25, MATH500, etc.), showing strong and consistent results.", "weaknesses": "1) The practicality of kvtc hinges critically on whether SVD components computed on calibration data can be effectively transferred to out-of-domain inference contexts. While the authors provide some analysis in Appendix C.6 on the effect of calibration data domain, I remain concerned about the potential error introduced when there is a significant distributional shift between calibration and actual input sequences. For instance, in Figure 8, the relative reconstruction error for \"Value Open R1 to FineWeb\" is notably higher than \"Value Open R1 to Open R1\".\n\n2) Table 4 reports the TTFT comparison between recomputation and decompression, which is informative. However, it would be valuable to also understand the latency in a more typical scenario where the KV cache is already present in GPU HBM (i.e., a cache hit). In such cases, what is the TTFT overhead introduced by kvtc’s decompression pipeline?\n\n3) It would be helpful to clarify the key technical distinctions of kvtc compared to prior SVD-based KV cache compression methods such as xKV in Introduction.\n\n4) xKV leverages inter-layer redundancy, whereas kvtc concatenates all attention heads across layers before compression. Given that Figure 2 shows block-wise cosine similarity patterns in value states rather than uniform cross-layer similarity, the rationale for global concatenation, rather than structured, layer-aware compression, needs further justification.\n\n5) Figure 3 appears largely illustrative and less directly tied to the core technical contributions of the paper.\n\n6) Across Tables 2 and 3, accuracy does not consistently decrease with higher compression ratios, in some cases, KVTC even outperforms the vanilla models. This counter-intuitive result is not analyzed.\n\n7) Baselines are compared using each method’s best reported configuration, which is reasonable to ensure strong baseline performance.However, this also makes it difficult to assess performance at equivalent compression ratios, as accuracy may not scale linearly with compression strength.\n\n8) Figure 8 demonstrates that cross-domain evaluation leads to larger reconstruction errors, yet no mechanism for adaptive or domain-agnostic calibration is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)