PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
Authors: Tengxuan Liu
Shiyao Li
Jiayi Yang
Tianchen Zhao
Feng Zhou
Xiaohui Song
Guohao Dai
Shengen Yan
Huazhong Yang
Yu Wang
Venue: ICLR 2026 Poster (2026)
Keywords: KV Cache Quantization
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs.

Abstract: Recently, significant progress has been made in developing reasoning-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) techniques.
However, this long-CoT reasoning process imposes substantial memory overhead due to the large Key-Value (KV) Cache memory overhead.
Post-training KV Cache quantization has emerged as a promising compression technique and has been extensively studied in short-context scenarios.
However, directly applying existing methods to long-CoT LLMs causes significant performance degradation due to the following two reasons: 
(1) Large cumulative error: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cumulative quantization error.
(2) Short-context calibration: Due to Rotary Positional Embedding (RoPE), the use of short-context data during calibration fails to account for the distribution of less frequent channels in the Key Cache, resulting in performance loss.
We propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs to address the above issues in two folds:
(1) To reduce cumulative error, we design a progressive quantization strategy to gradually lower the bit-width of KV Cache in each block. Then, we propose block-wise memory allocation to assign a higher bit-width to more sensitive transformer blocks. 
(2) To increase the calibration length without additional overhead, we propose a new calibration strategy with positional interpolation that leverages short calibration data with positional interpolation to approximate the data distribution of long-context data.
Extensive experiments on 7B–70B long-CoT LLMs show that PM-KVQ improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73–5.18$\times$ throughput over the original 16-bit LLMs.
Our code is available at https://github.com/thu-nics/PM-KVQ.

Published as a conference paper at ICLR 2026
PM-KVQ: PROGRESSIVE MIXED-PRECISION KV CACHE QUANTIZATION FOR LONG-COT LLMs
Tengxuan Liu 1.2, Shiyao Li Li,, 1 Jiayi Yang" Tianchen Zhao', Feng Zhou',
Xiaohui Song', Guohao Dai Shengen Yan", Huazhong Yang', Yu Wang!
Tsinghua University. "Infinigence-AI, *Columbia University, 4OPPO Al Center, Beijing, China, Shanghai Jiaotong University
ABSTRACT
Recently, significant progress has been made in developing reasoneng-capable Large Language Models (LLMs) through long Chain-of-Thought (CoT) tech- niques. However, this long-CoT reasoning process imposes substantial mem- ory overhead due to the large Key-Value (KV) Cache memory overhead. Post- training KV Cache quantization has emerged as a promising compression tech- nique and has been extensively studied in short-context scenarios. However, di- rectly applying existing methods to long-CoT LLMs causes significant perfor- mance degradation due to the following two reasons: (1) Large cumulative er- ror: Existing methods fail to adequately leverage available memory, and they directly quantize the KV Cache during each decoding step, leading to large cu- mulative quantization error. (2) Short-context calibration: Due to Rotary Po- sitional Embedding (RoPE), the of short-context data during calibration fails to account for the distribution of less frequent channels in the Key Cache, result- ing in performance loss. We propose Progressive Hixed-Precision KV Cache Quantization (PM-KVQ) for long-CoT LLMs to address the above issues in two folds: (1) To reduce cumulative error, design progressive quantization strat- egy to gradually lower the bit-width of the KV Cache in each block. Then, we propose block-wit memory allocation to assign higher bit-width to more sensi- tive transformer blocks. (2) To increase the calibration length without additional overhead, we propose new calibration strategy with positional interpolation that leverages short calibration data with positional interpolation to approximate the data distribution of long-context data. Extensive experiments on 7B-70B long- CoT LLMs show that PM-KVQ improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73 5.18x throughput over the original 16-bit LLMs. Our code is available at https: / // httgggitthhuugoo com hte//hhhnnne Sewpunnics/PM--M CS/PM-KV
INTRODUCTION
Recently, many pioneers have developed remarkable reasoning Large Language Models (LLMs) with long Chain-of- Chain-of-Thoughts (CoT) techniques, such as OpenAl-ol (OpenAl 2024). DeepSeek- RI (Guo et al. 2025), QwQ (Team 2025), and on. To achieve better algorithmic performance, these long-Co'l reasoning I.I.ss are trained to generate up to 128K tokens with multiple complex rationales from different perspectives (Guo et al. 2025). However, this long-CoT process demands significant memory overhead (~ (~10GB-I00GB) to store the Key-Value (KV) Cache as the history information, which limits the practical application scenarios for such long-CoT LLMs.
To mitigate the substantial memory overhead of long-CoT LLMs, various KV Cache compression
methods have been proposed (Liu et al. 2024c Yang et al. 2024 Su et al. 2025 Xiao et al.
"Equal contribution.
I Program leader.
1 *oorrsponniig author: Yu Wang (yu-wang@tsinggnn.cdd..))

---

Published as a conference paper at ICLR 2026
Fu et al. 2024). Among them, Post-training KV Cache Quantization is a promising compres- sion technique that has already been well explored in short-context scenarios (c.g., <8K tokens). QServe (Lin et al., 2024) and MiKV (Yang al. 2024) observe that the Key Cache has more out- liers than the Value Cache, leading to higher quantization error. More importantly. the outliers in the Key Cache persist in certain channels. To this end, they propose channel-wise equalization method to migrate the outliers from the Key tensor to the Query tensor, thereby significantly reducing the quantization error. KIVI (Liu et al. 2024c), SKVQ (Duanmu et al. 2024), and Intact (Liu al. 2024b) gain insights from the data distribution of the attention map and preserve the first or most recent tokens in higher bit- bit-width within the KV Cache to maintain the performance.
2023
However, directly applying the above choot-context-opimmize methods to long-CoT LLMs results in severe performance degradation. The reasons stem from the following two aspects: (1) Large cumulative error in long-Cor CoT LLMs: As a lossy compression method, directly quantizing the Key and Value tensors (Liu et al. 2024c [iin* et al. 2024. Yang et al. 2024, Duanmu et al.) 2024) introduces quantization errors at each decoding step when generating one token. As the number of generated tokens increases, the accumulated quantization error grows larger, leading to a sig- nificant performance degradation of long-Cor Coo- LLMs. (2) Short calibration data cannot reflect long-context data distribution: The Rotary Positional Embedding (RoPE) operator incorporates positional information into each channel of the Key Cache by rotating token embeddings using sine and cosine functions of different frequencies. For low-frequency channels after RoPE, which have a period of over 32K tokens, calibration using short sequences (c.g., 2K tokens) fails to accurately reflect the data distribution of the Key Cache, leading to more significant quantization errors.
In this paper, we propose Progressive Mixed-Precision KV Cache Quantization (PM-KVQ) to ad- dress the above two issues respectively. (1) To reduce cumulative error, we aim to fully utilize the memory budget of the target hardware through two strategies. On the one hand, we propose to quan- tize the KV Cache progressively. For example, to achieve extremely low-bit quantization, such as 2-bit, instead of directly quantizing KV Cache to 2-bit at each decoding step, we initially store KV Cache in 16-bit format and then gradually reduce the bit-width to 2-bit through shifting operations once the memory resource is fully occupied. On the other hand, we propose block-wiss wise memory al- location technique to allocate higher bit-widths for more sensitive blocks. Specifically, we formalize the bit-width allocation task as an Integer Programming problem, which can be effectively solved by existing solvers with negligible latency. (2) To increase the effective calibration length without intro- ducing additional computational or memory overhead, we retain the use of short-context data during calibration maintain low resource consumption. Furthermore, we propose leveraging positional interpolation (Chen et al. 2023) to embed long-context positional information into short-context data, thereby enabling more accurate estimation of the data distribution for long sequences.
To sum up, the proposed PM-KVQ mainly contains the following contributions:
We design progressive quantization and block-wise memory allocation techniques tailored for long-CoT scenarios to fully utilize the memory budget of the target hardware and effec- tively reduce cumulative quantization error.
We propose to use short-context calibration data with positional interpolation to increase the effective length without incurring additional computational or memory overhead.
Extensive experiments on long-Col CoT LLMs, ranging from 7B to 70B, show that the pro- posed PM-KVQ achieves up to 8% accuracy improvement over SOTA baselines on rea- soning benchmarks under 4-bit/2-bit KV Cache quantization settings, while delivering a 2.73-5.18x throughput improvement over the 16-bit model.
2 RELATED WORK
2.1 LONG COT LARGE LANGUAGE MODELS
Long-CoT
LLMs aim enhance multi-step reasoning capabilities for complex tasks like mathematical proofs, scientific reasoning, and multi-hop QA. Models such as OpenAl-ol (OpenAl 2024), QwQ (Team] 2025). and DeepSeek-R1 (Guo al. 2025) employ ad- vanced techniques to extend CoT reasoning depth. DeepSeck, specifically, integrates iterative self- refinement and tool-augmented reasoning (e.g., code execution and symbolic solvers) to maintain
2

---

Published as a conference paper at ICLR 2026
coherence across extended reasoning chains. Its architecture emphasizes hierarchical decomposition of problems and crror-correctio mechanisms, achieving sate-of-the--aee art performance.
Table 1: The memory overhead of the long-Cof LLMs. The batch size is 16, and the context length 32K.
Model
Weights (GB)
KV Cache (GB)
DeepSeek-LLaMAA-88
16
64
DeepSeek-Qwen-922 DeepSeck-LLaMA-70B
64
128
140
160
While long-CoT can significantly improve model performance, it introduces excessively more de- coding tokens (e.g.. >32K tokens per request) and large GPU memory overhead. As shown in Ta- T. despite employing efficient attention designs, such as Multi- Multi-Query Attention (MQA) (Shazeer, 2019 Group- Query-uerr Attention (GQA) (Ainslic al., (023), and Multi-head Latent Attention (MLA) (Liu et al.] 2024a), the memory overhead of the KV Cache in long-CoT LLMs remains significantly large, often surpassing that of the model weights. Consequently, reducing the memory overhead of the KV Cache is important for large batch sizes and long context requirements.
2.2 POST-TRAINING KV CACHE QUANTIZATION
To alleviate the large memory overhead with long reasoning contexts, many efforts have been made to reduce the KV Cache size. Post-training KV Cache quantization stands as promising technique for efficient inference. KV Cache quantization methods try use low bit-width integers represent the cached key and value states, instead of using high bit-width floating point values. Existing methods typically apply asymmetric uniform quantization for KV Cache:
XBF16 - -2
asym
Sasym
max (BF16) Z
S aaym
2
where XBFIE denotes the 16-bit brain floating point (BF16) Key or Value tensor, Xasym denotes the integer Key or Value tensor, Sasym and 7 min min (XB16)))) denote the scaling factor and the zero point respectively, b denotes the quantization bit-width, [-] denotes the rounding function.
Specifically, MKLV (Hariri et al. 2025) discovers that the sensitivity of Key and Value tensors are quite different, with the Key tensors being more sensitive to quantization than the Value tensors. Therefore, MKLV simply assigns higher bit-width to Key tensors and a lower bit-width to Value tensors. WKVQuant (Yue et al.) 2024) proposes to change the data flow of the previous KV Cache quantization by using the unquantized current Key and Value to calculate the attention operator, and then quantize the current Key and Value. SKVQ (Duanmu et al., 2024) further improves the WKVQuant by using a sliding window that stores the most recent 128 Key and Value features floating-poin format to reduce the cumulative quantization error. MiKV (Yang et al., 2024) is inspired by H20 (Zhang et al.) 2023) to use the heavy-hitter oracle to discover the important tokens in a higher bit-width and quantize the rest of the unimportant tokens into a lower bit-width. KIVI (Liu et al. (202cc) discovers that the Value tensors are much flatter than Key tensors, and the outliers Key tensors typically appear in certain channels. To this end, KIVI utilizes per-channel quantization for Key Cache and per-token quantization for Value Cache in group-wiss manner to reduce the quantization error. RotateKV (Su et al. 2025) combines the thannel-wise equalization and the rotation- based equalization with Hadamard matrices further reduce the quantization error.
In this paper, we adopt effective strategies from prior work, such as storing the first token INTI6 and using a sliding window for recent tokens. To further reduce quantization errors, we propose two improvements: (1) Progressive Quantization - initially store KV cache in higher precision and gradually lower the bit- bit-width memory memory becomes saturated; (2) Block-wise Memory Al- location - allocate more memory to sensitive transformer blocks when capacity allows, thereby preserving performance.
3

---

Published as a conference paper at ICLR 2026
Deroding
Step
Budget
Decoding Step
Budget
Magnitude
Short-context: Incomplete distribution
Waste
4K
16-bit
Waste
Bi-widhh Shrinking
8K
2it
8K
16 bit 8 bit
8
Taken ID Long context:
Waste
Bh-widh Strtesing
16K
2-bit
8-bit => 4-bit Bi-w Strincing
4-bit
Magnitude
Large calibration overhoad
32X
2-bit
2 bit
2-bit
(a) Progressive Quantization
Taken ID
Memory Budget
Block BBokk
1ett 2-tt
3
Bookn Waste!
Memory Budget
Short-context with Pl:
Memory Budget
DQM
Block 3 2.8
Blcck 4..8
Book3 2
Bock
2
Magnitute
Complete Distribution Small overhead
Rick)
Rock
Wedd
Fully utbilize memory resources
(b) Block-wise Memory Allocation
Token ID
is short for final bit-width progressive quantization)
[c] Calibration with Positional Interpolation
Figure I: Method Overview. (a) Progressive quantization: progressively shrink the bit-width of KV Cache to fully utilize the memory budget. (b) Block-wise memory allocation: we allocate a higher bit- withw those transformer blocks with higher sensitivity.
(c) Calibration with Positional Interpolation to approximate the distribution of context data with short-context data.
3 METHOD
3.1 PROGRESSIVE QUANTIZATION
As discussed in Section 2.2 existing post-training KV Cache quantization methods quantize every decoding step, causing large cumulative errors. A sliding window with high-precision cache allevi- ates this, but very low bit-widths (e.g.. 2-bit) still lead to severe accuracy loss long-CoT tasks. We show that existing KV cache quantization methods underutilize the memory budget and miss opportunities to reduce cumulative errors. As illustrated in the left panel of Figure aa), SOTA methods store 2-bit KV Cache at every decoding step, causing substantial memory waste when the budget is not fully used.
To address the above issue, propose progressive quantization strategy to make full use of the memory resources by gradually shrinking the bit-width the KV Cache, thereby significantly reducing the cumulative quantization error. For each transformer block, use "Fbit" represent the final bit-width of the progressive quantization process. In this case, we can easily calculate the memory budget for each transformer block based on the maximum context length of the target long- CoT L.L.M As shown in Figure a right, the Fbit in this example is 2-bit and the maximum context length is 32K. During generation, we initially store the KV Cache in 16-bit format alleviate the large cumulative quantization error. Once the memory budget is fully utilized, we apply a bit- width shrinking strategy accommodate more tokens by progressively reducing the bit-width of the existing KV Cache. Specifically, we use powers of two for quantization bit- widths, gradually decreasing them in the order of 16, 8, 4, and 2 bits.
In addition, for the bit-width shrinking strategy, we design an "Equivalent Right Shift" strategy that mathematically equivalent to de-quantizing the 2h-bit KV Cache and then quantizing it to b-bit. Here, b can be 8, 4, or 2, corresponding to shrinking the KV Cache from 16-bit to 8-bit, 8-bit to 4-bit, and 4-bit 2-bit, respectively. Specifically, we formulize the bit-width shrinking strategy by using integer addition and shifting as follows:
X ((2 -2¹ 2 + 1) + 2² >> 36,
where X, and X represent the b-bit and 20-bit tensor respectively. We keep the zero point un- changed (Z,- 7z)) and increase the scaling factor to S = (2 1)S2x preserve the dynamic range of the data distribution. The detailed proof of equivalence for Equation (3) is shown Sec- tion D Furthermore, we compare the effect of three different bit-width shrinking strategies and show that the "Equivalent Right Shift" strategy achieves better performance, as detailed in Section 4.4.1
3.2 BLOCK--IIS WISE MEMORY ALLOCATION
Existing KV Cache quantization methods typically apply a uniform bit-width across all transformer blocks, which may not fully utilize the memory resources of the target hardwares. As shown in
4

---

Published as a conference paper at ICLR 2026
Figure (bb left, in this example, the target hardware has sufficient memory store the KV Cache uniformly in 2-Fbit format, leaving a proportion of wasted memory. However, switching to uni- form 4-Fbit format may exceed the memory limit and trigger out-of- error. Therefore, using a uniform bit-width for KV Cache may not fully utilize the available memory across different scenarios with varying memory resources.
To fully utilize the memory resource in different scenarios for better performance, we propose a block-wise memory allocation strategy assign higher bit-width for more sensitive blocks. In- spired by existing mixed-precision quantization methods (Li et al., [023] Zhao et al., [024), we employ first-order Taylor approximation to estimate the sensitivity of the model output pertur-
bations in the Key Cache and Value Cache. Here, we take the Key Cache as an example:
((Q)(K))) = L(K) + GK, O (K; Qb(K,)).
(4)
where L. is the loss function, i represents the i-th transformer block, K; is the Key Cache, Qs(-) is the b-bit quantization function, Gk, the gradients of the loss function with respect to the K; O is the element-wise multiplication operator. The Value Cache follows similar formulation.
To minimize the effect of KV Cache quantization in each transformer block, we aim to minimize the following sensitivity term:
b |GK, O - -o((K,)) + ||Gv, o (V, Qo(V:))
where ,4 denotes the sensitivity of the KV Cache in the i-th transformer block to b-bit quantization.
Taking into account the sensitivity of all transformer blocks, our goal is to assign an appropriate bit- width to each block to minimize the impact on the loss function, subject to a given memory budget. To this end, we formulate the block-wise bit-width allocation as the following Integer Programming problem:
N
arg min 24,6
Σ
Σ
Lib 81,bs
(6)
N
Σ LID = 1, Σ
Σ b
Ri.b (Mem(Q.(K.) + Mem(Qu(V.)) < M,
(7)
L1b (0.1),bE
(8)
where N is the number of transformer blocks, Mem(-) is the function to calculate the memory usage of the quantized KV Cache, .M the memory budget for the KV Cache of all the transformer blocks, Ij,b is the one-hot vector that indicates the bit-width choice b of the i-th block, and B is the optional bit-width set, detailed Section 4.1. The proposed Integer Programming problem can be effectively solved by CVXPY (Diamond & Boyd 2016) within few seconds.
3.3 CALIBRATION WITH POSITIONAL INTERPOLATION
Previous studies have observed that the Key Cache of LLMs contains outliers in certain channels, which significantly increases the quantization error. Approaches such as QServe (Lin et al.) 2024) address this issue by introducing channel-wise reparameterizztioo method to transfer the outliers in Key tensors into Query tensors:
(9) where i is the channel index, A the reparameterization factor of the i-th channel, and Q(-) the quantization function. Generally, A; is calibrated using small dataset of sequences with a typical length of 512 tokens, which is much shorter than the maximum output length of 32K tokens. The calibration process follows Equation 10):
P P= (QA) Q ((KA ")7) A - diag(X.)
X; =
max Km,i) m
(10)
where m is the token position index, and a is the parameter to adjust the strength of outlier transfer, which can be set as fixed number obtained by grid search (Lin et al. 2024
5

---

Published as a conference paper at ICLR 2026
However, applying the above reparameterization technique to long- Cong-CoT LLMs using short calibra- tion data (e.g., 512) may fail to accurately capture the distribution of the Key Cache. This limitation arises because Rotary Positional Embedding (RoPE) (Su et al.) 2024) is used to inject positional information into the Key Cache, which introduces periodic variations across different channels:
Km K K
cos md; - sin m0:] sin m0; COS m0;
Km,s [ Km,+
cos(mm) + [0] K2 m. m.i+ sin(m0; + ()
V
K2
(1I)
where K and K denote the Keys before and after RoPE respectively, d is the hidden dimension of each attention head, and 0 denotes the rotary frequency of channel and i+ d/2. Since 8; 0 2:/d decreases with increasing i, the frequency of the sine curve is extremely low in channels with indices near d/2 and d. For example, in the D --recc-R--Distill---e------
the lowest frequency sine curve has a period of up to 54,410 tokens. Therefore, when using short sequences of 512 tokens for calibration, as shown in Figure [[c top, we cannot obtain the reparameterization factor that can completely reflect the sine-like data distribution.
Directly increasing the length of calibration data significantly increases both latency and memory costs due to the ( complexity of the self-attention operator. Instead, we embed long-context positional information into short calibration data by leveraging positional interpolation (Chen et al., multiply position scaling factor s to the position index in the rotary matrix of RoPE for positional interpolation, as shown below:
2023). Specifically,
Kmm
[oos(s: ml:) sin(s md.)
sin(s mb.)] cos(s m0.)
Km.d K km,+
cos(s ml: + 40) sin(s m0: ()]
K2 + K2
R Kmstg]
m.
(12)
As shown in Figure [c) bottom, by applying positional interpolation, we can increase the largest positional index by sx without additional computation and memory overhead.
3.4 METHOD PIPELINE
In this paper, the proposed PM-KVQ combines the above three techniques achieve better long- CoT performance with low bit-width KV Cache quantization. (1) Before the inference process, we first profile the sensitivity of each transformer block based on the calibration dataset, detailed in
Section[4.1.J and solve the Integer Programming problem to set the proper Fbit for each transformer
block, as discussed Section 3.2 Then, we apply the channel-wist reparameterization technique by using the calibration dataset with positional interpolation, as detailed in Section 333) (2) During the inference process, we apply progressive quantization to the KV Cache by gradually lowering the bit-width from 16-bit to the allocated Fbit, as shown in Section 3.1
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUPS
4.1.1 DATASETS
For the calibration dataset, we use the arXiv subset of RedPajama (Weber et al. 2024 as cal- ibration dataset. This subset consists of academic papers, containing mathematical Tormulas and reasoning process. We randomly select 512 samples, each with length of 2,048 tokens, for cal- ibration. For positional interpolation, we set s 8,192 context length to 2,048 tokens. We set in Equation (10) by grid searching over [0,1] for the optimal a that minimizes the reconstruction loss of the self-attention operator with a grid size of 20.
Equation T2p. which means we embed an
For performance evaluation, we mainly focus on evaluating the long-CoT LLMs on the math- ematical reasoning and code generation benchmarks with long generation contexts (> 16K). For mathematical reasoning, we use the AIME-2024/2025 (AIME) 2025) and CMIMC-2024 (CMIMC) 2025) datasets. For competition-leve code generation, we select coding problems released between January 2025, and April 6, 2025, from LiveCodeBench (Jain et al. 2024). Besides, illustrated in Section C.2 we also evaluate the proposed PM- MVQ on the IFEval (Zhou et al.] (2023) dataset with short generation contexts (~1 to demonstrate its strong generalizability across different

---

Published as a conference paper at ICLR 2026
context lengths. We sample 16 responses for each mathematical problem, 4 responses for each code generation problem, and response for each instruction following problem, using a temperature of 0.6, top-p of 0.95, and maximum output length of 32,768 tokens.
4.1.2 BASELINES AND MODEL CHOICE
For baselines, we compare PM-KVQ with SOTA KV Cache quantization methods, including the uniform bit-width methods RotateK (Su et all]|222000 2025), KIVI (Liu et al., 2024c), and mixed-precision quantization method MiKV (Yang et al. 2024), which retains the KV Cache of heavy hitters in BF16 format and uses low bit-width for other tokens. We also compare PM-KVQ with KVTuner (Li et al. 2023) in Section C.4 Similar to KIVI, PM-KVQ stores the KV Cache for the first and most
recent 128 tokens in INTI6 format to mitigate performance degradation. All model weights in our experiments are in BF16 format.
For model choices, we evaluate the different quantization methods above on the Deepseek-RI- Distill (Guo et al. (025) series as well as the QwQ-32B model (Team] 2025). Specifically, the Deepseek-RI-Distil series is an LLM family distilled from DeepSeek-RI. We choose Deepseek- R1 RI-Dissil--Qven--BB/1///333 and
ranging from 7B to 70B.
4.1.3 BIT-WIDTH AND BATCH SIZE SETUPS
For the bit-width settings, to demonstrate the effectiveness of the proposed PM-KVQ. we select quantization bit-widths that lead significant performance degradation when using baseline meth- ods for each long-CoT LLM. Specifically, we use 4-bit for DeepSeck-LLaMA--1 and 2-bit for other LLMs. Notably, the bit-width for the proposed PM-KVQ stands for the Fbit, as discussed in Section 3.1 In addition, for the optional bit-width set B in Section 3.2 we use B = {4.8} for DeepSeek-LLaMA--
and B = 4] for other long-CoT LLMs. We use asymmetric group- wise quantization for KV Cache with a group size of 128, as shown in Equation ( All of the performance results are conducted with fake quantization on an 8xA100-800 GPU server.
For the batch size setups, assign target GPU with different memory resources for different LLMs to show the memory constraints in real-world scenarios, as shown in Table 2 On the one hand, to demonstrate the effectiveness of progressive quantization, we set the batch size for each LLM such that all methods can fully utilize the memory resources of the target GPU. Specifically, batch size of 8 for LLaMA-8B with 4-bit KV Cache, 40 for Qwen-7B with 2-bit KV Cache, and 16 for the other L.Mss, as shown in Table 2 On the other hand, to evaluate the effectiveness of block-wise memory allocation, we smaller batch sizes allocate more memory per instance, ensuring that higher bit-widths cannot be directly used under the same constraints. In this setting, we use a batch size of for LLaMA-SB with 4-bit KV Cache, 32 for Qwen-7B with 2-bit KV Cache, and 12 for the remaining LLMs, as also shown in Table 2 Results across more target hardwares can be found in Section C.5
4.2 MAIN RESULTS
As illustrated in Table 2 for long-CoTi LLMS smaller than 10B, we compare PM-KVQ with Ro- tateKV, MiKV, and KIVI. For the 2-bit DeepSeet-R1--isiill-eeen MiKV causes the model unable generate meaningful responses. The SOTA method KIVI also suffers from significant performance loss by up to 9%. PM-KVQ outperforms KIVI by up to 8% when applying uniform Fbit for each transformer block (batch size = 40). When the batch size is reduced to 32, each sample receives larger memory budget. However, this budget is still insuf- ficient to apply uniform 4-bit quantization across all blocks. As result, KIVI constrained to 2-bit quantization, underutilizing the available memory. In contrast, PM-KVQ leverages block-wise memory allocation to better utilize the larger memory, achieving an additional performance gain of up to 0.84%. For the 4-bit DeepSeck-RA. -iitttRii--iiiLLaaa-la ods by up to 6.5% on AIME-2024, and even achieve better performance than the original LLM on mathematical benchmarks. Besides, for LLMs smaller than 10B, the average voting accuracy of PM-KVQ exceeds KIVI by up to 15.56%, demonstrating greater stability of the proposed method. We also compare PM-KVQ with KIVI of different bit-widths in Section
applying RotateKV or
PM-KVQ surpasses the SOTA meth-
For larger long-Cor Tong-CoTBie LLMS from 10B to 32B, we only compare the proposed PM-KVQ with KIVI because MiKV and RotateKV fail to generate meaningful information under 2-bit quantization, as
7

---

Published as a conference paper at ICLR 2026
Table 2: Main results of long-Cor Language Models on reasoning-relatte benchmarks with SOTA KV Cache quantization methods. "BS" is short for "batch size".
Models
Quantization
Bit-width
AIME-2024
AIMB-2025
CMIMC-2022
LiveCode
(Target GPU)
Methods
(K-V)
pass@p1
Voting
puss GB
Voting
pass
Voling
pas 1
16-16
41,04a571
63.33
30 30.00
36.67
27.29
43.33
26. 16
RotateKV (BS=32,40)
2-2
0.001am
0.00
0.001
0.00
0.00
0.00
0.001.00
DeepSeek-
MIKY (BS=32)
2/16-2/16
0.00
0.00
0.63
3.33
20
3.33
5.86
Qwen 7B (1 (x900-224))
MiKV (BS=40)
2-2
0.00
0.00
0.001
0.00
0.00.n00
0.00
0.001.00
KIVI (BS=32,40)
2-2
32,08
43.33
24.58
33.33
20.83
23.33
19.00
PM-KVQ (BS=32)
2/4-2/4
40.21=15:1
66.67
28.96=4.00
40.00
25. 23
40,00
14.
PM-KVQ (BS=40)
2-2
40.00=5.4
60.00
28. 12=471
33.33
26.462444
40.00
24.57=1443
16-16
44, 17:449
66.67
30.63-2655
50.00
26 24.67
36.67
32. 14:
RotateKV (BS=6.8)
4.4
42,92.1199
66.67
27. 18
40.00
26.46.1151
30.00
1..0
DeepSeek--
MiKY (BS=6)
4/16-4/16
35.63+714
66.67
24 79=372
36.67
25. +3:5
33,33
27. 1.300.3
LaMM--88
MiKY (BS=8)
4.4
41.67.1845
60.00
26 78
43.33
22.92.1444
26.67
29.71
(Ix4090-24G)
KIVI (BS=6,8)
4-4
41,255665
60,00
27.92
46.67
26,25
36.67
10,29
PM PM-KVQ (BS=6) PM-KVQ (BS=8)
4/8-118
47.71
73.33
31.25
50.00
28.13+440
36.67
31.71=0as
44
43,33e557
63.33
31.25
50,00
28,96:25.0
40.00
11,57=1177
DeepSeek-
16-16
68, 13
80.00
50.00=577
60.00
49. 49.58=4444
66.67
1
45.7
Qwen-148
KIVI (88=12,16)
2-2
48.13244
70.00
33. 33.962117
43.33
27.71
33.33
34, 34.43
(LXA100-10G)
PM-KVQ (BS=12)
2/4-2/4
67.71=594
80.00
46.67=1.66
60.00
47.71=420
60.00
42.14
PM-KVO (BS=16)
2:2
63.33
83.33
2858
60.00
46.67 1:20)
70.00
DeepScek-
16-16
72.08
86.67
53. 12
66.67
52.50
70.00
46 86
Qwen-32B
KIVI (BS=12.16)
2-2
63. 63.964559
83.33
45.42=338
60.00
40. 40.63+517
56.67
40.43+1110
(1xA100-80G)
PM KVQ (BS=12)
2/4 2/4
69.17.595
83,33
48. 18
60.00
51,25
66.67
13.
PM-KVQ (BS=16)
2-2
67.29
83.33
48.96=7.3
63.33
50,42=146
73.33
43.57=082
16-16
78.541435
86.67
67.71
76.67
21.25
80.00
14.71
QwQ-32B
KIVI (BS=12,16)
2-2
61.25
76.67
$1.67=537
63.33
48 48.33
63.33
11.86=121
(1xA100-80G)
PM-KVQ (BS=12)
2/4-2/4
66.462131
80.00
49.58:4439
63.33
54.58
66.67
45.14:000
PM-KVQ (BS=16)
2-2
67.29
76.67
49.79=629
70.00
56.67=3ml
73.33
44.57=040
discovered in the 2-bit BeepSeek-R1-Distine -wailll--ee-7BBB.. As shown in Table 2 PM-KVQ also demon- strates superior performance compared KIVI, improving average pass @ I and voting accuracy by up to 15.00% and 17.78% on various LLMs. Especially, for the DeepSeek-RI 1 --iistill-Qwen-14B, KIVI causes a performance degradation of 21. 21.87% on CMIMC-2024, whereas PM-KVQ has a sig- nificantly lower degradation of only 1.87% and 2.91 % under batch sizes of 16 and 12, respectively.
For the 70B-level long-Cof LLM, we evaluate the 2-bit DeepSeck-R--DisiiL-LLLM----- on the AIME-2024 benchmark. The original 16-bit model achieves a pass@ I of 14%. When the KV Cache is quantized 2- using KIVI, the pass@) drops significantly to 51.88%. In contrast, the proposed PM-KVQ enables the 2- model to achieve a much higher pass @ 1 of 64.79% under both batch sizes of 12 and 16, outperforming the KIVI baseline by 12.91%.
model
4.3 EFFICIENCY ANALYSIS
We evaluate 7B and 32B long-CoT LLMs on an A100-80G GPU, comparing the throughput of PM-KVQ (Fbit=2) against the original 16-bit LLMs and the 2-bit KIVI baseline. We adopt the official set- tings of KIVI (Liu et al. 2024c), using its inference engine and 472-bit CUDA ker- nels for efficiency evaluation. Besides, we implement 16/8-bit CUDA kernels and bit-width shrinking kernels to support PM- KVQ. To fully utilize the A A100-80G mem- ory, we set the batch sizes of the original 7B and 32B models to 18 and 1, respec- tively, while the quantized models allow larger batch sizes of 110 and 4.
Table 3: The throughput (in tokens/s) across different quantization methods and output lengths.
Quantization Method
Output Length
Model
16K
24K
32K
101.40
65.69
52.06
DeepSeek- Qwen-7B
KIVI
506.72
352.44
284.10
PM-KVQ
424.72
323.48
269.51
12.34
10.35
8.92
DeepSeek- Qwen-32B
KIVT PM-KVQ
35.65
33.28
31.59
33.74
32.15
30.81
As shown in Table B. across different model sizes and output lengths, PM-KVQ achieves a 2.13-5.18x throughput improvement over the original 16-bit LLMs. Compared with KIVI, the throughput of PM-KVQ is at similar level, with a slight reduction primarily due to the use of higher bit-widths during inference. Notably, the overhead of bit-width shrinking is negligible, as it is trig-
8

---

Published as a conference paper at ICLR 2026
4-bit
Bit-width Shrinking
2-bit
(a) Direct Right Shift
(b) Modified Right Shift (c) Equivalent Right Shift (Ours)
Figure Different bit-width shrinking strategies when the bit-width is reduced from 4-bit to 2-bit.
gered only when memory is fully utilized. Overall, PM-KVQ incurs throughput degradation of 2.45-16.188% % compared to KIVI but achieves a substantial relative accuracy improvement of 10.57-23.48%% To further evaluate the efficiency of the quantization procedure, measure the latency of block-wise memory allocation and calibration with positional interpolation. As shown in Section C.7 both the 7B and 32B LLMs complete these procedures within one hour using PM-KVQ.
4.4 ABLATION STUDIES
In this section, conduct detailed ablation studies show the effect of bit-wise shrinking strategies introduced Section B.I] the sensitivity of different transformer blocks discussed Section 3.2.
and the effectiveness the positional interpolation discussed in Section 3.3
4.4.1
THE EFFECT OF BIT-WIDTH SHRINKING STRATEGIES
Table 4: Ablation results of different bit-width shrinking strategies.
Bit-width
AIME-2024
Model
Bit-width Shrinking Strategy
(K-V)
pass
Voting
16-16
44.17
66.67
Direct Right Shift
4-4
12.08
23.33
DeepSek-LLLaMAS
Modified Right Shift
4-4
28.75
46.67
Equivalent Right Shift (Ours)
4-4
38.33
66.67
We compare three different bit-width shrinking strategies for reducing the KV Cache from 2b-bit to bit. Specifically, b can be 8, 4, or 2, corresponding to shrinking the KV Cache from 16-bit to 8-bit, 8-bit to 4-bit, and 4-bit to 2-bit, respectively.
By directly right-shifting by b bits, only the higher b bits of the original 2b-bit value are retained. As shown Figure (a), to preserve the dynamic range of the quantized values, keep the zero point unchanged (Zp Z25) and increase the scaling factor to S. (2 + 1)52s to compensate for the magnitude reduction caused by the right-shift operation.
(1) Direct Right Shift:
(2) Modified Right Shift: This strategy also uses b-bit right shifting strategy to perform the bit- width shrinking. However, instead of keeping the dynamic range unchanged, this strategy aims to ensure that quantization levels sharing the same upper bits before the shift can have their mean values directly mapped to the lower bit-width representation, as demonstrated Figure 2 (b). To achieve this, we change the scaling factor by S₁ = 2n. S2, and zero point by Z, Zzb (S. S26)
(3) Equivalent Right Shift (in Section 3.1): As shown in Figure 2 (c). this strategy is equivalent to directly de de-qquatizing the 2h- KV Cache and then quantizing it to b-bit.
We evaluate the above three bit-width shrinking strategies on the AIME-200 2024 benchmark with As shown in Table 4 both the Direct Right Shift and Modified Right Shift strategies result in significant performance degradation, reducing the pass 1 by 32.09% and 15.42%. respectively. In contrast, the Equivalent Right Shift demonstrates notable improve- ment over the other two strategies, increasing the pass@) @1 by 26.25% and 9.58%, and maintaining a lossless voting accuracy. Therefore, we adopt the Equivalent Right Shift strategy in PM-KVQ.
4.4.2 THE SENSITIVITY OF DIFFERENT TRANSFORMER BLOOCKS
We analyze the sensitivity and the memory allocation results across different models. For models with parameter size less than 1OB, as shown in Figure 3 we observe that the deeper blocks tend to be more sensitive quantization and receive a larger memory budget for the KV Cache. In addition,

---

Published as a conference paper at ICLR 2026
10a1
8-Fbit 4-Fbit
Sensitivity
10"
10
15
23
31
Block Index
(a) DeepSeek
10a1
4-fbit
Sensitivity
2-Fbit
10
10a
Block Index
(b) DeepSeck-RI-Distiie DieenillQQee--BB
Figure 3: Sensitivity quantization of KV Cache in different transformer blocks. Different colors
represents different memory budgets.
in the DeepSeek-RI -iii-------iiti-ee------- model, the first block is more sensitive than the other shallow blocks. Our memory allocation strategy accurately captures this feature, assigning a higher memory budget to the first block accordingly. The block sensitivity of larger models is detailed in Section C.I
4.4.3 THE EFFECT OF POSITIONAL INTERPOLATION
We evaluate the long-CoT performance across varying lengths of calibration data and position scal- ing factor s. We utilize the DeepSeck-RR-Distiine --i-------sill--LLMA--B-- to generate four responses for each problem in the AIME-2024-1 dataset. As shown in Table when the calibration sequence length is set to 2,048, applying positional interpolation with using positional interpolation, achieving accuracy comparable that obtained using calibration sequences of 8, 192 tokens. We also observe that when increases to 16, positional interpolation may lead performance degradation. This indicates that the computational savings of positional interpolation are not unlimited, and overly aggressive scaling can indeed performance drop.
improves pass by 1.66% compared to
Table 5: Ablation results of different calibration sequence lengths and position scaling factors.
Calibration Sequence
Position Scaling Factor
Effective Length
AIME-2022--
Model
Length
pass
Voting
2,048
1
2,048
46.67
60.00
DeepSeek-
2,048
4
8,192
48.33
60.00
LLaMA-SB
2,048
16
32,768
46.67
53.33
8,192
8,192
48.33
60.00
5 CONCLUSION
In this paper, we introduce Progressive fixed-precision KV Cache Quantization (PM-KVQ), post- training KV Cache quantization method designed for long-CoT L.MMs. To reduce the large cumula- tive error caused by uniform bit-width quantization, we design progressive quantization and block- wise memory allocation techniques. To increase the effective calibration length without incurring additional overhead, propose new calibration strategy with positional interpolation. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed PM-KVQ and each proposed technique. Overall, the proposed PM-KVQ significantly outperforms SOTA baselines by up to 8% on reasoning-re related mathematics and coding benchmarks and achieves 2.73-5.18% throughput compared to the original 16-bit LLMs.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164, 92364201), Tsinghua EE Xilinx Al Re- search Fund, and Beijing National Research Center for Information Science and Technology (BN- Rist). We thank for all the support from Infinigence-A..
ETHICS STATEMENT
This work focuses on reducing the substantial overhead caused by the linearly growing KV cache in long-context processing through KV Cache quantization. On the one hand, the proposed PM-KVQ better preserves model accuracy after low- low-precision KV cache quantization, making more accessi- ble for cost-constrained institutions, individuals, and application scenarios. On the other hand, as lossy compression technique, quantization can introduce distribution shifts and performance degra- dation, potentially leading to increased hallucinations instruction-followe failures. Therefore, additional caution and oversight are required during deployment.
REPRODICIBILITY STATEMENT
We describe the calibration and evaluation datasets, well as the data processing procedures, in Section 4.1.1 All datasets and models used in our experiments are publicly available. Detailed information the quantization bit-widths and batch sizes used for each long-CoT LLM is also provided in Section 4.1.3 To facilitate reproducibility, we also release our source code along with detailed guidelines.
REFERENCES
AIME.
American invitational mathematics examination, 2025.
URL
https:/ //
artoforobblemoolvingggcc com/|
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305. 13245, 2023.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint 2rXiv:2306.15555. 2023.
CMIMC. Carnegie mellon informatics and mathematics competition, 2025.
URL https:
7/ //cmimc.mmati wwt.............mmmmmmmm........ cw..cmm..e....m.m edu/math
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. http:////hhttbb..o///eee-o/ooppp wttpa//ppa.ccopppsp///cccaappppa 2023.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddeed modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5,2011
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding- window key and value cache quantization for large language models. arXiv preprint arXiv:2405.0m 06219, 2024.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang. Zixiao Huang, Shiyao Li, Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprini arXiv:2406. 14909, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 25..2.2211..... 12948, 2025.

---

Published as a conference paper at ICLR 2026
Mohsen Hariri, Lam Nguyen, Sixu Chen, Shaochen Zhong. Qifan Wang, Xia Hu, Xiaotian Han, and Vipin Chaudhary. More for keys, less for values: Adaptive kv cache quantization. arXiv preprint arXiv:2502.15075 2025.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.00 07974, 2024.
Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed-precision quantization for efficient Ilm deployment. In NeurlPS 2023 Efficient Natural Language and Speech Processing Workshop, pp- 1-5, 2023.
Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. Kvtuner: Sensitivity- layer-wise mixed-precision kv cache quan- tization for efficient and nearly lossless IIm inference. arXiv preprint arXiv:2502.04420, 2025.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang. Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Yujun Lin*, Haotian Tang*, Shang Yang*, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W4a8kv4 quantization and system co-design for efficient Ilm serving. arXiv preprint arXiv:2405.044 04532, 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.04 04434, 2024a.
Ruikang Liu, Haoli Bai, LIN Haokun, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. In The 62nd Annual Meeting of the Association for Computational Linguistics (ACL2024) 2024b.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache, arXiv preprint arXiv:2402.02750, 2024c.
OpenAI. Introducing openai o1, September 2024. URL htt.:att//pppaaa.ccooo comp///
Noam Shazeer. Fast transformer decoding: One write-head all you need. arXiv preprint 1rXiv:1911.02150, 2019.
Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 1670137063, 2024.
Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: Accurate and robust 2-bit kv cache quantization for 1lms via outlier-aware adaptive rotations. arXiv preprint arXiv:2501.16383. 2025.
Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context IIm inference. arXiv preprint arXiv:2406.1077 2024.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https: //wpn////////mmmmmmmmm wtt.wwwwww....u.a...ee wt/blo///b///oo qwtp://w333322
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NewrIPS Datasets and Benchmarks Track, 2024.
12

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXi::2399.1745. 2023.
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression importance-aware mixed precision quantization. arXiv preprini arXiv:2402.18800 2024.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint arXiv:2402.12001 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
Tianchen Zhao, Xuefei Ning, Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficien few-step text-to-image diffusion mod- els with metric-decoupled mixed precision quantization. In European Conference on Computer Vision, pp. 285-302. Springer, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instroction-folloiing evaluation for large language models, arXiv preprint 1.0091 2023.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In this paper, LLMs are only used assist in polishing the writing of this paper. The technical content, experiments, and conclusions are entirely conceived and conducted by the authors.
B
ADDITIONAL DETAILS OF EVALUATION
B.1 INTRODUCTION OF DATASETS
American Invitational Mathematics Examination (AIME) (AIME 2025) a mathematics com-
petition for high school students. It contains 30 challenging problems each year, designed to assess mathematical problem-solving skills across various topics, including algebra, combinatorics, geom- etry. number theory, and other subjects covered in high school curricula.
Carnegie Mellon Informatics and Mathematics Competition (CMIMC) (CMIMC) 2025 is an
annual mathematics contest for high school students, hosted by students from Carnegie Mellon University. The competition contains problems of algebra, combinatorics, and geometry, with each category including ten standard problems along with one tiebreaker. Our model evaluation focuses on the standard problem sets.
LiveCodeBench (Jain al., 2024) is extensive and continuously updated benchmark designed to evaluate the performance of LLMs in coding tasks. It continually gathers new problems from competition platforms. The benchmark encompasses four distinct scenarios: code generation, au- tomated code repair, code execution, and prediction of test outputs. In our experiments, we focus specifically on the code generation scenario.
IFEval (Zhou et al. 2023 is benchmark proposed to systematically evaluate the ability of LLMs follow natural language instructions. The dataset contains 541 prompts, each annotated with one or more verifiable instruction types such as word-count constraints, keyword frequency, formatting requirements, or prohibitions on certain symbols. These instructions were deliberately designed to be automatically checkable, enabling objective and reproducible evaluation without the need for human annotators.
C ADDITIONAL EXPERIMENTS
C. THE SENSITIVITY OF DIFFERENT TRANSFORMER BLOCKS
For models with parameter size over 10B, shown in Figure 4 KV Cache in deeper blocks tend to be more sensitive than shallower blocks. We also observe that for the Qwen-basee based models, the first block exhibits large sensitivity. In particular, the sensitivity of the first block is the largest among the first fifteen blocks different Qwen-based models. This phenomenon is not observed in the LLaMA-based models.
C.2 PERFORMANCE IN SHORF-GENERATION-CONNTE TASKS
To verify the scalability of PM-KVQ to short-generation-comeex tasks, we evaluate it on IFE- val (Zhou et al.] 2023), an instruction-follovin benchmark. We follow the experimental setup described in Section and adopt the evaluation metrics provided by OpenCompass (Contributors, 2023]. Compared to reasoning benchmarks in Table non-reasoning tasks are less challenging and generally involve much shorter outputs. For instance, the average output length of the DeepSeek- Qwen-7B model is 13,904 tokens on AIME-2024 but only 1,182 tokens on IFEval. As shown in Table 6 although our method is not specifically designed for short-output scenarios, it outperforms KIVI and achieves accuracy comparable the original 16-bit models.
C.3 COMPARISON WITH KIVI OF DIFFERENT BIT-WIDTHS
In our experiments in Section 4.2
PM-KVQ occupies memory than KIVI before the KV cache bit-width is reduced to the final Fbit. However, under uniform Fbit settings, both our approach and KTVI consume the same amount of memory once the KV cache bit-width is reduced Fbit. In fact,
14

---

Published as a conference paper at ICLR 2026
4-Fbit 2-Fbit
Sensitivity
104
10
35
Block Index
(a) DeepSeck- RI-Distill Qwen- 14B
104
4-Fbit
Sensitivity
2-Fbit
10y
Block Index
(b) DeepSeek- RI--istill Qwen-32B
4-Fbit
Sensitivity
104
2-Fbit
10
Block Index
DeepScck- R 11-Distill-QwQ--
10
4-Fbit
Sensitivity
2-Fbit
1
10
Block Index
(d) DeepSeck-RI--isiil-LLAlaaa
Figure 4: Sensitivity to quantization of KV Cache in different transformer blocks. Different colors represents different memory budgets.
15

---

Published as a conference paper at ICLR 2026
Table 6: Results of Long-CoT Language Models on non-reasoning benchmarks with SOTA KV Cache quantization methods. "BS" is short for "batch size".
Models
Quantization
Bit-width
1vvl
(Target GPU)
Methods
(K-V)
Prompt Strict
Prompt Loose
Instruct Strict
Instruct Loose
16-16
58.77
68.50
63.27
72.60
RotateKV (BS=32,40)
2-2
0.00
0.00
0.00
0.00
DeekSeck-
MiKV (BS=32)
2/16-2/16
57.30
66.17
61.18
70.06
Qwen-7B
MiKV (BS=40)
2-2
0.00
0.00
0.00
0.00
(1x4090-24G)
KIVI (BS=32,40)
49.29
60.31
54.57
64.57
PM-KVQ (BS=32)
2/4-2/4
57.35
68.03
62.09
71.65
PM-KVQ (BS=40)
2-2
57.58
68.34
62.09
71.81
16-16
$7.82
68.98
61.61
RotateKV (BS=6.8)
4.4
58.06
68.50
61.37
71.50
DeepSeck-
MiKV (BS=6)
4/16-4/16
44.08
56,38
46.92
59.53
LLaMA-8B
MiKV (BS=8)
4-4
55.69
66.61
59.95
70.39
(lx4090-24G)
KIVI (BS=6,8)
57.35
68.35
71.14
71.81
PM-KVQ (BS=6)
4/8-4/8
58.77
69.61
63.74
73.70
PM-KVQ (BS=8)
57.58
68.19
71.34
DeepSeck-
16-16
70.14
78.74
74.40
81.73
Qwen-14B
KIVI (BS=12,16)
2-2
67.54
77.01
72.04
80.47
(1xA100-40G)
PM-KVQ (BS=12)
2/4-2/4
73.70
80.47
77.73
83.46
PM-KVQ (BS=16)
2-2
73.46
80.47
77.49
83.46
DeepSeck-
16-16
74.41
81.73
78.20
84.41
Qwen-32B
KIVI (BS=12,16)
2-2
72.51
79.84
76.07
82.36
(1xA100-80G)
PM-KVQ (BS=12)
2/4-2/4
75.83
83.15
78.91
85.35
PM-KVQ (BS=16)
2-2
76.07
83.30
78.91
85.35
16-16
82.94
88.03
86.97
90.71
QwQ-32B
KIVI (BS=12,16)
2-2
73.22
80.00
78.67
84.09
(1xA100-80G)
PM-KVQ (BS=12)
2/4-244
81.99
86.77
85.55 85.55
89.45
PM-KVQ (BS=16)
81.75
86.61
89.45
Table 7: Results of long-CoT Language Models reasoning-related benchmarks with KIVI of different bit-widths. "BS" is short for "batch size".
Models
Quantization
Bit-width
AIME-2024
AIME-2025
(Target GPU)
Methods
(K-V)
pass @
Voting
pass@1
Voting
16-16
41.04
63.33
30.00
36.67
KIVI (BS=32,40)
16-16
21.88
36.67
21.67
26.67
DeepSeek-
KIVI (BS=32,40)
8-8
36.67
63.33
24.59
33.33
Qwen-7B
KIVI (BS=32,40)
4-4
39.38
63.33
26.46
36.67
(1 (1x4090-44G)
KIVI (BS=32,40)
2-2
32.08
43.33
24.58
33.33
PM-KVQ (BS=32)
2/4-2/4
40.21
66.67
28.96
40.00
PM-KVQ (BS=40)
2-2
40.00
60.00
28.12
33.33
under the same GPU and batch size constraints, if KIVI adopts higher quantization bit-width, will consume more memory and therefore exhaust the available memory budget before reaching the maximum output length. We evaluate KIVI across different bit-width settings. When KIVI fully utilizes the memory budget, we truncate its output accordingly. As shown in Table 7 our method still outperforms KIVI by 0.83%-18.333
C.4 COMPARISON WITH MIXED-PRECISION QUANTIZATION BASELINES
We conduct additional experiments comparing our method with KVTuner on DeepSeek- -I-bckk----iitilll Qwen- under the same target GPU and batch size settings. For KVTuner, adopt pernchannel asymmetric quantization for the Key cache and per-token asymmetric quantization for the Value cache. As shown in Table g the pass @ of our method surpasses KV'Tuner by 2.71-6.04%.
16

---

Published as a conference paper at ICLR 2026
Table 8: Results of long-Cof Language Models reasoning-related benchmarks with the mixed-
precision quantization baseline. "BS" is short for "batch size".
Models
Quantization
Bit-width
AIME-2024
AIME-2025
(Target GPU)
Methods
(K-V)
pass
Voting
pass@)
Voting
16-16
41.04
63.33
30.00
36.67
DeepSeek-
KIVI (BS=32)
2-2
32.08
43.33
24.58
33.33
Qwen-7B
KVTuner (BS=32)
2/4/8-2/4/8
34.17
56.67
26.25
33.33
(Xx409)))
PM-KVQ (BS=32)
2/4-2/4
40.21
66.67
28.96
40.00
C.5 PERFORMANCE ACROSS DIFFERENT HARDWARE CONFIGURATIONS
When the memory capacity of the target GPU increases, PM-KVQ can either maintain the per- request memory allocation by increasing the batch size allocate higher Fbit for each layer, allowing the KV cache to use higher bit-widths. We evaluate our method across different GPU memory capacities by increasing the Fbit accordingly. As shown in Table our method consistently achieves accuracy comparable to the original LLM under varying Fbit settings.
Table 9: Performance of PM-KVQ across different target GPUs. "BS" is short for "batch size".
Quantization
Bit-width
AIME-2024
Models
Target GPU
Methods
(K-V)
pass@)
Voting
PM-KVQ (BS=32)
2/4-2/4
40.21
66.67
1x4090-24G
PM-KVQ (BS=40)
2-2
40.00
60.00
DeepSeek- Qwen-7B
PM-KVQ (BS=36)
4/8-4/8
40.63
63.33
1xA100-40G
PM-KVQ (BS=58)
4-4
40.83
66.67
PM-KVQ (BS=108)
8/16-8/16
42.50
66.67
1 AXA100-80G
PM-KVQ (BS=164)
8-8
41.87
63.33
C.6 INTEGRATION WITH SPARSE ATTENTION
The progressive quantization and the block-wise memory allocation can applied naturally to sparse attention mechanisms. Calibration with positional interpolation is compatible with sparse attention mechanisms that rely on RoPE. We combine our method with QuestAttention (Tang et al. 2024) and evaluate the accuracy DeepSeck-RR RR--Disiil--wenn- model. As shown in Table 10 our method does not degrade the pass @ performance of QuestAttention.
Table 10: Results of PM-KVQ combined with QuestAttention. "BS" is short for "batch size".
Models
AIME-2024
Methods
(Target GPU)
pass
Voting
DeepSeek-
41.04
63.33
Qwen-7B
QuestAttention
33.33
63.33
(I (0X400-)4))
QuestAttentionnPMMMKOV
33.58
60.00
C.7 EFFICIENCY ANALYSIS OF PRE-INFERENCE PROCESS
Before the inference process, PM-KVQ performs block-wise memory allocation and calibration with positional interpolation preparation. Following the experimental setup in Section 4.1 measure the time required for these pre-inference procedures. As shown Table [I] compared with QServe (Lin* et al. 2024). PM-KVQ leverages positional interpolation to reduce calibration
17

---

Published as a conference paper at ICLR 2026
sequence length from 8,192 2,048 tokens, substantially reducing the calibration time by up to 77.21%. The additional block-wise memory allocation procedure account for 22.50-23.5344 pre- inference time.
Table 11: Latency of bise memory allocation and calibration. "Pl" is short for "Positional Interpolation".
Calibration
Memory Allocation
Model
Method
Time
w/o Pl
w/ Pl
QServe (search a))
52 min
DeepSeek- Qwen-7B
PM-KVQ (BS=40)
13 min
PM-KVQ (BS=32)
17 min
QServe (scarch a))
187 min
DeepSeek-
PM-KVQ (BS=16)
44 min
Qwen-32B
PM-KVQ (BS=12)
/
57 min
QServe (search ())
408 min
DeepSeek-
PM-KVQ (BS=16)
93 min
LLaMA-70B
PM-KVQ (BS=12)
/
120 min
C.8 ANALYSIS OF PROGRESSIVE QUANTIZATION
Long-CoT tasks exhibit highly variable response lengths. For example, Table [2] summarizes the number of responses of
falling into different length ranges.
Table 12: The number of responses falling into different length ranges.
Model
Response Length
AIME-2024
AIME-2025
0-4K
40
63
DeepSeek-
4-8K
99
67
LLaMA-8B
8-16K
138
144
16-32K
203
206
For shorter responses (e.g.. <16K tokens in our experiments), our method keeps the KV cache higher bit-width than Fbit throughout decoding. This reduces cumulative quantization error compared with using Fbit KV cache, thereby preserving performance. For longer responses (e.g.. >66 tokens in experiments), although the KV cache eventually shrinks to Fbit, the first 16K tokens are generated using precisson KV cache. Consequently, the model benefits from more accurate carly-stage generations.
D PROOF OF EQUIVALENT RIGHT SHIFT
Theorem D.1 (Equivalent Right Shift). Given 16-bit floating point tensor XBFI6. let X2b and Xs. denote the 2b-bit and b-bit quantized tensors of XBF16. respectively. Then
X, -(220 + 1) + +22 2 >> 3b.
(13)
Proof. Let the zero points be Zza = ZB = Z. According to Equation given by
2
the scaling factors are
Sz. max (BXI)) -
S max max(XBFId (BFF16)
(14)
22h -1
2
follows that
So (2 + 1)52b
(15)
Define
X 7
X 7
X 26
X,
(16)
S21
S
18

---

Published as a conference paper at ICLR 2026
Then the quantized tensors are obtained by rounding:
X
X26
Xx xx= [
(17)
and we have By the definition of rounding,
X21 (2 +1)Xa.
(18)
1
X
S X2b X2h +
(19)
2
2
Dividing both sides by 2 + 1 yields
2bb 2
X 26 +
< X6 <
(20)
2 + 1
2 +1
Perform the Euclidean division of 2xb by 26 +
X26 = 9(2 + 1) +r, with S q -1,
(21)
Then,
r 2 +
< X, 4 r+2 1
2
(22)
2 +
2b +
Now consider the expression:
(2 2 + 1)(((2))))))) + 11 + 26-1)
((225 - 2 1Xxx +2-
>> 3b
23h
(23)
9 (2 2b + 1)(r + +22011
q +
236
We proceed by considering two cases for the remainder
Case 1: 1:05r r 2
Then,
2b
1
< q
2
<
X,
< 4
=
+
(24)
2
2 + 1
2
+1
2
Hence, rounding gives X, =
X =
Moreover,
9 + (226 - 2 + 1)(r + 2b-1)
(226 2b + 1) 2
22b 2
1
0,
(25)
2
23h
2
2
and
9 + (2 2h + 1)(r + 2-111
2h - 1 + (2
2h + 12 1 + 20-13
236
23b
(26)
(2b 1)2
1.
23b
Therefore,
+ (2 - 2b + +r) + 20 2b-1)
= 0,
(27)
236
and thus
((226 2b + +)(22 + 222-1)) >> 36 = q == =X,,
(28)
Case 2: 20-1 +1 +1 2b_
Then,
1
2b-1
1
2 + 2
q
2 a+
2
X +
+ 1.
(29)
26 +1
26 +1
Thus, rounding gives X, =
X
+ 1.
19

---

Published as a conference paper at ICLR 2026
Moreover,
+ (2 2h + 1)(r + 2--1)
(2 26 + 1) + 11220
+1 2--1)
236
+
1
1,
(30)
235
236
235
and
9 (2 2ht + (r 2
2h (22h
2 + +2(bbb + +2h-1)
23h
23b
236 + 230-1
22
226-1 + 20+1
+ 26201
(31)
236
236 + 2
+
22h-1
2 -2 -h
+
= 2
< <2
236
Therefore,
+ (2 2 + + +
1,
(32)
236
and thus
((2 -2 2 +
1) 2b-11) >> 3b q 1 X...
(33)
In both cases, the desired equality holds, which completes the proof.
E LIMITATIONS
In this paper, do not consider all of the attention mechanisms, such the multi-head latent attention (MLA), which quite different from the widely used Group-Query Attention (GQA). Besides, do not combine the proposed PM-KVQ with other system-level optimization techniques and inference engines, which yields for future work.
20


Reviews:
[{"review_id": "cmn2moqHCW", "reviewer": "Reviewer_nh48", "summary": "This paper propose progressive mixed precision KV Cache quantization, of which a higher bit-width is assigned to more sensitive transformer blocks tailored for long-CoT scenarios, aiming for the low memory usage and quantization error. Extensive experiments on 7B–70B long-CoT LLMs show that the proposed block-wise and mixed precision quantization method improves reasoning benchmark performance by up to 8% over SOTA baselines under the same memory budget and achieves 2.73–5.18× throughput over the original 16-bit LLMs.", "questions": "1. How can this method used with PageAttention, for the quantized KV Cache management?\n2. How can this method used with sparse Attention, such QuestAttention, DSA or NSA?\n3. Can you draw the theoretical analysis, why such progressive shrinking is useful for long-cot tasks?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The progressive mixed precision quantization is interesting: 1) initially, the high bit (16-bit) quantization is used for the short-sequence; 2) then progressively shrink the bit width, while the high sensitive transformer blocks are maintained with the high bit width to narrow the quantization error; 2) and the memory allocation is block-wise, which is adaptive to the PageAttention.\n\nSecondary, the experiments show good, especially for the long-cot tasks, the proposed method improves reasoning benchmark performance by up to 8% over SOTA baselines.", "weaknesses": "The mixed precision quantization of KV cache is mature in the academia, such as KVTuner which allocate different bit width for different layer and K/V by optimized search algorithm. So the comparison with SOTA mixed quantization methods is not enough. And the practical benefit on the hardware is not given, such as the memory access saving and the throughput increase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03rLZTg7Go", "reviewer": "Reviewer_u27E", "summary": "The paper proposes PM-KVQ, a post-training quantization scheme for long-CoT LLM inference that combines: a. progressive quantization (start at higher precision and shrink the KV cache bit-width only when memory is about to run out), including an “Equivalent Right Shift” rule (`Eq. 3`) for precise bit-width shrinking, b. block-wise memory allocation, cast as a small integer program solved with CVXPY to allocate higher bit-widths to more sensitive transformer blocks; and c. calibration with positional interpolation to expose short calibration sequences to long-ctx RoPE phases. Reported results on DeepSeek-R1-Distill (7B-70B) and QwQ-32B show improved pass@1/vote accuracy over KIVI/RotateKV/MiKV at 2-4-bit KV cache, with throughput close to KIVI but below it.", "questions": "- **L41**: What is the reference for the claim “to generate 128K tokens”?  \n- **Table 1**: I am skeptical of using **BS** in your setting. What is the number of output tokens? Since the reasoning model generates long-CoT with the mentioned BS and GPU memory, it’s most likely that we get CUDA OOM.  \n- **Table 1**: **QwQ** has 32B parameters. How do you use it with one A100?  \n- **Sec 3.2**: Why is **CVXPY** used? What was the reason behind that?  \n- **Sec 4.1**: Your evaluation metric is **pass@k** where *k = 1*. What is *n* (number of independent trials)?  \n- **Sec 4.1**: Regarding **Voting**, how many samples did you draw?  \n- Could the progressive quantization step, while helpful for memory control, also hinder GPU utilization by blocking parallel execution?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Clear diagnosis of two long-CoT pain points (cumulative error, RoPE low-frequency channels), with concrete formulations (Eqs. 9-12) motivating the positional interpolation trick. \n- Simple but effective shrinking rule (Eq. 3) that avoids round-trip dequantization in implementation. \n- Block-wise allocation objective is standard, implementable, and explains gains when memory is partially free.", "weaknesses": "- The paper only reports FP16 results, omitting bf16, which is the de facto standard for inference. Since bf16 offers wider dynamic range and distinct hardware behavior, excluding it leaves uncertainty about PM-KVQ’s performance and compatibility in realistic deployment settings. \n- Accuracy under fake quant: Reporting accuracy without real 2-bit/4-bit kernels weakens the claim that PM-KVQ is robust in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NhqL0NFvH7", "reviewer": "Reviewer_WmCs", "summary": "This paper addresses KV cache quantization from a utility-driven perspective. The authors hypothesize that by fully utilizing the available memory and reducing the quantization bit-width only when the memory becomes full, one can achieve better CoT reasoning accuracy. Based on this idea, they propose a progressive quantization approach with an Equivalent Right Shift strategy. Specifically, all KV caches are stored in full precision initially, and once the memory limit is reached, the caches are progressively quantized by half using the proposed strategy. The authors further suggest assigning different memory budgets to each transformer block and formulate this allocation as an integer programming problem. To enable effective calibration for long contexts, they adopt positional interpolation to approximate long-context calibration using short-context data. Experimental results show that progressive quantization outperforms state-of-the-art fixed-precision quantization methods while maintaining comparable efficiency.", "questions": "1. How does the method compare to KIVI that also fully utilizes the available memory? Would there be a way to conduct such a fair comparison?\n\n2. For the Equivalent Right Shift strategy, is it reasonable to keep the zero point unchanged? Would it be possible for the zero point to vary across different quantization bit levels?\n\n3. In Table 4, when s increases to 16, the use of positional interpolation appears to cause performance degradation. Does this imply that the proposed method has inherent limitations? Overall, the method’s effect on pass@1 seems marginal.\n\n4. There is a typo around line 298: should it be CMIMC-2025 or CMIMC-2024?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Approaching KV cache quantization from a utility-driven perspective is interesting and practically relevant, as it reflects real-world deployment considerations.\n\n2. The figure illustrating the main idea of the paper is well-designed and easy to follow, effectively conveying the core concept.\n\n3. The authors conduct experiments on multiple models, demonstrating the broad applicability and effectiveness of the proposed method.", "weaknesses": "1. My main concern lies in the fairness of the experimental results presented in Table 1. The baseline comparison is rather limited, as KIVI serves as the only comparable baseline in most evaluations. Since KIVI uses a fixed precision while the proposed method can employ higher precision during generation, the comparison may not be entirely fair. It would be helpful to report the memory usage during generation for both KIVI and the proposed method to provide a more complete picture. It remains unclear whether the observed improvement primarily comes from the fact that the proposed approach occupies more memory overall during generation, which actually contradicts the reason why we apply quantization, as we want to reduce the memory usage.\n\n2. Because the proposed method depends on fully utilizing the available memory, it is important to evaluate its performance across different hardware configurations with varying memory capacities. Such an analysis would clarify how accuracy scales when the available memory changes.\n\n3. I find the use of positional interpolation insufficient to address the main challenge of long-context calibration. This method essentially distributes a small number of tokens over a wide range, leaving many positions without properly calibrated data. Moreover, according to Table 4, positional interpolation appears to offer limited improvement, which further supports this concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)