Channel-Aware Mixed-Precision Quantization for Efficient Long-Context Inference
Authors: Chengxi Liao
Zeyi Wen
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Long Context
Efficiency
Primary area: foundation or frontier models, including LLMs

Abstract: The key-value (KV) cache plays a vital role in accelerating autoregressive inference for large language models (LLMs). However, its linear memory growth with sequence length poses significant memory bottlenecks, especially in long-context scenarios.
Quantization offers a promising solution for memory efficiency. While existing methods typically apply channel-wise quantization to the key cache and token-wise quantization to the value cache, they suffer from severe performance degradation under low-bit configurations.
Our analysis reveals that quantization sensitivity varies across individual KV channels, presenting an opportunity for non-uniform bit allocation. Following this finding, we propose ChanMix, a mixed-precision quantization framework that supports channel-wise quantization on 2-bit setting with custom Triton kernels implementation. To improve low-bit quantization performance, we introduce a channel-aware bit reallocation strategy, which allocates bits across channel sensitivity.
Through extensive evaluation, ChanMix demonstrates superior performance across the NIAH, RULER, and InfiniteBench benchmarks for the Llama, Mistral, and Qwen model families, achieving improvements of at least 5 absolute percentage points on RULER compared to all baseline methods. Additionally, ChanMix enables a 2.3× increase in batch size and supports a 1.5× longer context length during inference. 
Our code is available at https://github.com/cxiliao/ChanMix.

Published as a conference paper at ICLR 2026
CHANNEL-AWARE MIXED-PRECISION QUANTIZATION FOR EFFICIENT LONG-CONTEXT INFERENCE
Chengxi Liao'
Zeyi Wen
'The Hong Kong University of Science and Technology (Guangzhou) The Hong Kong University of Science and Technology cliao1188coneer .hkust gz edu. venzeyi@hkust edu. en
ABSTRACT
The key-value (KV) cache plays a vital role accelerating autoregressive infer- ence for large language models (LLMs). However, its linear memory growth with sequence length poses significant memory bottlenecks, especially in long-context scenarios. Quantization offers promising solution for memory efficiency. While existing methods typically apply channel-wise quantization to the key cache and token-wise quantization the value cache, they suffer from severe performance degradation under low-bit configurations. Our analysis reveals that quantiza- tion sensitivity varies across individual KV channels, presenting an opportunity for non-uniform bit allocation. Following this finding, we propose ChanMix, a mixed precddprreision quantization framework that supports channel- wise quantization on 2-bit setting with custom Triton kernels implementation. To improve low- bit quantization performance, we introduce a channel-aware bit reallocation strat- egy, which allocates bits across channel sensitivity. Through extensive evaluation, ChanMix demonstrates superior performance across the NIAH, RULER, and In- finiteBench benchmarks for the Llama, Mistral, and Qwen model families, achiev- ing improvements of at least 5 absolute percentage points on RULER compared to all baseline methods. Additionally, ChanMix enables 2.3x increase in batch size and supports a 1.5x longer context length during inference. Our code is available at htttps:// com/ cxi 1 /iao/ ChanMix
INTRODUCTION
Large language models (LLMs) have advanced rapidly in recent years, with growing demand for complex and longer input tasks (Shaham al. 2023 Bai et al. 2025 Zhang et al., 2024). This has led to significant expansions in context length. making memory critical bottleneck during inference. For instance, Llama-3.1 (Grattafiori al. 2024) requires 125GB of memory per million tokens (Liu et al. 2024b), exceeding the memory capacity of most GPUs. Moreover, the memory footprint of KV cache restricts batch sizes, particularly for long-context scenarios. Efficient and effective KV cache compression thus remains key challenge in LLM serving (Jiang et al.) (025).
Approaches to reducing KV memory overhead can be divided into three categories. The first category is using shared attention heads across the KV cache by modifying the attention mecha- nism (Shazeer 2019; Ainslie et al. 2023 Liu et al., 2024a). While effective, these methods require fine-tuning limiting their universality. The second cate- gory is token eviction methods (Zhang et al. 2023 Li et al.) 2024 Ge et al., 2024), which discard less important tokens based scoring heuristics. Though memory-efficient, such methods incur significant performance degradation with permanent information loss. The third category is quan- tization methods (Yao et al. 2022 Liu al. 20224d] Hooper et al. 2024 Yang et al.) 2024b] Kang et al. 2024 Su et al., 20255), which compress the KV cache by using uniform bit allocation across channels. W While these quantization introduce minimal quality loss general tasks, they still struggle with specialized scenarios like long-context retrieval, especially in low-bit bit settings.
architectural changes and pre-training
ChanMix aims to boost the performance of quantization- based methods. We first investigate the
distribution of KV cache channels. While prior works (Hooper et al. 2024 Liu et al. 2024d Zhao
"Corresponding author.

---

Published as a conference paper at ICLR 2026
et al. 2024 exploit outlier channels in the key cache to alleviate quantization error, uncover that some robust channels with small- magnituee exist. This asymmetric sensitivity motivates us to allocate quantization bits in channel-aware manner to reduce overall quantization error. We then lo- cate one potential source of performance degradation in long-content scenarios. Recent studies (Wu et al. 2025 [Tang al. 2025 Xiao et al., 2025) have shown that certain channels are crucial for long-context retrieval. However, the impact of quantization on such channels remains unexplored. Our further experiments reveal that these retrieval-sensitive channels are highly vulnerable to quan- tization, and their bit precision disproportionatll affects long-come long-context performance. This finding explains the degradation of existing uniform quantization methods. Combining these two insights, we propose ChanMix, channel-aware mixed-precision quantization framework for the KV cache. We further introduce a novel bit allocation strategy that adapts to channel sensitivity: for sensitive channels (i.e., retrieval and outlier channels), higher bit precision is assigned to mitigate perfor- mance degradation; for robust channels (i.c., small magintd channels), lower bit precision used to save memory. This adaptive allocation enables ChanMix to significantly improve model perfor- mance while maintaining low memory footprint. ChanMix is fully compatible with existing token pruning methods (e.g.. (Xiao al. (025)) as well as the quantization error compensation methods (e.g., (Kang et al. 2024] Yankun et al.) (2025), ), making it a versatile solution for long- K cache compression. To summarize, our contributions include the following three key advancements:
ChanMix Framework: A channel-wise mixed-precision quantization framework for KV cache that supports 2-bit, with channel reordering for efficient 8- 8-bit-aligned packing and custom Triton kernel implementation.
Sensitivity-Awars Allocation: A novel bit allocation strategy that leverages a three-way asym- metry in channel consitivity-retrieev and outlier channels require higher precision, while small- magnitude channels can be quantized aggressively.
Extensive Evaluation: Comprehensive experiments on Llama, Mistral, and Qwen across NIAH, RULER, and InfiniteBencc show that ChanMix consistently improves accuracy over uniform quantization. Efficiency experiments further highlight our advantage by enabling 2.3x increase batch size and supporting a 1.5x longer context length.
RELATED WORK
2.1 LONG-CONTEXT LLMS
Long-context ability in LLMs becomes increasingly important as the demand for processing larger inputs grows. To extend the context length, one effective solution is fine-tuning the model with longer sequences. RoPE (Su et al.] 2024) and ALiBi (Press et al., 2022) are two widely used meth- ods which utilize relative position encoding, enabling the model to learn the relationships between tokens regardless of their absolute positions. Another solution is to use extra memory to restore the activation. ActivationBeacon (Zhang al. 2025) introduces a well-trained compression mod- ule that condenses activation tokens into beacon tokens, and then retrieves the relevant tokens from these tokens for generation, allowing the model to process longer sequences without fine-tuning. Existing models like Llama-3.1 (Grattationi et al. (024) can reach 128K context length with RoPE, while Gemini (Kavukcuoglu 2025) can achieve even 1M context length. The focus of long-context LLMS is shifting from extending context length to reducing the memory footprint of the KV cache. ChanMix aligns with this trend.
2.2 EFFICIENT LLMs INFERENCE
To reduce the inference cost of LLMss, the techniques can be broadly categorized into two directions:
Model Compression Model compression techniques aim to reduce the memory footprint of model weights and accelerate inference speed. Singular Value Decomposition (SVD) is a widely adopted low-rank approximation method that reduces matrix dimensions by approximating a given matrix using two smaller matrices. Recent studies, such as ASVD (Yuan et al. 2023). SoLA (Huang et al. 2025). and SVD-LLM (Wang et al. 2025). have enhanced SVD by incorporating weight im- portance metrics achieve better trade-off between model accuracy and memory efficiency. Quan- tization represents another major line of work. Some approaches, including LLM-QA QAT (Liu et al.] 2024ch and BitDistiller (Du et al. 2024). employ Quantization-Awue Awaree Training (QAT), retraining
2

---

Published as a conference paper at ICLR 2026
models with quantization constraints to minimize error. In contrast. methods such as GPTQ (Frantar & Alistarrh 2023), AWQ (Lin et al. 2024), and SqueezeLLM (Kim et al. 2024) utilize Post-Traine Training Quantization (PTQ), adjusting quantization parameters based on activation distribution to reduce ror. ChanMix is fully compatible with these weight compression techniques, as it specifically targets the reduction of memory consumed by the KV cache.
KV Cache Compression KV cache compression methods focus on decreasing activation mem- ory during inference. A prominent strategy token eviction. Recent work-including H2O (Zhagg 2023), Streaming LLM (Xiao et al., 2024), FastGen (Ge et al.
et al.
2024). SnapKV (Li et al,,
2024 PyramidKV (Cai et al. 2024), and DuoAttention (Xiao et al. 2025) observes that tokens contribute unequally to performance, and thus propose strategies to identify and prune less impor- tant tokens. However, such evietion approaches risk permanent information loss, particularly in multi-turn dialogue settings. Quantization has also been widely applied to KV cache compression. Methods like ZeroQuant (Yao et 2022). KVQuant (Hooper et al., 2024), KIVI (Liu et al. (024d), and WKVQuant (Yue et al. 2024) use uniform bit-width quantization across the KV cache, often employing channel-wise grouping for keys and token-wise for values, significantly reducing quan- tization error. ChanMix is fully compatible with these token pruning methods. For quantization, uniform bit allocation overlooks varying channel sensitivity, potentially leading to suboptimal per- formance. ChanMix addresses this limitation by adaptively allocating quantization bits according to channel sensitivity.
Mixed-Precision Quantization Numerous studies have employed mixed-precision techniques to enhance the inference efficiency of LLMs. For model weight compression, LLM-MQ (Li et al.) 2023) proposes layer-wise sensitivity-based precision allocation strategy for quantizing weight matrices. SliM-LLM (Huang et al. (2024) introduces a group-wise mixed-precision PTQ framework that assigns bit-widthh according weight salience. CMPQ (Chen et al. 2024) adopts a channel- wise quantization approach, adjusting precision based on activation distributions. PMPD (Chen) al., 2025) progressively reduces weight precision during inference balance efficiency and per- formance. For KV cache compression, MiKV (Yang et al. [024bb) ZipCache (He et al.] [0224) 2025a) identify salient or outlier tokens and allocate higher bit-widths to them. 2025) enables sensitivity-awe layer-wise mixed-prees precision quantization for the KV cache. QAQ (Cheng et al. 2025) adaptively allocates quantization bit-width based on out- lier characteristics and attention sensitivity. However, existing methods do not specifically address chamnel-sensitivity quantization of KV cache. ChanMix fills this gap.
and OTT (Su et al.
KVTuner (Li ct al.
3 BACKGROUND
In this section, we provide a brief overview of the transformer-based inference process and the concept of KV cache quantization.
3.1 INFERENCE PROCESS
The transformer-based (Vaswani et al., 2017) inference procedure consists of two distinct stages: pre-filling and decoding. In the pre-filling stage, the model processes the entire input prompt in one forward pass, computing and caching the KV states for all the tokens. This cached representation enables efficient attention computation during decoding. In the decoding stage, the model generates output tokens autoregressively. At each step of the decoding, the attention over the cached KV states computed, and then the next token conditioned on all the previously generated tokens is predicted. Finally, the KV cache with the new token's 's hidden states is generated.
Formally, let Xp E Rbxtxd denote the prompt embedding, where represents the batch size, I the prompt length, and dm the model dimension. The query, key, and value weight matrices W,. Wx, W, E Rm xds project these embeddings, with d, indicating the hidden dimension. In the pre-filling stage, the initialization of Q, K, and V is computed
= Xy
X,W,, V= VXXrW,
During decoding with KV caching, for each input embedding Xp E Rexxxdm, the key and value states are updated via concatenation:
K Concat K, XDWW), V Co
3

---

Published as a conference paper at ICLR 2026
The attention mechanism then computes:
QK" Vdh
Attention (Q, V) Softmax
3.2 KV CACHE QUANTIZATION
To reduce the memory footprint of the KV cache, quantization techniques are commonly applied. Early approaches such as group quantization (Yao et al. 2022) divide the KV cache into smaller groups and apply quantization within each group to mitigate quantization error. More recent studies (Liu et al. 2024d; Hooper et al.) 2024) investigate the distribution of KV activations and adopt mner-grained schemes, including channel-wiss quantization for the key cache and token-wise quantization for the value cache, which have demonstrated promising results. Formally, let denote the KV cache and b the number of quantization bits. For each group, we compute the zero point Z Cmin and scale factor S cache C is derived as:
Quantization Process
(Cmax Cminn) mm (2b 1). The quantized
Č Round (C-2 (c-2)
C SČ +
S
Here, Ö' is stored b-bit integer, while Z and S use full precision.
Quantization Error For a given group, the maximum quantization error Emar is defined as (detailed derivations are provided in the A.2
S
Emaz 2
(5)
which originates from both the rounding operation and the limited dynamic range of low-bit integers. Consequently, reducing the group scale factor S is essential for mitigating quantization error.
4 METHODOLOGGY
In this section, we first present our channel detection methods, then analyze the channel sensitivity of KV cache, and finally introduce our proposed ChanMix framework of KV cache quantization.
4.1 CHANNEL DETECTION
Outlier and Subnormal Key Channel We first sort the channel range from offline collected key cache, according to the quantization error analysis 5 Let K E IR NxC denote the key cache matrix, where N is the number of sample tokens and C is the number of channels. For each channel we compute its dynamic range R2 as:
R.
max Ki min Ki. ISISN
(6)
ISiSN
Then we use K-means clustering to cluster the channel ranges {Ri, R2...., Rc) } into three groups: outlier, normal, and subnormal. To ensure these three types of channel are clustering correctly, we set the three initial centroids {Rmin Rmedium, Rmar) ) for K-means clustering:
Clow, Cmedium, Chigh + K-means(BB) R Rc, k 3)
Channels in Chigh and Clow designated outliers and subnormal respectively.
Retrieval Channel In attention-based LLMs, retrieval heads play a pivotal role in the model's retrieval capability, which is a critical phenomenon revealed by recent studies (Wu al. 2025 Tang
et al. 2025 Xiao et al. 2025 Lee et al., 2025 Donhauser et al. 2025). We propose a simple yet efficient method, different from the previous works. Our approach stems from the hypothesis that model's long-context retrieval ability relies on capturing information between identical tokens across inputs (copy-paste operation (Wu et al. 2025)). So construct retrieval scenario to identify the retrieval heads. We first generate a semantically irrelevant sentence with tokens, and then around
repeat it times to create a sentence-level dependencies prompt. Here, n and t can be 100 and 30, respectively, according to our experience. Then we modify the attention mask (cf. Figure IT by removing local and sink attention scores (Xiao et al.) 2024). eliminating intra-sentence semantic dependencies. Finally, we input the constructed prompt to the model to compute each head's 's retrieval score using the retrieval mask.
4

---

Published as a conference paper at ICLR 2026
Formally, for sequence length and attention trix A E gixt we define the retrieval mask matrix M {0, 1 The retrieval score S for an attention head is computed as:
Retrieval Score Mask
The
sky
Is
blue
s - Aig My
8
The
sky
where o denotes the Hadamard product. Heads with higher aggregate scores are identified as retrieval heads, and channels in retrieval heads are defined as retrieval channels. Crucially, our method requires only one-shot inference, significantly reducing computational overhead compared to existing approaches.
The
sky
blue
The
sky
1s
blue
The
sky
i8
blue
The
sky
16
blue
4.2 CHANNEL SENSITIVITY ANALYSIS
Outlier and Subnormal Key Channel We analyzed key cache channels using 1,000 samples from WikiText- 2 and compiled the results. Figure (left) shows how the key cache values are distributed subnormal channels in blue, and outlier channels are in purple. There are clearly fewer outliers than subnormal channels. Then we evaluate the sensitivity of these channels in quantization bit width PPL metric. As Figure 2 (middle) shows, outlier channels become much worse with fewer bits, while subnormal channels stay stable. This phenomenon suggests that under the same bit-width budget, outlier channels should be allocated with more bits.
Figure 1: Our proposed A-shaped trieval mask removes noise score from attention sinks and local tokens to iso- late retrieval dependencies.
Retrieval Channel We also investigated the sensitivity of retrieval head channels to quantization bit-width, As shown in Figure (right), increasing the bit allocation for retrieval channels leads to more substantial improvement in retrieval accuracy, indicating that these channels are more sensitive to quantization performance.
Percentage of Entries
1.0
3.00
bit
Line Retrieva Accuracy
7.75
0.9
0.
7.50
0.8
0.-
bit
7.25
0.7
64
Channel
7.00
0.5
02
Outler
Suboormi
Minoo
6.75
Outiee Firtt
Subnomaa Fist
0.5
Settival Firet
Fandom Riss
6.50
0.4
0.0
128
896
1024
0.0
02
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Sorted Channel Index
2-bit Channel Ratio
4-bit Channel Ratio
Figure 2: Left: Stacked distribution (%) of key cache values with Llama-3.1 layer 20. Values categorized as outlier (top 25% of max layer range), normal (middle 50%), and subnormal (bottom 25%). For instance, Channel 128 (right) exhibits 40% subnormal and 60% normal entries. Middle: The perplexity on WikíTText-2, outlier channels are more sensitive to the decrease in the quantization bit. Right: The retrieval task accuracy, retrieval channels are more sensitive the increase in the quantization bit on the retrieval task.
4.3 CHANMIX FRAMEWORK
Overall Framework ChanMix applies straightforward quantization pipeline to the KV cache. During the prefill stage, after computing attention for each layer, the KV cache quantized into low-bit format- comprising integer tensors, scale, and zero-point parameters and stored in 8-bit aligned tensor. At the decoding stage, the quantized KV cache is dequantized back to full precision before attention computation.
Sensitivity-Aware Bit Allocation Following the analysis in Section 4.2 we propose two sensitive channel detection methods. As shown in Figure 3 (right), allocate 4 bits retrieval channels,
5

---

Published as a conference paper at ICLR 2026
Channel Index
Channel Quantization Bit
Nornal
Bit Reallocate
Sulmor
Taken Index
Channel-wwie Quantization
KV Cache
Outtier
Channel Reorder
Retrisval Channel
Token-wiis
Quantization
bit
Figure 3: Left: channel-wiss and token-witg quantization method. Right: bit reallocation for Chan- Mix (2-bit quantization). Reallocating channel bits by sensitivity: retrieval and outlier channels are allocated with bits, while subnormal with less. Channel reordering ensures efficient 8-bit- aligned storage of the quantized cache.
bits to outlier channels, and bit to subnormal channels. As shown in Section 4.2 the number of subnormal channels always exceeds that of outlier channels. Therefore, we reallocate only the same number of subnormal channels outliers to maintain 8 8-bit-aligned storage.
Implementation Details Following KIVI (Liu et al. 2024d), we apply channel-wise quantization to the key cache and token-wise quantization the value cache, Figure (left), and incorporate group-wise scaling and residual parameters to build our quantization framework. The quantized parameters stored in the float8 e4m3fnuz format minimize memory overhead. In the quantize- and-store step, a Triton kernel first efficiently quantizes the KV cache by calculating scale and zero-point parameters and converting the original cache into integer format. Another Triton kernel fuses reordering and storing operations to write the quantized KV cache into 8-bit aligned memory, minimizing memory copy overhead. During the Rad-aanddeeuantize step, Triton kernel reads the quantized cache from the 8-bit aligned tensor and dequantizes it to full precision. This kernel also merges reordering and dequantization reduce memory copy overhead. Note that our kernel implementation is fully compatible with FlashAttention, making it more general solution.
5 EXPERIMENTS
In this section, We first present the experimental settings, then the results and analysis of long- context and short-context experiments, followed by an efficiency analysis and ablation studies.
5.1 SETTINGS
Models We select 2iiaa-22-7b-22-Insttree (Llama-2) (together.ai) 2023) represent the multi-head attention (MHA) model, and -ertma------b--n-teeeee (Llama-3.1, 128K context length) (Grattafiori al.] 2024), length) (gradientai). 7b-
(Llama-3, 128k context (Mistral, 32K context length) (Jiang et al. 2023), and Awen-225514B--Inttree (Qwen-2.5, 32k context length) (Yang al. 2024a) represent the group- query attention (GQA) model, SrepSeek-Rl-LLammnn sent reasoning model in our evaluation.
(DS-RI-Llama) (Guo et al. 2025) to repre-
Datasets We evaluate ChanMix on the following benchmark to assess the long-context capa- bilities. The Ceedle-In-A-Hayaee
(NIAH) benchmark (Kamradi) 2023) is a synthetic benchmark designed to evaluate language model's capability to retrieve specific information from long-context scenarios. RULER benchmark (Hsieh et al. (0024) a benchmark designed to evaluate long-context language models with artificially generated synthetic dataset. InfiniteBench benchmark (Zhang ctt al. 2024) is designed to assess the ability of language models to handle long-context problems requiring processing, understanding, and reasoning over super long contexts (100K+ tokens) across real-world scenarios and synthetic scenarios. AIME is a mathematics reasoning benchmark fea- turing integer-anse answer problems that require multi-step symbolic derivations. While the input is short, it induces long reasoning outputs, evaluating sustained logical consistency and structured problem-solving ability. We also conduct experiments the following short-context benchmark.

---

Published as a conference paper at ICLR 2026
MMLU (Hendrycks et al. 20221 is a benchmark covering 57 subjects across STEM, humanities, and social sciences. MBPP (Austin et al. 2021) is dataset of beginner-level Python programming tasks where models must generate correct functions that pass unit tests. GSM8K (Cobbe et al., (0211 is math problem dataset focused on multi-step arithmetic reasoning.
Configurations ChanMix is implemented using PyTorch and Triton for fused quantization kemel operations. To detect the outlier channels, we randomly sample 10 entries from the Wikitext-2-raw: vI dataset (Merity et al., 2 as the profiling dataset. Both outlier and retrieval channel offline profile processes can be done in 10 minutes on a single GPU. We uniformly apply the following settings across all the experiments: quantization bits for subnormal, normal, outlier, and retrieval channels are set to 1. 2, 3, and 4 bits, respectively. with quantization group size of 32 and residual length of 128 for both key and value caches. All the experiments are executed on a single NVIDIA H800 GPU with 80GB memory.
Baseline methods To evaluate ChanMix's long-context capability, we compare against five state-of-the-art KV cache compression methods: ZipCache (He et al. 2024). KIVI (Liu et al.) 2024d), KVQuant (Hooper et al., 2024), DuoAttn (Xiao et al. 2025), and OTT (Su et al. (025b). For faimesss, all the methods (including the vanilla) use FlashAttention (Dao 2024) as the unified attention backend. All baselines use their original paper's default configurations and set KV size to around 20%.
Llama-2 (Context Length 32K Tokens)
Vanilla
ZipCache 20.5%
KIVI 19.1%
DuoAltn 21.8%
ChanMix 18.50%
Depth (%)
33
33
67 33
67 33
EE
100 67
69
100
1
Llama-3.1 3.1 (Context Length 80K Tokens)
Vanilla
ZipCache 20.4%
KIVI 18.9%
DuoAttn 20.7%
ChanMix 19.45%
Depth (96)
33
33
33
33
33
67
67
67
100 57
49
100
100
100
100
69K
gf
15k
30k
80k
Score
Figure 4: Bastle-i--a-Haystan (NIAH) benchmark results for Llama-2 and Llama-3.1 models,
compared with baseline approaches. ChanMix achieves the best for both model variants.
5.2 LONG-CONTEXT RESULTS
NIAH As shown in Figure ChanMix achieves near-vanilla model performance. Notably, it outperforms all the competing methods by large margin, effectively bridging the accuracy gap between the vanilla model and other quantization methods. This aligns with our design goal of minimizing memory overhead with minimal accuracy loss.
As shown in Table ChanMix maintains stable performance across diverse context lengths and models despite high compression ratios. Specifically, it incurs almost no accuracy degradation on Ilama-2, Mistral, and Qwen models, reducing the memory consumption by 81 %. In contrast, iiken-importance-baaee
RULER
methods (e.g.. ZipCache, OTT) exhibit significant accuracy drops, particularly for long contexts (longer than 16K). Quantization-basee KIVI and KVQuant show milder degradation but fail to match ChanMix 's efficiency-accuraa tradcoff. These results conclusively demonstrate that ChanMix maximizes long-context performance while minimizing memory overhead.
InfiniteBench Table 2 demonstrates that ChanMix achieves performance comparable to all the baselines across eight diverse tasks while significantly reducing KV cache size. These results vali- date that ChanMix maintains strong task-general capabilities while optimising memory efficiency.

---

Published as a conference paper at ICLR 2026
Table I: Comparison of ChanMix and other quantization methods the RULER benchmark. The best results are highlighted in bold.
Model
Method
KV Size
4K
8K
16K
32k
64k
128k
Average
Vanilla
100%
80.36
75.89
61.42
53.82
67.87
ZipCache
20.54%
65.11
54.87
33.71
13.91
41.90
Llama-2
KIVI
19.13%
68.34
65.03
53.31
41.31
57.00
32K
KVQuant
19.77%
70.84
66.28
59.43
49.6
61.54
DuoAun
21.81%
59.07
51.38
41.69
23.38
43.88
ChanMix
18.34% %
80.35
74.03
63.82
53.20
67.85
Vanilla
100%
91.66
87.64
86.96
81,69
86.99
ZipCache
20.62%
83.31
74.35
64.57
30.66
63.22
Mistral 32K
KIVI
19.19%
88.52
83.20
78.07
67.01
OTT
19.40%
77.49
70.14
65.57
55.17
67.09
ChanMix
19.59%
91.55
87.51
86.37
79.98
86.35
Vanilla
100%
88.73
87.89
81.86
76.86
71.59
70.2
79.52
Llama-3 128K
KVQuant
19.77%
83.88
81.71
73.08
65.79
62.05
62.18
71.45
ChanMix
19.45 %
87.8
86.4
81.75
76.15
71.7
70.01
78.97
Vanilla
100%
93.90
85.18
86.21
84.66
82.26
74.59
84.47
ZipCache
20.39%
87.69
76.31
68.49
60.67
OOM
OOM
73.29
Llama-3.1 128K
KIVI
18.90 %
85.42
80.10
76.07
72.58
68.36
OOM
76.51
DuoAttn
20.73%
73.11
67.36
62.33
60.82
55.99
34.58
59.03
OTT
18.98%
90.77
84.19
52.44
2.73
0.00
OOM
46.03
ChanMix
19.45%
90.64
86.01
83.66
82.68
80.55
69.41
82.16
Qwen-2.5
Vanilla 100%
97.27
95.53
94.11
92.17
94.77
32K
ChanMix
19.45%
97.3
95.68
94.14
91.4
94.63
Table 2: Comparison of ChanMix and other quantization methods the infiniteBench benchmark. The wold/underlined numbers indicate the first/second highest value in each column.
Model
Method
KV Size
Ea.Sum
En.QA
Er.MC
En.Dia
Zh.QA
Code.Debug
Math.Find
PassKey
Average
Vanilla
100%
20.50
12.70
40.17
5,00
18.44
25.63
13.14
27.12
20.34
KIVI
18.95%
17.41
13.72
43.67
7,50
18.77
25.89
11.71
21.19
19.98
1lama-2
KVQuant
19.76%
17.57
11.51
41,05
6.50
16.19
27.66
12.29
26.95
19.97
32K
DuoAmn
20.96%
13.50
16.91
39.74
0.00
11.69
20.05
0.86
15.76
14.81
ChanMix
18.23%
17.77
13.73
41,05
6.50
19.09
25.63
12.57
22.37
19.84
Vanilla
100%
23.31
9.39
$1.09
8.50
21.99
28.68
25.14
27.12
24.53
Mistral 32K
KIVI
18.95%
22.82
9.06
50,66
7,00
23.55
28.43
26.57
27.12
24.40
OTT
19.07%
21.84
8.91
$1.09
7.00
21.72
28.
26.00
27.12
24.05
ChanMix
19.48%
21.93
8.66
$1,09
7.50
23.43
29.19
26.00
27.12
24.37
Vanilla
100%
21.24
10.40
68.12
5.00
11.18
24.87
36.86
100.00
34.71
Llama-3 128K
KVQuant
19.76%
23.75
10.00
66.81
4.50
11.37
24.87
36.86
100.00
34.77
ChanMix
19.41%
23.02
9.97
66.81
5.00
12.22
24.87
36.57
100.00
34.81
Vanilla
100%
26.97
10.31
65.50
20.50
13.45
23.35
33.43
100.00
36.69
Llama-3.1 128K
DuoArt
20.26%
19.67
8.12
67.69
11.00
12.49
22.84
34,00
84,75
32.57
ChanMix
19.41%
24.23
9,41
65.94
16.50
12.93
23,35
33.43
100.00
35.72
Qwen-2.5 32K
Vanilla
100%
24.32
6.73
59.39
19.00
8.44
37.56
34.29
27.12
27.11
ChanMix
19.41%
24.66
7.01
60.70
21.50
8.64
37.56
33.43
27.12
27.58
Table 3 presents the performance comparison on the AIME24&25 benchmarks. ChanMix maintains comparable performance to other baseline methods. Notably, all three quantization-basee methods show performance degradation on these genggeeneattion tasks, indicating that low bit KV cache quantization remains key challenge for sustained reasoning.
AIME
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of ChanMix and KIVI the MMLJU, MBPP, GSM8K benchmark.
Comparison of ChanMix and other methods AIME bench- mark with pass@1 accuracy.
Table 3:
Model
Method
MMLU
MBPP
GSM8K
Method
AIME24
AIME25
Mistral
Vanilla
60.17
36.4
43.59
Mistral
KIVI
59.82
29.4
39.19
Vanilla
43.33
23.33
Mistral
ChanMix
60.15
33.4
42.46
KIVI
26.67
16.67
Llama-3.1
Vanilla
62.77
47.4
76.19
OTT 16.67
3lama-3.1
KIVI
62.57
44.2
74.75
ChanMix
26.67
Llama-3. 3.1
ChanMix
62.54
47.2
73.54
5.3
SHORT-CONTED RESULTS
We compare ChanMix with KIVI on three downstream short-context tasks: MMLU, MBPP, and GSM8K, covering general knowledge, code, and mathematical reasoning domains. As shown in Table 4 ChanMix performs better than KIVI in most settings. These results demonstrate the strong generalizability of ChanMix across diverse downstream domains and short-context scenarios.
5.4 EFFICIENCY ANALYSIS
We evaluate ChanMix's efficiency by comparing it with strong baseline, KIVI, and the vanilla model in three experiments. The memory presented peak memory usage collected during ference with PyTorch tools. In Figure 5 (left and middle), construct serving scenario (128 prefilling tokens + 368 decoding tokens), then increase the batch size. With our fused quantization and dequantization kernels, ChanMix outperforms KIVI with larger throughput and lower mem- ory usage, enabling 2.3x batch size than the vanilla model under the same memory. In Figure [ (right), fix the batch size to 1 and progressively increase the context length until out of memory (OOM) occurs. ChanMix supports 1.5x longer contexts than the vanilla model while consuming less memory. In contrast, the implementation of KIVI appears contain issues that might affect its long prefilling performance. These experiments show that our fused kernels sigmificantly reduce the memory copy overhead, achieving state-of-the-art performance in KV quantization.
10
Vanilla
Throughput token/s)
70
KM
Memory Usage (GBB)
60
ChanMix
Memory Usagg (GB)
Vanilla
30
Vanilla
2-
KIVI
30
KM
ChamMix
20
ChanMix
0
200
400
600
800
1000
10
200
400
600
800
1000
50K
100K
150K
200K
250K
300K
Batch Size
Batch Size
Context Length (tokens)
Figure 5: Throughput and memory usage. ChanMix can enable a larger context length and batch size with lower memory usage and achieve the best performance compared with KIVI.
5.5 ABLATION STUDY
Bit allocation strategy We conduct ablation studies to evaluate the effectiveness two sensitivity-aware bit allocation strategies. We use the RULER benchmark with LLaMA-3.1 and Mistral (Table 5) as the experiment setting. We begin with the base variant (ChanMix2), which applies 2-bit KV cache quantization without any enhancement: pure 2-bit KV cache quantization suffers significant accuracy degradation. Adding the retrieval channel component (ChanMix2+R) improves accuracy substantially while maintaining compression efficiency. The independent outlier

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study of ChanMix by combining types of sensitivity channels components.
Method
KV Size
Mistral
KV Size
Llama-3.1
Vanilla
100%
86,99
100%
84.47
ChanMix2
16.06%
72.13
15.78%
75.40
ChanMix2+R
19.59%
86.12
19.45%
81.50
ChanMix2+
16.06%
84.17
15.78%
81.13
ChanMix2+R+O
19.59%
86.35
19.45%
82.16
Table 6: Ablation study of ChanMix with different G and on RULER with Lama-3.1 model.
G
ChanMix2
ChanMix
G
R
Chan
ChanMix
G
ChanMix2
ChanMix
32
81.88
83.92
64
32
76.58
82.22
128
69.78
78.02
32
64
82.24
84
64
64
76.68
82.69
128
64
70.13
78.91
32
128
82.2
83.87
64
128
76.75
82.67
128
128
70.01
78.96
compensation variant (ChanMix2+O) also boosts accuracy without memory overhead. These results indicate that both enhancements are effective and independent. When combining both components (ChanMix2+R++)) we observe the best overall performance: 82.16 on Llama-3. and 86.35 on Mis- tral, which is remarkably close to the full -ree-ppecision results, confirming the complementary nature of the enhancements and the effectiveness of the full ChanMix framework.
Group size and residual length We further conduct experiments with different combinations of group size G and residual length on the RULER benchmark. Table shows that across nine configurations, ChanMix consistently outperforms ChanMix2, indicating that the observed perfor- mance gains stem from our proposed bit allocation strategy rather than specific choices group size or residual length. Moreover, ChanMix effectively mitigates the accuracy degradation under uniform low-bit quantization settings.
CONCLUSION
We have presented ChanMix channel-aware mixed-precision quantization framework for effi- cient long-contex. KV cache compression. Our framework starts from a novel aspect that allocates quantization bits by channel sensitivity, resulting in effective performance utilization of each quan- tization bit. With our bit allocation strategy, ChanMix achieves near-full-precisio performance the RULER benchmark, yielding least percentage-point absolute improvement over all baselines and models, while using similar memory. Efficiency results show that our custom Triton kernels support practical serving settings, allowing 2.3x larger batch sizes and J. 1.5x longer context lengths compared to the vanilla model. ChanMix consistently surpasses KIVI in both accuracy and efficiency, establishing stronger KV quantization baseline.
ACKNOWLEDGMENTS
This work is supported by National Key R&D Program China under Grant No. N0.2024YFA 1012700. It is also funded by the NSFC Project (No. 62306256) and the Natural Science Foundation of Guangdong Province (No. 2025A15150102611.
REFERENCES
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-hea head check- points. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
10

---

Published as a conference paper at ICLR 2026
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint 1777022000 2021.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang. Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang. and Juanzi Li. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computationaal Linguistics (Volume 1: Long Pupers), pp. 3639-3664, 2025.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406. 02069, 2024.
Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, and Stylianos Ve- nieris. Progressive mixed-precision decoding for efficient LLM inference. In The Thirteenth International Conference on Learning Representations, 2025.
Zihan Chen, Bike Xie, Jundong Li, and Cong Shen. Channel-wise mixed-precision quantization for large language models. arXiv preprint arXiv:2410. 3056, 2024.
Wen Cheng, Shichen Dong, Jayy Qin, and Wei Wang. Qaq: Quality adaptive quantization for I1m kv cache. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2542-2550, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning, In The Twelfth International Conference on Learning Representations, 2024.
Konstantin Donhauser, Charles Amal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, and Kartik Ahuja. Unveiling simplicities of attention: Adaptive long-context head identification. arXiv preprint 2044: 2025.
You Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. BitDis- tiller: Unleashing the potential of sub-4-bit LLMs via self-distillation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 102-116, Bangkok, Thailand, Au- gust 2024, Association for Computational Linguistics. doi: 1/2286633 httpps://cclantho.. wwtp::/..ooo020221111111111
URL
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In International conference on machine learning, pp. 10323-10337. PMLR, 2023.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024.
gradientai.
Llam-3-8l-iisttuc--aaaeeee----- 3aadiinntai/llaaa-----------
URL Settps://huggingeee
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint 207 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948 2025.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Ac- curate and efficient KV cache quantization with salient token identification. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Coleman Richard Charles Hooper, Schoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's ' the real context size of your long-context language models? In First Conference Language Modeling, 2024.
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14917 2024.
Xinhao Huang, You-Liang Huang. and Zeyi Wen. Sola: Leveraging soft activation sparsity and low-rank decomposition for large language model compression. In AAA1-25, pp. 17494-17502. AAAI Press, 2025. doi: 10. (669/AAALLV39911333999999
aaai v39116. 33923
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ann Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023.
Jiantong Jiang, Peiyu Yang, Rui Zhang, and Feng Liu. Towards efficient large language model serving: A survey on system-aware kv cache optimization. Authorea Preprints, 2025. URL hittp://da doi. wttp:/ooo..ogg11111/ 3tt.11..2222222tcheeeee
17604 63066066666666606060066606666666 6653101556111
Gregory Kamradt. Needle in haystack pressure testing llms, 2023. URL https://gtthht
Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. Gear: An efficient kv cache compression recipe for near-lossless generative inference of Ilm. arXiv preprint arXiv:2403.05522 2024.
Koray Kavukcuoglu.
Our Most Intelligent http:///w..gggggeee/eooooooooooo gemini model www.iiiiing-uuuattecc -aateeeeem-ar.... sanarchh2225.)
Gemini 2.5:
AI Model, 2025. deepminde)))
URL
Schoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In Interna- tional Conference on Machine Learning, pp. 23901-23923, PMLR, 2024.
Changhun Lee, Jun-gyu Jin, Younghyun Cho, and Eunhyeok Park. Seal: Scaling to emphasize attention for long-context retrieval. arXiv preprint arXiv:2501. 2025.
Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. Llm-mq: Mixed- quantization for efficient I1m deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurIPS, volume 9, pp. 3, 2023.
Xing Li, Zeyu XING, Yiming Li. Linping Qu, Hui-Ling Zhen, Yiwu Yao, Wulong Liu, Sinno Jialin Pan, and Mingxuan Yuan. KV'Tuner: Sensitivity-awar layer-wise mixed-precision KV cache quantization for efficient and nearly lossless LLM inference. In Forry-second International Con- ference on Machine Learning, 2025.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2026
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.044. 044444444 2024a.
Di Liu, Meng Chen, Baotong Lu, Huiqiang Jiang, Zhenhua Han, Qianxi Zhang, Qi Chen, Chen- gruidong Zhang, Bailu Ding, Kai Zhang, et al. Retrievalaatention: Accelerating long-context Ilm inference via vector retrieval. arXiv preprint arXiv:2409.10516 2024b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emnie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties:: ACL 2024, pp. 467-484, Bangkok, Thailand, August 2024c. Association for Computational Linguistics. doi: 10. 10.18653/w1/2224 findings-ache 2c1. https/t:::// wwl.ntholooy..ogg/000000 wincings-ac..22555
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. Forty-first International Conference Machine Learning, 2024d.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: zero-shot benchmark for long text understanding. In The 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint
arXiv:1911.021500, 2019.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang, and Min Zhang. Accurate KV cache quantization with outlier tokens tracing. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12895- 12915, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN doi: 10.18653/v1/2025 acl-long.631. URL htttps://cclatthott
979-8 879-8-891766 008-8-9976---1
orq /egg20225.0m acl acl-long........ 631/
Yi Su, Yuechi Zhou, Quantong Qiu, Juntao Li, Qingrong Xia, Ping Li, Xinyu Duan, Zhefeng Wang. and Min Zhang. Accurate kv cache quantization with outlier tokens tracing. arXiv preprint a5Xiv:2505. 10938, 2025b.
Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Danning Ke, Shikuan Hong, Yiwu Yao, and Gongyi Wang. Razorattention: Efficient KV cache compression through retrieval heads. In The Thirteenth International Conference on Learning Representations, 2025.
together.ai. Llama-2-7b-32k-instm
L.1 Llama- 7B- 32K- Instruutt 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2026
Xin Wang. Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, 2025.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanis- tically explains long-context factuality, In The Thirteenth International Conference on Learning Representations, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations, 2024.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang. Yao Fu, and Song Han. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. In The Thirleenth International Conference on Learning Representations, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.151155 2024a.
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Bunho Yang. Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-awee mixed precision quantization. arXiv preprint arXiv:2402.18096 2024b.
Hong Yankun, Li Xing, Zhen Hui-Ling, Yu Xianzhi, Liu Wulong, and Yuan Mingxuan. Svdq: 1.25-bit and 410x key cache compression for Ilm attention. arXiv preprint arXiv:2502 15304, 2025.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models, 2023.
Yuxuan Yue, Zhihang Yuan, Haojie Duanmu, Sifan Zhou, Jianlong Wu, and Liqiang Nie. Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint 12X05 2024.
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Long context compression with activation beacon. In The Thirteenth International Conference on Learning Representations, 2025.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. oeBench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402. 13718. 18, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy- hitter oracle for efficient generative inference of large language models. In Thirty-seventh Con- ference Neural Information Processing Systems, 2023.
Yilong Zhao, Chien-Ye Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6: 19992209, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LLM USAGE DECLARATION
We use LLMs for grammar correction and text refinement.
A.2 THEORETICAL DERIVATIONS FOR QUANTIZATION ERROR
For channel group, asymmetric quantization is performed by computing the zero point Z and scale (Cmax - (min)/ 1). The quantized cache
Cmin
(C (
Z S
Č Round
C SČ+Z
(9)
Let (C 2)/S, the quantization error defined as:
E
C (SÒ + B),
C (S Round (552) + Z),
(10)
S (æ Round(x)),
s
Therefore, the maximum quantization error Emax is: Fimax 2
(11)
A.3
EXPERIMENTAL DETAILS
A.3.1 EXPERIMENT SELECTION
Due to practical limitations, certain baselines were excluded or partially evaluated, as outlined be- low:
ZipCache: ZipCache encounters OOM errors for contexts exceeding 38k tokens. Consequently,
evaluated it only benchmarks with contexts under 32k tokens. We further excluded In- finiteBench due to its prohibitive execution time (estimated to be greater than one month per GPU).
KIVI: KIVI also fails with OOM beyond 64k tokens. Thus, we restricted its evaluation bench-
marks with contexts below this threshold.
DuoAttn: DuoAttn currently lacks support for the Mistral architecture. So, only evaluated it on Llama-2 and Llama-3.1 models.
KVQuant: KVQuant currently only support Llama-2 and Llama-3 models, so we only evaluate its performance on these two models.
A.3.2 KV SIZE CALCULATION
Table 7 summarizes the average token lengths across different benchmarks for the evaluated models:
Table 7: Average token lengths across benchmarks.
Model
NIAH
RULER InfinitBBench
Llama-2
17,000
14,874
32,000
Mistral
17,000
14,675
32,000
Llama-3.1 42,000
42,341
116,429
Given batch size 6, hidden dimension h. token length I, residual length r, and group size g, we derive the KV cache size under 2-bit quantization for Llama-3 with several baseline methods as follows:
15

---

Published as a conference paper at ICLR 2026
ZipCache: With 60% salient token ratio and 100:
RzipCache 26th 2 1.6 + 3/ 16 + 26l
16 2b
-hh-1 16
26lh 16
KIVI: With 32 and r 128:
26th 2 2.2Mh.16 16 2b
RKIV
26th 16
DuoAttn: With 20% sparsity, a sink size of 128, and 256 recent tokens:
2b(128 + 256 16 0.8 + 2bth 16 0.2
RauoAt
26th 16
ChanMix: With 30% retrieval head enhancement, 9 32, and r 128:
2blh 2 1.3 + 2-26th-8 + 2b 5 h- 16
RChanMix
g
26th 16
Here, each represents the normalized memory ratio compared full-precision KV cache, under the specific configurations of each method.
A.4 KEY VALUE CACHE VISUALIZATION
Figure 6 shows that in key cache, some fixed channels exhibit very large magnitudes, whereas in value cache. there no sigmificant pattern for outliers, this phenomenon suggests that we can apply channel-wise quantization for key cache, while it's not necessary for value cache.
Liama 2-78 Layer 12 Head 0 Key Cache
Llama-2-78 Layer 12 Head Value Cache
Liama 2 78 Layer 24 Head Key Cache
Liama 2 Layer 24 Head 0 Value Cache
10
b Tagg
bo og
200 Soreo
Charnel
0. 107
Chammal
200g 200 900
Charmel
200300 620
Charmel
Takon
Tokon
Tokan
Tokon
Uama Layer Head O- Key Cache
Liama 3.1. 88 Layer
Ulama 3.1 98 Layer 27
Liama-3. 3.1-6 Layer 27
Head D Value Cache
Head 0 Key Cache
Head Value Cache
1.5
14
1.0
10
& 25gg
2.60 2o
0 10
G00 Token
Creend
0%
100
g00
Chameal
100 Sai
Chamid
100
200 3m0
Charnel
Token
a00
Token
Token
Figure 6: Magnitude of key and value cache for Jama-2-7B-324 and
Table 8: Results on large size models.
Model
Method
nishuxl
minhha2
mishhct
mish.ml
mm2 niah.m3
minh.my nish.mg
coe
qa.l
qu.2
Average
Qwen-2.5
32K
Vanilla
100%
100
100
100
98
92
98.5
99.75
98.12
99.3
53.67
95.7
71
66.5 92.92
Qwen-2.5
32K
ChunMis
17.9%%
100
100
100
95
91.5
98
90.25
98.12
08.2
87.33
94
68.5
62.5
91.95
Llunna 8K
100%
99.5
99.4
Llama.3
ChanMix
19.78%
100
100
100
100
100
99
100
98.8
96
96.1
16

---

Published as a conference paper at ICLR 2026
A.5 RESULTS ON LARGE MODELS
We conduct experiments on RULER benchmark with two large size models: Qwen-2.5-32B and Llama-3.1-70B. Results table show that ChanMix perform well these large size models.
17


Reviews:
[{"review_id": "1prylks72S", "reviewer": "Reviewer_6Vs6", "summary": "This paper addresses the KV cache memory bottleneck for long-context inference. It identifies that the performance degradation of uniform low-bit quantization stems from asymmetric channel sensitivity. The authors propose ChanMix, a mixed-precision quantization framework that allocates bits based on a three-way channel classification: retrieval-sensitive channels (e.g., 4-bit), outlier channels (e.g., 3-bit), and robust/subnormal channels (e.g., 1-bit). The paper introduces an efficient one-shot method to identify these critical retrieval channels. \n\nChanMix is implemented with custom Triton kernels for efficient, 8-bit-aligned storage and dequantization. Experiments on Llama, Mistral, and Qwen show state-of-the-art results on RULER, NIAH, and InfiniteBench, significantly outperforming prior quantization methods.", "questions": "- How was the specific bit allocation (1, 2, 3, 4 bits) determined, and how sensitive is the model to changes in this policy?\n- How robust is the offline channel profile? Does a profile generated on Wikitext transfer to specialized domains (e.g., code, math), or is reprofiling required for optimal performance?\n- Why is the V cache not analyzed for channel-sensitivity and quantized channel-wise, similar to the K cache? Is the V cache less sensitive, or is this a design choice for simplicity?\n\nI will raise my score if authors give a good rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's core strength is the novel insight that \"retrieval-sensitive\" channels are a distinct category from magnitude-based \"outlier\" channels, and both require higher precision . This three-way sensitivity (retrieval, outlier, subnormal) is well-motivated by analysis and validated by strong ablation studies . \n- The proposed one-shot method for identifying retrieval heads is simple and efficient . The SOTA results (particularly the >5% gain on RULER) are significant. \n- I like the custom Triton kernel implementation part, which fuses channel reordering with (de)quantization, makes the method highly practical and efficient", "weaknesses": "- The primary weakness is the heuristic nature of the bit allocation policy (1, 2, 3, 4 bits). The paper does not provide an analysis of how this policy was derived or its optimality. \n- The generalizability of the one-time, offline channel profile is not thoroughly tested. It is unclear if a profile from Wikitext and a synthetic prompt  holds for all downstream tasks. \n- The paper asymmetrically analyzes the K cache (channel-wise) but not the V cache (token-wise), lacking justification for this design choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itiOIQfAri", "reviewer": "Reviewer_oLua", "summary": "The paper proposes ChanMix, a channel-aware mixed-precision quantization framework for compressing the key-value cache to enable efficient long-context inference. It identifies asymmetric sensitivities across KV channels, categorizing them as outlier, subnormal, and retrieval-sensitive, and allocates bits adaptively (1-4 bits) to minimize quantization error while maintaining low memory usage. The framework supports 2-bit to FP8 precision, includes channel reordering for 8-bit-aligned packing, and uses custom Triton kernels for implementation (but the code is not provided). Evaluations on Llama, Mistral, and Qwen models across NIAH, RULER, and InfiniteBench benchmarks show at least 5% improvement on RULER over baselines, with 2.3× larger batch sizes and 1.5× longer contexts.", "questions": "- Please address the items mentioned under Weaknesses. For example:\na. Lack of reproducibility\nb. Use of limited settings for group size and residual length\n\n- Your evaluation focuses mainly on long-context input-short-context output datasets. What is the performance of your method on long-context output tasks?\n- There is ambiguity in the “≥5% improvement” claim. The abstract states “improvements of at least 5% on RULER,” but the tables show absolute percentage-point gains compared to the baselines (e.g., +5-8 points). Please clarify whether “%” refers to percentage points or relative percent.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper's originality stems from its novel three-way channel sensitivity categorization (outlier, subnormal, retrieval) and adaptive bit allocation (4 bits for retrieval, 3 for outliers, 1 for subnormals; Figure 3, lines 290-294).\n- This assessment is based on the methodology (Section 4, lines 162-323) and claims in the abstract/introduction, emphasizing how it removes limitations from prior quantization works (e.g., uniform bits in KIVI, line 119) while the setting is very limited.", "weaknesses": "- Code is not provided, resulting in a lack of reproducibility.\n- The experimental setup does not clearly demonstrate the effectiveness of the proposed method. In `L-323`, a group size of 32 and a residual length of 128 are mentioned. The group size is relatively small, while the residual length is large. This configuration could be used to show the method’s effectiveness under less restrictive settings.\n- The retrieval-head detection hyperparameters (n, t) are fixed “by experience.” However, stability across different values and architectures is not explored", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Gnt7BNeEt", "reviewer": "Reviewer_HDr3", "summary": "This paper studies KV cache compression, which is important for LLM serving in real-world applications, especially in long-context scenarios. The authors claim that uniform bit allocation is the source of error for methods applying channel-wise quantization of keys and token-wise quantization of values. The authors propose ChanMix, a mixed-precision outlier-aware framework specifically designed for KV cache quantization. They evaluate their method on several benchmarks and develop a custom Triton kernel implementation to support their claims.", "questions": "1. How are outliers and subnormal channels defined? The first usage is Section 4.1 without proper definition.\n2. Can you please provide more explanation for the statement ‘Channel reordering ensures efficient 8-bit aligned storage of the quantized cache’? It is also not clear to me how this reordering is done.\n3. The largest models the paper has experiments with are Llama-3-8B and Qwen-2.5-14B. It would be good to test on larger models, such as Qwen3-32B, to see if the method scales well. Since sensitivity-aware bit allocation is a key contribution, I think the paper would benefit from adding QAQ into the comparison. Have the authors done such experiments?\n\nI am open to increasing my rating if the weaknesses and questions are resolved.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well written. The tables and figures are well presented.\n2.\tThe numerical results (e.g. Table 1) are promising: ChanMix outperform several existing methods in most cases.\n3.\tThe efficiency analysis is clear, and a kernel implementation is useful for the community.", "weaknesses": "1. Lines 17–19 and 53 overstate the findings. While the data is compelling, it does not support the claim that all quantization error stems from uniform bit-width allocation. I recommend softening the language to allow for nuance. Mixed-precision quantization is not new, and sensitivity variation across channels is expected. That said, this appears to be the first application of mixed-precision to KV, which is appreciated. The paper is well-motivated, but the novelty and breadth of the claims should be tempered.\n2. While ‘outlier’ is a widely used term in quantization literature, retrieval channel is a more recent, niche idea emerging from interpretability studies of attention heads, not a canonical term in the context of quantization. The corresponding part in section 4.1 should be expanded to provide more definition and background as retrieval channel is important in the motivation and implementation of the method proposed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)