KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
Authors: Zunhai Su
Kehong Yuan
Venue: COLM 2025 (2025)
Keywords: quantization
kv cache
transformer
llm
attention
TL;DR: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs

Abstract: Key-Value (KV) cache quantization has become a widely adopted optimization technique for efficient large language models (LLMs) inference by reducing KV cache memory usage and mitigating memory-bound constraints.
Recent studies have emphasized the importance of preserving the original precision of KVs for the first few tokens to ensure the protection of attention sinks.
While this approach has proven effective in mitigating performance degradation, its underlying principles remain insufficiently understood.
Moreover, it fails to address the recent discovery that attention sinks can emerge beyond the initial token positions.
In this work, we elucidate the underlying mechanisms of attention sinks during inference by examining their role in the cross-layer evolution of extreme activation outliers.
Additionally, we provide a comprehensive analysis of the interplay between attention sinks and KV cache quantization.
Based on our enhanced understanding, we introduce KVSink, a plug-and-play method that effectively predicts sink tokens with negligible overhead, enabling more thorough preservation. 
Extensive experiments demonstrate that KVSink outperforms the existing Preserve-First-N (PFN) strategy, offering more effective preservation of attention sinks during KV cache quantization.
Moreover, when applied to the well-established KVQuant method, KVSink further improves perplexity (PPL) and reduces reliance on 16-bit numerical outliers.

Published as a conference paper at COLM 2025
KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
Zunhai Su
Shenzhen International Graduate School, Tsinghua University zh-su23emails. tsinghua edu. cn
Kehong Yuan* *
Shenzhen International Graduate School, Tsinghua University yuankh@sz. tsinghua. en
Abstract
Key-Value (KV) cache quantization has become widely adopted opti- mization technique for efficient large language models (LLMs) inference by reducing KV cache memory usage and mitigating memory-bound con- straints. Recent studies have emphasized the importance of preserving the original precision of KVs for the first few tokens to ensure the protection of attention sinks. While this approach has proven effective in mitigating performance degradation, its underlying principles remain insufficiently understood. Moreover, it fails to address the recent discovery that attention sinks can emerge beyond the initial token positions. In this work, we clu- cidate the underlying mechanisms of attention sinks during inference by examining their role in the cross-layer evolution of extreme activation out- liers. Additionally, we provide comprehensive analysis of the interplay between attention sinks and KV cache quantization. Based on our en- hanced understanding, we introduce KVSink, a plug-and-play method that effectively predicts sink tokens with negligible overhead, enabling more thorough preservation. Extensive experiments demonstrate that KVSink outperforms the existing Preserve-First-N (PFN) strategy, offering more effective preservation of attention sinks during KV cache quantization. Moreover, when applied to the well-established KVQuant method, KVSink further improves perplexity (PPL) and reduces reliance on 16-bit numerical outliers.
1 Introduction
Transformer-based (Vaswani et al., 2017) large language models (LLMs), including GPT (Achiam al., 2023), LLaMA (Dubey et al., 2024), and DeepSeek (Liu al., 2024a; Guo et al., 2025), have revolutionized various domains of artificial intelligence research, including natural language processing (Hadi al., 2023; Zhao et al., 2023), computer vision (Zhang et al., 2024), and multimodal understanding (Liang et al., 2024). However, the impressive capabilities of LLMs come with significant challenges due their extensive size and com- putational demands (Zhu et al., 2024), along with the substantial Key-Value (KV) cache generated during inference (Li et al., 2024a), all of which hinder their deployment and practical application. KV cache facilitates LLMs inference by avoiding recomputation of past KVs. However, as the batch size and context length increase, the oversized KV caches become significant memory bottleneck (Liu et al., 2024c). KV cache compression has emerged as a promising direction to mitigate this challenge (Shi et al., 2024; Li et al., 2024a), encompassing broad array of techniques sechiques-including including quantization (Hooper et al., 2025; Su et al., 2025a), pruning (Xiao et al., 2023; Zhang et al., 2023), fusion (Liu et al., 2025;
*Corresponding author: Kehong Yuan

---

Published as a conference paper at COLM 2025
scores laye
output of decoder layer
decoder layer output acress layers
1400
700
2207
14
41
Layer index
(a) Attention sinks
(b) Activation outliers
(c) Stable outliers
Figure Visualizations of attention sinks and extreme activation outliers in LLaMA2-7B. (la) illustrates the presence of attention sinks at tokens 0 and 14. (1b) shows extreme activation outliers in the output of the decoder layer 3, which emerge attention sink tokens. (1c) shows the distribution of stable outliers across decoder layers. Unless otherwise specified, all visualizations use the following input from MMLU (Hendrycks et al., 2021): "The following are multiple-choice questions (with answers) about machine learning. nn'n..."
Ilama 2, 2_78 query_states layer_3
llama 2_7B key. states layer_
llama 7B value states layer
150-
135-
$ 310-
Figure 2: L2 norm distributions of Queries, Keys, and Values for all heads in the attention layer 3 of LLaMA2-7B across first 15 tokens. The sink tokens (0 and 14) display significantly smaller norms compared to non-sink tokens.
Wan et al., 2024), budget-aware allocation (Xiao et al., 2024; Cai et al., 2024), and low-rank decomposition (Chang et al., 2024; Saxena et al., 2024).
Xiao et al. (2023) characterizes the widely observed attention sink phenomenon, where LLMs tend to assign disproportionately high attention to the initial token, as shown in Figure 1a. Further research (An et al., 2025; Guo et al., 2024) suggests that attention sinks have a profound impact on the compression of LLMs. Specifically, the corresponding KVs require higher precision during quantization for adequate protection (Hooper et al., 2025; Su et al., 2025a; Duanmu et al., 2024; Su et al., 2025b) and must also be preserved during pruning (Xiao et al., 2023; Zhang et al., 2023). Additionally, as shown in Figure 1b, attention sink tokens have been found to exhibit extreme activation outliers (Bondarenko et al., 2021; Sun et al., 2024; An et al., 2025), which significantly impact activation quantization (Liu et al., 2024b; Li et al., 2024b). Although the practice of retaining the original precision of KVs for sinks tokens has proven effective mitigating performance degradation, existing practices and understanding still have several limitations: (1) inadequate systematic analysis of attention sinks and their related phenomena during LLM inference; (2) lack of in-depth understanding of the mutual impact between attention sinks and KV cache quantization; (3) current implementations focus on statically persevering the KVs of first few tokens, which has proven insufficient in light of recent findings (Sun et al., 2024; Yu et al., 2024) indicating that attention sinks can occur at other positions, as shown in Figure la.
In this work, we aim to deepen the understanding of the role of attention sinks during LLM inference and their interplay with KV quantization, addressing the aforementioned limitations and advancing research in LLM compression and interpretability. We begin by elucidating the role of attention sinks through an examination of the cross-layer evolution of various types of extreme activation outliers in Section Previous studies (Sun et al., 2024; Guo et al., 2024) have identified that extreme activation outliers manifest in the hidden states between decoders. As shown in Figure 1c, these outliers emerge and stabilize in the intermediate decoder layers, maintaining persistent presence at sink tokens and LLM-specific channels, exhibiting large and consistent magnitudes. Given their distinctive properties and their central role in our analysis, we refer to them as stable outliers for clarity. Interestingly, we further observe that stable outliers follow a structured progression across decoder layers, evolving through the stages of emergence, stabilization, and dissipation,
2

---

Published as a conference paper at COLM 2025
driven by extreme activation outliers originating from the down-projection layer in the feed-forward network. Additionally, attention sinks emerge and persist throughout the stabilization stage, serving the core mechanism for maintaining stability, As shown in Figure 2, this mechanism simultaneously imposes distinct numerical characteristics on the Queries, Keys, and Values of sink tokens, inevitably affecting KV quantization.
Then, Section 4, we provide a comprehensive analysis of the interplay between attention sinks and KV cache quantization. We quantitatively assess the impact of attention sinks on various quantization schemes through quantization error analysis. Our study also confirms that quantization significantly disrupts the implicit attention biases introduced by attention sinks. Finally, based on our enhanced understanding, we introduce KVSink, a plug-and-play method that effectively predict sink tokens by leveraging the intrinsic relationship between attention sinks and stable outliers. Our contributions are summarized as follows:
We advance the understanding of extreme activation outliers and attention sinks, elucidat- ing the role of attention sinks during the stabilization phase of stable outliers, and clarifying the mechanism by which attention sinks influence KV cache quantization.
To the best of our knowledge, this is the first work to thoroughly analyze and reveal the mutual influence between attention sinks and KV cache quantization. Our work not only deepens the understanding of attention sink preservation in KV cache quantization but also provides valuable insights for the development of more refined approaches the future.
Extensive experiments demonstrate that KVSink addresses the limitations of existing Preserve-First-N (PEN) strategy, providing more effective preservation with negligible overhead. Additionally, KVSink further refines the well-established KVQuant method, leading to improved perplexity (PPL) and reduced dependence on 16-bit numerical outliers.
Preliminary
Transformer decoder. LLMs are typically structured stack of
Transformer decoder blocks (Vaswani et al., 2017), each consisting of multi-head self-attention (MHSA) layer and feed-forward network (FFN) layer. We use the widely adopted LLaMA (Dubey et al., 2024) ar- chitecture an illustrative example, with a concise depiction provided in Figure 3. After tokenization and embedding, the input to the first decoder can be represented as H 4 hif.,, 11) E Roxd where h; denotes the i-th input token, is the embedding dimension, and n is the length of the tokenized input sequence. Then, the output of the I-th decoder block, H' Ruxd is given by:
down
gate
up
LayerNorm
out
H' = FFN (LN5fu (H")) + H",
(1)
Attention
RoPE
H" - H -' MHSA (INmissa (H--)).
(2)
Q
K
V
where 1 < 1 < L, with L denoting the total number of blocks. LN refers to layer normalization (with LLaMA employing pre-norm), Ol representing the output of the MHSA, and H" denoting the output of residual summations after the MHSA. Previous studies have found that Transformer-based models tend to learn structured activation out- liers in H' and H", which give rise to specific attention patterns that focus on special tokens, including BERT (Kovaleva et al., 2019), Vision Transformer (ViT) (Bondarenko et al., 2023), and LLM (Sun et al., 2024). A more detailed discussion of related works on this topic is provided in Appendix A.
LayerNorm
Figure 3: Trans- former decoder.
Multi-head self-attention. After LN, the input H is projected through the weight matrices W WR, W/ E dxd to generate the Queries, Keys, and Values, which are then divided into
3

---

Published as a conference paper at COLM 2025
K heads, denoted Q²*, KIA, vic, for Sk S K. The MHSA is computed as:
Q
A Softmax
vd + +M)
O Concat (AlkvlA) Wo.
(3)
where represents the attention mask, and dx = d/K, K. For simplicity, the rotation position encoding (RoPE) (Su et al., 2024) applied to the Queries and Keys is omitted here.
Feed-forward network. Next, H" is passed through a new LN and subsequently enters the feed-forward network layer:
H FFN (INggn(H")) + H²=H-= (o (INN (Hs))) @ LN (HN Wa HHH, (4)
where Wg, Wa, and Wa are the weight matrices for the gating, up-projection, and down- projection. denotes the activation function, and o represents the Hadamard product.
KV cache, The inference process comprises two stages: the prefill phase and the decoding phase. In the prefill phase, the LLM processes the token sequence generated from the input and produces the initial output token. Each attention layer I computes and caches the KV tensors K eache and v cache' In the decoding phase, the model takes the newly generated token as input. Let f' Rlxd denote the input embedding of the I-th attention layer. Each attention layer computes the Queries, Keys, and Values t/2, 1 and Hy as:
l We tk = == Wk. fyy = -'y.
(5)
Then, I/x and fly used to update the KV cache, with the complete KV supporting subse- quent MHSA computations:
Kl <ache <- concat(K' (kl <acher k),, vi cache coneat( Vl cacher 1t))
(6)
The growing size of the KV cache presents significant challenges in terms of memory usage and access latency, underscoring the need for efficient compression. Low-bit quantization has emerged as an effective approach, with related works discussed in Appendix
3 Attention Sinks and Extreme Activation Outliers
To enhance the understanding and preservation of attention sinks, first present our findings on the intrinsic relationship between the cross-layer evolution of different types of extreme activation outliers in Section 3.1. Subsequently, we elucidate the core mechanism through which attention sinks maintain the stability of stable outliers in Section 3.2.
3.1 Cross-Layer Evolution of Extreme Activation Outliers
We begin by specifying the activations in which each type of outlier occurs, using the same notation as in Section 2:
(1) The input to the down-projection layer, denoted X' in is given by
+ = a (LN c (Nffa(H") W
(7)
(2) The output of the down-projection layer, denoted as x X is given by
x out x in Wa
(8)
(3) The output of the residual summation after the MHSA, denoted as H""
(4) The output of the residual summation after the FFN, denoted as H', where stable outliers emerge. Notably, it satisfies H' X + H"
4

---

Published as a conference paper at COLM 2025
Signed Distribution Across Layers
Initial Layero
Emergence Layer1
Stabilication Layer2
Dissipation Layer30
Pastion
Layer31
ioput down-prejectin
X
Magnitude of Activation Values
W
of
projectn Xg
Residual summation MHSA
Stable Outilers Residual
summation after FPN 1f
Figure 4: Visualizations of various types of extreme activation outliers in LLaMA2-7B. The l side illustrates the cross-layer distribution of various types of extreme activation outliers, while the right side demonstrates their behavior in different stages.
While all these outliers appear at attention sink tokens and exhibit significantly larger magnitudes, they differ in their locations within the model and follow distinct cross-decoder- layer distribution patterns. As shown on the left side of Figure 4, outliers in x and Xo appear in the early and late layers, while outliers in H" and H' remain present across the intermediate layers. Interestingly, we found that the cross-layer evolution of these outliers unveils intrinsic relationships. Stable outliers, driven by outliers in X and X out undergo a structured progression, which we categorize into five stages: initial, emergence, stabilization, dissipation, and final (as shown on the right side of Figure 4).
In the initial stage, no noticeable outliers are present. Then, in the emergence stage, extreme outliers first appear in X d_in' subsequently propagating to X! "_ouut and ultimately inducing the emergence of stable outliers in H' through the residual connection. The stabilization stage extends across the intermediate layers, encompassing the majority of the decoder layers. In this stage, outliers persist in H" and H', while dii and x dout no longer exhibit extreme outliers. Instead, their magnitudes decrease significantly, resulting in minimal contribution to the variation in extreme outliers. During the dissipation stage, extreme outliers re-emerge in X positions as those d and their propagation to X! Lou generates outliers at the same during the emergence stage, with similar magnitudes but opposite signs. After the residual summation, this results in a significant reduction or disappearance of the stable outliers. Finally, in the last stage, no noticeable extreme outliers remain, and the model is ready generate the output. Overall, these various extreme activation outliers demonstrate systematic relationships and interactions. Experimental results on additional models and inputs can be found Appendix D.
3.2 Attention Sinks and the Stabilization of Stable Outliers
notable characteristic of stable outliers that during the stabilization phase, they remain consistently present, with their values varying only slightly. This implies that during the stabilization phase, the FFN and MHSA layers make minimal updates to the hidden states. Motivated by this and drawing inspiration from previous research (Bondarenko et al., 2023), we conclude that LLMs leverage attention sinks to achieve this behavior in the MHSA layer. Specifically, this process is governed by the following two key mechanisms:
QKV suppression. As shown in Figures 2 and 5, the Queries, Keys, and Values of sink tokens exhibit significantly smaller norms compared to non-sink tokens.
5

---

Published as a conference paper at COLM 2025
meni bayer head)
layer head
Jaknn
14
(a) Cosine similarity
(b) Query states
(c) Key states
lwar head
value dtates head O
0-
attertion outpptt layer fwaad
Toven
14.
Channel
(d) Attention scores
(e) Value states
(f) Attention outputs
Figure 5: (5b), (5c), and (5e) illustrate QKV suppression. (5a) highlights the high cosine similarity of QK. (5f) visualizes the attention output.
Quantization Schemes
Per- Token Key Quantization Frror
Per-Token Value
Per Coannel Key
Quantization Error
Quantization Frror
overall
w/o Sink tokens
w/
Overall
w/o
w/
Overall
w/o tokens
w/
Sink
tokens
Sink
Sink tokens
Dynamic
1.30
1.38
0.07
0.10
0.11
0.01
0.28
0.26
0.31
9.00
9.57
0.50
0.68
0.72
0.08
2/0
1.78
2:30
29.32
2.68
0.23
7.58
Bits
overall
Sink tokens
w/o
Overall
w/o Sink tokens
Overall
w/o Sink tokens
Static
86.73
16.39
3,10
5.17
89.00 60
20:41
10.09
3.45
10:24
8.26
107.36
46.14
11,30
6.35
15,83
11.44
Table 1: The experiments are conducted using round-to-neares. (RTN) integer quantization and the mean squared error (MSE) metric on the LLaMA2-7B model, with the values in the table scaled by factor of 100. The quantization group size is uniformly set to 16, and global min-max is used for static quantization. Under dynamic quantization, w/o Sink tokens refers to MSE for quantization groups that exclude sink tokens, while w/ Sink tokens refers to MSE solely for groups containing sink tokens. Under static quantization, w// Sink lokens indicates that sink tokens are excluded during both the calibration and quantization.
High cosine similarity of QK. Although the norms of Queries and Keys are small, the cosine similarity between the Queries of non-sink tokens and the Keys of the sink tokens remains high (Gu al., 2024), resulting large attention scores, as shown in Figure 5a.
Due to these two mechanisms, a small number of sink tokens exhibit extremely high at- tention scores but small Values, while the remaining tokens receive lower attention scores, resulting in attention outputs with small values, as shown in Figure 5f. This perspective aligns with previous research (Bondarenko et al., 2023) on extreme outliers in Pre-LLM Trans- formers, but we arrive at this conclusion from novel angle by analyzing the stabilization of stable outliers during LLM inference. Notably, the QKV suppression mechanism imposes distinct numerical characteristics on the Queries, Keys, and Values of sink tokens, which is the fundamental reason for their sensitivity to quantization. Additional experimental results QKV suppression and high cosine similarity of QK are provided in Appendix E.
4 KV Cache Quantization and Attention Sinks
4.1 Impact of Attention Sinks on KV Cache Quantization
Due to QKV suppression, the inclusion of sink tokens within a quantization group can expand the quantization range and exacerbate quantization errors, ultimately leading to further performance degradation. This arises from the trade-off between range and precision. Recent KV cache quantization methods adopt diverse quantization schemes (Liu et al., 2024c; Hooper et al., 2025; Duanmu et al., 2024), yet the impact of attention sinks across these schemes remains largely unexplored. In this section, we conduct an comprehensive study of the impact of attention sinks widely adopted per-token and per-channel approaches

---

Published as a conference paper at COLM 2025
Lhavel
Curea
Cla
Witerbe Feet
11
Head
(a) Average cosine similarity
(b) Attention biases visualization
Figure 6: (6a) depicts the average cosine similarity of Lies pid; across all tokens for each head on LLaMA2-7B. (6b) visualizes the attention biases for several example heads.
S/W NaR022
1ow72 2060
sntertain N4.1 lyver
loar
(a) 16-bit
(b) 8-bit
(c) 4-bit
(d) 3-bit
(e) 2-bit
Figure/ 7: The impact of KV cache quantization on attention scores and attention biases
across various bit-widths, employing static per-token quantization with group size of 16.
under both dynamic and static quantization through quantization error analysis (presented in Table 1). An overview of low-bit quantization is provided in Appendix C.
Per-token quantization. For per-token dynamic quantization, since quantization parameters are computed within each group corresponding to single token, the QKV suppression of sink tokens does not influence other tokens. However, in static quantization, where parameters are calibrated and fixed throughout quantization, the influence of attention sinks propagates across all quantization groups, exacerbating performance degradation. As shown in Table 1, excluding sink tokens during static per-token quantization significantly reduces quantization error, decreasing by up to 81.1% for Keys and 68.2% for Values.
Per-channel Key quantization. Based on the observation that the Keys exhibit outliers in specific channels, while Values do not, some research (Hooper et al., 2025; Liu et al., 2024c) employs per-channel quantization for Keys. In per-channel dynamic Key quantization, groups containing sink tokens experience larger quantization errors, while those without remain unaffected. As shown in Table 1, quantization errors increase by 16.3% to 29.2% compared to those without sink tokens. In per-channel static Key quantization, similar to per-token static quantization, excluding the impact of sink tokens leads to a reduction in quantization error of up to 42.7%.
4.2 Effect of KV Cache Quantization Attention Sinks
Several previous research (Sun et al., 2024; Gu et al., 2024; An et al., 2025) suggest that attention sinks and extreme activation outliers introduce implicit attention biases and demonstrate that incorporating explicit learnable biases during training can effectively eliminate outliers. Building on this insight, we first validate the presence attention biases induced by attention sinks during inference and then analyze the impact of KV cache quantization on these biases. The attention output of token can be expressed as:
Attention
-ptr, = V)
Eto, + Lp.., iES
(9)
iCS
where of represents the attention score of the Query token f and the Key token i, denotes the set of sink tokens and Dú denotes the Value of token i. As shown in Equation 9, the
7

---

Published as a conference paper at COLM 2025
presence of attention sinks influences the attention output of each token t through Lies pfo, To verify whether this represents attention biases, we calculate the average cosine similarity of Lies plu; across all tokens for each attention head. As shown in Figure 6, we find that for each head, Lies plu; remains highly consistent across all tokens when attention sinks emerge, confirming that represents the attention biases.
Next, we analyze the impact of KV cache quantization on attention sinks and attention biases. As shown in Figure/ 7, quantization significantly affects both, with the impact becoming more pronounced as the bit-width decreases. Notably, since the biases introduced by attention sinks persist across all subsequent tokens and may contain global or other crucial information (Darcet et al., 2023), their influence on attention computation remains continuous and significant.
4.3
KVSink
Given the profound interplay between attention sinks and KV cache quantization, it is crucial to implement effective preser- vation mechanisms during quantization. Existing approaches (Hooper et al., 2025; Duanmu et al., 2024) statically preserve the first few tokens (PFN), overlooking the potential presence of attention sinks at other positions. Moreover, relying on attention scores to identify attention sinks is not practical solution. First, dynamically identifying sink tokens incurs significant overhead. Second, attention computations rely on optimized CUDA kernels, such FlashAttention (Dao et al., 2022; Dao, 2023), which do not expose intermediate results. To address this challenge, we propose KVSink, with an overview illustrated in Figure 8. As discussed in Section 3, both stable outliers and attention sinks manifest on sink tokens. Building on this, KVSink initially identifies the emergence of stable outliers and subsequently uses them as indicators predict the positions of sink tokens. Notably, identification of outliers both highly efficient and accurate. (1) Outliers exhibit extremely large magnitudes and are sparse in occurrence, which enables their efficient detection through straightforward top-k sorting approach. (2) The identification needs to be performed only once during inference at the emergence stage. Additionally, the emergence stage layer can be pre-identified and treated as a static pattern, as it is input-independent (see Appendix D). (3) For a given LLM, outliers consistently emerge in specific fixed channels (Sun et al., 2024; An et al., 2025). This allows identification to be restricted to a single pre-identified channel. (4) Since the initial input sequence typically sufficiently long to capture all attention sinks, performing outlier identification only during the prefill phase offers a more computationally efficient approach. The complete algorithm for KVSink is presented in Appendix G.
tch
Emergance
FFN
MHSA
Figure 8: KVSink.
Overview of
Experiments
5.1 Experiment Settings
To assess the benefits of KVSink, we first compare it with the existing Preserve-First-N (PFN) solution across various LLMs and KV cache quantization schemes. Additionally, to further highlight the improvements KVSink provides over well-established methods, we conduct experiments using the KVQuant method (Hooper et al., 2025). The efficiency analysis of KVSink is presented in Appendix F.
Models, tasks, and datasets. We evaluate KVSink across seven models: LLaMA2- 7B/13B/70B, LLAMA2-7B-chat, Mistral-7B, LLaMA3-8B and -.laMA3.1-8B-nnstreee (Tou- vron et al., 2023; Chaplot, 2023; Dubey et al., 2024). The evaluation includes PPL tests conducted on the WikiText-2 and C4 datasets (Merity, 2016; Raffel et al., 2020).
Comparison of KVSink with the PFN strategy. We compare the PPL reduction achieved by KVSink and PFN using the same number of tokens for preservation, with values set to 0, 5, 10, 15, and 20. The evaluation conducted based three basic quantization schemes using
8

---

Published as a conference paper at COLM 2025
CL PEEIINEE CASTEREEEAA
Cat RoMAEa 00 with mooe
ucise Stoalaarelae
Soae DC
LAN 2a ant
Figure Comparison of KVSink with the PFN strategy. The orange line denotes the PPL of the FP16 KV cache, green indicates the use of KVSink, and the blue line represents the PFN method. PPL closer to that of FP16 reflects better performance.
2 PPL
C4 PPL
KV Arg.
Methods
LLaMA2
LLaMA2
LLaMA3
LLaMA3.1
LLaMA2
LLaMA2
LLaMA3
LLaMA3.1 88-Instruct
7B
13B
8B
88-Instruct
7B
13B
8B
16-bit
4.57
6.75
6.63
6.05
7.37
16
KVQuant
7.46
16.26
7,31
9.75
9.46
26.29
8.53
10:43
KVSink-5
6.52
5.10
6.77
8.83
7.93
6.47
7.74
9.57
2-2.02
KV XVQuant-0.1%
5.73
5,00
6.84
8.80
7.11
6.39
8.06
9.25
2.04-2.06
KVSSnkk55
5.60
4.85
6.56
8.25
7:10
6.28
7.87
8.80
KV KVQuant-D.5%
5.62
4.86
6.51
7.91
7.11
6.24
7.87
8.71
2.16-2.19
*VSinkk5
5.54
4.76
6.34
7.69
6.95
6.22
7.81
8.65
KVQuant-1%, 5.53
4.80
6.37
7.58
6.94
6.23
7.82
8.53
2.32-2.35
-VVSin-5
5.44
4.71
6.22
7.47
6.81
6.19
7.78
8.50
Table 2: Application to the KVQuant method. KVSink-5 denotes the use of 5 sink tokens.
RTN INT2/INTS quantization: per-token Key and Value static, per-token Key and Value dynamic, and per-channel Key with per-token Value static. The quantization group size is uniformly set to 128, and the Wikitext-2 dataset used for the PPL test.
Application of KVSink to the KVQuant method. KVQuant (Hooper et al., 2025) employs a range of advanced techniques for ultra-low-bit KV cache quantization, delivering state- of-the-art performance, including non-uniform quantization, per-vector dense-and-spane quantization for isolating numerical outliers, and per-channel quantization for Keys along with per-token quantization for Values. However, to preserve attention sinks, KVQuant only excludes the first token during calibration and quantization, a limitation that KVSink aims to enhance. We apply KVSink to the 2-bit KVQuant method with various settings for numerical outliers, inclucing 1%, 0.5%, 0.1%, and no isolation of numerical outliers. The evaluation performed using PPL tests on the Wikitext-2 and C4 datasets.
5.2 Main Results
Comparison with the PFN strategy. As shown in Figure 9, KVSink outperforms PFN in almost all cases. For instance, when evaluating LLaMA2-70B with per-token KV static 4-bit quantization, preserving only sink tokens with KVSink results in a PPL reduction of 163.2, with only marginal increase of 2.5 in PPL compared to the FP16 baseline. In contrast, the PFN strategy exhibits the PPL of 59.5, as it fails to account for sink tokens at other positions. Additionally, the experiments also show that preserving only 5 sink tokens with KVSink is sufficient in most cases.
Application to the KVQuant method. As shown in Table 2, integrating KVSink with the well-established KVQuant method provides two key benefits. First, while KVSink introduces only minimal modifications, it consistently improves PPL,, with the benefits

---

Published as a conference paper at COLM 2025
becoming more pronounced as fewer numerical outliers are preserved. Second, KVSink reduces the dependence on preserving FP16 numerical outliers. For example, with the 0.1% setting, the performance with KVSink is maintained even exceeds that of the 0.5%. Notably, reducing the preservation of FP16 outliers leads to more efficient compression.
Conclusion
In this study, enhance the understanding of attention sink preservation in KV cache quantization and propose KVSink to improve the existing PFN solution. We elucidate the intrinsic relationships in the cross-layer evolution of different types of extreme activation outliers and highlight the pivotal role of attention sinks during the stabilization stage of stable outliers. We also thoroughly analyze the mutual influence between attention sinks and KV cache quantization. Experimental results demonstrate that KVSink outperforms the existing PFN strategy and improves established KV cache quantization techniques.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arv preprint arXiv:2303.08774, 2023.
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. ar arXiv preprint ar Xrio:2305.13245 2023.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang- Systematic outliers in large language models. ar preprint ar Xii::2002.0641, 2025.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcom- ing the challenges of efficient transformer quantization. arXiv preprint arXtt:2109.12948, 2021.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systemıs, 36:75067-75096, 2023.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. ar preprint arXio:2406.02069, 2024.
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yo Yu Lin, Chong- Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S Abdelfattah, and Kai-Chiang Wu. Palu: Compressing kv-cache with low-rank projection. ar Xiv preprint ar Xiv:2407.21118, 2024.
Devendra Singh Chaplot. Albert q. jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed. arXiv preprint ar arXie:2300066225 2023.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. Xiv preprint ar Xio:1906.04431, 2019.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. ar Xiv preprint arXiv:2307.08691 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memort-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. Xit preprint 0rXt::2309.166 16588, 2023.
10

---

Published as a conference paper at COLM 2025
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conferenne of the North American chapter of the association for computational linguisties: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ar Xiv preprint ar arXiv:2010.11929, 2020.
Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. Skvq: Sliding-window key and value cache quantization for large language models. arxiv preprint ar xxXit:2405.06219 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. Xiv preprint ar (rXi:2247.211883, 2024.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arviv preprint ar ar::::4410 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arviv preprint Xr1::25001.12948, 2025.
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arxiv preprint ar Xix::410.1138. 13835, 2024.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 3, 2023.
Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. Zipcache: Accurate and efficient kv cache quantization with salient token identification. aiv preprint ar Xiv:2405.1 14256, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances in Neural Information Processing Systems, 37:1270-1303, 20252 2025.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. Xiv preprint arXiv:1908 X.t::1998.08593, 2019.
Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, and Lei Chen. A survey on large language model acceleration based kv cache management. arXiv preprint arXit:2412..1944. 2024a.
Qingyuan Li, Bo Zhang, Liang Ye, Yifan Zhang, Wei Wu, Yerui Sun, Lin Ma, and Yuchen Xie. Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference. ar preprint ar Xrxo:22412.04964 2024b.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405 409, 2024.
11

---

Published as a conference paper at COLM 2025
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arxiv preprint ar Kio:2442.11437,, 2024a.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Reza Haffari, and Bohan Zhuang. Minicache: Kv cache compression in depth dimension for large language models. Advances in Neural Information Processing Systems, 37:139997-140031 2025. 2025.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. Xrv preprint arXiv:2403.0124 2024b.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free free asymmetric 2bit quantization for kv cache. arXiv preprint ar xxx::2402.002550 2024c.
Stephen Merity. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 9, 2016.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://mmm....ooppprrr.rsee
waw.rrogggggaaeeaaee wttgg/appps//111/// 20022222000222 0004.077.m html.
Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Attention in low-rank space for kv cache compression. arXiv preprint ar 4rX1o:2408.05646,
Eigen attention:
2024.
Luohe Shi, Hongyi Zhang, Yao Yao, Zuchao Li, and Hai Zhao. Keep the cost down: A review on methods optimize Ilm's kv-cache consumption. arviv preprint ar arXiv:2407.18003, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan. Rotatekv: Accurate and robust 2-bit kv cache quantization for lims via outlier-aware adaptive rotations. arviv preprint arXiv:2501.16383, 2025a.
Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, and Kehong Yuan. Akvq-vl: Attention-aware kv cache adaptive 2-bit quantization for vision-language mod- 2025b.
els. ar Xiv preprint ar Xrx::2501111500
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. Xr preprini X1X7::2402.17762, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrXiv:2307.092880 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-context inference. arXiv preprint ar Xio:2246.18119, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. aro preprint ar Xro::2309.1444 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. Xrv preprint arXio:2410.10819 2024.
12

---

Published as a conference paper at COLM 2025
Jaewoo Yang, Hayun Kim, and Younghoon Kim. Mitigating quantization errors due to activation spikes in glu-based llms. arXiv preprint arXiv:2405.14428 2024a.
June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. ar Xiv preprint arXi::2402.18096 2024b.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. Xiv preprint ar arXiv:2406.15766 15765, 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong lian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-3477 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arviv preprint ar Xrxo:2233..1822, 122), 2023.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024.
13

---

Published as a conference paper at COLM 2025
A Related Work on Attention Sinks and Extreme Activation Outliers in Brasd Models
Previous studies have revealed the widespread presence of extreme activation outliers in Transformer-based models (Bondarenko et al., 2021), including BERT (Devlin et al., 2019; Kovaleva et al., 2019; Clark et al., 2019), Vision Transformer (ViT) (Dosovitskiy et al., 2020; Bondarenko et al., 2023; Sun et al., 2024; Darcet et al., 2023), and LLM (Sun et al., 2024; Guo et al., 2024; Yang et al., 2024a), with substantial attention concentrating on these outliers, forming attention sinks.
Numerous studies have sought to elucidate this behavior in Transformer models. In pre- LLM research, Bondarenko al. (2021), pioneering study, identified the bottleneck in activation quantization of Transformers caused by extreme outliers and uncovered the intrinsic relationship between attention focus pattern and these outliers. Clark et al. (2019) demonstrates that BERT-like Transformers tend to focus on the special [SEP] token. This behavior effectively acts as a "no-op" for attention heads that are unable extract the patterns they were trained to detect from the specific passage of text. Bondarenko et al. (2023) found that outliers and attention focus arise as attention heads attempt to learn "no-op" or partial update of the residual. In this process, strong outliers emerge due to the softmax function.
StreamLLM (Xiao et al., 2023) has conducted an initial investigation revealing the presence of attention sinks in LLMs. suggests that LLMs tend to treat initial token as attention sink because the model tends to dump unnecessary attention values to specific tokens. Sun et al. (2024) conducted an in-depth study on massive activations in LLMs and ViTs, demonstrating that these activations give rise to attention sinks and lead to implicit attention biases. Gu et al. (2024) found that attention sinks function more like Key biases, storing extra attention scores that may be non-informative and not contribute to the Value computation. An et al. (2025) categorize three types of outliers- activation outliers, weight outliers, and attention autliers-revealle attention mechanism.
their intrinsic connections and collective impact on the
Building on these works, we elucidate the role of attention sinks by analyzing the cross-layer evolution of different types of extreme activation outliers, offering novel perspective that has not been explored in prior research. Furthermore, our study delves into their interaction with KV cache quantization, offering valuable insights for future investigations.
B Related Work on KV Cache Quantization
Low-bit quantization reduces the bit-width of the KV cache representation, effectively decreasing its size and thereby mitigating memory usage and access bottlenecks. However, this process inevitably introduces quantization errors, leading to performance degradation. Existing methods explore various approaches to mitigate the impact of quantization errors in KV cache quantization. These methods can generally be classified into two main types based on the selected quantization dimension of the Keys.
KVQuant (Hooper et al., 2025) and KIVI (Liu et al., 2024c) both observe that Keys exhibit outliers in specific channels, while Values do not. Based on this observation, both methods utilize per-channel quantization for Keys to reduce the quantization difficulty. KVQuant employs non-uniform static quantization and incorporates several optimization techniques to address the challenges associated with static quantization. These techniques include Pre-RoPF quantization, per-vector dense-and-sparse quantization, and quantization, among others. KIVI employs dynamic integer quantization, where KV cache quantization is applied after accumulating a specified number of local tokens.
On the other hand, per-token KV cache quantization often uses token-level mixed-precisst quantization to preserve the precision of the KVs of critical tokens, thereby minimizing the loss of essential information. SKVQ (Duanmu et al., 2024), RotateKV (Su et al., 2025a), MiKV (Yang et al., 2024b) and ZipCache (He et al., 2024) are all approaches that focus on per-token mixed-precistt quantization. SKVQ introduces clipped dynamic quantization
14

---

Published as a conference paper at COLM 2025
Altention Score
Channel
Channel
Channel
Token
TOkOT
Nexd labae
Morteber:
Nart Ide
Tokana
Paniforsr Ouarizztion
Por Totoon Ovarlarion
Po-Channd Coorforlor
(a) Sinks
(b) Per-tensor (block)
(c) Per-token
(d) Per-channel
Figure 10: KV quantization based on different dimensionalities. The tokens highlighted in red represent the Keys or Values of sink tokens.
with channel reordering, preserving high precision for both the initial and most recent tokens. RotateKV utilizes outlier-aware Hadamard-transse transform-based rotation to reduce the quantization difficulty of the Keys, while preserving high precision for KV pairs associated with attention sink tokens. MiKV assesses token importance using accumulated attention scores, similar the approach used in H2O (Zhang et al., 2023), and subsequently employs relatively higher bit-widths to preserve the KVs of important tokens. ZipCache utilizes normalized attention scores to more accurately identify salient tokens and incorporates an efficient approximation of the saliency metric.
Although most of these methods preserve higher precision for the KVs of sink tokens or tokens with high attention scores, they lack clear explanation of the underlying principles. In contrast, our work offers a deeper understanding of the interaction between attention sinks and KV quantization, while also enhancing the existing Preserve-First-h solution.
C Overview on Low-Bit Quantization
In this section, introduce the quantization process using the commonly employed asymmetric integer quantization technique. The n-bit asymmetric integer quantization and dequantization processes, where IN, can be expressed as:
(1
X
zero, 0,0, - -1).
Q(X) clamp
(10)
Lccale
X' = scale (Q(X) zero),
(11)
scale clipped.max alippeddmint
(12)
2
zero clipped min(X
(13)
scale
where [-] indicates round operation. Q(X) and X' denote the quantized and dequantized values of X, respectively. The clamp operation ensures that the values are constrained within the specified range. The operations clipped_max(() and clipped-min(X denote the
operations that truncate the maximum and minimum values of X.
Quantization dimension. Quantization can be applied along various dimensions, with com- mon approaches in KV cache quantization including per-tensor, per-token, and per-channel methods, briefly shown in Figure 10. Per-tensor quantization utilizes single set of quantization parameters for the entire tensor (or block). Since LLMs perform autoregressive inference, token KV quantization has become a widely adopted approach (Duanmu et al., 2024; Su et al., 2025b; He et al., 2024). Recent studies (Liu et al., 2024c; Hooper et al., 2025) have identified outliers along the channel dimension in the Keys, with per-channel quantization shown to effectively mitigate quantization errors. Furthermore, quantization granularity can be improved by defining smaller groupings along the quantization dimen- sion, which reduces quantization errors but introduces additional overhead due to the increased number of quantization parameters.
Dynamic and static quantization. Dynamic quantization adjusts the quantization param- eters during inference, offering greater flexibility but potentially leading to less efficient performance compared to static quantization due to the additional computational demands
15

---

Published as a conference paper at COLM 2025
Emergence Stage Layer of Stable Outliers
Outlier Channels Stable Outliers
Model
Total Layers
Hidden Size
LLaMA2-7B
32
40%
2533, 1415
LLaMA2-13B
40
3
5120
4743, 2100
Mistral-7B
32
1
40%6
2070, 3398
LLaMA3-8B
32
1
4096
788, 1384, 4062
32
1
40%6
788, 1384, 4062
LLaMA3.2-1B
16
1
2048
400, 698, 2029, 1159
LLaMA3.2-3B
28
1
3072
588, 1016, 3046, 1731
Table 3: Emergence stage and outlier channels of stable outliers for several models.
of online adjustment. In contrast, static quantization involves estimating the range by pass- ing a few batches of calibration data through the model prior to inference. This approach enhances inference efficiency, as the quantization parameters are pre-calculated and remain fixed during inference. However, it may result in higher quantization errors due to the inability adapt to varying input distributions during runtime.
Impact of attention sinks on KV cache quantization. As shown in Figure 10, the abnormal value characteristics of sink tokens resulting from QKV suppression can significantly impact quantization when sink tokens are included in quantization groups or when quantization parameters calibrated using sink tokens are applied. The impact of sink tokens varies significantly across different quantization schemes, as discussed in Section 4.
D Additional Experimental Results on Cross-Layer Evolution of Extreme Activation Outliers
In this section, we present additional experiments on the cross-layer evolution of differ- ent types of extreme activation outliers across various inputs and models. We conduct experiments using two distinct prompts. These prompts are:
"The following are multiple choice questions (with answers) about machine learning. In |n A 6-sided die is rolled 15 times and the results are: side comes up times;"
prommp 1:
"Summer is warm. )n Winter is cold. (n Spring is mild. n Autumn is crisp. \n The sun rises early in the summer. in The days are short in the winter:)
prompt 2:
As shown in Figures 12, 13, and 17, different inputs do not affect the layers at each stage or the channels where stable outliers emerge. Therefore, they can be used as pre-identified static features during inference. Table 3 shows the emergence stage layers and outlier channels for several models used in KVSink.
We then validate the behavior of each type of outlier at each stage on additional models, including models of different sizes, fine-tuned models, and models using Grouped-Query- Attention (GQA) (Ainslie et al., 2023). As shown in Figures 17, 14, and 16, validation across additional models confirms
findings that stable outliers, driven by outliers in x din and X undergo a key process of emergence, stabilization, and dissipation. Building on this, the behavior varies slightly across models. For instance, in the LLaMA3 series models, the dissipation stage occurs in the final layer, with no distinct final stage. In the Mistral models, the dissipation stage spans multiple layers, whereas for the other models in our experiments, this stage is confined to a single layer. These minor differences do not impact KVSink's ss ability to predict sink tokens during the emergence stage.
E Additional Experimental Results on QKV Suppression and High Cosine Similarity of QK
In this section, we present additional experimental results on QKV suppression and the high cosine similarity of QK. First, as shown in Figures 18, 19, 20, and 21, QKV suppression persists across layers when attention sinks occur for different inputs (as detailed in Section 4). Second, as shown in Figures 18, 20, 22, and 23, QKV suppression persists across different models. A notable observation is the occurrence of unusually large norms in the Queries and Values of sink tokens in the final layer, phenomenon that has not been fully explored
16

---

Published as a conference paper at COLM 2025
CUI 1466
CUILEC ANd 1460
Figure 11: Additional visualizations of the attention scores and the high cosine similarity of QK in LLaMA2-7B.
LLMMA2-BB
LLMA2-138
MaMAA--708
Model
prefill time
KV Cache memory
prefill time
KV Cache memory
prefill time
kv cache memory
w/o
w/o KYSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSink-1
0.04
0.5
0.04
0.78
0.04
0.31
KVSink-5
676.78
0.04
2.5
109070
0.05
3.91
4481.0
0.05
1.56
XVSink-20
0.05
0.05
+ 15.63
+ 0.05
+ 6.25
Mistral-7B
LLaMA3-8B
LLaMA3.2-3B
Model
prefill time
KV cache memory
pretill time
KV cache memory
prefill time
KV cache memory
w/o KVSink
w/o KVSink
w/o KVSink
w/o KVSink
KVSink
KVSink
KVSink
KVSink
KVSink
KVSimk
KVSink
KVSink
KVSink-1
0.04
0.13
0.04
0.13
+004
0.11
KVSink-5
686.32
0.63
0.05
0.05
0.55
XVSink-20
0.05
2.5
+0.05
2.5
+ 0.05
+ 2.19
Table 4: Efficiency analysis of KVSink. KVSink-N indicates the preservation of N tokens. Time reported in milliseconds (ms) and memory in megabytes (MB).
in previous research. We hypothesize that this may suggest inherent differences among various sink tokens, and it could be explored part of future work.
In Figure 11, we present additional visualizations of the attention scores and the high cosine similarity of QK in LLaMA2-7B, confirming the correlation between these two factors.
F Efficiency Analysis of KVSink
This section presents experimental evaluation and analysis of KVSink's efficiency. We conduct experiments on multiple models using x A100 (80GB) setup and evaluate performance the Wikitext-2 dataset, with input sequences segmented into 4K tokens. For time efficiency, we measure the average prefill latency and the time required for KVSink's outlier identification operation under various configurations of sink token quantities. For memory efficiency, we theoretically estimate the original KV cache memory consumption under 2-bit quantization and quantify the additional memory overhead introduced by KVSink's preservation mechanism.
As shown in Table 4, leveraging our enhanced understanding of the attention sinks mecha- nism, KVSink involves only minimal computations, can be efficiently implemented using PyTorch, and exhibits negligible impact on time efficiency. The impact on memory effi- ciency is also minimal. As the context length increases, this impact could diminish further, as the number of preserved tokens remains fixed.
G KVSink Algorithm
The prefill phase with KVsink is illustrated in Algorithm 1. For clarity, the multi-head mechanism is omitted in the algorithm. Note that if static quantization is applied, sink tokens should also be excluded during quantization parameter calibration.
17

---

Published as a conference paper at COLM 2025
Algorithm 1 Prefill Phase with KVSink
1: Parameters:
Number of decoder layers: L, Emergence stage layer: !r, Hidden size: d, Outlier channel: c, Number of token length: n, Number of tokens for preservation: Weights: Wo, Wx, Wy, Wo.
2: Input: Input to decoder O: H E IR"xd
3: Output: Output of decoder L: H E Ruxd
4: Initialize: Suttiers 0, Siin Ø
5: for to L do
6: H' LLyeeroorm mdsaa ( 1H'
7: Q' t H' Wo, Kt H' WK. V +- H' Wy
8:
K tooen & Sink] + quantize( (K' (K' [token & Ssink))
9:
v /token & Siink] & quantize( (V' [token & Ssink))
10:
Kl Kuuanu V V
quant
11:
H' Attention Q', quantr Vi
K
Vquant)
12:
H' H'. Wo + H!-1
H' FFN (LNyyerNNorm// ((t')) H!
13:
ifl IF then
Soutliers + { (i,c) |H/ [op-
15:
16:
Ssink + {i (i,c) E Soutiers
17: end if
18:
end for
19:
return H¹.
LLaMAZ
Initial (Layer 0)
Emgerence (layer 1]
Stabilization (Layer 2-29]
Dissipation (Layer 30)
Final (Layer 31)
(prompt1)
Residaa summotion
If'
Output projectonttion Xeaue
Impat to down projection X
summation MHSA H
Attention output
Attention scores
Imput to decoder g-
Figure 12: Visualizations of the cross-layer evolution of extreme activation outliers in LLMMA2-BB with Prompt 1.
18

---

Published as a conference paper at COLM 2025
Emgerence (Layer 1
Stabilization [Layer 2-29)
Dissipation (Layer 30)
(prompt2)
78
(Layer 0)
(Layer
Readual summation after FFN x
Ourput of down- projecroection Ngaut
praneprojection
Resdual summation after MHSA Hi
Attention output n!
Attention
Figure 13: Visualizations of the cross-layer evolution of extreme activation outliers in LLMM22-7B with Prompt 2.
138 (prompt1)
[Layer 0-2]
Emgerence (Layer 3]
Stabilization (Layer 4-37)
Dissipation (Layer 38)
(Layer 39)
Recidual after FFN H²
3500
Output of down projection
Input projection Xe
Resiuua summation
#
Attention output
Attention
decoder -
Figure 14: Visualizations of the cross-layer evolution of extreme activation outliers in LLMMA2-13B with Prompt 1.
19

---

Published as a conference paper at COLM 2025
Emgerence (Layer 1]
Stabilization [Layer 2-30)
Dissipation (Layer 31)
BB (prompt1)
(Layer 0)
Readual after FFN x
Ourput of down projecrjection Ngasut
praneppajectiin
Resdual summation MHSA Hi
Attention output n!
Attention
Figure 15: Visualizations of the cross-layer evolution of extreme activation outliers in LLMM33-BB with 1rompt
LLaMA3.2
(prompt1)
38
(Layer 0)
Emgerence (Layer 1]
(Layer 2-26]
Dissipation (Layer 27)
Recidual after FFN #
30C
Output of down projection
Input projection Xe
Resiuul summation
#
Attention output
Attention
Figure 16: Visualizations of the cross-layer evolution of extreme activation outliers in LLMMA3.2-BB with Prompt 1.
20

---

Published as a conference paper at COLM 2025
2_78 (prompc))
Tamm, 2. 76 Jpromptl1 domenpro Input
1ama_2_78 dp pproppll
1amm___78 (prompr1)
resitual
MEDA
resizual after
Layor Idde
1amma__76 (prompc2)
laama down ond Input
1ama_ 78 up pprompeii
lamma_ 78 (prompe
Layer
Inow
Tama 138 (oromot.))
[lsma_2_1 2_138 (pmompt))
Tame 138 (promsc)) ShAn peol out
lame 2 138 (prompc))
redduul summation V845
resiuul summerion after
mixtral tprompti
mistral (prompti) down, geoi outbutt
nistral 78 (promptL)
residual simmariin after MisSA
down Input
resld ual Summmattoo after
Layer Srcee
Lapa-inde
Lipertnder
(aidd summi atenn after MHSA
dohn ord nout
doin proj oucout
rocidual Surtem ution afte FFN
Laye-Trde
Capa-Indec
Layeriree Sideri
Ilama 3-1, 3B Instruct (aromot))
Ilama,3 3-1 88, Instruct (crompt))
Ibmm_33 3-1. 8B Instrrcctttteeeee (prompt))
Ibma 3.1 Instruct (orompet)
Heema 3.2, (prompt)) residual Simmattinn after NoSA
Barra 322 38 (prompt)) down ord nout
Tamma 5.2. (grompt))
3.2 38 (prompti)
down Fral ouEout
residual Summi odinn ofter FFN
Layer Indexx
Lager
Lager
Figure 17: Distribution of extreme activation outliers across decoder layers in multiple models.
21

---

Published as a conference paper at COLM 2025
liama 2. 7B query, states layer
liama 7B key. states layer
layer
112-
12
Ilama 7B query states layer
Ilama 2 key states layer
layer
15.-
Tean 900
Ilama. 2.76 78 states layer
Ilama 2 78 key states layer
liama value states layer
115-
115-
100-
3
query states layer
laama_ 2_ states layer_22 22
llama 2 7B value states layer 22
11.4
124-
2010
Barna_2 2_7 query states layer_22 22
llama 2 7B key states layer 29
Ilama
Lama___700 2_7B value states layer
query states layer.
llama 7B key states layer 30
Ilama 2aaaa_2777 2.7B value states layer_ 30
11.5
125-
14
300 1 00
Barma_2 2_7B query states layer 31
llam 2 7. key states layer 31
Ilama_
aama___70 2,7B value states layer
35.
Figure 18: L.2 norm distributions of Queries, Keys, and Values for LLaMA2-7B using Prompt 1, with attention sinks occurring in layers beyond layer and 1, tokens and 14.
22

---

Published as a conference paper at COLM 2025
states head
states head
value head
Tokeen
Taun
14.
24
14.
Channel
Channel
Channel
query layer head
head
value kyee head
Taken
Totnn
14.
24.
14.
Channel
Channel
query states layar head
states boyer head
value byer head
Taken
Takons
14.
244
14.
Channol
Channel
states layer head
states 22 head
states head
Taken
14.
24
14
Channel
Channel
Channel
head
states 29 head
states layer head
Takenn
Toion
144
24
14
Channel
Channel
head
key layer 30
states layer head
Tallen
24.
14.
Channol
Channel
query head
states 31 head
states head
Taken
14
244
14
Channel
Channel
Figure 19: Queries, Keys, and Values for LLaMA2-7B using Prompt with attention sinks occurring in layers beyond layer 0 and 1, tokens and 14.
23

---

Published as a conference paper at COLM 2025
Ilama 76 query States layer
key layer
Ilama states layer
115-
10.0
Ilama query layer
Ilama 2 78 key states layer
layer
liama 7B query states layer
Ilama key layer
Ilama
layer 2
125-
15-
1.5
Ilama. query layer
Ilama_2 7B keeleeetteeeeeeeeeee states layer
layer
112-
15.-
160-
33
Iiama. query layer
Ilama 2, states layer 29
Ilama 2,7B value states layer 29
115.
10.0-
Ilama query layer
Ilama 2, states layer, 30
Ilama 2,7B value states layer 30
15.0-
14.-
13-
Ilama 78 query states layer 31
Ilama 78 key states layer 31
Ilama 2,7B value states layer 31
Figure 20: L.2 norm distributions of Queries, Keys, and Values for LLaMA2-7B using Prompt with attention sinks occurring in layers beyond layer and at tokens 0 and 4.
24

---

Published as a conference paper at COLM 2025
states head
states head
value head
Tokenn
14
24
14.
21
Channel
Channel
Channel
query layer head
layer head
value uyee head
Taken
Totnn
14.
24.
14.
Channel
Channel
Channel
query states layer head
states boyer head
value byer head
Taken
Takons
14.
244
14.
Channel
Channol
Channel
query states head
22 head
states head
Taken
Touon
144
24
14
Channel
Channel
Channel
query head
states 29 head
states layer head
Takeen
Toion
14
24
144
Channel
head
key layer 30 head
states layer head
Tollen
1.4
24.
14.
Channol
Channel
states head
states 31 head
states head
Taken
Tokon
14.
24
14.
Channel
Channel
Figure 21: Queries, Keys, and Values for LLaMA2-7B using Prompt 2, with attention sinks occurring in layers beyond layer 0 and 1, at tokens 0 and 4.
25

---

Published as a conference paper at COLM 2025
mistral 7b query states layer 0
key layer
mistral value_statlee states layer
12
query, layer
mistral 70 key states layer
mistral value, states layer
15.0-
1000
25-
mistral. 7b query states layer 2
mistral 70 key states layer
mistral value states layer
Saaiid 15
PC
mistral 7b query states layer 22
mistral 7b key states layer 22
mistral 7b value states layer 22
mistral, 7b query states layer 29
mistral 7b key states layer 29
mistral 7b value states layer 29
15-
mistral, 7b query states layer 30
mistral 7b states layer
mistral 7b value states
20.0-
11.-
15-
mistral 7b query states layer 31
mistral 7b key states layer 31
mistral 7b value states layer
19.00
12.5
10-01
50.
Figure 22: L2 norm distributions of Queries, Keys, and Values for Mistral-7B using Prompt 1, with attention sinks occurring in layers beyond layer 0 and 1, at tokens 0 and 15.
26

---

Published as a conference paper at COLM 2025
Iama 3.1.88 Instruct query states layer_0
llama key layer
Ilama_3. 8B, value states layer.
250-
200-
alama. query layer
ilama 3.1, 8B Instruct key States layer
BB layer
Baaa 3.1, 88 Instruct query states layer
Ilama_3. 3.1______. 88, insttuctt key. states layer 2
llama
8B value layer
2.-
D Dippo
20-
Ilama 3.1. instruct states layer
Ilama 3.1. instruct key states layer 22
Ilama 3.1, 8B Instruct value states layer
Iaama 3.1, 8B Instruct query states layer 29
3.1 Instruct key states layer
Ilama 3.1. 88 Instruct value states
layer
Ilama_ 3.1. 88 instruct query states layer 30
Baaaa 3.1. 88 Instruct key. states layor 30
Baaa 3.1, 8B Instruct value states layer
280-
ina
150
130
instruct query
layer
Ilama 3.1 8B instruct key states layer 31
Ilam
layer
Figure 23: 1.2 norm distributions of Queries, Keys, and Values for LLaMA3.1 using Prompt 1, with attention sinks occurring in layers beyond layer 0 and 1, at token 0.
27


Reviews:
[{"review_id": "4ZoDVWQFg6", "reviewer": "Reviewer_vwcZ", "summary": "This paper analyze the phenomenon of attention sink and explores its interaction with KV cache quantization. Building on the understanding of stable outliers, the authors propose an efficient method to predict sink tokens with minimal computation. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper study the attention sink phenomenon. This is an underexplored but significant behavior in LLMs. Studying its effect on KV cache quantization is a novel and meaningful direction, especially given the inherent trade-off between range and precision during quantization. The authors highlight how per-channel quantization can lead to large errors due to attention sink, and propose an algorithm that efficiently predicts these sink tokens by leveraging insights from stable outliers. The paper is insightful and well-written. Figures are illustrative.", "reasons_to_reject": "The evaluation is somewhat limited. The paper only compare against the Preserve-First-N baseline. Given the context, it would be more informative to include comparisons with additional baselines that also preserve a fixed number of tokens, which are common in KV cache pruning.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x8RTOW3Nt3", "reviewer": "Reviewer_G7Pw", "summary": "The paper analyzes why preserving only the first few tokens during KV-cache quantization is insufficient by tracing how attention sink tokens propagate extreme activation outliers through decoder layers. it shows that sink tokens can arise anywhere in the context and that their suppressed QKV norms plus high query-key similarity create implicit attention biases that are highly sensitive to quantization. The authors introduce KVSink, a lightweight procedure that detects stable activation outliers in a single early layer and uses their positions to mark sink tokens whose KV pairs must be kept at higher precision. Experiments on seven open-source Llama and Mistral models demonstrate that protecting just five KVSink tokens consistently lowers perplexity relative to the traditional Preserve-First-N strategy and further improves the state-of-the-art KVQuant method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work provides explanation of the causal link between activation outliers, attention sinks, and quantization error, provides subsequent design choices in clear empirical evidence; KVSink is model-agnostic, plug-and-play, and requires only a top-k magnitude scan in one pre-identified channel, so its computational and memory costs are almost zero at long context lengths.", "reasons_to_reject": "All reported benefits are measured in perplexity; The paper does not provide end-to-end decoding throughput or GPU memory results for real-time inference, so practical impact is unquantified. KVSink relies on a manually tabulated emergence layer and outlier channel for each model, which may need re-profiling after fine-tuning or architecture changes.", "questions_to_authors": "How does KVSink behave during streaming or speculative decoding, where sink tokens might appear after the prefill phase and the emergence layer has already passed; can the method be run without noticeable latency?", "extra_scores": {}}, {"review_id": "jbVrBL2mgG", "reviewer": "Reviewer_Uszr", "summary": "This paper introduces KVSink, a novel method for predicting sink tokens to enable their more thorough preservation during quantization. The work highlights the observation that attention sinks are not limited to initial token positions and that their effective preservation is crucial for maintaining model performance post-quantization. The paper presents the detailed analysis of extreme activation outliers and demonstrates the effectiveness of KVSink in preserving performance on downstream tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Efficient LLM inference is a critical area of research. This paper focuses on attention sinks and tackles the important challenge of performance degradation in LLMs due to KV cache quantization.\n2. This paper introduces a practical, plug-and-play method (KVSink) that can effectively predict and preserve these crucial tokens with low overhead.\n3. The authors provide extensive experimental results across various models and datasets to demonstrate the effectiveness of the proposed approach.", "reasons_to_reject": "See Questions to Authors below.", "questions_to_authors": "1. Does KVSink generalize well to a wider variety of LLM models, such as the Qwen series models.\n2. How robust is the KVSink method to varying sequence lengths? Are any limitations observed?\n3. How robust is the KVSink method to different types of LLM architectures, such as encoder-decoder architectures.\n4. Are the evaluation datasets, WikiText-2 and C4 enough? The paper KVQuant includes long context length evaluation.", "extra_scores": {}}, {"review_id": "vF1tlbgowe", "reviewer": "Reviewer_itTf", "summary": "KVSink enhances KV cache quantization by focusing on the preservation of attention sinks, thereby maintaining the model's ability to attend to initial tokens effectively. This approach strikes a balance between reducing memory usage and preserving model performance, making it particularly beneficial for applications requiring long-context inference.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "KVSink preserves the precision of the KV cache associated with attention sinks selectively. It maintains higher precision for the KV pairs corresponding to the initial tokens, and ensures that the model retains the necessary information to focus on these tokens during inference. This selective preservation mitigates the negative impact of quantization on the model's performance, particularly in tasks where early tokens are crucial.​", "reasons_to_reject": "- While KVSink selectively retains full precision for the attention sinks, it still quantizes the majority of the KV cache. If important information lies outside the preserved sinks (e.g., middle tokens in long documents), the model may still suffer from precision loss. This can hurt performance on tasks, where important context is spread throughout the sequence, such as summarization or reasoning across long documents.\n- Reliance on early tokens consistently attracting attention across layers and tasks, which isn't universally true. Some tasks or prompts may shift attention patterns, in which later tokens can become important depending on the question or query.\n- The method is mostly heuristic, e.g., retaining N tokens at the start of the sequence while quantizing the rest.", "questions_to_authors": "- How well does KVSink generalize to tasks beyond text generation, such as text classification, summarization, or question answering?\n- How does it compare with extreme quantization methods like INT4 or mixed-precision approaches?\n- Have you considered dynamic sink selection where the attention sinks could adapt based on the input sequence or the task?\n- Can you explain any limitations when quantizing the KV tensors? Are there any particular patterns or edge cases where KVSink doesn't perform as well?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents KVSink, a novel method for identifying and preserving attention sink tokens to improve the quality of KV cache quantization in LLMs. The work is motivated by the observation that attention sinks — tokens that disproportionately attract attention — are not necessarily among the initial tokens and are particularly vulnerable to precision loss under quantization. KVSink leverages the emergence of stable activation outliers to efficiently identify such tokens early, resulting in minimal overhead and consistent gains in perplexity across multiple models.

Overall evaluation:
- The technical depth and rigor of the analysis are strong. The authors provide a compelling explanation of how attention sinks and outlier activations interact with quantization errors. 
- The paper is clearly written, with well-structured explanations and informative figures. Reviewers appreciated the clarity of the causal link between activation outliers, attention behavior, and quantization error.
- The paper explores a novel and underexamined issue in LLM optimization: the role of attention sinks in quantization artifacts. 
- Efficient inference in LLMs is a highly relevant research area, and KVSink provides a practical contribution with potential downstream impact. 

The paper can be strengthened in the following aspects:
- Evaluation is limited to perplexity; Adding the results on LongBench during the discussion phase to the main paper is helpful. Also, provide end-to-end decoding throughput or GPU memory results for real-time inference.
- Justify the generalization to other model families (e.g., Qwen, encoder-decoder)
- Justify some heuristics (e.g., pre-defined emergence layers and outlier channels).

This is a high-quality and insightful paper that addresses an important efficiency problem in LLM deployment. Despite some missing evaluations and questions about generalizability, its conceptual contribution and empirical effectiveness reach the acceptance bar.
Decision: Accept