Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis
Authors: Qingyue Yang
Jie Wang
Xing Li
Yinqi Bai
Tong Xialiang
Hui-Ling Zhen
Jianye HAO
Mingxuan Yuan
Bin Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
KV Cache Compression
Attention Pattern
Primary area: foundation or frontier models, including LLMs

Abstract: Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce **Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations** from a temporally continuous perspective.
TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension.
Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at [https://github.com/MIRALab-USTC/LLM-TAPPA](https://github.com/MIRALab-USTC/LLM-TAPPA).

Published as a conference paper at ICLR 2026
WHY ATTENTION PATTERNS EXIST: TEMPORAL PERSPECTIVE ANALYSIS
A UNIFYING
Qingyue Yang", Jie Wang't Xing Lif,, Yinqi Bai', Xialiang Tong', Huiling Zhen', Jianye Hao', Mingxuan Yuan", Bin Li' MMOE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China Huuawii Technologies Co., Ltd.
*College of Intelligence and Computing, Tianjin University yangqingyue @mail. u edu. en, 1i. www.g2@huwwww.......... com
ABSTRACT
Attention patterns play crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such retrieval heads, sink heads, and diagonal traces, yet these observations re- main fragmented and lack a unifying explanation. To bridge this gap. we in- troduce Temporal Attention Pattern Predictability Analysis (TAPPA), uni- fying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations from a temporally continuous perspec- tive. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinc- tion be explained by the degree of query self-similarii along the tempo- ral dimension. Focusing on the predictable patterns, we further provide a de- tailed mathematical analysis of three representative cases through the joint ef- feet of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights tasks. Across these tasks, a simple metric motivated by TAPPA consistently im- proves performance over baseline methods. The code available at ht nts: / //github.
KV cache compression and LLM pruning
INTRODUCTION
Attention patterns matter for both LLM training and inference (Jiang et al., 2024; Geng al., 2025; Li et al., 2025; Yang et al., 2025). Prior studies have shown that attention heads exhibit structured and reusable forms, such as streaming heads, retrieval heads, and sink heads (Xiao al., 2023; 2024). Understanding why such patterns emerge critical for deeper conceptual understanding of attention mechanism and can directly inform the design of architectures and inference strategies that improve efficiency and robustness, for example, cache compression and pruning.
A substantial body recent research has empirically analyzed attention behavior. Prior analyses typically focus on single phenomenon, for example, the attention sink at the first token (Gu et al., 2024) or diagonal traces linked to high-frequency components of RoPE (Barbero et al., 2025). Other studies categorize heads by functional roles, such as retrieval and streaming (Xiao et al., 2023; 2024). Despite these advances, remains unclear what factors determine which attention pattern a head will adopt under the same attention formulation. Our goal is uncover unifying underlying mechanism that explains the emergence of these diverse patterns.
This work done when Qingyue Yang intern Huawei.
I Project lead. Corresponding author. Email: jiewangx wwwraangx@ustc.ccu.cmm

---

Published as a conference paper at ICLR 2026
TAPPA Theoretical Discoveries
Guided Downstream Tasks
Query temporal continuity induces predictable attention patterns.
① KV Cache Compression Budget Allocation
High Q-Similarity (Stable)
Predictable Putterns
9r-1
Query Erolliioon
Low Q S-imilaatty - More Dudgets
Qwan2.5 Gain over E4 (Nvidia, 2025)
4:
Cai
111.24
Fixed Historical Key Cache
Low Q Similarity (Randam)
Ungredictable Pattems
512
Budieet
Rc+1
2 LLM Structural Pruning
High Q-Similuriy - Higher Prunod Probubility
Massive channels and RoPE frequency control the diagonal interval.
High Channel Frequency
Low Channel Frequency
90
Small
Large
inserval
0MA0-1519
Figure I: Overview of Temporal Attention Pattern Predictability Analysis (TAPPA) Framework. Left: Theoretical discoveries. Query self-similarity (q-similarity) affects the predictable and unpre- dictable patterns. Within the periodic sequential pattern, the slash interval is affected by the joint effect of queries, keys and RoPE. Right: Q-similarity is applied to downstream tasks and achieves consistent improvements.
To address this gap, we adapt a temporal view of nuto-regressive inference and analyze how attention evolves over time. During inference, transformer LLM generates each token from the previously generated sequence, so the hidden states and attention scores across steps can be regarded as a temporal series. We then isolate the source of temporal variation in attention along the time axis. Conditioned on the past keys being fixed, changes in the attention distribution across steps are de- termined by the evolution of the query vector. this interaction, few embedding channels may dominate the inner product (Sun et al., 2024; Liu et al., 2024), which determines the shape of the at- tention pattern. Figure provides an illustration of how changes in queries and dominant embedding channels reshape the attention pattern.
Guided by the temporal view, we propose Temporal Attention Pattern Predictability Analysis (TAPPA), unified framework that interprets attention patterns through the temporal behavior of queries and the response the RoPE channels. We view the sequence of query vectors and the as- sociated attention distributions as a time series and characterize them using the notion of continuity. We mathematically show that temporal continuity of queries, measured by their self-similarrty, is the key factor distinguishing predictable patterns, characterized by clear regularities, and umpredictable patterns that lack stable regularities. Within the predictable regime, we further provide theoretical conditions for three representative patterns with the joint effect of queries, keys, and RoPE. Re- access patterns, where an attention head repeatedly focuses on a small set of tokens, require high query self-similarity and favorable initial query-key geometry. Sequential patterns, which appear as diagonals, are driven by high self-similarity in both queries and keys. In this case, we provide sufficient condition that does not require attributing the phenomenon exclusively components (Barbero et al., 2025). Seasonal patterns, which periodically repeat focus, arise when input periodicity combines with the periodic nature of dominant embedding channels. Since the computing attention from queries, keys, and RoPE is a common design in transformer-based mod- els, TAPPA both unifies diverse attention patterns and broadly applicable across LLMs.
To validate TAPPA, we evaluate it on downstream tasks. Prior works have shown that attention patterns are closely linked model's representational capacity (Li et al., 2025; Xiao et al., 2024) and can guide compression. Stable temporal behavior indicates redundant predictable attention allocation, which can be exploited for selective retention or pruning. Building on this view, we focus on two complementary compression settings: KV cache compression for stored states and LLM pruning for model weights. In both cases, a simple metric q q-similarity consistently outperforms baselines, demonstrating that these principles practically useful.
In summary, our contributions are as follows: (1) We introduce TAPPA, which provides the first systematic analysis of the shapes of attention patterns from unifying temporal perspective, analyz-
2

---

Published as a conference paper at ICLR 2026
Re-access
Legend: Formaaion Conditions
Predictable
Hligh Q Similarity
Low Sumilanty
Damirati 6,
High Similarit
Soquential
Lange Dominate 6,
Periodical Q.K
Sparse Attention
Simiarrty
(g++
Unpredictable
Periedical Sequential
Seasonal
5
Figure 2: TAPPA explains the formation of sparse attention patterns from a temporal continuity perspective. We first establish the fundamental Predictable and Unpredictable patterns in Sec. 4. We then detail the conditions that form the Re-access (Sec. 5.1), Sequential (Sec. 5.2). Seasonal (Sec. 5.4), and Periodical Sequential (Sec. 5.3) patterns in their dedicated sections.
ing unpredictable patterns alongside three predictable types: re-access, sequential, and seasonal. (2) Theoretically. we demonstrate that stable patterns emerge from the continuity of queries and keys combined with the RoPE mechanism. (3) We identify periodic sequential diagonals and explain them as consequence of the RoPE rotation period of the dominant channel. (4) We apply insights derived from TAPPA to downstream tasks, including KV cache compression and LLM pruning, achieving accuracy improvements.
RELATED WORK
2.1 ATTENTION PATTERNS
The sparse nature of attention mechanisms in Large Language Models (LLMs) is well-documented, giving rise to distinct, recurring patterns, Prior work has largely focused on identifying these patterns and using them for inference optimization. For instance, one widely discussed pattern is the attention sink, where high attention scores are consistently assigned to the initial tokens (Xiao et al., 2023), attracting significant research interest and analysis from various perspectives (Gu et al., 2024: Yu et al., 2024; Cancedda, 2024). Xiao et al. (2023) also highlighted the importance of attention to recent tokens, which form distinct diagonal trace in the attention map. The structured nature of these patterns has been widely exploited for KV cache compression and inference optimization by various methods, such as Minference (Jiang al., 2024), H2O (Zhang al., 2024), SnapKV (Li et al., 2024), AuoAttention (Xiao et al., 2024), and KVTuner (Li et al., 2025). Alongside these structured patterns, other works have identified retrieval heads (Wu et al., 2024: Xiao al., 2024). These heads appear to scan the entire context for semantically relevant information, resulting in seemingly random attention maps that are crucial for long-context reasoning and factuality (Xiao et al., 2024). However, these observations have remained largely fragmented, lacking unifying theory to explain the co-cxistence and emergence of these diverse patterns.
2.2 THE ROLE OF POSITIONAL ENCODING
growing body of work has sought a mechanistic explanation for these patterns by examining the role of Rotary Positional Embeddings (RoPE) (Su et al., 2024). Research has shown a direct link between RoPE's frequency components and specific pattern shapes. For instance, high-fi gigh-freuueccy components in RoPE have been demonstrated to be responsible for the formation of diagonal or previous-token patterns (Barbero et al., 2025). Conversely, other studies suggest that low cow-frequency components, or specific outlier channels with large magnitudes, may contribute to the emergence of attention sinks by creating a rotational offset that favors certain positions (Jonasson, 2025). While these studies provide crucial insights into how positional encoding shapes attention, they often ana- lyze RoPE's effects in isolation, without fully modeling its interaction with the dynamic content of the query and key vectors.

---

Published as a conference paper at ICLR 2026
2.3 THE INFLUENCE OF INPUT DYNAMICS
A parallel line of research investigates how the properties of the input tokens themselves influence attention patterns. AttentionPredictor (Yang et al., 2025) proposed that the temporal continuity of queries is a key driver for pattern formation, though it did not provide a deep mathematical analysis or consider the interplay with RoPE. Other works have corroborated the importance of input features, suggesting that attention sinks may arise from specific query-key angular relationships that are independent of position (Gu et al., 2024). Similarly, the continuity of queries between layers and constant massive channels of keys has also been noted (Lee et al., 2024; Liu et al., 2024), hinting at the inherent temporal consistency within the model. However, this line of inquiry has yet to be formally connected with the rotational effects of RoPE to provide complete picture.
In this work, we bridge the gap between these latter two perspectives. We propose TAPPA, a unifying theoretical framework that explains how input dynamics and positional encoding together influence attention patterns. Specifically, we demonstrate that variations in query self-similarity over time, when coupled with the rotational mechanics of RoPE, mathematically account for the diverse patterns observed in prior works.
BACKGROUND
Attention Mechanism. At the decoding step t, let the query be q: E IRd, the key matrix K E Igd with ky E IRd, and the unnormalized logits
Cr a=j=- Re
E IRT,
where Re-j is the Rotary Positional Embedding (RoPE) operator that rotates vector kg by relative phase proportional ())
The attention distribution is then
A, softmax(a.).
Since the softmax function monotonic with respect to the logits, preserves their relative order across positions. Therefore, for clarity, our discussion focuses on the logits a, and the resulting conclusions directly extend to the final attention distribution At.
RoPF encodes relative position information by applying channel-wise 2D rotations to pairs of embedding dimensions. For feature pair (m, m + d/ /2) at position m, the rotation cos((n - sin(n(m) sin(n6..) cos(nfm.)
R
where 6m = 2mm/d is the frequency of the m-th channel, d is the hidden dimension, and c is hyperparameter. While the original RoPE paper (Su et al., 2024) proposed pairing adjacent dimen- sions, this half-split pairing scheme is adopted by large-scale models like Llama and Qwen2 for greater computational efficiency.
Thus, for query and key ki the RoPE- attention score channel m is (m)
= 4r (m)T
Re-i, (m)
(4)
where qt (Tm) (4t,2m 4r,2m+1)
Decomposition View of Attention. Using the RoPE formulation, the attention logits dg,s be-
tween 9e and k; can be decomposed wise. Let ge = + M D (m) and k; = M (m)
E Ig² corresponds frequency channel with angular frequency where each pair (m) (m)
en = c 2m/d Then
Σ
11 ((m) (1)) || [|((m)] cos &i.4 + (i t)6mm)
(5)
where (m) denotes the angle between &f frequency channel contributes additively to the overall attention score, and how temporal shifts (i t) are modulated by channel-dependent phases On.
(m) and k. km())) This decomposition highlights how each
4

---

Published as a conference paper at ICLR 2026
RoPE Key Property. RoPE satisfies gelative-posittonn identity:
R Ry Rm-n+
(6)
which ensures that attention depends only the relative distance (t i), not absolute positions.
Attention Patterns. It is well-established that the attention mechanism is sparse and shows var- ious patterns. In this work, we focus on these sparse attention patterns, especially Llama-3. 1- 8B (Dubey et al., 2024) and Owen-2.5-7B (Yang et al., 2024a) with GSM8K (Cobbe et al., 2021) and AIGC (SoftAge-Al, 2024) datasets.
4 WHY PREDICTABLE AND UNPREDICTABLE ATTENTION PATTERNS EXIST
Previous works mainly analyze and utilize attention patterns, including retrieval/streaming heads or haaapp//wertcaalsssssblo--------
patterns, from the functionality geometric morphology view. In contrast, TAPPA provides a new and unifying time-series analysis perspective to theoretically understand the existence of diverse attention patterns (Figure 2), utilizing the underlying attention mechanisms. Under TAPPA, attention patterns fall into two temporal categories: predictable and unpredictable. Predictable patterns exhibit temporal continuity across decoding steps or along the temporal dimension, where the indices of high attention scores evolve smoothly over time. Unpre- dictable patterns display irregular jumps with little temporal consistency. This distinction matters because temporal stability enables inference optimization: stable patterns can be anticipated and efficiently compressed the KV cache, while unpredictable ones resist such treatment.
Empirically, retrieval attention heads exemplify the unpredictable case. Their attention often jumps across the entire context seemingly random fashion (Wu et al., 2024; Xiao et al., 2024; Li al., 2025), which is crucial for retrieving semantically relevant information but undermines pre- dictability. Predictable patterns, by contrast, correspond to heads that consistently attend to locally structured repeatedly accessed tokens, reflecting stable model behaviors that exploitable for compression and acceleration.
In TAPPA, the key differentiator behind these two regimes is query self-similarity, namely the similarity between queries at different time indices. When successive queries remain close in repre- sentation space, attention indices change smoothly and produce predictable attention maps. When queries drift strongly, the inequalities that define structured patterns are violated. Even with RoPE relative rotations, attention can jump unpredictably. To capture this distinction, we introduce quan- titative measure of query continuity, termed q-similariyy
In Appendix F.3, we further study the distribution of q-similarity across layers, heads, models, and datasets, and show that high-con high-continuity heads are common but not universal. High q-similarity correlates with stable, predictable heads, while low q-similarity leads to retrieval-like, unpredictable behavior. Figure 3 shows the attention patterns of the two models with high and low q-similarity scores. can seen that patterns with high q-similarity are more stable, while patterns with low q-similarity are more random.
Proposition 4.1. Let qt, 94+1 &R.. be consecutive queries, K [ki,...,, KT) the key matrix, and define the logits
Ot. = Ri
Oc+1,3 = 4241 Ret1-ski--
If 4t+1 qs has large norm and is not orthogonal to all rotated keys (Ret1-sksh, b then the differ- ence between the logit vectors a, and 4+41 is necessarily large. particular, there exist constants C1,02 C2 > 0 such that
||a,41 - -alloo 2 cillga c1l1g-11 gell c2.
Proposition 4.1 demonstrates that while low g-similarity leads to more random patterns, high q- similarity is necessary condition for predictable ones. In summary, q-similarity provides quan- titative indicator of whether an attention head behaves in predictable or unpredictable manner. In the following sections, the theoretical analysis focuses the predictable heads.
5

---

Published as a conference paper at ICLR 2026
Lama-3-1,, High Quammiar by 0.95
U.ma-3.1, L, Low Quimillrity 0.20
Owend.5. High Quaimileriy 0.99
Owenn-2.5 2,5, Low Q-similarity 0.38
Cuery Todan trdbrr
Quary Thlen Index
OUSS Taken Indox
Kay Iolmm tr
Day Token IndM
Kay Toomn Irdes
Kay liten Trides
Figure 3: Attention patterns at high and low Query similarity on the Llama and Qwen models. Stable patterns emerge under high similarity, whereas low similarity results in random patterns. There are random bright dots of critical keys in the second and fourth figures.
5 PREDICTABLE ATTENTION PATTERNS
In this section, we provide a temporal perspective analysis on predictable attention patterns, which rely on the temporal continuity of queries. The re-access pattern occurs when queries are highly self-similar, with low-frequency RoPE components helping to maintain alignment with fixed keys. We also discuss how this analysis relates to the conditions described in prior work (Gu al., 2024). Sequential patterns arise from the combination of high query and key similarity and the relative position property of RoPE. some cases, periodic sequential patterns appear. We provide clear calculation for the spacing between adjacent periods and verify it experimentally by varying the location of the dominant RoPE channel and the RoPE base parameter. Finally, we analyze seasonal pattern with periodical queries and keys.
These predictable patterss are useful for LLM inference acceleration. Methods that exploit such temporal regularities, including Minference (Jiang et al., 2024), H2O (Zhang et al., 2024) and SnapKV (Li al., 2024) can compress the KV cache with little loss in L.M performance, which empirically supports the claim that temporal stability is an important signal for effective KV com- pression (Jiang et al., 2024; Zhang et al., 2024; Li al., 2024).
5.1 RE-ACCESS PATTERN
The re-access pattern describes repeated attention small set of key tokens, appearing as vertical lines in the attention map and often referred to as attention sink (Xiao et al., 2023). Prior work has attributed this phenomenon query continuity (Yang et al., 2025) to the small angle between the first key and all queries (Gu et al., 2024), while others observed its correlation with lowfrrequeccy RoPE rotations (Jonasson, 2025). However, these explanations are partial, and TAPPA provides unified account of why they align under the same mechanism.
We propose that the stability of re-access pattern relies on two factors: (1) high self-similarity of consecutive queries, which prevents attention scores from drifting, and (2) the low-frequency com- ponents ROPE, which preserve alignment between queries and fixed keys even as time increases.
Theorem 5.1 (Vertical Stability of Attention). Suppose the channel-wise decomposition (Back- ground, Eq. 5) holds for the attention logits ar.i. Assume that the queries evolve continuously in the sense that ||q++1 - qell S E, while all keys k; remain fixed between steps t and t Further assume the existence of a dominant low-frequency channel m² whose weight Wm- dominates the other channels, and whose RoPE frequency Ome small, Then the per-key differences a,41,; - dt, uniformly small, and the attention logits are vertically stable.
When queries vary little over time or decoding steps, the only source of temporal change in equa- tion 5 is the RoPE-induced phase (i ()- a dominant channel with small 6m controls the sum, then shifting + t + 1 changes the cosine term only marginally, hence 01+1, A ar This yields vertically aligned attention weights. The empirical validation of the dominant-channel assumption for the re-access head is in Appendix F.1.
Connection to Attention Sink in the First Token. A well-known empirical phenomenon is the attention sink, which typically appears at the first token position. Prior work Gu et al. (2024) ob- served that queries and keys at the initial position tend to have a very small angle, and attributed this alignment the cause of the sink. TAPPA analysis provides complementary explanation: from

---

Published as a conference paper at ICLR 2026
Llama-3.1
Qwen-
Query May milarity
Q Self-Similariyy 0.74
6. KimiSirllrtyty 0.91
Query Token Index
Index
Indek
Figure 4: High self-similarity in Query (Q) and Key (K) matrices results in sequential attention patterns. An example from a Qwen-2.5 head (left) with high Q and K self-similarity (0.99 and 0.96) produces strong diagonal pattern in the attention map (far right). This phenomenon is also observed in Llama-3.1 (center right).
the decomposition in Equation 5, when the angle of!T) between q: (()) and k!!T) small, the cosine term cos( (m) (i - m) is close to Consequently, the logit contribution from that channel
approaches its maximum possible value ||g (m) [|]| [|]ls""]|1 making the overall attention score at,i large.
This alignment effect explains why high attention scores often emerge at positions where g and k are nearly aligned, particularly at the first token.
5.2 SEQUENTIAL PATTERN
Sequential patterns exhibit a shifting focus across tokens, typically progressing step by step along the sequence. The diagonal slash often observed near the main diagonal is commonly attributed to positional heads, which attend to tokens at fixed relative offsets. We argue that the sequential pattern arises from the combined effect of both high q-similarity and k-similarity and the relative-position property RoPE.
Theorem 5.2 (Sequential Patterns under High Self-similarity). Under the RoPE relative-posttion encoding, suppose queries and keys both exhibit high self-similarity, in the sense that
|1 gell Se,
||kit1 kill Se
for sufficiently small E > 0. Then the attention logits satisfy
|a1+1.i+1 - ar,il S Ce.,
for some constant C > Consequently, the attention logits exhibit approximate shift-invariance along the (+1,+1) diagonal, giving rise to sequential patterns in the attention map.
RoPE encodes relative positions through rotations. When queries and keys vary little across steps, this rotation structure preserves their interactions under a simultaneous shift, As a result, attention scores propagate along the (+1, +1) diagonal, producing sequential like) patterns.
High self-similarity in both query and key representations is sufficient con- dition for the emergence of Sequential patterns. Figure illustrates the patterns of heads with high query similarity and high key similarity, all of which clearly exhibit diagonal structures. Empirical results for separating the roles input dynamics and RoPE are in Appendix F.2.
Empirical Results.
5.3 PERIODICITY OF SEQUENTIAL PATTERNS
Empirically, we sometimes observe multiple parallel diagonal lines in attention maps, with a roughly constant spacing between adjacent lines (periodic sequential pattern). We attribute this periodicity to the rotation angle the dominant RoPE channel.
Theorem 5.3 (Periodic Sequential Pattern from a Dominant RoPE Channel), If a sequential pattern arises and the corresponding key exhibits massive channel at index m²", then the spacing between adjucent diagonals is determined by the rotation frequency of that channel:
2T mm
T
2t c 2mn"
7

---

Published as a conference paper at ICLR 2026
Gor O morsio Polls
Mastiee Dhanrel 124
Va Osannd
Maxdiv Cha tral
RCFC Case 100
CMPY Tobo
Dre
(a)
(b)
(c)
(d)
(e)
Figure 5: An illustration of how RoPE configuration affects attention patterns. (a) and (b) show a sequential pattern with a dominant channel at m 124. In (c) and (d), manually change the dominant channel to higher frequencies (m 2 and m = 5), which causes periodic diagonals to emerge. In (e), we change the RoPE base from c 000, 000
100, 000 with m 5.
Intuition.
When the massive channel is m, the attention score is dominated by that component:
at.j = |19; (1 (mm))
[14) cos "1 + ( t)0m)) m)
This term a cosine function of the relative offset (j ()-t) with angular frequency m.. Consequently, the diagonal lines in the attention map exhibit regular repetition with period T 2 2 ar/0mm, as given in equation 7. Since O.. 2m2m" /d higher channel indices m correspond to lower angular frequencies and therefore to greater spacing between adjacent diagonals.
We validate the theoretical mechanism with controlled manipulations on learned key vectors. We consider two axes of intervention: (i) relocating the massive channel across different indices, and (ii) varying the RoPE base hyperparameter
Relocating the massive channel. We first analyze a key vector kj whose attention map exhibits a single diagonal, shown in Figure 5 (b). We identify its massive channel index m 124 as shown in Figure (a). Given the Qwen2.5 RoPE hyperparameters (base c = 1, 000, 000, dimension d 128), this high-index channel corresponds to extremely low angular frequency. Its theoretical period is T 2m /d se 2.4 x 10 value so large that no repetition can be observed within practical context window.
To demonstrate the relationship between channel frequency and periodicity, we experimentally re- locate this massive channel to different target indices m, recomputing the GoPE-augmenter atten- tion for each case. The resulting attention maps with = and 3, visualized in Figure (c) and (d), show that periodic diagonals emerge as the massive channel is moved to lower-index, figgerrfreuuecy positions. Specifically, the channel index decreases, the angular frequency Om increases, shortening the period 7' and making the diagonals denser. This confirms the first find- ing of TAPPA: observable periodic diagonals require the key's massive channel to reside in high-frequency (low-index) position.
Furthermore, we observe that even for sigh-freuenccy channels, the diagonal patterns fade over long distances. This occurs because the self-similarity between queries and keys naturally diminishes as their relative distance increases, which disrupts the continuity required sustain the pattern.
Varying the RoPE base c. Independent of the channel index, the choice of RoPE base also controls the periodicity. To isolate this effect, we keep the same dominant channel m = 5 and repeat the above procedure for different values of the base (e.g., c = 1, 000, 000 and 100,000 in Figure 5 (d) and (c)). Since the channel frequency is given by 0m = c 2m/d decreasing directly increases 6n and hence reduces the diagonal period 2n m0m.
5.4 SEASONAL PATTERN
Seasonal patterns arise when attention maps repeat with fixed periodicity. This periodicity can manifest along either the temporal axis or the spatial axis. Due to the periodicity of the hidden states, the periodicity of queries and keys is often aligned, temporal and spatial repetitions typically occur simultancously and share the same period. We argue that the underlying cause of the seasonal pattern is that queries and keys exhibit periodicity, which is preserved and sometimes amplified by
8

---

Published as a conference paper at ICLR 2026
Table 1: The evaluation results on the LongBench dataset across different KV cache budgets.
Single DagumenDaeeei
MutisTeeumeetQAA
Summony
Fen-shed Leaming
Synthelle
Cofe
Budget
Methed
TRIE
Avcraget
Limme-311-40
31.05
53.78
55.04
27.49
43.61
55.65
Streamingl
25.44
27.
13.30
67:56
411.5
13.50
23.15
30.30
22.85
57.50
K7.60
42.06
97:00
60.51
51.38
41.75
I20
27:26
29.00
46:15
52.33 275
44:11
24:31
25.11
23.56
54.50
91.35 13355
42.11
6.36
99.40
62.30
54.35
44.39
512
PynnddVV
StopKy
30.76
42.03
52:13
46.14
30.51
24.98
24.
24.05
64.00
92.05
03.00.
59.50
62.63
54.90
86-92
11:02
42.99
26.37
63.50
42.52
99.50
TAPPA
29:47
03.06
51.63
54.53
$6.54
30.11
25:45
57
34.71
02.50
92.35
43:42
0:35
99.50
68.56
$7.35
47:31
SteraminaL.M
25.66
29.57
36.15
23.31
55.74
1824
DyomidkV
Snapk
38.55
22.54
63.64
65.34
12.38
12.75
55.29
46.53
46.19
50.57
$1.38
21.53
21.87
26.00
57
23.00
26.00
68.30
08:00
92.05
92.99
42.50
41.75
6.15
99.50
99.50
63.00
62:15
16.30
55.44
42.95
47.60
CAKE
22:33
44.98
12.18
$5.49
34.91
26
42.50
6.10
55.80
48.13
StraamingE
77:40
16.91
17.45
69 71
18.56
26.11
2551
21.67
22.17
67.50
90
62.49
6.12
$7.00
63/66
15:15
44.74
IE30
2461
18.54
46.90
23ST
23.06
22:31
04.90
43.06
6.11
16.91
Snapy
45.00
27.50
56.36
48.30
31.13
50.25
34.82
42.62
CAKE
30.79
4533
53.57
55.30
46.00
50.47
31,83
34.07
27.16
7055
91.48
43.48
6.00
99.50
63.23
56.64
48.43
TAPPA
30.70
45.00
$3.06
35.40
46.88
30.94
30.54
24:06
27.12
71,00
91.65
45.00
6.00
$9.50
64.93
58.90
48.73
Qwen2.5-78
Pull
Pull
20155
43.54
32.93
57.99
47.06
50.24
31.78
25.64
25:06
71.50
19.67
65.61
8.50
100.00
59.11
57.12
45.87
H20
34.17
21:35
22.00
512
PynandKY
Stap&V
23.54
27:33
43.70
38.14
50.40
50.15
55.90
55.73
44.21
14.28
27.53
22.12
24.42
22.24
21 16.6
22.74
21.00
10.54
66.50
65.00
56.56
56.36
44.14
43.60
3.00
9,00
59.00
50.50
59.87
57.59
$4.22
$1.00
45.51
45:34
CAKE
54.25
45.36
Strvaning
22.72
20.82
31.47
43.57
18.18
17.90
24.35
19.47
22.46
61.00
87.55
43.79
8.90
34.00
35.27
56.45
37.35
H20
36-55
31.44
40:40
08
43.00
22.30
71.90
41
55.00
87.83
15.00
8.90
56.18
59.77
08.88
4114
1924
PynaiiiiV
20.31
15.94
55.77
45.12
37.52
03.50
56.85
63.76
8.50
199.10
45.85
CAKE
70:47
47:71
12.12
16.11
6641
79.15
26.86
23.12
67.58
30.75
65.46
0.50
109.00
19:11
54.79
47:20
2964
43.99
11.91
WI
15.86
29.81
76.64
23.00
01.50
19.08
65.26
3.00
195.30
68.04
66.86
47.72
Straningg
23.18
36.33
45.64
45.30
40.10
19.74
31.57
23.50
76.41
33.06
15.00
53.73
37.07
H20
25,35
40.20
41.45
$3.49
46.46
27.00
26:93
22.96
13.79
63.50
58.50
46.16
8,00
195.09
61.06
51.50
46.95
2848
PyrmidKV
Snap&y
25.39
20.11
43.36
41.53
52:05
51.83
$7.17
75
46.26
30.9
29.49
23.23
23.84
71.59
$0.17
45.69
45.01
8.00
109.00
6002
57.96
51.31
48.12
48.17
DAPPA
29.18
44.0
51.34
$7.36
4537
30.19
25384
23:31
23.62
69.00
89.47
44.99
8,00
198.09
6595
68.06
48.46
RoPE through its relative-position encoding. Although the query condition does not exhibit temporal continuity, the pattern remains predictable over time and is therefore a predictable pattem..
Theorem 5.4 (Seasonal Attention Pattern from Periodic Keys and Dominant RoPE Channel), Sup- pose the query and key vectors are approximately periodic with interval L, in the sense that
|| L -qell S Eq.
||i4+L - k S Ek
for sufficiently small Ex > O, and that this interval is in near resonance with the dominant RoPE frequency, ie.., I. 2km I 8 for some positive integer i and sufficiently small 8 > O. Then
the attention logits satisfy
l - an,il S C2(Eg + Ek) + C28, |au,+L - at,ll S CzlEg + +E) C2
for some constants C1, C2, C3, C4 O, and therefore exhibit a seasonal pattern with period L along both query and key dimensions.
The seasonal pattern arises from two combined effects. First, the approximate periodicity of the input queries and keys induces corresponding periodicity in the attention map. This type of peri- odicity is common in structured data, such as looking at corresponding elements in consecutive lines of code data records. Second, when the interval L. is in resonance with the dominant RoPE fre- quency, the relative-position rotations align with the input periodicity, reinforcing the repetition and producing a stronger, more regular pattern. This dual condition pperiddic keys amplified by RoPE resonance- resonance-explains the emergence of clean, regularly spaced attention pattern. The observed inter- val L is therefore determined primarily by the period of the input data itself.
6 DOWNSTREAM TASKS
6.1 KV CACHE COMPRESSION
To demonstrate the practical value of TAPPA, we apply q-similarity, a metric derived from TAPPA, to the KV cache compression task, which aims to reduce the memory footprint of key-value caches during large language model inference while maintaining model accuracy. Based TAPPA, lower

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of TAPPA with ShortGPT under the same pruning ratios.
Model
Method
Pruned
Piqu
Hellaswag
Winegrande
Are Easy
Averuge (%)
ShonGPP
31%
63.33
45.94
61 .40
47.26
54.48
with TAPPA
31%
63.87
50.83
63,54
45.03
$5.82
Llama-2-7B
ShonGGP
34%
60.83
42.11
60.38
44.15
51.87
with TAPPA
34%
60.45
48.53
62.43
42.55
53.49
ShortGPT
28%
66.65
42.41
58.72
46.25
53,51
Llama-3.1-8B
with TAPPA
28%
64.69
55.09
63.77
52,90
59.11
ShonGGP
31%
64.96
37.69
58.41
42.76
50.96
with TAPPA
31%
65.51
42.22
62.51
46,59
54,21
ShonGGP
39%
63.17
41.83
50.59
44.32
49. 98
Qwen-2.5-7B
with TAPPA
39%
62.89
41.80
51.93
45.03
50.42
ShortGPT
43%
60.83
36.13
47.43
39.77
46.04
with TAPPA
43%
60.88
39.87
49.72
43.94
48.60
query similarity indicates higher likelihood of retrieval patterns. Since retrieval patterns attend to scattered and unpredictable key positions, they generally require a larger cache budget to pre- serve critical information (Xiao et al., 2024; Li et al., 2025). Therefore, we leverage q-similarity as a proxy signal to dynamically guide the per-layer cache budget allocation under limited mem- ory resources, improving inference efficiency while maintaining model accuracy. We provide the experiment details in Appendix G.I, and additional studies on the sensitivity to a and alternative similarity formulations for q-imiilarity are reported in Appendix H.
Results. As shown in Table 1, our method consistently outperforms CAKE and the other four base- lines across three different budget settings. These results confirm that q-similarity derived from TAPPA effectively reflects the likelihood of retrieval patterns, and by allocating more cache budget to layers exhibiting lower query similarity, we are able to preserve critical information more effec- tively, enabling efficient KV cache compression. More results comparing with DuoAttention (Xiao al., 2024) and Expected Attention (Devoto al., 2025) are in Appendix and J.
6.2 LLM PRUNING
To reduce the parameter size of LLMs and accelerate inference, structured pruning, which removes entire components such as layers, has emerged as a promising approach. Our specific goal is to design more effective proxy metrics to guide whole-layer pruming, under the same compression ratio. Based on TAPPA, higher q-similarity indicates more stable and predictable patterns. Such stability suggests that the layer extracts less novel information, making more dispensable. Consequently, layers with higher query similarity can be pruned with less impact on model performance, while low similarity layers, which are more likely to host retrieval-oriented and task-critical behaviors, are preserved. We provide the experiment details in Appendix G.2, and the sensitivity study of is reported in Appendix H.
achieve higher accuracy
Results. As shown in Table 2, our method consistently outperforms ShortGPT across different prun- ing ratios and models, validating the effectiveness of combining Block Influence with q g-iiiilarity as proxy signal for structured layer pruning. These results on LLM pruning validate our hypothesis regarding the connection between q-similarity and stable, predictable patterns. Layers with higher q- similarity exhibit greater redundancy due to their stability, and can therefore be pruned with minimal impact on overall model performance. Results about more baselines are in Appendix G.2.1.
7 CONCLUSION
In this work, we introduced TAPPA, a unifying framework to systematically analyze the diverse attention patterns within large language models. We demonstrated that the distinction between pre- dictable and unpredictable patterns can be explained by the temporal self-similarity of queries. Our theoretical analysis further elucidated that stable, predictable patterns arise from the combined ef- fects of query-key continuity and Rotary Positional Embeddings (RoPE), providing a clear explana- tion for phenomena like periodic sequential diagonals. The practical value of TAPPA is confirmed by applying its insights to downstream tasks. A simple metric inspired by TAPPA successfully im- proved performance in both KV cache compression and LLM pruning, validating our framework.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by National Key R&D Program of China under contract 2022ZD01 19801, National Nature Science Foundations of China grants U23A20388, 62021001, U19B2026, and U19B2044. This work was supported in part by Huawei as well. We would like to thank all the anonymous reviewers for their insightful comments. This research was also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
ETHICS STATEMENT
This research does not involve any personally identifiable information. All datasets used are publicly available and widely adopted in the community, and we have verified that their licenses permit research use. In accordance with the ICLR Code of Ethics, we ensure that our work adheres to principles fairness, transparency, and responsible Al research. We also disclose that LLMs were used for text polishing, while all conceptual contributions and validation remain the responsibility of the authors in Appendix L.
REPRODUCIBILITY STATEMENT
We will provide open access to all source code, configuration files, and preprocessing scripts, to- gether with detailed instructions to reproduce the main experimental results. All datasets employed are publicly available, and we specify the exact versions and preprocessing steps. Collectively, these resources and specifications enable reliable and faithful reproduction of our results.
REFERENCES
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024. URL https:
/ //openeeview dott//ooor
arXiv:2401.15024,
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about org abs/199/999 1911.
physical commonsense in natural language, 2019. URL 1166 11641.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, 2025.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 4792-4808, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: /aaxivv..... www.wiiww..aaaaaaeeeeaea
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Refichiro Nakano, Christopher Hesse, and John

---

Published as a conference paper at ICLR 2026
Schulman. Training verifiers to solve math word problems. arXiv preprint arXhv:2110.14168,
2021.
Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: Kv cache compression by estimating attention from future queries distribution. arXiv preprint arXiv:2510. 0.63,, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21755 2024.
Zijie Geng, Jie Wang. Ziqi Liu, Feng Ju, Yiming Li, Xing Li, Mingxuan Yuan, Jianye Hao, Defu Lian, Enhong Chen, and Feng Wu. Accurate KV cache eviction via anchor direction projec- tion for efficient LLM inference. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10781. 2024.
Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang. and Jiawei Chen. Rethinking entropy interventions in rlvr: An entropy change perspective. arXiv preprint arXiv:2510.1011 2025.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin- Yew Lin, al. Minference 1.0: Accelerating pre-filling for long-context Iims via dynamic sparse attention. arXiv preprint arXiv:2407.022490 2024.
André Jonasson. Rotary outliers and rotary offset features in large language models. arXiv preprint arXiv:2503.01832. 2025.
Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. (InfiniGinn:: Efficient generative inference of large language models with dynamic (KV} cache management. In 18th USENIX Symposium Operating Systems Design and Implementation (OSDI 24), pp. 155-172, 2024.
Xing Li, Zeyu Xing, Yiming Li, Linping Qu, Hui-Ling Zhen, Wulong Liu, Yiwu Yao, Sinno Jialin Pan, and Mingxuan Yuan. Kvtuner: Sensitivity-awar layer-wise mixed precision kv cache quan- tization for efficient and nearly lossless I1m inference, 2025. URL peetps://arxive 04420.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXiv preprint arXiv:2404.14466 2024.
Xize Liang, Lin Yang, Jie Wang, Yiyang Lu, Runyu Wu, Hanzhu Chen, and Jianye Hao. Boost- ing multi-domain fine-tuning of large language models through evolving interactions between samples. In International Conference on Machine Learning, pp. 37427-37441. PMLR, 2025.
Xize Liang, Lin Yang, Jie Wang, Rui Liu, Yang Lu, Jinliang Zeng, Hanzhu Chen, Dong Li, and Jianye Hao. Boosting multi-domain reasoning of lims via curvature- guided policy optimization. In The Fourteenth International Conference on Learning Representations, 2026.
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu, Kivi: A tuning-free asymmetric 2bit quantization for ky cache. arXiv preprint arXiv:2402.02255 02750, 2024.
Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hual Ling, and Jia-Chen Gu. cetturaationrrestainee sequential model editing. The Thirteenth International Conference on Learning Reprexenta- lions, 2025.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
12

---

Published as a conference paper at ICLR 2026
Xin Men, Mingyu Xu, Qingyu Zhang, Qianhao Yuan, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in large language models are more redundant than you expect. In Findings of the Association for Computational Linguistics: ACL 2025, pp- 20192-20204, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-33
Ziran Qin, Yuchen Cao, Mingbao Lin, Wen Hu, Shixuan Fan, Ke Cheng, Weiyao Lin, and Jianguo Li CAKE: Cascading and adaptive KV cache eviction with layer preferences. In The Thirteenth International Conference on Learning Representations, 2025.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap- Compressive transformers for long-range sequence modelling. arXiv preprint, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; An adver- sarial winograd schema challenge at scale, 2019. URL https::/aaxii..ottt http://aaaxiivooooa//////000 10641.
AoftAge-AI, 2024.
URL httpss//hhngiigffccccccrraaattts sertiiitundddtsssee
Jianlin Su. Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / /aaxivivvvv..////////aaaa/aaaivv wwg/oo////.o.o. //tg/bb/////ee /0077 09288.
Hong Wang, Zhezheng Hao, Jian Luo, Chenxing Wei, Yao Shu, Lei Liu, Qiang Lin, Hande Dong, and Jiawei Chen. Scheduling your Ilm reinforcement learning with reasoning trees. arXiv preprint arXiv:2510.24832. 2025.
Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mecha- nistically explains long-context factuality, 2024. URL https: arxiv. orw./orgggb.... /obbs/2444 15574.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang. Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. xrxiv preprint arXiv:2410.108/1 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang. Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, et al. Attentionprediettr:: Temporal pattern matters for efficient Ilm inference. arXiv preprint arXiv:2502.04077, 2025.
13

---

Published as a conference paper at ICLR 2026
Yifei Yang, Zouying Cao, and Hai Zhao. LaCo: Large language model pruning via layer col- lapse. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6401- 6417, Miami, Florida, USA, November 2024b. Association for Computational Linguisties. doi: 10. http://2..222.2seeeee/e findings-emn 372/.
URL https: / /tt//aaaaatttt otttchlooy..rgg//00
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Un- veiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. In International Conference on Machine Learning, pp. 57659- 57677. PMLR, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Com- putational Linguistics.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2026
A PROOF OF UNPREDICTABLE PATTERN
Proposition 4.1. Let qt, 90+1 E IRd be consecutive queries, K = [k1....,,yy KT] the key matrix, and define the logits
0t. 4: Re-j 0++1, 4it1 Be+1-zki--
q++1 - qe has large norm and is not orthogonal to all rotated keys Re+1-skij then the differ- ence between the logit vectors a, and d2+1 is necessarily large. In particular, there exist constants C1, c such that
||++1 az xe 2 ci cilla-11 qell
Proof. Let Aq: = 90+1 qt. For each position j, the change in the logit is
Daj a1+1, at,j (Aq) [ao)RRun Rit1-5ky (Ret1-) Re-
Denote the first term by Ti. and the second by T2.5.
Step 1: Bounding the RoPE difference term. Since Rm is an orthogonal rotation, its operator
norm is I, by the triangle inequality, || Rt-jjllo S || + R Re-illop S 2. If we assume the keys are bounded such that || S Bx for all j. then
T2, S | Re+1- Re-5llop || 2 Bg.
Step 2: Lower-bounding the query difference term. The first term can be written
Ti.s = || K|
Ret1-sk;)1-
The condition that Aq is not orthogonal to all rotated keys implies that the inner product is not always zero. We formalize this by assuming there exists an index j and a constant a > such that the normalized vectors have a significant projection:
K ||وو
Re+1-j-ky- /lk 2 a.
This condition essentially states that the direction of the query change aligns with at least one rotated key. Under this condition, and assuming a minimum key norm | || 2 Bg,mins we get
|Ti--1
>
a a. (ismmin|AAql Ag]l-
Step 3: Combining both terms. Using the bounds for the two terms at index j the reverse triangle inequality gives
Daj=1 [T15- - [T2 2 B Agll 2||
|x--
Since the infinity norm of vector is the maximum of the absolute values of its components, have
||a++1 - 0=|1.0 max |رyو | 2 a Bm. | 2| Bx.
This establishes the proposition with constants C₁ = aB and C2 = 2||qe] Bx- This completes the proof.
B PROOF OF RE-ACCESS PATTERN
Theorem S.1(Vertical Stability of Attention): Suppose the channel-wise decomposition (Eq. 5) holds for the attention logits 0t,5- Assume that the queries evolve continuously in the sense that ||4411 qell S while all keys ks remain fixed between steps and t 1. Further assume the existence of dominant tow-frequenae channel m² whose weight Wmm dominates the other channels, and whose RoPE frequency 0mn is small. Then the per-key differences a+ 02,5 uniformly small, and the attention logits are vertically stable.
Proof. We derive an explicit uniform bound for the per-key logit difference and show how it depends on the query increment and channel parameters.
15

---

Published as a conference paper at ICLR 2026
Using the channel decomposition from Eq. 5, write for each channel
Wm = 119: (m) [|(, Il,
w = l19
(m) |I [[(m [|,
(m)
and
(m)) Vm := 0z, + (i t)
Vn == d(m)) + (2 - (t + 1))8m.
Define the logit difference
D a,+1,i - 02,5.
Direct subtraction yields the exact identity
M
M Wm
A
Σ
(w/
Wm) COs ti
+
cos V - Cos tim)
(8)
m=l
We bound the two sums on the right-hand side separately. Let
E = ||4+1 - 9:ll-
First sum.
By the triangle inequality and the definition of Wm.
M
M
M Σ
Σ
(w/n wm) cos V
S
Iwm
wml =
|
|| - -11g (mm) ||/
m==1
m=1
m=1
Since the Euclidean norm is I-Lipchitz,
|19
(mm)
le
(mm)
1 (m)
(m)
$ 11aet1 - qell = E.
Hence
(wnn wm) cos v/n S
(m) ||-
m=1
m=1
Second sum.
Use the inequality cos cOs vl I
2/. so
cos vi COS
v/mm
I 1
vml = 10211,1
(m) (m) 0ml
S (10mm)
(m)
(mm) Dz.e + 1m
To control the angular difference, let rm == min (m) Il. |1 (m)) ||} and assume I' > 0, and denote [m)
{(m)_ lg(m)) q. (m) ||- In the 2D RoPE subspace, write (mm) rew, and l:+1 r++1 with
||u || = ||u+1 both vectors onto the circle of radius T' can only decrease their Euclidean distance while preserving the angle, so by elementary planar geometry, we have
and let Ag(()) Pt+1.; - (((m) be the angle between (m) and and] Projecting (m)
(m)
2rm sin
S (m)
2
Therefore
((())
x (m)
Pet1.s (m)(m))
(m) (m)
2 aresin
2rm
2 I'
I'
where we used (m) < l1 - 4t = & in the last inequality.
Therefore
Wm ( coss vo cos wm))
π < wm
+ 10mll)
2 mm
Summing over m yields
M
M
M
Wm cos vh - cos vm)
<
π
Wm
+
Wm
10ml-
(10)
2
mul
m=
16

---

Published as a conference paper at ICLR 2026
Combine bounds.
Inserting equation and equation 10 into equation 8 gives the explicit uniform
bound
M
M
M
S
[m)
+
π
W
Wm Am
(11)
2
T'
m=1
m==1
m=1
Define
M =
M
|
Wm
+
Wm 10ml
Tm
Thus A S 8 for every token index i.
Conclusion and asymptotics. Under the theorem hypotheses the keys are bounded and there ex- ists dominant channel m with Wm- much larger than the remaining {wm Imgem while Im is bounded away from zero and 10m=1 is small. In that regime, the two terms proportional to vanish - 0, and the last term is small because the dominant channel's frequency [0m, is small and the remaining channels carry only small total weight. Consequently å can be made arbitrarily small by taking + O, 10m- O, and by increasing the dominance of Wm- over other channel weights. Therefore the per-key differences A.; U++1,i at are uniformly small, which proves vertical stability.
C PROOF OF SEQUENTIAL PATTERN
Theorem S.2(Sequential Patterns under High Self-similarity); Under the RoPE relative-position encoding. suppose queries and keys both exhibit high self- sililaiiil,r in the sense that
|| - gell s,
||k++1 - kill Se
for sufficiently small e > Then the attention logits satisfy
|0441,++1 - dr,al S Cs,
for some constant C C> Consequently, the attention logits exhibit approximate shift-invariance along the (+1,+1) diagonal, giving rise to sequential patterns in the attention map.
Proof. Recall the attention logit
a,ii := 4s Re-sks:
where Ra is the RoPE rotation for relative offset A. By the RoPE identity we have R(+1))((41)) Ri-;; hence
0t+1,141 Re
Therefore, the difference can be written as
0++1,i+1 0r,1 = (q++1 4:) Re-shist1 + ge Re-i(kit1 - ki).
Taking absolute values and applying the Cauchy-Sehwarz inequality gives
0+41.1+1 0., < || qell - + [Iqell [1 k;]l
= || gell || + ||gel] || - kill.
where the last equality uses that each Rs is orthogonal (rotation), hence || = ||l|| Now impose the high self eeff-iimllarity hypothesis in the rigorous form
||gt+1 qell s, ||
k; S
for some Further assume the query/key vectors are uniformly norm- comn-bounded, i.e. there exist constants Q, K > 0 with || < Q and |A; || S K for all relevant t, i. Then
| azi || ||
s(K Q).
Setting C K Q yields the claimed bound
G an.l Ce.
Thus, under the stated assumptions, the attention logits approximately shift-invariant along the +1) diagonal (with error at most Cε), which produces the sequential diagonal structure in the logit map.
n
17

---

Published as a conference paper at ICLR 2026
D PROOF OF PERIODIC SEQUENTIAL PATTERN
Theorem 5.3 (Periodic Sequential Pattern from Dominant RoPE Channel): If a sequential pattern
arises and the corresponding key exhibits a massive channel at index m then the spacing between adjucent diagonals is determined by the rolation frequency of that channel:
2m
T
2% 2m
(12)
Proof. From the decomposition view of attention, the attention logits can be written as a sum over channels:
Σ
(m)
(m)) cos (ei.T) + (i 1)0m).
By assumption, channel m is massive, meaning its contribution dr dominates all other channels:
1
||k
|1ut (m))
[R( for all m + m
Hence, the logits approximately
'I cos (oes (m") + (i 1)0m.).
0e, Re
Consider positions and i+T. Assuming that the magnitudes ||b] | '1l and angles (ma") across consecutive tokens forming the sequential pattern, the attention pattern repeats whenever
vary slowly
1) (i T 1)0m. (mod 2T),
which yields
2T
T
By the definition of RoPE, 6n c 2mm/d and substituting m m² gives
2T
T
2m 2mm
Om
Therefore, the interval between adjacent diagonals in the attention map is exactly determined by the rotation frequency of the dominant channel, as claimed.
E PROOF OF SEASONAL PATTERN
Theorem 5.4 (Seasonal Attention Pattern from Periodic Keys and Dominant RoPE Channel): Sup- pose the query and key vectors approximately periodic with interval L, in the sense that
||ae+1 - qell S Eg:
||stt - kz S Ek
for sufficiently small Eq. >> O, and that this interval is in near resonance with the dominant RoPE frequency, i.e.,
L 2kT <
for some positive integer k and sufficiently small > 0. Then the attention logits satisfy
|at+L,3 ar,il Ci ex) + C28, |ar,4t an,il S Cs(ey Ex) C.d
for some constants C₁ C2, Cz2 C4 > O, and therefore exhibit a seasonal pattern with period L along both query and key dimensions.
18

---

Published as a conference paper at ICLR 2026
Proof. We again use the channel-wise RoPE decomposition. For each channel m, let Ry(") and R! denote the 2 x rotation matrices induced by RoPE at positions and with angular frequency Om. We define the post-RoPE query and key components as
= R m)) q1
K
R m)) x(m)
By construction, RoPE is an orthogonal transformation, so ll mm)
|| = l1 (m))
and [|A:TT] = -|//())
The logit contributed by channel can be written as a dot product
M
(m) or
(m) [(m)) (9t
(m)
We first bound the variation of the dominant channel m along the query dimension. For arbitrary vectors u, u', v, v', we use the standard dot-product inequality
1u T v ='''' - ||l|| || - w'll + ||d4 10 wo1--
(x)
Applying (*) with a u+L u/ alm²)) and D' =k!m²) gives
la a a || KalTi' ,i((///)) (at R((m))
(m²) (m=")
I [ 11 qt 1mn1)1
(13)
It remains to control (m(mm ((m')
mg6m 111 Using the definition of Te
((m²)
we have
(m*)
= =R 90t R
[((m)) (m²)
(m=)) (m=)
4:
= =R R/m²)) (9++L 4: (m²)) ") + (Ri42 )p
(14)
Taking norms and using orthogonality of n!(m²) yields (m=)
(m
s 119(42 qi (m
(m=e)
'11 + || (R(#1)
gg 11
(15)
The first term is controlled by the assumed L- I-periodicity of the queries:
(om")
de
fm
'11 S Eg.
For the second term, we use the near-resonance condition. By the definition of RoPE, R(m²)
R(m²) " p(=²') where R!!"") is a rotation by angle L in the channel-m* plane. The hypothesis ILOm. 2km| S ō means that 12 is in fact rotation by an angle of magnitude at most d around the identity. For planar rotation by angle 7, we have [[(()) - /|| = sin( < 17/. so
(m))) (B G+L
(mm))] "||
| (m(mm) (R( (mmm)) 1)gg" [m="]] "|
S || (1/f(m²) - /11 1l (m=) s 5 1| (m²) 1|-
(16)
Combining equation 15 and equation 16 gives
1 am || S Eg + 8 119 (m "I1--
Substituting this into equation 13 and recalling || (m '11 |||( yields
1 - 04.5 'I [| "ammm"lle, + (1a("=>)1 "I11(bb (m" 116 =: cf ey Cl.
(mm))
An entirely symmetric argument, exchanging the roles of and and using the L-periodicity of the keys [[
{(m²) < Eg. shows that
(
ch+) c
for some constants dlt), cftt) > 0 depending only on the norms of q: (m²) and k(m²)
Finally, recall that channel m² is assumed to be massive: its contribution |/ (m²) ||| (m²)) dominates the contributions of all other channels. The residual variation coming from non-dominant channels {m # m²} } is therefore uniformly bounded and can be absorbed into the constants CI...., Ca- Renaming the constants and noting that Ey + Ek 2 Ey and Ey + EK 2 Ek, we obtain the bounds stated in Theorem 5.4:
|ac+L, a s Ci + Ex) + Cză, lat,i-L arii S C3 + Ex) + C48.
This shows that the dominant component of the attention logits approximately repeats every L steps along both query and key dimensions, giving rise to seasonal pattern with period I..
19

---

Published as a conference paper at ICLR 2026
F
EMPIRICAL SUPPORT
F.1
EMPIRICAL VALIDATION OF THE DOMINANT-CHANNELL ASSUMPTION OF RE-ACCESS
PATTERN
Pee-channel weight
0.5
dint=5)
0.4
0.3
0.0
10
20
KPPE channel
a
(b)
Figure 6: Empirical validation of the dominant-channel assumption for a re-access head. (a) is an attention heatmap of the re-access pattern. (b) plots the RoPF-channel weights of attention at the sink position (dark vertical stripe), showing that single low-frequency channel m² accounts for most of the total weight.
Theorem 5.1 assumes that the attention logits of re-access heads are dominated by a single low- frequency channel. To directly examine this assumption, we perform simple spectrum analysis on a head whose attention map exhibits a clear re-access pattern in Figure 6(a).
For this head, we focus on the key position corresponding to the re-access stripe, namely the attention sink. We decompose the query and key vectors into M = D/2 RoPE channels, where each channel groups the two feature dimensions that share the same RoPE frequency. For every channel we aggregate its contribution over the decoding steps and then normalize the resulting values so that they sum to This gives a one-dimensional spectrum {pm) J
Figure 6(b) plots the weight of each attention channel. The horizontal axis is the RoPE channel index m (0 < m < M), and the vertical axis the normalized channel weight Pm, i.e., the rel- ative contribution of each channel to the attention logits the sink position. We observe highly concentrated pattern: a single channel m carries about Pm- se 51% of the total mass, while the remaining channels form long tail with much smaller weights. The dominant channel m² lies in the low-frequency half of the RoPE spectrum, consistent with the dominant low-frequency channel assumption used in Theorem 5.1.
Together, these observations provide direct empirical evidence that, for the re-access heads we ana- lyze, the attention logits are indeed governed single low-frequency channel.
F.2 DISENTANGLING QUERY DYNAMICS AND ROPE SEQUENTIAL PATTERN.
To empirically separate the roles of input dynamics and RoPE, we conduct controlled ablation on a single attention head that exhibits a clear sequential pattern. For this head, the average cosine sim- ilarity between consecutive queries is approximately 0.99, and the full model (with RoPE enabled) produces an almost perfectly smooth diagonal attention pattern.
We construct three variants using the same head and the same input sequence (Figure 7):
1. High q-similarity with RoPE (full model). In the original model, both the queries and keys have high temporal self-similarity, and RoPE applied as usual. The resulting at- tention map shows a clean, nearly raaslation-invariann diagonal stripe: high-attention region shifts along the (+1,+1) direction with very little distortion. This behavior is consistent with the analysis of TAPPA, which predicts that when both qe and k; vary smoothly in time, RoPE induces approximate inittinnvaiaane along the main diago- nal.
increases, the
20

---

Published as a conference paper at ICLR 2026
High Q-similarity With RoPE
High Q-similarity Without RoPE
Low Q-similarity With RoPE
Query Index
Query Index
Query Index
Key Index
Key Index
Key Index
Figure 7: Ablation of query dynamics and RoPE on a head with a strong sequential pattern. Left: original head with high q-similarity and RoPE enabled. Middle: high q-similarity without RoPE, which retains a rough, broken diagonal with additional vertical streaks. Right: RoPE with perturbed q, where the diagonal tendency is overlaid with scattered, unpredictable activation spikes.
2. High q-similarity without RoPE. In the second variant, we disable RoPE for this head by replacing the rotation matrices with identity, while keeping the original queries and keys unchanged. The attention map still exhibits diagonal bias, reflecting the strong local similarity in the queries and keys. However, the diagonal becomes noticeably rough: is broken into segments and is superposed with vertical streaks. This indicates that high q-similarity alone is sufficient to encourage local, near-diagonal attention, but does not guarantee the smooth, globally shift-invarian diagonal pattern observed in the full model.
3. Perturbed q-dynamics with RoPE. In the third variant, we keep RoPE enabled but mildly perturb the temporal dynamics of the queries by randomly resampling their time indices within the same sequence. This reduces the average cosine similarity between consecutive queries from 0.99 to 0.97, while leaving the keys and RoPE parameters unchanged. The resulting attention map still contains visible diagonal tendency, but it now overlaid with many scattered, seemingly random activation spots. other words, the attention pattern becomes a mixture of predictable diagonal component and unpredictable spikes.
Across these three conditions, observe that: (i) high q-similarity without RoPE yields coarse, locally diagonal pattern, (ii) RoPE with perturbed q-dynamics produces partially diagonal but no- ticeably more unpredictable attention, and (iii) only when smooth q-dynamics and RoPE are both present do we obtain the clean, stable sequential pattern seen in the full model. This ablation sup- ports the mechanism identified by TAPPA that sequential attention patterns arise from the joint effect of smooth input dynamics and RoPE, and that these two factors play complementary roles: input dynamics control whether the pattern is predictable or unpredictable, while RoPE shapes the predictable component into regular, shift-invariant diagonal structure.
F.3 Q-SIMILARITY DISTRIBUTION
To better understand the behavior of q-similarity, we compute per-head q-similarity scores across all layers for two models (Llama-3.1 and Qwen2.5-7B) two representative datasets (GSM8K and AIGC). As shown Figure 8, we have following observations:
Overall high q-similarity supporting temporal continuity. Across all heads and layers, the aver- age q-similarity high for both models (around 0.80 for Llama-3. and 0.86 for Qwen2.5-7B). This empirically supports the assumption of TAPPA that queries tend evolve in temporally continuous manner in a large portion of the network.
Model-specific but layer-structured distributions. Each model exhibits its own characteristic distribution of q-similarity values, indicating that the q-similarity distribution reflects moeel-specific properties and thus naturally calls for per- per-model calibration. the same time, within given model, we observe a clear and consistent structure: heads in the same layer have very similar q q-similarity scores (forming tight clusters), whereas the average q-similarity differs significantly across layers.
21

---

Published as a conference paper at ICLR 2026
Uama-3.1 gsmak
Uama-3.1 (with
meeh
mea
64
1e o
muama m IN ENEME oaw
20-ir uuaan reRak
Layer
hanne
anonue
15 geea 0-00000 pasva. o in
15 gen mamuea: aas aomaanannnn connnnassenn
i. nea Se14 ge
10 BeSsEL i1 2a waw.
4.
13 -i---eE u 20 220g20w0 wan
UN7 monnns n2
-sxena 1230
meme
10
meen
Owen-2.5 gsmmik
Qwen-2.5 (with mean!
mean
moan
25
25 RL
20
20-m mane
Jas mommnneee Man D TE n 100
Layor
15
Tam-u.. aanan Canann DROD
Reseuu REDE
15 Doun W.
mom SOMP HOL W 1 m2
12
D 15 3
1s
10
15
15
20
moon
10
15
20
mian
Hleed
Hleed
Figure 8: Head-wise q-similarity heatmaps for Llama-3.1 and Qwen2.5 on GSMSK and AIGC. For readability, we show only the two decimal digits of each q-similarity value (e.g., "83" denotes a q-similarity of 0.83).
22

---

Published as a conference paper at ICLR 2026
This justifies the design choice in TAPPA of operating at the layer level by using layer-wise averages when building downstream metrics and policies.
Stable across datasets for the same model, enabling lightweight calibration. For fixed model, the q-similarity distribution is highly consistent across datasets. For Llama-3.1, the average q- similarity GSM8K and AIGC differs by only about 0.01. For Qwen2.5-7B. the absolute mean difference between the two datasets is about 0.07, but the overall shape and ranking of layers/heads very similar. In particular, the relative ordering of heads is largely preserved, so percentile-based selection strategies such as selecting the top x% most continuous heads are unaffected. This indi- cates that q- g-similarity has good stability and generalization across datasets, and that only small amount of data is needed to calibrate q-similarity for given model, without requiring separate tuning for each task.
G EXPERIMENT DETAILS
G.1 DETAILS FOR KV CACHE COMPRESSION
Implementation details. Following CAKE (Qin et al.. 2025), we introduce an adjusted per-layer performance score that incorporates q-similarity derived from TAPPA:
P/ Pi + a(1 - S)).
(17)
where Pt denotes the original layer preference score based entropy and variance of attention patterns (as defined in Equation (6) of CAKE), S₁ is the cosine similarity among queries within recent window, which instantiates q-similarity in TAPPA, and a hyperparameter controlling the contribution of q-similarity. Formally,
S ==i(((Q
(18)
The intuition is that lower q-similarity indicates a more random and dispersed attention pattern, which generally requires allocating a larger budget. By adjusting Pt with (1 Si). we bias the score toward layers exhibiting retrieval- like behaviors.
Finally, following the allocation rule in CAKE, we normalize the adjusted scores to distribute the total budget across layers:
P Σ PE
Bi
B
(19)
LLMs, benchmark and baselines. We evaluate our method on Llama-3.1-8B (Dubey et al., 2024) and Qwen2.5-7B (Yang al., 2024a), using the LongBench (Bai et al., 2024) benchmark, which covers 16 long-context understanding tasks.
G.1.1 BASELINES OF KV CACHE COMPRESSION
Baselines include StreamingLLM (Xiao et al., 2023), H2O (Zhang et al., 2024), SnapKV (Li et al., 2024), PyramidKV (Cai et al., 2025), and CAKE (Qin al., 2025). We provide detailed de- scriptions of these baselines in Appendix G.I.I. In the KV cache compression task, we eval- uate our method against five representative baselines. Based on whether the budget allocation across layers is uniform, these baselines can be categorized into Uniform Allocation, represented by StreamingL.MM (Xiao et al., 2023), H20 (Zhang al., 2024), and SnapKV (Li et al., 2024), and Non-Uniform Allocation, represented by PyramidKY (Cai et al., 2025) and CAKE (Qin et al., 2025).
StreamingLLM: retains the first and most recent tokens.
H20: prioritizes tokens with high cumulative attention.
SnapKV: leverages observation window at the end of the input to cluster and preserve important KV positions for each head.
PyramidKV: allocates larger budgets lower layers and smaller ones to higher layers with SnapK eviction indicator.
CAKE: introduces a periericce-iooritizzed adaptive allocation strategy, dynamically adjust- ing budgets across layers.
23

---

Published as a conference paper at ICLR 2026
Table Comparison with additional structured pruning baselines on Llama-2-7B. Results of LLM- Pruner, SliceGPT, LaCo, and ShortGPT are quoted from ShortGPT, and the LLMPruner setting excludes post-training as ShorGGPT.
Method
Pruning Ratio
PIQA
HellaSwag
WSC
BoolQ
RACE-B H
Avg.
L.MMrrnrr
27. 27.00%
71.22
56.46
36.54
55.20
22.56
48.40
SliceGPT
26.40%
66.21
50.27
36.54
38.32
21.07
42.48
LaCo
27.10%
69.80
55.69
40.38
64.07
22.61
50.51
ShortGPT
27.10%
66.43
53.02
52.46
74.71
32.25
55.77
TAPPA
28.10%
66.76
55.97
68.13
62.17
33.88
57.38
G.2 DETAILS FOR LLM PRUNING
Implementation details. Building on the Block Influence (BI) metric proposed by ShortGPT (Men al., 2025), design adjusted proxy score:
BI' BI
(20)
where / is hyperparameter and q-qii importance score derived from q-similarity & in TAPPA.
Following ShortGPT's pruning pipeline, we use the PG19 dataset (Rae et al., 2019) as calibration set. First, we collect hidden states and queries from each layer while running inference on calibration data. Next, we compute the proxy scores for all layers based the adjusted BI score. Finally, we sort the layers in ascending order of scores and remove those with the lowest scores. The number of pruned layers can be adjusted to balance efficiency gains and accuracy preservation.
LLMs, benchmark, and baselines. We evaluate our method on Llama-2-7B (Touvron et al., 2023), Alama-3.1-80 (Dubey et al., 2024) and Qwen2.5-7B (Yang et al., 2024a). Using the procedure described above, we first evaluate how redundant each layer is and decide which layers are to be pruned. Then
perform zero-shot task classification on common sense reasoning datasets: PIQA (Bisk et al., 2019). HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019) and ARC-easy (Clark et al., 2018) at different pruning ratios. In the main experiments, we compare our method with ShortGPT as the primary baseline. We further include additional structured pruning baselines in Appendix G.2.1. We list the removed layers in Table 4 of Appendix G.2.2.
G.2.1 COMPARISON WITH ADDITIONAL STRUCTURED PRUNING BASELINES
To provide more comprehensive comparison for structured LLM pruning, we additionally consider three representative baselines, namely LLMPruner (Ma et al., 2023), SliceGPT (Ashkboos et al., 2024), and LaCo (Yang al., 2024b). LLMPruner gradient- based structured pruning method that removes non-critical coupled structures to preserve model functionality. SliceGPT is a post- training pruning scheme that compresses the model by applying principal component analysis to hidden states in each layer and reducing the embedding dimension. LaCo is structured pruning approach based on layer collapse, which gradually merges similar layers while using a threshold to avoid excessive collapsing. Following the evaluation protocol in ShortGPT (Men et al., 2025), we report results on PIQA, HellaSwag, WSC, BoolQ, and RACE-H. The results of LLMPruner, SliceGPT, LaCo, and ShortGPT quoted from ShortGPT, and the LLMPruner setting excludes post-training as in ShortGT for fair comparison.
As shown in Table ratio. In particular, improves the average score over ShonGGT while maintaining strong accuracy on WSC and RACE-H, indicating that the proposed pruning criterion remains effective in more challenging structured pruning regimes.
method achieves the best average performance under higher pruning
LIST OF REMOVED LAYERS G.2.2
In the LLM Pruning downstream task, we evaluated our pruning method on different LLMs and pruning ratios. we list the removed layers in Table 4.
24

---

Published as a conference paper at ICLR 2026
Table 4: Removed layers for different benchmark models, using PG19 calibration dataset.
Model
Method
Pruning Ratio
Removed Layers
ShortGPT
31%
21, 22, 23, 24, 25, 26, 28, 29,
TAPPA
31%
22, 23, 24, 25, 26, 27, 28, 29
Llama-2-7B
ShortGPT
34%
19, 23, 24, 25, 26. 27, 28, 29, 30 19, 21, 22, 23, 24, 25, 26, 27, 28, 29
TAPPA
34%
ShortGPT
28%
20, 22, 23, 24, 25, 26, 27, 28, 29 22, 23, 24, 25, 26, 27, 28, 29
TAPPA
28%
Llama-3.1-8B
ShorGGPT
31%
20, 21, 22, 23, 24, 25, 26, 27, 28, 29 19, 22, 23, 25, 26, 27, 28, 29
TAPPA
31%
ShortGPT
39%
4,5,6, 7, 10, 11, 12, 14, 15, 20 40, 11, 12, 13, 14, 15, 16, 17, 18, 20
TAPPA
39%
Qwen2.5-7B
ShorGGP TAPPA
43%
11, 13, 14, 15, 16, 18, 19, 20, 21, 23 4,5, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20
43%
Table 5: Comparison of eight q-similarity formulations on Llama-3.1-4 under the 1024 KV budget on LongBench subsets.
Method
MF-en
HotpotQA
QMSum
TriviaQA
Lec
Avg.
q sim cosine
52.63
55.45
24.59
92.04
64.58
57.86
q sim dot
52.57
55.12
24.71
91.87
64.52
57.76
q s_.sim_poe pearson
52.68
54.84
24.80
92.03
64.91
57.85
su.ssm_ccchee cuclidcan
52.58
54.85
24.76
91.62
64.59
57.68
s1
52.57
54.76
24.69
91.79
64.56
57.67
52.59
55.05
24.87
91.86
64.82
57.84
q_simarbf rbf
52.40
54.63
24.43
91.61
64.82
57.58
q.sim_kl
52.54
55.56
24.56
91.48
64.54
57.74
H ADDITIONAL ABLATIONS AND HYPERPARAMETER SENSITIVITY
H.1 ALTERNATIVE SIMILARITY FORMULATIONS FOR Q-SIMILARITY
Throughout the paper, we instantiate q-similarity using cosine similarity, which is used to compute the layer-wise score for KV cache compression. To assess whether the downstream gains rely particular similarity formulation, we replace cosine similarity with seven alternatives and re-evaluate KV cache compression representative LongBench subsets. Specifically, we consider dot-product similarity, Pearson correlation coefficient, Euclidean distance, LI distance, angular distance, radial basis function (RBF) kernel similarity, and Kullback-Leibler (KL) divergence. For distance-based and divergence-basee based measures, lower values indicate higher similarity and are used accordingly in the layer scoring.
The performance variation across similarity formulations is small, indicating that the method does critically depend on particular metric choice. Cosine similarity achieves the best average performance and is therefore used as the default throughout the main experiments.
H.2 SENSITIVITY TO a IN KV CACHE COMPRESSION
We study the sensitivity to the weighting coefficient a the adjusted layer score used for KV cache compression. Table 6 reports results two LongBench subsets under fixed 1024 KV budget. The performance remains stable across a wide range of values, and larger a generally yields slightly better results, consistent with the usefulness of the q-similarity signal.
Hyperparameter selection follows a lightweight per-model strategy. For each model, we evaluate (0.1, 1, 000 a small validation split and select the best a. The selected is then fixed for all
25

---

Published as a conference paper at ICLR 2026
Table 6: Sensitivity to a on Llama-3.1-8B under the 1024 KV budget LongBench subsets. The setting a = O corresponds to CAKE without q-similarity. The setting a oo corresponds to using only g-similarity.
MF-en
HotpotQA
Avg.
(CAKE)
52.16
55.43
53.80
0.1
52.19
55.43
53,81
0.2
52.17
55.39
53.78
0.5
52.17
55.39
53.78
0.8
52.22
55.43
53.83
1
52.14
55.43
53.79
1.5
52.19
55.42
53.80
2
52.27
55.42
53.84
5
52.11
55.56
53.84
10
52.24
55.54
53.89
00
52.41
55.44
53.93
Table 7:
Per-layer computational overhead comparison between q-similarity computation and CAKE under different context lengths. Latency is measured in ms, and memory is measured in MB.
CAKE
q-sim
Improvement Latency
Length
Latency
Memory
Latency
Memory
Memory
4K
0.198
72.12
0.195
8.69
2%
88%
8K
0.308
136.12
0.197
8.69
36%
94%
16K
0,506
264.12
0.199
8.69
61 %
97%
32K
0.874
520.12
0.196
8.69
78% %
98%
datasets and KV budgets for that model the full evaluation. The setting a = oo corresponds to using only the q-similarity term to rank layers.
H.3 COMPUTATIONAL. OVERHEAD OF Q-SIMILARITY COMPUTATION
This subsection evaluates the runtime and memory overhead of computing q-similarity scores dur- ing inference. We measure the per-layer overhead of our q-similarity computation and compare with CAKE, which maintains attention score-based statistics for eviction. All measurements are conducted Llama-3.1-8B with window size of 32, under different context lengths up 32K tokens.
As shown in Table 7, the overhead of q-similarity is small and stable across all context lengths. In particular, the per-layer latency stays below 0.2 ms, and the additional memory consumption remains about 8.69 MB. This behavior is expected because q-similarity only computes query sim- ilarities within fixed window, so its complexity is effectively independent of the total sequence length. In contrast, attention score-based methods like CAKE update and store per-token statisties to derive eviction signals, so both runtime and memory overhead increase with the context length. At 32K tokens, q-similarity reduces the per-layer latency overhead by % and the additional memory consumption by 98 % compared to CAKE, highlighting the efficiency advantage of a query-based signal for budget allocation.
H.4 SENSITIVITY TO 33 IN LAYER PRUNING
We conduct sensitivity study for the weighting coefficient 3 used in layer pruning. As shown in Table 8, the performance varies moderately across B, with a plateau for / E [0.2, 0.4]. Identical results for nearby 3 values occur when the induced pruned layer sets unchanged.
26

---

Published as a conference paper at ICLR 2026
Table 8: Sensitivity 3 Llama-2-7B under a 34% pruning ratio. Identical results for some
values arise when the corresponding pruned layer sets are the same.
β
Piqa
Hellaswag
Winogrande
Arc Easy
Average
0
60.83
42.11
60.38
44.15
51.87
0.1
60.83
42.11
60.38
44.15
51.87
0.2
60,45
48.53
62.43
42.55
53,49
0.3
60.45
48.53
62.43
42.55
53.49
0.4
60,45
48.53
62.43
42.55
53.49
0.5
61.10
46.47
57.77
39.69
51.26
Hyperparameter selection follows a per-model strategy. For each model, we evaluate 3 e {0.1, 0.3, 0.5} small validation split and select the best 3. The selected 3 is then fixed for all datasets and pruning ratios for that model in the full evaluation.
I COMPARISON WITH DUOATTENTION
In this section, we provide detailed comparison between the q-similarity based method derived from TAPPA and DuoAttention (Xiao et al., 2024), a recent baseline that explicitly distinguishes retrieval heads and streaming heads for KV cache compression.
1.1 BASELINES AND METHODOLOGY ADAPTATION
DuoAttention is an optimization- based method that explicitly identifies retrieval heads via training. It assigns a learnable scalar, which we denote as Oljoo, to each attention head to represent its retrieval importance.
To conduct direct comparison between the q-similarity metric derived from TAPPA and DuoAt- tention's learned importance for the layer-wise budget allocation task, we adapted their scoring mechanism into the same layer-wise budget allocation scheme. Specifically, we calculate the im- portance score for each layer by averaging Oldnn values across heads that layer. We then compute the allocated budget Bi for layer using formulation analogous to Eq. 19:
B
duo
B
(21)
(R) duo
kk=
where a the average score of layer 1. This setup allows us fairly evaluate the effectiveness of the two metrics in identifying layers that require higher KV cache budgets.
Table 9: Performance comparison with DuoAttention LongBench using Llama-3.1-8B. For each budget, the best score on each subset is highlighted in bold.
Suggle DecomentiA
Bodgit
Methed
Fall
45.4%
$1.18
47.14
11.39
5.00
4006
512
TAPPA
29.47
42.56
$1.63
51.70
34,53
46.64
45.98
38.81
25.48
34.57
24.31
63.50
91.35
42.42
6.15
99.59
64.56
$7.35
56.87
47.31
DrsAsteation
3011
44.17
53-72
46.5
31.01
16
35.02
26.22
67.00
91.89
6.50
64.7%
55.84
45.11
TAPPA
55.14
36-65
2043
TAPPA
38.70
90.61
45.69
45.19
53.06
5362
$5.49
55.49
46.8
46.3
38.94
10.32
3061
30.54
24.65
34.99
27.25
27.12
71,00
70.10
91.65
91.49
43.74
43.00
6.80
6.00
99.59
99.59
64.96
64.95
51.30
55.82
45.33
46.6
I.2 POTENTIAL FOR HIGHER COMPRESSION RATIO
It is crucial to highlight the fundamental difference in how the two methods categorize attention patterns and the resulting impact on the compression scope. DuoAttention operates on a binary premise where it differentiates Streaming Heads that necessitate only sink and recent tokens from
27

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison of Expected Attention (EA) with and without our q q-similarity based budget allocation on LongBench. For each budget, the best score on each subset is highlighted in bold.
Single DecamentiA
Malii DecumeeeiAA
Summary
Few- Slbee Louniisg
Syuthelic
Cosle
Badget
Methae
Averaage
Lisma-A1-40
512
EA
TAPPA
21.53
34.45
31.22
40.01
46.75
47.59
4092
48:39
17.10
31:17
22.99
26.65
25.43
47.59
72.82
35.60
2.17
94.50
$1.50
$2.80
5I.39
30:37
1004
EA with TAPPA
EA
21.35
26.17
31.53
48.79
62.72
43.13
$2.30
50.30
44.53
45:91
33.75
18.19
38.
29.66
23.90
73335
27.13
27.31
53.50
55.59
$6.70
$7.70
36.86
36.35
4.79
4381
98.58
$3.00
52.99
55.78
54.35
53.42
43,40
42.64
2048
EA
TAPPA
27.37
Qwe62.5-7B
TAPPA
38.41
24.39
44.66
35.59
1004
EA
TAPPA
18:53
35.71
42.03
00
36.93
35.81
36.31
35.43
17.08
18.70
39:57
30.07
21.40
20.68
25.07
25.09
51 31.50
$2.48
43.51
37.06
6.25
51.33
97.00
12.66
52.31
$7.50
36.57
2348
EA with TAPPA
EA
20.34
23.07
43/15
39:45
06.85
97
45.46
45.36
19.47
36.35
35.11
16.36
31.47
31.45
21 64
21.00
35.43
25.66
55.00
55.15
13.99
$4.50
16.18
6.11
98.25
07.33
41.88
14.76
55.01
41:12
42.49
Retrieval Heads requiring full history retention. focus on heads exhibiting streaming behavior.
Consequently, its compression efforts primarily
In contrast, TAPPA provides more detailed categorization. The q-similarity metric derived from TAPPA distinguishes complex Retrieval patterns from variety of regular attention patterns, includ- ing Re-access, Sequential, and Seasonal patterns. Crucially, TAPPA identifies these regular patterns as compressible. This effectively expands the scope of compressible heads beyond just streaming heads. By compressing these additional heads that might otherwise be preserved, this strategy could achieve a higher compression ratio while maintaining model performance.
1.3 EXPERIMENTAL SETUP
We compare DuoAttention and TAPPA under strict KV cache budgets. For DuoAttention, the co- efficient Olduo is set to the official value trained on Llama-3.1 3.88BB and released by the authors.' All results are evaluated on the LongBench benchmark with 16 subsets. We report performance at KV cache budgets of 512, 1024, and 2048 tokens, and also include the full context baseline.
1.4 RESULTS AND ANALYSIS
The results are in Table 9. Across all three budgets, the q-similarity based method achieves higher average performance than the DuoAttention based allocation. At the subset level, the advantage is most visible on multihop reasoning and retrieval-oriente TAPPA improves HotpotQA from 54.58 to 55.43 and TREC from 67.00 69.50. At the 2048 budget, the q-similarity based method reaches an average score of 48.73, which is close to the full context baseline 49.06.
tasks. For example, at the 1024 budget,
COMPARISON WITH EXPECTED ATTENTION
Expected Attention (Devoto et al., 2025) is a training-free KV cache compression method that ranks and prunes KV pairs by analytically estimating how future queries are expected attend cached keys. Expected Attention uses uniform layerwise budget allocation and then applies its expected attention-based importance score within each layer to perform KV pruning. To evaluate the com- patibility of the budget adjustment strategy derived from TAPPA with other compression frame- works, we integrate the q-similarity based layerwise budget allocation into Expected Attention by replacing the uniform allocation while keeping the original Expected Attention scoring and pruning mechanism unchanged. We use the official hyperparameter settings of Expected Attention from the released implementation and do not tune them for integration.
wtw//////w.bb...././//oo/o/ooott http.////m//i//////r/eeeee. Phttps:// tttpss://gii http://ggtttuuucee com
28

---

Published as a conference paper at ICLR 2026
Table 10 reports results LongBench with 16 subsets under KV budgets of 512. 1024, and 2048. Across both backbones and all budgets, adding TAPPA budget allocation improves the average performance Expected Attention. Specifically, the improvement is approximately 46.8% on Qwen-2.5 with the 512 KV budget, increasing the average score from 24.25 to 35.59. These results indicate that the proposed q-similarity based temporal signal tion component that strengthens Expected Attention without modifying its core expected attention estimation.
a plug-in budget alloca-
K BROADER RELEVANCE OF ATTENTION PATTERN ANALYSIS.
Attention pattern analysis benefits both training and inference of large language models. During inference, long reasoning Wang et al. (2025) operates over long contexts, yet only small subset of tokens and attention routes are informative for each step. Distinguishing stable allocation from retrieval-oriented allocation can prioritize evidence tokens and reduce redundant computation while preserving reasoning-relevant information flow. For training, RL training Liang et al. (2025; 2026); Hao et al. (2025) is increasingly influential but remains costly due to long horizon rollouts and repeated forward passes; pattern-bas based diagnostics can guide architectures and objectives that im- prove information density and yield models that more readily admit sparsification quantization. Attention patterns also support model editing Ma et al. (2025) by localizing the attention routes that mediate specific behaviors, enabling targeted interventions with fewer unintended side effects.
L
THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large Language Models (LLMs) were employed solely for the purpose of enhancing the linguistic clarity and stylistic refinement of this manuscript.
29


Reviews:
[{"review_id": "me4PtEqSGk", "reviewer": "Reviewer_UTJo", "summary": "This paper proposes a framework for analysing attention patterns in Transformer-based models; author identify three types of patterns (namely re-access, sequential, and seasonal) from a subset of the heads (referred to as \"unpredictable\") based on query self-similarity and positional embeddings. Based on the proposed framework, authors then propose a method for efficiently allocate KV cache budgets and structured layer pruning, improving over baselines like CAKE and ShortGPT.", "questions": "- Can you please expand more on the runtime/costs of computing q-similarities and what kind of overheads they add to methods like CAKE?\n- Are there cases where q-similarity fails to misidentify retrieval heads? What does it happen in those scenarios?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The analysis in e.g. Proposition 4.1 that links attention stability to query self-similarity, and how query drift induces changes in the logit changes, is novel and interesting; likewise for Th. 5.2 and 5.3, which provide conditions under which sequential/periodic diagonals appear\n- Using q-similarity in CAKE and ShortGPT yields sigificant improvements in several settings", "weaknesses": "- Improvements in CAKE (Tab. 1) seem very marginal, are they statistically significant? Averages are not clearly reported\n- Computing q-similarities for every layer/head seems computationally expensive, but runtimes/costs are not discussed in-depth", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMpi5uzyrL", "reviewer": "Reviewer_yc4e", "summary": "This paper proposes a framework to explain diverse attention patterns in LLMs through temporal continuity analysis. The authors categorize patterns as predictable (re-access, sequential, seasonal) or unpredictable, attributing the distinction to query self-similarity. They provide mathematical analysis of how query/key continuity and RoPE jointly produce these patterns, and validate their framework through KV cache compression and (fewer) LLM pruning experiments. While the paper makes a reasonable attempt to unify attention pattern analysis, the contributions are incremental over existing work (particularly AttentionPredictor). The theoretical analysis, though rigorous, doesn't yield sufficiently novel insights—the sequential pattern analysis overstates its departure from prior work, and the seasonal pattern analysis lacks empirical grounding. The downstream experiments show only marginal improvements.", "questions": "- What is the computational overhead of computing q-similarity scores during inference?\n- What is the precise impact of this KV cache compression method on memory footprint ? And on latency ? \n- How are the hyper-parameters selected ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The temporal continuity perspective provides a systematic way to understand previously fragmented observations about attention patterns. The decomposition view connecting query similarity to pattern stability is intuitive.\n- Rigorous mathematical treatment: the theorems provide formal proofs for the emergence of different pattern types, with explicit bounds relating pattern stability to query/key properties and RoPE parameters.\n- Novel insight on periodic sequential patterns: The analysis of diagonal spacing and experimental validation by manipulating dominant channel locations is particularly interesting.\n- The evaluation on KV Cache compression is performed with different budgets.", "weaknesses": "- Limited novelty: the observation that query continuity drives attention stability was already made by AttentionPredictor (and the authors acknowledge that). While this paper provides mathematical formalization, the fundamental insight is not new. \n- KV cache compression: the improvements over CAKE are marginal and seem to be within noise margins. Other state of art methods such as DuoAttention, Expected Attention could be a stronger baseline.\n- LLM pruning is only compared against a single baseline. There are no other comparisons to other structured pruning methods. If the authors think this makes sense, could they explain why they considered only this one baseline.\n- The hyperparameters  appear hand-tuned without ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUiwhv5pHw", "reviewer": "Reviewer_V8Hz", "summary": "This paper analyzes sparse attention patterns from the perspective of query similarity. It derives theoretical relations between the similarity of consecutive query vectors and the corresponding changes in attention values. The paper further shows how certain query distributions lead to typical attention patterns. It shows potential applications of these insights in attention budget allocation and layer pruning.", "questions": "1. Is the query similarity computed after applying RoPE?\n\n2. Could you provide more justification for using attention patterns to guide layer-wise FFN pruning in Section 6.2? In particular, why does high query similarity (stability) imply that “the layer extracts less novel information”", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a new perspective for explaining existing attention patterns from the query point of view.\n\n2. The paper demonstrates how the proposed query-level observations can inform the design of sparse attention and layer pruning, adding practical value to the theoretical analysis.", "weaknesses": "1. The claim of analyzing the *joint effect of input dynamics and positional encoding* seems overstated. While it would be valuable to disentangle and quantify their respective contributions, the paper instead merges them into the query with post-encoding. This makes the connection to the original input less clear than the abstract and introduction suggest.\n\n2. Several assumptions used in the derivations are not carefully validated, which raises concerns about the reliability of the conclusions. For instance, the assumption of a dominant channel weight in Theorem 5.1 requires empirical support.\n\n3. The empirical section lacks comparisons with more direct and recent baselines, such as DuoAttention [1], which explicitly distinguishes retrieval heads.\n\n4. Some key concepts (e.g., continuity, predictability) are introduced without sufficient explanation in the introduction. Brief definitions would help prevent confusion.\n\n[1] Xiao, Guangxuan, et al. \"Duoattention: Efficient long-context llm inference with retrieval and streaming heads.\" arXiv preprint arXiv:2410.10819 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2g4ngq3cbv", "reviewer": "Reviewer_Qj6A", "summary": "This paper proposes a unifying temporal perspective to explain the emergence of diverse attention patterns in LLMs. The authors categorize patterns into predictable (re-access, sequential, seasonal) and unpredictable (retrieval-like) types, attributing the distinction to variations in query self-similarity over time. They provide theoretical analyses linking pattern formation to query-key continuity and RoPE, and validate their framework through applications in KV cache compression and LLM pruning. Experiments on models like Llama and Qwen show consistent improvements over baseline methods.", "questions": "1.How does q-similarity vary across different layers and heads? Is it consistent across models, or does it require per-model calibration?\n2.The paper claims that high q-similarity implies redundancy in pruning. Is this always true? Could some stable patterns be critical for certain tasks (e.g., syntax parsing)?\n3.In the proof of vertical stability (Theorem 5.1, Appendix B), a crucial step bounds the change in the angle between the query and a fixed key:\n$$|\\phi_{t+1,i}^{(m)}-\\phi_{t,i}^{(m)}|\\leq\\frac{\\|q_{t+1}^{(m)}-q_t^{(m)}\\|}{r_m}$$\nHowever, consider a simplified scenario in 2D: let q_{t}=(1,0) and q_{t+1}=(cos(2arcsin(ε/2)), sin(2arcsin(ε/2))). Here ||q_{t+1}-q_{t}|| ≤ ε， the angle change is 2arcsin(ε/2) ＞ ε. This suggests the above inequality does not hold. \n\nIn the proof of Theorem 5.4 (Seasonal Pattern), the derivation for non-dominant channels contains a critical error. The term (i-t)θ_m is incorrectly repeated in both cosine functions when calculating |a_{t+L,i}^{(m)} - a_{t,i}^{(m)}|. The correct expression for a_{t+L,i}^{(m)} should have a phase of (i-t)θ_m - Lθ_m. More critically, the standard inequality used to bound the difference for non-dominant channels is misapplied. This inequality, | ||u|| ||v|| cos φ - ||u'|| ||v'|| cos φ' | ≤ ..., is valid only when φ and φ' are the geometric angles between the vector pairs (u,v) and (u',v'), respectively. In your proof, you apply it with angles φ = φ_{t,i}^{(m)} + (i-t)θ_m and φ' = φ_{t+L,i}^{(m)} + (i-t)θ_m. However, these are not pure geometric angles but include an additive positional phase. This misapplication renders the subsequent bound on non-dominant channels invalid.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized, with clear motivations, methodical explanations, and thorough appendices.\n- The paper offers a coherent perspective that integrates previously fragmented observations (e.g., sink heads, diagonal patterns, retrieval heads) under a single temporal continuity lens.\n- The paper provides detailed mathematical proofs for each pattern type, clearly linking RoPE mechanics and query-key dynamics to observable attention structures.\n- The proposed q-similarity metric is effectively applied to downstream tasks (KV cache compression, pruning), demonstrating improved performance over strong baselines.", "weaknesses": "- The entire framework hinges on the assumption of temporal continuity in queries and keys. While this is likely a reasonable assumption for many layers and tasks, its universality is not thoroughly explored. The analysis might be less applicable in layers or for inputs where representations change abruptly. A discussion of the boundaries of this assumption would strengthen the work.\n- The q-similarity metric is central to the applications, but its specific formulation (e.g., the choice of cosine similarity) lacks a comprehensive ablation study. It remains unclear how sensitive the performance gains are to these choices, or if an even more effective metric derived from the same theory could be designed.\n- The proofs provided in the appendix, while a valuable effort, contain significant weaknesses that undermine their theoretical rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)