Deconstructing Positional Information: From Attention Logits to Training Biases
Authors: Zihan Gu
Ruoyu Chen
Han Zhang
Hua Zhang
Yue Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Position Encoding; Toeplitz Matrix; Attention Logit.
Primary area: interpretability and explainable AI
TL;DR: We propose a unifying perspective on the role of positional encoding and discover that rope training exhibits implicit biases.

Abstract: Positional encodings, a mechanism for incorporating sequential information into the Transformer model, are central to contemporary research on neural architectures. Previous work has largely focused on understanding their function through the principle of distance attenuation, where proximity dictates influence. However, the interaction between positional and semantic information remains insufficiently explored, and the complexity of mainstream corpora hinders systematic, comparative studies of these methods. This paper addresses these challenges through a deconstruction of the attention-logit computation and a structured analysis of all mainstream positional encodings. A key focus is placed on Rotary Positional Embedding (RoPE), whose product-based structure uniquely facilitates a direct interaction between position and content. To probe this characteristic, we designed a novel synthetic task that explicitly demands a strong synthesis of positional and semantic information. As theoretically predicted, RoPE demonstrates a significant performance advantage over other encodings on this specialized task. Concurrently, this targeted evaluation uncovers an implicit training issue: a hidden bias manifesting as a distinct information aggregation phenomenon in the model's shallow layers, which we term the "single-head deposit pattern." Through subsequent ablation studies, we analyze this pattern and identify a method for its mitigation. These findings highlight the need for a deeper investigation into the training dynamics of positional encodings to bridge the gap between their theoretical design and practical implementation.

Published as a conference paper at ICLR 2026
DECONSTRUCTING POSITIONAL INFORMATION: FROM ATTENTION LOGITS TO TRAINING BIASES
Zihan Gu Gu Ruoyu Chen Han Zhang", Hua Zhang 1 Yue Hu
Institute of Information Engineering, Chinese Academy of Sciences;
2 School of Cyber Security, University of Chinese Academy of Sciences;
3 "Antai College of Economics and Management, Shanghai Jiao Tong University. Iguzihan, chenruoyu, zhanghua, huyue @ @iie.ac.cm
ABSTRACT
Positional encodings enable Transformers to incorporate sequential information, yet their theoretical understanding remains limited to two properties: distance attenuation and translation invariance. Because natural language lacks purely po- sitional data, the interplay between positional and semantic information is still underexplored. We address this gap by deconstructing the attention-logit compu- tation and providing structured analysis of positional encodings, categorizing them into additive and multiplicative forms. The differing properties of these forms lead to distinct mechanisms for capturing positional information. To probe this difference, we design a synthetic task that explicitly requires strong inte- gration of positional and semantic cues. As predicted, multiplicative encodings achieve clear performance advantage this task. Moreover, our evaluation reveals a hidden training bias: an information aggregation effect in shallow lay- ers that term the single-head deposit pattern. Through ablation studies and theoretical analysis, we proved that this phenomenon is inherent in multiplica- tive encodings. These findings deepen the understanding of positional encodings and call for further study of their training dynamics. Our code is available at https: /ttp:////ttttut/tt com Qq hta.icci21111 2tte////////---eeeeeeeeeero -rrepssit-Paatern- Patterr--eeearcch
INTRODUCTION
Positional Encoding (PE) (Vaswani al., 2017; Su et al., 2024) endows Transformer architectures with an understanding of sequence order, compensating for the germutation-invariae nature of self- attention ( (Vaswani et al., 2017). This component is critical their success in both language (Vaswani et al., 2017) and vision (Dosovitskiy et al., 2021) I) tasks. The design of PE has evolved substantially, from the additive sinusoidal embeddings (Vaswani et al., 2017) of the original Transformer to trainable absolute embeddings (Shaw et al., 2018) and more sophisticated relative schemes like T5 biases (Raffel al., 2020) and Rotary Positional Encoding (RoPE) (Su al., 2024). This progression reflects a sustained effort to enhance generalization, extend context length, and improve model efficiency.
Different positional encodings model distance decay and translation invariance in different ways, but these differences are difficult to observe in the face of complex data with human grammatical rules. Current evaluation paradigms, which rely aggregate metrics such as perplexity (Frmo et al., 2025) or extrtll-xtraaootttion benchmarks (Kazemnejad et al., 2023), document these performance differ- ences but fail to illuminate the underlying mechanisms of how positional signals are processed (Peng et al., 2024; Ermo et al., 2025). Therefore, theoretical research on positional encoding relies on synthesis tasks (Zuo et al., 2025; Kazemnejad et al., 2023), but some unexplained phenomena have emerged. For example, despite its strong theoretical properties, such as the attenuation characteristic that supports length generalization, RoPE can underperform simpler relative PEs even models with no PE on certain tasks (Kazemnejad et al., 2023). This discrepancy highlights a critical gap in our understanding: How, precisely, do different PE schemes mediate the interaction between
Corresponding Author

---

Published as a conference paper at ICLR 2026
Generalization 6op
Hidden Shote
Li
CH
RoPE
Wa
Wx
Difference of Two Modes
Attention Machanism
GE K
Hadamard production
Relative PE
Addition
-Epcch
Testing accuracy on tasks that nequire both recognition relative pesitions.
Attention Logit Matrix
Two Modes of PE
Relative PE
Toepirtz Hattrix
Deposit Pattern
G + Co Gg
Ger.
RoPE
Teepiny Hattris
Vadymord production
leyers, position
information is concentrated single head.
(Gy.k Gap + Gp. + Gp Go
Head
Comporentt pure attention mechanism
: Components introduced by PE
The "deposit" head
Figure 1: Schematic Overview. (Left) Each token is viewed as 24 + Ps. After the Wo. WK projections, their interactions form the four inner-product terms of the logit matrix. The framework unifies PE methods by how they introduce Toeplitz drraslation-invariane (Absolute, T5, ALiBi) contribute a Toeplitz bias B or a Toeplitz Grr,ko; RoPE applies relative- position Toeplitz kernel G. multiplicatively through a Hadamard product. (Right) These two modes produce distinct behaviors: RoPE's multiplicatiie coupling gives faster learning and smaller generalization gaps, yet concentrates shallow-layer positional information into a "deposit head".
structure: additive PEs
token content and position? This lack of mechanistic clarity hinders both the explanation of existing performance trade-offs and the principled design of future PE schemes.
To bridge this gap, our approach is grounded in a core principle of positional encoding: the relationship between any tokens should depend on their relative distance, not their absolute positions. This property of translation invariance is formally captured by the structure of a Toeplitz matrix, in which all elements along any given diagonal are identical. This insight allows us deconstruct the attention logit calculation and propose a unified, Toeplitz-based framework for analyzing PE mechanisms (Figure 1). our framework, each token's embedding composed of content component (what it is) and a position component (where it is). The central idea is that these position components collectively induce Toeplitz structure in the final attention scores.
This framework clearly distinguishes between two primary classes of PE. Additive methods, such as relative position biases, operate by adding a static Toeplitz matrix of position scores directly to the attention logits. In contrast, multiplicative methods like RoPE dynamically integrate positional information into the query and key representations. This creates a powerful content-position interaction, where the positional influence on attention scores is conditioned on the token's content. While this coupling provides a significant advantage on tasks where meaning is tightly bound to relative location, argue that it also introduces a strong inductive bias. This bias manifest drawback, concentrating the learning of positional logic into a small, specialized subset of attention heads.
Observations based on the theoretical framework directly triggered our experimental design: we conduct experiments on a series of carefully designed synthetic tasks. We generate random sequences from small vocabulary and use labeled "anchor" tokens to define two contrasting objectives: a rosition-sensitive task requiring reasoning about the relative positions of anchors, and position- agnostic task dependent only on token counts. This controlled setup enables us to precisely isolate and evaluate the model's 'ss positional reasoning capabilities. Our experiments confirm that RoPE decisively outperforms other methods on the sesssivv-eensiivee task while underperforning on the position-agnostic one. More importantly, this setting reveals a striking artifact unique to RoPE: the "single-head deposit pattern"', where nearly all positional processing in the shallow layers becomes concentrated into a single attention head.
Further investigation confirms this phenomenon is unique to RoPE on tasks with strong content- position correlation. Through rigorous ablation studies and theoretical derivation such as injecting absolute positional encodings selectively applying RoPE to a subset of heads-we demonstrate that the deposit pattern is an intrinsic property of RoPE's multiplicative structure, not an incidental training artifact. This finding also explains why Transformers using RoPE tend not to form the latent
2

---

Published as a conference paper at ICLR 2026
positional representations observed in models with additive PE (Zuo al., 2025; Kazemnejud et al., 2023). We posit that this intense, structurally- induuce specialization is primary cause of the gap between RoPE's theoretical promise and its practical performance.
Our contributions can be summarized as follows:
We propose unified analytical framework using Toeplitz matrices that categorizes positional encodings into additive and multiplicative mechanisms, clarifying their distinct effects on attention logits.
We empirically identify and analyze RoPE's performance paradox through targeted synthetic tasks, revealing a unique 'single-head deposit pattern' where positional logic becomes highly concentrated in shallow layers.
We conduct causal analysis to demonstrate that the deposit pattern is an intrinsic property of RoPE's multiplicative architecture, offering mechanistic explanation for its observed performance paradox.
RELATED WORK
Positional Encoding in Transformers. Positional encoding was introduced alongside the Trans- former architecture to endow self-attention with sequence order information (Vaswani et al., 2017). Early approaches employed learnable positional embeddings (Shaw et al., 2018), while subsequent work integrated position biases directly into the attention logits, giving rise to relative position encoding and its extensions (Press al., 2022: Li al., 2024; Chi al., 2023; 2022: Raffel et al., 2020; Su et al., 2024; Zhao et al., 2025). Among these, Rotary Positional Encoding (RoPE) (Su et al., 2024) stands out by applying a multiplicative bias to the QK" matrix, and has been widely adopted in modern open-source models. Related efforts sliding-windowd and -ingth-generalization designs are also often framed as positional encoding variants (Kiyono et al., 2021; Chen et al., 2024a;b; Ding al., 2024; Peng al., 2024). Recent RoPE-based improvements include Frequency-Partitionee Partitioned Encoding (FoPE) (Ermo et al., 2025) and the Multi-head Latent Attention (MLA) mechanism in Deepseek- V3 (Liu et al., 2024).
Mechanistic Analyses of Positional Encoding, Empirical studies have probed how Transformers internalize positional signals. Some works demonstrate that even without explicit positional embed- dings (NoPE), models can learn positional distinctions (Wang et al., 2024; Zuo et al., 2025; Haviv al., 2022; Barbero et al., 2024), occasionally outperforming PE-equipped counterparts on select tasks (Kazemnejad et al., 2023). Explanations range from implicit causal masking (Zuo al., 2025) to dataset-dependent effects (Barbero et al., 2024). Other analyses document characteristic behaviors of RoPE, such as "massive value" phenomena and rotation artifacts (Jin et al., 2025; Jonasson, 2025).
Toeplitz Structure and Spectral Theory. The action of positional encoding in attention can be unified under Toeplitz-matric -atrixx representations (Böttcher & Silbermann, 2006; Grenander, 1958; Gray, 2006; Oppenheim & Schafer, 1999). Classical results on Toeplitz spectral distributions, including Szegō's 's theorem, inform our theoretical framework. Moreover, the impact of matrix spectra on optimization convergence is well studied in numerical linear algebra and convex optimization contexts (Horn & Johnson, 2012; 1994; Boyd & Vandenberghe, 2004; Nesterov, 2013).
METHODOL.OGY
3.1 PRELIMINARIES: POSITIONAL ENCODING IN ATTENTION
The Transformer attention score between a query 9 at position i and a key ky at position j is based on their inner product, q. ky. Since this computation does not depend on sequence order, Positional Encoding (PE) schemes introduce positional information by modifying either the token representations or the attention logits.
Absolute PE (Vaswani et al., 2017) adds position vector P; to the initial embedding ez:
h; Pi.
The query and key vectors derived from h; therefore mix content and position inside the representation, making the attention score depend on the absolute token positions.
3

---

Published as a conference paper at ICLR 2026
Relative PE (Raffel et al., 2020) incorporates positional information directly into the attention logits through a learnable cestance-dependee term:
Oij X exp(1, q Ri + bz-a)
defining feature of relative PE is translation invariance: the positional contribution depends only on the displacement i, not on the absolute locations of and j. Within this family, T5 (Raffel et al., 2020) uses learned table indexed by (j i), while ALiBi (Press et al., 2022) applies fixed slope that increases linearly with distance. Thus both are translation-invariant, but differ in whether the distance mapping is learned (TS) or predetermined (ALiBi).
Rotary PE (RoPE) (Su et al., 2024) introduces position by rotating the query and key vectors with position-dependent matrix R, A key identity shows that the resulting inner product depends only on the relative displacement:
(Reg:) (Rykk) = (R -, R
Thus RoPE implements positional encoding through structured rotation that couples content and position via relative position, rather than through additive shifts or explicit bias terms.
3.2 A UNIFIED FRAMEWORK VIA TOEPLITZ MATRICES
To analyze these mechanisms within a common framework, we introduce two foundational assump- tions. The first is a conceptual decomposition of token representations.
Assumption 3.1 (Representation Decomposition). Any token representation z; can be conceptually decomposed into a position-independee component ps. such that v4 = G + P:-
content component o; and position-dependent position
Remark 3.1. This decomposition is not a modeling constraint but structural lens. It is supported in four complementary ways: (i) Design: Absolute PE explicitly detines position-dependent vectors ps (Vaswani et al., 2017); (ii) Emergence: Architectures lacking PE still develop internal positional directions (Zuo et al., 2025); (iii) Functionality: Ablating P: hurts performance more than ablating random vectors of equal norm (Section 5.4); (iv) Generality: We treat the removal of as manifold-based coordinate subtraction rather than a rigid linear projection, ensuring compatibility with diverse PE schemes.
This allows us analyze how PE schemes construct or interact with the positional component pi- The core principle uniting relative PE schemes is translation invariance: the positional component of the attention score depends only on the relative distance j i. This principle is perfectly embodied by specific matrix structure.
Definition 3.1 (Toeplitz Matrix). A matrix T is a Toeplitz matrix if its entries are constant along each diagonal, i.e., Tı,j = 0i-j for some sequence {ax). It is defined by 2N unique values for an N x N matrix.
Such matrices naturally arise whenever a quantity depends only on the relative displacement j i, making them the canonical representation of translation-invarian positional interactions. This leads to our second assumption, which connects the positional components of tokens the structure of the attention matrix.
Assumption& 3.2 (Toeplitz Structure from Positional Interaction). Since the positional contribution to attention depends only the relative displacement i, the Gram matrix formed by the position- dependent components naturally takes a Toeplitz form.
Remark 3.2. Assumption 3.2 can be viewed less as a modeling assumption and more as a working hypothesis that captures the intended innvariionnnvaariant decay, and that the p: decomposed from Assumption 3.1 should have such property. This assumption could be checked by sinusoidal absolute PEs and is explicitly enforced by construction in relative PE biases. It provides powerful analytical tool for understanding the structure of positional information in the attention matrix.
Under Assumptions 3.1-3.2, the attention logits can be decomposed into interpretable content and position interactions. Let of ky be query/key vectors derived from content components and qf, k2 be those from position components. Let Gr denote the Gram matrix where (Gu,u)i. = u Wj.
4

---

Published as a conference paper at ICLR 2026
For additive mechanisms (e.g., Absolute Relative PE), the logit matrix is sum of components:
Additive ke Gge, KP + Gop Ggp.Ao ke B
(I)
Here, B denotes the explicit relative-biat bias matrix used only by relative encodings such T5 and ALiBi. For absolute or learned absolute PEs, we instead have B = 0, and the Toeplitz positional structure is provided solely by the term Ggr, Thus, in all additive mechanisms the Toeplitz component arises either from Gor. kv (absolute PE) or from B (relative PE), while the cross-terms Gye,kr kr + Ggp,te remain the only channel through which content can interact with relative position.
In contrast, multiplicative mechanisms like RoPE induce a different structure. Following its formulation using complex numbers in Su al. (2024), the logit matrix becomes:
LROPE Re [G + Gge k0 + Re (gr Ge}
where the Hadamard product and Ge is a Toeplitz matrix.
Remark 3.3. The complex form of RoPE is an important component of the original paper (Su et al., 2024). This form, combined with the Abel summation, can prove an upper bound on its distance decay. A detailed derivation of how to convert the complex RoPE form into the Toeplitz matrix form above is provided in Appendix B. This formulation highlights that RoPE does not add position information; instead, it modulates all content interactions through shared position-dependen kemel.
This formal analysis reveals critical distinction. Additive methods combine content and position signals, but RoPE's multiplicative structure uses Toeplitz kernel (Ge) to modulate the entire content- content and content-position interaction. This enables a more direct and expressive coupling, allowing the model to learn attention patterns where the relevance of token's 's content is conditioned on its relative position to another. This type of task cannot be achieved through pure translation invariance; instead, it requires that specific translation distance be directly affected by the content feedback. We hypothesize that this powerful coupling mechanism comes with a drawback: by routing the learning of positional dependencies through this multiplicative kernel, RoPE may create a strong inductive bias that encourages positional specialization, concentrating this logic into a small subset of attention heads. This hypothesis motivates the experiments in the following section.
4 EXPERIMENTS
To empirically test our hypothesis about the distinct behaviors of additive and multiplicative positional encodings, we design two synthetic tasks. These tasks allow us to isolate the model's ability to couple token content with positional information in a controlled setting. We use a 6-layer Transformer decoder our base model and evaluate six PE configurations: Absolute (Vaswani et al., 2017), T5 Relative Bias (Raffel et al., 2020), ALiBi (Press et al., 2022), RoPE (Su et al., 2024), NoPE (no explicit PE) (Kazemnejad al., 2023), and a Random initialized embedding baseline. Our experiments are averaged over five seeds, and the variance is less than 5%. Full implementation and training details provided in Appendix D.
4.1 SYNTHETIC TASK DESIGN
Task 1 Position-Sensitive:
Relative Distance Classification. This task is designed to require strong content content-position position coupling, the theorized strength of multiplicative mechanisms. Each input sequence contains two unique "trigger" words drawn from vocabulary of 1,000 tokens. The model must predict the relative distance between them, formulated as classification problem over binned distances. Success requires the model to identify what the trigger words are and determine where they are in relation to each other. We generated a dataset 100,000 examples for this task.
Trigger Word Counting, This task serves as control, designed to penalize any inflexible positional bias. The model must predict the total number of occurrences of a predefined set of 20 trigger words within sequence, regardless of their positions. Because positional information is a nuisance variable, an ideal model should learn to ignore it. We generated a dataset of 50,000 examples for this task.
Task 2 Position-Agnostia
5

---

Published as a conference paper at ICLR 2026
4.2 PERFORMANCE AND GENERALIZATION ANALYSIS
We first analyze the learning dynamics and generalization performance of each PE method on our two tasks. The results are presented in Figures 2 and 3.
Training Accuracy
Testing Accuracy
Absolute PE
Accuracy
Relative PE
ALiBi
0.5
0.5
Random Initialization
NoPE
RoPE
25
50
75
100
25
50
75
100
Epoch
Epoch
Figure 2: Training and test accuracy on Task 1 PPositionSSnsstivve RoPE's multiplicative coupling
enables it to significantly outperform all other methods in both convergence speed and final general- ization.
On the Position-Sensitive Task, RoPE Excels, As predicted by our framework, RoPE (Figure 2) demonstrates superior performance. It converges fastest and achieves the highest test accuracy with minimal generalization gap. This supports our theory that its multiplicative content-position coupling is uniquely suited for tasks demanding this form of reasoning. In contrast, additive methods show mixed results. Absolute PE performs second best, as its initial positional signal can be adapted by subsequent layers. The Random embedding baseline learns the training set but fails to generalize, as its non- To-pllppi structure lacks the translation invariance required to learn a general rule. Methods like ALiBi, with its fixed and data-agnostic Toeplitz bias, struggle to adapt, while NoPE and standard Relative PE fail entirely, lacking any mechanism to bind content to position.
Training Accuracy
Testing Accuracy
Absolute
Accuracy
0.8
Relative PE
ALIBI
0.5
0.4
Random Initialization
NoPE
50
100
150
50
100
150
RoPE
Epoch Figure 3: Training and test accuracy Task (Position-Agnostic) Strong, inflexible positional
Epoch
biases like ALiBi are detrimental. RoPE's adaptive nature allows it to perform well, nearly matching methods with no positional bias.
On the Position-Agnostic Task, Strong Positional Biases Detrimental. Figure confirms second prediction: on a task where position is irrelevant, strong positional inductive bias is harmful. ALiBi, with its rigid. non-adaptive bias, suffers the most severe performance degradation. Conversely, NoPE and Relative PE (which learns to attenuate its bias towards zero) perform well, as they do not impose counterproductive positional structure. RoPE also performs strongly, demonstrating its adaptability; its multiplicative mechanism allows the model to learn to down-weight the positional signal when it is not useful. Absolute PE incurs small penalty, while the Random baseline again overfits, learning spurious correlations between positions and counts in the training data.
4.3 EVIDENCE OF THE LINGLE-HEAD DEPOSIT PATTERN IN RoPE
Our theory suggests that RoPE's multiplicative structure concentrates positional reasoning into few specialized heads. To find direct evidence for this phenomenon, we conduct a head-wise causal ablation study on the models trained on Task 1. For each head in the network, we zero out its output and measure the resulting drop in test accuracy. large drop indicates that the head critical the task.
The results, shown in Figure 4, provide striking confirmation of our hypothesis. In the RoPE model, nearly all positional logic is localized to a single head in the first layer. Ablating this one head causes a catastrophic drop in accuracy (==60%). while ablating other heads has a negligible effect. We term this phenomenon the "single-head deposit pattern." Crucially. this pattern is unique to the combination of the RoPE architecture and a position-sensitive task. As shown in Figure 4, the pattern does not emerge in the NoPE model on the same task, does it appear in the RoPE model

---

Published as a conference paper at ICLR 2026
RoPE, Teek 1- Laver
1.0
Deposit Fatiem
0.5
0.8
0.0
NoPE, Task 1 Layer
1.0
Accuracy
0.6
RoPE, Task NoPE, Task RoPE, Task Baselines
Average Patter
0.5
0.4
0.0
RoPE, Task Layer
0.2
Head
Layer
Figure 4: Head-wise ablation reveals unique "Deposit Pattern" in RoPE. Each violin shows the distribution of accuracies after ablating each head in that layer; long vertical tails correspond to outlier heads whose removal causes large accuracy drop. The three insets illustrate the characteristic shapes of these distributions: a "deposit" shape with single deep outlier (RoPE- Task 1), flat "average" shape (NoPE-Task 1), and noisy "shock" shape (RoPE-Task 2). These shapes match the contours of the main violins. Full per-head curves are provided in Appendix D.2.
trained on the vosition-agnostic Task 2. This confirms that the deposit pattern not a generic artifact of Transformer training but a direct consequence of RoPE's 's multiplicative content-powt mnteettppositon coupling mechanism. We therefore interpret the deposit pattern as a training bias rather than clean modularity, although full causal link to length generalization is left for future work (Appendix E).
5 ABLATION STUDIES AND MECHANISTIC INSIGHTS
The discovery of the "single-head deposit pattern" raises critical questions about the underlying mechanics of RoPE. Is this pattern an inevitable side effect of multiplicative coupling, an efficient specialization, or a wasteful bottleneck? To elucidate the mechanisms driving this phenomenon, we design three targeted ablation studics, each testing a specific hypothesis.
5.1 ABLATION 1: DOES ROPE SUPPRESS LATENT POSITIONAL REPRESENTATIONS?
We hypothesize that RoPE's explicit, multiplicative positional signal is so potent that it inhibits the model from forming its own implicit positional representations-aa known capability of standard Transformers (Zuo al., 2025; Kazemnejad al., 2023) thereby causing other heads to remain position-agnostic.
Hypothesis:
To test this, inject a redundant signal by augmenting a RoPE-equipped Transformer with an additional Absolute PE at the input layer. This directly provides an explicit p: component before the RoPE-enabled attention layers. We then observe if this alters the deposit
Experimental Design:
pattern.
Task
Testing Accuracy
Task Testing Accuracy
Layer
1
0.8
Accuracy
0.5
0.4
RoPE
RoPE
RoPE-Abs
50 Epoch
100
0
50
100
0.9
Layer
Epoch
0.8
0.7
For RoPE-Abe, Deposit Pattern diseppear Leyer and
0.6
Layer
12
16
Figure 5: (Left) Performance of hybrid RoPE Absolute PE model. (Right) The deposit pattern is initially suppressed in Layer 1 but re-emerges in deeper layers, demonstrating the strong inductive bias of RoPE.
Results: The results, shown in Figure 5, are revealing. The explicit Absolute PE signal disrupts the deposit pattern in the first layer, distributing positional responsibility more evenly. However, the pattern re-emerges in deeper layers. This strongly suggests that while an initial explicit signal can be utilized, the powerful inductive bias of RoPE's multiplicative coupling eventually reasserts itself,
7

---

Published as a conference paper at ICLR 2026
overriding the additive signal. This confirms our hypothesis: RoPE's mechanism is dominant that suppresses the model's 's tendency to form or utilize other forms positional representation.
5.2 ABLATION 2: Is A SINGLE
HEAD SUFFICIENT?
The deposit pattern implies that for tasks like ours, single RoPE-enabled head is sufficient for positional reasoning. rendering additional RoPE heads redundant.
Hypothesis:
Experimental Design: We test this by systematically reducing the number of attention heads that utilize RoPE, from all heads down to just one, with the rest operating as NoPE heads. We evaluate performance on Task 1.
Task
Task
0.8
Accuracy
0.2
4
10
14
16
8
10
12
14
(NoPE)
Head
(RoPE) (NoPE)
Head
(RoPE)
Figure 6: Performance on Task when RoPE is applied to a diminishing subset heads. Performance with just
remains near-optimal
two RoPF-enabled heads.
Results: As shown in Figure 6, performance on Task remains near-optimal even when only one or two heads are equipped with RoPE. This confirms that a small number of specialized heads are indeed sufficient to capture the necessary positional information for this task. The result supports our explanation for the deposit pattern: once head effectively learns the content- poseenttoooiioonn interaction via the multiplicative mechanism, other heads become inert with respect to this task, as there is no learning incentive develop redundant positional capabilities.
5.3 ABLATION 3: CAN HYBRID ARCHITECTURE MITIGATE INEFFICIENCY?
The deposit pattern, while effective, is an inefficient use of model capacity. We hypothesize that a hybrid architecture can mitigate this by explicitly dedicating resources, retaining RoPE's strengths while improving robustness.
Hypothesis:
Experimental Design: To explore this, we evaluate the Multi-Laaeen Attention (MLA) architecture from DeepSeek (Liu et al., 2024). MLA formalizes hybrid approach by creating parallel pathways for position-agnostic and position-sensitive processing within single head. Formally, the query-key inner product is structured as concatenation of NoPE component and a RoPE component:
(g., ky) (Wurqz; R, WoR)). (WUKI, W,RR)),
where (: denotes vector concatenation and the projection matrices for each pathway are distinct. This structure ensures that each attention head computes both NoPE-style and RoPE-style logit simultaneously. Interpreted through our theoretical framework, this mechanism effectively modifies the logit matrix by creating a weighted combination of standard content matrix and RoPElmodulated one:
LMLA ccomplx (Gq.k + Gq.p + Gp.k + Gp) (aG. + I).
This prevents the purely multiplicative signal from dominating and encourages more distributed representation position. We replaced our standard attention with the MLA module and evaluated on both tasks.
Method MLA 88.34
Task
Task 2
Accuracy
0.8
97.41
RoPE
92.64
69.43
NoPE
3.51
77.69
0.2
Layor
Table 1: MLA performance comparison on the two tasks.
Figure 7: Head-wise ablation violin plot under MLA.
8

---

Published as a conference paper at ICLR 2026
The MLA architecture successfully mitigates the deposit pattern. Figure 7 shows that single head is indispensable, and positional responsibility is diffused across the model. Furthermore, Table 1 shows that MLA nearly matches RoPE's performance on Task while dramatically improving on Task 2. This demonstrates that by preventing over-specialization, the hybrid model achieves a more robust balance.
Results:
5.4 ABLATION 4: WHERE Do ADDITIVE AND MULTIPLICATIVE PES STORE POSITIONAL INFORMATION?
Hypothesis. Given the decomposition 2; = C. + p: (Assumption 3.1), we ask whether different PE mechanisms retain an explicit positional direction ps the activations, whether this information absorbed into the query-key parameterization. Our hypothesis is that additive PEs preserve activation-level ps, while RoPE rapidly suppresses it.
Experimental Design. For each layer we ablate either (1) the fixed positional vectors ps (Absolute PE) or (2) norm- mete-md random vectors, and measure test accuracy. A larger drop under pe-removal indicates that the model relies on an explicit positional direction. For RoPE, we repeat the same intervention on the hybrid RosP model (Section 5.1), where ps is added only at the embedding layer.
Absolute
Absolute RoPE
100
Accuracy
Accuracy
Baseline (B-6811)
Baseline (0.3323)
0.2
Nean
0.7
Muan
Absolltte
Absoluse PE
0.0
0.6
Layer Index
Layer Index
Figure 8: Laser-wise ablation absolute positional vectors versus norm- matched random vectors. Left: Absolute PE shows consistently larger drops under p:-removal, confirming that additive PEs preserve explicit positional directions in the activations. Right: For Absolute+RoPE, this difference vanishes after Layer 2, indicating that RoPE suppresses activation-level positional directions and shifts positional structure into the attention parameterization.
Results. Fig. 8 shows a clear contrast: (1) For Absolute PE, removing pt causes a consistently larger accuracy drop than removing norm-metched random vectors, indicating that additive PEs preserve explicit positional directions the activations: (2) For Absolute+RoPE, this difference disappears after Layer 2. showing that RoPE suppresses activation positional directions and shifts positional information into the query-key parameterization.
Summary. Additive PEs store position in explicit activation directions pt, whereas RoPE rapidly eliminates these directions and embeds positional structure in the frequency-modubee parameters. This is why the structure of MLA can eliminate this phenomenon.
6 THEORETICAL ANALYSIS OF THE DEPOSIT PATTERN
The empirical discovery of the "single-head deposit pattern" in RoPE (Su et al., 2024) invites formal theoretical explanation. In this section, we provide a mathematical argument demonstrating that this phenomenon is an inherent consequence of RoPE's design. We analyze the gradient signal for the relative distance classification task (Task 1). Let (is. je) be the positions the trigger words for a sample s, with the task being to predict the relative distance d, := lJ. i.l- The algebraic intuition behind our proof this: for a sample labeled l jl. the optimization rewards the distance signal j - Jl and penalizes other distance signals. For addition, the reward and penalty always antagonistic because the position bias lacks context information. However, for multiplication, the penalty is attenuated by frequency, making the reward cumulative. This accumulation ultimately leads to an initial top-level bias in the gradient signal, which is amplified by backpropagation. As comparison with RoPE, we also analyze ALiBi (Press et al., 2022) on this task.
Our analysis relies on the following idealized assumptions, which are empirically plausible in the context of early training.
Assumption 6.1 (Formal Setup for Gradient Analysis).

---

Published as a conference paper at ICLR 2026
(A1) Jacobian Stability. Each inter-layer Jacobian j has singular values bounded in [1 0n, 1 + oq] for aq E (0, consistent with architectures using pre-normalization.
(A2) Monotone Anchor Path. For each sample there is an "anchor" row u, where the gradient signal is positively correlated with the target token's value vector. Formally, (h.L), ( 2 n > O. This formalizes that a basic learning signal exists.
( uesi
(A3) Projection Separability (for ALiBi). The projection matrices Wo, WK. , Wy are such that ker Wq n ker Wx & ker Wy. This is generic condition in overparameterized models.
Our analysis demonstrates a fundamental difference in how RoPE and ALiBi structure the gradient signal for relative position tasks.
Proposition 6.1 (RoPE Gradient Seed Coherence). Under Assumption (A2), the aggregated anchor
gradient for RoPE, H (d), has a deterministic lower bound. A strictly positive seed (HL (h) (d) > 0)
guaranteed if the learning signal sufficiently strong to satisfy the condition ai (d) TJ+ (d) >
(b)
(h)
CO) XL.
Proposition 6.2 (ALiBi Gradient Cancellation). Under Assumptions (A3), for any empirical distance
distribution Pu. there exists a batch of samples realizing jAx such that the aggregated anchor gradient for ALiBi vanishes for all heads and distances: H (h) (d) 0.
The existence of a non-cancellable seed in RoPE is the starting point. The following theorem shows how this seed inevitably leads to the deposit pattern.
Theorem 6.1 (Exponential Amplification and Specialization). Let U be the subspace of gradient directions corresponding the positive RoPE seeds from Proposition 6.1. The signal-to-noise ratio of the gradient within this subspace is amplified exponentially during backpropagation. Crucially, the margin between the gradient signal for the dominant head and that of the aecond-dominan head also grows exponentially.
L-1
Margin; 2 Margin L
II Tke
where each gain factor Vik >>
(5)
Implication. Theorem 6.1 provides the theoretical basis for the single-head deposit pattern. Even a minuscule initial advantage for one head in learning the position-sensitive task (a non-zero Marging) becomes exponentially magnified as gradients propagate to lower layers. This dynamic strongly incentivizes the optimizer allocate the entire task of positional reasoning that single head, as its parameters receive a disproportionately larger learning signal, thus explaining the observed specialization. The detailed proofs for these results are provided in Appendix C.
7 CONCLUSION AND LIMITATIONS
In this work, we introduced a novel framework that leverages spectral theory to deconstruct how posi- tional encoding (PE) mechanisms operate within the Transformer attention matrix. We demonstrated that this framework provides a powerful lens, allowing us to predict learning dynamics and uncover previously hidden processing patterns. Our primary discovery, the "single-head deposit pattern" in RoPE, serves as clear mechanistic explanation for the observed paradoxes in its performance. By identifying the distinct signatures of additive and multiplicative coupling, our analysis not only diagnoses the cause of this over-specialization but also points toward solution. The success of hybrid architecture validates our theory and suggests that controlled mixing of positional signals promising strategy for achieving both spectral stability and robust, distributed representations. Furthermore, we exploit gradient flow to verify that this is an intrinsic property of the RoPE.
Firstly, the connection hypothesize between the deposit pattern and RoPE's known struggles with length extrapolation requires direct empirical validation. Secondly, future work should probe the limits of these positional mechanisms more complex, algorithmic tasks such sequence reversal or Dyck language recognition. However, most of these tasks are beyond the capabilities of transformers, more complex controlled analysis is a serious challenge for future work.
Limitations:
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62372448.
REFERENCES
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205 2024. 3
Albrecht Böttcher and Bernd Silbermann. Analysis of Toeplitz operators. Springer, 2006.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. 3. 13, 14
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. CLEX: Continuous length extrapolation for large language models. In The Twelfth International Conference on Learn- ing Representations, 2024a. URL httph:tt///// httpss////ppeeeeew..ooo net/
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongloRA: Efficient fine-tuning of long-context large language models. In Inl. Conf. Learn. Represent. (ICLR), 2024b.
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 2022.
Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Association of Computational Linguistics (ACL), 2023. 2,
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond million tokens. arXiv preprini 133Y::3402.13733, 2024. 3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Ncil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent. (ICLR), 2021. 1
Hua Ermo, Jiang Che, Lx Xingtai, Zhang Kaiyan, Ding Ning, Sun Youbang, Qi Biqing, Fan Yuchen, Zhu Xuekai, and Zhou Bowen. Fourier position embedding: Enhancing attention's periodic extension for length generalization. Int. Conf. Mach. Learn. (ICML), 2025. 1,3, 24, 25
Robert M Gray. Toeplitz and circulant matrices: A review. Foundations and Trends in Communica- tions and Information Theory, 2(3-4):155-233 239, 2006.
Ulf Grenander. formss and their applications. Univ of California Press, 1958. 3, 14, 15
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. 3,21
Roger A Horn and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012. 3
Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. In Int. Conf. Mach. Learn. (ICMI), 2025. 14, 25
11

---

Published as a conference paper at ICLR 2026
André Jonasson. Rotary outliers and rotary offset features in large language models. arXiv preprint arXiv:2503.01832. 2025. 3
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy, The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 2023. 1, 3, 5, 7, 21, 25
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. 3
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli.. Functional interpolation for relative positions improves long context transformers. In Iut. Conf. Learn. Represent. (ICLR), 2024.
3
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. 3,8
Yurii Nesterov. Introductory lectures on convex optimization: basic course. Springer Science, 2013.
Alan Oppenheim and Ronald W Schafer. Discrete-Time Signal Processing. Prentice Hall, 1999.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. 3,
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In Int. Conf. Learn. Represent, (ICLR), 2022. 3, 4, 5,
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. 1, 4,
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv: 1803 1803.02 055, 2018. 1,3
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 2024. 1, 4, 5, 9, 14
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 4,
Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. In Findings of the Association for Computational Linguistics: ACL 2024, 2024. 3
Jianbo Zhao, Taiyu Ban, Zhihao Liu, Hangning Zhou, Xiyang Wang, Qibin Zhou, Hailong Qin, Mu Yang, Lei Liu, and Bin Li. Drope: Directional rotary position embedding for efficient agent interaction modeling. arXiv preprint 25X::22003.5029, 2025. 3
Chunsheng Zuo, Pavel Guerzhoy, and Michael Guerzhoy. Position information emerges in causal transformers without positional encodings via similarity of nearby embeddings. In Proceedings of the 31st International Conference on Computational Linguisties, 2025. 1, 4, 7, 25
12

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT OF LLM USAGE
During the preparation of this manuscript, large language models (LLMs) were employed in a limited and auxiliary capacity. Specifically, their usage was restricted to the following three aspects: (1) checking grammar and expression at the sentence level, thereby providing local linguistic refinement; (2) performing global polishing after the draft was completed, ensuring that the overall exposition conforms to idiomatic English usage: and (3) improving the readability of the proof details presented in the appendix.
At no stage were LLMs used for generating research ideas, developing arguments, or modifying the substantive content of this work. Their sole role to assist in enhancing clarity and effectiveness of communication.
A NOTATIONS LIST
The notations used throughout this article are summarized in Table 2.
Table 2: Some important notations used in this paper.
Notation
Description
Real vector in IR20
24
X;
Complex vector in R.
Content label for a token hidden activation Position label for token or a hidden activation
6,
Absolute position encoding for The bias decided by the relative position encoding The rotary matrix for position of RoPE The gram matrix for the vector columns {ur), {ur}
or B
R
Gu
Logit value of attention layer
(x(a)
Toeplitz Matrix for 0; -N, -N,... ..... N
B DETAILED METHOD ANALYSIS
In this section of the appendix, we provide detailed theoretical analysis and derivations supporting the non-trivial conclusions presented in the main text. Our analysis relies idcalized proof's that simplify the complex optimization dynamics neural networks, aiming to reveal a theoretical explanation for the observed experimental phenomena. While the actual simulation and analysis of non-convex optimization in full Transformer models is extremely difficult, this theoretical framework offers valuable insights into the underlying principles.
Our analysis predicated on core theoretical link regarding the role of spectral properties in learning. We rely on the established principle that favorable spectral properties of matrices involved in optimization
as a more contracted eigenvalue spectrum (tighter bounds) or a smaller spectral norm-coorelate with faster and more stable convergence of learning algorithms. Building upon this, our theoretical framework posits that the rapid and stable convergence of the positional coupling function, driven by these favorable spectral properties, will lead specific phenomenon during training: the localization of this function within a limited set of attention heads in shallow layers, resulting in the empirically observed deposit pattern Boyd Vandenberghe (2004).
Proposition B.1 (RoPE Logit Form): This proposition formally derives the mathematical form of the Rotary Positional Encoding (RoPE) attention logit as presented in the main text (Equation 9), showing how RoPE's rotation mechanism translates into a specific inner product structure involving content and relative positional information. We would prove that
LOOPE complex (Gq. Gq.p + Gp.k Gp) Ge-
13

---

Published as a conference paper at ICLR 2026
We use the fomula in Su et al. (2024) that logit value of and in comples 18
d/2-1 (0)6110))0
q kj (0)
0=
and there are t and 22 that
d/2
d/2-1
4/2-1
Σ
q a ky
<
Σ k=0
0 q
S Σ
1 ky 110,,
O
km
Due to continuity and the mean value theorem, there exists t/-y that
d/2-1
d/2-1
Σ
q 1 ky (0)
((i-)))
Σ k=
qi k
(0 = qikje" (1-
k=
Then we can use e (ft-))0,,_, to generate Ge to finish our proof.
This proposition analyzes and demonstrates the higher degree of freedom and expressive power of the RoPE attention logit form (Equation 9) compared to the non-RoPE additive logit form (Equation 8). It highlights why RoPE's structure is theoretically better suited to capture the complex content-relative positional relationships required by tasks like Task 1.
Proposition B.2 (Logit Expressiveness):
Task 1 is directly related to the following optimization problem:
For - s s d - and constant C1 and C2.
Relative Position Encoding
qi ki +bi-j > Ci C2 Gotsr Kjta +bi-jo
2. RoPE
Re(< Kj > (i--))0(--)) C1 > C2 > Re( qits, kits > ((00)))0--1).
It is easy to verify that the solution space of RoPE is much larger than that of Relative Position Encoding. This can be considered from the following two aspects. First, the expression of RoPE can make the key-value pairs with a fixed distance meet the requirements as long as the angle distribution meets the requirements, while the expression angle distribution because the bias has no significant effect. Second, RoPE actually only needs significant angle to easily meet the conditions, which actually corresponds to the cause of Massive Value described in Jin et al. (2025).
Relative Position Encoding requires more stringent
The emergence of the deposit pattern may share a common theoretical origin: the spectral properties of the attention logit matrix. Gradient- based optimization is known to be more stable for matrices with tightly clustered eigenvalues (Boyd & Vandenberghe, 2004). As suggested by Szegō's theo- rem (Grenander, 1958), the multiplicative interaction in RoPF is theorized to contract the eigenvalue spectrum of the positional component more effectively than additive methods.
Proposition B.3 (Spectral Contraction): This proposition formally proves the spectral property claimed the main text. By applying theorems like Szegö's theorem and inequalities concerning the Hadamard product, it demonstrates that the Toeplitz matrix structure associated with multiplicative coupling (as in RoPE) has a more desirable eigenvalue spectrum (c.g., more compact or tighter bounds) compared to Toeplitz structures associated with additive coupling mechanisms.
The structure of the attention logit matrix L, particularly the properties of its Toeplitz or Toeplitz-lkke components, provides crucial insights into the stability and dynamics of positional information processing. The width of the eigenvalue spectrum often has a great impact on the optimization process.
For Toeplitz matrices, classical results such as Szegö's theorem connect matrix structure to asymptotic eigenvalue distributions. We recall relevant form below:
14

---

Published as a conference paper at ICLR 2026
Theorem B.1 (Szegö's Theorem on Eigenvalue Distribution (From Grenander (1958))), Let (a)
Da
be an N x Toeplitz matrix with (Tv(a))ss - -a= and let a(e**) = Σ
age sk9 be its symbol.
If a(c")) is real-valued and continuous, then the eigenvalues (N) of of (a) asymptotically fill the interval (min a(e"), max max a(e"))) N ->
Applying this theorem and related spectral analysis techniques to the Toeplitz structures induced by various PE mechanisms:
In Non- Non-RoPE methods, the Toeplitz components Ggr, Ar and B contribute additively the logit, and their spectral ranges are determined by their respective coefficient sequences.
In RoPE, the complex logit involves Hadamard product with the Toeplitz matrix Ge. This multiplicative interaction theoretically contracts the eigenvalue range relative additive compositions, resulting in tighter spectral bounds.
We simplify the proposition into the following: Let W, E be N x N positive definite Toeplitz matrices. Then the eigenvalue spectrum of the Hadamard product W E is more compact than W.
Here, each element of a complex number with a membranc length of 1. We can directly use Schur's inequality prove that the membrane length of the eigenvalue of W E must be less than equal to the membrane length of the eigenvalue of W.
However, strict comparison requires citation of Szegö's Theorem B.1 on Eigenvalue Distribution Grenander (1958). So let us calculate the generating function (sign function) corresponding to W E and W respectively.
N-1
Symble(W o.W.oo E)(
Σ
w.e
2Re ws) cos(ii: + i0)
-+=
i=
N Σ
(ym
Σ
wie0(
2Re we (cos(i0)
1=-++1
==0
We will use the following lemma to prove that the range of the above formula must be smaller than that of the following formula, and thus Szegö's Theorem shows that its eigenvalue spectrum is more compact.
Lemma B.1 (Amplitude bound and aligneet criterion). Fix an integer N 2 and non- negative weights wi 2 0. Define
N-1
f(0)
Σ 1=0
2w, cos +0,
+ i0),
g(0)
2w; cos(
R.
Then
max f(0)
max DER
Σ
W/.
OER
Equality holds iff all phase offsets coincide modulo 2m:
0n = 01 ... ON-1 (mod 2T).
Consequently the peak-to-pead range of (width Σ wi) never falls below that of f. and both
ranges coincide only under complete phase alignment.
Proof. Denote
N P(z) = Σ
N Q(z) =
G2 G: :=
i0
Σ
w
|2| =
1=0
1=
15

---

wi. Then at some 0* both previous inequalities are Published as a conference paper at ICLR 2026
Upper bound.
For any |2| = 1 the triangle inequality gives
N-1
N
IP(2) - Σ |ll
Σ 1=0
wy.
==
Hence
N-1 1g(0)/-
[s(0)| 2| S P(
Σ w;
Conversely, [co)(10) implies |(0)] s Σ wi for every while g(0) = Σ
Wy shows that
this bound is attained, so max lg(0) 2 Σ Wü Combining the two yields the desired inequality.
Equality case. tight:
Suppose max
/[0)==2 Σ
0
P(e)0)) )|=0)1=> Σ Wis
RP((" P(())))) )/.
The first equality forces **all terms in the sum for P to share a common phase**, i.c. (((0++i00)) is same for every i; the second forces this common phase to be 0 or which does not affect alignment. Thus
0g i0 = 0u (mod 2x) vi
8, Po (mod 2x) vi.
Conversely, if all 0g are equal, choosing = - 0p gives f(-00) = g(0) = Σ achieved.
wi. so equality is
The lemma follows.
C DETAILED PROOFS FOR THEORETICAL ANALYSIS
This appendix provides the detailed proofs for the propositions and theorems presented in Section
C.I PRELIMINARIES AND NOTATION
We consider Transformer with I. layers and H heads. For batch of N samples, (1uy Js) the token positions for sample 8, and d, = Jss ig is the relative distance. The empirical measure of
N
distances is juN (d) = Σ Hbis - = dd. The attention mechanism is defined by g(h.f)
2=1
g0.10 /dh,, dh, (h,.) softmax,ow (S (h.h))), and (h.)) ((b,), We use the following constant, derived from bounded operator norms and loss gradients:
(th) = |Wolla (A, ||2 sup |00./0yl22
We also denote the sharpness as Xz max (1. ||2 and the minimum anchor attention as
(b)
inf St dd
A L..L) Awyss
C.2 SOFTMAX GRADIENT CALCULUS
Lemma C.1 (Row-softmax Gradient Identity). For any layer 1, head h, sample 8, and row i,
as(h. ig (A (h.1)
Ol, =
(h,3) (h,1)
(h.l)
, 1)
where (1.1) (.
16

---

Published as a conference paper at ICLR 2026
Proof. The result follows from the chain rule. The Jacobian of row-wise softmax is J; diag(Ar As.)) - A.. The upstream gradient is 0e/aa = (ae (Be/aY)V.. Combining these yields the identity.
Lemma C.2 (Quantitative Anchor Gain), j be an anchor column for row i. Let Jj be the softmax Jacobian for row i. For the canonical basis vector Ej we have;
[Pae. J ej- Asg (1 Acgx) and (|P(ey)) J. eerl12 S A15 At.-))
(8)
Proof. A direct computation gives J. Cj. Ci. At, The component along Cj*
is Aeg. (1 - A.gr). The squared norm of the orthogonal component is A
Σ Al,x
S
kデj"
A A2,x)² Al.j ). Taking square roots yields the result.
C.3 PROOF OF PROPOSITION 6.1 (ROPE TOP-LAYER SERD)
Proof. From Lemma C.I, for a sample s with anchor u, and target je:
de, (.
(Aur. Au,..
(L.L)
(h.)) A!!
(9)
By Assumption (A3), the first term A (h,L)
Keds = (Bl,) (0l./Ox(h..)) ( y(h. 2 =n For the second term,
uas
Cauchy-Schwarz and the definition of ((h) yield:
(A Au.:)l S ||Au,,:l2 |l /06./ OY ((hh2) (2 12x1 xL S x(
(10)
Combining these, and using (1.L) 1:
Oe, ( A.... 1 (
2 A (1. (h)
A Ar.j
((b)) XL 2 a (d.))) (da) (ds) c(,)xL. XL.
(11)
Summing over all samples s with d, to get H(())) (d) gives the result:
(d) 2 N (x(()) (1. (d) nh
(1) (d) c( x1.)
(12)
This is strictly positive if the learning signal term a, outweighs the interference term CXL.
C.4 PROOF OF PROPOSITION 6.2 (ALIBI CANCELLATION)
Proof. We provide a constructive proof. Fix a distance bucket d and partition its samples Sa into pairs (s, s'). For each pair, we construct their token embeddings to achieve cancellation.
1. Identical Attention Matrix: By (A3), choose / such that Wol = Wx = 0 and Wv + 0. For base token embedding zig set the anchor token embedding for sample s to be x. Zo + t and for s' to be I = zо f. All other token embeddings are identical for s and s'. Since the query and key projections of are zero, all dot products qi kg are identical for both samples. As ALiBi adds the same bias by. (j i). the scores (.h.L) and attention matrices A (h,L.) are identical for s and s'.
2. Flipped Gradient Signal: Since Wy of O, the value vector at the anchor row u, is flipped: v(h.D) (s') se u,.: (s) 2Wrt. By appropriate choice of xo. this can be made an exact sign flip. This implies that the term A A (s.))(())
A
(). (s).
3. Pairwise Cancellation: Applying Lemma C.1 to the anchor edge js) for both samples, we find their respective gradients are equal and opposite, summing to zero. Repeating this for all pairs makes H/hh (d) = 0.
17

---

Published as a conference paper at ICLR 2026
C.5 PROOF OF THEOREM 6.1 (EXPONENTIAL AMPLIFICATION)
Let U/ be the subspace spanned by gradient directions corresponding to the seed-positive buckets. Let
gX be the gradient vector at layer I, so gd) = J()TT g g(++1)
Lemma C.3 (Layerwise Directional Advantage). Under the assumptions. there exists Bi 2 0 such that for any u + O,
JUJ ull2
04 3t)
2
(13)
P ull2 1
The gain & is lower-bounded by term proportional to the summed strength of the positive seeds from Proposition 6.
Proof. The Jacobian J0) is decomposed into the within-attention part Ol)) and the rest of the block R(), By (AI), the R(() part contributes the factor (1 oz)/(1 + ot). The gain 3r arises from O Lemma C.2 establishes directional gain for the anchor component within the softmax Jacobian.
When composed with the value projection and the upstream gradient from (A2), the anchor direction
receives coherent signal. Aggregating over the anchor rows in subspace U yields net directional advantage, which extends from the basis vectors of 4 to any vector u by a convexity argument. n
Proof of Theorem 6.1. Applying Lemma C.3 to the backpropagation recurrence n0hh = J()T ((+1;
[() g(+1)) ||2
2 [Pag
SNR, 1 g(++1) 2|
Pu+=g(+D11> " SNR++1
(14)
Iterating this inequality from layer L down to yields the exponential product. The same multiplicative logic applies the vector components corresponding to the top two heads within the subspace 4, proving margin amplification.
C.6 REMARK: CONCEPTUAL LINK TO CONVOLUTION KERNELS
It is helpful conceptually frame the aggregated gradient H. (d) discrete convolution. If define the empirical distribution of distances as a signal fuN, and the expected gradient contribution for a given relative distance A as "deposit kernel" x (h) (A), then the total aggregated gradient is their convolution: (1,,)) ( ())
From this perspective, our proof demonstrates a key difference in the structure of these implicit kernels:
For RoPE, the kernel x. (A) has a trigonometric structure due to the rotational mechanism. This structure is analogous to positive-definite kernel, which resists being driven to and ensures a positive "seed" is deposited.
For ALiBi, the kernel x. (²)) (A) is merely affine (linear plus a constant). This simple struc- ture allows for exact cancellation when convolved with symmetric distance distribution, explaining why no seed is guaranteed.
This intuitive framing aligns with the rigorous proof and reinforces the conclusion that the deposit pattern is an inherent property of RoPE's 'ss multiplicative design.
D SUPPORT EXPERIMENT
D.1 EXPERIMENT SETUP
All experiments were performed on a single NVIDIA RTX 4090.
The experimental parameter settings of non-MLA are shown in Table and the experimental parameter settings of MLA are shown in Table 4.
Remark D.I. Note that all our experiments used dropout, so the deposit pattern is not phenomenon that can be improved by ordinary dropout.
Since Task 2 did not have a position effect, subsequent experiments were all on Task 1.
18

---

Published as a conference paper at ICLR 2026
Setting
Value
Vocabulary size (vocab__ss _size)
574
Model dimension (dmodel)
256
Feed-forward dimension (dz)
512
Number of attention heads (nur_heads)
16
Number of decoder layers (rum_
6
Dropout rate
0.1
Maximum sequence length Positional encoding types tested
128
nope, abso Abbsolut,,, al ibi, relative, random, rope 100 (Task 1),150 (Task 2)
Number of epochs
Batch size
512 or 1024
Optimizer
AdamW (Ir=le-4, betas=(0.98, 0.9), weight decay__ecay=le-5) Cosine schedule with warm-up (6% of total steps) Cross-entropy 70% 30%
Learning-rate scheduler
Loss function
Train/Test split
Random seed
0,42,70,113.130
Table 3: Hyperparameters and settings (PE).
Setting
Value
Vocabulary size (vocab_st size)
574
Model hidden dimension (dmodel)
256
Feed-forward dimension (dg)
512
Number of logical MLA heads (num heads)
16 (8 in fact)
Compression dimension (dcompress)
128
Number of decoder layers (rum_layers)
6
Dropout rate
0.1
Maximum sequence length (max_len)
128
Positional embedding
Rotary (RoPE)
Number of epochs
100 (Task 1); 150 (Task 2)
Batch size Optimizer
1024
AdamW W (Ir=le-4, betas=(0.95) 0.9). weight_decay-ee Cosine warm-up (6% of total steps)
Learning-rate scheduler
Loss function
Cross-entropy 70% 30%
Train/Test split
Random seed
0,42,70.113.21
Table 4: Hyperparameters and settings (MLA).
D.2 COMPLETE VISUALIZATION LAYER-BY-LAYER HEAD ABLATION
For completeness, we provide the full layer-by-layer head ablation results for the RoPE models trained Task 1. These figures expand upon the summary violin plots in Fig. 4, showing the per-head accuracy after zeroing out each attention head at every layer.
How read the plots. Each subplot corresponds one Transformer layer, and each point plots the test accuracy after ablating single head. Large vertical drops indicate heads whose removal severely harms performance, revealing where positional reasoning concentrated. Flat curves indicate heads that contribute minimally and whose ablation produces no measurable degradation.
6-layer RoPE model. Figure 9 shows the full ablation for the 6-layer model. The first layer contains single dominant head whose ablation causes large accuracy collapse, while all remaining heads exhibit near-flat profiles across layers. This fine- iin-nnei visualization matches the condensed "deposit pattern" presented in the main text.
19

---

Published as a conference paper at ICLR 2026
Layer
Layer
Layer
Layer
Layer
Baseline (0.967)
Figure 9: Full head-ablation curves for the 6-layer RoPE model on Task Each subplot shows the test accuracy after ablating single attention head in the corresponding layer. A single head in Layer produces a catastrophic accuracy drop, while all other heads across all layers induce only minor changes. This confirms that RoPE concentrates positional reasoning into early-layer head, even in shallower architectures.
8-layer RoPE model. Figure 10 reports the same analysis for an 8-layer RoPE model. Despite the increased depth, the pattern qualitatively identical: single early-layer head carries almost all positional reasoning, and deeper layers show only small, noise-level fluctuations across heads. This demonstrates that deposit pattern is not sensitive to depth and remains stable across architectures.
Summary. These complete ablation maps confirm that the deposit pattern a robust and highly localized effect: RoPE consistently routes positional reasoning into head in the earliest layers, with all other heads acting effectively position-agnostic for this task.
D.3 EXPERIMENTS OF HEAD ABLATION ON OTHER PE METHODS
In this subsection, and Relative PE.
violin plots show the ablation experiment results of Alibi, Absolute PE,
In Figure 1. find that the position information of relative position encoding and ALiBi are scattered, while the absolute position encoding has some significant heads and some insignificant heads in the shallow layer, and the whole is oscillating.
20

---

Published as a conference paper at ICLR 2026
Layer 1
Layer
Layer
Layer
200
0.8
0.8
Accurary
0.7
0.7
0.8
0.6
0.6
0.6
0.5
0.5
0.5
0.5
0.4
0.4
0.4
0.4
9
31
15
7
9
15
5 7
33 15
11
Head Index
Head Index
Heed Index
Head Index
Layer
Layer
Layer 7
Layer
1.0
1.0
1.0
W
0.0
0.5
6.5
0.5
0.4
0.4
0.4
7
9
31
13
15
5
7
11
13
15
1
3
0
11
13
15
7 9
31
13
35
Head Index
Head Index
Head Index
Head Index
Baseline (0.975)
Figure 10: Full head-ablation curves for the 8-layer RoPE model on Task 1. The same localization pattern reappears larger depth: unique critical head in Layer 1 causes large accuracy collapse, and all remaining heads across Layers 2-8 have negligible impact. The persistence of this topology across depth demonstrates the stability and robustness of the RoPE deposit pattern.
The similarities among NoPE, Relative PE, and ALiBi are consistent with previous studies on how NoPE obtains position information through causal masks Kazemnejad et al. (2023); Haviv et al. (2022).
D.4 SOME EXPERIMENTS ON PARTIAL ROPE
is foreseeable that MLA can fully alleviate the deposit pattern at the expense of a little generalization ability. We also conducted some ablation experiments in the experiments of RoPE for some heads and NoPE for some heads. Interestingly, we found that the most significant deposit pattern is not necessarily in the first layer, but may be in the second or third layer. In some cases, the first layer does not even have deposit pattern. We can imagine that this is the effect of using NoPE attention
21

---

Published as a conference paper at ICLR 2026
1.0
Abb. Took 1-Lowe1
0.0
Ascolute PE Task Layer
Accuracy
VWW
0.6
Alibi, Task 1 Reletive PE, Task Baselines
0.5
0.4
Logo
0.2
5
Head
Layer
Figure 11: Ablation experiments other PE methods.
heads in deeper layers. This inspires us that the way the FFN affects the position encoding is far more complicated than we imagined. We give the partial ablation line graph the first heads doing RoPE layer below.
0.40
0.
1.00
Figure 12: h: 2, 1,2.
0.2
0.55
Figure 13: k = 3, 1,2.
D.5
WHY POSITION EMBEDDING COMPLEX?
We conclude this section by explaining why understanding positional encodings is difficult and almost impossible decouple.
We first give our conclusion that the position component the token comes from manifold embedding rather than subspace decomposition, which means that extracting the position component requires us
22

---

Published as a conference paper at ICLR 2026
Layoted
037
0.5
2.0
0.00
0.75
2.13
0000 5000
1000
Figure 14: k = 5, 1,3.
t-d Laperv llad
cnd Layen- Acceracy
05.25
0.78
Figure 15: = 6, 1,2.
to have an algorithm to solve the changes in the position manifold. In an extremely ideal case, we can use the Devinson-Durbin algorithm in signal processing to recursively solve through the value of the Toeplitz matrix. However, extracting Toeplitz matrix from general matrix is not unique, and we cannot determine whether the extracted Toeplitz matrix truly represents the position information.
This conclusion comes from very simple observation: the position embedding of absolute position encoding is not orthogonal to word embedding. Therefore, there is no subspace decomposition that can completely separate the two components.
we denote as continuous embedding from S1 & SI.. & S1 R and g as a continuous embedding from R,, to S1 @ S1... @ 51, then the essence of absolute position encoding is actually f()(1). In other words, the explicit position encoding that we can understand essentially manifold embedding.
The general method of dealing with manifold embeddings requires a specific embedding mapping. For general positional encodings, this mapping is not solvable, so it is almost impossible to decouple the positional encoding.
E RELATIONSHIP TO LENGTH GENERATION
Here, we give a more detailed qualitative statement to describe our thinking on the relationship between deposit patterns and length generation.
Why the deposit pattern reflects a training bias rather than modularity. Although the deposit pattern superficially resembles a form of head-level modularity, our evidence suggests that it is instead a bhining-inducce bias specific to RoPE's multiplicative structure. First, multiplicative PE suppresses activation-lewe positional directions (Section 5.4), causing carly-layer heads to compete for the positional signal. Once single head captures the content-posiin interaction, RoPE's s strong inductive bias quickly eliminates positional gradients for the other heads, preventing them from
23

---

Published as a conference paper at ICLR 2026
0.55
039
8.5
10
0.5
Figure 16:
13, / = 1,2.
Lone Hoad
Lopord
03
Figure 17: 15, 2,4.
acquiring similar capabilities. This is fundamentally different from deliberate modularization: the specialization emerges not because the model decomposes the task across heads, but because the training dynamics collapse positional reasoning into the earliest head that happens to align with the RoPE kernel.
Second, the deposit head does not learn a general positional module; it learns a narrow set of frequency bands that are amplified by RoPE's rotation mechanism. As noted in Ermo al. (2025), only a subset of frequencies is sufficiently trained, and their influence is later propagated through feed-forward mixing. This leads to a compressed, low-rank representation of positional structure, which is efficient for the training distribution but fragile outside it.
Taken together, the deposit pattern better interpreted as training bias toward early collapse of positional responsibiliry, rather than evidence for stable or interpretable modular decomposition of positional reasoning. RoPE's multiplicative interaction creates a "winner-takes-all" dynamic, where one head monopolizes positional information not because the architecture encourages modularity, but because the optimization process and frequency structure favor such collapse.
Further evidence from tasks beyond pairwise position reasoning. RoPE enables efficient pair- wise position content coupling but does not equip the model with the compositional machinery needed for whether the deposit head learned by RoPE represents reusable "positional module," we designed family of tasks requiring the model to combine multiple pairwise distances (e.g., predicting the sum of several trigger- distances). Despite the task being a straightforward extension of our synthetic setup, all RoPE-based models failed to generalize beyond the training length, even when the training accuracy was near-perfect. In contrast to the pairwise distance task- task-where positional responsibility collapses into single head the multi- multi-trigger task requires the model compose several independent positional relations, a capability that is architecturally challenging for standard Transformers.
tasks that require combining several positional relations. To test
24

---

Published as a conference paper at ICLR 2026
This observation aligns with prior findings on tasks such as Dyck languages, context-free composition, and stack-like dependencies (e.g. Chi et al. (2023)): Transformers struggle when a task requires manipulating multiple positional relations simultaneously. The failure
retrieving, combining, of RoPE to generalize in our multi-trigger setting therefore does not indicate a lack of training or capacity, but rather reveals that the single-head deposit behavior does not constitute modular or compositional positional representation. Instead, the deposit head encodes a narrow, task-specific mapping tied to the training distribution, and-unlike true module-ii it cannot be recombined or composed when the task demands multiple positional deductions.
Taken together, these results reinforce that the deposit pattern reflects training- collapse of positional responsibility rather than the emergence of reusable module. RoPE enables efficient pairwise position- content coupling but does not equip the model with the compositional machinery needed for length-generalizing tasks that require combining several positional relations.
More than one study has shown that transformer's understanding of position information starts from the first layer Zuo et al. (2025); Chi et al. (2023); Kazemnejad et al. (2023). The key to successful length generalization is that responses to relative position information within the training length can be naturally transferred to a longer length (test length). Therefore, the lower the coupling between location and content, the better the scalability. If the coupling degree is high, it is necessary to learn the coupling mode. At the same time, Deposit Patterns shows that only some frequency bands are fully trained during training like Ermo et al. (2025). These frequency bands are passed through the fully connected layer, spreading the influence to most of the frequency bands in the deep layer, which means that in fact only specific relative position differences are considered by the training process, so the length generalization ability is drastically affected.
Relationship with massive value in RoPE. Recent empirical studies have reported that Trans- formers trained with RoPE often develop small number of extremely high-norm rows Wq and Wk (Jin et al., 2025), behavior not observed under additive positional encodings. Although this phenomenon is not the focus of our work, the Toeplitz formulation naturally explains its underlying mechanism.
RoPE mixes content and position multiplicatively through a frequency-indent indexed Toeplitz kernel Ge- In our formulation, each logit entry takes the form
(g,, ky)REP (gi, ky) (Ge)i-j-
where (Ge) is a complex rotation encoding relative displacement. Importantly, each frequency band of RoPE corresponds to fixed Toeplitz diagonal, and gradients flowing through that diagonal are accumulated entirely on the rows of Wo, Wx associated with the same frequency.
This creates "frequency bottleneck":
if particular diagonal of the Toeplitz kernel becomes predictive early in training, then its associated frequency bands receive disproportionately large gradient flow,
causing the corresponding rows of Wo. Wx to grow rapidly in norm,
while other frequencies remain under-trained.
Thus the massive-value rows arise direct consequence of gradient concentration onto small subset of Toeplitz diagonals, not from instability of the optimization process. Additive PEs do not exhibit this effect because their position signal enters only through additive py-based cross-terms, so no lequuency-indexen diagonals receive concentrated multiplicative amplification.
Under our unified framework, the massive-vale valuue phenomenon is therefore a natural by-product of RoPE's multiplicative structure, in which positional information is stored directly in selected frequency bands of Wo, Wx rather than in explicit activation-level positional vectors.
25


Reviews:
[{"review_id": "KE0SbyfDt5", "reviewer": "Reviewer_hmEf", "summary": "The paper presents a novel analytical and empirical framework for understanding positional encodings (PEs) in Transformers, with a focus on the difference between additive and multiplicative (e.g., RoPE) mechanisms. The authors use a Toeplitz-matrix formulation to characterize translation invariance in positional interactions, and both predict and empirically demonstrate that multiplicative PEs (especially RoPE) produce a distinctive single-head deposit pattern, where positional reasoning becomes localized to a single head in early layers.\n\nTo explore this, the paper introduces two synthetic tasks—a position-sensitive task (relative distance classification) and a position-agnostic task (trigger counting)—and performs causal ablation studies to confirm that RoPE’s multiplicative nature induces strong specialization. The authors further provide theoretical proofs (based on gradient dynamics and spectral analysis) showing that this phenomenon arises inevitably from multiplicative coupling, and argue that DeepSeek's MLA (Multi-Latent Attention) architecture mitigates over-specialization.\n\nThis is a very good paper that would be probably be suitable for ICLR in its current form. However, I do have some questions for the authors for my own understanding. Additionally, there are several points that could be improved to make this a strong contender for spotlight or oral presentation.", "questions": "- Why weren't learned additive PEs analyzed? Is this because they would not meet assumption 3.2 of Toeplitz structure? \n- Relatedly, I was reading the introduction, I thought the author's might analyze the validity of the Toeplitz structure of positional interactions in models where it is both baked in (baseline), and models where it must be learned. Could the authors discuss this, and if relevant, add some additional analyses in this vein? \n- The authors state that Relative PE learns to attenuate its bias towards zero in Task 2. Can they report empirical numbers to demonstrate this? \n- Section 5.3: why is the U included in the notation for weight matrices? it seems to add nothing? \n- Are MLA and RoPE parameter-matched in 5.3? If not, can you perform an additional parameter-matched experiment to convincingly demonstrate the benefit for MLA? \n- Proposition 6.2: would it be possible to assess the initial advantage empirically and thereby predict the head that develops the deposit pattern?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Relevant topic with a clear gap in the literature, whereby the differences across different strategies for positional encoding are poorly understood, making application to new tasks ad-hoc and reliant upon hyperparameter tuning, rather than theoretical insight. This paper aims to provide theoretical insight into a very clear and directly stated question: \"how, precisely, do different PE schemes mediate the interaction between token content and position). The answers are interesting in their own right and may be useful for ML practitioners moving forward. \n- The paper is very clearly written, integrating both theoretical analysis and careful synthetic experimental analyses, along with appropriate ablations. The structure of the paper is easy to follow and guides the reader along a compelling journey. \n- The framework via toeplitz matrices is clever and appreciated. \n- The author's conclusions are strongly supported by their analyses, in nearly every case (see minor exceptions below). Preliminary answers to the motivating questions are provided. Clear follow-ups to this paper can be executed in future work.", "weaknesses": "- It wasn't clear to me why the author's posit that \"intense structurally-induced specialization is a primary cause of the gap between RoPE's theoretical pmroise and its practical performance.\" This theme comes up many times (e.g. line 387 \"the deposit pattern, while effective, is an inefficient use of model capacity\"), and I never felt the author's fully explained their rationale. Modularity is often praised for its benefits, in contrast to this position. \n- Some aspects of prior literature were explained so coarsely to have impossible for an unfamiliar reader to follow, e.g., the mention of \"massive value\" pehenomena, rotation artifacts, T5, AliBi, etc. With the additional page in the revised version, I suggest expanding relevant descriptions.\n- Assumption 3.1 (linear separability of content and position components) is questionable. The authors motivate it only with absolute PE and NoPE, but do not discuss its validity for multiplicative mechanisms, which seems more fraught. \n- For Task 1, it wasn't clear why a classification approach was used vs. the more natural regression framing. \n- 4.2: the authors suggest that their framework predicts that RoPe will exceed in the position sensitive task. This wasn't immediately obvious to me. Can the authors formalize their predictions for different models based on the framework?\n- Figure 4, and other related plots: I don't think the violin plots are the right choice of visualization, personally. The point is not about the shape of the distribution (which is quite small for a violin plot anyways, at 16 samples), but rather, the presence of a particular outlier. I would prefer to see individual data points, possibly in addition to the violin plots already plotted. This is currently plotted for the first layer on the left. However, the recommended approach would do this for each layer and remove the need for the plot on left. In any case, the plot on left would be more effectively visualized with heads sorted by accuracy, since the x-axis is meaningless currently. \n- The authors claim that Figure 6 validates their prediction of only 1 head being necessary. Clearly, it is 2 heads. This may be because this is applied across all layers, where the single-head deposit pattern is restricted to earlier layers, but this should be explained more accurately. \n- Unclear whether the benefit for MLA is due to larger parameter count (three matrices in Q,K projections vs. two in RoPE), or the hybrid approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1cTd4Na1XP", "reviewer": "Reviewer_TXbo", "summary": "Position Encoding (PE) plays a critical role in LLMs, different PE methods have been proposed, fixed or learnable, additive or multiplicative. However, the understanding of those PE methods are still under exploration, and their efficacies are normally tested on aggregate metrics. In this study, a new framework is proposed to analyze PE more systematically.\n\nTwo simple and isolated tasks are proposed, position-sensitive and position-agnostic tasks. These two tasks can better test if PE is needed or which PE methods can deliver a better result. Based on the tasks, this study tries to analyze the key reason behind those PE methods using Toeplitz matrices. This Toeplitz framework can group those PE methods into additive and multiplicative, showing the theory behind why multiplicative PE can better inject position information. Finally, this study also shows only a few heads focus on the multiplicative PE, denoted as the deposit pattern.\n\nGenerally, this study proposes a new framework to study and compare the previous PE methods. A new finding is presented on the multiplicative PE, RoPE.", "questions": "Following the weakness of clarity, here is a list of questions regarding the paper:\n1. Assumption 3.1 and in Fig 1, x = c + p, should I consider this a conceptual idea that the input can be disentangled into content and position so that we can study separately? Or this is a more strict formula that the Topelitz analysis is based on. For the latter case, does the Gram(c+p) equation also apply to ALibi, where the merge happens after taking the matrix product? (I might miss the proof for this part in the literature.)\n2. Based on the experiment and the conclusion, the proposed Toeplitz-based framework can distinguish how additive and multiplicative PEs work. It does not matter if the additive is fixed or learned, absolute or relative, they all can be summarized by Eq(1). (This is my understanding, please correct me if I am wrong). If my understanding is correct, the introduction and Sec3.1 could be a little confusing. It seems this study tries to cover or touch every aspect of PE, but it is less clear what the proposed Toeplitz framework actually solves in the context. Based on Fig 1, it seems RoPE and Relative PE are the two components for comparison, but my understanding from the experiment is the two modes should be “Additive” vs “Multiplicative”(RoPE)? And for additive PE, the mode includes both relative and absolute. If my understanding is right, it could be better to re-organize the figure and Sec3.1 for clarity.\n\n3. Following the above question, would this Toeplitz framework categorize “AliBi” to additive or an exception? Based on Fig 1, “AliBi” seems an exception to the “Relative PE”(additive to me), (Eq1 cannot explain “AliBi”??), then how is “AliBi” processed or captured by the framework? I can find a bunch of experiments for “AliBI”, but I cannot find a detailed explanation of “AliBi” based on the proposed framework. This question could be related to Q1, handling the addition before or after Gram computation.\n\n4. A naive question for Eq(1), G(qp, kp) is always Toeplitz for all additive PEs?(did I miss border cases?), what is the main purpose of the explicit bias term B?\n\n5. The de-couping framework, Eq(1) and Eq(2), demonstrates how a PE is involved with the content. Please help me understand how Toeplitz matrices play a key role in the context?\n\n6. It is interesting to see that how “Alibi” work on the position-sensitive task, Fig 2, line 269, “Methods like ALiBi, with its fixed and data-agnostic Toeplitz bias”, Is the reason behind “AliBi” due to fixed and data agnostic? Does the “absolute PE” also inherit those properties? \n\n7. Regarding “SINGLE-HEAD DEPOSIT””, this seemed an important finding by the framework. However, this finding could be over extrapolated. Given the position-sensitive task can be inferred mainly on position information, and RoPE can better inject this discriminant feature(position information) based on Fig 2. Then the explored question in Sec 4.3 is essentially finding, if there is an attention head that can capture the injected feature(position information). Again, the injected feature is a discriminant shortcut for the model to reach the correct prediction. And it is not surprising that different heads could capture different input features, and networks tend to lazily rely on easy features. Therefore, NoPE on Task 1 does not offer that feature to capture, and the feature from RoPE becomes not discriminant for Task 2 in Fig 2. This explanation could be a bummer to the finding, or please correct me if the above understanding is wrong.\n\nAlthough many points remain unclear in the draft, I still appreciate the effort the authors have made and I have learned something new from this study.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PE is used in almost all LLMs (except NoPE variants), therefore this is an important topic to analyze for downstream studies. \n\n2. The framework proposed in this study can be used to investigate those widely used PE methods. Thus the finding in this paper is applicable to a wide range of LLM studies.\n\n3. The tasks used to analyze PE are simple, and the simplicity is also very important for XAI. The proposed position-dependent and position-agnostic tasks can directly isolate the effects of each PE; Thus the tasks could be a more clear metric than those aggregated ones.\n\n4. It seems technically sound using Toeplitz matrices to distinguish the additive and the multiplicative PEs. This framework can theoretically analyze the differences between the two types of PEs.\n5. A series of ablation studies are included to further explore the new finding on the multiplicative PE, RoPE.\n\nThere exist XAI studies which combine different metric tasks and analysis steps to investigate a phenomenon. However, they often lead to less clear conclusions when stitching something without a good reason. The greatest strength of the proposed framework is the simplicity, the tasks are directly designed on position information.", "weaknesses": "Overall this study seems novel and interesting. However, the presentation of the current draft could be improved. This study aims to organize multiple things, absolute PE, Relative PE, RoPE, Alibi, fixed PE, learnable PE, additive PE, multiplicative PE. The current presentation seems not clear enough to organize all of those, especially in Fig 1. It took me several “hops” to combine those pieces, and what Toeplitz matrices are used for, but I could be wrong or miss something. \n\nAnother weakness is the explanation behind those PEs based on the two tasks. The discussion on the reason seems less solid. I will list all of my questions in detail below.\n\nDespite the weaknesses, I still believe this study is intriguing, opening a new door to this topic. And I am not expecting one study to explore the whole universe behind the door.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oIxPJrWnL", "reviewer": "Reviewer_7AJz", "summary": "This paper presents a unified Toeplitz-based framework for analyzing how positional encoding schemes influence the composition of the attention matrix in Transformers, distinguishing the distinct effects of additive and multiplicative mechanisms. It shows that multiplicative encodings, such as RoPE, modulate the QK interactions between content and positional factors of model representations through a unified multiplicative term that incorporates relative positional information, thereby inducing stronger content–position interactions. Synthetic tasks reveal that RoPE excels at position-sensitive reasoning but is outperformed by certain additive embedding methods in tasks where positional information is inconsequential. Furthermore, RoPE is found to develop a “single-head deposit pattern,” in which one attention head monopolizes positional computation. Through causal ablations and hybrid designs, the authors demonstrate that this specialization is an intrinsic property of RoPE’s multiplicative structure and relate it to redundant attention heads and new attention designs such as multi-latent attention.", "questions": "1. Could you explain how to deduce from the Toeplitz formulation of the RoPE attention matrix in (2) that “by routing the\nlearning of positional dependencies through this multiplicative kernel, RoPE may create a strong\ninductive bias that encourages positional specialization, concentrating this logic into a small subset\nof attention heads”? For me, it seems impossible to conclude that the $G_{e}$ term will be subsumed by a subset of heads *a priori* solely from the form of (2).\n\n2. Could you explain the surprisingly strong performance of absolute PE in your two synthetic experiments? While they are all additive PE methods that should behave similarly according to your Toeplitz formulation, in Task 1 it significantly outperforms relative PE, and in Task 2 it significantly outperforms ALiBi. This appears counterintuitive to me, since absolute PE is as rigid and inflexible as ALiBi, given that both are not learnable.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides clear insights by formalizing the mechanistic distinction between additive and multiplicative approaches to incorporating and modulating interactions between positional and content information in model representations, particularly in relation to relative positional encoding. Its controlled experimental settings effectively demonstrate the various idiosyncrasies of RoPE as consequences of its multiplicative interaction mechanism, most notably the single-head deposit phenomenon. The experiments in Section 5 examining variations of this behavior under different input and structural perturbations are especially thought-provoking, aligning closely with the authors’ intuitive hypotheses and theoretical characterizations. Together, these results convincingly illustrate how RoPE’s multiplicative structure both enables strong positional reasoning and induces positional over-specialization.\n\n2. The paper further strengthens its empirical observations with a theoretically rigorous proof that explains why positional specialization emerges (at least during the early stages of training). This theoretical grounding provides valuable causal insight, transforming the observed empirical regularity into a predictable and interpretable property of RoPE’s multiplicative dynamics.", "weaknesses": "1. The second synthetic task the authors design (trigger word counting) seems rather off the point. It essentially tests to what extent different position encoding methods can suppress themselves and perform the “no-op” behavior, which is not related to the authors' characterization of the particular attention matrix component to which the relative positional information is injected via the Toeplitz formulation. From my perspective, the second task should be content-agnostic instead of position-agnostic (or more specifically, it should be agnostic to the interplay of content and relative position), which would allow for a more interesting comparison between the additive and multiplicative PE methods.\n\n2. The setting of the paper is rather limited. While it claims to “deconstruct positional information,” it actually only focuses on how multiplicative PE methods better express the interplay between relative positional information and content and shows that training on the highly specialized relative distance classification task, which exactly captures this correlation, would lead to the single-head deposit phenomenon. It is highly debatable whether such tasks exist in the pretraining corpus of LLMs at all, and the paper also does not discuss the settings of training on multiple tasks (not even the two synthetic tasks proposed), which is the usual practice of training LLMs. Furthermore, the paper does not discuss the mechanistic significance of ROPE and other PE methods in accounting for some truly intriguing LLM phenomena that concern the joint utilization of relative positional information and semantic content in more generalized settings. Notable examples include in-context learning (particularly recency bias, which means that the last demonstration's label is more likely to be predicted for the query), chain-of-thought, and in-context information retrieval (needle-in-a-haystack and such). The lack of discussion of the implications of the experimental findings for characterizing those LLM behaviors limits the significance of the findings.\n\n3. The citation format used in the paper appears problematic in various places, where the author names and publication years of the cited papers appear side by side with the main text without being enclosed in parentheses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J92SJe60pA", "reviewer": "Reviewer_qxxu", "summary": "This work offers a unified framework on how positional signals are represented and processed in transformers using different positional encoding methods. The proposed framework classifies a range of methods by the way in which positional information interact with content information and contribute to attention logits. The relative merit and failures of the methods are shown through two synthetic tasks. The authors also find that RoPE results in high head specialization in positional information , and present detailed and theoretical analysis to understand this phenomenon.", "questions": "I'd be curious to hear the authors intuition on whether such strong concentration of positional processing in RoPE would show up in larger models trained to model language. Since in language modeling, supposedly models are trained under much more diverse objectives, would there be more competition for a very local representation for position?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper offers a set of important intuitions on the mechanisms and relative advantages between different positional encoding methods.\n- The discovery of the strong head specialization in RoPE is novel and interesting, and the accompanied analyses on the factors leading to this phenomenon are quite illuminating.", "weaknesses": "- As the empirical experiments are performed in a synthetic setting, whether similar results generalize to a language modeling setting remains unclear. The paper would be greatly enhanced by some ablation analysis on whether similar head specialization are observed in transformers trained under language modeling with RoPE (e.g. changes in on some position-sensitive tasks), or at least, some discussions on the implications of these results for larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLW2l6rnIT", "reviewer": "Reviewer_sR2j", "summary": "This paper aims to understand the role of positional encodings by both empirical investigations and a theoretical framework based on Toeplitz modeling. Particularly, the paper contrasts additive and multiplicative encodings, using experiments and analysis to illustrate their distinct effects on model accuracy across different synthetic tasks.", "questions": "Please see the section Weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is very well written and organized, which has a clear motivation.\n2. The problem studied in this paper--the role of positional encodings--is of fundamental importance in the field, and this paper does a good job of filling the gap between practical success and limited theoretical understandings. In addition, this paper leverages intuitive numerical examples to illustrate theoretical results and conjectures, which makes this paper very easy to follow.\n3. The “single-head deposit pattern” is an interesting and reproducible phenomenon, providing a plausible explanation for the gap between RoPE’s theoretical advantages and its inconsistent empirical performance.", "weaknesses": "1. [Assumption 1] This assumption--the additive decomposition of each token representation--seems a bit strong, as it rules out any interaction between content and position components. Although there is a remark that connects this assumption to existing literature, I was wondering if this assumption can be (approximately) justified via experimental/simulated results. It would be very helpful if the authors can elaborate more on this part.\n\n\n2. Overall, the writing in Section 3.2 is a bit rushy, and similar to the previous question, the remark below Assumption 2 is also not detailed and it would be better to elaborate on how sinusoidal absolute PE meets this assumption and why it could be (approximately) representative for a certain class of PEs.\nIn addition, it would be easier to understand the role of assumptions if there could be a simplified/informal result in Appendix B on how RoPE can be converted to Toeplitz matrices under Assumptions 1-2.\n\n3. Theorem 6.1 holds under Assumptions (A1-3), and I was wondering if Theorem 6.1 is novel beyond showing exponential amplification under generic Jacobian stability assumptions. Additionally, I am curious if the desired property of RoPE under these Assumptions would suggest new designs of PEs.\n\n4. Can the deposit pattern or its variants also be observed in other tasks? Also, based on the analysis in this paper on the favorable scenarios for RoPE, are there any intuitions behind which scenarios would, instead, be favorable for additive PEs, which would be very helpful to add to the discussion of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)