Decoupling Positional and Symbolic Attention in Transformers
Authors: Felipe Urrutia
Jorge Salas
Alexander Kozachinskiy
Cristian Buc Calderon
Hector Pasten
Cristobal Rojas
Venue: ICLR 2026 Poster (2026)
Keywords: Transformers architecture
positional encodings
Transformers theory
large language models
Primary area: foundation or frontier models, including LLMs

Abstract: An important aspect subtending language understanding and production is the ability to independently encode positional and symbolic information of the words within a sentence. In Transformers, positional information is typically encoded using Positional Encodings (PEs). One such popular PE, namely Rotary PE (RoPE), has been widely used due to its empirical success. 
Recently, it has been argued that part of RoPE's success emerges from its ability to encode robust positional and semantic information using large and small frequencies, respectively. In this work, we perform a deeper dive into the positional versus symbolic dichotomy of attention heads behavior, both at the theoretical and empirical level. We provide general  definitions of what it means for a head to behave positionally or symbolically, prove that these are two mutually exclusive behaviors and develop a metric to quantify them.  
We apply our framework to analyze Transformer-based LLMs using RoPE and find that all heads exhibit a strong correspondence between behavior and frequency use.  
Finally, we introduce canonical tasks designed to be either purely positional or symbolic, and demonstrate that the Transformer performance causally relates to the ability of attention heads to leverage the appropriate frequencies. In particular, we show that we can control the Transformer performance by controlling which frequencies the attention heads can access. Altogether, our work provides a detailed understanding of RoPE, and how its properties relate to model behavior.

Published as a conference paper at ICLR 2026
DECOUPLING POSITIONAL AND SYMBOLIC ATTEN- TION BEHAVIOR IN TRANSFORMERS
Felipe Urrutia
Jorge Salas
CENIA & Faculty of Mathematics UC
CENIA
Santiago, Chile
Santiago, Chile
jorge. www.s@cenia.......oooomo.......m
Alexander Kozachinskiy
Cristian Bue Calderon
CENIA
CENIA
Santiago, Chile
Santiago, Chile cristian wtw.stii.nuucccnnaaa... buc@cenia.cl
bexxxnder.kozachinseee www..kccsssssyyiceeniiiaaaa alexander
Hector Pasten
Cristobal Rojas
CENIA & Faculty of Mathematics UC
IMC UC & CENIA
Santiago, Chile
Santiago, Chile luis rojaseuc.cl
hector.pasteee pawtrrrpttteeeucecll.....
ABSTRACT
An important aspect subtending language understanding and production the ability to independently encode positional and symbolic information of the words within sentence. In Transformers, positional information is typically encoded us- ing Positional Encodings (PEs). One such popular PE, namely Rotary PE (RoPE), has been widely used due to its empirical success. Recently, it has been argued that part of RoPE's success emerges from its ability encode robust positional and semantic information using large and small frequencies, respectively. In this work, we perform a deeper dive into the positional versus symbolic dichotomy of attention heads behavior, both at the theoretical and empirical level. We provide general definitions of what it means for a head to behave positionally or sym- bolically, prove that these metric to quantify them. We apply our framework to analyze Transformer- based LLMs using RoPE and find that all heads exhibit strong correspondence be- tween behavior and frequency use. Finally, we introduce canonical tasks designed be either purely positional or symbolic, and demonstrate that the Transformer performance causally relates to the ability of attention heads to leverage the ap- propriate frequencies. particular, we show that we can control the Transformer performance by controlling which frequencies the attention heads can access. Al- together, our work provides detailed understanding of RoPE, and how prop- erties relate to model behavior.
two mutually exclusive behaviors and develop a
INTRODUCTION
Positional Encodings (PEs) play a critical role in how modern Transformer-basee Large Language Models (LLMs) function. Such criticality is, for example, illustrated by the fact that encoder-only Transformers with No Positional Encoding (NoPH) are invariant under permutations of prompt's tokens (Pérez al.) 202I), severely limiting their expressivity. While decoder-only Transformers using NoPE can in theory recover positional information (Kazemnejad et al.) (023).. they are unable to learn some matrices observed in practice (Barbero et al. (022).. Regardless, at the practical scale, adding a component to explicitly inject positional information is the current standard choice state- of-the-art Transformer-base LLMs.
Rotary PE (RoPE) has increasingly become one of the most popular choices to inject positional information. These PEs are given by sequence 0,,...,0y; O4/2 of d/2 angles (where the embedding

---

Published as a conference paper at ICLR 2026
dimension d is assumed to be even), each of which acts corresponding rotation different rotational plane. By rotating query and key vector pairs, they account for the relative positional distance between them, thereby incorporating the positional information into the computed repre- sentations. Conventionally, RoPE is believed to be helpful because it facilitates token dependency decay with increasing distance. Yet, more recent evidence suggests that RoPE's utility lies in its ability to leverage different frequency bands for different purposes. In Barbero et al. (2024) for instance, the authors show that while certain models largely prefer to use low frequencies (which they conjecture to implement "information channels"), they also find that specific particular heads use high frequencies to produce "robust positional attention patterns" that also play a key role in the overall workings of the model.
Moreover, the choice of the base in RoPE (which determines the total frequency range) has also been shown to play an important role when extrapolating models to longer contexts. On the one hand, there is evidence that improved performance in long-context fine-tuning can be achieved by decreasing the base (i.e., increasing the frequencies) in RoPE (Liu et al. (023), a phenomenon often attributed to the fact that higher frequencies bias the Transformer towards attending more to closer tokens. On the other hand, decreasing the base negatively affects the ability to accurately retrieve information in such long contexts (Men et al. (024). Indeed, information retrieval (in long contexts) requires attending to similar tokens even when their relative distance is large, an ability suggested to be improved by selecting larger (rather than smaller) bases (Men et al. 2024), thereby shifting the range towards lower frequencies.
At a conceptual level, all these works seem to point towards the fact that different frequency bands in RoPE are used by attention heads implement two opposite core abilities: one more concerned with positional information and one with symbolic information. creating a tension between the two. Achieving a deeper understanding of this tension is a crucial step towards improving the mechanisms of Transformers by optimally striking the positional-symbot balance in given task context.
In particular, the following questions remain open:
What are the mathematical properties that underlie these two core abilities?
How can we measure if an attention head's behavior is exhibiting either of these abilities?
How do these abilities relate to the use of the different frequencies in RoPE?
How is model performance affected by the choice of these frequencies?
Contributions. In this work, we perform detailed theoretical and empirica I study of all these questions. Due to space constrains, we defer our further related work discussion to Appendix [A] We rely on the GEMMA-2, QWEN-2, and LLAMA-3 families of large language models for our empirical analysis. In summary:
We provide formal definitions of what it means for an arbitrary attention head to behave purely positionally or purely symbolically iors are mutually exclusive, unless the attention pattern follows a uniform distribution. Moreover, we prove that certain intrinsically positional operations cannot be implemented by heads that behave symbolically, whereas certain symbolic operations cannot be imple- mented by heads that behave positionally.
given input. We prove that these behav-
2. We introduce novel metric intended to provide scores that reflect the extent to which a given head behaves positionally or symbolically on a given input. Our metric can be flexibly applied to analyze given model at different levels of granularity, from specific heads on specific inputs and at specific frequencies in the case of RoPE- attention, single pair of scores per head, enabling the characterization and visualization of the poxitional-symbolic profile of the model.
3. We apply our metric to inspect the behavior of attention heads in Transformer- LLMs using RoPE, specifically the GEMMA-2, QWEN-2, and LLAMA-3 families. By analyzing these scores along the different frequency ranges, we reveal a surprisingly sharp connection between frequencies and attention head behavior: lower frequencies preferred by sym- bolic heads, while moderately larger frequencies tend to correspond to positional heads. On
'Code for our experiments be found furrut f -and- wiselllgg-cclcc r2022
2

---

Published as a conference paper at ICLR 2026
the other hand, larger frequencies used by heads achieving high values in both scores, a phenomenon that forces an approximately uniform distribution attention weights. 4. We then study attention heads in order understand how frequency choices affect their performance canonical tasks designed to intrinsically be either positional or symbolic. We find that head can act positionally (and thus perform well on the respective task) only when given access to relatively large frequencies, while forcing it to work with lower frequencies inevitably leads poor performance. We show that in this case, a U'-haped accuracy pattern tends to emerge. Similarly, attention heads can perform well on intrinsi- cally symbolic tasks if given access low frequencies. Interestingly, forcing it to rely on higher frequencies also harms performance, but this time leading to an inverted U-shaped accuracy pattern. Our observations are supported by both theoretical and empirical results.
2 PRELIMINARIES
Definition 1 (Attention Head). A d-dimensional decoder-only attention head is a function H: (I*')= - (IE")) given by a continuous "Logits function" L: (R4 N)² - R. continuous "Value function" VAL: Rd - R,, and continuous "activation function" F: Rd x R - Rd. Given an inpul sequence of vectors
= (II,...,,In) E (R , the head outputs a se- quence of vectors y (y1,...,,h, ..n) = H(), computed as follows. First, the "attention weights" " are computed as w/ = exp()))/ Eas, exp(At), with X3 = L(xe, 2y, then, a se- quence (a1....,,a)) an) of "attention vectors": a, -== E)=1 w3- VAL(Z,); and finally, one sets: y/ = F(a,, 24), L,...... n.
Transformer architecture. (decoder-only) transformer MEMB for the vocabulary Σ is composed of embedding function EMB: E - RRd where d is the embedding dimension, finite sequence of d-dimensional Attention Heads II...., He, and function Id R Given sequence s (o.,.... o,)) in the vocabulary Σ, the transformer M returns a probability distribution on as follows: Define 2; EMB(O,). Then, apply Hę
H1 to (z1,..., xr,) to obtain (#1,...,, Un))
Subsequently, define r(u,) R and finally the desired probability distribution is given by μ = softmax(v). When j is maximized at a unique symbol E Σ for an input s, we set MEMB (s) aa
About the Logits function. Of particular interest for us are the Logits functions of following form: L(z,, i, J) = (U' KI,, U'Qz)), 1.j=1, 1,..., j i, where Q and K are the (learnable) "query" and "key" matrices and Rd + Rd is unitary linear operator whose role is to incorporate the relative positional information between æ;; and æ given by positional inputs > 0 and > 0. We note that both NoPE and RoPE are special cases.
(1)
Rotary Positional Encoding (RoPE). We assume even hidden dimension d 2 2. For RoPE (Su where 91 is the fastest
ct al. (2024h) we consider the set of angles = (gk = 0 22-12=1)/ddk rotation angle/frequency and 9a/2 is the slowest For given angle gk. denote the rotation matrix p(gx) as
[cos(gk) sin(gx)) sin( cos(gi) [
p(gx)= =
To account for different rotation angles, we define a matrix R$ = Dd/2 p(gx)) E IRddd as the concatenation of the i-th exponentiation of rotation matrices p(g)) in a block diagonal matrix. Since each matrix p(gx) is rotation matrix, we get that p(g))' = p(igh). Finally, for query vector x; position and key vector z, position j. we define the RoPE logits function as ix,, (R's x2) (R's; x R R
3 POSITIONAL VERSUS SYMBOLIC ATTENTION
As done in previous transformer explainability Ali et al. 2025 Sakarvadia et al., 2023 Ferrando ett al. 2023 Ameisen al. 2025) and RoPE analysis work (Barbero al.) 2024), and for the sake of simplicity, we choose to base our theoretical analyses directly on the Logits function, while attention weights are used for empirical observations. All proofs are deferred to the Appendix.
3

---

Published as a conference paper at ICLR 2026
3.1 DEFINITIONS
In this section, we provide formal definitions of what it means for a head to behave positionally or symbolically given input. In the following, S; will denote the set of permutations of [t].
Definition 2 (Positional and Symbolic attention). We say that a head H acts positionally on an input I- (#1. F2, ..., Fn) at query i I n if its logit sequence is invariant under permutation of the key vectors. That is, for all x E S
Vj<i.
On the other hand, we will say that H acts symbolically on (I), I2,,.., un) E (IRd)" at query i S nu if its logit sequence is equivariant under permutations of the key vectors. That is, if
L T()) L(z,, (..))
VIT St-1, Vj <i.
In other words, suppose we are querying on a given input from position to all positions i. A head H is positional when L(x,, i, x) depends only on the position j. and not on the value of the key vector zj. In contrast, H will be symbolic when L(x, i, .y. only depends on the key-vector Tr (the "symbol""), regardless of the position j. See Figure 4 for an illustration (Appendix B.1]
Let us mention a few important examples. First, it is easy to see that a NoPE head acts symbolically on every input (the logits function does not depend on j in this case). Second, if an input F is such that the key vectors Kæ, are all the same for j I...i 1, say equal to some vector v, then any head with Logits in equation [ will act positionally (u. U²=:Qx;), which only depends on j. Note that for such an input, NoPE head will act both positionally and symbolically. In this case, the logits Lxx,, 8, ()) will have the same value for all ar with j < i, yielding uniform attention pattern.
it, simply because L(x,, i, (..
3.2 THE POSITIONAL-SYMBEN
EXCLUSION PRINCIPLE
Let L be a logit function and ā (x1, ..., In) an input. Here, we assume we are querying from i n. For permutation Sn-1 and E [n 1], define
h [] L(xn, n, x))) L(ang n, 23,5);
8aym L, [] = L( n, Fy ग (j)) - L(z,, n, Iz.
We view tht and o2 as real ((n- - (nn-1)!- (1n-nnnn)-iiinnn dimeennnnla vectors with coordinates, indexed by elements of S, x [n 1]. By definition, a head acts positionally (respectively, symbolically) at i on the input if and only if 6p = 0 (respectively, 07,mm 0). One could imagine that some attention heads in real-life transformers, although not being exactly positional or symbolic, have one of the vectors 8pos or 57,3 really close to O, thus exhibiting "almost" positional or symbolic behavior. Based on this idea, the next section we define positional and symbolic scores.
We now use these norms in a result that intuitively states the following: If the head's behavior on B close to both positional and symbolic, then the attention weights form an approximately constant sequence. This is an instance of the non-focused attention, we can have both positional and symbolic behavior.
duality: only at the cost of a uniform
exclusion principle). Let H be an arbitrary attention head with the logit function L. Let ā = (I.....,Ia) be an input and let X = (AI, An-1) be the sequence of logits on this input. excluding L( n. Ina n): namely, >g L( n, Nz.)). Then, denoting by the average value of X;² 's, we have
Theorem 1 (The resitional-symmole
Var(A) 1 Ea, 11)²
10 + 1 112 (n 1)! (n 1 1)
3.3 POSITIONAL AND SYMBOLIC SCORES AND THEIR APPLICATION TO REALISTIC MODELS
In this section, introduce scores to measure the extent to which each head in a model behaves po- sitionally or symbolically, and apply them to inspect the google gemma-2 22-2biit t model (simi- lar analyses on four other models are presented in the Appendix B.5) Our evaluation is grounded in
4

---

Published as a conference paper at ICLR 2026
the binding task, which tests whether the model can correctly associate entities with their attributes (Feng & Steinhardt 2023). Specifically, we construct inputs consisting of n = 256 untit--attribnn pairs, where entities are proper names and attributes are colors (e.g., Alice likes the color Red; Bob likes the color Blue). At the end of the sequence, a query probes the attribute of one entity (e.g., What color does Alice like the most?). We choose this task for two reasons. First, it can intuitively be approached by a combination of positional and symbolic mechanisms. Second, we have control over the location at which a given entity-attribute pair appears in the prompt. This allows for clearer comparison of the model behavior's 's when the pair location ranges from the beginning to the end of the prompt.
From Logits to Attention weights. Given arbitrary head H with logits function L and an input T. we define the logits produced when querying the final token In as L(T) = (L n, x3, i S n). Then, we define the corresponding attention weights as D(z)= = softmax(L(T))
Computation of Positional and Symbolic Scores. We partition the sequence I into contiguous m blocks and compute the sequence d = of DTT) over the block i. To analyze how attention behaves under input permutations, we focus on simple block swaps (e.g., exchanging block i with block j). For each such permutation, we represent the corresponding block averages before and after the swap into two-dimensional vectors Vay (de, d (before the swap) and bly (di, d') (after the swap, but on the same block locations). Using cosine similarity on the two-dimensional vectors v and v', we define two scores: the positional score Spos (ō), which measures the stability of block averages under permutations (i.e, we compare each 1'y against Ugy), and the symbolic score (x), which measures whether block averages move consistently with the permutation (i.e. we compare each o/g against Uyz). The full details of how the metric is computed can be found in Appendix B.3
dm), where d; is the average attention weight
Visualizing heads in the Positional-Symbolic PS(H, Ø) == (Spos (2), (z)) defines a point in what we call the positional-symin In order to visualize the behavior of an entire model on a given task, we can fix a prompt á = (o1,..., o,) and compute the scores for all heads in the model based on their corresponding inputs 0 = (of 11 ..... .....,,'-" )) with é defining the layer number. This allows us to reveal the behavior of the entire model and visualize its behavior's 's profile in the positional -yymollli plane.
Plane. The pair of scores
plane.
model on the binding task. Each dot on Figure [IA represents the location of an attention head on the positional-symbol plane (with color coding layer depth). We observe that early laver heads are associated to high positional scores, whereas late layer heads tend to be more symbolid Figure 3 illustrates an important feature of our work. Compared to previous studies (Barbero et al. 2024). we now can depict full model attention head behavior snapshot, for all heads and layers. This snapshot provides a strong empirical support for our theoretical positional-symbott negatively correlated (Pearson correlation coefficient: r = -0.91, p-value < 0.0001). Furthermore, Figure 7 shows the predominance of symbolic head behavior when querying block and how the positional and symbolic scores gradually revert when the queried blocks gradually move from block to 256.
Figure shows this profile for the entire gemma-2-225
exclusion principle. Indeed, the scores in both heatmaps are
Positional and Symbolic Scores in RoPE. We now take closer look into how attention heads use the different RoPE frequencies. We do this by first decomposing a given head H into m distinct projection heads, each of which can be seen as individual 2-dimensional head with single asso- ciated frequency (see Appendix [B.3 for details of logits per frequency). By computing the positional and symbolic scores for each projected head, we can visualize how the behavior of the combined head H depends on each of the frequencies in RoPE. As can be seen in Figure [ there is a sharp correspondence between attention head behavior type (i.e., symbolic or positional) and frequency, with relatively high frequencies associated to positional behavior and lower frequencies to symbolic behavior, mirroring the results observed in some specific heads by Barbero et al. (2024). but now
TTo assess the statistical significance of this pattem,, we conducted a median test comparing early (layers 1-13) and late (layers 14-26) layers. For positional scores, early layers showed a higher median (0.83) than late layers (0.56; p-value < 0.0001). Conversely, symbolic scores were higher in late layers (median 0.67) than in early layers (median 0.49; p-value < 0.01).
5

---

Published as a conference paper at ICLR 2026
A
B
C
All heads in Pos/Sym Plane
Positional Score
L-B, H=D
L-12 H=D
1.0
1.0
1.
0.8
Hoad
0.5
0.5
Symbolic Store
0.6
100
Symbolic Score
L-15. H=0
L=23 H=2
1.0
Hoad ID
Layer Depth
0.5
0.0
0.0
0.0
0.0
0.2
0.4
0.5
0.8
1.0
10
15
20
25
100
100
Positional Score
Layer
Froquency
Proquoncy
D
F
Fraquency ID=127
Frequanccy ID-63
Frequency IDeo
Avg. Norm of Logits per Frequency
Attention Weght
1 1-+160
200
Pas:
Pos:
Pos:
Pas:
0.00
0.00
0.00
Sym::
5ym::
Sym:
Sym:
1.00
1.00
75
100
125
1000
1000
1000
1000
Frequency
F
Freq. Pos/Sym Plane
Attention Weight
++
128
1+0
1.0
Pos:
Pos:
Pos:
1.30
1.00
0.8
Sym:
Sym:
Sym:
Sym:
Symbolic Scare
0.00
0.00
0.6
1000
1000
1000
1000
0.4
256
96
256
+16
256 224
Attertion Weight
0.2
Pos:
Pos:
Pos:
Pas:
0.99
0.60
Sym:
Sym:
Sym:
Sym:
0.0
p.96
0.98
0.98
0.0
0.2
0.4
0.8
0.0
1.0
2000
0
1000
1000
1000
Positional Score
Token
Token
Token ID
Token ID
Figure Global and local analysis of attention head behavior. A. Each head in the posi- tional-symbolic plane. B. Heatmaps of positional and symbolic scores for each head across all layers. C. For the same heads, we plot their positional and symbolic scores as a function of RoPE frequencies. By convention, lower frequency IDs correspond to higher angular frequencies, and conversely, higher frequency IDs correspond to lower angular frequencies. D. norms of the logits at each frequency for head (12:0). E. Attention weight patterns as a function of permutations. Top rows (green traces): Behavior of a symbolic head whose attention weight mass follows the permutations. Middle rows (red traces): Behavior of positional head whose attention weight mass is invariant to the permutations. Bottom rows (blue traces): Behavior of a (mix) attention head with both high symbolic and positional scores, displaying uniform mass with low attention weight scores. Note that symbolic, positional, and mix head behavior ar associated with low, relatively large, and the largest frequencies, respectively. F. Location the positional-symbolic plane of head (12:0) function of the selected frequency.
generalized to all heads and layers (see Figure 5). Our method allows to reveal new empirical ob- servation. We find that for the highest frequencies, some heads display large positional and symbolic scores. According to Theorem these should correspond to heads exhibiting a uniform attention pattern.
We then took an even deeper look at the first head from layer 12 (head 12:0; see top right of Figure [C). T depicts the behavior of this head for three different frequencies exhibiting symbolic (green traces), positional (red traces) and both (blue traces) behaviors, respectively. As expected from our definitions, symbolic behavior is reflected by changes in the attention weight mass that moves with the permutation, whereas the attention weight profile remains unchanged for the po- sitional behavior. One can also clearly see a uniform attention pattern for the frequencies where the head displays both behaviors. Finally, head (12:0) also allows understand the underlying

---

Published as a conference paper at ICLR 2026
reasons for the location of heads on the positional-symine symbollli plane, as this head exhibits both high symbolic and positional scores (see red star on Figure TA ). For this head, we analyzed the norms of (iggure[p)), and noted that most of the norm mass is concentrated
the key vectors at each frequency on the smallest frequencies, which therefore contribute more to the global head's behavior, and ex- plains relative high symbolie score. Additionally, some mass is distributed over large range of intermediate frequencies, in turn explaining a relatively high positional score.
4 POSITIONAL AND SYMBOLIC CANONICAL. TASKS
In this sectior we aim at deepening our theoretical understanding of positional and symbolic heads. In particular, we introduce non-trivial tasks that can be solved by pure positional (symbolic) heads using 1-layer attention-only transformer. Our goal is to show how simple models can give us important clues with respect to the fundamental tension between tasks requiring the extraction of either positional symbolic information from the input.
Index Task. We now introduce task intended to be intrinsically positional. Consider an input sequence Spos = (oo, oI,..., In 1, where each og is a symbol from finite vocabulary Σ, and j is an integer index in the range 0 S n. The goal of the task is to output the symbol at position j (determined by the last token in ), Jz. Thus, to solve this task, one needs a transformer that computes the function fPos defined by nism allowing to solve this task should be highly sensitive to the position being asked for, while can ignore the actual symbols. We now proceed prove that fpos can never be solved by I-layer decoder-only transformer MEMS with a single head H that acts symbolically.
((..,,,......,
oj. Intuitively, any mecha-
Theorem 2. For an arbitrary attention head H, if H acts symbolically on a non constant input if then MEMB+ + fPos-
Note that Theorem 2 does not assume any specific form in which the positional information is incorporated into the logit function. On the other hand, our next result shows that fros can be solved by simple and purely positional attention head.
Theorem 3. For each > O there is purely positional attention head Hpos that uses RoPE with single angle and no value or activation function, such that MEMB = fros every Spos of length n.
Even though solving fres seems rather simple, Theorems and characterize it as a purely posi- tional task. Interestingly, not every angle of RoPE is appropriate for the construction of Hpos- In fact, if the angle is greater than 2T /n, attention weights are not maximized in just one position of the input, which could lead to confusion on the model's output. Later we will see how this effect emerges also empirically, having a direct impact on a model's 's capacity for solving tasks like fPos.
empiriold
Information retrieval Task. Here, we consider inputs of the form SSYM (01 02thi, in where is is an integer for every h < n. The goal for this task is to retrieve the correct symbol o, #ij. associated to the last symbol o, (the query). We assume for this task that σ; appears just once in the sequence before the query. Formally, the solution function [syM is defined as (so Clearly, this task only depends on the symbols and positional information is irrelevant to the answer. Similar to the previous task, we can show that any simple head that acts positionally can not compute fSYM. Here, by H being simple mean that VAL is the identity, the activation function F is just a projection, and that EMB- = ONEHOT.
o 1 -n. H) = a,tij
Theorem 4. Let H be any simple attention head. If H acts positionally on an input I, then MONEHtor fSYM-
We remark that if we allow F to be a general MLP, then one can use it to override the attention mechanism and produce a (rather artificial) counterexample (see Appendix section We proceed by showing that purely symbolic attention head Hl can naturally solve fsvu-
Theorem S. For every n > 0 there exists a purely symbolic attention head HsvM without value or activation cocettionfnnttioo function, such that MonEHor = JSYM on every 88yM of size n.
All proofs for this section can be found in Appendix C
7

---

Published as a conference paper at ICLR 2026
Experiments positional versus symbolic heads. Our results show that simple 1-head trans- former can perfectly solve fros and fSYM- Namely, the 1-RoPE Hpos for fros and the NoPE HsvM for fSYM- In order to test if such heads can be learned during training and at the same time explore how the choice of the angle 0 affects accuracy, we trained a simple 1-RoPE architecture for different values of 0 both tasks. While we observe that 1-RoPE converges appropriate angles (Figure we also find that the chosen RoPE angle has a direct impact on the learning abilities of the model. Indeed, as expected from our theoretical results, small frequencies (large ID) are not suitable for solving the Index (positional) task, while larger ones are inadequate for the Information Retrieval (symbolic) one. Note the striking resemblance of these aceuracy curves with the positional and symbolic scores curves from Figure C
proper solution for the
A
Iter 64
Iter 256
Iter 448
Epoch 1
Epoch 2
Epoch S
Epoch 100
Pestoval
Acuureyy
8
Iter 84
Iter 256
Iter 448
Epech 1
Epech
Epoch
Epoch 100
Accurncy
60
Frecuency
Frecuency
Frecuency ID
Frecuency
Frecuency
Frecuency
Frequency ID
0.01 2nth
0.10 2mgn
2
Angle (log scale)
Figure 2: Performance the canonical tasks across training iterations and epochs. A. Tension between Index (positional) and Retrieval (symbolic) tasks. B. Accuracy the partial induction task for 1-frequency and 2-frequency models. C. Frequency IDs mapped to the log-scaled angle axis.
Partial induction Task. Finally, we present a partial induction task with inputs of the form SMIX = H1, #2 -n-1 o #) where ig is an integer for every k < n. We as- sume for this task that o, appears least two times in the sequence, in positions f1 and ja, and that integers in and ijs are different. The goal for this task is to retrieve the correct symbol-integer pair associated with the last occurrence of a, (notice that the case with a single occurrence subsumes the information retrieval task). We denote by fMIX the corresponding solution function. The following result (which follows directly from our previous results) show that if a I-layer transformer computes fMX. then it cannot behave in purely positional purely symbolical way.
Corollary 1. If a simple model MoNcHot with head H computes MMN then for every input M H does not behave positionally (or symbolically) on Mix.
On the positive side, we demonstrate that a single attention head with two RoPE angles suffices.
Theorem 6. For every n. > O there exists an attention head H&t with two RoPE angles such that MonEHor fMi on every sMix of length n.
Experiments for the partial induction task. Figure 2B, shows that toy model with 1-RoPE angle can not solve the partial induction task. On the other hand, a model with angles 0 = 0 and 82 (flexible), solve it when 6g is not too big. This is expected from the proof of Theorem 6 since bigger angles for O2 confuse the model's output. Finally, the fact that the 1-RoPE version of this model has decreasing accuracy as angles get bigger suggests that our version of the partial induction task has more prominent symbolical difficulty. This could be explained by the fact that solving fMI requires to detect only two positions of the input (both occurrences of or), but several symbols need to be distinguished at the same time.
Namedd as such in allusion to induction heads operations OOssson
2022
8

---

Published as a conference paper at ICLR 2026
5
ON THE RELATION BETWEEN FREQUENCIES AND ACCURACY SHAPE
A
min
6a 0.01 min
Oa 2nin
L-12, Hed, ID:65
100
Accunacy
20
Oimentonn
-2
Aocusiy
-.0
80
-15
-20
Qvey
Posiioon
-3 Dimenson 1
4.0
Ornensoon
50
03
Figure 3: A. Shapes of Accuracy for the Index (in red) and Retrieving (in green) tasks, per Epochs. B. Query (in blue) and key (in orange) vectors trajectories during training for the Index Task. C. Query/Key vector projections a rotational plane from a gemma 22-2------- Head for some Binding Task's ss input (64 pairs).
As we have seen, a head required to behave in a certain way needs to access certain frequencies. In this section, operate with the wrong frequency is affected. We thus analyzed the accuracy of our trained I-layer Transformer on both of our canonical tasks, focusing on how it depends on the location of the answer within the prompt. Interestingly, as it can clearly be seen in Figure 3 (upper graphs), the loss in accuracy when using frequencies that are too low for solving the Index task (red) more severe when the answer lies at the middle of the prompt, and less severe when they closer to the border positions, resulting in a U-shaped accuracy pattern. This is reminiscent of a phenomenon that has been reported before as "lost in the middle" (Liu et al. 2024). On the other hand, when solving the Retrieving task (green) using frequencies that are not low enough, the opposite phenomenon is observed, and inverted U-shaped accuracy pattern emerges.
interested in understanding how the performance of head that is forced to
Figure 3 shows the trajectories of the query (blue) and key (orange) vectors during training. These trajectories reveal the emerging mechanism the trained model relies on to solve the task. The query vectors are configured so that their angles code the different positions they are querying to, while the key vectors arrange into single direction, which explains why the head behaves positionally. In fact, this is the exact same mechanism the theoretical solution Hpos implements. On the other hand, Figure 3 shows the projections of the query (blue) and key (orange) vectors on selected rotational plane for a frequency where head 12:0 from gemma- - 2 2-2b- it behaves positionally. One can see the striking resemblance to the toy model.
By further studying the mechanisms implemented by our theoretical toy models, we are able to provide an explanation for the observed accuracy shapes, at least at the attention level pattern. Let us say that function f [n] + IR is u-shaped if it has local maxima at 0 and n and over this interval it is first decreasing and then increasing. Similarly, we say inverted-u-ssaped if -f is u-shaped. Now, for either of our canonical tasks (index and information retrieval) let be the location of the answer within the input prompt, and let Wmax (j) be the largest attention weight value of the heads Иooo and Hssm from the proofs of Theorem 3 and [5] respectively. For the next Theorem, we consider a modified version HSvu of Hsyy where we allow for the use of an angle 0. From the construction of these heads it follows that Wmax is precisely the weight position j, which is moreover equal for every such input. Besides our empirical findings, in the Appendix we give rigorous mathematical proof of the following:
Theorem 7. The function Wmax is u-shaped for Hpos and inverted-uu-supped for simplified version of H
6 CONCLUSIONS AND FUTURE DIRECTIONS
We have carried a thorough analysis of two core abilities that attention heads in Transformers require in order to work effectively. By providing precise definition of positional and symbolic behavior,

---

Published as a conference paper at ICLR 2026
came up with metric detect working head's positional symbolic nature. We were able to disentangle these mechanisms and mathematically understand why these two abilities are in tension with each other. By analyzing both real and toy models, we explained why heads prefer different frequencies in RoPE depending on the mechanisms they need to implement to process given input, and finally related the resulting accuracy of model its ability to employ the appropriate frequencies.
We believe the characterization of model by its positional-symbolic profile is powerful tool to analyze how particular models behave on different tasks. Even though in this paper we have con- centrated on the Binding Task (Feng Steinhardı 2023). we foresee a promising strategy in sys- tematically performing similar analyses for wider set of tasks and models. For example, one could use this tool to characterize the nature of models and tasks in terms of the "amount" of positional or symbolic mechanisms they tend to implement (for models) rely on (for tasks).
In fact, the invariant properties that underlie positional or symbolic behavior can be thought of as inductive biases that the model has to learn to solve the task at hand. In other architectures such as CNNs or GNNs for example, these type of inductive biases are imposed into the architecture by design, thereby improving generalization properties and learning efficiency (Maron et al. (2019); Petrache Trivedi 2023); Deng et al. (2022)). ). It is interesting to wonder whether a similar strategy could be applied to Transformers, for instance by imposing the appropriate degree of positional or symbolic behavior to selected heads. is tempting to conjecture that these two core abilities consti- tute in fact the fundamental building blocks of any head, in the sense that their action can always be somehow decomposed into simpler heads behaving purely either positionally or symbolically, Such theory could constitute the base for designing architectures targeted specific tasks, We see this as an exciting avenue for future work.
ACKNOWLEDGMENTS
All authors are funded by the National Center for Artificial Intelligence CENIA FB210017, Basal ANID. Kozachinskiy supported by ANID Fondecyt Iniciación grant 11250060.
REFERENCES
Riccardo Ali, Francesco Caso, Christopher Irwin, and Pietro Liò. Entropy-lens: The information signature of transformer computations. arXiv preprint 2rXiv:2502. 16570, 2025.
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, et al. Circuit tracing: Revealing com- putational graphs in language models. Transformer Circuits Thread, 6, 2025.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06200) 2024.
Lihu Chen, Gaël Varoquaux, and Fabian Suchanek. The locality and symmetry of positional encod- ings. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14313- 14331, 2023a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023b.
Siyu Chen, Heejune Sheen, Tianhao Wang, and Zhuoran Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. arXiv preprint arX1v:2402.19442, 2024.
Yiting Chen and Junchi Yan. What rotary position embedding can tell us: Identifying query and key weights corresponding to basic syntactic or high-level semantic information. Advances in Neural Information Processing Systems, 37:54507-55528. 2024.
Weijian Deng, Stephen Gould, and Liang Zheng. On the strong correlation between model invariance and generalization. In S. Koyejo, S. Mohamed, A. Agarwal,
10

---

Published as a conference paper at ICLR 2026
D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural Information Process- ing Systems, volume 35, pp. 28052-28067. Curran Associates, Inc., 2022. URL
https: http://roociiiinn.eeeeeeeeeeeeee coppapr httpaapeer_eeepppeeeeeeeeee Files/paper/2022 file/
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint 1rXI::2402133755 2024.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? arXiv preprint arXiv:2310. 17/91, 2023.
Javier Ferrando, Gerard Gállego, loannis Tsiamas, and Marta R Costa-Jussà. Explaining how transformers use context build predictions. arXiv preprint arXiv:2305. 12535, 2023.
Ruiquan Huang, Yingbin I. Liang, and Jin Yang. Non-asymptotic convergence of training transform- for next-token prediction. Advances in Neural Information Processing Systems, 37:80634- 80673, 2024.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892-24928, 2023.
Guolin Ke, Di He, and Yan Liu. Rethinking positional encoding in language pre-training, arXiv preprint 2rXiv:2006. 15595, 2020.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. arXiv preprint arXiv:1805.01052. 2018.
Alexander Kozachinskiy, Felipe Urrutia, Hector Jimenez, Tomasz Steifer, Germán Pizarro, Matías Fuentes, Francisco Meza, Cristian B Calderon, and Cristóbal Rojas. Strassen attention: Unlock- ing compositional abilities in transformers based on new lower bound method. arXiv preprint arXiv:2501.19215. 2025.
Yingcong Li, Yixiao Huang, Muhammed E Iidiz, Ankit Singh Rawat, and Samet Oymak. Mechanics of next token prediction with self-attentis and Statistics, pp. 685-693. PMLR, 2024a.
International Conference on Artificial Intelligence
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason Klusowski, Jianqing Fan, and Mengdi Wang. One-layer transformer provably learns one-nearest neighbor in context. Advances in Neu- ral Information Processing Systems, 37:82166-82204, 2024b.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:23 2023.
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gast- par, and Chanakya Ekbote. Local to global: Learning dynamics and effect of initialization for transformers. Advances in Neural Information Processing Systems, 37:86243-86300 2024.
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A curious case of single-layer transformers. The Thirteenth International Conference Learning Representations, 2025.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph 09902
networks, 2019. URL htttpp:://axxvv.com http::::////xiiooo////////////// absr///b///////
Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. arXiv preprint arXiv:2405. /459/, 2024.
11

---

Published as a conference paper at ICLR 2026
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11888 2022.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprini vrXiv:2309.0007) 2023.
Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing-complete. Journal of Ma- chine Learning Research, 22(75):1-35, 2021.
Mircea Petrache and Shubhendu Trivedi. Approximation-generllizzzzz der (approximate) group equivariance. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. 61936-61959. Curran Associates, Inc., 2023. URL
trade-offs un-
neurips 635f6226c6611111550 Caperr
Philip Quirke and Fazl Barez. arXiv:2310.3312,, 2023.
Understanding addition in transformers. arXiv preprint
Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: tool for mechanistically interpreting the attention head information retrieval mechanism. arXiv preprint arXiv:2310.162770 2023.
Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. Advances in Neural Information Processing Systems, 36:36677-36707, 2023.
Clayton Sanford, Daniel Hsu, and Matus Telgarsky. One-layer transformers fail to solve the induc- tion heads task. arXiv preprint arXiv:2408. 14332, 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputiigg, 568:127063. 2024.
Yuandong Tian, Yiping Wang. Beidi Chen, and Simon S Du. Scan and snap: Understanding training dynamics and token composition in I-layer transformer. Advances in neural information process- ing systems, 36:71911-71947 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog. Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.1600 2023.
Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, and Acyr Locatelli. Rope nope and back again: A new hybrid attention strategy. arXiv preprint arXiv:2501.1879. 2025.
Tong Yang, Yu Huang, Yingbin Liang, and Yuejie Chi. In-context learning with representations: Contextual generalization trained transformers. Advances Neural Information Processing Systems, 17785877-85897 2024.
Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention net- works can process bounded hicrarchical languages. arXiv preprint 111iv:2105.11115, 2021.
12

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
Given the success of RoPE, several studies have dived into its mechanisms to provide: (i) better long-context performance, and (ii) deeper knowledge of how the RoPE's mechanisms relate the type of information encoded in the Transformer.
Extending context windows. Much work has been done to modify or extend the mechanisms of RoPE in order to extend the context window length of LLMs. It has been shown down-scaling position indices RoPE, within the pre-trained context-length limit and alongside small amounts of fine-tuning, avoids catastrophic attention scores blow-ups seen in naive extrapolation, thereby promoting extrapolation in unseen ranges (Chen et al. 2023b). In the same vein, |Png et al. (2023) propose YaRN, method that combines Neural-Tangeet- KeenneenttKeeelllee aware position interpolation with at- tention scaling improve long context window performance. Other researchers have found that in- creasing the RoPE base frequency improves long-context information retrieval (Xiong et al. 2023). More recently. LongRoPE proposes the addition of non-umine non-uniform position interpolation with a fine- tuning curriculum strategy as shown to further increase the ability of long-com cong-context performance (up to 2,048k tokens) (Ding et al. 2024). Finally, by analyzing attention scores in the Needle A Haystack [] benchmark, Yang et al. (2025) observed that NoPE was related to higher attention mass over the needle (compared to RoPE), leading them to propose RNoPE, i.c., alternating NoPE and RoPE layers. RNoPE resulted in better performance, particularly within question-answee con- texts.
Mechanisms of RoPE and their relation to information encoding. Despite its importance to im- prove model performance, much less work has focused on generating deeper theoretical knowledge of the mechanisms subtending RoPE, and how these relate to the behavior of attention heads, and in turn to model performance. [Chen & Yan (2024) observed that the 2d pair rotation direction highly depends on Query/Key pair weight angle. These authors observe empirically, and theoretically demonstrate, that non-orthogonal weight vector pairs are more sensitive to positional information, while orthogonal ones focus rather on semantics. Their theoretical advances was then used to devise an efficient fine-tuning method (Angle-based Weight Masking) that primarily focuses on updated comantics-related weights. Closest our work is that Barbero al. (024). In their work, they theoretically prove (and empirically observe in Gemma 7B) that RoPE can learn specific positional (diagonal and off-diagonal) attention head patterns, which not the case for NoPE, Furthermore, they relate the ability to learn these position attention patterns as leveraging high frequencies in RoPE. Conversely, semantic attention patterns are attributed leveraging low fiequencies. Further- more, they suggest to truncate low-level frequencies to avoid arbitrary (irrational) rotation values in long contexts, and observe better performance when doing so (when keeping 75% of the highest RoPE frequencies). In works like Kitaev & Klein| (2018) they hard wire the separation of posi- tional and symbolic (or content) information present in the input, by forcing this separation in the initial embeddings. Ke et al. 2020), the other hand, incorporates particular projection matrices for positions and symbolic information independently, which are then aggregated as two different components of the logits function. In this work, we do not force any separation of positional or symbolic information in the models. Rather, we relate these qualities to the behavior of the logits function, with a particular interest in the context of RoPE as the state-of-the-art positional encoding. We found that, in decoder-only Transformers, there is an emergence of heads tending to either adopt positional or symbolic behavior (which is learned and not forced in the architecture) suggesting that many underlying mechanisms can be decomposed into purely symbolic and purely positional ones. In this sense, our work provides strong theoretical and empirical reasons for the study of architec- tures with the capacity of learning separate positional and symbolic logit functions. Finally, in Chen jet al. (2023a) researchers investigate attention patterns and defines interesting metries of locality and symmetry of attention patterns in Transformers. In comparison, instead of considering attention weight patterns as they stand, our metric analyzes how these patterns change when permuting the input vectors, thus capturing mechanistic properties of a different nature. In particular, our metric is in principle agnostic to how focused these patterns are, and for instance both symbolic and positional behavior perfeetly compatible with focused attention.
httpp://githhb..r htte:://ggthb...cmm////aaaaaarts Needle nAHaystack
13

---

Published as a conference paper at ICLR 2026
Our work extends current knowledge by: (7) mathematically defining positional symbolic at- tention head behavior, (ii) offering novel metric to quantify these behaviors at different levels of granularity (from heads to full model), (itii theoretically proving that position and symbolic at- tention heads are mutually exclusive, (iv) theoretically proving that some positional tasks cannot be performed by symbolie heads, and vice-versa, and (v), empirically demonstrating that causally control the accuracy pattern in by gating access to specific frequencies.
Moreover, described in the main text, we use toy 1-head decoder-only transformer to gain deeper understanding of the RoPE properties. Such a model allows us to have a strong hold on the interpretation of the RoPE mechanism, otherwise lost when working with deeper and wider mod- els. Other works following this approach have demonstrated important features of Transformers. For instance, such toy models have been used to characterize transformer loss landscapes (Makkuva et al.
2024 2025). Furthermore, 1-layer transformers have also been used to evaluate their expres- sivity Li et al. (20246); Quirke Barez (2023); [Sanford al. (2023) 2024); KKozachinskiy et al.
2025 Moreover, these models have been used to understand the learning dynamics and properties
in transformers (Tian al. 2023
et al. 2024a
Huang et al. 2024 Yang et al. 2024 Chen al.
2024).
14

---

Published as a conference paper at ICLR 2026
B ADDITIONAL ILLUSTRATION AND PROOFS OF SECTION 3
B.1 ILLUSTRATION OF ATTENTION HEAD BEHAVIOR DEFINITION
Logit Values
Invariant Logit Values
Equivariant Logit Values
traneformation
L(xuixtt.))
X1
X2
X1
X:
Xk
Xi
Xi
Xx
Xx
Figure 4: Illustration using the same sequence as in Definition Middle: Example of invariant logit values after simple permutation (swapping æy with rg). Right: Example of equivariant logit values under the same permutation.
Left:
Original logit values.
B.2 PROOF THEOREM
Denote dk.j LZx. n, Th. j). Take index j E [n-1] and permutation Sn-1 independently and uniformly at random. Note that (j, (()) is uniformly random pair from [n 1]² (under any fixation j, the distribution (j) uniform). Observe that:
Var(A) = Var(agy),
|
2
E((a(s))) - azy)²
(n 1)! (n 1)
1 2
(n 1)! (n 1) E((ja(()) -aj}²²
Since (j. x(j)) is a uniformly random pair from [n- 112, the pair (r( has the same distribution, meaning that:
|
(n - 1)! (n 1) B(a+()s (n())))))
Using the fact that Var(X) E(X Y)2 // for any random variable X and its independent copy Y. we get:
Var(A) Var(a,;) E(a,s - (
Finally, using an inequality (x y) < 2x 2y² for x. E TR, we derive the statement of the theorem
Var(A) Elaai a=()=()) / Blay ax(r)) E(en(s) an())a()))
|| 112 saym 11 L.a
(n 1)! (n 1)
B.3 GENERAL METRIC TO TEST POSITIONAL. OR SYMBOLIC HEADS
Blocks. We define a fixed partition of the index set [n]] into m contiguous intervals (blocks)
h = [a, by], 1a = [02, b2], I = [am, 6m]s
a1 = 1, 6m
with 4+41 by 1. For block I& define its average attention weights under z
de
Σ D(T).
tEL
We collect these into the block-average vector d(x) (di (I),... ..., dn (x)) R
15

---

Published as a conference paper at ICLR 2026
Block permutations with dynamic intervals. Let the original sequence be partitioned into tiguous blocks .,....,Im, as above, with lengths |/ [m A block permutation π E Sm rear- ranges these blocks, producing new sequence Tz)). We then define block intervals I'...... ! by preserving the original relative positions but adjusting the boundaries according to the lengths of the permuted blocks. More precisely, we let
& (x) where the boundaries given recursively by
T [ak bk]+
d 1,
= a 147-1
d +
Thus, the k-th block position in the permuted sequence has the same length as the block that was moved into that position. The block averages under the permuted sequence are
(x(T²) = I
Σ D(T(z)), TET
dimennina representation. A simple type of permutation a are a swap between block and block j. We represent the relevant attention mass by the vector
Ut ( (T), d, (z)),
and after the permutation by
Uky (T(()( ( (x(T))))
Permutation weights. Not all block swaps equally informative: weight each permutation ı according to the amount mass it moves,
(T) softmax [z(x) - /r)z))/T),
where + > 0 is temperature parameter controlling how sharply the softmax focuses on high-mass swaps.
Positional and symbolic scores. permutations using cosine similarity remain under a permutation:
We can now quantify how the head behaves with respect to block The positional score measures how stable the block averages
Σ
a(x) cos sim (asimi1(( (1)(((())), Viy (z)
Spos
The symbolic score instead measures whether the block averages transform exactly as the permuta- tion prescribes:
SSvM (I)
E [(( cos sim (1)(T(()), Uja (z)(
Scores per frequency. Even though previous definitions are applicable to any logits function L, in the case of RoPE have a more refined understanding of the positional and symbolic behavior of each RoPE's 's frequency as single angle attention heads.
leverage its frequency decomposition (as Hadamard attention head) to
We begin by isolating the contribution of a single attention head at a given layer, and decomposing its logits into frequency components induced by RoPE. Consider layer & and head h. The logits between query position n with token In and key position with token 2, are given by
t (I n, 23.
-KK' 1K
R 12
d/2
th 1
Ro 2th 22,2441
4/2
Σ
I (In, n, D).
"cos sim(a,b)
(0.8)
[a]
16

---

Published as a conference paper at ICLR 2026
Here each term
Lt (In: 2z.) 2k 22 (2.22(112)
corresponds to the logit contribution from frequency O, where the metrics Spos (T) or (T) be measured individually.
B.4 TEMPLATE PROMPTS FOR REAL-WORLD MODELS BINDING TASK
In template prompts, we consider the span-block format defined as follows:
span_block block "\n". join [f" (name) likes the color (color)" for name, color in sequence))
See bellow the template prompts for each model.
gemma -2-it
sonpcstart_of_tueee _o___uuruuuuerr
You are helpful assistant that can answer questions about entities and their associated attribute.
Concext:
([spen_block])
Queation: What color does ((entity(il)) like the most ourr <start_
Answer: [[entity(5]) likes the color
11ama-3.2-i2 2-i
<lbegin_ot faxt /s</st Seaerrt_hhaar____e _ ssystems aad header_ Idi> _dl> You are helpful assistant that can answer questions about entities and theie associated ottribute.<leo__ff
Concext:
[[span_block)]
Queation:
What
[eeniiy((]]]]
[istart_header... rassistante sdl_
Answer: [(entity(i]: color
qwen2-it
<limsstart |saystee
helpful assistant answer questions entities their associated
attribute. [|i
<lim_start (suser
Consext:
[[spen_block))
Question: What color does ((entity(jl)) like the (est?k/mmmedd
<lim_start /sassistant
Answer: [(antityld]] likes the color
B.5 GEMMA-2-22 1-AMA-2-BB-ISSTRUCE ON THE BINDING TASK
For the google/ prrolle/eemaa-- ggmma-mma222 --iima---bb-a 2--2biit model, we compute global positional and symbolic scores for each attention head, and for each head we compute the scores of the projected heads for every frequency. The same analysis is also performed for other models, with results deferred to the next sections. Our evaluation is grounded in the binding task, which tests whether the model can correctly associate entities with their attributes (Feng Steinhardt 2023). Specifically, we construct inputs consisting of
256 entity-atribbte pairs, where entities are proper names and attributes are colors (e.g., Alice likes the color Red; Bob likes the color Blue). At the end of the sequence, query probes the attribute of one entity (e.g., What color does Alice like the most?). The choice of the task made based on two reasons. First, it a task that can intuitively be approached by a combination positional and symbolic mechanisms. And second, have complete control of the location of
17

---

Published as a conference paper at ICLR 2026
the correct answer the question at the end of the prompt requires the model to find. This allows for a clearer comparison the model behavior's when we change this location by changing the query.
We segment cach imput into 256 blocks, with each block containing exactly entity-attribute pair. Then we select 9 uniformly spaced blocks. For a given query targeting the k-th block, we generate alternative inputs by permuting the position of i-th block with different j-th block, for all j = 1..9. This procedure yields nine permutations per query, and total of nine queries per input.
For each attention head, compute the positional and symbolic scores across RoPE frequencies. Recall that Gemma employs RoPE, where frequency identifiers are mapped to angular frequencies according to 0g = 10000-2000 2-t/dd with the frequency index and d = 256 the head dimension. By convention, lower frequency IDs correspond higher angular frequencies, and conversely, higher frequency IDs correspond to lower angular frequencies. This mapping allows us to analyze the contribution of each frequency to the model's binding behavior.
We first evaluate the metrics separately for each frequency. Figure shows the location of selected aftention heads in the positional-symbolic plane, together with their detailed frequency profiles. Since the model uses a head dimension of 256, this corresponds to 128 RoPE frequencies. By convention, higher frequency IDs correspond lower angular frequencies, while lower frequency IDs correspond to higher angular frequencies.
We then aggregate these measurements by summing logits across frequencies, yielding global positional-symboli profile. The results, presented in Figure provide an overall view across all layers and heads of the model.
NG
Figure 5: Positional-symbolic profiles of all attention heads in google/gemma -2-mmma-2-2bi..
Each subplot corresponds an attention head, 26 layers, heads per layer.
18

---

Published as a conference paper at ICLR 2026
Selected PasttanaliSymmbll Plane
L-1.
Posional
HR2
Syribelc
LOS HEO
100
1.50
1.00
1.00
0.60
0.50
0.50
0.25
0.25
Symboti Scre
420
L-12. HeO
L-15,
04
1.00
1.00
1.00
+0
0.50
0.50
0.50
0.25
0.25
100
Postional Scom
Frequency
Frequarcy ID
Freeuancy
Figure 6: Positional-Symbolic Plane across Frequencies. Visualization of six attention heads from the google / gemma- 2-2b-it model. (Left) Each point represents head in the posi- tional-symbolic plane, labeled by its layer and head index. (Right) For the same heads, we plot their positional and symbolic scores as a function RoPE frequencies.
Query
Query
Query black
1.00
1.00
1.00
Symbolic Score
0.50
0.90
0.50
0.25
0.00
0.00
0.00
0.0
0.5
0.0
0.5
1.0
0.0
0.5
Pos tinnal Score
Post Score
Postioonl Score
80
80
40
40
20
20
Scoree
Score
Siorre
Figure 7: Heads when the block moves to 9). (Above) Heads the positional symbolic plane for each block position. (Bellow) Histograms showing how positional and symbolic scores change as the relevant block moves.
C ADDITIONAL ILLUSTRATION AND PROOFS OF SECTION
C.1 ILLUSTRATION OF CANONICAL TASKS
See Figure for an illustration of how the canonical tasks presented in Section should work.
Index Task
Retrieval Task
Partial Induction Task
x--
R--
2
n-
b
2
Out:
Out:
Figure 8: Illustration of the canonical tasks.
19

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF THEOREM
First, we will prove that whenever head H = (L, VAL, F) acts symbolically on input T. the output of the model MEMR is invariant over permutations of the input.
Lemma 1. For fixed vocabulary E = {81, 82....81111)) fttte and input 8* (Sig ... st..) such that H acts symbolically on EMB(s*), ). then
MEMB (s") MEMB (T((")) VT E S
Proof. Set EMB(s*)= = (#1,... Fn) and fix a position k S n. Now, we have that the attention weight at position & with input s* is
(nk((") = exp(L(xn, n, (k, k))
exp(L(Xn: n, Fk. π (k)))
(ox (TT(S)) (TT
exp(L((n.,,,) n, 21,))
exp(L(an. 2x(1,
because H acts symbolically on s", Now, if we look the attention vector for the last position with input s we get
Σ Wnyl (s" VAL(())
Σ
W (i) (sn((()"))
(x")")=
jSn
J.n
(2)
Σ Zjsn (n)
Ejgn (nn(()(s"))
Snn
(3)
Ejsn Wnm(j) (nnl)) (T((*))
(TTS)))
(4)
Which implies that MEMB has the same output for s" and (8*)).
Consider an arbitrary vocabulary Σ and E such that = EMB(s) = (x1, ..: Fn) is input where H acts symbolically. Since E is not constant, there exists permutation π S such that (Tos()())) * (s) (we just change what in the position j being asked). Then, from Lemma it follows that MExu * (s) or MEsu (T()) fros (Ts)).
C.3 PROOF OF THEOREM
The goal is to design an attention head Hpas such that the model MoseHor using Hoos returns the symbol oj at position with input Spos, i.c. performs an indexing operation.
We fix sequence length n > 0, and define the desired inner product condition:
K.K Rd 'Qxnii= = N TTt (n) cos (0(j - i))
for i = 0, ...,.
In order to achieve this, first we fix vector = (1/2, T/2) and the rotation angle 0 T v/n (so maximum reached just for = j). We define the query vector as:
(i) Rd "
Therefore, get query matrix QPos such that:
Qrose; 4m (j)
where e; is the one-hot encoding the symbol j. For our solution, we use the ONEHOT embedding function. On the other hand, we define constant key matrix Kpos such that for any position the key embedding of the symbol at that position is:
K ONEHOT(O.) v.
20

---

Published as a conference paper at ICLR 2026
With these choices, the result is one-layer transformer MosEHor with attention head Hpos Pos Qros. Id, TT) that always maximizes its attention at the right position. Therefore, MONEHOT perfectly copies the symbol at the position asked by input Spos, least when Spos has length n. Now, by Theorem 3 we know that Hpos can not be symbolic head for any input. We formally state its positional nature in the following lemma.
Lemma 2. For every input spos, Hpos is positional head on ONEHOT(SPOS)
Proof. We just apply the definition of positional head on y, E (x), ,
In In- J. i E [n - 1] then
L (i), n, = cos (0( - -)) [r (i), n,
Lemma 3. For the head HPos and any seos with (NEEHOT(SPos) then
MONEHor (Spos) = fPos (spos).
Proof. First, we see that logits are maximized at position j.
arg max LIxn(i), n, arg max cos(O(j - -()) j. (E(n-1]
k (e(n-1)
Therefore, the biggest attention weight is ws/ i.e
max W 1 = w.
Now, given that the value function VAL id, then we have that
Ti:
which implies that arg max SOFTMAX(a,); 0.E0
ONEHOT (xz) ay concluding the proof.
C.4 PROOF OF THEOREM 4
Suppose there exist an input SsyM (s1, 8n), an attention head H = (Q, K, U, id, πι) that acts positionally on F ONEHOT (SsoM (H)) and Mosettor fsYM. Now, by defining the function
exp(L(En, (z. M,
D(In, 2z,))
kSn
we can express the attention vector an a (SsyM (H)) as
an (ssYM(
Σ
D(E n,, 1y
jSn
We also have that
MoNeHor(Ssvvl arg max SOFTMAX(M xn,)); o.E2
(5)
=ONEHoT max D(I (1,
25
(7)
by exploiting the fact that {-,}5=1 1 is an orthonormal basis and क (a,, x.) = an In this case, i" is the position of the input $syx (H) where the answer appears. Since MONEHor solves fsyM, there is unique vector z;- that maximizes the output distribution. Therefore, there exists position & < with & * i* such that
D(En, n, Rk, k) D(En, n, xrr,
21

---

Published as a conference paper at ICLR 2026
Now, applying the positional behavior of H, get that D 2ir, d") (z Ik, i*) and D( n. 2k. k)) Dz n, Fin, k).
Finally, consider permutation -i E 5n that just interchange 8;. with 8k. We have that
On (F ( = Σ
Dan n, F5 + Dan, n, Iky )akks
which implies that
MONaHor -* = arg max (4)
(8)
0.E3
=OnEHoT (arg max (Zn, n, H3,
(9)
Sk
(10)
And therefore,
MONEHtor (8SYM (H) # MoNEttor (Ty - (8srM (H)),
but fsym (H)) = -svm = (H)) which is contradiction.
C.5 PROOF OF THEOREM
Fix sequence length n > and a vocabulary {01# #1 m #k) with m, k: 0. We want matrix Ksvm such that for each symbol there exist vector
Vo Ksv ONEHOO(HHHaaOO
for any a E (1,..., k) and KYvM ONEHoT(O; = O for any dj. On the other hand, we want matrix Qsym such that
QsYM (o, ()) Rg (2"""o "vo,
In essence, we want that vectors Vo; E TR2 codify all symbols 8 Σ such that a; appears 8. In general, this can be done for < in order to avoid overlapping in the vectors vo, after the rotation with Rg. For example, we can fix the vector Vo (1, 0) and an angle 8 i e [2,.... m] we have that Vo; Uo
m
2mt Then, for m
Having such matrices for fixed 6, we get logits function defined as
L(ONEHOT( a #), n, aa)
= xt T "K Rg n- QurmE
(11)
= U.. RT (n
(12)
-Vg. R
T 2
(13)
fort 0,...,
refer to the attention head using this logits function as H& Notice that when 0, H&SM uses the NoPE (No Positional Encoding) since the operator Rg becomes the identity. In consequence, HSv purely symbolic for any input.
In the case that 0 0, Re introduces a rotation that perturbs the query vectors. To maintain sym- metry of the inner product structure, the vectors must be centered the middle the context. This requirement naturally leads the choice | = m_1 However, in this case the H does not act symbolically given that logits depends on the positions.
To formalize construction for 0 0, prove the following lemma. Lemma 4. For the head HSym and any ssyM with |ONEHoT(asyw)) n then Mosetorr(Svva) fsvu
Proof. Using the fact that
arg max (I,,(oy),,, n,2,,, arg max 1 vo, = iE|n-1]
vE[n-1]
and the argument follows from the same idea of the proof of Lemma
22

---

Published as a conference paper at ICLR 2026
Finally, by Theorem we know that H can not be positional head. Now proceed show that it is indeed a symbolic one.
Lemma S. For every input 8SyM the head HS is symbolic on ONEHOT(8SYw!
Proof. For i,j E [m]], y ONEHOT(O and, ki, K2 E [1e - (xn((a))) ys k1) vo vo; = L n, n,
C.6 A COUNTER EXAMPLE WITH MLP LAYER
Counter Example. Fix the sequence length to be 3 and consider vocabulary consisting of symbols {a, bj and integers (0,1}; We take as input a sequence SSYM with associated embedded soft-vectors
SSYM = Hi, #2 o#), = (SSEH))) = (3/1 y22 T).
Attention Head Construction. Consider head H with logit function L satisfying:
[xx 3, yr, 1) (25( =1, Læ, 3, y/. 2) = cos(x) = =- [xx, 3, x, 0.
This can be achieved by choosing RoPE with 0 = n, and matrices Q, K such that
Qx Ky = (1, 0, 0, (for positions j 1, 2),
Kr = (0, 0, 0, 0, 0) (at the last token).
With VAL = id, the attention vector at the last position is
03 (r(o#1,), ((H2, (b#1,T), (bH2, (at,)), ((H,,)).
where 7 is defined as follows
if o #i occurs twice, if a fo never occurs, where f. d2 E and Àa J. with -
() = >a Ilo -z)), ((oyHig,
A
if o fo occurs exactly once.
Final Output Function.
Define the function F(ag) R by
F(as)a = AND(EO()(O,tt.))) z), As),
F),, A1) 1Q, (2)))
and F(as)a,# 0. The logical operations AND, OR and Eq be implemented by MLP with ReLU activations and fixed precision with sufficient depth (see Lemma A.1 in Yao et al. (202)).
C.7 PROOF OF COROLLARY
By Lemma we know that if H acts symbolically on an input SMIX (H) (oo i #i ...., -n- #), then for every permutation of SM) the answer of the one-head decoder-only transformer model MOnEHor using is invariant. However, the answer of fMx must change if permute both appearances of oj in SMi((H) (H) which means Mosettor can not solve fMIX-
On the other hand, by using the argument the proof C.4 know that if H acts positionally on Minx then we can force its model's output to change for permutations of smx where fxix is invariant.
23

---

Published as a conference paper at ICLR 2026
C.8 PROOF OF THEOREM 6
In this case we mix the ideas we used to define heads Hpos and Hy by using RoPE with frequencies. First, fix the vocabulary Σ {ar σi #1, a #2....., Tm with m, k 0. In the same way as we did for HYS we can define vectors vo, E IR² that codify the presence of 0; in symbol of Σ. On the other hand, as we did for Hpos, we set constant vector = (1/2, 1/2) Now, we define a matrix K E R [4x such that
KMI ONnEHoTTT(o [Vo o]. for any (1,,.,,), ki, while (N:HoT(o;:)KKa)) 0 for any i 11..,.. For the query matrix QMix. we define a query matrix such that
MIx ONEHoo gn ()) [R] Ea,, 2).
Finally, for angles 0, 82, we get logits function
ON Ry
MIx (NEHoT(O,##) =
T na- V. Rb, leaa-qq(ee (o() + Rgg (14)
vg
T u + cos (02(n - i))
(15)
If we set 01 (NoPE), then our logits function becomes
L(ONEHOT(o,#),11 n, ONE)oT(o;#+)),) k) = vo Va, + cos (02(n k)).
with 0 < 0nn < / (n|E)) properly chosen to define our solution head H
Lemma 6. For H 0.00 and any mix with (ONEHOT(SMIX) n then MoNcHor = fMII()MIX).
Proof. Notice that:
02 < 2n 2TTT(m/ (n|_l) = cos(2 Σ < cos (0yn)
and v lo, < <cos(27/|22) for o, + o. Then
T vo Voz + cos (02(n k)) < cos(2 [I) + cos (02(n k))
cos (82n) + cos (02(n k))
S cos (02n) +1
cos (02 jp)) +
cos (02(n ip)) + vo Vas
for all & E [n 1] 12) with 1,2, where Vo V0 Vos,
Using this and j1 < 12, we can conclude:
arg max [(x, n, 2., = arg max (E(n-1]
(vo Va - cos (0z(n - +)))
sE|n-1]
arg max (v Vo, cos (02(n i)))
arg max cos (0z 1))
Je0
j2-
And the result follows from the fact that the most attended symbol of the input the output for this architecture as seen in the proof of Lemma 3
24

---

Published as a conference paper at ICLR 2026
D APPENDIX FOR SECTION 5
D.1 SPECIALIZED HEADS FOR INDEX, INFORMATION RETRIEVAL AND PARTIAL INDUCTION TASKS
To further disentangle the role of positional and symbolic information, we train from scratch family of minimal transformer models, each consisting of a single layer, a single attention head, and single RoPE frequency. Each model is trained independently with different frequency choice, and performance is evaluated on two tasks: an index task and an information retrieval task (see Section 4 for precise definitions).
All experiments are carried out on sequences of length n 32 + where 32 tokens form the context and the additional token is the query. Both tasks are defined over mixed symbol-integer sequences. In the information retrieval task, the model must retrieve the integer associated with the query symbol, while in the index task, the model must instead return the symbol associated with the query integer (which directly encodes a position). We fix a vocabulary of 16 symbols and 32 possible integers.
We sweep over range of base angles for the RoPE:
[0.0, 0.01, 0.1, 0.25, 0.4, 0.5, 0.8, 1.0, 1.5, 2.0].
Here, 0.0 corresponds to NoPE model (no rotary position encoding), while 2.0 corresponds to RoPE configuration that completes one full angular lap over the n 33 tokens of the sequence. More generally, the effective frequency is given by
base_angle,
n
and the number of laps is defined as n8/ n8/2T.
Training is performed for 100 epochs with batch size 64, using 40,960 training samples and 20,480 validation samples. Figure reports the resulting accuracies as a function frequency. The results show clear tension: index task and information retrieval achieve their best performance different frequency ranges, indicating that the model's 's inductive bias over RoPE frequencies favors one type of task at the expense of the other.
D.2 PROOF OF THEOREM 7 U-SHAPE AND INVERTED U-SHAPE.
U-shape accuracy. Let us start with the statement Wmax is U-shaped for Hpos- As established in the logits of Hros are of the form:
the proof of Theorem
[[x,, Ti, i) cos(O(j - i)):
where 0 < 2x and j is the location of the answer. Thus, we obtain the following expression for Wmaxx (j):
exp(1)
Wmax (J) exp(cos(0(j 1)) + exp(cos(0() n)))
What happens when we go from Wmax(] (j) to Wmax( + 1)? We add exp(cos(()) and delete (O from the denumerator. That is, if cos(Oj) cos(O(j nn)), the value of W (j) decreases, and otherwise, it stays the same or increases.
have cos( > cos(O(j n1) if and only if [i| I n/.. Hence, starting from j = when [i] is much smaller than j n//. the function Wmax is decreasing, having local maximum j = Wheh j is about n/2, the absolute values |j] and | - become equal and then the first one becomes larger, and afterwards the function Wmax starts increasing, reaching another local maximum j = n in the form of a U-shape, as required.
Since <
Inverted U-shape accuracy. We consider simple RoPE head with one angle, equal to T/nn, as the one used for the empirical results depicted in Figure 3 where the inverted U-shape is observed in the retrieving (symbolic) task. We will rigorously prove this is the case in simplified scenario. The simplified task will be the following: we have two possible tokens Tp, Ti and we consider prompts Ti Fn consisting of copies of 7p, except at positions where it has Ti. These positions
25

---

Published as a conference paper at ICLR 2026
the query (j n) and some & the requested token in position L, which means that we expect distribution of attention wights a; (j L,..., m) with small values except for a peak j L. This must hold for every prompt of this form.
1. The task for the attention head is to detect
As we have empirically demonstrated, RoPE cannot be expected behave symbolically for this choice of angle (it is too big) so we accept a weak solution of the symbolic task: the logit sequence Ats Myy Am negative, except for a positive value when we reach the desired token TI. We also want that the logits are bounded away from O (positive or negative) as much as possible. Since the query position fixed, if q = QIn IR2 is the query vector, then we can write the logit function as
[ n, J. = L(x, = xKU
(U is the RoPE matrix for angle π n/n). For any embedding vector R let = E R2. Let 2 be the embedding vector of Tú for = 0, 1. We make the simplifying assumption that q, z and zi have norm -te discussion can be adapted to a more general case, but choose keep the computations simple while still exhibiting the main features of the inverted U-shape phenomenon.
Let y(a)) be the angle between y and the vector x' æK for E Rdd Then the formula for the RoPE logit and the assumption |/lll = 1 gives
[xx ||=='|| cos(mj/n (1)))
In particular, our assumption that this logit function gives an approximate (in the sense of sign) solution to the symbolic task described above, with logits bounded away from 0 as much as possible, gives
(=0) -
T(21) x/2
so that
{ [CoS(TT] (Tj / T 7/2)
if = 0
[(,,j)
[cos(T]/mm x x/2)
i
(here we used the simplifying assumption || = 1). We note the symmetry L(21.j) = (
Let
mi(i) = (xp(((zz,,))
and note that vo({)ui (f) = -1. Define
S= n-
Σ Po(
Recall that the position of the sought token is l. Then the attention weight at j = é is maximal (due to the sign of the logits -this is the only positive logit for the given prompt) and is
v (t)
W mmxx
=
S
+ (1() - (c
This is the only peak of the attention weights for the prompt with sought token position é. Finally, we can give a rigorous proof of the inverted U-shape phenomenon in this setting. Theorem (Inverted U-shape). Assume n 2 5. As functio E, the sequence Wmax(E) is increas- ing for 0 < < n and decreasing for n/2 < < n.
Proof. We can use the same formula defining Wmaxx (1) on real variable t instead of the discrete variable E, and then restrict back to integers l.
Let [()
1/umaxX( (1). It suffices to prove that ('(1) negative for < < n/2 and positive for n/2 < L <
From the equation vo(f)u we have
((1) 1 + Svo (1) un(1)
Then
f'(1) Su)(1) (1)(()))) 16(1) (S (1))
26

---

Published as a conference paper at ICLR 2026
If n 2 5 and t l is an integer, the parenthesis is strictly positive because substracting different sumands from (here we use n is odd). On the other hand, we explicitly compute r/(()):
(t(()
_ vo( sin(nt/ti + T/2).
n
As po(t) > always, the sign of the previous expression is negative for Tt/n E 70, and positive for nt/n (T/2, x). The result follows.
27

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS FOR REALISTIC MODELS
E.1 LLAMA-3.2- 1-B-SSTRRUUI
Selected PosifooaiiSSmmblll
Posioonal
H45
Syrbbelc
1.0
1.00
1.00
20:0
0..00
0.50
0.50
0.25
Syntolle S0om
06
344
L-52 H=1
L-15,
0.4
1.00
1.00
40
0.00
0.50
0.50
0.25
0.00
0.8
20
(a) Sysibional-Symmoli Plane across Frequencies.
Visualization of attention heads from the (Left) Each point represents a head in the posi-
met aetta-11ae aallamm/iia /Ll ama-3. .2.22262 InB-Instuuct model. Konal-symbolic plane, labeled by its layer and head index. (Right) For the same heads, we plot their positional and symbolic scores function of RoPE frequencies.
Query block 1/258
Posilional
Positional Score
1.0
150
2033
Hosol
0.8
20
Symboic
12
0.0
0.5
3.0
10
Symbolic
Symbolic Score
150
40
100
Laye Dosth
0.4
0.6
0.8
1.0
0.0
0.5
1.0
10
Positonal Score
Score
Layer I
(b) All heads, fixed first block. (Left) Each head in the cesitional-symbolie plane. (Middle) Histogram showing predominance of symbolic over positional behavior. (Right) Hcatmaps of positional and symbolic scores for each head across layers.
Query block 1/256
Query block 128/256
Query black 258/256
Smeole Score
0.75
0.75
0.75
0.25
0.25
0.25
0.00
Soore
Positionl
0:50
200
0.54 0.5.0
1.50
150
150
Count
100
50
50
0
0.0
0.5
1.0
0.0
0.5
10
0.0
0.5
1.0
Score
Scoree
(c) Heads when the block moves (1 9). (Above) Heads in the ceatiional-symuule plane for each block position. (Bellow) Histograms showing how positional and symbolic scores change as the relevant block moves.
Figure 9: Positional-Symbon Plane across Heads. 16 Layers, 32 Heads per layer. Since the model uses a head dimension of 64, this corresponds to 32 RoPE frequencies.
28

---

Published as a conference paper at ICLR 2026
E.1.1 LLAMA-3.2-3B 3.0-------B--
Selected Heads SyeiioraillSmmblli Plane
L-d. Hed
Pestonal
L-5. H-S
Syritotc
L=S. H
1.0
1.00
1.00
1.00
0.75
0.00
0.30
0.25
0.25
0.25
Symbote Scom
4-1
d0
50
25
L-12, H-1
L-15, H+0
L-15 HH-20
0.
1.00
1.00
1.00
0.75
0.00
0.50
0.25
0.25
0.25
0.2
0.4
0.8
26
50
25 Pregusncy
50
Pos vonitoaal Score
Prequency
Fisgueay
(a) Positional-Symbolie Plane across Frequencies.
Visualization of six attention heads from the (Left) Each point represents a head in posi- (Right) For the same heads, we plot their positional
peeta-liama LL ama-3. 3.-a--..--BBI--- Ins-nnstruue -.-----IInttuuc tional-symbolic plane, labeled by its layer and head index. and symbolic scores as function of RoPE froquencies.
model.
Query block 1/255
Posttional
Positional Score
200
Heaf
100
0
0.0
0.5
3.0
Symbrllic
Symbollc Score
2033
Hexdl
15
Laye: Dupth
2003
10
0.0
0.4
0.8
0.8
1.0
0
0.0
0.5 Soore
1.0
10
Positoona Score
Layer
(b) All heads, fixed first block. (Left) Each head in the positional symbolic plane. (Middle) Histogram showing predominance of symbolic over positional behavior. (Right) Heatmaps of positional and symbolic scores for each head across layers.
Query block 1/256
Query block 128/256
Query black 256/256
1.00
1.00
1.00
Simbalc Score
0.75
0.75
0.75
0.50
0.25
0.25
0.00
0.00
0.00
0.0
0.5
0.0
0.5
1.0
0.0
0.5
Positional
Positional
Postiionl
QP/8.8
250
200
150
150
0.67
150
0.51
0.69
100
100
100
50
90
0.0
0.5
0.0
0.5
10
0.0
0.5
Score
Score
Score
(c) Heads when the block moves (1 9). (Above) Heads in the positional-symbolic plane for each block position. (Bellow) Histograms showing how positional and symbolic scores change the relevant block moves.
Figure 10: Positional-Symbolle Plane across Heads. 28 Layers, 24 Heads per layer. Since the model uses a head dimension 128, this corresponds to 64 RoPE frequencies.
29

---

Published as a conference paper at ICLR 2026
E.2 QWEN2-1.55 1.--2-1..B--AA --B--SSTTUCT
Symbolie
Selected Heads Posthional/yymbille Plane
Lnd, Hin
Lick Hr2
Ln6, HeO
1.00
1.00
1.00
g12
0.75
0.75
0.75
0.00
0.50
0.00
30.0
Symbo
a21
8o
L=12,
H-2
1.00
1.00
0.2
0.50
0.50
0.50
0.25
0.25
0.25
0.0
0.00
0.00
0.00
0.0
0.2
0.4
0.6
0.8
1.0
25
00
25
50
25
50
Positinnll Score
Frequency
Frequency
Preq annryyyyy
(a)
Positional-Symbolie Plane across Frequencies. Visualization of attention heads from the qwen/ / Qwen2-1 50-55B----- rustr model. (Left) Each point represents head in the positional-symbolic plane, labeled by its layer and head index. (Right) For the same heads, we plot their positional and symbolic scores function of RoPE frequencies.
Query block 1/256
Positional
Postionnl Soore
1.0
100
0.6
60
10
0.6
Symbolic
Symbolic
Symbalic Score
0.64
Layer Depth
50
0.0
0.1 Positional Score
1.0
0.0
(b) All heads, fixed first block. (Left) Each head in the positional-symooii plane. (Middle) Histogram showing predominance of symbolic over positional behavior. (Right) Hcatmaps of positional and symbolic scores for each head across layers.
Query block
Query 126/256
Query block 256/256
1.00
1.00
1.00
symbolc Sooe
0.75
0.75
0.50
0.50
0.50
0.25
0.00
0.00
0.00
0.5
0.5 Positional Soore
Positional Soore
Posttionll Soore
125
0.54m02.2
125
125
100
0.54
75
75
90
50
25
25
Score
Score
(c) Heads when the block moves (1 to 9). (Above) Heads in the positional-symbolic plane for each block position. (Bellow) Histograms showing how positional and symbolic scores change the relevant block moves.
Figure II: Positional-Symbolic Plane across Heads. Layers, 12 Heads per layer. Since the model uses head dimension of 128, this corresponds to 64 RoPE frequencies.
30

---

Published as a conference paper at ICLR 2026
E.3 QWEN2-0.5B -
Symbooie
Selected Heads Pasttiona//ymmbill Plane
Led, Hen
Li3k H+2
Ln6, HeO
1.00
1.00
1.00
0.75
0.75
0.75
0.00
0.50
0.00
Symbo
1.00
1.00
0.2
0.50
0.50
0.50
0.25
0.25
0.25
0.0
0.00
0.00
0.00
0.0
0.2
0.4
0.6
0.8
1.0
Positionll Score
Frequencyy
Frequency
Freq anquanyy
(a)
Positional-Symbolic Plane across Frequencies. Visualization of attention heads from
qwen/ / Qwwn2-0...... 0.. 5B- Inst ruct model. (Left) Each point represents head in the positional-symbolic plane, labeled by its layer and head index. (Right) For the same heads, we plot their positional and symbolic scores function of RoPE frequencies.
Query block 1/256
Positional
Posiionnl Soore
1.0
100
0.6
0.6
Symbolic
Symbollc
Symbalic
0.57
Head
Layer Death
80
0.4 Posilional
1.0
0.0
(b) All heads, fixed first block. (Left) Each head in the positional-symooii plane. (Middle) Histogram showing predominance of symbolic over positional behavior. (Right) Hcatmaps of positional and symbolic scores for each head across layers.
Query block
Query 126/256
Query block
1.00
1.00
1.00
Symbolc Sooe
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.00
0.00
0.00
1.0
Positional Soore
Posttionll Score
Postional Score
4.0
100
100
Score
Score
(c) Heads when the block moves (1 to 9). (Above) Heads in the positional-symbolic plane for each block position. (Bellow) Histograms showing how positional and symbolic scores change the relevant block moves.
Figure 12:
Positional-Symbolic Plane across Heads. 24 Layers, 14 Heads per layer. Since the model uses head dimension of 64, this corresponds 32 RoPE frequencies.
31


Reviews:
[{"review_id": "WevcmbKyTr", "reviewer": "Reviewer_x2bi", "summary": "Authors formalize definition of positional and symbolic heads, define metrics, canonical tasks, prove several theorems and perform analysis of several open weight and toy models.  Characterization of a model by its positional-symbolic profile is a potentially powerful tool to analyze how particular models behave on different tasks.", "questions": "Is score input dependent? Could the head act symbolically on one sequence and positionally on another?\nHow the score is affected by block boundaries?\nAre you considering only swaps but not all permutations of input sequence? How does this affect score?\nHow expensive is calculation of such scores in practice?\n\nline 389. nit: Frecuency ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Theoretical Formalization: The authors provide formal definitions for when an attention head behaves purely positionally (depending only on position, regardless of token value) or purely symbolically (depending only on the token value, regardless of position) and prove several theorems.\n\nIntroduce a novel metric is introduced to quantify the extent to which any given attention head exhibits positional or symbolic behavior on a specific input. This metric allows for granular analysis, creating a positional-symbolic profile for the entire model, visualized in the positional-symbolic plane.\n\nApplying the metric to real-world LLMs (GEMMA-2, QWEN-2, and LLAMA-3 families) reveals a sharp correspondence between RoPE frequencies and attention head behavior.\n\nVery interesting finding about high frequencies in RoPE", "weaknesses": "Analysis done on limited set of tasks for real world LLMs ( binding task for real models)\nIt is not clear how reliable and computationally expensive metric in practice", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEe4yrXmaO", "reviewer": "Reviewer_uiez", "summary": "This work aims to study the positional and symbolic properties of the Rotary Positional Encodings (RoPE). To this end, the authors provide formal definitions of the two properties and introduce a metric to assess the extent to which a given head behaves positionally or symbolically on a given input, which enables analyzing the positional and symbolic behaviors of a given model at different levels of granularity, and also visualizing of the positional-symbolic profile of the model.\n\nFirst, they apply the metric to the transformer-basd LLM, Gemma-2m,  and reveal that RoPE with a high frequency tends to be very positional while a low frequency corresponds to symbolic patterns. \n\nSecond, they introduce two tasks, Index and retrieval,  that can be solved by pure positional (symbolic) headsusing a 1–layer attention-only transformer. Empirical studies show that a head can only act positionally when it uses fairly large frequencies. If we force it to use lower frequencies, its performance drops.Similarly, attention heads can perform well on intrinsically symbolic tasks if given access to low frequencies.", "questions": "* the key concept of frequency is not introduced in the preliminary, which hampers the understanding of the subsequent sections. I would suggest introducing the basic background of the RoPE in section 2.\n* in line 145, *\"its logit sequence is invariant under permutation of the key vectors.\"* I would like to confirm if the key vectors or input vectors (with regard to $x$) are shuffled? I asked this since self-attention uses the input vectors as key, query, and value. I am confused by performing the permutaion of only the key vectors. \n* some figures are confusing. What are the score, frequency IDs, and the two line plots in Figure 1C? What is the number, e.g, 1->32, in Figure E? What are the three points in Figure F?\n* In Figure 3A, why a low frequency results in a U-shaped acc patterns? Why an encoder with a low frequency can deal with indexes of the beginnig positions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This work provides a formal definition of positional and symbolic behavior of attentions and introduces metrics to quantify the two scores for LLMs, which would be useful for following studies\n* They provide both theoretical analyses and empirical studies to support their findings. Extensive experiments have been conducted, including multiple LLMs, studies across real LLMs and toy transformers.  \n* The presented tasks, index and retrieval, are interesting, which offer a doable way to study the functions of positional and symbolic properties", "weaknesses": "* Some key related papers are not discussed in the current manuscript, which strongly undermine the novelty and contribution of this work. \n[1] and [2] have already discussed the separation of positional and semantic encodings. More importantly, [3] reveals the attention weights in transformers can be divided into two components: positional weights and semantic weights. It further introduces two properties, locality and symmetry, to measure the two behaviors. Basically, a high (or low) locality corresponds to positional (or symbolic) behaviors. In other words, the locality here can depict the positional-symbolic profile of a language model, i.e., a high locality is aligned with the RoPE with a high frequency. However, the authors did not discuss the main distinction of this work compared to prior studies, which undermines the contribution of this work.\n\n* The presentation requires further improments. (1) This work aims to study and interpret the patterns of RoPE, but the authors do not provide a basic background in the preliminary section while they did present the background of the transformer architecture. I would suggest introducing the basic concepts of RoPE, e.g., what is the frequency. (2) I appreciate that they provide dense figures, but some figures are hard to understand. The lack of clear titles and axis texts lead to a bad readability (see questions below).\n\n* They have conducted extensive theoretical analyses and empirical studies, but it is not easy for me to derive the main conclusions from this manuscript, and it is quite vague about how to use their findings to improve the current RoPE \n\n\n\n[1] Constituency Parsing with a Self-Attentive Encoder. ACL 2018\n  \n[2] Rethinking Positional Encoding in Language Pre-training. ICLR 2021 \n\n[3] The Locality and Symmetry of Positional Encodings. EMNLP 2023 Findings", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hA4c89uPtD", "reviewer": "Reviewer_tJZ7", "summary": "This paper investigates how Transformers decouple positional and symbolic information in attention heads. It provides formal definitions for positional and symbolic behavior and a theorem showing they are mutually exclusive unless the attention pattern is uniform. The authors introduce a metric to quantify the amount of positional and symbolic information, using it to analyze RoPE models. The main finding is that low-frequency angles from RoPE are used for symbolic tasks, while high-frequency angles are used for positional tasks. This is demonstrated by training a toy model in canonical positional and symbolic tasks. Moreover, this metric is also applied to pre-trained models, showing that it can be used to decouple positional and symbolic behavior in attention heads.", "questions": "**Questions:**\n1. What are the limitations of basing the theory on logits but the empirical metric on softmax weights?  \n2. Why use blocks? Is it just for efficiency? What's the impact of this approximation?  \n3. What is the model's accuracy on the binding task? Are you only analyzing instances where the model was correct?\n4. Can you add statistical tests (e.g., correlation) for the claims in Lines 268-269 (layer depth) and Lines 271-272 (negative correlation)?  \n\n**Suggestions:**\n- \"Fine-Tuning Enhances Existing Mechanisms\" (https://arxiv.org/abs/2402.14811) is a relevant paper that could be cited. They identify the circuit that solves entity tracking as having heads that handle positional information and others that handle content information.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The main takeaway is very interesting: attention heads use RoPE frequency as a tool to implement either positional or symbolic functions. \n- The paper has originality in providing a theoretical analysis of the problem.\n- The choice of training toy models trained in canonical tasks is a clear way to test the positional and symbolic functions of attention heads.\n- The experiments are done in multiple pre-trained LLMs, which shows how generalizable the proposed metric is.", "weaknesses": "- Some of the claims lack statistical tests:\n    - The claim that early layers are positional and late layers are symbolic (Lines 268-269) is based on visual inspection, not a statistical test.  \n    - The \"negatively correlated\" claim (Lines 271-272) also needs a proper correlation test with a p-value.  \n- The theory is on logits, but the metric uses softmax weights. The paper doesn't discuss this switch. Softmax is a nonlinear function and has a relativistic behavior, ie, when analyzing a token $d$ attention to another token $s$, the logit value itself is not the only important component, but how large this logit is in comparison to the other tokens $j \\leq d, j \\neq s$. I wonder if this property would somehow impact the analysis.\n- The usage of blocks is not motivated. Clearly, it is important for efficiency, but this is not discussed in the paper, which makes the transitions between paragraphs in Section 3.3 a bit rough to read. Block sizes were not discussed either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFfdqY1HEU", "reviewer": "Reviewer_Fhmh", "summary": "The authors formalise what it means for a head to behave positionally or symbolically. They then measure and study in detail both mathematically and empirically how positional and symbolical heads function. The paper focuses particularly on RoPE and performs interesting experiments to support their claims.", "questions": "I was wondering if the authors have an opinion given their analysis on the proposed method of p-RoPE (Barbero et al. 2024) where the p% lowest frequencies are set to spin to 0 to construct robust symbolic channels? \n\nOn a related note, do the authors have an idea of how the study could point towards possibly improving RoPE or for example the understanding of the choice of the wavelength hyperparameter. This is particularly important as many frontier models have been choosing different wavelengths for RoPE to deal with long context.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I found the paper to be insightful. The formalisation of positional / symbolic heads using invariance and equivariance is clever and the resulting analysis is very interesting. I found Section 5 to be a good sanity check of the theory and intuition. \n\nI find this work to be a really strong and useful deep dive into RoPE consolidating and improving our existing understanding of how RoPE functions and how internal frequencies are used. As RoPE is the current most popular LLM PE, I find this paper to be on an important topic.", "weaknesses": "The main weakness I see is that the paper is dense at times especially in notation. It might be useful to add a small summary at the end of the sections to help convey what the authors believe should be the main message. This is mostly a minor suggestion to help with readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)