On the Emergence of Position Bias in Transformers
Authors: Xinyi Wu
Yifei Wang
Stefanie Jegelka
Ali Jadbabaie
Venue: ICML 2025 poster (2025)
Keywords: attention mechanism
transformers
position bias
positional encoding
deep learning theory
Primary area: deep_learning->attention_mechanisms
TL;DR: We show that causal masking biases attention toward earlier tokens as layers deepen, while relative positional encodings balance distance-based decay with early-position dominance, providing deeper insights into position biases in transformers.

Abstract: Recent studies have revealed various manifestations of position bias in transformer architectures, from the "lost-in-the-middle" phenomenon to attention sinks, yet a comprehensive theoretical understanding of how attention masks and positional encodings shape these biases remains elusive. This paper presents a graph-theoretic framework for analyzing position bias in multi-layer attention.  Modeling attention masks as directed graphs, we quantify how tokens interact with contextual information based on their sequential positions. We uncover two key insights: First, causal masking inherently biases attention toward earlier positions, as tokens in deeper layers attend to increasingly more contextualized representations of earlier tokens. Second, we characterize the competing effects of the causal mask and relative positional encodings, such as the decay mask and rotary positional encoding (RoPE): while both mechanisms introduce distance-based decay within individual attention maps, their aggregate effect across multiple attention layers—coupled with the causal mask—leads to a trade-off between the long-term decay effects and the cumulative importance of early sequence positions. 
Through controlled numerical experiments, we not only validate our theoretical findings but also reproduce position biases observed in real-world LLMs. Our framework offers a principled foundation for understanding positional biases in transformers, shedding light on the complex interplay of attention mechanism components and guiding more informed architectural design.

On the Emergence of Position Bias in Transformers
Xinyi Wu Yifei Wang?
Stefanie Jegelka 32 Ali Jadbabaie
Abstract
weigh input elements based on their relevance, enabling ef- ficient handling of complex dependencies (Bahdanau et al., 2015; Kim et al., 2017). However, despite this widespread success, many questions remain unanswered regarding how these mechanisms process information and the artifacts they may introduce. Developing a deeper theoretical understand- ing of their inner workings is essential not only to better interpret existing models but also to guide the design of more robust and powerful architectures.
Recent studies have revealed various manifestations of position bias in transformer architectures, from the fost--n-thhemmddle phenomenon to attention sinks, yet comprehensive theoretical understanding of how at- tention masks and positional encodings shape these biases remains elusive. This paper presents a graph- theoretic framework for analyzing position bias in multi-layer attention. Modeling attention masks as directed graphs, we quantify how tokens interact with contextual information based on their sequential po- sitions. We uncover two key insights: First, causal masking inherently biases attention toward earlier posi- tions, as tokens in deeper layers attend to increasingly more contextualized representations of earlier tokens. Second, we characterize the competing effects of the causal mask and relative positional encodings, such as the decay mask and rotary positional encoding (RoPE); while both mechanisms introduce distance-bart based de- cay within individual attention maps, their aggregate effect across multiple attention layers-couplee with the causal mask leads to a trade-off between the long- term decay effects and the cumulative importance of early sequence positions. Through controlled numeri- cal experiments, we not only validate our theoretical findings but also reproduce position biases observed in real-world LLMs. Our framework offers a princi- pled foundation for understanding positional biases in transformers, shedding light on the complex interplay of attention mechanism components and guiding more informed architectural design.
One particularly intriguing aspect that demands such the- oretical investigation is position bias, i.c., the bias of the model to focus on certain regions of the input, which sig- nificantly impacts the performance and reliability trans- formers and LLMs (Hou et al., 2024; Wang et al., 2024; Zheng et al., 2023). For instance, these models often suffer from the "lost-in-the-mideee -i-theemmddlee problem, where retrieval ac- curacy significantly degrades for information positioned in the middle of the input sequence compared to information at the beginning or end (Guo & Vosoughi, 2024; Liu et al., 2024; Zhang et al., 2024). Similarly, in-context learning is highly sensitive to the order of illustrative examples: simply shuffling independently and identically distributed (i.i.d.) ex- amples can lead to significant performance degradation (Lu et al., 2022; Min et al., 2022; Zhao et al., 2021). Moreover, recent research has also revealed that attention sinks (Gu et al., 2025; Guo et al., 2024; Xiao et al., 2024) posi- tions that attract disproportionately high attention weights arise at certain positions regardless of semantic relevance, suggesting an inherent positional bias.
These empirical findings suggest that while transformers ef- fectively encode and process positional information through the combined use of attention masks and positional encod- ings (PEs) (Fang et al., 2025; Wang et al., 2024), these design elements also appear to introduce systematic posi- tional biases, often independent of semantic content. This raises a fundamental and intriguing question about the role of positional information in attention mechanisms:
1. Introduction
The attention mechanism is central to transformer architec- tures (Vaswani et al., 2017), which form the backbone of state-of-the-art foundation models, including large language models (LLMs). Its success lies in its ability dynamically
How do attention masks and positional encodings shape position bias in transformers?
'MIT IDSS & LIDS 'ITT CSAIL 'TU Munich. Correspon- dence to: Xinyi Wu <xinyiwu@ mit.edu?.
To address the question, we develop a graph-theored frame- work for analyzing attention score distributions in multi- layer attention settings. Related to the empirical tool of attention rollout (Abnar & Zuidema, 2020) and building
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Emergence of Position Bias in Transformers
Table Summary of our results and their connections to empirical observations on position bias reported the literature.
Empirical Observations on Position Bias
Our Results
Positional information induced by the causal mask (Barbero al., 2024a; Kazemnejad al., 2023; Wang al., 2024)
Theorem 4.1, Section 5.2
Decay effects induced by relative PEs (Su et al., 2023)
Lemma 4.4-4.6, Section 5.1
Interplay between the causal mask and relative PEs (Wang et al., 2024)
Theorem 4.5-4.7, Section 5.1
Attention sinks (Gu et al., 2025; Xiao et al., 2024)
Theorem 4.1.4.3, Appendix K.4 Section 5.2
The "lost-in-the-middle" phenomenon (Guo & Vosoughi, 2024; Liu et al., 2024)
upon the theoretical analysis of attention from a graph per- spective (Barbero et al., 2024a; Wu et al., 2023; 2024), we model attention masks as directed graphs, enabling rigorous mathematical analysis of attention patterns. This approach proves particularly powerful for studying multi-layer atten- tion mechanisms, it allows us to precisely quantify how each token's 's contextual representation is composed from in- formation at different positions in the sequence. By tracking the information flow through the attention layers, we can systematically examine how positional biases emerge and propagate across layers, providing insights into the complex interplay between attention masks, PEs, and the network's depth.
2. Related Work
Position bias in transformers Position bias in trans- former models has emerged as a critical challenge across diverse applications. In information retrieval and ranking, Guo & Vosoughi (2024); Hou et al. (2024); Liu et al. (2024); Zheng et al. (2023) demonstrated systematic degradation of performance due to positional dependencies. Similarly, in in-context learning, model performance can vary dramat- ically based solely on the order of examples (Fang et al., 2025; Lu et al., 2022; Min et al., 2022; Zhao et al., 20. While mitigation strategies such as novel PEs (Kazemnejad et al., 2023; Zhang et al., 2024), alternative masking tech- niques (Fang et al., 2025; Wang et al., 2024) and bootstrap- ping (Hou et al., 2024) have been proposed, they remain task-specific and empirically driven. This gap between em- pirical observations and theoretical understanding highlights the need for rigorous analysis of how transformers process and integrate positional information through attention.
Our contributions are summarized as follows:
We develop a graph-theoretic framework that unifies and advances understanding of position bias in trans- formers, offering deeper insights into diverse empirical observations documented in the literature (Table I).
We show that causal masking in transformers inher- ently biases attention toward earlier positions in deep networks. This happens as tokens in deeper layers attend to increasingly more contextualized representa- tions of carlier tokens, thereby amplifying the influence of initial positions. We derive analogous results for the sliding-wing wwiddnggwwidowwwwooo mask and the prefix mask, highlighting the generalizability of our framework.
The effect of attention masks and PEs in transformers The role of attention masks and PEs in transformers has been explored from various perspectives. Yun et al. (2020b) analyzed the function approximation power of transformers under different masking schemes, while Wu et al. (2023; 2024) investigated the role of attention masks in mitigating rank collapse. Moreover, Gu et al. (2025) empirically exam- ined how attention masks affect the emergence of attention sinks. As for PEs, Kazemnejad et al. (2023) studied their role in length generalization, and Barbero et al. (2024b) ana- lyzed RoPE's use of feature dimensions. Additionally, Wang et al. (2024) empirically observed that both causal masking and RoPE introduce position dependencies in L.Mss. De- spite these advances, fundamental questions remain about the mechanisms through which attention masks and PEs enable transformers to process and integrate positional in- formation, as well as the nature of the systematic positional biases that emerge as a result.
We uncover a nuanced interaction between causal masking and relative PEs, such as decay masks and rotary positional encoding (RoPE). Our findings high- light a trade-off in multi-layer attention networks, where local decay effects within individual layers are counterbalanced by the cumulative importance of early sequence positions. These results provide deeper understanding of how PE and masking interact in deep attention-based architectures, with design implications about how to balance local and global context.
We support our theoretical findings with experiments, empirically validating that deeper attention layers am- plify the bias toward earlier parts of the sequence, while relative PEs partially mitigate this effect. Through carefully controlled numerical experiments, we fur- ther investigate the role of data in shaping position bias and how causal masking implicitly leverages positional information.
3. Problem Setup
Notation We use the shorthand [m] == (1...., n). For a matrix M, we denote its i-th row by M,. and its j-th column by M.j- Throughout the analysis in the paper, we formalize the attention mask to be a directed graph G. Formally, we represent directed graph with N nodes by g
2

---

On the Emergence of Position Bias in Transformers
and let E(g) be the set of directed edges of G. A directed edge (i.t) F(G) from node j in 9 means that in the attention mechanism, token j serves as a direct context for token i or token attends to token j. The set N₁ of all neighbors of node is then {k: : (ke i) E E. We say a node u is reachable from node u in a directed graph G if there is a directed path (u, n1); R22); (ng; v) from u to v. In the attention mechanism, this means that token u serves direct indirect context for token v.
Then applying the decay mask is essentially
A decay = softmax (4) W (x(" + D)- (2)
Note that while the decay mask formulation follows AL- iBi (Press et al., 2022), it can be generalized to more com- plex variants such as KERPLE (Chi et 2022).
Rotary Positional Encoding (RoPE) Another way to en- code the relative positional information is through RoPE (Su et al., 2023), which applies rotation to query and key em- beddings by an angle proportional to the token's 's position index within the sequence. Formally, the rotation operation applied to each query or key Xs,: ((e.K) can be written as
Furthermore, we will be using the following graph- theoretc terminology (see Figure 1 for a schematic illustration):
Definition 3.1 (Center Node). A node e from which every node in the directed graph G is reachable is called a center node.
(K)K = X.,. WL R
3.1. (Masked) Attention Mechanism
where
Given the representation X E RNxd of N tokens, the raw attention score matrix is computed as Z XXQ(XWx)" / where Wo. WK E Raxd' are the query and the key matrix, respectively, and vdok is a tem- perature term to control the scale of raw attention scores. Without loss of generality, we assume dok ysis. To enforce masked attention, we create sparse atten- tion matrix A E RNN based on Z whose sparsity pattern is specified by a directed graph G: we normalize Zig among all allowed token attention interactions ik) E EG) such that if (j, :) E E(g),
ros sin i0
sin coe il
0
0
cus 69
sin 10
0
R
sin
CDs 19
DOG 2/5
clss 19422
in our anal-
is the rotation matrix with a set of pre-defined base rotational
angles θ = {00 < 01 S .... S 02/2]. Then the raw attention scores under RoPE ZROPE become
(ZRoPE)ss (X., (X,
= =X,, R 30
exp(Z.;)
Ai = softmaxg(Zs)) = EkEN, exp(Zit)
which distorts the original raw attention scores based on the relative token distances. The final attention scores under RoPE are calculated as A RoPE = softmaxg (RRPEE)
and Ay = 0 otherwise.
3.2. Attention Update
4. Main Results
For our analysis, we consider single-head (masked) self- attention networks (SANs). The layerwise update rule can be written as
In the transformer model, the attention mechanism is the sole module that allows tokens interact with one another and incorporate contextual information from the sequence. It iteratively refines the contextual representation of each to- ken across layers, allowing information to flow and accumu- late based on relevance. This concept of contextualization through attention has its origins in the development of at- tention mechanisms, which predate transformers (Bahdanau et al., 2015; Kim al., 2017). From the perspective of con- textualization, the attention mechanism can be expressed in the following form (Kim et al., 2017):
(0)) softmaxgun (x /V
((++)== (1
(1)
where W(O) E dxd is the value matrix. For simplicity. throughout the paper, we assume that d = d' and g(t) = g. Yet the results can be easily generalized to the case where masks are time-varying and satisfy regularity conditions.
3.3. Relative Positional Encoding
N
x((+1)
(A
(0) Dis
X W
(() (4)
w((0
Decay Mask The decay mask represents the relative dis- tance between two tokens by introducing an explicit bias favoring more recent tokens. Formally, it can be written as:
F=1 p(U x|
(
where 2; is a categorical latent variable with a sample space [1....,N] NN that selects the input X,.: to provide context for token i. In this formulation, (²) represents the attention
- (i - -m j <i
Dij
0
otherwise.
3

---

On the Emergence of Position Bias in Transformers
matrix layer 1, (z; = X x denotes the cumulative probability of selecting input token the context for token i at depth t, and (-) is learned transformation function.
This probabilistic formulation reveals two key aspects of the attention mechanism: acts as both a context selector and a feature aggregator. As a selector, assigns probabilities pp(() that quantify the relevance of each token to target token i at depth t. As an aggregator, combines these selected contexts weighted by their respective probabilities IP(() to form the contextualized representation X (4), Since position bias fundamentally manifests as systematic preferences in how tokens select and incorporate context from different positions, analyzing the attention mechanism's 's behavior is crucial for understanding these biases. By examining how attention masks and PEs affect the probability distribution p(t), investigate how position-dependent patterns emerge and propagate through multi-layer attention in trans- formers.
Causal
Sliding-window
Prefix
Figure 1: Three types of attention masks and their correspond- ing directed graphs G used in the analysis (self-loops are omitted for clarity). directed edge from token j to i indicates that i attends to j. The center node(s) (Definition 3. 1). highlighted in yellow, represent tokens that can be directly or indirectly attended to by all other tokens in the sequence. As depicted in the top row, the sapp-theeretic formulation captures both direct and indirect contributions of tokens to the overall context, providing a compre- hensive view of the token interactions under multi-layer attention.
Attention rollout The quantity analyzed in our frame- work, (z, = j x coincides with the attention roll- out metric proposed by Abnar& & Zuidema (2020). However, our work was developed independently. and the motivation is fundamentally different: whereas attention rollout serves as empirical visualization tool applied to specific input sequences, we treat the same quantity as a theoretical object and analyze it across arbitrary inputs to reveal the model's inductive biases.
Through the graph-theoretic view, our first result states that for causal mask G, as tokens in deeper layers attend to increasingly more contextualized representations of earlier tokens, the context of each token converges exponentially toward the first token in the sequence.
Finally, we adopt the following assumptions in our analysis:
A1 There exists C E R such that
max EEN
[lxx 11,
||
12) sc.
Theorem 4.1. Let be the causal mask. Under A1-A2, given X E Nxx, for every token i E [N].
A2 The sequence {Il I Wi
| is bounded.
lim p(t = 1|x((0))= =1. L+2
Al assumes that the key and query weight matrices are bounded, which is crucial for efficient attention computation in practice (Alman & Song, 2023), whereas A2 is to ensure boundedness of the node representations' trajectories (1) for all 2 0 (Wu al., 2024).
Moreover, there exist 0 < C, < where Ne such that
(= 1 s C(1 - ()-1)))'
for all <j < and 12 0.
4.1. Attention Masks: A Graph-Theoretic View
Theorem 4.1 reveals that in multi-layer causal attention, positional bias toward earlier sequence positions intensi- fies with depth - regardless of semantic content. This phe- nomenon arises from the nature of multi-layer attention: starting from the second layer, tokens no longer attend to raw inputs but instead contextualized tokens, i.e., repre- sentations transformed by prior attention layers. Combined with the sequential structure of the causal mask, this iter- ative process amplifies the role of earlier tokens, they influence later ones not only as direct context but also in- directly through intermediate tokens along the path. We discuss a few intriguing implications below.
We first analyze the case without PEs, focusing on the effect of attention masks. A graph-theoretic perspective offers a powerful framework for analyze multi-layer attention: the flow of attention across tokens can be represented as paths in directed graph defined by the mask, where each path captures how information is transmitted between tokens (see Figure for an illustration). The number of steps in a path corresponds to the number of layers. By accounting for all such paths, we can quantify the cumulative influence of each token in the context computation of other tokens.

---

On the Emergence of Position Bias in Transformers
The role of softmax The key property that leads to the above result is that the softmax operation in the attention mechanism cannot fundamentally disconnect any directed edge in the graph G. As result, center nodes (Defini- tion 3.1), which appear in the context directly or indirectly for all tokens in the sequence, will eventually gain a dom- inant role in the context as their direct and indirect contri- butions propagate through the graph. Empirically, Gu al. (2025); Xiao et al. (2024) found that changing softmax to ReLU, which can disconnect edges in the graph, indeed mitigates the emergence of attention sinks.
Theorem 4.2. Let 9 be the sliding-window mask with width
w 2 2. Under A1-A2, given X(0) E NRdx for every token e [M].
lim (2) = 1|x(0))= =1.
Moreover, there exist 0 < C. < where Ne E +l < such that
p(()) (z+ = )1x S 0(1 (jj-1j---)) 1)e 8 (y((rr))
for all 1 < ind 0.
The above result suggests that smaller window size w helps mitigate the model's bias toward early tokens in the sequence. However, such a moderating effect has its limit - the contextual information will still exponentially converge toward the first token over successive layers, though at a rate determined by the ratio between the sequence length N and the window size w.
How No PE induces positional information Previous works have observed that the causal mask alone amplifies the position bias (Wang et al., 2024; Yu et al., 2024). Despite these observations, it remains insufficiently understood how the causal mask captures positional information and what in- formation is being captured. One hypothesis in Kazemnejad et al. (2023) suggests that the causal mask may be simu- lating either an absolute PE or a relative PE with specific weight matrices.
Finally. for the case of prefix mask, where the first K tokens in the sequence serve as prefix and all subsequent tokens attend to them, contextual information exponentially converges toward these K tokens rather than being dom- inated by just the first one, with each of these K tokens having a non-trivial influence.
Theorem 4.1 offers a different perspective. The causal mask results in earlier tokens being utilized more frequently dur- ing computation, inducing sequential order. This bias aligns with the token order in the sequence. To validate this perspective, we present additional experimental results in Section 5.2, providing empirical evidence that the causal mask is not simulating any PE but instead just exhibits a bias toward the earlier parts of a sequence.
Theorem 4.3. Let G be the prefix mask with the first K lokens being the presix tokens. Under Al-A2, given X R xd for every token i E [N].
lim pp(( E [1]]x10)) 11 10000
Trade-off between representation power and position bias Theorem 4.1 also highlights trade-off between rep- resentational power and positional bias the depth of attention layers increases. While numerous studies have demonstrated that deeper attention models are crucial for improving representation power (Li et al., 2024b; Merrill & Sabharwal, 2022; Sanford et al., 2024; Yun et al., 2020a), our findings reveal that these benefits come at a cost. As the model depth increases, the initial tokens in sequence utilized more frequently, amplifying the positional bias toward the beginning of the sequence. This trade-off un- derscores the importance of carefully balancing depth and positional bias in the design of attention-basse architectures.
and there exists 0 such that
lim in (2+ (() 2 n.
Ak: [K].
Moreover, there exist 0 < C,E < where Ne < 1 such that
=
C(1 (j (j- K))o)
for all K < S and t t 0.
Attention sink and center node The above result con- nects the emergence of attention sinks to the structural role of center nodes in the graph G defined by the mask. Specifi- cally, in Gu et al. (2025), the authors observed two interest- ing phenomena: 1) when using the sliding-window window mask attention sinks still appear on the absolute first token the sequence, but not on the first token within each context win- dow; 2) when using the prefix mask, attention sinks emerge on all prefix tokens, rather than just on the first token.
Theorem 4.1 on the causal mask can be generalized to en- compass other types of attention masks, notably the sliding- window mask (Beltagy al., 2020: Jiang et al., 2023) and the prefix mask (Lewis et al., 2020; Raffel et al., 2020). In the sliding-window window mask, each token is allowed to attend to a fixed number of preceding tokens. Let w denote the width of the Buding-window, representing the maximal number of tokens each token can access. The following result shows how limiting the context window size affects the propaga- tion of contextual information in attention mechanism.
These empirical results align well with Theorems 4.2 and 4.3. Our results suggest that the absolute first token and the prefix tokens act as center nodes for the sliding- window mask and prefix mask respectively. The context for each
5

---

On the Emergence of Position Bias in Transformers
token, after multi-layer attention, exponentially converges to these center nodes. This connection between attention sinks and center nodes suggests that attention sinks are not arbitrary artifacts but arise naturally from the underlying graph structure induced by the attention mask.
Compared Lemma 4.4, while the decay mask imposes a stronger decay effect on earlier tokens within individual attention layers, these tokens gain more cumulative im- portance across multiple layers. This trade-off between layer-wise decay and cross-layer accumulation transforms the initially monotonic decay pattern within each attention map into a more intricate, non-monotonic behavior when aggregated throughout the network.
4.2. Relative PEs: A Competing Decay Effect
Having analyzed how attention masks bias the model toward the beginning of the sequence, we now shift our focus to studying PEs, the other key mechanism for representing positional information in transformers.
4.3. A Closer Look at RoPE
Having analyzed the effect of the decay mask, which directly incorporates a distance-based decay into the attention score calculation, we now turn our attention another popular form of relative positional encoding: RoPE (Su et al., 2023).
Relative PE, as the name suggests, incorporates positional information by modifying the original attention scores in way that reflects the relative positions of tokens. Among these. the decay mask (Press et al., 2022) explicitly intro- duces a distance-based decay effect into the attention mech- anism. We begin by examining the effect of the decay mask on individual attention layers.
RoPE's s inherent complexity has made clear theoretical understanding challenging. However, recent empirical ob- servations in Barbero et al. (2024b) suggest that in practice, LLMs tend to predominantly utilize feature dimensions that rotate slowly. This phenomenon introduces additional struc- ture, enabling a more refined analysis of RoPE's effects by focusing on these slowly rotating feature dimensions. For simplicity and without loss of generality, we consider the case where only the slowest-rotating feature dimensions with base rotational angle 01 are used by the model, i.e. effectively reducing the embedding dimension to = 2. See Appendix for more results on the general case d 2 2.
Lemma 4.4, Consider the decay mask in (2) where G is causal. Under A1-A2, given X(() E R.Nx xd. there exists CIAA, Cmin > such that for all si [M] and 2 O,
e -1-j)m I $(A (1) decuy)) )ij S Cmaxee (i-5)m
Lemma 4.4 demonstrates that the decay mask introduces an exponential decay effect into each attention map, with the strength of the effect determined by the token distances. However, while this result characterizes the behavior of individual attention layers, the interaction between layers in a multi-layer setting leads to more intricate behaviors. Building on Lemma 4.4, Theorem 4.5 examines the cumu- lative effect of the decay mask across multiple layers when combined with the causal mask.
Recall from (3) that RoPE operates by rotating the original query and key embeddings by an angle proportional to the token's 's position index within the sequence. Similar to the decay mask, which incorporates distance- based decay into attention scores, RoPE adjusts raw attention scores via these rotations. To formalize this relationship mathematically, we define the original angle between query q;
(t) := x w
Theorem 4.5. Consider the decay mask in (2) where G is causal. Fix T 2 0. Under A1-A2, given X RRNx it holds for all j < i E [N] and & T,
and key k.") = ((())) as 0 Then the following result analyzes how RoPE's daiiiinn-eeeedeett rotations systematically modify the computation of attention scores.
(1+i-)) -j (((tt-))) ) j
Lemma 4.6. Let G be the causal mask and d 2. Suppose
IP(t)
(2: = 51x - docay
-((-Sm)
(t) for t 2 0, 11g 122 1|, ||2 > O. and @ < 88,, where 8 > and (8 + N - -1)0g S n. Then under A1-A2, given
(0) Nxd, there exists Craax Cmin C, d' > 0 such that
Notably, if we denote
for all j SiE [N].
L(x) = log ((++2) am
- S (4 ROPE)) S - 111-3307
x
The result shows that by solely leveraging feature dimen- sions that rotate slowly, RoPE effectively induces a distance- based decay effect, which aligns with the intuition in Su et al. (2023). However, it is worth noting that the decay effect induced by RoPE is significantly smaller compared to that of the decay mask (Lemma 4.4). This is because the base rotational angle 9 is typically chosen to be small, i.c. se 1/10000 per token (Dubey & et al., 2024; Su et al., 2023), resulting in a more gradual decay.
then L(x) is not a monotone function of the distance ir between two tokens. More precisely, under Stirling's ap- proximation, the critical point, where the highest attention score occurs, is at x" = 1/(eT - 1). This means that in- creasing the decay strength m decreases I², making the model more biased towards recent tokens, whereas increas- ing the number of attention layers increases x", making the model more biased towards initial tokens.
6

---

On the Emergence of Position Bias in Transformers
However, similar to the case of the decay mask, when con- sidering the effect of RoPE across multiple layers of atten- tion, the long-term decay effects within individual layers are counteracted by the increasing influence of earlier tokens given by the causal mask.
specific Fq introducing position-dependent bias in the data, or randomly assigned to the class of any Ti, simulating a scenario without position bias in the data.
Tracking position bias To quantify position bias, we eval- uate model performance using sequences containing novel classes not seen during training. Specifically, by generat- ing new class centers for the Gaussian mixture and ran- domly assigning one of the L existing labels to these novel classes, we ensure that the model relies on contextual infor- mation rather than memorized class features. Crucially, we can systematically vary the position of the correct answer within test sequences to measure retrieval accuracy changes, thereby isolating and quantifying position- biases in the model's behavior.
Theorem 4.7. Fix T > O. Under the same conditions as in Lemma 4.6 for < T, given X(0)= E RNNxd, there exists >> such that for all j S E [N] and t T,
=)x(00)) = θ (1
i i-j (((+---) - --((--))P0f
plt) ROPE
(2 (2
Again, if we write
x c c-0209 )
[(x)= log
Network architecture The input sequences are passed through an attention-only network followed by a classifier. Each attention layer has one attention head. The classifier is then a three-layer MLP with ReLU activations and a softmax layer which predicts the probabilities of the L labels.
then, by implicit differentiation, the critical point x" is increasing function of the depth t and decreasing function of the base rotational angle O1 (see Appendix H). This im- plies that increasing the base rotational angle 0, reduces the optimal distance x², amplifying the long-term decay effect and causing tokens to focus more on nearby tokens. In con- trast, increasing the number of attention layers increases xx and hence deeper models become more biased toward initial tokens.
Following Reddy (2024), we set n 8 and d = 64. Ad- ditional experimental details are provided in Appendix J. Despite our use of a simplified experimental setup, we ob- serve the emergence of key phenomena documented in real- world L.LMs, such as the "lost-in-thee-mddlee"" phenomenon (Section 5.2) and the formation of attention sinks (Ap- pendix K.4). This convergence between our controlled envi- ronment and real-world observations validates our choice of abstraction, suggesting that we have preserved the essential mechanisms driving position bias while enabling systematic investigation.
5. Experiments
In this section, we validate our theoretical findings via care- fully designed numerical experiments'. To ensure a con- trolled setup that enables precise manipulation of positional biases in the data, we adopt the synthetic data-generating process and simplified self-attention network framework proposed in Reddy (2024). This setup allows us to system- atically isolate and examine the effects of different compo- nents on the emergence of position bias.
5.1. The Effects of Depth and Relative PEs
To investigate the position bias arising solely from the archi- tectural design of the attention mechanism, we use training sequences without positional bias, where the position of x, sharing the same class as Equery is uniformly random in {1, 2, n.). We evaluate two architectural variants: one without residual connections, and one with standard residual connections applied after each attention layer. The no-residual variant serves as theoretical probe, isolating the contribution of attention alone to positional bias. In contrast, the residual variant reflects standard architectural design and enables us to examine how residual connections, in addition to attention, influence the emergence and ampli- fication of positional bias.
Task structure Following Reddy (2024), we adopt the following information retrieval task: The model is trained to predict the label Squery of a target zquary using the cross- entropy loss, given an alternating sequence of items and labels: I1, .i....... In, Nn, quuery. The sequence is embed- ded in d dimensions. Each 2; is sampled from Gaussian mixture model with K classes (see Appendix I. for results under a fixed-vocabule vocabulaayy setup), and y, is the corresponding class label assigned prior to training from the total L. labels (L < K). The burstiness B is the number of occurrences of 24 from a particular class in an input sequence. Impor- tantly, at least one item in the context belongs to the same class as the query. To control position bias in the training data, quer can either be explicitly assigned to the class of a
To evaluate the position bias in the trained model, con- struct test sequences of the form (a. bj. Here, the bolded term a explicitly marks the correct position, ensuring yo matches Uquery: while position serves as a baseline. In these sequences, Da and TG are identical vectors, allowing us to control for the influence of semantic information on the
Our code is available at gihab..cm/iiniwwDDpssstoooobbbbs in-attention.
7

---

On the Emergence of Position Bias in Transformers
No PE
Decay Mask
RoPE
depth
41
depth res
depth res
Accuracy
150
first vs. mi diddle first vs. last middle vs. last Figure 2: Position bias arising solcly from the architectural design of the attention mechanism, with no positional bias in the training data. è denotes the gap for the case (a, b] -b,, a), where bar magnitude indicates gap size, positive indicates bias toward earlier position, and negative indicates bias toward later position. Deeper attention amplifies the bias toward earlier tokens, regardless of the PE used. Furthermore, decay mask introduce stronger distance-based decay effects that increase focus on recent tokens than RoPE, Residual connections affect positional bias in nontrivial, depth- and PE-dependent way (see Appendix K.2 for more details).
first vs. middle vss last middle last
first vs. middle first vs. last middle vs, last
RoPE
Accuracy Gap
"OOE AM
no mask
middle
vs. last middle first vs. middle vs.
vs. middle
middle vs.
Figure 3: Position bias when trained data biased toward the first and last positions. Compared with no mask, causal mask without PE indeed introduces positional dependencies. However, pure causal mask captures positional bias only at the first position but not the last, whereas both sin PE and RoPE successfully capture biases at both ends. Moreover, the performance under PEs also displays a "lost-in theethe-in thedmiddee" pattern, which is absent under other types of positional bias in the training data (see Appendix K.3 for more details).
model's retrieval accuracy. We then measure the retrieval accuracy gap between pairs of sequences where the content at positions and is identical, but the correct position varies. This gap, defined as [a, b] [b, a]. quantifies the model's positional preference independent of semantic in- formation. To perform this evaluation, we construct three pairs of test sets, each containing 10, 000 sequences: [first, middle] vs. [middle, first], [first, last] vs. [last, first], and [middle, last] vs. [last, middle]. Here "first" (position 1), "middle" (position n/2), and "last" (position n.) denote fixed positions within sequence.
sequence, information near the sequence's 's end becomes less distinguishable, consistent with the patterns observed in Barbero al. (2024a). Furthermore, both the decay mask and RoPE introduce distance-based decay effects that reduce the bias toward the beginning induced by the causal mask and increase the focus on recent tokens. However, the decay effect induced by the decay mask substantially more pronounced than that by RoPE, as predicted by our theory.
Effect of residual connections on position bias Resid- ual connections appear to interact with positional bias in a nontrivial way. While position bias still emerges in mod- els with residual connections, its relationship with depth becomes non-monotonic and be either amplified or re- duced, depending the positional encoding used and the depth regime. We provide more detailed analysis in Ap- pendix K.2.
Figure 2 shows the average results over five runs, where a vs. b denotes the gap [a. b] |b, a). The magnitude of each bar represents the size of the performance gap. and the sign of each bar reflects the direction of the bias: positive sign indicates a bias toward earlier positions, while negative sign indicates a bias toward later positions. We highlight several key observations. First, without residual connections, increasing model depth consistently amplifies the bias toward earlier parts of the sequence, regardless of the PE used (see Appendix K. for additional results). Also note that the performance gap between the middle and last positions notably smaller than that involving the first position. This aligns with our theory, which suggests that as the model focuses more the initial part of the
5.2. Can Causal Mask Induce Usable Positional Information?
Next, we empirically examine how the causal mask lever- ages positional information. Kazemnejad et al. (2023) hy- pothesized that without PE (No PE), the causal mask can implicitly simulate absolute or relative PE through specific
8

---

On the Emergence of Position Bias in Transformers
weight matrix configurations. To test this hypothesis, we train models with residual connections sequences with positional bias at either the beginning the end. Specifi- cally, in the training data, Equery is assigned to the class of di or In with equal probability. We then evaluate two types of attention masks: no mask (G is complete) and causal, and three types of PEs: No PE (Kazemnejad et al., 2023), absolute sinusoidal PE (sin PE) (Vaswani al., 2017), and relative PE using RoPE. For evaluation, construct six types of test sets as described in Section 5.1, each with 10, 000 sequences.
6. Conclusion
In this paper, we study position bias in transformers through probabílistic and graph-theoretic lens, developing a theo- retical framework that quantifies how positional information influences context construction across multi-layer attention. Our analysis reveals two key findings about position bias in transformers: the causal mask's ' inherent bias toward car- lier tokens, as deeper layers increasingly attend to these positions through iterative attention, and the interplay be- tween causal masking and relative positional encodings, which results in a nuanced, non-monotonic balance between distance-basee based decay effects and the cumulative influence of earlier positions. These findings open several promising directions for future work. One potential direction is lever- aging these insights to design bias-free transformers, miti- gating positional biases to improve model robustness and generalization capabilities. Alternatively, our framework can also inform the strategic exploitation of positional bias in specific applications, such as emphasizing early positions for text summarization or prioritizing recent interactions in recommendation systems. Another important direction is to extend our analysis to other transformer components- such as residual connections, value projections, and ML.ss- which may interact with attention in nontrivial ways, as suggested by our results in Section 5.1 and analyses based vector norms (Kobayashi al., 2020). By deepening our understanding of how architectural choices in transformers shape positional dependencies, our work provides foun- dation for designing attention mechanisms with predictable and task-aligned positional properties.
Figure shows the average results using a 2-layer network over five runs. Notably, in the left subplot, the causal mask without PE demonstrates a clear position bias toward the first position compared the no mask without PE. This indicates that the causal mask indeed introduces a notion of position. However, when strong positional biases are present in the training data, both sin PE and RoPE allow the model to effectively capture these biases at both ends, regardless of the mask used. In contrast, a causal mask without PE only enables the model to learn a position bias at the beginning of the sequence. If the hypothesis by Kazem- nejad al. (2023) were correct, that the causal mask uses positional information by simulating PEs, then the model should be able to capture positional bias at any location. This discrepancy suggests that the causal mask does not in- herently implement PE but instead introduces a bias toward earlier positions via iterative attention, capturing positional bias only when it aligns with this predisposition.
Acknowledgements
The role of data in creating positional bias It is worth noting that in Figure 3, we observe the "lost-in-the-middle" phenomenon (Liu et al., 2024), where information retrieval accuracy follows U-shape relative to the position of the answer, with performance at the beginning of the sequence slightly better than at the end. More experimental results under different types of positional bias in the training data can be found in Appendix K.3. Notably, this phenomenon does not occur when the training data lacks positional bias (Figure 2)² or contains other types of positional bias consid- ered (Appendix K.3). This suggests that specific types of positional bias in the training data also play role in how the model learns to prioritize positions within sequence.
The authors would like to thank Gautam Reddy for the help with the experimental setup and acknowledge the MIT Su- perCloud and Lincoln Laboratory Supercomputing Center for providing computing resources that have contributed to the research results reported within this paper.
XW and AJ were supported by ONR N00014-23-1-2299 YW and SJ were supported by ONR N00014-20-1-202. (MURI ML-SCOPE), NSF CCF-21 12665 (TILOS Al Insti- tute), and an Alexander von Humboldt Professorship
Impact Statement
An additional observation from the fixed-vocabulary setting (see Appendix L for details) is that the geometry of the token embeddings can significantly influence task difficulty. This geometric structure may also affect the emergence and nature of positional bias, shapes how the model learns to distinguish and attend to tokens.
This paper presents work whose goal is to advance the field of Machine Learning. There are potential societal conse- quences of our work, none which we feel must be specifi- cally highlighted here.
Witt the sole exception occurring in the deep-layer regime with RoPE; see Appendix K.2.
9

---

On the Emergence of Position Bias in Transformers
References
Guo, T., Pai, D., Bai, Y., Jiao, J., Jordan, M. I., and Mei, S. Active-dormant attention heads: Mechanistically demys- tifying extreme-token phenomena in llms. 2024.
Abnar, S. and Zuidema, W. Quantifying attention flow in transformers. In ACL, 2020.
Guo, X. and Vosoughi, S. Serial position effects of large language models. ArXiv, 2024.
Ait-Saada, M. and Nadif, M. Is anisotropy truly harmful? case study text clustering. In ACL. 2023.
Halliday, M. A. An Introduction to Functional Grammar. 2004.
Alman, J. and Song, Z. Fast attention requires bounded entries. In NeurlPS, 2023.
Hollenstein, N., Pirovano, F., Zhang, C., Jäger, L., A., and Beinborn, L. Multilingual language models predict hu- man reading behavior. In NAACL, 2021.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., and Zhao, W. X. Large language models are zero-shot rankers for recommender systems. In ECIR, 2024.
Barbero, F., Banino, A., Kapturowski, S., Kumaran, D., Ara' ujo, J. G., Vitvitskyi, A., Pascanu, R., and Velivek- ovi'c, P. Transformers need glasses! information over- squashing language tasks. In NeurIPS, 2024a.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Velivckovi'c, c, P. Round and round we go! what makes rotary positional encodings useful? ArXiv, abs/2410.06205, 2024b.
A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b. ArXiv, 2023.
Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. In NewrIPS, 2023.
Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. ArXiv, 2020.
Cai, X., Huang, J., Bian, Y.-L,, and Church, K. W. Isotropy in the contextual embedding space: Clusters and mani- folds. In ICLR, 2021.
Kim, Y., Denton, C., Hoang, L., and Rush, A. M. Structured attention networks. In ICLR, 2017.
Kobayashi, G., Kuribayashi. T., Yokoi, S., and Inui, K. Attention is not only a weight: Analyzing transformers with vector norms. In EMNLP, 2020.
Chi, T.-C., Fan, T.-H., Ramadge, P. J., and Rudnicky, A. Kerple: Kernelized relative positional embedding for length extrapolation. In NeurlPS, 2022.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., rahman Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequene pre-training for natural language generation, translation, and comprehen- sion. In ACL, 2020.
Dubey, A. and et al. The llama 3 herd of models. ArXiv, abs/2407.21783,, 2024.
Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP, 2019.
Li, Y., Pazdera, J. K., and Kahana, M. J. Eeg decoders track memory dynamics. Nature Communications, 2024a.
Fang, L., Yifei Wang, K. G., Fang, L., and Wang, Y. Re- thinking invariance in in-context learning. In ICLR, 2025.
Li, Z.,, Liu, H., Zhou, D., and Ma, T. Chain of thought powers transformers to solve inherently serial problems. ArXiv, abs/2402.12875. 2024b.
Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. Representation degeneration problem in training natural language generation models. In ICLR, 2019.
Liu, N. F. Lin, K., Hewitt, J.. Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2024.
Glanzer, M. and Cunitz, A. R. Two storage mechanisms in free recall. Journal of Verbal Learning and Verbal Behavior, 1966.
Godey, N., de la Clergerie, E. V., and Sagot, B. Anisotropy is inherent to self-attention in transformers. ArXiv, abs/2401.12143. 2024.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In JCLR, 2019.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp. P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022.
Gu, X., Pang, T., Du, C., Liu, Q., Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges in language models: An empirical view. In ICLR, 2025.
10

---

On the Emergence of Position Bias in Transformers
Merrill, W. and Sabharwal, A. The parallelism tradeoff: Limitations of log-precision transformers. Transactions of the Association for Computational Linguistics, 11:531- 545, 2022.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks. In ICLR, 2024.
Yu, Y., Jiang, H., Luo, X., Wu, Q., Lin, C.-Y., Li, D., Yang. Y., Huang, Y., and Qiu, L. Mitigate position bias in large language models via scaling single dimension. ArXiv, hass2400.022336 2024.
Min. S., Lyu, X., Holtzman, A., Artetxe. M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
Yun, C., Bhojanapalli, S., Rawat, S., Reddi, J., and Kumar. S. Are transformers universal approximators of requence-to-sequenee functions? In ICLR, 2020a.
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E.., Detito, Z., Rai- son, M., Tejani, A.. Chilamkurthy, S., Steiner, B., Fang. L., Bai, J., and Chintala, S. Pytorch: An imperative style, serforerfrrmance deep learning library. In NeurIPS, 2019.
Yun, C., Chang, Y.-W,, Bhojanapalli, S., Rawat, A. S., Reddi, S. J., and Kumar, S. O(n) connections are ex- pressive enough: Universal approximability of sparse transformers. In NeurIPS, 2020b.
Zhang, Z. A., Chen, R., Liu, S., Yao, Z., Ruwase, O., Chen, B., Wu, X., and Wang, Z. Found the middle: How language models use long contexts better via plug-and- play positional encoding. ArXiv, abs/2403.04797, 2024.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Zhao, T. Z.,, Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging Ilm-as-a-judae with mt-bench and chatbot arena. NeurlPS, 2023.
Reddy, G. The mechanistic basis data dependence and abrupt learning in an in-context classification task. In ICLR, 2024.
Sanford, C., Hsu, D., and Telgarsky, M. One-layer trans- formers fail to solve the induction heads task. ArXiv, abs/2408. 144332,,,,,, 2024.
Su. J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. 2023.
Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.. and Polosukhin. I. Attention is all you need. In NeurlPS, 2017.
Wang, Z., Zhang, H., Li, X., Huang, K.-H., Han, C., Ji, S., Kakade, S. M., Peng, H., and Ji, H. Eliminating position bias of language models: A mechanistic approach. ArXiv, abs/2407.011000 2024.
Wu, X., Ajorlou, A., Wu, Z., and Jadbabaie, A. Demys- tifying oversmoothing in attention-based graph neural networks. In NeurIPS, 2023.
Wu, X., Ajorlou, A., Wang, Y., Jegelka, S., and Jadbabaie, A. On the role of attention masks and layernorm in transformers. In NewrIPS, 2024.

---

On the Emergence of Position Bias in Transformers
A. Proof of Theorem 4.1
A.I. Auxiliary results
Lemma A.1. Under A1-A2, there exists >> 0 such that A 2 for all t 2 0. (i.i) e E.
Proof. Writing (1) recursively, get that the token trajectories
x
".. X w; (..
(5)
stay uniformly bounded for all t 2 0 by A2. Then it follows from A1 that there exists C E R such that for all 2 0,
(x()
x
W Q
sc,
2:12
2
(6)
(x(
x
w
SC.
K
1.112
Hence for all i,j E [M]
-C² S <x10 (x(" C
This implies that there exists such that A 2 c for all (j,1) E.
A.2. Proof of Theorem 4.1
denote p(O) = A(O... A suffices to show that there exists C and 0 < < such that
C(1 1)))
(7)
for all 1 < < and t 2 0.
The proof will go by induction:
Base case By Lemma A.1, it follows that
P!0) S (1 - c)
< S Then let := e.
Induction step
Assume that (7) holds, follows that for all < S i.
p(++))
( 10p (1 -0 1) (j 1)6) C(1 (j - 1)6)
k=j
From above, we conclude the theorem.
B. Proof of Theorem 4.2
For to f,, we denote
(br4t) (6)) A..A(to))
Without loss of generality, we assume in the following proof that N 1 can be divided by w 1.
B.1. Auxiliary results
Lemma B.1. Let G be the sliding-window window mask with the window size w 2 2. Then there exists > 0 such that for all
l 2 0,
< A (to+ N-1 1:to)
11, SiEi
12

---

On the Emergence of Position Bias in Transformers
Proof.
Given the connectivity pattern of the sliding-window mask and Lemma A.I, it follows that for all
A (to+ I:to) a lower triangular matrix. Moreover, since Agg (fo+ (fo+@-1:tt)) counts the aggregate probability of the walks of length N-1 w-=
N-1
between token and token j where by Lemma A.1, each walk has probability at least e=-1,
Thus we conclude that there exists c 0 such that for all to 2 O.
A (lo 11:t0) 20, (j
B.2. Proof of Theorem 4.2
For k 2 0, denote
A (((+))( -
and
P(R) A(h) A(0)
Then by Lemma B.I and Theorem 4.1, we get that there exists < C' < 1 and 0 < c << such that for all k 2 0
p()) S C(1 (i-1-j-1)e) Visie(M
Denote 0
max ((t) 18 1j
and (/,0)
max P(lk) 1SiSN ij
Then it follows that for all k: 2
20hh S C(1 (j 1)cc**
Observe that
E [N], p < 10
(8)
and
Pj NN, 0 ((+1) < 00
(9)
Let 0j = (1 1 -jj-j - 1))c -
Then for k > 1 and
< N-1
w-1-
*
+r
(1 (j 1)c))*
q
This implies that for all / N-1
p(() < o = C(1 (j - (2
bj
As for t < N-1 notice that p,0)) < 1 - e for all j Si [N] by Lemma A.1. Then by 8 and 9, we deduce that
p s1 #
Vjsie [N].
We thus conclude the statement.
13

---

On the Emergence of Position Bias in Transformers
C. Proof of Theorem 4.3
First note that the first and third statements:
lim (z: [K]]X(()
(10)
for all E [N] and
PP0 (2, = -| S C(1 - (j (j- K)()*
for all K <jS and 2 O, follow immediately from Theorem 4.1 by regarding the first K tokens super node in the causal graph G and aggregate the edges accordingly. Thus it suffices to show that there exists x >> 0 such that
lim inf (pl (z+ k|x(") 2 Ak E [K].
(11)
1-000
Fort > 0, consider
PM Σ
(((:1) A
(t:11
(0)
1=1
Then for ki,ko E [K],
p(t)
max(i.K) } p(t:1) (0)
[kg
21
Alk
p(t)
maaxfi,K) p(t:1)
6k2
1=1
Ak2
which then follows
p(t)
Aig
sky
min Islsmin(t.K)
p(t]
1515
A
skz
Then by Lemma A. there exists C > 0 such that for all k1, E [K],
p(()
C < lim inf
iks
10000
p(t)
sk2
Since vom Σx_ p!0
by (10), we deduce (11) as desired.
D. Proof of Lemma 4.4
Fix / 2 0. Let
2 - -x(0 (x. hi
Following from Lemma A.1, there exists I Imax E IR such that for all j s E [N],
2 [Tuinn [max].
Consider the denominator in the softmax(-) operation in the caleulation of (A decay Dayi
(A
(1)) e
(i-A)m k)m
>
Juin
e (i-k)m k)m
k=
e (1+1)m
Jiin
e-m m
c 2m
Iin
1 - C
minn
(1 + m
14

---

On the Emergence of Position Bias in Transformers
and
2
Σ Jom
c 2!l kim-))m S e mmx
kmm
Amax
172
It follows that
Amar (1-5)m
) S elmia (1 + e-
-maxt (1-0m
and
(6-i--(6-))mm
(A (is efmax (11 e) Cmine -f-ee----ppi (f-j)m
where Cmax == elfmas Immn) /(1 m and Cmin == (1 (1-e- me -max)) Imax)
E. Proof of Theorem 4.5
Note that in the causal graph G, there are (( paths of length / + from token j to token i.
Since going from token to token i in the causal graph, the connectivity patterns ensure that the token indices along the path are non-decreasing, i.e. if we denote the directed path as 11) (4, .2...... (le, i), it holds that j s 1 S la S Together with Lemma 4.4, we conclude the theorem statement.
S 4 S i.
F. Proof of Lemma 4.6
(1)
Then it follows from the definition of RoPE that Fix t 2200 0. Denote the angle after rotation to be vij
- - (i-j)0
Thus
1 10:9 (i 1)01 11(i (1)0))11--10 - 10 2 10 - () - 810
W ( = 1010 (i j)0,1 S (i j)0,1 + 100 < (i- j + 8)0
Let the original query and key j embeddings be " = x(" and k(tt) x w:( respectively, and the correspond- ing query i and key j embeddings after rotation be (()) and k respectively.
Since (1, (0) _k3 k000 |10 [|al| |1allk/012 2 cos v:(() it follows from that there exists Cininy Cmax such that for all i,j E [N].
Cmin cos((i - + 8)01)= S (q. k00) I Cmax Cmaxxco((( - 810;). Since for |ar| S π there exists e > such that 1 - x²² E cos & - x² /ce, we get that
Cmin (1- ((i ()+++)+0 5)²67
(t) M < Cmax (1- ((i - (()
< (q
2
and hence
(1 ((i (6)-))/2-88) 82)09
Cuin (1 -82207 (i 2))))) S (g: (t) k(0)) < Cmax
c
Consider y!! = Ei= c44 100 (400)) Then by (6), we get that there exists Ymax, Ymin > 0 such that
Ymax S x < Ymin.
We thus conclude the statement.
15

---

On the Emergence of Position Bias in Transformers
G. Proof of Theorem 4.7
Notice that
(PRoPg)is
E
S (1-2) 1 4-141-2 ... A
(12)
4 SS-
Given that when G is the causal graph, due the connectivity the directed path of length from token j to token must be non-decreasing, i.e. j $ 1 S .... S 4-11 & i, and there would be in total (*t22) such paths. For each such path j S li 2 4-1 S i, notice that by Lemma 4.6, we get that fix T 2 O, there exists Cuin. Cmax 0 such that
レ 14-1 1-
... A 2 e c(0-- + -4- -2-1))+++++))) + )88
(13)
and
A [t-l) A (t-2)
14111 1-11112
1 e -' ((t-)
(14)
From (13), since
4 $ l <
4-1 we further get that
1 (1-1) 1l1-2)
1/4-1 Ae-idh--
1 2 Cuine -
c(8-51903
(15)
and similarly
((-)) (t-2) Au 4-14-
1 I S (1-5))
(16)
H. Implicit differentiation of with respect to 0 and
Recall that under RoPE,
--000f 2'09 (
L(x) log
x
Then by Stirling's approximation.
L(x) = ((t + x)) log(t+
+ - (t + x))
(x log x - x) -
and thus
+ x
L'(x) = log
20 x.
Taking implicit differentiation
i
+
t
dx
1
log
Ōt
x
x(x + t)
Ot
+ t
and
dx
20 201
at
So let
1
1
dx
(20) +
+1
(x f)xx
Ht
and thus
dr
=
20 (I + t) + > 0.
Øt
Taking implicit differentiation of 0::
i log
t+x
-t
dr
00,
zxx + t) 00,
16

---

On the Emergence of Position Bias in Transformers
and
dr 2x +
20
01.
001
0
So let
dz -2r
201 +
2(1 +
00,
and thus
dr
-2x
0.
001
201 + =(z+t)
Hence we observe that x is an increasing function of and a decreasing function of 0₁. This implies that increasing the base rotational angle 0, reduces the optimal distance x", amplifying the long-term decay effect and causing tokens to focus more on nearby tokens. In contrast, increasing the number of attention layers increases x and hence deeper models become more biased toward initial tokens.
The effect of RoPE: case for d 2 2
In this section, we present generalized version of Theorem 4.7 for the case d 2 2.
Let the query y and key & be vectors in Rd, where is even, and & be the angle between & and k, which we assume be well-defined, with:
cos Φ (g. k)
|1
Define the length-2 segments of query q and and key l as
q (12-11 42),
bn (921-1:921)
for l E [d/2], and let di be the angle between and kj. with:
(qi kz)
cos Φi
|
Without loss of generality, we make the following assumption:
A3 There exists By. 3g such that ||4 10 2 (*) 1l and ||&] ||x|012 Ba|lA for all e [d/2] for all
The condition means that all segments makes a nontrivial contribution to the norm. In practice, since LLMs tend to tend to predominantly utilize feature dimensions that rotate slowly (Barbero et al., 2024b), the effective d/2 tends to be a small number.
Given the pre-defined set of base rotational angles Ô {0 S 01 S
S 02/2), we reparametrize 0, a,0,
I.1. Results
present the general version of Lemma 4.6 and Theorem 4.7 as follows:
Lemma L.. Let G be the causal mask and A1-A3 hold. Suppose for t 2 0. |1a:ll2. | O. and lo s 80,, where
5 > 0 and
(V
ByBk 8x + 2(N - 1)04//2 0 S 2x
Then there exists Crax, Omin, d' > O Osuuch that
Cmine (I () < (A (A (opp))s S Crave E1/2((0 E117(1-3)0000 22
17

---

On the Emergence of Position Bias in Transformers
Theorem 1.2. Fix T > 0. Under the same conditions as in Lemma 1.1 for S T, there exists e such that for all 1 $ T,
(P (ROPE)>s =6 (((++))) e -e --22
1.2. Proofs of Lemma 1.1
We first show the following auxiliary result: Lemma I.3. Under A3, it holds that
π 1 -
V/
13 3k
for all E [d/2].
Proof. By definition, since
d/2
Σ
cos ||//1/11/||22o cos
then the Cauchy-Sehwarz inequality implies that
4/2
Σ
ot) S 11/122/14112 c)
(17)
By A3, (17) becomes
d/2
1
Σ (1
cos Φ1) S
(1
cos c)
8y
1=1
Given the trigonometric identity
cos 2r 2 sin x, we get that
4/2
Φ1
sin
5
sin"
(18)
Byg
3k
2
Notice for all E.
sin 2 S x²
(19)
and for all |x|| S n/2,
72 x S sin2 x.
(20)
Apply (19) and (20) to (18), we get that
4/2
Σ
2 <
0²
11
43, 3k
1=1
Hence for all E [d/2), it follows that
|0x <
61.
3r33
Denote the angle after rotation to be Vi.j.1- Then it follows that
thg.11 = Di.j. - 1)a01.
18

---

On the Emergence of Position Bias in Transformers
follows similarly in the proof of Lemma 4.6 that
Cmin (1 8 (i (d-j))a j - ((d)), (kf))) s Cmax [ ((i- j) (1-(6--)Pa222 833
c
where 8' = 2/ By
6, for all / [d/2].
Since
4/2
(d) k3) Σ ((de), (K))1),
1=1
we conclude the statement.
1.3. Proof of Theorem 1.2
The result is a direct corollary of Lemma 4.6 and Theorem 4.7.
J. Experiments
Here we provide more details on the numerical experiments presented in Section 5, All models were implemented with PyTorch (Paszke et al., 2019).
Parameterizing the data distribution As defined in Section the input data distribution is modulated by tuning various parameters. In addition to the parameters described in the main text, for the Gaussian mixture with K classes, each class k is defined by d-dimensional vector μt whose components are sampled i.i.d. from a normal distribution with mean zero and variance 1/d. Then the value of H; is given by A + en:/ /V 2, where Y) is drawn from the same distribution as the Mg 's and e sets the within-class variability. Each class is assigned to one of L labels (L S K). The contents of the labels are drawn prior to training from the same distribution as the uag ss
In Reddy (2024), the author found that different configurations of the data generating process give rise different learning regimes. To enable better information retrieval ability of the model, choose the configuration suggested by Reddy (2024) that corresponds to the difficult in-weight learning and easy in-context-leaming regime to ensure the information retrieval ability of the model. Specifically, we set 7 0.75, K = 2048, L = 32, and B 4.
Relative PE hyperparameters For the decay mask, we set m = log( ~ 0.223. For RoPE, we set 0 10000 2(--)/ as in Su al. (2023).
Compute We trained all our models on Tesla V100 GPU.
Training details In all experiments, we used the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 10-3, weight decay of 10 10-6, batch size of 128, and trained for 100, 000 iterations.
K. Additional Experimental Results
K.1. The effect of depth and relative PEs
Here, we present additional experimental results the effect of attention depth on positional bias. Since models without residual connections become increasingly difficult to train as depth increases, we limit our experiments to depths of 2, 4, and The results are shown in Figure 4. We observe that deeper attention amplifies the bias toward earlier tokens, regardless of the PE used. Furthermore, both decay mask and RoPE introduce distance-based decay effects that increase focus on recent tokens, but the one induced by decay mask stronger.
K.2. The effeet of residual connections on position bias
In this section, we present additional experimental results on the effect of residual connections on the positional bias exhibited by the model. The results are shown Figure As discussed Section 5.1, we observe that with residual
19

---

On the Emergence of Position Bias in Transformers
No PE
Decay Mask
ROPE
1100 00:1
depth depth
0.077
Accuracy
8:00
EXITELL
420
230
42X
middle vs. last Figure 4: Results with pure attention layers: position bias arising solely from the architectural design of the attention mechanism, with no positional bias in the training data.
fss middle first vs. last middle last
first middle
first vs. last
middle vs. last
vi middie
vis last
Decay Mask
RoPE
0.77
depth
Accuracy Gap
depth 10
3009
<XI
first vs. last middle vs. last Figure 5: Results with residual connections: position bias arising solely from the architectural design the attention mechanism, with no positional bias in the training data.
vs. middle
first vs. last
middle vs. last
first vs. middle
first vs. last
middle vs. last
vs middle
connections, the relationship between model depth and positional bias becomes non- non-monotonic, and the strength of the bias can be either amplified or reduced depending on the positional encoding used and the depth regime. These findings suggest that residual connections modulate the accumulation of positional effects across layers in complex manner than with pure attention alone. It would be interesting to further explore and characterize how residuals influence the emergence and amplification of positional bias.
K.3. The role of training data on positional bias
In this section, we present additional experimental results building the experiment described in Section 5.2, but focusing on cases where positional bias in the training sequences is introduced at other positions. Specifically, we consider three types of training sequences where Equery is assigned the class of 1) æ (the first position). 2) 2n,2 (the middle position), or 3) In (the last position). The corresponding results are shown Figure 6, Figure 7, and Figure 8, respectively.
Observe that, compared with no mask, the causal mask without PEs indeed introduces sense of position across all cases. Specifically, it enables the model to learn a positional bias favoring the beginning of the sequence, as earlier tokens tend receive more attention through the mechanism of iterative attention. In contrast, both sin PE and RoPE allow the model to effectively capture different positional biases regardless of their location in the training sequences.
Interestingly, when comparing this behavior to the case shown in Figure 3, we note that the "oat-in-the-middle" phenomenon only emerges when the training sequences are biased toward both the beginning and the end (the only other exception occurs in the deep-layer regime with RoPF, despite the absence of positional bias in the training data; see Appendix K.2). This suggests that specific types of positional bias in the training data could also play crucial role in shaping how the model learns process and prioritize positions within a sequence.
As the structure of positional bias in natural language remains unclear, this observation raises the following question:
Does positional bias natural language sequences shape the "lost-in- -iithe-mmdddee phenomenon in a similar way to what we observe in this simplified case?
This question connects to broader inquiries about the parallels between artificial and human attention. In neuroscience, the primacy-recency effect highlights that human attention often gravitates toward the beginning and end of sequences (Glanzer Cunitz, 1966; Li al., 2024a), phenomenon that may have influenced the structure of human languages, where critical
20

---

On the Emergence of Position Bias in Transformers
No PE
Sin PE 5:00 Gass
RoFE
mask
Accuracy Gap
60223889 middle vs. last
1/st middle
first last
middle vs. last
first middle
first vs. last
first middle
first last
middle vs. last
Figure Position bias exhibited by the model when trained data biased toward the first position.
No PE
sin PE
RoPE
CS4e C552
138
no mask
causal mask
1250
Accuracy
first middle
last
dddde
first middle
first last
middle
first middle
first
middle last
Figure Position bias exhibited by the model when trained on data biased toward the middle position.
No PE FIE
sin PE
RoPE
no mask
causal mask
Accuracy Gep
1.5
-E3221115
vs.
first vs. middle
vs. middle
Figure 8: Position bias exhibited by the model when trained on data biased toward the last position.
information is frequently placed in these positions (Halliday, 2004). As demonstrated in Section 5.2, when such patterns are present in training data, attention-based architectures seem to develop analogous biases (Hollenstein et al., 2021), aligning with natural language characteristics for improved performance. This raises deeper, perhaps philosophical questions: To what extent are these biases intrinsic to effective sequential processing? How closely should neural networks emulate human cognitive patterns? Investigating these connections can deepen our understanding both human and artificial intelligence while guiding the design of more effective machine learning models.
K.4. Attention sinks
Despite our use simplified experimental setup in this work, we observe the emergence of key phenomena documented more complex settings. In addition the "lost-in-the-min th--elllmdddle" phenomenon discussed in Section 5.2 and Appendix K.3, in this section, we report the formation of attention sinks our setting.
Figure 9 shows an example of the attention maps of a two-layer self-attention networks under the causal mask without PEs. where the sequences used for training and inference are all free of position bias. We observe the similar phenomenon of attention sinks as reported in Xiao et al. (2024).
More guantitatively, following Gu et al. (2025), we calculate their metric for measuring the emergence of attention sinks, over 10, 000 sequences free of position bias. Specifically, denote the adjacency matrix of the mask G be M. Then the
21

---

On the Emergence of Position Bias in Transformers
Layer
Causan Layer2
Pud.ll cae
Figure 9: Example of the emergence of attention sinks in our experimental setting. In particular, the sequences used for training and inference are all free of position bias.
metric for attention sink at token j is calculated as
T T Σ
N Σ 144 >..
Attention Sink,
Mij ==1
The threshold r we choose is 0.2. The results for the causal mask, the sliding-window masks (with = 5,9, 13), and the prefix masks (with K 2, 4, 6) are shown in Figure 10, Figure 11, and Figure 12, respectively. In particular, we make the following observations:
1. Attention sinks emerge on the absolute first token under the causal mask.
2. Attention sinks tend emerge the absolute first token when the window size is larger, under the sliding-window mask.
3. Attention sinks emerge on the K prefix tokens, not just on the first token alone, under the prefix mask.
All of these phenomena have been observed in real-world LLMs in Gu al. (2025). This alignment between our controlled setup and real-world observations affirms the validity of our abstraction, indicating that we have captured the key mechanisms underlying position bias while facilitating systematic analysis.
22

---

On the Emergence of Position Bias in Transformers
Causal
Attention Sink (%)
Index
Figure 10: Attention sinks emerge on the first token under the causal attention mask.
Sliding Window, w
Sliding Window, w-9
Sliding Window w-13
Attention Sink (%)
Attention Slnk (%)
Attention Sink (%)
Token Index
Token Index
Token Index
Figure 11: Attention sinks lend to emerge on absolute token when the context window size w larger, under the Bldding-windood mask.
Prefix, K-2
Profix, K=.
Prefix, K=6
Attention Sink (%6)
Attention (%6)
-12
Attention Sink (%)
Token Index
Token Index
Token Index
Figure Attention sinks emerge on the K prefix tokens, not jotjust on the first token alone, under the prefix mask.
L. Additional Results under Fixed- Setting
In addition to the Gaussian mixture setting discussed in the main text, we also present results for a fixed-vocabulary setting: we define fixed of K class vectors (#1, ......... "K ), and assign each input x; *o((), where c(i) {1...., K) the class index. Each class vector U2 is then defined by = (μx + /w)/ /I + X2, where μa and are d-dimensional vectors whose components are sampled i.i.d. from a normal distribution with mean zero and variance 1 1/d. Note that the shared component 1) makes Us not completely orthogonal to each other, resembling token embeddings in practice (Ait-Saada Nadif, 2023; Cai et al., 2021: Ethayarajh, 2019; Gao et al., 2019; Godey et al., 2024). À is scalar that controls the strength of anisotropy, where we set 0.75.
The corresponding results are shown in Figure 13-Figure 24, respectively. Most of the findings presented in the main text continue to hold in this setting. Notably, even with residual connections, increasing depth consistently amplifies positional bias, as shown in Figure 14.
One key difference between the two settings is that, compared to the orthogonal embeddings used in the Gaussian mixture setting of Reddy (2024), the anisotropic embeddings in our setup appear to make the task significantly more challenging.
23

---

On the Emergence of Position Bias in Transformers
No PE
Decay Mask
RoPE
418
depth depth
2044
Accuracy
C021
first middle first last
middle vs. last
first ddie
first vs. last
middie last
first vs. middle
first last
middle vs. last
Figure 13: Results with pure attention layers under the exca-vocabulary setting: position bias arising solely from the architectural design of the attention mechanism, with positional bias in the training data.
Decay Mask
RoPE
depth
0.100
0.99
depth
3.171
depth 10
0/993
Accuracy
2:101
2007
40
middle vs. last Figure 14: Results with residual connections under the vocabuaayy setting: position bias arising solely from the architectural design of the attention mechanism, with no positional bias in the training data.
vsstvv. middle
first vs. last
middle vs. last
first vs. middle first vs. last
visst middle first last middle vs. last
We note that in preliminary experiments under the fixed-vocabulary setting with À 0, the results closely resemble those from the Gaussian mixture setup, suggesting that embedding geometry plays critical role in task difficulty. It is possible that residual connections behave differently when the task becomes difficult. Future work could further investigate how the geometry of the input embeddings influences both task difficulty and the emergence of positional bias.
24

---

On the Emergence of Position Bias in Transformers
mask causal mask
A224
Accuracy Cap
IM
first vs. middle
vs. last
middle vs. last
first vs. middle
firstvs. vs. last
middle vs. last
first vs. middle
first vs. last
middle vs. last
Figure 15: Position bias exhibited by the model when trained data biased toward the first and last positions.
No PE
sim
RoPE
causal mask
Accuracy Gap
2090
Arst middle
first vs. last
middle vs. last
first vs. middle
First vs. last
middle vs. last
first vs. middle
first vs. last
middi vs. last
Figure Position exhibited by model when trained on data biased toward position.
No PE
sin PE
RoPE
mask
am
Accuracy
first middle
first vs. last
middle last
first middle
first last
middle last
first middle
first last
middle last
Figure Position bias exhibited by the model when trained data biased toward middle position.
No PE
sin PE
RoPE
2.00
Accuracy
causal mask
first
first middle virst
first
Figure 18: Position bias exhibited by the model when trained on data biased toward the last position.
Figure 19: Results under Exed-vocabulary setting
25

---

On the Emergence of Position Bias in Transformers
Figure 20: Fixed- vocabulary setting: example of the emergence of attention sinks in our experimental setting. In particular, the sequences used for training and inference are all free of position bias.
Causal
Attention Sink (56
Figure 21: Attention sinks emerge on the first token under the causal attention mask.
Sliding Window, w=
Sliding Window, w
Sliding Window, w=13
100-
Adtention Sink (%)
Attention Sink (%)
Attention Sink (%)
Token Index
Token Index
Token Index
Figure Attention sinks tend to emerge on absolute token when the context window size w larger, under the sliding- window mask.
Prefix, K-2
Prefix, Km4
Prefix, K-6
Attention Sink (%6)
Attention Sink (%)
Attention Sink (96)
Token Index
Token Index
Taken Index
Figure 23: Attention sinks emerge on the K prefix tokens, not just on the first token alone, under the prefix mask.
Figure 24: Attention sinks under the -ocauuucyb setting.
26


Reviews:
[{"review_id": "7xuHdqpge8", "reviewer": "Reviewer_dVk1", "summary": "This paper aims to analyze the effect of attention masks, such as the causal mask, of the observed attention patterns. In particular, the authors suggest modeling the possible paths that the information can flow using attention as edges of a graph. By looking at this graph, they obtain certain bounds which they map to practical observations such as bias towards early tokens in the sequence. Furthermore, the paper looks into the effect of relative and rotary positional encodings on these bounds and show how these encodings can lead to different biases.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing thoughtful feedback. Below, we provide detailed responses to the comments.\n\n> Theorem 4.1 only yields an upper bound on the weight given to a token. \n\nThank you for the comment. We agree that Thm 4.1 provides an upper bound rather than a strict inequality, and we do not claim it guarantees a strict bias toward earlier tokens in every setting. Rather, it characterizes an emergent tendency, rooted in the causal mask, for earlier tokens to accumulate more attention as the number of attention layers increases.\n\n- Importantly, while the upper bound is not always tight, it is not vacuous: it decays exponentially to $0$ for tokens $i\\geq 2$ as attention deepens, meaning it cannot be arbitrarily large. This suggests that—regardless of specific token embeddings—later tokens become progressively less influential, while earlier tokens retain more aggregate attention. One can derive more precise and stronger results by imposing additional assumptions regarding token embeddings or model weights. For example, for sequences of identical tokens, the upper bound becomes exact. \n\n- Moreover, our empirical results support these core insights from Thm 4.1. As shown in Fig. 2, even in the absence of explicit positional bias in the data, models with causal masking exhibit clear early-token bias—an effect that strengthens with depth. This empirical observation closely aligns with our theoretical findings, suggesting that the upper bound meaningfully captures the inductive bias induced by causal masking.\n\n> Attention patterns look at how attention is distributed in a single layer.\n\nThank you for the comment. \n\n- While attention patterns are often analyzed at the single-layer level, such analyses may miss how information propagates and accumulates across layers—a key aspect of how transformers build contextual representations. A central contribution of our work is to model this global effect of attention across layers. In your example, while token 2 does not directly attend to token 0, it inherits information from token 0 through token 1. Such multi-hop influences are not captured by per-layer attention patterns but are critical for understanding deep model behavior.\n\n- Our graph-theoretic framework is designed precisely to capture these *multi-step dependencies by modeling the global effect of attention across layers*. This complements prior layerwise analyses and aligns with recent work (Abnar et al., 2020; Barbero et al., 2024; Wu et al., 2024) that emphasizes the importance of analyzing multi-layer attention compositionally.\n\nBy tracing these cumulative token interactions, our framework offers a complementary and more holistic view of attention dynamics. We hope that our methodology can serve as a useful tool for the community and help guide future investigations into multi-layer attention mechanism.\n\n> In Kazemnejad et. al. the authors explicitly show a set of weights for a transformer model that lead to it learning positional indices whereas the paper is claiming this can not be achieved based on experiments. \n\nThank you for the comment. We would like to clarify the relationship between our work and the findings of Kazemnejad et al.:\n\n- Kazemnejad et al. provide a theoretical construction showing that positional information can, in principle, be encoded using only causal masking via hand-crafted weights. However, their result does not reflect what typically emerges through standard training.\n\n- Our work builds on this insight by asking a complementary question: Do transformer models actually learn positional encodings in practice when trained without explicit positional signals? We address this through a controlled task where position is critical and observe whether models develop positional awareness under causal masking alone.\n\n- As shown in Sec. 5.2 and Fig. 3, models trained with explicit positional encodings (sin PE or RoPE) capture both start and end-of-sequence biases, while models using only causal masking consistently fail to capture end-of-sequence patterns. This suggests that while position can theoretically be represented via causal masking, *it does not seem to emerge naturally in practice*, even when position is highly informative. Our theory supports this by showing that causal masking induces an inductive bias toward earlier positions.\n\nThus, rather than contradicting Kazemnejad et al., our work provides a more practical and deeper understanding of how position bias actually manifests in Transformers under different PEs. We will clarify this distinction more explicitly in the final version of the manuscript.\n\n\nWe appreciate your questions and comments very much. Please let us know for any further questions.\n\n---\nReferences\n\nAbnar et al. Quantifying Attention Flow in Transformers\n\nWu et al. On the Role of Attention Masks and LayerNorm in Transformers\n\nBarbero et al. Transformers need glasses! Information overSquashing in Language Task", "claims_and_evidence": "In addition to the major issue I mention in the next section, some of the claims are not fully justified. For example, theorem 4.1 only yields an upper bound on the weight given to a token. While the upper bound is smaller for later tokens, this doesn't mean that there is necessarily a bias towards earlier tokens as the upper bound can be arbitrarily large and not be strict. Therefore, I find these claims to be misleading.", "methods_and_evaluation_criteria": "Based on my understanding, the graph model that is being used, looks at the weight given to each token overall not per layer. What it means is that if token 1 pays attention to token 0 in layer 1, and token 2 pays attention to token 1 in layer 2, this is counted as attending to token 0. However, these weights are not what is looked at when talking about attention patterns. Attention patterns look at how attention is distributed in a single layer.  As such, I do not believe that the mapping of the obtained results (that are based on the graph model) to", "theoretical_claims": "Look at the above", "experimental_designs_or_analyses": "In Kazemnejad et. al. the authors explicitly show a set of weights for a transformer model that lead to it learning positional indices whereas the paper is claiming this can not be achieved based on experiments. Since the given weights in Kazemnejad et. al. actually allow learning positional indices, I believe these results require further investigation to understand what is different.", "relation_to_broader_scientific_literature": "No comments.", "essential_references_not_discussed": "No comments.", "other_strengths_and_weaknesses": "No comments other than above.\n\n\n## update after rebuttal\nMy concerns remain largely unaddressed. For example:\n\n1) It is possible that understanding the distribution of attention scores over original tokens would be beneficial. However, this paper makes many claims about the relevance of results with the per-layer attention patterns. For example, Table 1 tries to suggest relevance between the theorems and the attention sink phenomenon. These claims are made regularly in the paper and are the essence of how the results are shown to be important. However, these are in no way valid since the thoerems do not apply to per-layer attention patterns at all. As such, I find them extremely misleading to the community.\n\n2) I am still unsure about experimental results showing that the models can not learn positional biases since previous work also include empirical work that show it is possible to decode position information when no explicit positional encoding is provided. I believe a mismatch in settings might be the culprit and I would like to ask the authors to carefully investigate this. \n\n3) Additionally, conclusions drawn from the upper boud remain questionable. For example in the comment \"later tokens become progressively less influential, while earlier tokens retain more aggregate attention\", the upper bound only shows the first one. However, it does not mean the second one is necessarily true unless a lower bound is drawn. If all the scores go equally to 0, the upper bound would still hold. It might be possible to show such lower bound as well, given that the scores sum to 1 but this needs to be done before making the second part of the claim mentioned above.\n\nMy score remains unchanged.", "other_comments_or_suggestions": "No comments other than above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0fJ7H64qL", "reviewer": "Reviewer_SCvk", "summary": "This paper analyses position bias in transformers, both theoretically and experimentally.\nThe paper first proposes to analyse a transformer as a graph, with attention weights representing weighted-edges between two tokens’ representations in adjacent transformer layers; an *attention flow* can then be computed between tokens $t$ and $t’$ at layers $\\ell$ and $\\ell’$, respectively, by summing the attention over all possible paths between them.\nThe paper then uses this graph-based framework to show how the attention flow changes as the depth of a transformers grows to infinity.\nThey consider models with autoregressive, sliding, and prefix attention masks. \nImportantly, they show that for these three models all attention flow converges to the first token position as the depth of a model grows.\nThey also consider models with no position embeddings, with ALiBi, and with ROPe.\nThey then experiment on a synthetic task, with results supporting their theoretical analysis.\n\n# Strengths\n\nThis paper studies an interesting phenomenon—position bias in autoregressive language models—providing new theoretical results justifying their existence.\n\nThis paper then supports this theoretical analysis with well designed experiments.\n\n# Weaknesses\n\nThe paper misses some critical literature in interpretability and analysis of language models, which could make analyses stronger if considered. In particular, the paper fails to cite Abnar et al. (2020), who first proposed the graph-theoretical analysis of transformers used in this paper (termed attention flow there). This framework has later been criticised/expanded by, e.g., Kobayashi et al. (2022), who point out that considering value-vector norms is important for more meaningful analyses involving attention flow. Further, discussing papers such as Jain et al. (2019) and Wiegreffe et al. (2019), and their relation to the analyses proposed here could significantly strengthen the paper.\n\nThe theoretical analysis ignores residual connections, which are integral to modern transformers. Residual connections can be interpreted as increasing the effect attention by one on diagonal entries, i.e.,: $A = \\widehat{A} + I$. Given this extra term, the condition necessary to prove Theorem 4.1 (i.e., $P_{ij} < 1- \\epsilon$) does not hold. So, do the theoretical analyses here only hold for transformers without residual connections? Furthermore, were residual connections used in experiments?\n\n* Jain et al. (2019). Attention is not Explanation. https://aclanthology.org/N19-1357.pdf \n* Wiegreffe et al. (2019). Attention is not not Explanation. https://aclanthology.org/D19-1002.pdf \n* Abnar et al. (2020). Quantifying Attention Flow in Transformers. https://aclanthology.org/2020.acl-main.385.pdf\n* Kobayashi et al. (2022). Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. https://aclanthology.org/2020.emnlp-main.574.pdf", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive assessment and constructive feedback, which have helped strengthen our work. Below, we provide individual responses to the comments you raised. \n\n> The paper misses some critical literature in interpretability and analysis of language models. \n\nThank you for the pointers to these important references. In particular, we will clarify how our approach aligns with, yet differs from, the “attention rollout” and “attention flow” methods proposed by Abnar et al. (2020). While those methods conduct post-hoc graph-based analyses on specific inputs to understand information flow, our work takes a more general and theoretical perspective: we formalize attention masks themselves as directed graphs (Sec. 3) and analyze how their structure shapes the flow of information across layers—independent of semantic content.\n\nThis allows us to rigorously trace how positional information is integrated across general sequences, highlighting the inductive bias introduced by masking alone. We hope this theoretical framework—which connects architectural design to emergent behavior—offers a useful new tool for understanding and probing attention mechanisms in future research.\n\nWe appreciate your feedback and will work diligently to ensure that our final version properly suits our contribution within the broader literature on understanding attention.\n\n> The theoretical analysis ignores residual connections, which are integral to modern transformers.\n\nThank you for your comment. One way to incorporate residual connections into our analysis is to follow the renormalization approach used in Abnar et al. (2020), where each attention matrix $A^{(t)}$ is adjusted to $A^{(t)}\\_{res} = 0.5 A^{(t)} + 0.5I$. This ensures $A^{(t)}_{res}$ remains a valid stochastic matrix and thus retains interpretability.   Under this adjustment, our theoretical results still hold; but the convergence rate would slow down, aligning with results in Dong et al. (2021) showing that residual connections slow down the rank collapse rate in token representations.  We will add a remark to the final manuscript detailing how to handle residual connections in our analysis and their effect. \n\n> Should matrices $W_q$ and $W_k$ have size $d_{QK}$ instead of $d'$? \n\nGood catch! Indeed, in many practical implementations, $d'$ matches $d_{QK}$​, However, we keep them distinct in our formulation to accommodate more general scenarios where these dimensions may differ (e.g.  Wang et al. (2024) proposes adjusting $d_{QK}$​​ at inference time to handle longer contexts more effectively).\n\n> Attention sink and center node: do your analysis does not actually justify the existence of attention sinks, right? Or does it?\n\nThank you for the thoughtful question. Our analysis does not directly justify the existence of attention sinks—defined as single-layer heads that concentrate all attention on a single token. Rather, our goal is to provide structural insight into where attention sinks tend to emerge and why they become more prominent in deeper layers.\n\n- In particular, we observe that the positions where attention sinks are most likely to appear (e.g., the first token under causal and sliding window masks, or the first K tokens under prefix masks; see Xiao et al., (2024); Gu et al., (2025)) coincide with the center nodes in the attention-mask graph defined by our framework. These nodes act as structural attractors in the multi-layer attention dynamics, helping to explain why they accumulate disproportionate influence over time.\n\n- Thus, while attention sinks remain a per-layer phenomenon, our graph-theoretic perspective offers a complementary explanation for their emergence in specific positions and their amplification in deeper layers.\n\nWe appreciate your questions and comments very much. Please let us know if you have any further questions.\n\n\n----\nReferences\n\nWang et al. (2024)  Length Generalization of Causal Transformers without Position Encoding.\n\nXiao et al. (2024) Efficient Streaming Language Models with Attention Sinks.\n\nGu et al. (2025) When Attention Sink Emerges in Language Models: An Empirical View.\n\nDong et al. (2021) Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth.\n\nAbnar et al. (2020). Quantifying Attention Flow in Transformers.", "claims_and_evidence": "Yes. The paper supports its claims with both theoretical analyses and practical experiments.", "methods_and_evaluation_criteria": "Yes. The only issue in this regard is the lack of residual connections in the theoretical analyses.", "theoretical_claims": "I checked the two first proofs and did not immediately find any issues.", "experimental_designs_or_analyses": "All experiments seemed sound and valid to support the paper's claims.", "relation_to_broader_scientific_literature": "I believe the lack of connection with prior work is the largest issue in this submission. The paper misses some critical literature. In particular, the paper fails to cite Abnar et al. (2020), who first proposed the graph-theoretical analysis of transformers used in this paper (termed attention flow there). This framework has later been criticised/expanded by, e.g., Kobayashi et al. (2022), who point out that considering value-vector norms is important for more meaningful analyses involving attention flow. Further, discussing papers such as Jain et al. (2019) and Wiegreffe et al. (2019), and their relation to the analyses proposed here could significantly strengthen the paper.\n\n* Jain et al. (2019). Attention is not Explanation. https://aclanthology.org/N19-1357.pdf \n* Wiegreffe et al. (2019). Attention is not not Explanation. https://aclanthology.org/D19-1002.pdf \n* Abnar et al. (2020). Quantifying Attention Flow in Transformers. https://aclanthology.org/2020.acl-main.385.pdf\n* Kobayashi et al. (2022). Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. https://aclanthology.org/2020.emnlp-main.574.pdf", "essential_references_not_discussed": "Yes. The paper claims one of its contributions is proposing a new graph-based framework for analysing transformers, but this framework was already proposed by Abnar et al. (2020). In particular, the main value analysed in this paper $P_{ij}^{(t)}$ is what Abnar et al. (2020) term attention flow.\n\nAbnar et al. (2020). Quantifying Attention Flow in Transformers. https://aclanthology.org/2020.acl-main.385.pdf", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* Line 125: Should matrices W_q and W_k have size d_{QK} instead of d’? The same question for W_v in line 146.\n* Attention sink and center node: As I understand it, attention sinks are not about the limiting behavior of attention flow as $t \\to \\infty$. Attention sinks are typically defined as a specific (single) attention head which puts all attention mass on a single previous position. If that’s the case, then your analysis does not actually justify the existence of attention sinks, right? Or does it?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fM8jgNi5ZU", "reviewer": "Reviewer_hruX", "summary": "This paper presents a graph-theoretic framework to analyze how position bias emerges in transformer architectures. The authors mathematically model attention masks as directed graphs to understand how tokens interact based on their sequential positions across multiple layers of attention.\nThe authors support their findings with experiments that reproduce position biases observed in real-world LLMs, including phenomena like \"lost-in-the-middle\" and attention sinks. Their framework helps explain why increasing model depth amplifies positional biases, why some positions gain disproportionate attention, and how different masking approaches affect information flow.\n\nThe main claims of the paper are:\n- Causal masking inherently biases attention toward earlier positions in deep networks. This happens because tokens in deeper layers attend to increasingly contextualized representations of earlier tokens, amplifying their influence.\n- There's a nuanced interaction between causal masking and relative positional encodings (like decay masks and RoPE). While these encodings introduce distance-based decay within individual attention maps, their aggregate effect across multiple layers creates a trade-off between local decay effects and the cumulative importance of early sequence positions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive assessment and constructive comments, which have helped strengthen our work. Below, we provide responses to your comments.\n\n> The paper primarily analyzes the attention mechanism without deeply exploring how other transformer components might interact with position bias.\n\nThank you for your thoughtful comment.  We agree that these other transformer components could contribute to position bias in an nontrivial way. We view this as an important direction for future work.\n\nThat said, our focus on the attention mechanism in this work is aligned with a common practice in the theoretical analysis of transformers: isolating the attention mechanism to better understand its intrinsic inductive biases. Many prior studies have adopted similar simplifications by omitting or abstracting away certain components in order to enable precise analysis [1-5].\n \nNevertheless, our analysis can be naturally extended to account for residual connections. Following the renormalization technique in [6], we can redefine the attention matrix at layer t as $A^{(t)}\\_{res} = 0.5 A^{(t)} + 0.5I$. This ensures $A^{(t)}_{res}$ remains a valid stochastic matrix and thus retains interpretability.   Under this adjustment, our theoretical results still hold; but the convergence rate would slow down, aligning with findings in [1] that residual connections slow down rank collapse rate under attention.  We will add a remark to the paper detailing how to handle residual connections in our analysis and their effect. \n\n> ...would benefit from more direct empirical validation with actual LLM attention patterns rather than just the controlled experiments. \n\nThank you for the comment. Natural language lacks ground-truth annotations for position bias, making it difficult to disentangle architectural effects from semantic content. To enable precise control, we adopt the synthetic setup from Reddy, which allows us to study positional bias in isolation. Despite this abstraction, our setup reproduces key behaviors observed in real LLMs, such as the “lost-in-the-middle” effect (Sec. 5.2) and attention sinks (App. K.2). Tab 1 further shows alignment between our results and empirical observations on position bias reported in the literature.\n\nWe agree that validating these results directly in real models with natural language data is an important next step. As noted in App K, one direction is to quantify position bias in LLMs and relate it to known linguistic structures [7]. We will highlight these directions in the updated version.\n\n\n> The experimental design has some minor limitations.\n\nThank you for your comment. The choices n = 8 and d=64 follow from Reddy. In line with your suggestion, we have added more variants for the decay masks and RoPE for the experiment in Sec 5.1 under the same setup:\n\nDecay \n| m | depth | first vs. middle | first vs. last | middle vs. last |\n|---|---|---|---|---|\n|  | 2 | -.025 | -.092 | -.070 |\n| 0.511 | 6 | -.057 | -.059 | -.006 |\n|  | 10 | -.044 | -.049 | -.002 |\n|  | 2 | -.043 | -.064 | -.022 |\n| 0.223 | 6 | .000 | .039 | .042 |\n|  | 10 | .030 | .075 | .039 |\n|  | 2 | .010 | .009 | -.011 |\n| 0.105 | 6 | .079 | .121 | .044 |\n|  | 10 | .110 | .148 | .073 |\n\nRoPE\n\n| $\\theta$| depth | first vs. middle | first vs. last | middle vs. last |\n|---|---:|---:|---|---|\n|  | 2 | .005 | .001 | -.002 |\n| 1/100 | 6 | .051 | .070 | .008 |\n|  | 10 | .075 | .088 | .012 |\n|  | 2 | 0.006 | .002 | -0.009 |\n| 1/1000 | 6 | .064 | 0.84 | .015 |\n|  | 10 | .079 | .086 | .018 |\n|  | 2 | .005 | .013 | -0.013 |\n| 1/10000 | 6 | .078 | .088 | .015 |\n|  | 10 | .092 | .104 | .013 |\n\nThe results align with our findings in Sec 4, with greater $m$ or $\\theta$ inducing greater decay, while deeper attention amplifying the bias toward earlier tokens.\n\nWe also present here the standard deviations for the results in Fig 2:\n \n|  |  | first vs. middle | first vs. last | middle vs. last |\n|---|---|---|---|---|\n|   | 2 | 0.016 | 0.014 | 0.009 |\n| no PE | 6 | 0.026 | 0.013 | 0.021 |\n|  | 10 | 0.011 | 0.013 | 0.005 |\n|  | 2 | 0.016 | 0.015 | 0.007 |\n| decay mask | 6 | 0.017 | 0.018 | 0.028 |\n|  | 10 | 0.010 | 0.019 | 0.011 |\n|  | 2 | 0.016 | 0.017 | 0.010 |\n| RoPE | 6 | 0.025 | 0.027 | 0.010 |\n|  | 10 | 0.020 | 0.025 | 0.005 |\n\n\nThe standard deviations for Fig 3 show a similar trend that they are small compared to the averages. This supports the robustness of our reported trends.\n\n\nWe sincerely appreciate your feedback and welcome any further suggestions.\n\n----\nReferences\n\n1. Attention is not all you need: pure attention loses rank doubly exponentially with depth\n\n2. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse\n\n3. A mathematical perspective on transformers\n\n4. The emergence of clusters in self-attention dynamics\n\n5. On the Role of Attention Masks and LayerNorm in Transformers\n\n6. Quantifying Attention Flow in Transformers\n\n7. Large-scale evidence of dependency length minimization in 37 languages", "claims_and_evidence": "The connection between center nodes in their graph-theoretic framework and attention sinks in real models is theoretically elegant but would benefit from more direct empirical validation with actual LLM attention patterns rather than just the controlled experiments.", "methods_and_evaluation_criteria": "The authors' methodological choices seem good:\n\n- Using a graph-theoretic framework is appropriate for analyzing information flow in attention mechanisms.\n- The controlled experimental setting from Reddy (2024) allows for isolating the effects of different architectural components on position bias.\n- The evaluation metric (accuracy gap between different positions) directly measures position-dependent performance differences.\n\nEven though, focusing only on syntetic data leaves the question on whether the same dynamics would emerge in natural language data with more complex semantic structures.", "theoretical_claims": "The theoretical claims seem sound.", "experimental_designs_or_analyses": "The experimental design is generally sound but has some minor limitations:\n\n- The experiment uses very small model sizes (n=8, d=64) compared to real LLMs, raising questions about whether the observed effects scale appropriately to larger models.\n- While the authors test three different position encoding schemes, they use only a single implementation of each. Testing variants (like, different decay rates for decay masks or base frequencies for RoPE) would strengthen the generalizability of the findings.\n- The statistical analysis is minimal, with results presented as averages over five runs but without confidence intervals or significance testing, making it difficult to assess the reliability of the observed effects.", "relation_to_broader_scientific_literature": "I think this paper is valuable: the paper builds a bridge between empirical observations of positional phenomena in transformers and theoretical understanding of their architectural causes, while in the literature these phenomena were observed but not fully explained.\n\n- The paper connects to the literature on the \"lost-in-the-middle\" phenomenon described by Liu et al. (2024), Zhang et al. (2024), and Guo & Vosoughi (2024) - the authors provide a theoretical explanation based on the interplay between positional biases from the architecture and biases in the training data, showing how specific patterns of position-dependent performance emerge under different conditions.\n\n- The paper connects to recent work on attention sinks by Gu et al. (2025) and Xiao et al. (2024) - the authors provide a theoretical explanation by showing that attention sinks naturally emerge at center nodes in the directed graph defined by the attention mask. This explains why sinks form at the beginning of sequences under causal masking or at all prefix tokens under prefix masking.", "essential_references_not_discussed": "I'd suggest the authors to check these papers:\n\n- Stolfo, Alessandro, et al. \"Confidence regulation neurons in language models.\" Advances in Neural Information Processing Systems 37 (2024): 125019-125049.\n\n- Cancedda, Nicola. \"Spectral filters, dark signals, and attention sinks.\" arXiv preprint arXiv:2402.09221 (2024).", "other_strengths_and_weaknesses": "other weaknesses:\n- The paper primarily analyzes the attention mechanism without deeply exploring how feed-forward networks, residual connections, and other transformer components might interact with position bias.\n\nother streanghts:\n- The work successfully connects multiple empirical observations about position bias (attention sinks, lost-in-the-middle effect) within a coherent theoretical framework, offering deeper understanding of these phenomena.\n- The authors provide formal proofs for their theorems, making their claims verifiable and establishing a solid foundation for future theoretical work in this area.\n- The controlled experiments are well-designed to isolate the effects of different components (mask type, positional encoding, model depth) on position bias, validating the theoretical findings.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6k7u1EP5zT", "reviewer": "Reviewer_qhRD", "summary": "This paper studies the \"position bias\" in transformers, that is, the bias of the model to focus on certain regions of the input. The authors investigate how causal mask and positional encoding impact this position bias. To that end, they leverage a graph-theoretic formalization of the attention module to study the position bias. In particular, the authors unify under their study this bias and its empirical observations shown in prior works. The authors show that causal masking biases attention towards earlier positions and conduct this analysis for several types of masking used in practice. The authors also study how positional encoding and masking interact and provide insights into how to design them to trade-off between local and global context. Finally, the authors validate their findings experimentally in a controlled setting.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and positive assessment of our work. Below, we provide individual responses to the comments and questions you raised.\n\n> I wonder how MLPs would impact the current analysis. \n\nThank you for the question. Our analysis focuses on the self-attention mechanism, as it is the primary component responsible for information exchange across tokens. In contrast, MLP layers are typically applied independently to each token and do not directly contribute to inter-token communication. As such, they are unlikely to affect position bias in the same way that attention mechanisms do.\n\nThat said, we acknowledge that MLPs, being universal function approximators, can theoretically induce position-dependent effects under specific conditions. For instance, one could construct a scenario where an MLP maps $x_i$ to $x_{n-i+1}$ for a sequence  $x_1, x_2, x_3, x_4, …, x_{n-1}, x_n$​, thereby indirectly altering token interactions. However, such behaviors would require highly specific weight configurations that are unlikely to arise under standard training, and they would generally lack robustness to input variation.\n\nOur goal in this work is to isolate and understand the role of attention masks and positional encodings in shaping position bias—factors that more directly govern how positional information is integrated across tokens. Nonetheless, extending our framework to investigate whether and how MLP layers might interact with or amplify these biases is a valuable direction for future work, and we will add a discussion of this point in the revised manuscript.\n\n> What do the authors think of other families of \"sparse\" softmax, like sparsemax or fenchel-young derived softmax. Would it make sense to study them through the lens of avoiding attention sinks?\n\nThank you for the question. Our current findings suggest that sparser attention mechanisms may help mitigate attention sinks by slowing the convergence of attention flow. In our framework, this is captured by Theorem 4.2, which shows that increased sparsity (e.g., shorter sliding windows) causes center nodes to accumulate influence more gradually, making them less likely to dominate early on. Consistently, we observe empirically that attention sinks are less likely to emerge under sparser masking conditions (Figure 10).\n\nThis connection suggests that sparse alternatives to softmax could act as implicit regularizers, dampening the formation of attention sinks and promoting more balanced information flow across tokens. Extending our analysis to formally validate and characterize these effects would be a valuable direction for future work, and we will note this in the revised manuscript.\n\n> Could the authors elaborate on larger-scale experiments that could validate their theory or that can benefit from their findings?\n\nThank you for the question. A key contribution of our theoretical framework is that it helps identify and quantify the sources of attention bias introduced by masking and positional encodings. These insights can inform the design of alternative transformer architectures, by motivating positional encoding schemes or masking strategies that encourage more uniform or task-aligned attention distributions.\n\nLarger-scale experiments could build on our findings to evaluate how different architectural choices impact position bias and downstream performance on different tasks. Conversely, one could also explore whether alignment between a model’s learned position bias and natural language structure (e.g., recency effects in Futrell et al.) correlates with improved language modeling performance.\n\n> Given that the graph-theoretic framework of attention was introduced in Wu et al. (2024), I think that the mention of \"novel graph-theoretic framework\" in the abstract is not adapted.\n\nThank you for the thoughtful comment. We agree that the use of graph-theoretic tools to analyze attention mechanisms has been explored in prior work, notably by Abnar et al. and Wu et al. We will revise the abstract to more accurately reflect our contribution. We will revise the wording in the abstract to more precisely reflect our contribution.\n\nThe novelty of our work lies not in the use of graphs itself, but in how we apply this perspective to analyze the flow of information and the emergence of positional bias across multiple layers of attention. Our framework leverages the structure induced by causal and other attention masks to provide theoretical insight into how positional preferences arise independent of semantic content. We will clarify this distinction in the revised abstract to better situate our contributions. \n\nWe sincerely appreciate your feedback and welcome any further suggestions.\n\n---\nReferences\n\nAbnar et al. (2020) Quantifying Attention Flow in Transformers.\n\nFutrell et al. (2015) Large-scale evidence of dependency length minimization in 37 languages.", "claims_and_evidence": "The theoretical claims are well supported by clear and detailed proofs, and the authors also provide experimental validation of their theory in a controlled setting.", "methods_and_evaluation_criteria": "The authors provide theoretical results to better understand position bias in attention blocks. They conduct experiments to validate their findings. I believe the method and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "The proofs are well-detailed and clear.", "experimental_designs_or_analyses": "The authors provided the code to reproduce their results. A quick inspection of the code seems to indicate that the experiments were well-conducted, although I did not re-run the experiments. The experimental design is sound, and results are analyzed, although the experiments are only done in a controlled, hence not very realistic, setting.", "relation_to_broader_scientific_literature": "I find that related work and prior works are well introduced and compared. Although the graph-formalization was introduced in a prior work [1], the submission's contributions seem to be novel regarding the understanding of position bias in attention layers by leveraging the graph formalization of attention layers.\n\n*References*\n\n[1] Wu et al. On the Role of Attention Masks and LayerNorm in Transformers, NeurIPS 2024", "essential_references_not_discussed": "To the best of my knowledge, there are no essential references not discussed in the current submission.", "other_strengths_and_weaknesses": "**Strenghts**\n\n- The paper is very well written with detailed prior work, clear notations and technical background\n- I find the graph formalization very elegant and innovative. \n- The theory is sound, with theorems contextualized and their implications explained\n- The contributions summarized in Table 1 are impressive, especially given the many connections to empirical findings in prior works\n\n**Weaknesses**\n\nI list below what I think are weaknesses, but I would be happy to be corrected if I misunderstood some important aspects of the authors' contributions.\n- The current analysis focuses on self-attention-only networks, while transformers include feed-forward layers. Although tokens are only interconnected in the attention blocks, the influence of feed-forward layers is not negligible, and I wonder how it would impact the current analysis. Could the authors elaborate on this point?\n\n- In the experiments, the network and data considered are simple, which, I believe, is due to the need to be in a controlled setting to investigate properly the position bias. Could the authors elaborate on larger-scale experiments that could validate their theory or that can benefit from their findings (I do not ask the authors to conduct them, but rather to add a discussion for future work in more practical settings)? For instance, this could be in a limitation or discussion section at the end of the paper.\n\n- Given that the graph-theoretic framework of attention was introduced in [1], I think that the mention of \"novel graph-theoretic framework\" in the abstract is not adapted. I acknowledge that this does not reduce the novelty of the contribution since it is the first time it is used to study the position bias, but I would appreciate if the author could elaborate on this if I misunderstood something or remove this term in the abstract if the framework is indeed not new.\n\nOverall, I find the paper interesting and the analysis well conducted. Although the model is simplified and the experiments are limited to controlled settings, I believe this is valuable work to better understand position bias in transformers. This is the reason for my current score and I would appreciate if authors could clarify the points mentioned above.\n\n**Update after rebuttal**: I increased my score from 4 to 5.", "other_comments_or_suggestions": "None", "questions_for_authors": "1) Could the authors elaborate on how the analysis conducted in the current submission could be extended or influenced by considering feed-forward layers? In particular, does the position bias only depend on the attention layers since this is the only block where tokens are interconnected, or can the MLPs also influence it, maybe in some indirect manner?\n\n2) In the paragraph after Theorem 1, the authors discuss the role of softmax. If I understood correctly, softmax cannot lead to disconnected graph components since it is not sparse and cannot have exactly zero entries. The authors mention that, empirically, replacing softmax with ReLU can mitigate the emergence of attention sinks. What do the authors think of other families of \"sparse\" softmax, like sparsemax [1] or fenchel-young derived softmax [2]. From Theorem 1, would it make sense to study them through the lens of avoiding attention sinks?\n\n3) Could the author add a discussion/limitation section regarding the scope of their study (controlled setting, simplified transformer) and some room for future work in more practical settings?\n\n*References*\n\n[1] Martins et al. From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification, ICML 2016\n\n[2] Blondel et al. Learning with Fenchel-Young Losses, JMLR 2020", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has received a wide range of reviews. On one hand, some reviewers acknowledge the contributions of this paper, including the new theoretical results for position bias in autoregressive language models and new explanations about how model depths affect positional biases. However, on the other hand, reviewers have raised strong concerns about the mismatch between the theory and empirical observations. Specifically, the authors develop a theory for the attention scores over original tokens, while the empirical observations, such as attention sink, are reported in terms of per-layer attention patterns. Additionally, the paper only states a correlation between attention sinks and the developed theoretical results, without providing an underlying explanation for why attention sinks emerge.

Overall, despite the debates and potential issues with this paper, most reviewers have provided positive evaluations and believe that the identified issues can be addressed in the camera-ready version. The authors should seriously consider these critiques and make the necessary modifications accordingly.
Decision: Accept (poster)