What are you sinking? A geometric approach on attention sink
Authors: Valeria Ruscio
Umberto Nanni
Fabrizio Silvestri
Venue: NeurIPS 2025 spotlight ()
Keywords: Attention sink
Transformer geometry
Position encoding
Self-attention
Primary area: deep_learning
TL;DR: Attention sink in LLMs serves as geometric reference frames that anchor token representations in high-dimensional space, emerging during training as optimal solutions to the coordinate system problem, shaped by architecture and position encodings.

Abstract: Attention sink (AS) is a consistent pattern in transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of a fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in high-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.

What are you sinking? A geometric approach on attention sink
Valeria Ruscio, Umberto Nanni, Fabrizio Silvestri
Sapienza University of Rome
ruscio@diag. uniromal it, fsilvestri@digg, uniromal.
Abstract
Attention sink (AS) consistent pattern transformer attention maps where certain tokens (often special tokens or positional anchors) disproportionately attract attention from other tokens. We show that in transformers, AS is not an architectural artifact, but it is the manifestation of fundamental geometric principle: the establishment of reference frames that anchor representational spaces. We analyze several architectures and identify three distinct reference frame types, centralized, distributed, and bidirectional, that correlate with the attention sink phenomenon. We show that they emerge during the earliest stages of training as optimal solutions to the problem of establishing stable coordinate systems in figh-dimensional spaces. We show the influence of architecture components, particularly position encoding implementations, on the specific type of reference frame. This perspective transforms our understanding of transformer attention mechanisms and provides insights for both architecture design and the relationship with AS.
Introduction
Transformer-be models exhibit an interesting phenomenon called "attention sink" where beginning- of-sequence tokens receive substantial attention (30-40%) regardless of semantic relevance (Xiao et al. 2023). Removing this pattern degrades model performance, suggesting these allocations serve an essential function beyond content processing. We postulate that attention sinks represent transformers' reference frames, coordinate systems within their representational manifolds. Transformer operations rely dot products between query and key vectors, making angular relationships crucial for information flow. Reference frames provide fixed geometric anchors that allow tokens to establish relative positions through consistent angular relationships, solving the challenge of maintaining stable geometric relationships in high-dimensional spaces. Without these anchors, token representations would lack consistent orientation, making reliable encoding of positional and semantic relationships impossible. We show that reference frames emerge as mathematically optimal solutions to constraints imposed by the softmax operation on the probability simplex. We categorize them into three classes: centralized, distributed, and bidirectional. While prior work [Gu et al.) 2024] Barbero et al. [025] treated attention sinks model-specific phenomena, our geometric interpretation unifies these observations alternative solutions to the same fundamental challenge, advancing understanding of transformer geometry.
Related works
The attention sink phenomenon, where tokens allocate substantial attention to specific tokens regard- less of semantic relevance, first identified by Xiao et al. [2023). who discovered the beginning- of-sequence token consistently receives disproportionate attention. This aligns with broader efforts to develop mechanistic understanding of transformer models (Elhage et al. 2021 While Yu et al.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2024 and Cancedda [2024] found attention sinks emerging beyond sequence beginnings (which our framework explains as distributed reference frames), Gu et al. [2024] observed that high cosine similarity between queries and the first token's 's keys creates attention sinks despite the keys' small f2-norm, a phenomenon our theory reframes as the low-norm keys establishing a distinguished point in the representation manifold. /Zhan et al.] (2025) 's "catch, tag, and release" mechanism aligns with our finding that reference frames emerge early in training as mathematical necessities, while Barbero ctt al. [2025] frames attention sinks as preventing information 'over-mixing" in deep networks.
Position encoding significantly influences attention patterns: [Su et al. 2024 Rotary Position Embedding creates bias toward first-token attention (facilitating centralized reference frames), while altematives like ALiBi [Press et al., [02T] demonstrate how encoding changes affect attention patterns. Ruscio and Silvestri [2024] found wavelet-like patterns emerging among attention heads to overcome RoPE limitations, and Kazemnejad et al. 2023] showed scaled RoPE variants dramatically change attention patterns, which our framework explains as enabling distributed reference frames through reduced positional bias. Research by [iu et al. (2024). Voita et al. (2019), and Mohehbi al. [023] supports our prediction that architectural design directly influences reference frame formation. Finally, Darcet et al. [2024) observed that absolute position embeddings create topological complexity supporting bidirectional reference frames (consistent with our encoder-only findings), while Sun et al. [2024) demonstrated that explicit bias parameters can mitigate attention sinks, which our framework interprets altering the manifold's geometric structure.
3 Reference Frames
It has been shown by Moschella et al. [2022] that in high-dimensiona representation spaces, latent vectors need a consistent way to relate to each other by establishing reference frames acting as canonical coordinate systems to anchor the representation manifold. Without stable reference points, the model struggles to consistently determine relationships between tokens, and distances and directions become ambiguous. At its core, a reference frame in a transformer is a structure R = (M, p) where M the representation manifold (a smooth, locally Euclidean topological space), P {pI, P2, ..... ............. } is set of reference points that act distinguished locations on the manifold, and á .M x P + Rd is mapping function that relates any point to the reference points, providing a coordinate chart for the manifold. This structure provides a means to consistently measure relationships between token representations regardless of their semantic content or position.
This definition allows us to formalize what constitutes an "attention sink" mathematically: an attention sink is a token position j for which the attention weight oij exceeds threshold r across many source tokens
sink(j) (j)
=
(a1)27)
27
(I)
n2 i=1
where 1 is the indicator function, T is typically set to the 90th percentile of attention weights, and ? frequency threshold (typically 0.3-0.5).
Reference frames emerge through ceff-organization during training rather than being explicitly programmed. However, this self-organizztion occurs within architecture aecitecture-specffic channels that significantly influence the resulting geometric structures. This guided emergence can be formalized as optimizing loss function L over an architecture-specific inductive bias B.
These inductive biases don't deterministically program specific attention patterns but rather create the conditions where certain geometric structures naturally emerge as optimal solutions during training. The architecture shapes the loss landscape such that gradient descent naturally converges toward specific reference frame types. This explains why attention sinks consistently form during training without being explicitly encoded the architecture.
3.1 Vector geometry of reference frame types
We identify three distinct reference frame types, each characterized by specific geometric orga- nization within the attention mechanism's vector space. Our analysis shows that attention sinks tokens consistently receive disproportionate attention weight regardless of their semantic content, functioning as geometric anchors in the representation space. These attention sinks fall into two distinct categories: they appear either special tokens (like [BOS] or [CLS] and [SEP]) that mark

---

sequence boundaries or as regular vocabulary tokens (such as commas or articles) that serve syntactic functions the text.
Centralized Reference Frames emerge in decoder-only architectures with standard RoPE (LLaMA 3.1 and 3.2, Mistral v0.1, Gemma). The soanininnnoo--uueeenne token [BOS] becomes a universal origin point where each token's query vector maintains high cosine similarity with this reference token's s key vector, even as the reference key's magnitude (42-norm) remains small. This creates a computational hub where all tokens can efficiently establish their relative positions through single comparison operation. When processing "The cat chased the mouse", attention from "mouse" would focus substantially (30-40%) the [BOS] token rather than semantically related tokens, creating an efficient comparison path "mouse ++ [BOS] ++ cat". This can be expressed as a transformation where token representations oriented primarily through their relationship to the central reference point:
h Br,BOS VBOS + Σ aiyV
B
where og typically ranges from 0.3-0.4 (30-40%) regardless of semantic relationship. Distributed Reference Frames emerge in architectures with modified positional encoding schemes (Qwen 2.5, Phi-2). Multiple tokens serve reference points, creating system. At the vector level, multiple key vectors maintain moderate cosine similarity with various query vectors across the sequence, creating network of local reference points. In our example, attention from "mouse" would be allocated (10-15%) to several anchoring tokens, creating multiple computational paths: "mouse + [Ref] <> cat" and "mouse t> structure creates multiple, mmaller-weigh transformations:
flexible coordinate
"the" + cat." This distributed reference
h = Σ as, +
Σ Lasy JER
FER
where R is the set of reference tokens and oi, typically ranges from 0.1-0.15 (10-15%) for each reference.
Bidirectional Reference Frames emerge in encoder architectures with absolute position embeddings (BERT, XLM-RoBERTa) Both the beginning and end tokens serve as reference points, with attention patterns shifting through network depth. Early layers establish relationships with the beginning token's ' vector, while deeper layers increasingly reference the end token's vector, creating a dynamic coordinate system that changes through network depth. This dynamic reference structure can be formalized as layer-dependent transformation:
h (
Σ
3 a
Σ
Oli V
jE (start,end))
jd (start,end)
where 5100 are layer-specific weighting factors that shift from start-dominant early layers to end-dominant in later layers.
Centralized
Bidirectional
Distributed
Ref
Bo5
Bo5
EoS
Ref
Taken
Reforncepeiit
layers
Lata layers
Figure 1: Geometric interpretation of reference frames: (left) centralized framme with a single dominant reference point serving as a universal origin; (center) distributed frame with multiple weaker reference points creating a flexible coordinate system; (right) bidirectional frame with dual-anchor structure and layer-wie wise specialization.

---

When a token attends to a reference point, it performs vector operation that orients its representation within the shared coordinate system:
h Σ any (Wyx)) = as W Wxee
Σ (Wy())
(5)
jytref
This explains why removing attention sinks often disrupts performance despite their seeming semantic irrelevance, they provide the geometric infrastructure that makes consistent representation possible. The mathematical significance attention matrices. Dominant eigenvectors align with reference tokens, creating stable subspaces that UV where
reference frames becomes clear the eigendecomposition of
serve as coordinate axes. This alignment is captured the low-rank approximation columns of U' correspond to reference points that minimize the Frobenius norm error A UV |r- These reference structures enable transformers to perform implicit basis transformation operations that maintain geometric consistency across sequence positions.
3.2 Information Geometry and the Probability Simplex
The softmax operation in attention fundamental to reference frame formation, as creates geometric constraint through the probability simplex [Zuhri et al. 2025
A n-1 = {(11 a2.... ..., an) E R
aj 2 for all j, and Σ aj 11
(6)
This constraint mathematically necessitates the emergence of reference frames by creating two critical geometric effects: i) it transforms unbounded attention logits into bounded manifold with intrinsic curvature, and ii) it enforces conservation law that makes attention a zero-sum resource, more about in A The resulting geometry privileges sparse attention distributions that concentrate probability mass on minimal set of tokens, precisely the pattern observed in reference frames. Each attention distribution a, represents a point on the probability simplex A n-1 inducing a Riemannian metric through the Fisher information matrix Fú E log p(z) ({))a 3 log p(z)]. This metric determines how distances are measured in the representation space and creates the conditions for attention sinks to emerge as geodesic reference points.
In the context of attention mechanisms, the information bottleneck principle (Tishyy et al. 2000 translates to optimizing attention distributions on the probability simplex:
min
Σ 1
uig wiy (ai(ala)))) + (R,a)),..,a,)
Gr
where Wiy represents the semantic relevance between tokens and and R is regularization term. The attention distribution from each token defines probability measure on the manifold, inducing a metric structure through the Fisher information matrix. This metric determines how distances measured in the representation space.
3.3
The Emergence Attention Sinks
The attention sink phenomenon be formalized through the interaction between position encoding and the self-attention mechanism. In self-attention, the attention score between tokens at positions and is:
exp(q: kj/vd)
Oij E explq: k//V))
where qi = Wox; and kj = W kX, are the query and key vectors. With RoPE, this becomes:
RRo, Wox,) (Ro, WRX,)/V)
Oij Σ Σ (Ro W Wox:) (Ro, W Wxx1)/Vi)
For the first token (j = 1), RØ, I (the identity matrix). creating computational advantage that biases attention toward this position. This mathematical property, combined with the simplex constraint of the softmax, naturally leads to the formation of an attention sink at the beginning-of- sequence token in models with standard RoPF. Specifically, when k, has a small form relative to other keys but maintains high angular alignment with queries, it creates the conditions for an attention

---

sink. This can be expressed through the dot product decomposition: q; K1 = =|: |k cos (g This balance between small norm and high cosine similarity represents an optimal trade-off in the attention mechanism's s geometry. A small | ensures the reference token doesn't ' dominate the output representation magnitudes, while high cos(Og,.)) k1 ensures consistent geometric relationships. This explanation clarifies why attention sinks aren't 't merely artifacts but optimal geometric solutions that emerge consistently during training despite different initialization conditions.
3.4 Position Encoding and Reference Frame Formation
Position encoding implementations fundamentally shape the geometric organization of transformer attention, directly influencing which reference frame type emerges during training. Our analysis shows thatthe mathematical structure of position encoding creates specific inductive biases that guide attention patterns toward distinct geometric configurations. Different encoding schemes establish different coordinate geometries through their modifications of token embeddings:
Standard Rotary Position Embeddings (RoPE) used in architectures like LLaMA, apply rotation matrix R. with frequency-depenee angles: X; + Rg,x, where 0; = w for fixed frequency w. This creates fundamental geometric asymmetry: the first token (position 0) receives no rotation at all (f,- = 0), making its rotation matrix the identity matrix (Ry- = I). This mathematical property gives the first token a privileged computational status in the attention mechanism. As other tokens undergo increasingly larger rotations based on their positions, their query vectors maintain higher cosine similarity with the first token's key vector compared other tokens similar semantic distances. During training, the model naturally exploits this computational advantage, developing a centralized reference frame where the first token becomes a universal origin point in the representation space. Topological analysis of these models reveals star-like attention graphs with dominant central node, reflected in low Betti, values (few cycles) and strong negative correlation between algebraic connectivity and degree centralization. This structure corresponds to a pointed manifold (M, Po) with a distinguished point po serving as universal origin. This geometric configuration optimizes the probability simplex constraint by placing high probability mass on single token, creating a sparse attention distribution that enables efficient long-range dependency modeling.
NTK-aware scaled RoPE emmloyed by models such Qwen 2.5 and Phi-2, introduces critical geometric modification to standard rotary embeddings. By applying a scaling factor a < 1 to the rotation angles: x; + R, X,, X,. This approach fundamentally alters the manifold geometry of attention. The reduced rate of angular separation diminishes the computational advantage of the first token by making the relative rotations between tokens more uniform throughout the sequence. This seemingly minor mathematical change has profound effects on reference frame formation. With the positional bias toward the first token weakened, the model no longer converges on single dominant reference point during training. Instead, multiple tokens can effectively compete as reference points, creating a distributed network of local coordinate systems. Spectral analysis of these models reveals a characteristic flippiiiig correlation pattern between connectivity and centralization metrics that reflects this fundamental reorganization of the attention manifold. The geometric transformation can be understood as a shift from pointed manifold (M, Po) with single distinguished origin to a multi- pointed manifold (M, {pI: P2, ..., Pm H). This distributed structure sacrifices some of the efficient hub-and-spoke compression of centralized frames for greater contextual adaptability, enabling more flexible modeling of diverse linguistic structures.
Absolute position embeddings implemented in encoder architectures like BERT and XLM- RoBERTa, employ fundamentally different approach to encoding position: x; + X; + P:. where P; is a learned embedding for position i. Unlike rotational methods that modify the geometric relationships between tokens through angular transformations, absolute embeddings directly in- ject position-specific information into each token's representation. This creates different kind of inductive bias, rather than establishing implicit reference points through computational advan- tages, absolute embeddings create explicit position markers throughout the sequence. This approach naturally supports the emergence of bidirectional reference frames with reference points at both
'TK stands for Neural Tangent Kernel, in the context of transformers, this approach modifies position encodings based on network capacity rather than using fixed frequencies, allowing models to better generalize to lengths beyond their training distribution.
5

---

sequence boundaries. Topologically, these models display remarkably high initial complexity with elevated Betti, values, indicating numerous cycles in early layers, a strong contrast to the star-like structures in decoder models. The spectral analysis shows peak connectivity in early but not initial layers, with dramatic connectivity drops between early and middle layers, suggesting rapid geometric reorganization. As the network deepens, attention systematically shifts between reference points, creating a dynamic coordinate system that changes through network depth. This bipolar manifold (M,,tttrt: Pena) implements layer-specific optimization strategies that shift attention mass between special tokens, enabling simultaneous access to both sequence endpoints, a critical capability for tasks requiring bidirectional context integration.
Pstart:
4 Methodology
Geometric and Topological Analysis Standard attention analysis cannot distinguish structured coordinate systems from semantic similarity patterns. We quantified the topological structure of atten- tion networks through persistent homology and Betti numbers, revealing how connectivity patterns evolve across network depth. Our approach draws on the growing application of topological data analysis to neural networks Naitzat et al. 22200 Moor et al. 2020) and the established methodology of persistent homology for analyzing high nigh-dimensionl data structures [Edelsbrunner al. 2008
(Distance Matrix = 1 - Attention Matrix).
The Ripser algorithm computed persistent homology on these distance matrices, tracking Bettig (con- nected components) and Betti, (cycles) across layers. Persistence values measure the significance of topological features, with higher values indicating more stable structures. This approach revealed characteristic topological signatures for each reference frame type.
To complement topological features with algebraic characterization, we examined the spectral proper- ties of attention graphs' Laplacian matrices to assess connectivity patterns and reference structures (L - - A where D is the degree matrix and A is the adjacency matrix derived from thresholded attention weights). We computed these matrices at multiple thresholds (0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2), measuring: i) Algebraic connectivity (Fiedler value) to quantify graph connectedness; ii) Star-likeness, to measure proximity to ideal star topologies, directly testing for hub-based reference frames; iii) Gini coefficient to quantify inequality in attention distribution; and iv) Degree centraliza- tion assess concentration of attention. We analyzed correlations between these metrics identify mathematical signatures of different reference frame types.
Information- Theoreeic Analysis While topology reveals structural patterns, understanding whether reference tokens merely aggregate information or fundamentally establish coordinate systems re- quires quantifying their functional impact. We first quantified attention distributions, particularly how attention sinks affect information flow: KL Reduction KL (originiall) - KL((ithout sinks). Our implementation identified attention sinks using percentile thresholds (0.8, 0.9, 0.95), measuring the changes in KL divergence when attention sinks were removed, the attention sink concentration across network layers, and the layer-specific patterns in how reference points influence information geometry. Large KL reductions indicate that sinks establish organizational principles rather than superficial attention patterns.
-eoometrr properties of
We then examined how reference frames manifest in value space geometry through complementary quantitative approaches. We measure the influence of reference tokens by calculating their relative magnitude, the ratio between reference token keys' fnrmorm and average keys (l 2/ + E || [k/l2)), revealing how these points establish distinguished positions in representation space. We capture directional guidance through cosine similarity between reference values and transformation vectors ( Ei= cos( h/ h;)), while quantifying their structural importance via KL divergence be- tween original and iiferrence-emovve attention matrices (DRI. (AI A.er)) To distinguish between reference frame types, we identify the number of significant reference points across architectures. We further investigate the relationship between attention and geometric transformations by examining attention entropy (- Σ, aiy log aij), transformation magnitude (l h; and their correlation (corr( H (A.). ||h; h,llz)), We also measure geometric-semante alignment (corr(a:j) cos(v;, v))) to determine whether attention follows semantic relationships or prioritizes geometric organization. The directional influence metric is particularly interesting as it directly quantifies how reference tokens shape the geometric structure of the manifold. High values indicate that reference tokens actively guide the orientation of all other token representations, functioning as coordinate axes rather than merely aggregating contextual information. Similarly, gcometric-semantee alignment measures

---

whether attention patterns follow semantic content (positive values) or abstract geometric principles (negative values), providing insight into how reference frames balance content processing with coordinate system establishment.
To understand how reference frames affect learning dynamies and parameter sensitivity, we computed the Fisher information matrix, which provides natural Riemannian metric on the manifold of probability distributions, measuring how sensitive model outputs are to parameter perturbations. For
each model, we computed the layer-wise Fisher norm Fell v /Ex [Felly where Fe represents
the Fisher information matrix for parameters in layer f. This quantifies the information content and learning capacity cach layer with respect to the attention distribution manifold, revealing whether properties.
reference frame layers exhibit distinctive information-geomeett
Random Matrix Theory Analysis of Attention Evolution The temporal emergence of reference frames shows whether they represent fundamental architectural necessities or learned optimizations. If reference structures emerge early and consistently across initializations, they likely constitute essential computational primitives. To study this, we employed Random Matrix Theory (RMT) to analyze how
attention structures emerge from initially random patterns: PMP(2) v (b-=a)a-a)
a S b
2
where a = (1 - / b (1 + v and is the aspect ratio of the matrix. Deviation from this Marchenko-Passee distribution indicates the emergence non-random structure. We quantified this
Participation Ratio C, A.) A emergence through several metrics: Spectral Gap = Xa
Σ.
DKL OKL(PempllMM)) = Σ Pemp (t) log
Pemp
(10)
PMP(1)
where X; are the eigenvalues of the attention matrix, and Pemp is the empirical eigenvalue distribution. For each checkpoint during training, we extracted attention matrices from all layers and heads, computed their eigendecompositionn and tracked the evolution of these metrics. This methodology allows us to identify when reference frames begin to form and how they develop through training, providing insights into the fundamental role these structures play in the learning process.
We tracked these metrics across training checkpoints Pythia models from the earliest stages (step0 to step8) to later points (step9000 through step1 143000), identifying when reference frames begin to form and how they develop through training.
Experimental design We analyzed diverse set of transformer models to investigate architecture- specific and architecture-invarinl patterns in reference frame formation: decoder-only models LLaMA-3.2 (IB, 3B) and 3.1 (8B- Instruct, 8B), Phi-2, Qwen-2.5 (3B, 7B, 7B-Instruct), Mistral-7B- v0.1, Gemma-7B, Pythia (1.4B, 2.8B, 6.9B, 12B); and encoder-only models baeRT-bae--nccaedd. XLM-RoBERTa--arge
For topological, spectral graph, value space and KL divergence analyses, we used dataset of STEM-focused Wikipedia sentences (mathematics, chemistry, medicine, physics) ranging from 6 to 50 tokens. We processed 500 samples for topology, spectral and Fisher information analysis, and a subset of 50 samples for KL divergence analysis. For the temporal RMT analysis, examined 100 samples across training checkpoints of Pythia models. All experiments were conducted using Google Colab with T4 A100 GPUs.
5
Analysis Results
As we can see from table [ each type exhibits characteristic mathematical signatures across our analytical methods, yet all establish stable coordinate systems for representation learning.
5.1 Centralized Reference Frames
Centralized reference frames emerge in decoder-only architectures with standard rotary position embeddings (RoPE), including LLaMA, Mistral, and Gemma. Our spectral graph analysis reveals strong negative correlation between algebraic connectivity (Fiedler value) and degree centralization, indicating that centralized attention structures prioritize information concentration over distributed connectivity, as shown in table This creates a star-like topology with a dominant central node. The KL divergence in table analysis consistently shows negative KL reduction values when attention
7

---

Table 1: Topological Analysis of Reference Frame Types
Property
Llama-3.2-3B
Qwen2.5-7B
XLM-RoBERTa
Reference frame type
Centralized
Distributed
Bidirectional
Connected
Early layer (Bettin)
26.43
26.16
22.68
Components
Final layer (Bettia)
26.43
17.97
1.69
Change
0.00
-8.19
-20.99
Cycles/Loops
Early layer (Betti:) Final layer (Betti;)
0.00
0.00
19.69
0.00
0.00
3.11
Change
0.00
0.00
-16,58
Topological Persistence
Early layer persistence Final layer persistence
0,0573
0.2381
0.0521
0.1620
0.1543
0.0156
Change
40.1046
-0.0838
-0.0365
Attention Head Specialization
Token specialization Specialized heads Top layer specialization
100% on BoT
65.4% on "."
100% on 
/</>>
120
7 Layer 0 (7 heads)
77
Layer 0 (24 heads)
Layer 18 (16 heads)
Attention Standard Deviation
Early layer
0.1219
0.0737
0.0197
Middle layer (max) Final layer
0.1620 (layer 21)
0.1316 (layer 14)
0.0950 (layer 12)
0,1332
0.1182
0.0602
sinks are removed, demonstrating the reference point's critical role in maintaining geometric stability, Fisher information metries table [ show extreme early-layer concentration, with approximately 60% of total Fisher information concentrated in the first layers, quantifying precisely how these architectures establish their coordinate system through a dominant early reference point. The full results for LLaMa E, Mistral v0.1 and Gemma G and Pythia are in the appendix.
Table 2: Spectral graph signatures reference frame types
Property
Centralized (LLaMA-3.2-3B)
Distributed (Qwen2.5-7B)
Bidirectional (RoBERTa)
Position encoding
Standard RoPE 0.04 (1/28 layers)
NTK-aware RoPE 0.25 (7/28 layers) 16.2 12.8/ 11.0 18.8 (1)
Absolute 0.38 (9/24 layers) 38.3 /1.2/11///2 22.5 55.3
Algebraic C Connectivity (Fiedler Value)
High threshold effectiveness Early / Middle / Late layers
12.7/11777.00
Maximum value (layer)
17.7
likeness Measure
Low thrsh. (0.001): E//L High thrsh. (0.1): E/UL
0.54 /..53 0.54 597/0.96/0.9
0.54 0.53 / 0.50 0.78/0.93 0.88 Yes
0.35 0.36 0.39 0.85 / 0.94 Yes
Middle-laye layer peak (high this.)
No
Degree Contralization and Variance
Centralization: F/M/L.
0.54/0.55/066
0.52/0.54 0.50 130.3 117.4/ 104.5
/0.1/
Variance: E/M/L
114.9/11../88
98.3 90.3/50.8/ 89.7
Signature Correlation Patterns
Fiedler vs. Centraliz. (0.001)
Strong negative (-0.95)
Negative (-0.46)
Positive (0.32)
Fiedler vs. Centraliz, (0.1)
Positive (0.34)
Strong positive (0.61)
Negative (-0.33)
Correlation sign Hip
Yes (.0.95 + +0.34)
Yes (-1.46 +0.61)
Yes (0.32 - -0.33)
5.2 Distributed Reference Frames
Distributed reference frames emerge in architectures with modified positional encoding schemes, such Qwen 2.5 and Phi-2s NTK-aware scaled RoPE. Our spectral analysis identifies distinctive sign-flipping correlation pattern between Fiedler values and centralization metrics, negative at low thresholds transitioning to positive higher thresholds, shown in table 2 a reliable signature of distributed reference frames. KL divergence measurements in table reveal characteristic three- phase pattern: positive KL reduction early layers, stronger negative reduction in middle layers, and moderated negative values in late layers. This signature indicates that reference points serve different functions at different network depths. Fisher information with multiple significant peaks across different network depths, reflecting fundamentally different approach to establishing coordinate systems. The complete results for Qwen 2.5 and Phi-2 are in the appendix.
& lower peak concentration

---

Table 3: Architectural Differences in Attention Sink Properties
Property
LLaMA-3.2-31
Qwen2.5-7B
XLM-RoBERTa
Reference frame type
Centralized
Distrihuted
Bidirectional
Attention Sink
Avg. KL. Reduction
-0.0974
-0,0088
-0.1017
Properties (t=0.8)
Avg. Sink Concentration
82.93%
69.92%
66.86%
Max Sink Concentration
96.40% (L.25)
85.75% (1.23)
85.22% (L21)
Layer-wise
Early Layer Pattern
Strong negative
Positive
Mixed KL
Distribution
Middle Layer Pattern
Moderale negative KL
Mixed KL
Strong negative Variable KL
Deep Layer Pattern
Strong negative KI.
Mixed KL
Architectural
Sink Formation
Consisteut across layers
Variable by layer
Strong middle-layer focus
Implications
Threshold Sensitivity
High
Moderate
Moderate
Early-layer Context
Strong sink focus
Weak sink formation
Very weak sink formation
5.3 Bidirectional Reference Frames
Bidirectional reference frames emerge in encoder architectures with absolute position embeddings, such BERT and XLM-RoBERTa. Our topological analysis reveals remarkably high initial com- plexity with high Betti, values indicating numerous loops/cycles in early layers, contrasting sharply with decoder models, shown in table The spectral analysis in table shows peak connectivity in early but not initial layers, with dramatic drops in connectivity from early to middle layers suggesting rapid geometric reorganization. The most distinctive signature appears in our KL divergence analysis, in table which reveals characteristic U-shaped profile with positive KL reduction in both first and final layers, confirming the dual-anchor nature of the reference structure. Fisher information in table 8 peaks in middle layers rather than at the beginning, indicating a fundamentally different approach information distribution. The comprehensive results for BERT and XML-RoBERTa H are in the appendix.
5.4 Value space analysis
Our vector geometry analyses reveal how reference frames function as coordinate systems within transformer representation manifolds. As shown in Table 4 each reference frame type implements a distinct geometric strategy for balaneing representational stability with flexibility. Centralized frames
Table 4: Value Space Characteristics Across Reference Frame Types
Property
LLaMA-3.2-3B
Qwen2.5-7B
XLM-RoBERIa
Reference frame type
Centralized
Distribated
Bidirectional
Directional
First Layer
0.9672
0.7508
0.7310
Influence
Middle Layer
0.5058
0,5000
0,9808
Last Layer
0.5012
0.5000
0,9274
Evolution Pattern
Sharp decrease
Early decrease,
Increase, stable
Geometric Semantic
First Layer
-0,0383
0.0569
0.2065
Alignment
Middle Layer
-0,2999
-0.0771
-0.3545
Last Layer
.0,2614
0.0000
-0.3982
Evolution Pattern
Consistently negative
Near zero throughour
Positive to strongly negative
Information Cuntent Change
410.97
7161.51
Std Dev
126.99
3649.72
4.40
Moderate
Very high
Reference Token Structure
Mean Count
1,00
1.23
1.60
Maximum Count
2.00
8:00
10.00
Distribution
Single token dominant
Multiple tokens
Varying tokens by layer
Attention-Valle
Entropy-Magnirdd Magnitude Corr. Farly-to-1 ate Layer Shift
0.23
-0.23
0.23
Relationships
Transformation Magnitude
2.46 to 84.07
15.03 to 222.39
19.74 to 23.81
establish a strong initial coordinate origin that gradually accommodates more nuanced transformations while maintaining a single reference point. The negative geometric-semante alignment confirms these frames prioritize geometric organization over semantic relationships. Distributed frames employ multiple reference points with substantially higher information content change, resembling differential geometry's 's use of local coordinate charts for complex manifolds. This approach trades computational efficiency for greater adaptability to semantic structure. Bidirectional frames exhibit a remarkable phase transition in geometric-semanne alignment, implementing dual-phase computation where

---

early layers leverage semantic relationships before deeper layers reorganize representations according more abstract geometric principles.
These patterns confirm that reference frames represent optimized solutions to the challenge of establishing stable coordinate systems high-dimensional spaces, with architectural choices directly influencing which strategy emerges.
Our Random Matrix Theory analysis further supports this interpretation by demonstrating that reference frames emerge during the earliest training steps, well before task performance begins to converge, indicating their fundamental role in representation organization.
5.5 Temporal Emergence of Reference Frames
Our Random Matrix Theory analysis in table shows how reference frames develop during the earliest stages of training across different model scales. The spectral gap metric shows a non- monotonic relationship with model size, smaller models exhibit minimal changes while mid-sized models (particularly 6.9B) demonstrate the most pronounced increases. This suggests reference frames establish themselves most efficiently at certain parameter scales rather than scaling linearly with model size. Participation ratio measurements show that while smaller models develop more distributed attention during training, larger models progressively concentrate attention fewer dimensions. The dramatic increase in participation ratio change between 6.9B and 12B models suggests a phase transition in how the largest models organize their representational geometry.
Table 5: Evolution of Random Matrix Theory Properties During Pythia Model Training
Metric
Pythia-1.4B
Pythia-2.8B
Pythia-6.9B
Pythia-12B
Average Changes from Step 0 Step
Spectral Gap
-0.0002
0.0014
0.0031
0.0007
Participation Ratio
0.0073
-0.0111
-0.0118
-0.0727
Attention Entropy
0,0005
0.0003
0.0011
0.0018
Sink Concentration
0.0001
0.0002
0.0001
0.0013
Layer-wise Distribution of Changes
Largest Spectral Gup Increuse
0.0163 (L18)
0.0139 (L28)
0.0288 (L330
0.0322 (L,23
Largest Spectral Gap Decreuse
-0.0177 (L17)
-0.0138 (L31)
-0.0250 (L23)
-0.0227 (L335
Lurgest Part. Ratio Increase
0.1530 (L13)
0.1792 (L19)
0,4147 (L23)
0.3061 (L25)
Largest Part. Ratio Decrease
-0.1353 (L.7))
-0,2898 (L23)
-0.4957 (L25)
-0.8785 (L34)
Layer specialization becomes increasingly pronounced as models scale up, with early layers remaining relatively stable, middle layers showing divergent patterns, and deep layers demonstrating dramatic evolution attention structure. The dual trends in attention entropy and sink concentration metrics reveal that while attention generally becomes more uniformly distributed across tokens, larger models simultaneously develop more pronounced attention sinks. The emergence of increasingly structured attention patterns in larger models suggests that scale enables more sophisticated geometric representations, mathematical necessity rather than an architectural accident.
6 Limitations and Conclusion
Limitations Our topological and spectral analyses focus attention patterns specific network snapshots rather than continuously tracking their evolution throughout training.
Conclusion Our work reframes attention sinks from architectural quirks to fundamental aspects of transformer geometry. The reference frame perspective offers three key contributions: (1) unifying
diverse attention patterns across architectures through a single geometric principle; (2) providing
insights into how architectural choices influence attention organization; and (3) establishing foundation for deliberate reference frame engineering. By demonstrating that attention patterns reflect optimal solutions to the challenge of establishing stable coordinate systems in high-dimensional spaces, our framework opens new directions for transformer optimization. Future works will explore using attention sink tokens as anchoring points for transfer learning, potentially enabling more efficient knowledge transfer while preserving geometric stability across different model architectures.
10

---

7 Acknowledgments
Research supported in part by European Union Next Generation EU namely by the MUR-PRIN 2022 project "2022REWNTE- Artificial Intelligence algorithms to track and detect Covid-19 vaccine- related infodemic on social media" CUP no. B53D23020690066; in part by the projects FAIR under Grant PE0000013 and SERICS under Grant PE00000014 14 under the MUR National Recovery and Resilience Plan funded by the European Sniion-NxtGeneraatioEEU, and in part by the project Neural Reasoning over Open Data (NEREO) funded by the Italian Ministry of Education and Research (PRIN) under Grant 2022AEFHA, and in part by the project SEED funded by Sapienza University of Rome.
References
Federico Barbero, Alvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Razvan Pascanu, et al. Why do llms attend to the first token? arXiv preprint arXiv:2504.00733. 2025.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv preprint arXiv:2402.09221, 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, In The Twelfth International Conference on Learning Representations, 2024. URL
Herbert Edelsbrunner, John Harer, et al. Persistent homology-a survey, Contemporary mathematics, 453(26):257-282, 2008.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. arXiv preprint arXiv:2410.10788 10781, 2024.
Amirhossein Kazemmejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 36:24892-24928 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (9296-26306, 2024.
Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupata, and Afra Alishahi. Quantifying context mixing in transformers. arXiv preprint arXiv:2301.12977, 2023.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pages 7045-7054. PMLR, 2020.
Luca Moschella. Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209. (tt.::22299.4343000 15430, 2022.
Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. Journal of Machine Learning Research, 21(184):1 2020.
Ofrr Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 /2409, 2021.
Valeria Ruscio and Fabrizio Silvestri. Beyond position: the emergence of wavelet-like properties in transformers. arXiv preprint
2024.

---

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568: 127063, 2024.
Chuanhao Sun, Zhihang Yuan, Kai Xu, Luo Mai, N Siddharth, Shuo Chen, and Mahesh K Marina. Learning high-frequency functions made easy with sinusoidal positional encoding. arXiv preprini arXiv:2407.09377, 2024.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physsics/0004057, 2000.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint
2019.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. / 2023.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 156.6555.... 2024.
Stephen Zhang, Mustafa Khan, and Vardan Papyan. Attention sinks and outlier features: A' N'catch, tag, and release' mechanism for embeddings. arXiv preprint wwXiv22502.00919 2025.
Zayd MK Zuhri, Erland Hilman Fuadi, and Alham Fikri Aji. Softpick: No attention sink, no massive activations with rectified softmax. arXiv preprint arXiv:2504.20996 2025.
A Geometric Effects of the Softmax Operation
The softmax operation transforms raw attention logits into probability distributions through the equa- tion Gy exp(z,)/ Σ, exp(z). This transformation has two fundamental geometric consequences that directly drive reference frame formation.
First, softmax transforms unbounded attention logits into a bounded manifold with intrinsic curvature. While input logits 2j can range from -oo to +00 in Euclidean space R., the output attention weights aj are confined to the probability simplex A=-1, This mapping from Euclidean space to the simplex introduces non-Euclidean geometry with positive curvature under the Rish metric. The curvature of this manifold means that geodesics (shortest paths) differ from straight lines, creating geometric pressure toward the vertices and edges of the simplex. Mathematically. this corresponds to sparse attention distributions where most probability mass concentrates The dimensional reduction from n-dimensional logit space to an (n - d)-dimensional simplex creates an information bottleneck that forces the network to prioritize certain geometric relationships over others during training.
small subset of tokens.
Second, softmax enforces conservation law that makes attention zero-sum resource through the constraint Σ, a = 1. This constraint means each token has exactly one unit total attention to distribute across all tokens in the sequence. Any increase in attention weight one token must be precisely balanced by decreases to others, creating economy of attention where tokens compete for finite resources. This competition creates evolutionary pressure during training toward allocation patterns that maximize the computational utility of each attention unit. 'he simplex constraint drives the model toward solutions that balance the trade-off between distributing attention for content processing and concentrating attention for geometric reference. The mathematical optimality of allocating substantial attention to a small set of reference tokens emerges naturally from this constrained optimization problem.
Together, these geometric effects create the necessary conditions for reference frames to emerge as optimal solutions to the challenge of establishing stable coordinate systems while respecting the mathematical constraints imposed by the softmax operation. The different reference frame types we identify represent alternative solutions to this same fundamental geometric problem, each balancing the trade-offs between computational efficiency and representational flexibility in mathematically distinct ways.
12

---

B Statistical Analysis
Our statistical methodology employed multiple complementary approaches to validate the existence of distinct reference frame types. We performed pairwise t-tests between transformer layers to identify significant changes in topological metrics (Betti numbers and persistence values), using 0.05 our significance threshold. For spectral analysis, computed Pearson and Spearman correlations between Fiedler values and graph properties (star-likeness, centralization, variance, density) across seven attention thresholds (0.001 0.2), allowing us to detect threshold-ddppendett correlation patterns. To quantify information-theoredt between original and sink-removed attention distributions, testing the statistical significance of KL reduction values and their correlation with sink concentration metrics. Fisher information analysis involved computing Frobenius norms of Fisher matrices across layers and correlating these with layer depth using both Pearson and Spearman methods to capture linear and monotonic relationships respectively. All correlation analyses included p-value calculations to assess statistical significance, with < 0.05 considered significant. When comparing multiple metrics across numerous layers, we report the proportion of significant comparisons to evaluate overall pattern reliability.
differences, we calculated KI. divergence
Table 6: Core Statistical Metrics Across Reference Frame Types
Property
L.aMA-3,2-3B
Qwen2.5-7B
Xo.-RoBEKTa
Reference frame type
Contralized
Distributed
Bidirectional
Topological
Bettig constancy
26.41(p 1.00;
16.1 1.000)
22.60
1.00pp 0.0001]
Stahility
Dim0 Persistence range
0.032 0..162(p 0.0001)
0.067
00,418pp < 0.0001)
0.003
0.155 < 0.0001)
Bettiy evolution
0 (constant)
(constant)
20.93 - 00p < 0.0001)
Spectral
Star-Ikknnes (t=0.001)
y= -0.32, 0.10
r= -0,50, p < 0.001
r= -0.76, p = 0.008
Correlations
Star-likeness (t=0.02)
0.HH. 0.016
0.51, 0.001
0.57, 0.004
(Fiedler)
Centralization (1=0.001)
-0.48, < 0.001
p 0.008
0.31.
Centralization (I=0.02)
-0.57, x 0.001
-0.63,3 0.031
-0.68.8. 0.038
Information Metrics
KL reduuctoo range
-027746o 0.013
-0.219to 0.110
-0,284to + 0.088
Kocconsenttattinn correlation
-0,78, 0.001 96.00%
-0.76, p 0.0015 86.40%
-0.70, 0.008 85.52%
Peak sink concentration
Learning Dynamics
Fisher-depth correlation
0.39. 0.042
0.799. p 0.001
0.38. p 0.071
Fisher depih Spearman
0..97. 0.001 12.7%
0.82, < 0.001
0.38, p 0.066
Atlention parameters (5)
35.4%
72.8%
B.1 Reference frames statistical analysis
In table 6 is our comprehensive statistical analysis provides quantitative validation for the three distinct reference frame types identified in transformer architectures. Centralized reference frames (LLaMA-3.2-3BB exhibit remarkable topological invariance with Bettio remaining constant at 26.41 across all layers (p= = 1.000) and zero Betti, values throughout, while Dim0 Persistence evolves significantly from 0.057 in early layers to 0.162 in the final layer (all p < 0.0001), accompanied by spectral correlation reversals where Fiedler-star correlations shift from negative (r = -0.32) at low thresholds to positive (r 0.50, p 0.001) at higher thresholds, consistently negative KL reduction values across layers (ranging from -0.03 to -0.274) with strong negative correlation between KL reduction and sink concentration (r -0.78, p < 0.001), and significant negative Fisher information correlation with layer depth (r= -0.39, p = 0.042) showing extremely strong monotonic decrease (Spearman p -0.97, p < 0.001).
Distributed reference frames (Qwen2.5-7B) demonstrate a distinctive U-shaped DimO Persistence pattern dropping from 0.418 to 0.067 before recovering to 0.261 (all transitions p 0.0001), systematic spectral correlation reversals across thresholds where star-likeness correlations change from negative (r -0.59, p 0.001) to positive (r = 0.51, p 0.05) while centralization correlations strengthen negatively, a three-phase KL reduction pattern with positive values in early layers (+0.110) negative in middle layers (-0.219), and mixed in deep layers, alongside the strongest Fisher information depth correlation (r= -0.79, p < 0.001) indicating efficient learning distribution across multiple reference points.
Bidirectional reference frames -XLM-ROBERTE uniquely show dramatic topological evolution with Bettip decreasing from 22.69 to 1.00 and Betti, from 20.93 to near-zero (all < 0.0001), layer-specific spectral behaviors varying from strong negative star-likeness correlations early layers (r = -0.92, p < 0.001) to positive correlations in deeper layers, characteristic U-shaped KL profile with positive
13

---

reduction at extremes (+0.071 and +0.088) and negative values in middle layers (minimum -0.28), combined with no significant Fisher information correlation with depth (r= p= p= 0.071) reflecting maintained learning capacity throughout the network, where attention parameters comprise 72.8% of total parameters compared to 12.7% centralized and 35.4% distributed frames, demonstrating how each reference frame type implements mathematically distinct solution to coordinate system establishment in high-dimensional transformer representations.
B.2 Value space statistical analysis
Table 7: Value space statistical analisis. p < 0.05, **p < 0.01, *** < 0.001).
The asterisks indicate statistical significance levels (*
Property
LLaMA-3-2-3B
Qwen2.5-7B
XLM-RoBERTa
Reference frame type
Contralized
Distributed
Bidirectional
Reference Point
Relative Magnitude (mean)
0,5413
0.4893
0.5918
Metries
Directional Influence (mean)
0.5419
0.5523
0.9397
Information Content Change
41118
7084.96
13.74
Reference Token Count
1.00
1,23
1.58
Max Reference Count
2
7
I Layer Evolution
Relative Magnitude Pattern
Mixed (r= -0.37)
Mixed (r= 0.29)
Mixed (r = 0.33)
(Correlations)
Directional Influence Pattern
Mixed -0.37)
Decreasing (r -0.60)
Mixed -0.46)
Information Content Pattern
Mixed 0.46)
Increasing 0.53)
Mixed -0.03)
Cross-Layer
Directional Influence Change
-0.082
-0.105*
-0.028
Differences
(Early vs Late)
(p = 0.091)
(p = 0.045)
(P 0,290)
Information Content Change
+119.66*
+2452.50
0.024
(Early vs Late)
0.014)
0.061)
0.990)
Key Reference
MagnitudeIInlleeme
0.9997*44
-0.1264**
0.0018
Correlations
Correlation
(p < 0.001)
(p < 0.001)
0.843)
Magnitude-Inie Info Content Correlation
-0.9147***
0.2285***
0.6000
(p < 0.001)
( 0.001)
0,001)
Influence-Info Content Correlation
+**9153***
-0.7708***
0.6268
(p < 0.001)
(p < 0.001)
(p < 0.001)
Attention-Vall Value Relationships
Attention Entropy (meun)
1.0522
1.4513
2.1871
Value Transform Magnitude
9.7099
16.1056
8.1544
Geometric-Semmee Semannii Aligmment
0.2923*44
0000455**
0.0019
(p < 0.001)
(p < 0.001)
(p (p=0..5 0.561)
Marrop--Magn Correlation
0.2279
-0
0.2346***
(p S. 0.001)
< 0.001)
(p < 0.001)
Layer-wise
Attention Entropy Change
-0.4068***
10860
-1.0198a*
Evolution
(First Last)
(p < 0.001)
< 0.001)
< 0.001)
Transform Magnitude Change
+81.6071.0*
206 206.3281***
+1.0779a99
(First Last)
(p < 0.001)
(p < 0.001)
(p 0.001)
Geometric Alignment Change
-0.2224***
-0.0575***
-0.6038
(First to Last)
(p < 0.001)
(p < 0.001)
(p < 0.001)
Statistical
Sample Size (texts)
500
500
500
Properties
Total Data Points
14. 000
14. 000
12, 0.000
Significant Correlations
37/40(92.5%)
36/40(90.0%)
33/ 03/40(82.5%)
Our statistical analysis of value space provides quantitative validation for the three distinct reference frame types, with measurements across 500 text samples generating 12, 000 - 14, 000 data points per architecture. Centralized reference frames (LLaMA-3.2-33B demonstrate the most coherent reference structure with near-perfect correlation between relative magnitude and directional influence (r 0.9997, p 0.001), maintaining single dominant reference token (mean count 1.0006) whose directional influence dramatically decreases from 0.97 the first layer to 0.50 in the final layer, while information content change significantly increases from early (351.35) late layers (471.01) with a mean difference of 119.66 (t = -2.76, = 0.014), accompanied by consistently negative geometric-semenn semanicc aligmment (mean -0.29, p < 0.001) and positive entropy-magninn correlation (r = 0.23, < 0.001) indicating that diverse attention patterns produce larger value transformations, Distributed reference frames (Qwen2.5-7B) employ multiple reference points (mean count = 1.23, max 7) with substantially higher information content change (mean 7084.96, SD 3641.75), showing distinctive negative correlation between relative magnitude and directional influence (r = -0.13, p < 0.001) that contrasts sharply with centralized frames, while directional influence significantly decreases from early (0.60) to late layers (0.50) with p = 0.045, accompanied by near-zero ecometric-semannt alignment (mean -0.04, p 0.001) and negative entropy-
14

---

magnitude correlation (r= -0.23. < 0.001) suggesting that focused attention patterns drive larger transformations in this architecture. Bidirectional reference frames XXLM-RoBERTa) maintain the highest overall directional influence (mean 0.94, S1>= 0.07) with the most complex reference structure (mean count = 1.58, max 9), uniquely showing no correlation between relative magnitude and directional influence (r = -0.002, = 0.84) while both metrics correlate positively with information content change (r = 0.60 and r 0.63, both p < 0.001), exhibiting the only non- significant gcometric-semantic alignment (mean 0.002, 0.56) and positive antropy-magnitue correlation (r = 0.23, p < 0.001). Cross-layer evolution reveals dramatic and statistically significant changes across all architectures: attention entropy decreases substantially (centralized: -0.41, distributed: 22111, Sddirectioaal--100 all < 0.001), value transformation magnitude increases exponentially (centralized: 81.61-fold increase from 2.46 84.07, distributed: 206.33-fold from 15.03 to 222.39, bidirectional: 4.08-fold from 19.74 to 23.81, all < 0.001), while geometric- semantic alignment becomes increasingly negative (centralized: -0.22 decrease, distributed: -0.06 decrease, bidirectional: -0.60 decrease, all < 0.001), with layer-specific correlation patterns showing remarkable consistency within each architecture type, centralized frames exhibit correlation reversals from negative (r = -0.53) to positive (r = -0.57) entropy-magnitude relationships, distributed frames show systematic progression from strongly negative (r -0.70) to near-zero (r = -0.05) correlations, and bidirectional frames demonstrate the most dramatic shifts with correlations ranging from r = -0.16 r 0.64 across layers. These quantitative patterns provide robust statistical evidence that reference frames not only organize attention patterns but fundamentally shape value space transformations, with each architecture implementing mathematically distinct strategies that manifest in measurable differences in how information flows through the network, how strongly reference points influence transformations, and how semantic and geometric properties interact throughout the model depth.
C Fisher Table
Table 8: Comparative Analysis of Fisher Information Distribution Across LLaMa, Qwen and RoBERTa
Property
LLaMA-3.2-3B
Qwen2.5-7B
XLM-RoBERTa
Reference frame type
Centralized
Distributed
Bidirectional
Component Importance
Attention Mechanism
12.7% (78,164)
29.7% (46,040)
71.7% (8.363.327)
MLP Components Embedding
85.5% (527,103)
51.8% (80,380)
27.8% (3,244,160)
1.8% (10,892)
2.1% (3,247)
0.2% (26,120)
Layer Distribution
Peak Layer
Layer I: 371,465
Layer 18,791
Layer 9: 2,504,416
Secondary Peak
Layer 39,429
Layer 26-27: 6,039
Layer & 2,371,86 ,845
Pcak to Minimum Ratio
143:1
26:1
303:1
Structural Features
Total Fisher Norm
616,694
155,080
11,668,807
Early Layers (0-9) Concentration
82.7%
64.7%
59.3%
Middle Layers Concentration
13.5%
14.9%
39.8%
Final Layers Concentration
3.8%
3.9%
0.9%
D Attention Sink Data
The analysis of attention sink patterns across 500 Wikipedia samples provides strong evidence for the theoretical predictions regarding positional embedding geometry. Ilama 3.2 3B, employing standard rotary position embeddings (RoPE), shows the predicted pointed manifold structure with the beginning-of texx token serving as the dominant attention sink in 100% of analyzed samples, achieving an average sink score of 0.9968. This universal reliance on single reference point directly reflects the geometric asymmetry introduced by RoPE, where the first token experiences no rotation (fg = O, R, = I), giving a privileged computational status that the model exploits during training. In stark contrast, Qwen 2.5 3B, which implements NTK-aware scaled RoPE with reduced angular separation (a < 1), has a fundamentally different attention topology consistent with universal origin, Qwen distributes attention sinks across diverse tokens- with the most common sink (The) appearing in only 17.2 7.2% of samples and the top 10 sinks cumulatively accounting for 53.4% of cases. This distributed
a multi-pointer manifold (M, {P1, P2, ......... ...,,,,)) )). Rather than converging
15

---

Table 9: Attention sink distribution summary across 500 Wikipedia samples. Analysis performed
with 7 0.3 (sink threshold) and set the 90th percentile of attention weights.
Model
Samples
Most Common
Occurrence
% Coverage
Avg Sink
Top Sink
Score
Llama 3.2 3B
500
< <lbegin_ _fettt____tet
500
100.0% 17.2%
0.9968
Qwen 2.5 3B
500
The
86
0.8892
Table 10: Distribution of top 10 attention sink tokens for Llama 3.2 3B (standard RoPH) and Qwen 2.5 3B (NTK-aware scaled RoPE). Llama exhibits extreme centralization with 100% coverage by single token, while Qwen shows distributed sink selection across diverse tokens.
Llama 3.2 3B (Standard RoPE)
Qwen 2.5 3B (NTK-aware RoPE)
Rank
Token
Freq.
Rank
Token
Freq.
%
500
100.0
The
17.2
medicine
10.2
3
matter
3
0.6
3
In
40
8.0
4
include
3
0.6
4
This
19
3.8
5
MRI
2
0.4
For
17
3.4
pressure
0.4
3.0
7
logy
2
0.4
7
These
10
2.0
8
Nunes
2
0.4
8
of
10
2.0
9
GIS
2
0.4
A
10
2.0
10
conversion
0.2
One
1.8
Cumulative (Top 10):
101.2%
Cumulative (Top 10):
53.4%
pattern, where different contexts select different reference tokens, confirms that NTK-awar scaling successfully mitigates the positional bias inherent standard RoPE, enabling the formation of multiple local coordinate systems rather than a single centralized reference frame.
E
Llama Family
As we can see from table [T] all Llama variants, regardless of parameter count or fine- tne-tuning, exhibit 100% specialization on the beginning-of-sequeee reference frame in this kind of architecture. The stable Bettin numbers (connected components) across all layers in all models confirm the predicted behavior of centralized reference frames, they maintain consistent topological structure rather than merging components as seen in distributed frames. All models show monotonically increasing persistence values from early late layers, and this indicates that the centralized reference point becomes more significant in deeper layers, suggesting increasing reliance on this coordinate system as information flows through the network.
token. This is strong evidence fot the centralized
The remarkable consistency across different parameter scales (IB, 3B, 8B) suggests that reference frame formation is fundamental architectural property rather than emergent behavior dependent on model size. Despite some difference, the instruction-tunet model still maintains the fundamental centralized reference frame signature, perfect BOS specialization, stable topology, and monotonically increasing persistence.
that all Llama models exhibit remarkably similar spectral properties despite their varying parameter counts (IB to 8B) and fine-tuning status. This consistency supports the thesis that reference frame type is fundamentally determined by architectural choices rather than scale or training objective. The consistent peak in algebraic connectivity in middle layers (1.01x- 1.61x higher) aligns with our theory that centralized reference frames optimize information flow through the network via a coordination layer structure. The strongest peak the smallest model (1.61x IB vs. 1.01x in 8B) suggests that smaller models rely more heavily on this coordination mechanism.
In table 12 we
The strong negative correlation between algebraic connectivity and degree centralization (-0.9790 to -0.9865) distinctive mathematical signature of centralized reference frames. This indicates that as
16

---

Table 11: Reference frame signatures across Llama model variants
Property
Llama-3.2-IB
Llama-3.2-3B
Llama-3.1-8B
Llama-32-3B-Instrrue
BOS specialization
100%
100%
100%
100%
Bettig (early)
26.39
26.43
26.55
26.37
Bettip (late)
26.39
26.43
26.55
26.37
Bettig change
0.00
0.00
0.00
0.00
Dim0 persistence (early)
0.0748
0.0573
0.0591
0.0410
Dim0 persistence (late)
0.1651
0.1620
0.1612
0.1350
Persistence increase
+0.0903
+0.1046
+0.1021
+0.0940
Max attention StdDev
0.1400
0.1620
0.1635
0.1622
connectivity increases, attention becomes more evenly distributed, but critically, this happens only specific thresholds, maintaining the overall centralized structure.
Table 12: Spectral Signatures in Llama models
Property
Llama-3.2-1B
Llama-3.2-3B
Llama-3.2-3B-I
Llama-3.1-8B
Position encoding
RoPE
RoPE
RoPE
RoPE
Threshold effectivvnness (0.1)
Algebraic connectivity (early)
13.3843
12.4644
12.9546
12.6150
Algebruic connectivity (middle)
13.9363
11.6529
12.7520
10.7345
Algebraic connectivity (late)
8,5773
6,9879
8,3986
7,8740
Maximum connectivity
18.4033 (layer O)
17.6574 (layer 0)
18.2137 (layer 0)
18.0901 (layer 0)
Star-likeness (early)
0.5348
0.5352
0.5368
0.5345
Star-likeness (middle)
0.5351
0.5340
0.5358
0.5327
Stur-likeness (late)
0.5357
0.5371
0.5394
0.5343
Maximum star-likeness
0.5377 (layer 14) 0.5517 (layer 22)
0.5526 (layer 22)
0.5455 (layer 25)
Degree centralization (early)
0.5383
0.5435
0.5418
0.5436
Degree centralization (middle)
0,5315
0.5502
0.5424
0,5582
Degree centralization (late)
0.5796
0.6126
0,5909
0,5979
Maximum centralization
0.6169 (layer 13)
0.6988 (layer 22)
0.6720 (layer 22)
0.6683 (layer 25)
Degree variance (early)
I 119.9924
114,9265
119.7555
115.2534
Degree variance (middle)
124.9291
111.6136
118.7679
106.3977
Degree variance (late)
100.6093
85.1385
97.0134
89.3467
All models show high threshold sensitivity, with meaningful graph structures only emerging lower thresholds (S 0.05). This sensitivity itself a signature of centralized frames, where attention is concentrated specific tokens, creating sparse attention distributions that become disconnected at higher thresholds. All models exhibit sign-flip in correlation patterns at approximately the same threshold (== 0.2), transitioning from negative to positive correlation between Fiedler values and centralization metrics. This threshold uniformity across model scales suggests it represents fundamental property of centralized reference frame organization rather than scale-dependent phenomenon.
In table 14 we can see that unlike the three-phase pattern observed in Qwen's 's distributed reference frames, Llama models exhibit a consistent pattern of negative KL reduction values across nearly all layers, with only layer 0 showing small positive values. This consistent negative KL reduction pattern (averaging between -0.07 and -0.13 across different network regions) indicates that removing attention sinks in Llama models consistently reduces the uniformity of attention distributions. This provides strong evidence for the centralized reference frame hypothesis, where a single dominant reference point serves universal coordinate system for all token representations.
critical finding is the extremely high sink concentration values across all network depths in Llama models (77-89%), substantially higher than the early-layer concentrations in Qwen models (39- 48%). This indicates that I.lama establishes strong reference points immediately and maintains them throughout the network.
17

---

Table 13: Key Fiedler value correlations across LLaMA models at different thresholds
Threshold
Property
Mean Correlation
Pattern
0.001
Centralization vs. Fiedler
-0.6361
Strong negative
0.01
Centralization vs. Fiedler
-0.8795
Very strong negative
0.02
Centralization vs. Fiedler
-0.8621
Very strong negative
0.1
Centralization vs. Fiedler
0.4720
Sign flip to positive
0.001
Density vs. Fiedler
0,4808
Moderate positive
0.01
Density vs. Fiedler
0,8221
Very strong positive
0.1
Density vs. Fiedler
0,5293
Moderate positive
Several key patterns provide additional evidence for centralized reference frames: for example, the layer with maximum sink concentration is consistently deep in the network (layer 25 for three models), reaching extraordinary values of 96+ %. This suggests the reference point becomes increasingly important for coordinate stabilization as representations grow more complex through network depth. The most negative KL reduction (strongest effect from removing sinks) occurs either in very early layers (LI) or very deep layers (L25), revealing a bimodal pattern where reference points are established early and then heavily leveraged in later processing. The base and instruction-tunee 3B models show remarkable similarity in their reference frame signatures, suggesting the centralized reference structure is fundamental to the architecture rather than task-dependent.
The consistency of these patterns across model scales (IB to 8B) demonstrates that centralized reference frames represent a stable architectural solution to the geometric organization problem.
Table 14: Attention Sink Analysis Across Llama Model Variants
Property
Llama-3.1-8B
Llama-3.2-1B
Llama-3.2-3B
Llama-32-3B--nstruuc
KL Reduction (0-3 layers)
-0.1093
-0.0959
-0.1271
-0.1213
KL Reduction (5-15 layers)
-0.0726
-0.0741
-0,0692
-0.0803
KL Reduction (17+ layens)
-0.1275
N/A
-0.1208
-0.1179
Max KL reduction
0.0118 (LO)
0.0221 (L.)
0.0072 (L.)
0.0370 (L.)
Min KL reduction
-0.2245 (L25)
-0,2040 (L.)
-0,2175 (L1)
-0,2225 (L1)
Sink concentration (0-3 layers)
77.09%
75.91%
83.29%
83.77%
Sink concentration (5-15 layers)
83.36%
75.98%
79.94%
82.58%
Sink concentration (17+ layers)
89.22%
N/A
87.51%
88.76%
Max sink concentration
96.60% (L25)
95.33% (L13)
96.40% (L25)
96% (L225
Layer pattern type
Consistent
Consistent
Consistent
Consistent
Optimal threshold
0.8
0.8
0.8
0.8
Analyzed layers
9
15
15
F
Qwen Family
As we can see in table 17 all Qwen models exhibit the distributed reference pattern, with moderate specialization (36.6%-65.4%) on common linguistic elements rather than beginning-of-sequeen token. This contrasts sharply with the centralized reference frames in Ilama models where BOS specialization is consistently 100%. A fascinating pattern emerges in Bettig changes: the smaller 3B model maintains stable component counts (26.18 26.18). While both 7B models show significant component integration, with similar reductions (-8.19 and -9.27) This suggests that larger models develop more flexible reference structures where components merge in deeper layers, potentially enabling more complex reasoning by integrating information across multiple reference points.
All Qwen models show decreasing persistence values through network depth and this confirms that distributed frames initially establish stronger reference points that gradually weaken as information flows through the network. The instruction-tune model shows slightly enhanced component inte- gration (-9.27) compared its base counterpart (8), suggesting that instruction tuning further optimizes the distributed reference mechanism for improved reasoning.
18

---

Table 15: Fisher information distribution across Llama models
Architectural Pattern
LLaMA-32--BB
LLaMA-3.2-3B
LLaMA-322-3--1
LLaMA-3.1-88
Component Importance (Percentage of Total Fisher Norm)
Attention Mechanism
11 11.0%
12.7%
23.3%
47.9%
MLP Components
86.9%
85.5%
70.1%
30.2%
Embeddings
2.1%
1.8%
6.4%
20.7%
Layer Distribution Patterns
Two Layers
66.7%
66.7%
65.8%
61.6%
Middle Layers (4-11)
18.7%
15.2%
11.5%
6,4%
Deep Layers (12+)
12.3%
18.1%
22.6%
10.7%
Layer Decay Characteristics
Layer 1 to Layer Ratio Initial Final Layer Ratio Decay Rate*
19,4:1
19.2:1
25.8:1
9.5:1
41.0:1
112.9:
65.1:1
20.5:1
Moderate
Steep
Very Steep
Moderate
Mpdel-Specific Patterns
Fisher Info per Parameter
Highest Front-loaded
Medium Pront-loaded Steady decline
Lowest Front-loaded
Medium Most balanced Final layer bump
Layer Distribution
Unique Feature
Layer 9-10 bump
Extreme decline
*Decay rate describes how quickly Fisher information diminishes acmoss layers
Table 16: Value space analysis across Llama model variants
Property
Llama-3.2-1B
Llama-3.2-3B
Llama-3.1-8B
Llama-3.1-88 3.1-8I-Intstruct
Value Space Metricy
Relative Magnitude (Mean)
0.6662
0.5415
0,5480
0.5384
Directional Influence (Mean)
0. 0.6694
0,5421
0.5495
0,5400
Directional Influence (Median)
0.5000
0.5000
Information Content (Mean)
249.6769
410.9697
349.2795
373.5666
Information Content (Median)
282.5689
440.9005
373.0212
396,0780
First-to-Last Layer Influence
-0,39
-0.45
-0.40
-0.40
Attention- Value Correlation
Atlention Entropy (Mean)
1.0517
1.0058
Value Transformution Magnitude
11,2118
9,7103
10,3658
10.1119
First Layer Magnitude
2.6703
2.4579
1,2628
1,2258
Last Layer Magnitude
106.1408
84.0693
133.1640
132.8909
Alignment
-0.3096
-0.2921
-0,3238
-0,3599
First Layer Alignment
-0.0318
-0.0383
-0,0839
-0.0875
Last Layer Aligmment
-0.3128
-0.2614
-0,3456
-0.3961
Entropy-Maan Magintud Correlation
-0.0366
0.2273
0.3050
0,3229
Fisst-to-Last Layer Shift
-0.5148 -0.5186
-0.5349 to -0.5665
-0.5002 to -0.5549
-0.4974 -0.5565
Table 17: Reference Frame Signatures Across Qwen Model Variants
Property
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-7B-Instruct
Token specialization
"Gthe" (36.6%)
"." (65.4%)
"," (64.4%)
Bettig (early)
26.18
26.16
26.15
Bettig (late)
26.18
17.97
16.88
Bettig change
0.00
-8.19
-9.27
Dim0 persistence (early)
0.4183
0.2381
0.2382
Dim0 persistence (late)
0.2629
0.1543
0.1572
Persistence change
-0.1555
-0.0838
-0.0810
Max attention StdDev
0.1274
0.1316
0.1296
In table T8 three Qwen models exhibit peak star-likeness in middle layers across multiple thresholds, with remarkably similar ratios (1.00 1.05x higher in middle layers). This consistency across
19

---

model scales (3B vs. 7B) and training objectives (base vs. instruct) suggests that the middle-layer organization is a fundamental characteristic of distributed reference frames.
Both 7B models show similar algebraic connectivity patterns, with peaks only at high thresholds (0.1-0.2). The 3B model also shows peaks these thresholds but with slightly higher magnitude (1.08 1.26x vs. approximately x.00x in the 7B models). This suggests that while distributed reference frames consistently organize connectivity patterns differently than centralized frames, there are subtle scaling effects on the strength of these patterns.
Table 18: Spectral Signature of Distributed Reference Frame Signatures in Qwen Models
Property
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-7B-Instruue
Position encoding
NTK-aware RoPE
NwK-aware RoPE
NWK-aware RoPE
Threshold effectiveness (0.1)
0.64
0.25
0.32
Algebraic connectivity (early)
18.1862
16.2441
16.2151
Algebraic connectivity (middle)
12.3066
12.7960
12.9784
Algebraic connectivity (late)
11,5426
11.0412
11.2764
Maximum connectivity
20.1 1863 (layer I)
18.7700 (layer 1)
18.4620 (layer 1)
Star-likeness (early)
0.5349
0.5361
0.5360
Star-likeness (middle)
0.5349
0.5336
0.5337
kar-likeness (late)
0,5398
0,5041
0,5057
Maximum star-likeness
0.5655 (layer 33)
0.5415 (layer 26)
0.5431 (layer 26)
Degree centralization (early)
0.5346
0.5237
0.5229
Degree centralization (middle)
0,5413
0.5372
0.5344
Degree centralization (late)
0,5528
0.4967
0,4932
Maximum centralization
0.6033 (layer 33)
0.5625 (layer 26)
0.5656 (layer 26)
Degree variance (early)
119.9341
133.2675
126.5460
Degree variance (middle)
114.8901
117.3620
115.2299
Degree variance (late)
115.2607
104.4904
102.9542
The sign-flipping correlation pattern between Fiedler values and centralization metrics emerges as the most reliable signature of distributed reference frames. All Qwen models show negative correlations at low thresholds transitioning to positive correlations at higher thresholds, with the sign-flip consistently occurring around the 0.05 threshold. The 3B model shows slightly higher threshold sensitivity than the 7B models, remaining effective up to 0.1 rather than 0.05. This suggests that larger models with distributed reference frames may develop more specialized attention patterns that become disconnected at lower thresholds.
All three Qwen models show peak degree centralization in middle layers, but with interesting variations: the 3B model shows peaks primarily at high thresholds (0.05-0.2), while both 7B models show peaks across multiple thresholds. This suggests that scale may influence how distributed reference frames organize their centralization patterns, with larger models developing more consistent middle-layer centralization.
The table 20 shows distinctive attention sink patterns across the Qwen model family, providing quan- titative evidence for distributed reference frames. All models demonstrate a three-phase pattern with positive KL reduction in early layers (+0.078 to +0.109), stronger negative reduction in middle layers (-0.026 -0.051), and moderated negative values in late layers (-0.008 to -0.018). 3). This indicates that reference points serve different functions at different network depths, initially establishing geometric stability, then actively shaping information geometry in middle layers, before stabilizing deeper layers.
Maximum sink concentration occurs in deep layers (23-25) rather than early layers, reaching 84-92% a key signature of distributed reference frames. The 3B model shows stronger contrasts between layer regions, particularly with more negative KL reduction in middle layers (-0.0514 -0.0266/- 0.0299), suggesting smaller models may rely more heavily on distributed reference structures for computational efficiency.
The layer with minimum KI. reduction (most negative impact when removing sinks) is consistently layer 5 across all models, while maximum concentration appears deeper in the network. This
20

---

Table 19: Key Fiedler Value Correlations across Qwen models at different thresholds
Threshold
Property
Mean Correlation
Pattern
0,001
Centralization vs. Fiedler
-0,4383
Moderate negative
0.02
Centralization vs. Fiedler
-0.6152
Strong negative
0.05
Centralization vs. Fiedler
-0.0482
Near zero (transition point)
0.1
Centralization vs. Fiedler
0.5565
Sign flip to positive
0.001
Density vs. Fiedler
-0.0996
Slight negative
0.02
Density vs. Fiedler
0.6835
Strong positive
0.1
Density vs. Fiedler
0.1147
Weak positive
separation between maximum effect and maximum concentration further supports the multi-pointed manifold structure hypothesized for distributed reference frames, where coordination is achieved through multiple specialized reference points rather than single dominant one.
Table 20: Attention Sink Analysis Across Qwen Model Variants
Property
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-77-Instruct
KI. Reduction (0-3 layers)
+0.1090
+0.0859
+0.0780
KI. Reduction (5-15 layers)
-0.0514
-0.0266
-0,0299
KL Reduction (17-35 layers)
-0.0084
-0.0177
-0,0134
Maximum KL reduction
0. 1329 (Layer
0. 136 (Layer I)
0. 1045 (Layer I)
Minimum KI. reduction
-0.3058 (Layer 5)
-0. 1.26 (Layer 5)
-0.1515 (Layer 5)
Sink concentration (0-3 layers)
39.59%
48.58%
48.13%
Sink concentration (5-15 layers)
78.03%
77.88%
76.93%
Sink concentration (17-35 layers)
77.60 77.60%
74.42%
72.95%
Maximum sink concentration
91.89% (Layer 25)
85.75% (Layer 23)
84.27% (Layer 23)
Optimal threshold value
0.8
0.8
0.8
Number of analyzed layers
19
15
Sequence length
128
128
128
Table 21: Fisher Information Distribution in Qwen Models
Architectural Pattern
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-7B-1
Component Importance (Percentage of Total Fisher Norm)
Attention Mechanism
10.4%
29.7%
30.2%
MLP Components Embedding
84.7%
51.8%
56.1%
4.9%
2.
1.7%
Layer Distribution Patterns
Key Processing Layers
Layer 2 (53.7%)
Layers (1- (54.9%) 64.7%
Layers 0-6 (56.6%)
Early Layers (0-9) Middle Layers (10-27)
78.7%
63.9%
11.0%
14.9%
17.1%
Final Layers
7.2% (28-35)
3.9% (26-27)
3.9% (26-27)
Layer Decay Characteristies
Peak Layer Value
Layer 2: 136,593
Layer O: 18,791
Layer 1: 18,359
Peak to Minimum Ratio
432:1
26:1
18.6:1
Decay R
Very steep after peak
Steady, gradual
Steady, moderate
Model-Specific Patterns
Fisher Info per Parameter
Medium Single peak & steep drop Layer bump (14.067)
Lowest
Low Heavy & gradual decline Smoother distribution
Layer Distribution
Heavy gradual decline Final layers bump
Unique Feature
DDecyy rate describes how quickly Fisher information diminishes layers
21

---

Table 22: Value Space analysis across Qwen model variants
Property
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-7B-Instrut
Value Space Metrics
Relative Magnitude (Mean)
0.4655
0.4890
0.4897
Directional Influence (Mean)
0,5313
0,5521
0,3525
Directional Influence (Median)
0.5000
0,5000
0.5000
Information Content (Mean)
1792.7866
7161.5074
7162.1374
Information Content (Median)
2054.0861
8431.1512
8519. 9060
First-to-Last Layer Influence
-0.15
+0.18
+0.18
Attention Value Correlation
Attention Entropy (Mean)
1,5390
1.4449
1.4711
Value Transformation Magnitude
31.6244
16.2740
14,9721
First Layer Magnitude
21.4661
15.0259
14.7112
Last Layer Magnitude
358.9528
222.3867
191.2135
Geometric-Semantie Alignment
-0,1472
-0.0426
-0.0395
First Layer Alignment
0.0676
0.0569
0.0578
Last Layer Alignment
-0.0349
0.0000
0.0000
Correlation
-0,0842
-0.2325
-0.2329
First-to-Last Layer Shift
-0.4763 to -0.2981
-0.7045 to -0.0459
-0.6958 to -0.0501
G Gemma and Mistral
As we can see from table 23 both models show 100% specialization on their respective beginning-of sequence tokens (
 in Mistral, <bos> in Gemma). This perfect specialization is the hallmark of centralized reference frames, establishing a single dominant reference point that serves as the universal origin for the representation manifold. Both models maintain identical Bettig counts from early to late layers (27.24 + 27.24 in Mistral, 26.11 - 26.11 in Gemma), indicating complete topological stability of component structure. This stability is characteristic of centralized frames, where the component organization remains fixed throughout the network. Both models show monotonically increasing persistence values through layers, with Mistral showing +0.0796 increase and Gemma showing a remarkable +0.3163 increase. This strengthening of reference point significance through network depth is defining property of centralized frames that differentiates them from distributed frames (which show decreasing persistence) and hybrid frames like Pythia. Like other decoder models, both Mistral and Gemma show no loops (Betti, = 0.00 = 0.00) at any layer, contrasting sharply with the complex cyclic structures found in bidirectional encoder models like BERT and RoBERTa.
Table 23: Centralized Reference Frame Signatures in Mistral and Gemma Models
Property
Mistral-7B-v0.33
Gemma-7B
Position encoding
RoPE
RoPE (Modified)
Token specialization
100%
100% on <bos>
Bettig (early)
27.24
26.11 11
Bettip (late)
27.24
26.11 1
Bettig change
0.00
0.00
Betti, (carly)
0.00
0.00
Betti (late)
0.00
0.00
Betti, change
0.00
0.00
Dim0 persistence (early)
0.0310
0.2015
Dim0 persistence (late)
0.1105
0.5179
Persistence change
+0.0796
+0.3163
Max attention StdDev
0.1503 (layer 24)
0.1404 (layer 21)
22

---

Despite sharing the same fundamental reference frame type. Mistral and Gemma show important differences in how they implement their centralized frames: for example, gemma starts with much higher persistence values (0.2015) compared to Mistral (0.0310), indicating stronger initial reference point structure. This suggests Gemma establishes a more dominant centralized reference from the earliest layers. The most striking difference is in the magnitude of persistence growth. Gemma's s persistence increase (+0.3163) is nearly four times larger than Mistral's (+0.0796), resulting in an extremely high final persistence value of 0.5179. This suggests Gemma's s centralized reference becomes exceptionally dominant in deeper layers. While both models show peak attention standard deviation in similar layers (24 for Mistral, 21 for Gemma), the overall attention distributions and specialization patterns across layers likely differ in subtle ways not fully captured in the topological metrics.
In table 24 we can see the algebraic connectivity (Fiedler values) across both models shows a distinctive pattern: for Mistral-7B, we observe a clear "inverted U" pattern where connectivity peaks in middle layers (13.3343) compared to early (13.0303) and late layers (12.0898). This pattern suggests that middle layers serve as critical transition point in the reference frame structure- they balance information flow between the initial reference frame establishment and the subsequent semantic processing. In contrast, Gemma-7B shows a monotonic decrease in connectivity from early (13.0545) to middle (1 (11.0557) to late layers (9.9455). This different pattern suggests that Gemma implements distinctly different reference frame strategy, potentially establishing stronger reference points earlier in the network.
Table 24: Spectral Graph Analysis of Reference Frame Structures
Property
M7stral-7B-v0.3
Gemma-7B
Most effective threshold
0.001
0.001
Algebraic Connectivity (Fiedler Values)
Early layers (avg)
13,0303
13.0545
Middle layers (avg)
13.3343
11.0557
Late layers (avg)
12.0898
9.9455
Maximum value
20.7132 (layer 0)
15.1970 (layer 1)
Like-iie
Early layers (avg)
0.5371
0.5354
Middle layers (avg)
0.5345
0,5344
Late layers (avg)
0.5372
0.5344
Maximum value
0.5441 (layer 2)
0.5382 (layer 14)
Degree Centralization
Early layers (avg)
0.5475
0.5357
Middle layers (avg)
0.5411
0.5488
Late layers (avg)
0.5564
0.5650
Maximum value
0.5942 (layer 2)
0.5948 (layer 22)
One of the most striking findings in the Mistral data is the strong negative correlation (-0.9405) between algebraic connectivity and degree centralization. This trade-off represents a fundamental tension in transformer architecture design, between establishing stable reference points (high central- ization) and enabling efficient information flow (high connectivity). The fact that this correlation is so strong (-0.9405) suggests this is not incidental pattern but core organizing principle. Both models show remarkably similar star-likeness metrics at the baseline threshold (0.001), because both models converge on very similar star-like structures despite their different architectural designs. The fact that star-likeness metrics remain consistently high (>0.53) across all layers in both models indicates that reference points remain essential throughout the entire network depth, not just in early stages.
The strong positive correlation (0.9481) between Fiedler values and degree variance in Mistral suggests that the model balances compression and relevance (as measured by connectivity), it simultancously increases the variance in how information is distributed, creating specialized processing structure.
In table 25 we can see the predominantly negative KI. reduction values throughout middle layers (-0.0557 for Mistral, -0.0590 for Gemma) indicate that removing attention sinks increases the KL
23

---

Table 25: Attention Sink KL Divergence Analysis for Reference Frame Models
Property
Mistral-7B-v0.1
Gemma-7B
KL Reduction (1=0.8)
Early layers (0-5)
-0.0576
0.0105
Middle layers (7-19)
-0.0557
-0,0590
Late layers (21-31)
-0.0703
-0,0330
First layer
0.1342
0.1435
Final layer
0.0111
0.0576
Sink Concentration (t=0.8)
Early layers (0-5)
89.62%
73.59%
Middle layers (7-19)
81.55%
80.28%
Late layers (21-31)
84.90%
80.75%
First layer
86.95%
61.46%
Final layer
63.64%
53.51%
Layer-specific Patterns
Highest sink concentration
95.44% (layer 21)
91.57% (layer 21)
Lowest sink concentration
63.64% (layer 31)
53.51 % (layer 27)
Most negative KI. reduction
-0.1907 (layer 3)
-0.1218 (layer 11)
Threshold Effects (final layer)
KL. reduction (t=0.8)
0.0111
0.0576
KL reduction (1=0.9)
-0.0257
0.0391
KL reduction (t=0.95)
-0.0413
0.0285
Threshold Effects (sink concentration) 008 to 1=095 change
-23.9% points
-18.8% points
divergence between attention distributions. What's particularly fascinating is that both models show positive KL reduction values their first layer 13.1342 for Mistral, 0.1455 for Gemma) and final layer (0.0111 for Mistral, 0.0576 for Gemma). This U-shaped pattern suggests that reference frames serve different informatiio-geomerre functions different depths in the network
Table 26: Key Fiedler Value Correlations in Mistral-7B and Gemma-78 7 Models
Threshold
Property
Gemma-7B Mistral-7B
0.01
Centralization vs. Fiedler
-0.7787
-0.7429
0.05
Centralization vs. Fiedler
-0,
-0,3591
0.10
Centralization vs. Fiedler
0.5893
0.2162
0.01
Density Fiedler
0.7345
0.6935
0.10
Density vs. Fiedler
0.5764
0.4387
Mistral shows extremely high sink concentration in early layers (89.62% at t=0.8), which then decreases in middle layers (81.55%) before slightly increasing again in late layers (84.90%). This pattern aligns with your description of centralized reference frames where single dominant reference point establishes a universal origin. Gemma shows a markedly different pattern with significantly lower sink concentration in early layers (73.59% at (=0.8), which then increases in middle layers (80.28%) and remains stable late layers (80.75%). This suggests a more distributed reference frame that develops progressively through the network depth.
One of the most striking differences between the models in their first-layer sink concentration. the difference (25.49 percentage points) likely reflects the different position encoding implemen- tations you described your theoretical framework. Mistral's standard RoPE implementation creates stronger positional bias toward the first token, facilitating more centralized reference frame. Gemma's modified position encoding appears to reduce this positional bias, enabling more distributed reference frame with multiple weaker reference points.

---

The fact that both models ultimately achieve similarly high maximum sink concentrations (95.44% vs. 91.57%, both layer 21) suggests that strong reference points are mathematically necessary despite these architectural differences.
Table 27: Fisher Information Distribution in Mistral and Gemma Models
Architectural Pattern
Mistral-7B-- -B-11.
Gemma-7B
Component Importance (Percentage of Total Fisher Norm)
Attention Mechanism
58.8%
High Very High Low
MLP Components Embedding
31.8%
8.7%
Layer Distribution Patterns
Key Processing Layers
Layer (44.0%)
Layers 26-27 (77.2%)
Early Layers (0-9) Middle Layers (10-19)
73.9%
High* Medium
11.6%
Final Layers (20-31)
4.4%
Very High*
Layer Decay Characteristies
Peak Layer Value
Layer I:
3,053,101
Layer I: 13,804,803
Peak Minimum Ratio
165:1
High U-shaped distribution
Decay Pattern
Sharp drop, slow decline
Model-Specific Patterns
Fisher Info Distribution
Front-loaded, with minor bump at end
Strong bi-modal distribution
Layer Distribution
Steep initial drop, then plateau
Sharp drop after Layer in final layers Extreme values layers and 27
Unique Feature
Small humps layers 12 and
"Exact values affected by overflow in Gemma computation (Infinity reported some components)
Table 28: Value space analysis for Gemma and Mistral models
Property
google/gemma-77
mistrralai/Miittral-BB-.000
Value Space Metrics
Relative Magnitude (Mean)
0.5796
0.9758
Directional Influence (Mean)
0.5830
0,9771
Directional Influence (Median)
0.5312
0,9827
Information Content (Mean)
307.7589
179. 179.1840
Information Content (Median)
309.7665
189.7641
First-to-Last Layer Influence
+0.33
+0.01
Attention- Value Correlation
Attention Entropy (Mean)
1.2289
1.1524
Value Transformation Magnitude
28.2835
14.4851
First Layer Magnitude
205.1822
0,3779
Last Layer Magnitude
446.5816
355.8180
Geometric-Semantic Alignment
-0.4012
-0.1386
First Layer Alignment
-0.6124
-0.1006
Last Layer Alignment
-0.3177
-0.3320
Entropy-Magnitude Correlation
-0.2408
0.2264
First-to-Last Layer Shift
-0.3443 to 1464
-0.4777 to -0.1529
H Encoder only (BERT and Roberta)
In table 29 we can see that both models exhibit striking pattern of dual specialization where attention heads focus on different special tokens depending on layer depth: early layers show perfect (100%) specialization on beginning tokens ([CLS] in BERT, <S> RoBERTa); and middle/late layers show perfect (100%) specialization ending tokens ([SEP] in BERT, 
 in RoBERTa). This layer-specific specialization pattern creates "bipolar" reference strueture that differs fundamentally from both the single-point centralization in Llama and the distributed multi-poin structure in Qwen.
25

---

Both encoder models exhibit remarkably high initial topological complexity: high Betti values (15.40 in BERT. 19.69 RoBERTa) indicate numerous loops/cycles early layers, and this contrasts sharply with decoder models (Llama, Qwen) which show virtually no loops (Betti = 0.00). This high initial complexity reflects how bidirectional attention creates rich interconnected structures that allow tokens to reference each other in complex ways rather than primarily referencing a central point or distributed landmarks.
Table 29: Bidirectional Reference Frame Signatures in Encoder Models
Property
BERRTbase-uncaseed
XIM-RRBEKKTa-arrge
Token specialization
Early: 100% on [CLS]
Early: 100% on 

Middle: 100% on [SEP]
Late: 100% on 

Bettip (early)
4.48
22.68
Bettip (late)
4.33
1.69
Bettig change
-0.14
-20.99
Betti; (early)
15.40
19.69
Betti, (late)
2.79
3.11
Betti: change
-12.61
-16.58
Dim0 persistence (early)
0.0453
0.0521
Dim0 persistence (late)
0.0194
0.0156
Persistence change
-0.0259
-0.0365
Max attention pattern
Layer O: 0.3362
Layer O: 0.1731
Layer 6: 0.8765
Layer 11: 0.4731
Layer 12: 0.8135
Layer 23: 0.3540
As information flows through the network, bidirectional reference frames undergo dramatic topo- logical simplification, for example, we can see substantial reduction in loops (Betti, decreases by 12.61 in BERT, 16.58 in RoBERTa) Component integration in RoBERTa is particularly interesting, since Betti, decreases from 22.68 1.69. This simplification pattern suggests that bidirectional frames start with complex relationships that gradually consolidate around key reference points as information processed.
Both models show decreasing persistence values from early to late layers, similar to distributed frames but unlike centralized frames:
BERT: 0.0453 > 0.0194(-0.0259)
RoBERTa: 0.0521 + 0.0156(-0.0365
This suggests that feature significance weakens through layers as the model transitions from complex initial representations more specialized final representations.
In table 30 we can see that while decoder models (Llama, Mistral, Gemma) show their maximum connectivity in layer 0, these bidirectional models show maximum connectivity in early but not initial layers (layer 2 for BERT, layer for BoERRa) This subtle shift indicates a fundamentally different geometric organization strategy. What's 's more remarkable is the dramatic drop in connectivity from early to middle layers (46.69 + 23.79 in BERT, 38.27 + 31.25 in XLM-RoBERTa), which contrasts sharply with the gentle declines even increases seen in decoder models. This suggests rapid geometric reorganization as information moves through the network.
The U-shaped algebraic connectivity pattern in BERT (high-llw- -medium)) differs completely from both centralized reference frames This suggests fundamentally different geometric organization. Both models show much lower baseline star- star-likenn likeness values (0.33-0.39) compared decoder models (0.53-0.54), suggesting less reliance on single reference points. This aligns with your theory that bidirectional models establish dual reference points rather than a single dominant one. Unlike decoder models, these architectures show peak degree centralization in middle layers at the low threshold (0. 1579 for BERT, 1274 for XLEM-,)E indicating that the reference structure evolves substantially through network depth. The effectiveness scores drop sharply at the 0.1 threshold (0.42 for BERT,
-iihh-mmiuumu) and distributed reference frames
(high- niigh-mmeiium--oon
26

---

much Table 30: Bidirectional Reference Frame Signatures in Encoder Models
Property
BERT-base-uneased
XLM-RoBERTa-arre
Position encoding
Absolute
Absolute
Most effective threshold
0.01
0.01
Threshold effectiveness (0.1)
0.42
0.38
Algebraic connectivity (early)
46.6913
38,2726
Algebraic connectivity (middle)
23.7880
31,2481
Algebraic connectivity (late)
27.3752
22.5304
Maximum connectivity
52.4496 (layer
55.3003 (layer 1)
Bireliienesss (early)
0.3274
0.3491
Star-likeness (middle)
0.3680
0.3550
Star-likeness (late)
0,3585
0,3941
Maximum star-likeness
0.4121 (layer 7)
0.4383 (layer 20)
Degree centralization (early)
0.0467
0.0786
Degree centralization (middle)
0.1579
0.1274
Degree centralization (late)
1208
0.2234
Maximum centralization
0.2730 (layer 7)
0.3556 (layer 19)
Degree variance (carly)
28.7433
98.2550
Degree variance (middle)
50.4899
50.8467
Degree variance (late)
52.4320
89.7002
0.38 for XLM-RoBERTa) compared decoder models, indicating more complex attention patterns that can't 't be simplified to binary relationships.
that the most insteresting pattern in the data is the distinctive U-shaped KL reduction profile across network depth. Both models show positive KI. reduction in their first layer (0.0392 for BERT, 0.0635 for XIoM-RBERRTa) and final near-final layers (XI.-RRBEERT: shows 0.0774 in layer 23). The larger model, roberta, shows more dramatic U-shaped pattern, with stronger positive KI reduction in the final layer (0.0774 -0.0179). This suggests that increased model capacity allows for more distinct reference points at sequence boundaries. Also, Roberta shows its peak negative KL reduction in a deeper layer (layer 9 vs. layer 5 in BERT), indicating that larger models can maintain the initial coordinate system longer before transitioning to the integration phase.
In table 32
Table 31: Key Fiedler Value Correlations in Encoder Models with Bidirectional Reference Frames
Threshold
Property
BERT-base
XIM-RoBERTa
0.001
Centralization vs. Fiedler
0.1801
0.3242
0.02
Centralization vs. Fiedler
-0,8442
-0.6672
0.1
Centralization vs. Fiedler 0.3233
-0.3302
0.2
Centralization vs. Fiedler
0,2859
0.2347
0.01
Density vs. Fiedler
0.8227
0.6146
Both models show substantial degradation in KI. reduction at higher thresholds in early layers (-45.2% for BERT, -58. 58.1% for XLM-RoBERTa from t=0.8 to t=0.95). This suggests that the initial reference point relies on broad distribution of attention weights. Both bidirectional models show remarkably low sink concentration in their first layers (32.07% for BERT, 28, 19% for lower than the 80-90% concentration typically seen in decoder models. This indicates fundamentally different approach to establishing initial coordinate systems.
The clear differences between bidirectional and decoder models confirm that architectural choices fundamentally shape the geometric organization of the representation space. The use of absolute position embeddings in these models corresponds to their bipolar reference structure, supporting your claim about position encoding implementations influencing reference frame types. And the U-shaped KL reduction pattern provides perhaps the clearest evidence yet for the characterization
27

---

Table 32: KL Divergence Analysis of Bidirectional Reference Frames
Property
BERT-base-uncased
XLM-RoBERTh--arge
KL Reduction (1=0.8)
Early layers (0-3)
-0.0243
-0,0306
Middle layers (5-9)
-0.1121
-0,2156
Late layers (11+)
-0.0179
-0,0826
First layer
0.0392
0.0635
Final layer
-0,0179
0.0774
Maximum negative
-0.1733 (layer 5)
-0.2574 (layer 9)
Sink Concentration (t=0.8)
Early layers (0-3)
51.10%
51.06%
Middle layers (5-9)
71.73%
75.06 75.06%
Late layers (11+)
77.20%
73. 1%
First layer
32.07%
28.19%
Final layer
77.20%
44.39%
Maximum concentration
77.20% (layer 11)
85.22% (layer 21)
Layer-specistc Patterns
KI. U-shape (first-final)
Yes (+)
Yes (+)
Mid-layer KL reduction peak
Layer 5
Layer 9
Late layer low concentration
No
Yes (44.39%)
Threshold Effects
0.8 to t=0.95 deepening
-137.9%
-15.5%
Early KL reduction change
-45.2%
-58. 1%
Middle KL reduction change
-22.2%
+15.5%
Late KL reduction change
-117.9%
-2.1%
of bidirectional models establishing a bipolar manifold with reference points both sequence boundaries.
Table 33: Architectural Pattern Analysis Based on Fisher Information Distribution in Encoder Models
Architectural Pattern
BERT-base-uncased
XLM-RoBERTa-large
Component Importance (Percentage of Total Fisher Norm)
Attention Mechanism
22.5%
71.7%
MLP Components Embedding
44.5%
27.8%
31.3%
0.2%
Layer Distribution Patterns
Key Processing Layers
Layers 0-1, 11 (32.7%)
Layers 7-10 (64.7%)
Early Layers (0-3)
35.9%
10.5%
Middle Layers (4-8)
16.1%
52.4%
Final Layers (9+)
13.0%
36.9%
Layer Decay Characteristics
Peak Layer Value
Layer 1: 666
Layer 9: 2,504.416 302.6:1
Peak to Minimum Ratio
5.9:1
Decay Pattern
Gradual decline with end spike
Inverted U-shape with end spike
Mpdel-Specific Patterns
Fisher Info Distribution
Relatively balanced Early and final emphasis High embedding importance
Highly concentrated in middle Middle-heav with final spike Dramatic middle-layer concentration
Layer Distribution
Unique Feature
28

---

Table 34: Value space analysis for BERT and XLM-RoBERTa models
Property
BEKTTbase-uncased
XLM--oBERTTa-argge
Value Space Metrics
Relative Magnitude (Mean)
0.5289
0.5928
Directional Influence (Mean)
0,8141
0.9394
Directional Influence (Median)
0,8239
0.9683
Information Content (Mean)
4.0998
13.7629
Information Content (Median)
4.3081
14.3960
First-to-Last Layer Influence
+0.19
+0.20
Attention- Correlation
Attention Entropy (Mean)
2.3200
2.1957
Value Transformation Magnitude
8.6058
8.1538
First Layer Magnitude
9.4719
19.7373
Last Layer Magnitude
11.5637
23.8058
Geometric-Semantie Alignment
-0.1200
0.0013
First Layer Alignment
0.1246
0.2065
Last Layer Aligmment
-0.3608
-0.3982
Entropy-Magniudde Correlation
0.2940
0.2326
First-to-Last Layer Shift
-0.1935 to -0.1043
-0.1552 to 0.6409
I Phi
The results from microsoft/phi-2 shown in table reveal clear distributed reference frame archi- tecture characterized by distinctive patterns across multiple analytical methodologies. The Fisher information distribution demonstrates balanced allocation between attention mechanisms (37.4%) and MLP components (32.7%), contrasting with the attention -ilenito-dommnntedd profiles typically observed in centralized reference frame models. Rather than concentrating information processing in early layers, phi-2 exhibits distinctive triple-peate -pakee distribution with significant Fisher norm values in layers 0-1 (5.5%). secondary peak around layer 26 (262.81), and pronounced concentration in final layers 29-31 31 (49.6% of total Fisher norm). This multi-peaked pattern represents fundamental departure from the early-layer concentrated processing found in centralized reference frame architec- tures like LLaMA. The KL divergence analysis provides particularly compelling evidence for the distributed reference frame classification through its characteristic three-phase pattern. Early layers (particularly layer 0) show positive KL reduction values (+0.1416 at threshold 0.8), indicating that removing attention sinks actually improves information flow at these stages. This transitions sharply to negative values in middle layers (ranging from -0.1525 to 1995), followed by stronger negative values in deeper layers (peaking at -0.2787 in layer 25). This progression reflects a fundamentally different approach to utilizing reference points compared to centralized models, which typically show consistently negative KL reduction across all network depths. The attention sink concentration metrics further reinforce this pattern, showing progressive buildup from relatively low values in early layers (35.23% in layer 0) to very high concentration in deep layers (97.1995 in layer 25), rather than the consistently high concentration characteristic of centralized reference frames. Spectral graph analysis identifies the mathematical signature that definitively marks phi-2 as implementing distributed reference frame. The model exhibits a characteristic sign-flipping correlation pattern between Fiedler values (algebraic connectivity) and centralization metrics, shifting from negative correlation low thresholds (-0.4518 0.001) to strong positive correlation higher thresholds (+0.6009 at 0.1). This pattern closely mirrors the signature observed in Qwen models (-0.46 + +0.61) that use similar NTK-aware scaled rotary position embeddings. Layer-specific correlations reveal particularly strong negative correlations in early layers (-0.8796 in layer 0) at low thresholds, with progressive transition positive correlations at higher thresholds through network depth. Topological analysis supports these findings, showing substantial evolution in attention structure across network depth. The Bettig numbers (connected components) increase dramatically from 0.0 layer 0 to 25.99 in layer 31, indicating progressive fragmentation of attention rather than the stable topolog-
29

---

Table 35: Reference Frame Analysis of Microsoft/Phi-2
Analysis Category
Microsoft/Phi-2
Fisher Information Distribution
Attention Mechanism
37.4% (3905.88)
MLP Components
32.7% (3419.73)
Embedding Other Components
1.0% (105.28)
28.9% (3012.81)
Layer Distribution Patterns
Key Processing Layers
Layers 0-1 (5.5%), 29-31 (49.6%)
Early Layers (0-10)
19.8% (2066.12)
Middle Layers (11-28)
17.4% (1818.65)
Final Layers (29-31)
49.6% (5181661)
Layer Decay Characteristics
Peak Layer Value
Layer 30: 2973.23 Layer 342.85, Layer 26: 262.81 30.3:1 (Layer 30 vs Layer 17) Yes (Farly, Middle, Late)
Secondary Peaks
Peak to Minimum Ratio
Multi-peaked Pattern
KL Divergence Analysis
Early Layer KL Red. (t=0.8)
Positive (Layer 0: +0.1416)
Middle Layer KL Red. (t=0.8)
Negative (Layer 11: -0.1525)
Deep Layer KL Red. (t=0.8)
Stronger Negative (Layer 25: -0.2787)
KL Pattern
Three-phase (Positive + Negative + Stronger Negative)
Attention Sink Concentration
Early Layer Concentration
Low (Layer 0: 35.23%)
Middle Layer Concentration
High (Layer 11: 92.37%)
Deep Layer Concentration Concentration Growth
Very High (Layer 25: 97.19%)
Progressive (35.23% + 97.19%)
Spectral Graph Analysis
Low Threshold Correlation (0.001)
Negative (Hiedler vs. Centralization: -0.4518)
High Threshold Correlation (0.1)
Positive (Fiedler vs. Centralization: +0,6009)
Comrelatinn Pattern
Sign fign-fliping (-0.4518 +0.6009)
Signature Feature
Early layer strong negative correlation (-0.8796 layer
Topological Features
Bettip Progression
0.0 25.99 (Layers 0 31)
Betti Values
Consistently 0.00 (No loops) +25,99 (significant fragmentation) Progressive disconnection across depth
Connected Component Change
Structural Evolution
Reference Frame Classification
Frame Type
Distributed Reference Frame NTK-awarc scaled RoPE
Position Encoding
Key Evidence
Three-phase KL pattern, Sign- wiippiiippingg..cm. correlation. Multi-peaked Fisher information, Progressive sink concentration
ical structure maintained in centralized reference frame models. This fragmentation reflects how distributed reference frames implement more flexible coordinate system that adapt different computational needs across network depth.
J
Pythia Family
Examining the Pythia family of models (Table 37 reveals how centralized reference frames evolve with increasing model scale. While Pythia models exhibit the defining characteristics of centralized reference frames, we observe systematic scaling relationship between model size and reference
30

---

Table 36: Value space analysis for Microsoft Phi-2
Property
microsoft/phi-2
Value Space Metrics
Relative Magnitude (Mean)
0.5041
Directional Influence (Mean)
0,5334
Directional Influence (Median)
0.5000
Information Content (Mean)
593.5976
Information Content (Median)
710.1380
First-to-Last Layer Influence
-0.23
Attention- Value Correlation
Attention Entropy (Mean)
0.9237
Value Transformation Magnitude
23,9090
First Layer Magnitude
22.6574
Last Layer Magnitude
0,0000
Geometric-Semantee Alignment
-0.1478
First Layer Alignment
0,0370
Last Layer Alignment
0,0000
Entropy-Magniun Correlation
0.1305
First-to-Last Layer Shift
-0.5548 to 0.0000
point strength. As model scale increases from 2.8B to 12B parameters, token specialization the most attended token (consistently "Gthe") increases proportionally from 31.0% to 42.2%. The number of specialized attention heads also scales with model size, increasing from to 4 across the family. This suggests that larger models develop more pronounced centralized reference structures, potentially enabling more efficient information routing through the network. Topologically, Pythia models maintain remarkably stable Bettip numbers 25.77 across all model scales and through network depth, confirming the centralized reference frame pattern. However, we observe consistent decrease in persistence values from early to late layers, with larger models showing more dramatic reductions in persistence (-0.1065 2.8B to -0.1671 12B). This counter-intuitive finding suggests that while larger models establish stronger reference points, they simultaneously develop more nuanced relationships between these reference points and contextual tokens.
The maximum attention standard deviation shifts to earlier layers as model scale increases (from layer 16 in 2.8B layer in 12B), indicating that larger models establish their reference structures more efficiently and earlier in the processing pipeline. This aligns with our temporal emergence findings that reference frames develop more rapidly during training in larger models.
In table 37 we can see that as model size increases from 2.8B to 12B parameters, there is a clear strengthening of centralized reference frame structures. For example, the consistency of attention to the token "Gthe" increases systematically with model scale (31.0% - 36.8% + 42.2%). This suggests that larger models develop more robust reference points, supporting the theory that reference frames are fundamental geometric adaptations. Then, we can see that the number specialized heads increases from just 1 in the smallest model to 4 in the largest. This indicates that with greater capacity, models allocate more resources to establishing reference frames, underscoring their importance.
The Betti numbers show consistency in the basic topological structure: all three models maintain nearly identical Bettig values ( (25.7) across both carly and late layers. This stability across model scales suggests that this particular topological configuration represents an optimal solution to the geometric organization. The consistent Betti, value of 0.00 across all models and layers indicates that these centralized reference frames organize tokens in tree-like structure rather than forming loops, aligning with the description of star-like topologies that optimize information routing.
Interestingly, initial persistence values slightly decrease as model size increases (0. 18.1 - 0.1801 + 0. 0.1671), suggesting that larger models might initially establish more nuanced or distributed reference structures. Both larger models (6.9B and 12B) show complete persistence decay to 0.0000 in late layers, while the smallest model retains some persistence (0.0760). This suggests that with sufficient capacity, models more fully optimize the geometric organization through the network depth. All
31

---

models show negative persistence changes, contrasting with the positive changes in Mistral/Gemma comparison. This difference might indicate architectural variations in how reference frames evolve through the network depth.
All three models achieve similar maximum attention standard deviation values (0.1342 0. 0.1471), suggesting consistent degree of attention concentration regardless of scale. The layer of maximum attention concentration varies (layer 16 + 9), with larger models generally achieving peak concentration in earlier layers.
Table 37: Centralized Reference Frame Signatures Pythia Models
Property
Pythia-2.8B
Pythia-6.9B
Pythia-
Top token specialization Number of specialized heads
31.0% on Gthe
36.8% on Gthe
42.2% on Gthe
3
4
Bettio (early)
25.72
25.71
25.73
Bettio (late)
25.71
25.71
25.73
Bettio change
-0.01
0.00
0.00
Betti (early)
0.00
0.00
0.00
Betti (late)
0.00
0.00
0.00
Betti, change
0.00
0.00
0.00
DimO persistence (early)
0.1825
0.1801
0.1671
DimO persistence (late)
0.0760
0.0000
0.0000
Persistence change
-0.1065
-0.1801
-0.1671
Max attention StdDev
0.1342 (layer 16)
0.1465 (layer 8) 0.1471 (layer 9)
Table 38: Architectural Pattern Analysis Based on Fisher Information Distribution in Pythia Models
Architectural Pattern
Pythia-2.8B
Pythia-6.9B
Component Importance (Percentage of Total Fisher Norm)
Attention Mechanism
60.8%
77.6%
MLP Components
11.6%
12.0%
Embedding
27.5%
10.4%
Layer Distribution Patterns
Key Processing Layers
Layers 24, 27-29 (30.9%)
Layers 23-27, (38.1%
Early Layers (0-9) Middle Layers (10-20)
6,4%
7.1%
10.9%
23 23.9%
Final Layers (21-31)
57.5%
59.0%
Layer Decay Characteristics
Peak Layer Value
Layer 24: 971.90
Layer 26: 3,100.50
Peak Minimum Ratio
29.4:1
45.0:1
Decay Pattern
Steady rise toward end
Gradual rise, steep end increase
Model-Specific Patterns
Fisher Info Distribution
Heavily back-loaded Minimal early importance
Heavily back-loaded More balanced with strong final focus Strong ramp-up starting at layer 18
Layer Distribution
Unique Feature
Dual peaks (24 and 27)
32

---

Table 39; Key Fiedler Value Correlations Across Pythia Models by Size
Correlation Pattern
Pythia-2.8B
Pythia-6.9B
Pythia-12B
Centralization Fiedler Values
At threshold 0,01
-0,0491 0.2199
-0.5235 0.0004
-0,5895 0.5083
At threshold 0.1
Correlation strength scaling
Weak
Moderate
Strong
Density vs. Fiedler Values
At threshold 0.02 Pattern consistency
0.1069 Inconsistent
0.5971 Moderate
0.6782 Highly consistent
Table 40: Value space analysis across Pythia model variants
Property
Pythia-2.8P
Pythia-6.9B
Pythia-12B
Value Space Metrics
Relative Magnitude (Mean)
0.4934
0.4759
0.4782
Directional Influence (Mean)
0,5614
0,5506
0,5432
Directional Influence (Median)
0,5000
0,5000
0,5000
Information Content (Mean)
485.9178
950.4851
930.9070
Information Content (Median)
598.1627
954.3322
0.1200
First-to- Layer Influence
-0.06
-0.25
-0.25
Attention Value Correlation
Attention Entropy (Mean)
1.0975
0.9041
0.6942
Value Transformation Magnitude
33,0226
25.9536
24.9152
First Layer Magnitude
38.0535
56.1840
84.1770
Last Layer Magnitude
67.5896
0.0000
0,0000
Geometric-Semant Semantic Alignment
0.0483
-0.0421
-0.0687
First Layer Alignment
0.0512
0.0506
0.0277
Last Layer Alignment
0.0215
0.0000
0.0000
Matropy-Magnituee Correlation
0.0110
-0,2509
-0.2332
First-to-Lass Layer Shift
-0.4454 to -0.1034
-0.4288 to 0.0000
-0.3498 0.0000
K NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly articulate our three main contributions: (i) reframing attention sinks as geometric reference frames, (ii) identifying and characterizing three distinct reference frame types, and (iii) demonstrating how architectural choices determine which reference frame emerges. These claims are substantiated through our methodology (Section 3 and 4) and analysis results (Section 5).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals not attained by the paper.
33

---

2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper explicitly acknowledges limitations in Section 5 (paragraph 'Lim- itations"), ) noting that our topological and spectral analyses focus on attention patterns at specific network snapshots rather than continuously tracking their evolution throughout training, which may miss important transitional dynamics reference frame formation.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper states theoretical assumptions and provides complete mathemat- ical formulations in Section 3, particularly in the subsections Reference Frames (3.1), Emergence of Attention Sinks (3.2), and Position Encoding (3.3). These sections present our full mathematical framework with clearly defined notation, formal equations for each reference frame type, and rigorous derivations how different architectures lead to specific geometric structures.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
34

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Section 4 (Methodology) provides comprehensive details our experimental design, including model specifications (LLaMA-3.2, Qwen2.5, etc.), dataset composition (500 STEM-focused Wikipedia sentences), implementation of topological and spectral analyses, and computational resources used (Google Colab with T4/A100 GPUs). These specifications, combined with the detailed metries and formulas presented, enable full reproduction of our experimental results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttps://nips.co cc/
for more details.
35

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wttpc//ggiiisss/odeeee Sat////ooSSSSiiaaeeeeeeee www.ssssssniiiionioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section (Methodology) provides comprehensive details of our experimental setup, including the diverse set of transformer models analyzed (LLaMA-3.2, Phi-2, Qwen- 2.5, BERT, XL.M-RoEERTa, etc.), dataset specifications (500 STEM-focusee sentences ranging from 6-50 tokens), analysis methods (topological, spectral, information- theoretic), and computational environment (Google Colab with T4/A GPUs). Our experimental design subsection specifically details the threshold values used (0.001-0.2), dataset sampling approach, and measurement metrics each analysis type.
Wikipedia
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While we don't include explicit error bars, we report standard deviations for key metrics (e.g., in Table 4's Information Content Change) and ensure statistical reliability by analyzing patterns across multiple architectures, confirming our findings are consistent rather than due to random variation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
36

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should he careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section (Methodology) specifies that all experiments were conducted using Google Colab with T4 A100 GPUs. While the paper doesn't 't include exact execution times, it provides sufficient computational context by detailing the sample sizes (500 samples for main analyses, 100 samples for temporal analysis) and model scales (ranging from 1B to 12B parameters) that define the resource requirements needed for reproduction.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work focuses on analyzing existing, publicly available language models understand their internal mechanisms (reference frames), without creating potentially harmful capabilities, using private data, or enabling applications that could lead societal harm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdictio).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: While our paper primarily focuses on theoretical analysis of transformer architectures, briefly mention in the conclusion that understanding reference frames has potential positive implications for developing more principled approaches to transformer engineering.
Guidelines:
37

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles. surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release new models, datasets, or tools that could be potentially
harmful.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our paper properly credits the original model creators (e.g., Meta Al for LLaMA, Mistral AI for Mistral, Google for Gemma) when analyzing their architectures. We used publicly available model checkpoints accessed through the Hugging Face platform in accordance with their respective licenses, and our "TEM-focused Wikipedia dataset was derived from Wikipedia content under CC BY-SA 3.0 licensing.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
38

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our paper doesn't 't introduce new datasets, or models as downloadable assets. We released the code and its documentation supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This research doesn't involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research doesn't involve any human subjects, data collection from individ- uals, or human evaluation of results.
39

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research methodology does large language models components of our core research methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy http:///eeerippsscco//eeeeeeeeee for what should or should not be described.
40


Reviews:
[{"review_id": "DxjVuJHtkZ", "reviewer": "Reviewer_uQoB", "summary": "This paper introduces a geometric interpretation for attention sinks in transformers, proposing that these sinks emerge as optimal 'reference frames' under softmax constraints in an emergent coordinate system. Crucially, they emphasize that these are not accidental artifacts but fundamental outcomes of geometric constraints on the learning dynamics of transformers. The authors identify three reference-frame types—centralized, distributed, and bidirectional—linked to inductive biases from positional-encoding schemes, and validate these patterns across 10+ model checkpoints (e.g. LLaMA, Qwen, BERT) using topological, spectral, and information-theoretic analyses", "questions": "### Questions or Suggestions\n- The authors state that reference frames (and the specific reference frame types they observe) are *optimal solutions* under the softmax constraint to \"establishing stable coordinate systems in high-dimensional spaces\". In what sense do they mean \"optimal\" here, optimal with respect to which exact objective? Could the authors elaborate a little bit on this.\n- Could you clarify the definition of a specialized head is in Table 1? I'm trying to relate the dataset size to these measures, e.g. does it have to meet the reference frame attention threshold for all samples in the dataset? \n- What precisely is \"Token Specialization\" in Table 1?\n\t- e.g. What does 100% on BOS token mean?  I assumed it meant that all specialized heads (ones that demonstrate attention sinks) only attend to the BOS. \n\t- What about the *65.4% on \",\"*, for the distributed frame? What do the other specialized heads attend to? Are these commas in the same position (absolute or relative--i.e. the first comma in the sentence) across all sentences, or different.\n- As mentioned in the weaknesses section, could you provide charts of the distribution of attention sink patterns for each reference frame type across  the sentences tested (e.g. what tokens end up as attention sinks across all 500 sentences and what percentage of specialized heads focus on them, how many attention sink tokens per sentence, how many specialized heads etc) . I'm particularly interested to see how distributed among different tokens the \"distributed reference frame\" really is.\n- How were the thresholds for what constitutes an attention sink (Section 3) set?", "limitations": "The authors include a brief limitations section. An extra limitation that readers find helpful is that the KL analysis on ablated sink tokens may not translate to impact on downstream tasks.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths \n\n- The geometric analysis of attention sinks in transformer models presents a novel framing of these sinks as sites for an emergent coordinate system. Providing insight into what is going on at these tokens and why they emerge.\n- The experimental setup used is thorough, while the domain of sentences used to is somewhat narrow (i.e. just *stem-oriented* sentences), multiple architectures and models are tested in a comprehensive manner.\n- The conceptual framing of the paper is well presented (Sections 1-3) and the optimality perspective is interesting.\n- The authors bring to bear a rich set of analytic tools such as persistent homology, spectral graph metrics, KL/Fisher information to investigate their hypotheses.  \n\n\n### Weaknesses\n- **Sections 4-5 seems to be overly terse and a reads a bit like an information dump.** The authors introduce numerous analytic tools (persistent homology, Betti numbers, Ripser algorithm, persistence values, spectral properties, algebraic connectivity, star likeness, Gini coefficient, degree centralization, information theoretic measures, Fisher information matrix etc.) in a short span without guiding the reader to what question each tool is geared towards answering. It might be more helpful if these were presented in sections answering particular research questions / showing evidence for particular hypotheses, or at least some schematic of tool to information inferred.\n- **Metric values are not contextualized for readers:** in the various result tables there are many different metrics being presented without much contextualization of why these particular numbers imply the conclusions drawn. For example, Table 1 reports Betti₁ values but does not explain what constitutes a “high” vs. “low” Betti₁ or why changes across layers are meaningful.\n- **Some simple baseline analyses are missing:** \n\t- While it is natural that some background in the subfields used for analysis would be needed to fully understand the results, it would be helpful to start with some simpler statistical analyses that demonstrate evidence for the reference types found and then extend to the more in-depth analytic methods (potentially leaving some entirely for the appendix given the space constraints).\n\t- For example an initial visualization/table of the distribution of attention with respect to the sinks for each reference frame type across all the sentences tested (e.g. percentage of attention on sinks in specialized and possibly non-specialized heads, number of sinks per sentence). This is summarized in the \"Token specialization\" line in Table 1. But I would have expected to have seen that broken out in a bit more detail as initial evidence for the attention patterns described in prior sections.", "was_revised": true, "final_justification": "I think this is a quite interesting paper providing a lens on the attention sink phenomenon from a number of different analytical angles while providing a theoretical motivation for why these sinks appear. While the analysis is dense I think it gives a path for future work to build upon its findings to, for example, explore the necessity or relative advantages/disadvantage of attention sink formation as an architectural component.\n\nWith regards to the rebuttal, my questions were primarily around clarity of presentation and elaboration of a few definitions in the paper and the authors have responded adequately.", "rebuttal": "We thank the reviewer for the accurate review.\n\n$\\textbf{Regarding weakness 1 and 3: info dump and missing simple baselines}$\n\nWe agree with the reviewer's opinion. In our effort to be comprehensive, the presentation in sections 4 and 5 became overly dense. \n\nOur Plan for Revision:\n\nWe will revise Section 4 (Methodology) to explicitly frame our analysis around a series of research questions. This will guide the reader and justify why each tool was necessary, transforming it from an \"information dump\" into a clear investigative path. The structure will be:\n\n- RQ1: What is the global shape of the attention graph? -> Tool: Topological Analysis (Persistent Homology, Betti numbers).\n- RQ2: How is information concentrated within that shape? -> Tool: Spectral Graph Theory (Fiedler value, centralization).\n- RQ3: How functionally critical are these structures? -> Tool: Information Geometry (KL Divergence, Fisher Information).\n- RQ4: How do these structures orient representations in value space? -> Tool: Value Space Analysis (Directional influence, etc.).\n\nAs suggested, we will add a new subsection in Appendix (due to space constraints) dedicated to a more detailed, foundational analysis of attention sink patterns. This new section will include the requested visualizations, showing the distribution of attention sinks for each reference frame type across our test dataset. \n\n\n$\\textbf{Regarding weakness 2: metric values are not contextualized for readers.}$\n\nThis is a fair and important criticism. We thank the reviewer for pointing this out. To stick to page limits, the tables in the main body of the paper present a high-level summary of our findings. We would like to note that the complete, detailed tables for all experiments are provided in the appendix for full transparency.\n\nThat said, we agree that the summarized values in the main text must be better contextualized to be interpretable. We will revise Section 5 to ensure that for each value presented, we explicitly explain its significance, for example what constitutes a \"high\" versus \"low\" value and why the observed numbers support our conclusions.\n\nExample for Table 1: When discussing $Betti_1$ values, we will add a clarification: \"A $Betti_1$ value near zero, as seen in Llama-3.2, indicates a tree-like or star-like graph with very few cycles, which is the topological signature of a centralized structure. In contrast, the significantly higher initial Betti₁ of 19.69 for XLM-RoBERTa indicates a highly complex graph with numerous loops, a distinctive feature of the initial layer connectivity in bidirectional frames.\" We will apply this principle to all key metrics in the main text.\n\n$\\textbf{Regarding Question 1: what do you mean by \"optimal\"? - definition of a \"specialized head\"}$\n\nBy \"optimal,\" we do not mean a globally optimal solution in a mathematical sense, but rather an emergent optimum found by the model during training via gradient descent.\n\nThe objective being optimized is the standard cross-entropy loss, which, as we argue in Section 3.2 (citing Tishby et al., 2000), can be viewed through the lens of the information bottleneck principle (Equation 7). The \"optimality\" of reference frames arises from the interplay of two things:\n\n- The softmax constraint: This pushes attention distributions towards sparsity on the probability simplex to efficiently compress information.\n- The positional encoding bias: This creates a specific \"loss landscape\" where certain tokens are computationally cheaper to use as anchors.\n\nA reference frame is \"optimal\" in the sense that it is the most efficient strategy the model discovers to satisfy the need for a stable, sparse coordinate system (from the softmax) within the specific geometric constraints imposed by its architecture (the position encoding). We will revise the abstract and introduction to state this more precisely.\n\nA head is defined as \"specialized\" if it has learnt to consistently form attention sinks across our dataset. So, we classify a head as specialized if, for a high percentage of the test samples (for example, >80%), it produces an attention pattern that meets the $\\textsf{sink}(j)$ definition from Equation 1. This ensures we are analyzing heads that have adopted a stable, structural role rather than ones that only occasionally exhibit sink-like behavior.\n\n$\\textbf{Regarding question 2: meaning of \"Token Specialization\".}$\n\nThis metric in Table 1 describes the behavior of the specialized heads we identified.\n\n- \"100% on BOS token\" for a centralized frame means that every single one of the specialized heads in that model uses the Beginning-of-Sequence token as its primary, consistent attention sink.\n\n- \"65.4% on ','\" for the distributed frame means that among that model's specialized heads, the comma token is the most common anchor, accounting for 65.4% of their observed sink behavior. The remaining percentage is distributed among other tokens that serve as secondary anchors, such as articles (\"the\", \"a\") or periods.\n\nIt's interesting to note that in the second case the \"special tokens\" are function-dependent, not position-dependent. A comma acts as a reference point because of its syntactic role, regardless of where it appears in the sentence.\n\nWe are gonna add a proper explanation regarding this point in the paper.\n\n$\\textbf{Regarding question 3: extra charts for attention sink distribution.}$\n\nWe agree that providing these visualizations would greatly strengthen the paper and make the concept of a distributed frame much clearer. We plan to add a new section (probably in the appendix) dedicated to this. It will include charts showing the distribution of tokens that become sinks for each reference frame type across our dataset. \n\n$\\textbf{Regarding question 4: how thresholds were set.}$\n\nThe thresholds $\\tau$ (attention weight) and $\\gamma$ (frequency) in Equation 1 were set empirically to be conservative and robust.\n\n- $\\tau$ (90th percentile): This was chosen as a standard statistical convention to isolate only the most extreme, outlier attention weights, ensuring we focus on truly disproportionate attention.\n\n- $\\gamma$ (0.3-0.5): This frequency threshold ensures that we only identify tokens that are consistent attractors across a large portion of the dataset, filtering out anomalies that might appear in only a few samples.\n\nOur method identifies tokens that receive a high absolute quantum of attention, regardless of rank. A token could be the #1 attended-to token for every query, but if the attention distribution is very flat, its absolute weight might never cross the 90th percentile threshold. Conversely, a token like [BOS] might consistently receive 30% of the attention from every other token. It would always be highly ranked, but more importantly, it consumes a disproportionate, absolute share of the probability mass.\nThe idea is that attention sinks function as reference points in a coordinate system. For this, we need to find the \"gravitational wells\" of the attention map, like the points that command a large, absolute portion of the attention budget from across the sequence. Our definition, by focusing on outlier absolute weights, is designed to locate these anchors."}, {"review_id": "zq7iPsja37", "reviewer": "Reviewer_JUDb", "summary": "The paper presents a geometric interpretation of the attention sink phenomenon in transformer models, arguing that attention sinks are not merely architectural artifacts but manifestations of fundamental geometric principles related to the establishment of reference frames in high-dimensional spaces. The authors identify three distinct types of reference frames—centralized, distributed, and bidirectional—that correlate with the attention sink phenomenon and emerge as optimal solutions for establishing stable coordinate systems during training. They analyze how architectural components, particularly position encoding implementations, influence the specific type of reference frame that emerges. The paper's contributions include unifying diverse attention patterns across architectures under a single geometric principle, providing insights into how architectural choices affect attention organization, and laying the foundation for deliberate reference frame engineering in transformer models.", "questions": "1. You have provided extensive analysis on several model families (LLaMA, Qwen, BERT variants, etc.) at various parameter scales. However, most analyses focused on models up to ~12B parameters. Could you discuss whether and how your findings (e.g., centralized, distributed, bidirectional reference frames) scale to significantly larger models (70B+ parameters, or GPT-class models)?\n2. Your analysis provides strong theoretical and empirical insights into attention sinks. Can you elaborate more concretely on how these insights might directly inform architectural choices or training strategies in practice?\n3. Your paper utilizes advanced methodologies (persistent homology, spectral graph theory, information geometry) extensively. Could you clarify why each analysis was necessary to support your core conclusions? Could any of these analyses be considered redundant or confirmatory?\n4. You briefly suggest in your conclusion the intriguing idea of \"reference frame engineering\" using attention sinks. Can you expand more explicitly on potential methods or experiments for deliberately manipulating attention sinks to achieve desired model behaviors?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The submission is technically robust and thorough. It provides a well-founded theoretical explanation for attention sinks, rooted in geometry and information theory. \n2. The paper’s empirical analysis is impressively comprehensive: it spans multiple model families and sizes, and uses complementary analysis techniques to validate each claim. \n3. The submission is generally well-written and organized. \n4. This work advances the mechanistic understanding of transformers by reframing attention sinks as necessary geometric structures rather than model artifacts.\n\nWeaknesses:\n1. One minor concern is that some of the analysis, while enlightening, is quite complex (e.g. persistent homology on attention graphs, information geometry formulations). This raises the question of whether all these analyses were necessary to support the core claims or if a subset would suffice – however, the inclusion of multiple perspectives does strengthen confidence in the findings.\n2. Although the analysis covers various transformer models and scales (up to ~12B parameters), generalizability to significantly larger models (e.g., 70B+ parameters) remains unclear. Clarifying theoretical expectations or providing additional empirical insights regarding larger-scale models would further strengthen the paper’s impact.\n3. The contribution primarily centers on theoretical understanding and conceptual insights rather than immediate practical improvements to transformer architectures or their performance metrics.\n4. The paper’s density of content is quite high. Readers who are not familiar with certain fields (e.g., topological data analysis or information geometry) might find parts of the presentation difficult to follow. A bit more gentle explanation or an appendix tutorial could help a broader audience.", "was_revised": true, "final_justification": "Resolved Issues:\n1. Complexity and Necessity of Analyses: The authors clarified that the use of diverse analytical tools (persistent homology, spectral graph theory, information geometry, etc.) was not redundant but complementary, each answering a specific research question. They committed to revising the methodology section to clearly map each method to its respective insight, which significantly improves accessibility.\n2. Lack of Practical Implications: The rebuttal provides specific pathways for using reference frame insights in architectural decisions.\n3. Scalability to Larger Models: While empirical validation on >12B models is missing due to compute limits, the authors provide a clear theoretical rationale for scale-invariant behavior of reference frames, based on architectural primitives (e.g., softmax constraints, RoPE properties). They acknowledge this limitation and appropriately classify scalability as a testable future hypothesis rather than an overclaim.\nRemaining (Minor) Concerns:\n1. The lack of large-scale empirical validation (e.g., 70B+ models) limits the immediate practical generalizability, though theoretical support is persuasive.\nDespite a few open questions (e.g., scale-up validation), the paper is methodologically sound, conceptually rich, and forward-looking. The rebuttal was detailed, thoughtful, and addressed concerns directly. Given the novelty, depth, and clarity of response, I believe this paper makes a meaningful and lasting contribution to understanding transformer architectures and deserves a spot at NeurIPS.", "rebuttal": "We thank the reviewer for the accurate review.\n\n$\\textbf{Regarding weakness 1: complexity and necessity of the analyses}$\n\nOur goal was to show that the evidence for reference frames is not an artifact of one particular analytical lens, but a consistent signal across topological, spectral, information-theoretic, and vector-space perspectives. In our revision, as detailed in our answer to Question 3, we will improve the methodology section to more clearly motivate why each tool was necessary to answer a specific research question, thereby guiding the reader through our analytical narrative. \n\n\n$\\textbf{Regarding weakness 2: scalability to larger models}$\n\nOur analysis was computationally constrained to models up to ~12B parameters. However, as we elaborate in our answer to Question 1, our core theoretical argument is based on architectural \"primitives\" (softmax and position encodings) that are scale-invariant. We therefore hypothesize that our findings will generalize, and perhaps even become more pronounced, in larger models. We will explicitly state this in the limitation section.\n\n$\\textbf{Regarding weakness 3: theoretical vs. practical contributions}$\n\n Our paper moves the conversation about attention sinks from \"bug\" to \"feature,\" opening up new avenues for deliberate architectural design and training strategies. We outline several concrete examples of how this theoretical insight can translate into practical applications in our answers to Question 2 and Question 4.\n\n$\\textbf{Regarding weakness 4: the density of the presentation}$\n\nWe agree that the paper's density could be a barrier. We will add more explanatory text to contextualize the results in our tables (as discussed in our response to Reviewer uQoB) and restructure Section 4 to better motivate our methodology. We will also add a section to the appendix to provide more background on the key concepts from topological data analysis and information geometry.\n\n$\\textbf{Regarding question 1: scaling on bigger models}$\n\nUnfortunately for this project we didn’t have the resources to test on models of that size. But, while our direct empirical analysis was computationally constrained to models up to ~12B parameters, we have theoretical reasons to believe our findings will generalize to much larger models.\n\n- Our core thesis is that reference frames are not an emergent property of scale, but a fundamental consequence of architectural primitives. The geometric principles we identify, the simplex constraint of softmax and the mathematical properties of position encodings, are intrinsic to the architecture, regardless of parameter count. A 70B LLaMA-family model still uses standard RoPE, which mathematically privileges the first token. Therefore, we hypothesize it will continue to exhibit a centralized reference frame. In fact, as model capacity and the complexity of the learned representations increase, the need for a stable, efficient coordinate system to anchor the high-dimensional manifold likely becomes more critical, not less.\n- Empirical observations from the community regarding larger models appear to support this. For instance, strong, persistent attention to the [BOS] token is a widely noted phenomenon in very large decoder-only models. Our framework provides a formal explanation for these observations.\n\nSo, we acknowledge that direct empirical validation on 70B+ models is an important next step. We will add to the paper that while our theory predicts scalability, this remains a hypothesis to be confirmed by future work with greater computational resources.\n\n$\\textbf{Regarding question 2: how the knoledge about the reference frame could be used in practice}$\n\nOur findings suggest that the choice of position encoding is, in effect, a choice of geometric inductive bias. This allows for more principled architecture design. For example:\n- For tasks requiring robust long-range dependencies anchored to a consistent context (e.g., summarization, few-shot prompting), an architecture promoting a centralized frame (like standard RoPE) may be optimal due to its efficient, hub-and-spoke information routing.\n- For tasks requiring high contextual flexibility and sensitivity to local syntax (e.g., creative writing, complex reasoning), an architecture promoting a distributed frame (like NTK-scaled RoPE) might be better, as it creates a more adaptable, locally-aware coordinate system. This allows designers to match the geometric properties of the model to the geometric properties of the task.\n\nKnowing that models dedicate significant capacity to forming reference frames (as shown by our Fisher Information analysis), we could devise strategies to accelerate this process. For example, a pre-training or warm-up phase could use a curriculum or a specific regularization term to explicitly encourage the formation of a stable reference frame, potentially leading to faster convergence on the primary task.\n\n$\\textbf{Regarding question 3: is the analysis redundant?}$\n\nWe view our analysis not as redundant, but as providing a form of cross-validation, where each method offers a unique and necessary piece of the puzzle. The complexity arises from the need to build a robust case from different angles.\n\nHere is a breakdown of why each component was essential:\n\n- Topological Analysis (Persistent Homology): This was necessary to understand the high-level shape of the attention graph. It answered the question: \"What is the global connectivity structure?\" For example, the discovery of high $Betti_1$ values (loops) in early layers was a unique signature that immediately distinguished bidirectional frames from the star-like ($Betti_0 \\approx$ 0) structures of decoders. \n- Spectral Graph Theory: This analysis quantified the properties of the connectivity structures revealed by topology. It answered: \"How is information concentrated and distributed within that shape?\". Metrics like the Fiedler value and degree centralization allowed us to quantitatively differentiate between centralized (single dominant node) and distributed (multiple smaller nodes) frames. The sign-flipping correlation pattern we identified became a key quantitative signature.\n- Information Geometry (KL & Fisher): This moved beyond structure to function. It answered: \"How critical are these reference points for the model's information processing?\" The KL reduction experiment directly measured the functional cost of removing sinks, proving their importance. The Fisher information analysis revealed where in the network (i.e., which layers) the model invests its learning capacity to build these frames, highlighting the early-layer focus of centralized frames versus the mid-layer focus of bidirectional ones.\n- Value Space Analysis: This final piece connected attention patterns to their ultimate purpose: geometric transformation. It answered: \"How do these reference points actually orient other tokens in the value space?\" The directional influence metric was crucial here, showing that reference tokens act as coordinate axes that actively guide other representations, rather than just passively aggregating information.\n- Random Matrix Theory Analysis (in the Appendix): it clarified that the reference frames emerge in the first stages of the training, showing that the reference frames aren't just an artifact, but a useful feature to achieve proper processing. \n\nWe will, however, take the reviewer's feedback to heart and revise the methodology section (section 4) to more clearly state the unique question each analysis answers, and better justifying the inclusion of each and guiding the reader through the logic.\n\n$\\textbf{Regarding question 4: potential methods or experiments for \"reference frame engineering\"}$\n\nBuilding on our answer to Question 2, we can envision several concrete experimental ideas:\n- Explicit anchor tokens: it could be possible to introduce new, non-semantic \"anchor\" tokens into the vocabulary. During training, a regularization term could be added to the loss function to explicitly encourage (or discourage) attention to these tokens, effectively creating programmable reference points. An experiment could test if a model can be trained to use [ANCHOR_1] as a centralized sink and [ANCHOR_2] as a secondary, distributed sink.\n- Attention head specialization: Our analysis shows that specific heads specialize in attending to reference points. it could be possible to perform experiments where, during fine-tuning, these specialized heads are either frozen (to preserve the original geometric structure) or selectively pruned/retrained (to force the model to adopt a new geometric strategy). This could be a powerful tool for efficient task adaptation.\n\nOnce again, we thank the reviewer for their deep engagement with our work. Their feedback has been valuable in helping us clarify our contributions and strengthen the paper."}, {"review_id": "1bGJImH6X8", "reviewer": "Reviewer_ATpd", "summary": "This paper studies transformer \"attention sinks\" as geometric reference frames that establish coordinate systems in representation space. The authors identify three types: centralized (single dominant anchor), distributed (multiple weaker anchors), and bidirectional (dual anchors), each determined by position encoding choices. Using topological, spectral, and information-theoretic analysis across multiple transformer architectures, they show these reference frames emerge naturally during training as optimal solutions to geometric constraints, serving as essential infrastructure for stable token relationships rather than inefficiencies.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nI'm not an expert on attention sink and related topics, but I found this explanation very interesting and novel. Understanding the attention in LLM could be very important to the field. \n\nThe paper is clear about the assumption, experiment and analysis, making it clear about the general idea without knowing the theoretical details. \n\nIn the experiment, the authors run a comprehensive cross-architecture analysis. The breadth of architectures tested demonstrates generalizability across different model families and scales.\n\nWeakness\n\nWhile the paper shows strong correlations between position encoding types and reference frame emergence, the authors don't perform controlled experiments changing only position encoding while keeping other architectural elements constant to definitively prove causation. Also, it lacks a fundamental theoretical explanation for why these specific patterns emerge.", "was_revised": true, "final_justification": "I'm not an expert of this topic, but I'd like to maintain my positive score.", "rebuttal": "We thank the reviewer for the review.\n\n$\\textbf{Regarding weakness 1: lack of experiments to prove causation}$\n\nWe agree that a perfectly controlled experiment, training multiple large-scale models from scratch where only the position encoding scheme is varied, would provide the most definitive proof of causation. However, the huge computational cost is beyond the resources available for this work.\n\nInstead, we sought to build a strong case for a causal link through two complementary lines of evidence:\n\n- Consistency across many architectures: our study was designed to show that the link between position encoding and reference frame type is not an artifact of a single model but a consistent principle. We demonstrate that all analyzed models using standard RoPE (LLaMA 3.1/3.2, Mistral, Gemma) develop centralized frames, all models with NTK-aware scaled RoPE (Qwen 2.5, Phi-2) develop distributed frames, and all bidirectional encoder models with absolute embeddings (BERT, XLM-RoBERTa) develop bidirectional frames. The consistency of this pattern across different model families, sizes, and training datasets provides robust correlational evidence that strongly suggests the position encoding scheme is the primary causal factor.\n- A mechanistic explanation: our paper provides a direct mathematical explanation for why this causal link exists. In Section 3.4, we explain the underlying mechanism for each encoding type. For instance, we explain that standard RoPE's formulation results in an identity matrix transformation for the first token ($R_0 = I$), giving it a unique, privileged computational status that naturally leads to its emergence as a centralized reference point (lines 167-169). Similarly, we explain how the scaling factor in NTK-aware RoPE diminishes this advantage, creating the conditions for distributed frames (lines 181-184).\n\nWhile we did not perform the interventional experiment suggested, we believe our combination of consistent, large-scale observational evidence and a clear, mathematically-grounded causal mechanism provides a compelling argument for the claims made. We will, however, add a note to the limitations section acknowledging that this type of controlled experiment would be a valuable direction for future research.\n\n\n$\\textbf{Regarding weakness 2: lack of theoretical explanation on the emergence of the phenomenon}$\n\nWe respectfully believe that our paper does propose a fundamental theoretical explanation, which is rooted in the interaction between two core principles of the transformer architecture. Perhaps we can state it more directly here and ensure it is properly written in the final version.\n\nOur theoretical argument is that reference frames (and thus attention sinks) emerge as an optimal solution to a geometric problem created by the interplay of:\n- The information geometry of the softmax function: as explained in section 3.2 (and further in Appendix A), the softmax function constrains attention outputs to the probability simplex ($\\Delta^{n-1}$). We argue that this is not just a normalization step but a fundamental geometric constraint. This constraint, governed by the Fisher information metric, creates a curved representational manifold that privileges sparse distributions, that is, it creates a mathematical pressure to concentrate attention probability mass onto a small number of tokens. This is the general force that necessitates the creation of \"sinks.\"\n- The inductive bias of position encodings: while the softmax creates the pressure to form sinks, the specific mathematical structure of the position encoding scheme provides the inductive bias that determines the pattern of these sinks. Section 3.4 is dedicated to this point, providing a detailed theoretical account for each reference frame type:\n1. Centralized frames emerge because standard RoPE's math gives a unique computational advantage to the first token.\n2. Distributed frames emerge because scaled RoPE modifies this math to weaken the first token's advantage, allowing other tokens to serve as local anchors.\n3. Bidirectional frames emerge because absolute position embeddings explicitly inject position markers that the encoder architecture can leverage as stable start and end anchors.\n\nSo, our theory says that attention sinks are not an accident, but a necessary geometric feature. The softmax creates the \"reason\" (a need for sparse, stable reference points on the probability simplex), and the position encoding provides the \"way\" (the specific geometric biases that shape the final structure). \n\nWe will revise the introduction and abstract to make this two-part theoretical argument more explicit."}, {"review_id": "rMNmpRiW14", "reviewer": "Reviewer_idkt", "summary": "This paper presents a framework for studying *attention sinks*, and often discussed but rarely\nformally defined and studied property in attention-based models in which tokens corresponding to\nseemingly neutral or irrelevant information attract disproportionate attention.\n\nIt studies reasons why sinks emerge, formally defines sink tokens, and classifies different\nbehaviors in \"reference frames\", which are claimed to occur as a result of learning\nhigh-dimensionality latent spaces. Choices in the architecture such as the use of positional\ninformation, and the specific type of positional info result in different reference frame behaviors\n(centralized, bi-directional, distributed).\n\nThis framework and these findings are a great first step towards understanding attention sinks, and\nto my knowledge have rarely been investigated in prior works. Sinks present a wide range of\nopportunities and challenges in foundational models, from potential pruning methods for efficiency,\nto efforts in robustness and more effective quantization.", "questions": "1. Section 3.3 suggests that the emergence of sinks is directly related to positional embeddings /\ninformation. Does that mean that models without positional biases / embeddings do not form sinks?\nAppendix A suggests that the softmax probability distribution mapping is one of the factors behind\nforming of sinks, so perhaps positional information are what bias the emergence of sinks to earlier\ntokens?\nCould we say that sinks emerge due to softmax and positional information? This is a more interesting\nfinding (assuming it's true), and I think it belongs in the abstract, where it currently says sinks\nare not an architectural phenomena (more on this in the next question).\n\n2. There are many instances in the paper that claim sinks are not an architectural phenomena.\nHowever, given that it only occurs in attention, I could argue that only architectures with\nattention (primarily transformers) exhibit this behavior. This is also discussed in the paper. For\nexample lines 27-30:\n\n> While prior work [...] treated attention sinks as model-specific phenomena, our geometric\n> interpretation unifies these observations as alternative solutions to the same fundamental\n> challenge, advancing understanding of transformer geometry.\n\nIn the end, it's all about transformers / NNs with attention. Therefore I recommend changing the\nlanguage to clarify this.\n\n\n3. I may be misunderstanding this, so maybe you can point me in the right direction. Line 56 says\nthat the study by Moschella et al. shows that NNs with high dimensional latent spaces naturally\nestablish reference frames. However, after going over the paper, based on what I understand they're\nproposing establishing a data-dependent coordinate system as to make it easier to compare latent\nspaces, among other things.\nCan you clarify this? If possible point out where Moschella et al. discusses the emergence of\nreference frames?\n\n4. Line 17:\n> Removing this pattern degrades model performance\n\nCould you please add references? I'm personally not convinced that eliminating sinks is necessarily\nbad for performance, as there are many different approaches that eliminate (or mitigate) the effect.\nOf course it's obvious that eliminating the tokens entirely post-training will affect quality, but\nit's not clear to me why designing a reasonable architecture that is not prone to this would be\ninferior in performance.\n\n5. I have a few issues with the definition of sinks (Sec. 3, Eq. 1). I understand that this may be a\nwidely accepted definition, but I still take issue with it, because it is too dependent on those\nthresholds, specifically $\\tau$. Why is the value of the 90th percentile a good choice? Is there a\nstudy on false positives?\nIf sinks are considered \"context tokens that attract disproportionate attention from most or all\nquery tokens\", then would it not make more sense to rank the attention weights for a single query\ntoken, and instead look for context tokens that appear in the top P%?\n\nI don't think this paper needs to discuss this, but I think it might be useful to point out that\nthe definition of sinks in Eq. 1 may not be the only one or the most reliable one, or if you believe\nit is, it would help to cite prior works using this definition or briefly discuss why.\n\n6. Can you comment on approaches such as Softpick or other similar modifications to the softmax\nfunction and what could be their effect on the emergence of sinks?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Formal definition of attention sinks, with reasonable arguments as to why they occur. This has\n  been a rarity in papers discussing sinks directly in my experience.\n\n2. Investigation of the effects of different types of positional information / embeddings on the\n  types of reference frames and therefore sink behavior.\n\n3. Well-written and clearly organized, and relatively easy to ready and understand.\n\n### Weaknesses\n1. The major limitation in my opinion is a lack of background references. It would certainly be\n  helpful to readers in other research areas (including myself) to read prior works.\n  Geometric properties such as reference frames could use more extensive definitions, or references\n  for those unfamiliar (again including myself).\n\n2. As pointed out in the paper as well, the study is limited to only certain checkpoints of the\nmodels being studied. It would be interesting to see this study conducted throughout training. It\nwould also be interesting to find out whether much smaller models trained with limited resources\nexhibit similar sink behavior, and that way the study could be done more efficiently.\n\n3. (Minor) Experiments are limited to various LLMs, and not other foundation models in the visual\n  or audio space. While I think having more breadth would certainly help, I don't think it's a\n  necessity. More specifically, I think the findings affect those other applications as well, and\n  not just LLMs, so if at all possible, I recommend looking into other foundation models beyond\n  language.\n\n\n### Minor Notes:\n\n1. Multiple citations appear to be out of date or referencing the preprint instead.\n  Moschella et al. \"Relative representations enable zero-shot latent space communication.\" was published in ICLR 2023.\n  Xiao et al. \"Efficient streaming language models with attention sinks.\" was published in ICLR 2024.\n\n2. Other works prior to Xiao et al. (2023) have also discussed the behavior of sinks without using\n  the term:\n  * Clark et al. \"What does BERT look at? An Analysis of BERT’s Attention.\": https://aclanthology.org/W19-4828.pdf.\n  * Kovaleva et al. \"Revealing the Dark Secrets of BERT.\": https://arxiv.org/abs/1908.08593.\n\n3. In Figure 1, could you change the color or just the visualization of connections for Centralized\n  and Distributed, since they are not layer-specialized? It can be confusing as the straight line is\n  labeled \"Early layers connection\" in the legend. Or is that intentional?", "was_revised": true, "final_justification": "All of my concerns were addresses by the authors, and as long as the changes that were requested by me and other reviewers are integrated, I recommend the acceptance of this paper.", "rebuttal": "We thank the reviewer for the accurate review.\n\n$\\textbf{Regarding the \"minor notes\":}$\n\n- $\\textsf{Out of date citations}$: Thank you for catching this. We will update all citations, including Moschella et al. (2023) and Xiao et al. (2024), to their correct, published versions.\n- $\\textsf{Prior work on sink-like behavior}$: While Xiao et al. coined the term \"attention sink,\" the underlying behavior was indeed observed in earlier foundational work on BERT interpretability. We will absolutely add citations to Clark et al. (2019) and Kovaleva et al. (2019) in our related works section to provide a more complete historical context.\n- $\\textsf{Figure 1 visualization}$: Thank you for pointing out this potential confusion. The visualization was intended to show that connections in Centralized and Distributed frames are not layer-specific, unlike the Bidirectional frame. We will revise the figure and its legend to make this distinction clearer.\n\n\n$\\textbf{Regarding weakness 1: lack of background references}$\n\nWe will add a new section in the Appendix that provides an introduction on the key concepts from information geometry and topological data analysis.\n\n\n$\\textbf{Regarding weakness 1: study on static checkpoints}$\n\nThis is a key limitation that we acknowledge. A full analysis throughout the entire training process is computationally prohibitive for models of this scale. Your suggestion to study smaller models is very good and our Random Matrix Theory analysis on Pythia checkpoints (in Appendix B, due to space constraint) is a first step in this direction, showing that reference frames emerge in the earliest stages of training. We will highlight this finding more prominently and explicitly state in our limitations section that a more granular temporal study, perhaps on smaller, more efficiently trainable models, is a crucial direction for future work.\n\n\n$\\textbf{Regarding weakness 1: limited Scope (LLMs only)}$\n\nOur work focuses on LLMs as this is the primary domain where attention sinks have been formally studied. However, we agree that our geometric framework is general and likely applicable to any Transformer-based architecture. We thought that this paper was rather dense in information already, so we decided to focus on text and save the other modalities for a possible future work.\n\n\n$\\textbf{Regarding question 1 and 2: causality (Softmax + Positional Embeddings) and \"not an architectural phenomenon\"}$\n\nWe agree our wording could be more precise.\n\n- Causality: You are absolutely correct. Our central argument is that sinks emerge due to the interplay between the softmax function and the inductive biases of position encoding. The softmax creates the geometric pressure for sparsity on the probability simplex (as discussed in Appendix A), and the position encoding provides the specific bias that determines where these sparse points (sinks) will form. This is indeed a core finding, and we will revise the abstract and introduction to state this two-part mechanism more explicitly.\n- \"not an architectural phenomenon\": Your critique of this sentence is fair, when we wrote this, we meant that sinks are not a bug or an arbitrary quirk specific to one model implementation (e.g., LLaMA vs. Mistral). Rather, they are a fundamental, predictable consequence of the core components of the Transformer architecture itself. In that sense, they are an architectural phenomenon. We will revise our language to be more precise, stating that reference frames are a fundamental geometric principle within the Transformer architecture, emerging predictably from the interaction of its core components. Thank you for pushing for this clarification.\n\n$\\textbf{Regarding question 3: interpretation of Moschella et al. (2023)\"}$\n\nThank you for this very careful reading. Moschella et al. propose using relative representations to establish a shared coordinate system, they do not claim that such frames emerge naturally in standard training. Our intention was to cite their work as providing the conceptual inspiration for why reference frames are such a powerful and necessary structure in high-dimensional spaces, which motivated our search for their emergent counterparts. We will revise line 56 and related text to state that Moschella et al. demonstrated the utility and proposed the creation of such frames, which provided a key motivation for our work.\n\n$\\textbf{Regarding question 3: reference for performance degradation\"}$\n\nThe primary reference for performance degradation upon removing the initial tokens (the attention sink) is the original paper by Xiao et al. (2024), \"Efficient Streaming Language Models with Attention Sinks\". They demonstrate that for streaming applications, where a fixed-size cache is used, preserving the attention sink tokens in the cache is crucial for maintaining model performance. We will add this citation explicitly at line 17.\nWe also agree with your nuanced view: our claim is not that any architecture designed without sinks is inferior. Rather, simply removing the sink tokens post-hoc from a model that was trained to rely on them is detrimental. Designing an architecture from the ground up that accomplishes the same geometric stabilization without a sink is an exciting research direction. We will clarify this important distinction in the text.\n\n$\\textbf{Regarding question 3: definition of sinks\"}$\n\n$\\textsf{On the use of thresholds and the 90th percentile}$\n\nWe acknowledge the reviewer's concern that any threshold-based definition carries a degree of arbitrariness. Our approach was guided by two principles: aligning with the conceptual understanding of the phenomenon and choosing a conservative measure to ensure the robustness of our findings.\n\n- The term \"attention sink\" implies a token that attracts a disproportionately large share of the total attention budget. Our definition aims to capture this by looking for attention weights ($\\alpha_{ij}$) that are outliers in the global distribution. The 90th percentile for $\\tau$ was chosen as a standard statistical convention to isolate these extreme values, the top 10% of all attention links. This ensures we are not just flagging moderately high attention, but truly exceptional concentrations of it.\n- The role of the frequency threshold ($\\gamma$): We want to emphasize that the $\\tau$ threshold does not operate in isolation. The second threshold, $\\gamma$ (frequency), is critical and directly addresses the reviewer's point about consistency. A token is only identified as a sink if it exceeds the high attention threshold $\\tau$ for a large fraction ($\\gamma \\ge$ 0.3) of all source tokens. This combination ensures that we identify tokens that are not just attended to strongly by one or two queries, but that act as consistent, sequence-wide attractors. While we did not conduct a formal false-positive study, this dual-threshold requirement is designed to be highly resistant to false positives; a token must be both an outlier in attention weight and highly consistent in its role as an attractor.\n\n$\\textsf{On the proposed alternative definition}$\n\nThe reviewer's suggestion to rank attention weights for each query and look for tokens that consistently appear in the top P% is an excellent and valid method for identifying tokens that are consistently important to other tokens. We believe this method would be highly effective for identifying stable semantic or syntactic relationships. However, we chose our definition because it is tailored to identify a slightly different phenomenon that is central to our paper's thesis: the existence of universal geometric anchors. The reviewer's method identifies tokens that are consistently high-rank. For example, for the query \"cat,\" the token \"mouse\" might consistently be in the top 5% of attended-to tokens. This captures relative importance.\n\nOur method identifies tokens that receive a high absolute quantum of attention, regardless of rank. A token could be the #1 attended-to token for every query, but if the attention distribution is very flat, its absolute weight might never cross the 90th percentile threshold. Conversely, a token like [BOS] might consistently receive 30% of the attention from every other token. It would always be highly ranked, but more importantly, it consumes a disproportionate, absolute share of the probability mass.\n\nThe idea is that attention sinks function as reference points in a coordinate system. For this, we need to find the \"gravitational wells\" of the attention map, like the points that command a large, absolute portion of the attention budget from across the sequence. Our definition, by focusing on outlier absolute weights, is specifically designed to locate these universal anchors. The reviewer's proposed method, while very good, is better suited for analyzing pairwise token importance.\n\nWe will add a note in the appendix clarifying the rationale for our definition in contrast to other valid alternatives, like the one proposed.\n\n$\\textbf{Regarding question 4: effects of softmax modifications\"}$\n\nStandard softmax creates the pressure for the kind of sparsity we observe in classic attention sinks. Modifying this function would not remove the fundamental need for a reference frame, but would change its implementation.\nApproaches like Softpick or sparse-softmax enforce a \"harder\" sparsity, often forcing the model to select a small, discrete set of tokens to attend to. This would prevent the formation of a \"lazy\" sink where one token passively collects a soft distribution of attention from all others. Instead, we hypothesize that these methods would force the model to implement a more dynamic reference frame. For any given query, the model would be forced to make a hard choice, selecting a small set of tokens to receive 100% of the attention. These tokens would become explicit, context-dependent reference points for that specific computational step."}]

Author rebuttal:

Meta-review: This paper advances a clear, novel geometric account of attention sinks as emergent reference frames (centralized, distributed, bidirectional) shaped by softmax geometry and positional encodings. Evidence spans multiple architectures and checkpoints (e.g., LLaMA/Qwen/BERT) with complementary analyses (topological, spectral, information-theoretic). Reviewers are positive (two “Accept,” one “Borderline-Accept”), citing strong conceptual contribution and thorough cross-architecture validation; residual concerns (presentation density, absence of fully controlled causality experiments, very-large-scale validation) are primarily expository and scope-related. The rebuttal directly addresses these with concrete camera-ready commitments: restructuring methodology around explicit research questions, contextualizing metrics, adding background/visualizations, clarifying “optimality” and sink definitions, and precisely situating claims (softmax + positional encodings). Given the strength and the reviewers’ post-rebuttal endorsements, the AC decides to accept the submission.
Decision: Accept (spotlight)