The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Authors: Matteo Saponati
Pascal Josef Sager
Pau Vilimelis Aceituno
Thilo Stadelmann
Benjamin F Grewe
Venue: ICML 2025 poster (2025)
Keywords: Transformer models
Self-attention
Deep Learning Theory
Training Dynamics
Self-supervised training
Initialisation techniques
Mechanistic Interpretability
Primary area: deep_learning->attention_mechanisms
TL;DR: We introduce a new theoretical perspective on self-attention matrices, showing that bidirectional and autoregressive training induces symmetric and directional matrices, respectively. We show how to leverage these structures for boosting performance.

Abstract: Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models — including ModernBERT, GPT, LLaMA3, and Mistral — and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models.

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Matteo Saponati"
Pascal Sager 21
Pau Vilimelis Aceituno
Thilo Stadelmann 23 Benjamin Grewe
Abstract
in understanding these models is unraveling the structures of self-attention, which is essential to Transformer archi- tectures. Current literature largely overlooks the nature of the self-attention weight matrices during autoregressive training, where the model predicts the next token se- quence given previous ones (Radford et al.. 2019; Black et al., 2021; Touvron et al., 2023) and bidirectional training, where the model predicts a missing token given the full se- quence (Devlin et al., 2019; Bao et al., 2022; Warner et al., 2024). Understanding self-attention requires answering two fundamental questions: How can we interpret the structures learned in the self-attention matrices? What is the impact of different objective functions on these matrices?
Self-attention is essential to Transformer archi- tectures, yet how information is embedded in the self- attention matrices and how different objec- tive functions impact this process remains unclear. We present mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this frame- work, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models - including ModTmBERT, GPT, LLaMA3, and Mistral - and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on lan- guage tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby im- proving the interpretability of Transformer mod- els.
Previous work used sparse auto-encoders to identify inter- pretable features (Huben et al., 2024; Bricken et al., 2023), circuit analysis to interpret Transformer components (Olah et al., 2020; Elhage et al., 2021; Olah, 2022), and tech- niques like the logit lens to analyze self-attention mecha- nisms (Geva al., 2021; Dar et al., 2023) (for detailed discussion, see Section 5). However, these methods do not reveal the structural patterns self-attention matrices the transformations they encode. Crucially, how autoregressive and bidirectional training shape specific weight structures remains unclear.
1. Introduction
To address this gap, we introduce a novel framework for analyzing self-attention matrices and understanding how different objective functions define their weight updates. We then use this framework derive understandable math- ematical structures that should emerge from such updates. Finally, we verify these interpretable structures numerically on many pre-trained and custom models across different modalities, supporting the universality of our results. Identi- fying these universal structures fundamental not only for improving the performance of Transformer models, but also for their safety, alignment, and interpretability (Olah, 2022).
Transformer models now achieve state-of-the-ard perfor- mance across wide range of tasks and domains (Radford et al., 2019; Dosovitskiy et al., 2021; Radford et al., 2023). Despite their success, the internal mechanisms governing their decision decision-making processes remain poorly understood, raising concerns regarding model alignment, reliability, and safety (Wang et al., 2023; Yao et al., 2024). A key challenge
"Equal contribution IInttitute of Neuroinformatics, ETH Zürich and University of Zürich, Zürich, Switzerland 'Centre for Arti- ficial Intelligence, Zürich University of Applied Sciences, Win- terthur, Switzerland 'ECLT European Centre for Living Technol- ogy, Venice, Italy "ETH AI Center, Zürich, Switzerland. Corre- spondence Mattco Saponati <masapo@@nn tni.ettz.ch>, Benjamin ini. Grewe www.bwewiii..thh.cccom cth...
Specifically, we connect the matrix W qk W W of self self-attention with bilinear forms, offering novel insights compared to studying query and key matrices alone. We reveal structured patterns in the implicit weight updates of Wyk. uncovering key differences between encoder-only and decoder-only models:
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
1.
Decoder-only models: Training with an autoregressive objective produces few columns with disproportion- ately high norms, introducing directionality in W qk.
We observe that X W X corresponds to a bilinear form (see Definition A.2). Specifically, the entry dij = -Aaliy can be formulated in two equívalent ways: (1) as the canonical dot product between a query qi and a key k, (like in stan- dard Transformer models), (2) as the dot product between tokens and x, under the bilinear form W gh:
2. Encoder-only models: Bidirectional optimization in- duces symmetric structures in W reflecting the bal- anced nature of the training objective.
3. We validate these theoretical findings across diverse Transformer architectures and imput modalities, show- ing that they generalize across models and tasks.
Big (q: ky) (xi, W gh xx) = (x,,
4. Empirically, we find that symmetric structures in W enhance training efficiency for encoder-only models, leading to higher accuracy and faster convergence in language tasks.
Intuitively, this equivalence shows that the matrices W, and Wa together define an alternative metric in the embed- ding space, which quantifies the score of x; and x, without explicitly constructing the query and key vectors. Let us consider a specific input token x; E Rdd The self-attention layer maps it an updated token X; RR as follows,
2. Autoregressive and bidirectional training leads to directional and symmetric weight updates
Bas Xj W..
In this section, we introduce novel framework that links the self-attention matrices Was to bilinear forms, enabling to analyze how an objective function influences their structure. This approach reveals fundamental patterns in Wyg that are not apparent when examining Wg and Wx k separately. For example, we prove that autoregressive and bidirectional training induce directional and symmetric structures in the W gk matrices. In the following sections, we define and formalize these concepts.
where the coefficients {āez} in the second term represent the projection of x; onto span (X) (the subspace spanned by X (xx] x ,..., in the transformed embedding space defined by W yk.
Σ Eaz = Xxx, X5.
(4)
Because the softmax function preserves the order of its input values, the ranking of the raw attention scores âij is maintained in the final normalized scores aij.
2.1. Interpreting self-attention with bilinear forms
Self-attention (Vaswani al., 2017; Radford et al., 2019) is a type of score function A R...d x R, + R.N.N that maps sequence of N token embeddings with dimension d into a matrix of attention scores. Except for the row-wise softmax function (.). self-attention is a linear transforma- tion of the embedded tokens. In particular,
Oij < Oigy e bi < dss Vi,
(5)
As a result, the attention weights 0zj define a convex com- bination of the token vectors Xj, meaning the output lies within the convex hull of the input sequence Conv(X) C span [X} (see also Appendix A.2). We note that W. is a linear transformation that is applied independently each projection in the sum and thus does not influence our deriva- tion.
1
A(X)
A(X)
Vd
1
QKT
In the following sections, we demonstrate how this equiva- lent formulation of self-attention provides a useful frame- work for analyzing the training of Transformer models. Specifically, we show that the choice of the objective func- tion such autoregressive prediction (Radford et al., 2019) or bidirectional training (Devlin et al., 2019) produces dis- tinct structural patterns in Wok-
Vd
1
XW qh X"
Vd
where A is the linear part of self-attention (raw un- scaled attention scores), X = [x], xM] E W the sequence of N token embeddings x, Rd, and W p = W, W E Idd This equation shows that the linear transformation Wg and Wt are always combined to compute attention scores with one single matrix Wak- While the matrices W, and Wa are defined separately for computational efficiency, this formulation remains mathe- matically equivalent (see also Elhage et al., 2021; Olsson et al., 2022; Dar et al., 2023).
2.2. Deriving the gradients of self-attention with bilinear forms
To show the connection between the objective function and the structures of self self-attention matrices, we derive a conve- nient formulation for the weight update of W
Wgk- We first for- mulate a sequence modeling problem with self-supervised
2

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
L(U)
Proposition 2.1. (The implicit weight update sum of rank-1 matrices). Let U (fr,..., ......,,] be sequence of N tokens, and let L(U:W) denote the negative log- likelihood of the sequence under a Transformer model with learnable parameters W,
backward
5
Kij
L(U:W)- = Σ ((t))= ((t))=- log log pwlti tp.),
(7)
where Dj S [I....,NN} is the set of indices used condi- tion the prediction of token 1,. Let the self-attention function be defined as in Equation (1). Following the gradient of C(U W). the implicit gradient-batt based update to the bilin- ear weight matrix W qk at the 1-th self-attention layer is proportionally equivalent to:
Cj
Wsk
1. The sum of contributions from each token tj ; jje used to predict t;,
forward
x x;
sequence of embedded tokens
AW
9b
ΣΣAW JEC.
EΣ
X
Litt,
i JEC.
(8)
where Cs denotes the set of context indices for predict- ing ty.
N
AW qh
Figure 1. Illustration of the computation the self-attention score between token and token x, (forward pass, see Equation (3)), and its corresponding contribution to the weight update of Wak (backward pass, Equation (8) and Equation (9)). The symbols "0", "0", and "o" refer to the addition, multiplication, and dot product operations, respectively.
SEP;
2. The sum of contributions from each prediction target t P3 when predicted with tj.
Aw gh x
ΣΣAW' qh
-Σ
iEP;
iEP;
(9)
training as follows. Let U (t1...... (N) be sequence N)
where Pj denotes the set of tokens for which tj is included the context.
of N tokens. For each position i, let D; S denote the conditioning sel, that is, the set of indices cor- responding to the tokens used to predict the token t, This formulation makes possible to isolate and analyze the contribution of each token t, E Dj the prediction of t,. Let LL)) be the negative log-likelihood of each token 4, expressed as,
Here, B15 is scalar that quantifies the contribution of the token embedding Xj to the prediction error for x; at the 1-th layer, and the matrix K!-- E Mi is a rank-1 matrix defined by the outer product of the token embeddings at the previous layer.
N C(U:W)
K 1--1
(10)
Σ C(I.) =
Σ log (o
(6)
We provide proof for this proposition with related remarks in Appendix A.4, and an illustrative description of the forward and backward pass in Figure 1.
where Wi is the set of trainable parameters, and (t: (D.))
are the conditional densities parameterized by the model (sec also Appendix A.3).
2.3. How context and prediction impact the gradient differently
Here, demonstrate that the updates to the matrix W follow structured pattern: the contribution of the token in the prediction of the embedding results in adding rank-1 matrix Kij to the matrix W gh- As result, the total weight update to W is expressed as linear combination of these rank-I I matrices. We formalize this observation in the following proposition.
Next, show how the formulation of AW gk enables the analysis of the contribution of any given token L" to the weight updates and how this affects the properties of W gk- Indeed, Proposition 2.1 indicates that a token t" impacts the updates Wa differently when serving as context for predicting other tokens or being itself predicted.
3

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
When token 1* serves context (1; = ("), the embeddings of all predicted tokens contribute to the column space of W gh, where the update of every k-th column w., is given by
2.4. The relation between objective functions and structures in self-attention matrices
In this final section, we show how to relate these properties to the specific objective function. Crucially, the number of times token t" appears as context or as prediction depends on the training objective. Autoregressive training implicitly introduces directionality by predicting each token solely on its preceding tokens. In contrast, bidirectional training uses tokens as context and as predictions symmet- rically. This fundamental difference affects the weight up- dates of W gk. and consequently, the structures encoded in its rows and columns.
Aw 1==!" = [xxe=] JA:
Σ Biax
(11)
VEP:
Only the embedding of l" is instead added to the row space, where the update of every m-th row Awm, is given by
A AWm, =t=t"
Σ iEP,
Bim bxilx
X
(12)
Intuitively, using t* as context increases the dimensionality of the column space proportionally to the embeddings of the predicted tokens, while reducing the row space along the direction of the embedding of t". Conversely, when 1" is being predicted (t; t²), all token embeddings from the context are added to the row space of W ak while only the embedding of t* is added to the column space. Con- sequently, the role a token plays during training affects its contribution to the weight update differently. We formalize this in the following proposition.
We formalize this relationship in the next two theorems, un- der the following assumptions: (a) tokens exhibit statistical correlations, leading to partial alignment in their embed- dings; (b) the entries of Wek are i.id. at initialization with finite mean and variance; (c) some tokens tend to occur earlier in the sequence and are more predictive of future content; (d) bidirectional training induces approximately symmetric error signals. In this setting, we prove how the objective function influences the internal structure encoded by self-attention.
Proposition 2.2. (Different implicit updates for context and prediction). Let U = (ti.....tt tx} be sequence of tokens and let x, E Fd be the embedding of the i-th token. Let AW qk. be the weight update from Proposition 2.1. I. Let t be a given token U. When using t" as context, the k-th column of A Wgg
Theorem 2.3. (Autoregressive training induces direction- ality) Let V (to....,ty' be a vocabulary of tokens, and let U C V. denote the sample space of all sequences of length N. Let U {t1...., .ty] E U be random vari- able with U ~ P(U). Define Pr[tg = t²) the marginal probability that the token at position equals t" E V. Let (XI...., .xy} be the token embeddings corresponding to the elements of where each embedding x; ~~ is drawn ii.d. from a distribution DD with zero mean and covariance matrix Cov(x;) Σ.. Let W qj be query-key matrix of self self-attention mechanism, and let AW gl denote its gradient update as defined in Proposition 1. computed under an autoregressive objective as in Definition A.3.
gk is given by
Aw .k
[x
Σ Broxi SEPL
(13)
1==
while the m-th row of A W qb is given by
AWmy
Σ
Bix (m
Xy.
(14)
iEP
It follows that the contribution of the token t after training satisfies,
When predicting t", the k-th column of A AWak ok is given by
ED [ I Aw
Tr(E)
A Wk.
Σ
Bagl [x]
X
(15)
Ep
Wk, Vm s.t. Em,m <
SEC,-
(17)
while the m-th row of A qr is given by
Moreover, there exists scalar E Rs proportional to the product of the row and column norm variances, x Var (l|wm,-l) I1) Var (ll such that for all w > %
Awm, 1,=t"
[xx=]m Imm
Σ JEC,
En Buxx]
(16)
Pr [[ 1 w] Pr [Il > w)
(18)
We provide proof for this proposition with related remarks in Appendix 1. Note that all mathematical results derived so far rely solely on the structure of self-attention and make no assumptions about the input data. In the next section, we build on these results to link structural patterns in W qt with the specific form of the objective function.
that is, columns of W qb are more likely than rows to have high norm under autoregressive training, indicating a direc- tional bias in gradient updates.
Theorem 2.4. (Bidirectional training induces symmetry). ). Let U = [tt,...,to ,ty} be a sequence of tokens. Let W ale be the query-key matrix of a self-attention mechanism, and
4

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
let AW gk denote its gradient update as defined in Propo- sition 2.1, computed under bidirectional objective as in
where || 112 is the L2 norm, ||m.,.lll is the norm of the k-th column, A4c = [B|||m.,. and o. = v //ar] |ar||m..x 2/22 are the mean and standard deviation of the column norms, y is a scaling factor, and similarly I'M is the sum of the norm of the rows that are higher than a given threshold,
Definition A.3. It follows that,
N
N Big Kij.
AW qk
(19)
;=1==1
FM
Σ kER
mmm with R [Ik ||m ll2 > Pr pr+
and that every pair (i.j) with i # j contributes the weight update with term
(26)
with |mmm as the norm of the k--th row, (4, = [E| m ||2]
AW We Kig BycKiz"
(20)
and v Var mm,--1a as the mean and standard deviation of the row norms.
that is approximately symmetric.
AW Aage AW qb tetty
(21)
Here, positive and negative directionality scores indicate the dominance of high norm rows or columns, respectively (see Appendix A.7). Finally, we compute the matrix W qs for every layer, calculate the median symmetry and directional- ity score across layers, and analyze the differences between encoder- and decoder-only variants.
We provide proof of these two Theorems, and the related Propositions and Lemmas in Appendix A.5.
3. Symmetric and directional structures are predominant in Transformer models
We find that encoder-only models remarkably show a higher degree of symmetry than decoder-only (Figure 2a). This difference is consistent across multiple families of models and input modalities, such as BERT (Devlin et al., 2019), GPT (Radford et al., 2018; 2019), LLAMA3 (Touvron et al., 2023), Phi (Hughes, 2023; Abdin et al., 2024), MISTRAL (Jiang et al., 2023), ModemBERT (Warner et al., 2024), and many others (see Figure S1 for vision and audio models). Strikingly, we observe that decoder-only models have higher degrees of directionality than encoder-only models (Figure 2b). Again, this difference is consistent across all the models and input modalities we consider. We show in Figure S2 that a similar pattern is observed when including full encoder- decoder Transformers (e.g. the language T5 models (Xue et al., 2021)), despite these models having an overall lower degree of directionality.
In this section, we validate our theoretical findings by quanti- fying empirically the degree of symmetry and directionality in different families of open-source Transformer models. To do so, we define two scores for symmetry and directionality in square matrices. First, we define the symmetry score E R as follows,
Definition 3.1. (Symmetry score). Given square matrix M E Ml, we define the symmetry score,
| |M 1
2
(22)
|M
where ||-1|P is the Frobenious norm, and M, and M,, are the symmetric and skew-symmetric parts of the Toeplitz decomposition of M. respectively,
1 M, - [M+M+ MT) M, 2 [M- - (M-M)), (23)
4. Experiments
In this final section, we test if using structural priors based on our previous results can improve the pretraining of Trans- former models. To do so, we train Transformer models from scratch and perform a series of experiments analyze how symmetric and directional structures develop during training across layers.
Here, positive and negative symmetry scores indicate the presence of symmetric and Akew-symment structures, re- spectively (see Appendix A.6). Second, we define the direc- tionality score E R as follows,
Definition 3.2. (Directionality score). Given a square ma- trix M E M., we define the directionality score,
4.1. Evolution of symmetric and directional structures during learning
7M ÕM
(24)
FM + TM
To test the applicability of our result, we first train 12-layer transformer models in both encoder and decoder modes and quantify the median symmetry and directionality scores across epochs. At initialization, the symmetry and direc- tionality score of the matrix Wa at any layer is zero (see Definition 3.1 and Definition 3.2 and related Appendix A.6
where GM is the sum of the norm of the columns that are higher than given threshold, as follows,
TM- Σ immeelp with C c=6 {k |lm, > Me to 2ET
(25)
5

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
b
1.0
rows
1.0
sym
encoder-only
decoder-only
directionality score
symmetry score
0.5
embedding dim
0.5
embedding dim
0.0
0.0
anti-sym
-0.5
columns
-0.5
encoder-only decoder-only
embedding dim
embedding dim
-1.0
10?
10
10
-1.0
107
10a
10
# parameters
# parameters
Figure 2. a) Left) Median symmetry score of the matrix W at function of the total number of parameters. Each dot corresponds to interquartile range across layers of given pre-trained model (see Tables in Appendix 5). Right) Example structures
the median in the W matrix of encoder-only model (BERT Tiny, layer (Ture al., 2019)) b) Left) Same in for the median directionality score the matrix Wat. Right) Example of structures in the W qb matrix of decoder-only model (TinyStories GPT, layer (Eldan Li, 2023))
and A.7). The incremental update of Wgt t described in the previous sections predicts that decoder-only models develop high-norm columns incrementally during training (see Theorem 2.3). Likewise, as symmetric weight updates are added to Wy in encoder-only models, Theorem 2.4 pre- diets that symmetric structures emerge incrementally during training.
Table 1. The final loss the end of training and the speed-up for the 4. 12, and 24-layer models trained on the Jigsaw dataset (cjadams et al., 2017), Wikipedia (Foundation, 2022), and Red Pa- jama (Computer, 2023), with and without symmetry initialization (see Appendix B.1). Speed-up (%) is calculated by subtracting the epoch at which the symmetrically initialized model reaches the non-symmetric model's final loss from total number epochs, and then dividing by the total number of epochs. For example, 50% speed-up means that the model with symmetric initialization achieves the final loss of the non-symmetric model in half the number of training epochs.
Consistent with results pre-trained models, encoder- only models show higher degree of symmetry than decoder-only models (Figure 3a). In contrast, decoder-only models have higher directionality score (Figure 3b). We observe this difference on all datasets we tested (Jigsaw (cjadams et al., 2017), Wikipedia (Foundation, 2022), Red Pajama (Computer, 2023), see Figure S3). Furthermore, late layers of encoder-only models more symmetric and con- verge faster than early layers when training bidirectionally. At the same time, decoder-only models learn almost non- symmetric matrices with strong skew-symmetric matrices in the middle layers (Figure 3c). When training unidirec- tionally, both encoder and decoder models show a higher degree of directionality for late layers, which is remarkably stronger for decoder-only models (Figure 3d). We observe similar differences across layers with all the datasets we tested (Figure S4), despite these models having less signif- icant differences in directionality scores. See Appendix B for a detailed description of the experiments.
MODEL
Loss
SPEED-UP
4-LAYER MODEL
JIGSAW
2.782
JIGSAW (+ SYMM)
2.758
26 %
WIKIPEDIA
0,984
WIKIPEDIA (+ SYMM)
0.812
73 %
RED PAJAMA
1.106 0.907
RED PAJAMA (+ SYMM)
69 %
12-LAYER MODEL
JIGSAW
1,419
JIGSAW (+ SYMM)
1,430
0
WIKIPEDIA
0.256
WIKIPEDIA (+ SYMM)
0.247
20
RED PAJAMA
0.297 0.274
4.2. Enforcing symmetry at initialization improves the training of encoder-only models
RED PAJAMA (+ SYMM)
35 %
24-LAYER MODEL
The previous section showed that symmetric structures in- crementally emerge during training the W matrices of encoder-only models. Here, we first provide evidence that these findings can be exploited speed up training using
JIGSAW
0.786
JIGSAW (+ SYMM)
0.739
14%
WIKIPEDIA
0.192
WIKIPEDIA (+ SYMM)
0.166
44%
RED PAJAMA
0.209
6
RED PAJAMA (+ SYMM)
0.189
34%

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
a
1.0
1.0
encoder-only decoder-only
directionality score
symmetry score
0.5
0.5
0.0
0.0
-0.5
-0.5
-1.0
-1.0
0
1000
2000
0
1000 epochs
2000
epochs
d
encoder-only
decoder-only
encoder-only
decoder-only
1.0
1.0
directionality score
symmeety score
0.5
0.5
layers
0.0
0.0
-0.5
-0.5
-1.0
-1.0
1000 epochs
2000
1000 epochs
2000
0
1000 epochs
2000 0
1000 epochs
2000
Figure 3. Evolution of symmetry score during training. We train Bert-base-uneased model with 12 layers the Wikipedia dataset (Foundation, 2022) in encoder-only (blue) and decoder-only (purple) mode, see legend. Shown are the median and the interquartile range. Shown are the median and the interquartile range. b) Same in panel for the median directionality score. Evolution of the median symmetry score across layers the encoder-only (left) and decoder-only (right) models. Each layer is color-coded as shown on the legend. d) Same as panel e for the median directionality score.
symmetry as an inductive bias. Specifically, we explore how symmetric initialization influences the training dynamics of the model and whether it enhances learning efficiency and overall performance.
improvements were not significant compared the gains observed in language models (see Appendix C for detailed analysis). These results highlight that embedding symme- try inductive bias across all Transformer layers can enhance training efficiency and model performance.
We train 4-layer, 12-layer, and 24- 24-layer encoder-only mod- els, comparing two initialization strategies: initialize the self-attention matrices independently versus initializing the W, and Wn in each self-attention layer to ensure that W yb symmetric (see Appendix B). We report the results of our experiments in Table We observe that enforcing symme- try at initialization leads to lower loss values at the end of training for most of the models. Importantly, symmetric initialization significantly accelerates convergence, reaching the final loss value faster than those with random initializa- tion (up to 75% faster for l-yer models, 35% faster for 12-layer models, and 44% faster for 24-layer models; see also Figure S5a). Moreover, we observe that self-attention matrices initialized symmetrically lose symmetry during training but converge to higher symmetry levels than ran- dom initialization (Figure S5b) This symmetric initializa- tion decreases the gap in symmetric scores between layers compared to random initialization (Figure S5c). When we tested symmetric initialization on vision Transformers, the
5. Related work
Mechanistic Interpretability (MT). In contrast inter- pretability approaches that focus on explaining specific data instances by analyzing features (Wu et al., 2020; Lundstrom et al., 2022). attention scores (Hoover et al., 2020; Barkan et al., 2021; Yeh et al., 2023), or output variations (Jin et al., 2020; Wang et al., 2022a), mechanistic interpretabil- ity (MI) seeks to provide a more general understanding of Transformer models. MI based on the study of "cireuits," analyzing the between activations across different compo- nents of Transformer (Olah al., 2020). Following the categorization by (Rai et al., 2024), MI techniques include: (i) The logit lens (nostalgebraist, 2020; Geva et al., 2021) projects layer activations or weights into the vocabulary space V. allowing the derivation logits and revealing the influence of individual components on the prediction. This

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
technique can also be applied to query-key matrices Wap gk to study how attention heads transform source into target tokens (Dar et al., 2023); (ii) Probing techniques allow to identify correlations between layer activations and features by training a linear classifier or shallow neural network (the probe) to predict the presence of a feature in layer activations (Dalvi et al., 2019; Gurnee et al., 2023); (iii) Sparse autoen- coders map activations into a nigher-dimensional yet sparse representation, facilitating the identification of independent (monosemantic) features (Huben et al., 2024; Bricken et al., 2023); (iv) Visualization techniques facilitate the analysis of attention scores (Olsson et al., 2022: Lieberum et al., 2023) and neuronal activity patterns (Elhage et al., 2022) by rendering them in graphical format. However, their utility often depends on human comprehension of the resulting visualizations: (v) Ablation studies assess the importance of model components by systematically removing or modi- fying them and observing the resulting behavioral changes (Olsson et al.. 2022; Wang et al., 2022b); (vi) Causal media- tion analysis (CMA) analyzes the importance of components (Vig et al., 2020; Meng et al., 2022) or connections (Wang et al., 2022b; Goldowsky- et al., 2023) between them by introducing perturbations (e.g., noise) and selectively patch- ing them to measure the recovery of lost capabilities. Our work adds a new perspective on MI by providing scalable and generalizable approach to the mechanistic understand- ing of self- attentntnniin. In contrast to existing work, it is not limited to analyzing fully trained models but investigates the influence of learning and can analyze models of different sizes across all modalities.
broader approach, confirming our findings in the image domain.
6. Discussion
In this work, we demonstrate how bidirectional and au- toregressive objective functions influence the structure of the query-key matrix Wyt in self-attention, enhancing our understanding of Transformer models. Our mathematical framework shows that bidirectional training induces sym- metric structures in W gh: whereas autoregressive training results in matrices characterized by directionality and col- umn dominance. To empirically validate our analysis, we develop and apply symmetry and directionality scores to various Transformer encoder and decoder models across multiple modalities, including text, audio, and images. Our results reveal that bidirectionally trained encoder models ex- hibit high symmetry, while autoregressively trained decoder models demonstrate strong directionality, thereby support- ing the predictions of our mathematical framework. This suggests that self-attention inherently reflects these struc- tural properties, contributing to the mechanistic interpretabil- ity of Transformer models. Finally, we leverage our findings to improve convergence speed during bidirectional train- ing by initializing W, and Wa matrices such that Wok is symmetric. While our findings mark an initial step to- ward leveraging symmetry for more efficient Transformer training, further research is required to assess the scalabil- ity of symmetric initialization in large-scale models and across diverse domains. Furthermore, is important to ex- plore strategies for leveraging the directionality structures of decoder-only models. For instance, incorporating structural constraints into the objective function or weight regular- ization could enhance training efficiency and stability in autoregressive settings.
MI for model enhancement. Insights from MI have been instrumental in various applications, including improving model safety (Belrose et al.. 2023), updating the model's s learned knowledge (Meng et al., 2022), and guiding the generation process of generative models (Geva et al., 2022). One of the most exciting applications is leveraging MI tech- niques to improve model performance. For instance, Skean and geometric
Intriguingly, we found that there are similarities between the structure of the attention weights that we find and existing connectivity patterns in cortex, areas dedicated and image comprehension show symmetric connectivity patterns (Song et al., 2005; Wang et al., 2006), while areas dedicated to language are highly directional (Peng et al., 2024). Further studies could use those patterns to connect self-attention mechanisms and models of information processing in the brain (Gershman et al., 2025).
et al. (2025) conduct information-thhoree analyses to show that intermediate layers exhibit a favor- able trade-off between compression and signal preservation. Leveraging this insight, they demonstrate that features ex- tracted from these layers consistently outperform final-layer representations across a broad range of downstream tasks. Similarly, Trockman & Kolter (2023) observe that query-key matrices (W₂t) gk)) frequently exhibit pronounced negative diagonal, prompting them to initialize it with approximately the identity matrix, leading to enhanced accuracy in im- age classification tasks. Similar to (Trockman & Kolter, 2023), we demonstrate that findings about the structure of W can inform initialization strategies that improve Trans- former performance. However, since the identity matrix is one instance of symmetric matrix, we consider the work by (Trockman & Kolter, 2023) special instance of our
By bridging theoretical insights with practical improve- ments, our work not only advances the interpretability of self-attention but also provides a foundation for optimiz- ing Transformer architectures. Ultimately, these findings contribute to a deeper understanding of the mechanisms governing self-attention, paving the way for more reliable and efficient Transformer-based models.
8

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Authorship contributions
plaining Transformers via Gradient Self-Attention Maps. In Proceedings of the 30th ACM International Confer- ence Information & Knowledge Management, CIKM, Pp. 2882-2887, Virtual Event Queensland Australia, Oc- tober 2021. ACM. ISBN 978-1-4503-8446-9, doi: 10.11453359633.3482222
Conceptualization: MS, TS, BG. Mathematical analysis: MS, PVA. Analysis of pre-trained models: MS. PS. Experi- ments: PS. Writing: MS, PS, BG. Supervision: TS, BG.
Code availability
Belrose, N., Furman, Z., Smith, L.., Halawi, D., Ostrovsky, I,, McKinney, L., Biderman, S., and Steinhardt, J. Eliciting Latent Predictions from Transformers with the Tuned Lens, March 2023, arXiv:2303.081
The code to reproduce our results and all the Figures in the main text and Supplementary Materials is freely available giinth..co/mmieesspppt/atttttttt geometry.
Besag, J. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society Series D: The Statistician, 24(3):179-195, 1975.
Declaration of Interests
The authors declare no competing interests.
Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Model- ing with deeh--Tnsoofloo. GPT-Neo: Large Scale Au- loregressive Language Modeling with Mesh- Beees-T-nssrflow Zenodo, March 2021. doi:
Acknowledgements
MS was supported by an ETH Postdoctoral fellowship (reference nr. 23-2 FEL-042). PVA was supported by UZH Postdoctoral fellowship (reference nr. K-76207- 01-01). TS and PS were supported by DIZH Fellowship "Stability of self-organizing net fragments as inductive bias for next-generation deep learning" from the ZHAW. This work was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps. We thank Yassine Taoudi Benchekroun, the Grewe lab, Alexander Nedergaard, Thomas Hofmann, and the Data Analytics lab for their helpful comments and insightful discussions.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., et al. Towards Monosemanticity: Decom- posing Language Models With Dictionary Learning. Transformer Circuits Thread, October 2023. URL monosemantic- htt.://mn...cttttttttteeeeeettta
cjadams, Sorensen, J., Elliott, J., Dixon, L., McDonald, M., nithum, and Cukierski, W. Toxic Comment Classification Challenge, 2017. URL https:
Impact Statement
classi cossificationnchi lenge.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Computer, RedPajama: an Open Dataset for Train- ing Large Language Models. https: thub.com/ togethherccompter//Reee RRedPa jama--ataaaaaaa Data, October 2023.
References
Dalvi, F,, Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., and Glass, What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 33(01 (0000))6600-6311, July 2019. ISSN 2374-3468. doi: 10. 0330:...33300003331133
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, August 2024. arXiv:2404.14219
Dar, G., Geva, M., Gupta, A., and Berant, J. Analyzing Transformers in Embedding Space. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics, volume of ACL., pp. 16124-16170, Toronto, Canada, 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/2023.ad.eo
Bao, H., Dong, L., Piao, S., and Wei, E. BEÎT: BERT Pre- Training of Image Transformers. In Proceed- ings of the 10th International Conference on Learn- ing Representations, ICLR, 2022. URL https:// openteviee.net/fooun wtwp:/nnee/oonumuememe wtt.eeeeeteooooendeeeeeeeeeeee.e -eiiidddds--eedce.
893.
Barkan, O., Hauon, E., Caciularu, A., Katz, O., Malkiel, I,, Armstrong, O., and Koenigstein, N. Grad-SAM: Ex-
Deng, J.. Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database.

---

Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. Trans- former ForwaFo Layers Build Predictions by Pro- moting Concepts in the Vocabulary Space. Proceed- ings of the 2022 Conference on Empirical Methods in The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 248-255, 2009. doi: 10.1109/ N1RR22099.2068445
Natural Language Processing, EMNLP, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v// 2022.emnlp-main..
Devlin, J. Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transform- ers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguistics, vol- ume of NAACI., pp- 4171-4186, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10. 10.11663/v1/N19 1423.
Goldowsky-Die Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing Model Behavior with Path Patching, April 2023. #rXiv:2304.05969
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding Neurons in a Haystack: Case Studies with Sparse Probing. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835-8856. URL https: pttps://openrevieewe forum?id=JYs1R9INN
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., et al. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In Proceedings of the 9th International Conference on Learning Representations, ICLR, 2021. URL https: // //openreview.net/eeeeeee ootp://///oooommmmooooooooooo d.TTT.
Hoover, B., Strobelt, H., and Gehrmann, S. exBERT: A Vi- sual Analysis Tool to Explore Learned Representations in Transformer Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics: System Demonstrations, ACL, pp. 187-196, Online, 2020. Association for Computational Linguistics. doi: 10. 1
Eldan, R. and Li, Y. TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, May 2023. arXiv:2305.07755
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse Autoencoders Find Highly Inter- pretable Features in Language Models. In Proceed- ings of the 12th International Conference on Learn- ing Representations, ICLR, 2024. URL https:// openreview.ned wttp://wwwweeee/fotieeeeee-e forumfod-urrprree 6bwRS1.e.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Con- erly, T., et al. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, December 2021. URL circuits ppb.ss.ppp//... http///aaaewwwwwnneeeee.e
Hughes, A. Phi-2: The Surprising Power of Small Language
Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., Das- Sarma, N., Mann, B., et al. Softmax Linear Units. Transformer Circuits Thread, June 2022. URL
Models, December 2023.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., Casas, d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., al. Mistral 7B, October 2023. arXiv:2310.06825.
solu/index.html
Foundation, W. Wikimedia Downloads. https:// Cumps.wikimediia.rs. 2022.
Jin, D., Jin, 7.,, Zhou, J. T., and Szolovits, P. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05); 8018 81888025, April 2020. ISSN 2374-3468. doi: 10.1609/ aaai. v34.05.631..1
Gershman, S. J., Ficte, I., and Irie, K. Key-value memory in the brain. Neuron, (13 11:16)))-110000 1707, 2025.
Geva, M., Schuster, R., Berant, J., and Levy, O. Transformer Feed-Forward Layers Are Key- Value Memories. In Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v11 101553vv12000 1/2021 wttp://ee00/2111meeeeea mm..
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. ALBERT: A Lite BERT for Self- supervised Learning of Language Representations. In Proceedings of the 8th International Conference on Learning Representations, ICLR, 2020. URL https: // wwwppneevew.neee/eeeooo d= eAAEEVV..
10

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Lieberum, T., Rahtz, M., Kramár, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilitiess in Chinchilla, July 2023. arXiv:2307.09458.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving Language Understanding by Generative Pre-Training June 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models Are Unsupervised Multi- task Learners, 2019.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Ap- proach, July 2019, arXiv:1907.1
Radford, A., Kim, J. w., Xu, T., Brockman, G., Mcleavey, C., and Sutskever, I. Robust Speech Recognition via Large-Scale Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning, vol- ume 202 of ICML, pp. 18492-285188, PMLR, July 2023.
Loshchilov, and Hutter, F. Decoupled Weight De- cay Regularization. In Proceedings of the 7th In- ternational Conference on Learning Representations, ICLR, 2019. URL https:// http::://peenzeview...tttt forum? 1 ddm.mggki--BK....
Rai, D., Zhou, Y., Feng, S., Saparov, A., and Yao, Z. A Practical Review of Mechanistic Interpretability for Transformer-Bssee Language Models, July 2024. ar arXiv:2407.02644 07.44466..2..
Lundstrom, D. D., Huang, T., and Razaviyayn, M. A Rigor- ous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributions. In Proceedings of the 39th International Conference on Machine Learning, vol- ume 162 of ICML, pp. 14485-14508. PMLR, July 2022.
Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., and Shwartz-Ziv, R. Layer by layer: Uncov- ering hidden representations in language models. arXiv preprint arXiv:2502.020113 2025.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and Editing Factual Associations in GPT. In Advances in Neural Information Processing Systems, volume 35 of NeurlPS, pp. 17359-17372. Curran Associates, Inc., 2022.
Song, S., Sjöström, P. J., Reigl, M., Nelson, S., and Chklovskii, D. B. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology, 3(3): e68, 2005.
nostalgebraist. Interpreting GPT: the logit lens, 2020. /pstss/ AcCRBeuuppaalvttrr//eeerrrrteeed the-logit-lens
Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, volume 30 of NeurIPS. Curran Associates, Inc., 2017.
URL https:
Olah, C. Mechanistic Interpretability, Variables, and the
Importance of Interpretable Bases, June 2022. URL
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, al. LLaMA: Open and Efficient Foundation Language Models, February 2023. arXiv:2302.1397.
attps://trannsffrmer-iicooo ott.://aaaattttaaaaoooooooaa///a ppb//222// mech--interp-essay/nddx.btt
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024.00 March 2020. ISSN 2476-0757. doi: 10.2391 1.d22395//disill..om 5/diisiill.0022.000.
Trockman, A. and Kolter, J. Z. Mimetic Initialization of Self-Attention Layers. In Proceedings of the 40th Inter- national Conference on Machine Learning. volume 202 of ICMI., pp. 14456-34468 PMLR, 2023.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y. Chen, A., et al. In-context Learning and Induction Heads. Transformer Circuits Thread, 2022. URL bettps://transfommet www.ts..tttttttt.oocoooo in in-context- -ew.nnttex---ereenng and- induction heads/index.html
Turc. I., Chang, M.-W., Lee, K., and Toutanova, K. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models, September 2019. arXiv: 1999008882....
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, I.., and Polosukhin, Atten- tion is All you Need. In Advances in Neural Information Processing Systems, volume 30 of NeurIPS. Curran Asso- ciates, Inc., 2017.
Peng, Y., Bjelde, A., Aceituno, P. V., Mittermaier, F. X., Planert, H., Grosser, S., Onken, J., Faust, K., Kalbhenn, T., Simon, M., et al. Directed and acyclic synaptic con- nectivity in the human layer 2-3 cortical microcircuit. Science, (84(6693) 343, 2024.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. Investigating Gender Bias in

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems, vol- ume 33 of NeurlPS, pp. 12388-12401. Curran Associates, Inc., 2020.
and privacy: The Good, The Bad, and The Ugly. High- Confidence Computing, 4(2):100211, June 2024. ISSN 26672952. doi: 10. 10.10166j.hcc.202.. 100211.
Yeh, C.,, Chen, Y., Wu, A., Chen, C., Viégas, F., and Watten- berg, M. Attention Vizz:ionVii: A Global View of Transformer Attention. IEEE Transactions Visualization and Com- puter Graphics, pp. 1-11, 2023, ISSN 1077-2626. doi: 10.1
Wang, B., Xu, C., Liu, X., Cheng, Y., and Li, B. SemAttack: Natural Textual Attacks via Different Semantic Spaces. In Findings of the Association for Computational Lin- guisties. NAACL, pp. 176-205. Seattle, United States, 2022a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2222..nm http://22222.nnnnnnnaaaaaaa 14.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy train strong clas- sifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, October 2019.
Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the Wild: Circuit for Indirect Object Identification GPT-2 Small, November 2022b. arXiv:2211 1..119..09..
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz. D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, ICLR, 2018. URL https: http///ppnneeeeeeee//// forum? Srrum?id=r1ddpp..
Wang, Y., Markram, H., Goodman, P. H., Berger, T. K., Ma, J., and Goldman-Rakie P. S. Heterogeneity in the pyramidal network the medial prefrontal cortex. Nature neuroscience, 9(4):534554 542, 2006.
Wang, Y., Zhong, W., Li, L., Mi, F., Zeng, X., Huang, W., Shang, L., Jiang, X., and Liu, Q. Aligning Large Language Models with Human: A Survey, July 2023. arXiv:2307.12966.
Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., al. Smarter, Better, Faster, Longer: Mod- Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference, December 2024.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. HuggingFace's Transformers: State-of-the Nat- ural Language Processing, July 2020. arXiv:1910.03771.
Wu, Z., Chen, Y., Kao, B., and Liu, Q. Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 4166 4176, Online, 2020. Association for Computational Linguisties. doi: 10. 1/2p:/11101200mmmmmmaaa
Xue, I.,, Constant, N., Roberts, A., Kale, M., AI-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mT5; A Mas- sively Multilingual Pre-Tree Trained Text-to-Tees Text Transformer, March 2021. arXiv:2010.
Yang, Z., Dai, Z.,, Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Yao, Y., Duan, J.. Xu, K., Cai, Y.. Sun, Z., and Zhang, Y. survey on large language model (LLM) security
12

---

F that is The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
A. Mathematical proofs
A.1. Preliminaries
Following the notation in (Vaswani et al., 2017; Radford et al., 2018), we define a Transformer architecture as, Definition A.1. (Transformer architecture). Let U RRN. be matrix representing the sequence of N one-hot encoded tokens of dimension V. Transformer architecture consists of stacked attention blocks, each one composed of self-attention layer and feedforward layer, as follows,
Xo(U) UW + WP
xx=x; = + ar (i-i W o
W W [; X (1: W W2)
4 [1,7]
(S1)
o (Z) oX W.))
where W. Vid represents the linear transformation from the vocabulary space the embedding space of dimension d, Wp E Vdd represents the positional encoding, Xo R. is the initial embedding of the sequence, ar(-) is self-attention function given by
a ( = ((X
(S2)
where the matrix of attention scores A (X1 is given by
(X W
K (X-1) = X W
(S3)
( 0 (22 (22KK"
where 1/vī is constant normalization factor, and W E R.ddd and W Rdd represent linear transformations within the embedding space, v'(x-1) = X W represents a linear transformation within the embedding space, my(-) is a position-wise feedforward layer with hidden dimension d, and Icarnable matrices W e Rddy and W E Rdrdd W. Rdd.V represents the linear transformation from the embedding space back the vocabulary space, a(-) is the row-wise softmax function, and o (Z) E R...V is the estimated probability distribution over the vocabulary. We omit layer normalization and biases for simplicity (see also (Elhage al., 2021)).
Furthermore, we use the following definition of a bilinear form,
Definition A.2. (Bilinear form). A bilinear form on vector space V over field F is a map M : V X linear in each argument separately, that is,
M(ax 2) = bM M(x, ay + bz) a.((xy)) bM 2)
(S4)
for all x,y, E and a, E F. Let {ei,.... ea} basis for the vector space The matrix M such that [M]y M(e,, 0,) the matrix of the bilinear form on this basis, and it follows
M(x y) = My
(S5)
Finally, we provide the following definition of autoregressive and bidirectional training objectives, Definition A.3. (Autoregressive and hidirectional objectives). Let U ((1...., sequence of tokens. The joint probability of U is factorized autoregressively as follows,
Pr[U] = Pr|t...... = N. Pr(ts]ti..... (.-)).
(S6)
During autoregressive training, model with of parameters W is optimized to minimize the following negative log-likelihood
N
(UU:W) = Σ
(w [ty): <)
(S7)
13

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
where the conditional probabilities modeled with learnable parameters W. During bidirectional training, model with set of parameters W is optimized to minimize the following negative log-likelihood
((U: - Σ log pw [t)]: +
(S8)
In practice, only a subset M [I....,N N] of the tokens are predicted as in Masked Language Modelling (MLM) (see Devlin et al., 2019; Warner al., 2024), leading to the following negative log-likelihood
W) = Σ
log :tyhjs : M).
(S9)
EE M
A.2. Related remarks to Section 2.1
Remark A.4. Each element of the sum in Equation (4) implicitly defines an operator in the subspace spanned by X (x] xT....,,NN given the transformed embedding space defined by W
x
gh:
Σ ais (W)) = Σ x W X,
PWww. on X)),
(S10)
where (- (1))) are operators over the subset X. The set X is in general linearly dependent, and the number of tokens N in the sequence differs from the embedding space dimension d. Furthermore, Wgs gh represents general bilinear form (see Definition A.2), which may not satisfy all the defining axioms of a formal inner product - namely, linearity, conjugate symmetry, and positive definiteness. Finally, the operators Pw,, ( (-x)) are not formal projection operators since they are not nilpotent (Pw,, (-x, x;) o PW,,. (-x;) o Pw ( x))). Nonetheless, the bilinear map Wa R x Rd - IR still associates any pair of vectors with a scalar value quantifying their alignment as determined by the geometric relations encoded Wat- We Therefore, self-attention computes a generalized decomposition x, on Conv (X) in the transformed embedding space defined by Wgk- convex combination ensures that the resulting vector remains within the region enclosed by the basis vectors X.
Remark A.5. Following Definition A.I, multi-head attention consists of parallelizing the self-attention operation across H different heads with an embedding space d₁, < d,
X = X + concat 1,, A2V2, A,
(S11)
k. X²) is the self-attention of the h-th head, Wihh Rd.d, WEA Rddd, and where A 1/2 X q.h W Wk,
W b E Rd.dd are the query, key, and value matrices of the h-th attention head, respectively, and W. E gdd is linear transformation (Vaswani et al., 2017). Operationally, the self-attention computation is performed in parallel by factorizing the W, and We matrices into H rectangular blocks, as follows,
W q
W 9.1 W Wg
W H
(S12)
We
W
W
W
and performing the matrix multiplication W. a WT k per every h-th head independently in one step. It follows that the full W ql: matrix is given by the sum of the bilinear forms W akeh of every head, as follows,
WoB = W, = Ewase a. WT
Σ Watch h
(S13)
h
where each W Wh Bddd is square matrix with rank W (h.h)) projections onto Conv(X) that are then summed together, as follows,
S dn- Therefore, each head perform independent
E Σ x W W
Σx
x
W Σ
Wykth gk.h X2 X, ΣΣ (x,, W Wotnhx)x Xj.
(S14)
thus performing the same operations as in Equation (4).
14

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
A.3. Related remarks to Section 2.2
Let V- (f,,..,, } be a vocabulary of tokens, and let D be dataset of sequences Un E V of length N, where D = (Un = (Ii .......,. xx) ) C VX, For a single sequence Uu with joint distribution p(Un), the chain rule implies that for any permutation a of (I,...; J,, the joint probability can be written as,
plU,) II p(="0 (o(1):- 2 ())-)))
(S15)
Given a set of conditional parents DD C (I,..., N} \ {i), define the **ocaa negative log-likelihood** as:
N
C(U:W)- Σ
log pwr rD.).
(S16)
where W denotes the model parameters. Let G be directed graph where each edge is D. + i. We distinguish two cases:
1. If G a directed acyclic graph (DAG), then a topological ordering t exists such that Dz()) S (r(I),..., 7(i - 1)}. In this case, the joint distribution (z( II N (x TD is properly normalized and the local negative log-likelihood in (S16) is equal to the true negative log-likelihood. For example, during autoregressive training, each token is conditioned only on its predecessors,
N
L(U;W) =- Σ fogu(lal Ii,..., xi-1)
logpw(a; res)-
(S17)
2. If G contains directed cycles, the product II pw(a, TD,) is not guaranteed to be normalized, and thus does not define a proper joint likelihood. In this case, one can instead minimize the following pseudo log-likelihood,
((U:))=-\ = logpw((; Ti,..., 41-1 24 NN)
Σ log x-1),
(S18)
where Z-₁ denotes all tokens in the sequence except x4. Following Besag's theorem (Besag, 1975), the maximum pseudo-likelihood estimator consistent under standard regularity conditions, and (S18) remains compatible with stochastic-gradient optimization even though is not the true negative ligelikeli (Yang et al., 2019).
A.4. Proof of Proposition 2.1 and related remarks
Proof. Let = (4,,..., ( a sequence of tokens. Let L) be the negative log-likelihood each token 4 expressed as
C)U:W)= = Ec(t) - = - log
(S19)
where D, C (0.1. NJ is the conditioning set, that is, the set of indices defining the set of tokens (1, j E D.} of the conditional probability distribution. Let U= (tu. ti..... tx] be the sequence of N one-hot encoded tokens t, E RV associated with U, where the dimension the vocabulary. Let L(t,) be the cross-entropy of the one-hot encoded token t, and the estimated probability distribution (zz) BR as follows.
N C(U)
C(t))
Σ log(o(z))) d=1
(S20)
i=1
where we let z; the prediction of the i-th token t; from the representations in the last layer of a Transformer model, following Definition A.
xx- =t; Tw + We
xf" = =[((s-11)
vi [1,
(S21)
(2i) = o (xt Tw.),
15

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
where x _ = Fi(x)-) is a short notation for the self-attention and multi-layered perception transformation of the /-th layer,
x = -x--" + (x(xx=))
Fixx
(S22)
x = =x + +m(())
where the self-attention function is given by
(x
)
dly((o)) (w0)) x T w JED;
Σ
(S23)
Let attention coefficients of = = of the I-th layer be parameterized with general parameter w'. Let the gradient of [t,,) w.r.t. wi (the parameterization of the attention scores) be factorized as follows,
V [wi)( = aC(t,) Ooky dal, Ow
(S24)
follows that,
Oc(t.)
@L(t.) dz, 0x 0x!
daly
Oz.
ox! Ox! d 2g
(S25)
dxp o%
T W = (t, oz())
w
Σ
JED.
where the term 0xt /8x, includes the set of partial derivatives that define the gradient of the representation x at the last layer w.r.t the self-attention representation x/ the l-layer, follows,
0x! 0x!
L-1 Σ E((())) ) (1 + mi(
1+
(S26)
m=l
where
3 Fi("
m((x)) ① ml (x/)-
F(("
(S27)
0x!
Ox!
Let 6% be the error at the last layer propagated to the self-attention function the I-th layer, as follows,
Ox
o =
W
(S28)
thus obtaining the following equation for the gradient
da
V C(t)) 85
(S29)
Dw
JED:
Let the attention scores be computed without the row-wise softmax operation and explicilly with the bilinear form W gk. such that the following expression gives the score between the i-th and jth token,
a
aly x; 1-1
W
xx
(S30)
from which we obtain
duly ow 9k
= x
(S31)
where K! M. is square rank-1 matrix given by the outer product between the i-th and j-th token from the I - 1-th layer. It follows that the total gradient of L(t,) is given by
V W C(t:) 55
Σ
x
k! ij
1-1
(S32)
JED.
16

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training where we notice that, for every j, the term 8 x
1-1 is a scalar quantity that we define as S. thus obtaining,
Vw C(t;) = Σ
3
K ij
(S33)
JED.
and therefore,
V L(U) = -
- 8 JED:
Σ
K
1
VW gke LE [1,
(S34)
We can rewrite the double summation either as,
V ww L(U) ΣΣ JEC.
K
V ok: E [1,
(S35)
o/
where Cj C (0. N is the set of indices defining the set of tokens
{t,} used to predict 6, or equivalently as,
Ww; ((U)
Σ iEP;
Σ
B
K
VW gh: LE [],
(S36)
where P, C (,,,,.... .] is the set of indices defining the set of tokens {4i that are predicted by ty. thus concluding the proof.
In standard Transformer models, the bilinear form W is not directly computed, and as such, it is not explicitly updated through gradient descent. Nonetheless, W qt is implicitly updated with combination of the weight updates of W q and WR having the same form as in Proposition 2.1, see the following.
Remark A.6. Let L(U) be the negative log-likelihood of given sequence of one-hot encoded tokens U. Let W and W be the query and key transformation matrices of the [-th layer of Transformer models, see Definition A.1. Let W and A be updated via gradient descent, that is, W - w! + +nVw;((U) and W + W + + ((U), where is the learning rate. It follows that the matrix W gk W w is implicitly updated following.
W
W 9b + (W, + w: w/ ((U)) W ((V))
-W' + ( W C (U) Vw; ") + +
-W 9k + Σ Σ B ( w K,J + K w/ W. ) + o(7²)
(S37)
jEC.
W gk + AW oh + o W gk + AWl
gk
assuming that the learning rate 77 is small. Therefore, the implicit weight update of W gk following gradient descent is given by
AW 2h = 17 Σ Σ
b (Ww k' (K', wh
Σ Σ K
(S38)
JEC:
JEC:
where both W W! k! and K!
W W!W' are rank-1 matrices,
-Ww" W x >x,"" =x 1-1 K w W! T =x (x (x)-1
w/w:
7 T
(S39)
T W WWwkk] )) -x(-1x/-1
A.5. Formal proofs of Theorem 2.3 and 2.4: the connection between objective functions, directionality, and symmetry In this section, we provide formal proofs of Theorems 2.3 and 2.4, which characterize the directional and symmetric structures that emerge during autoregressive and bidirectional training, respectively. To this end, we introduce a series of intermediate Propositions and Lemmas, following these steps:
17

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
1. We begin by showing that when 1a is used context token predict other tokens, all predicted tokens contribute the column space of W gt: while only E² contributes the row space. Conversely, when t is the token being predicted, only t contributes to the column space, whereas all context tokens contribute to the row space (Section A.5.1).
2. Next, we show that under reasonable assumptions about the statistical distribution of token embeddings, these structural properties imply that the expected norm of column updates exceeds that of row updates when t" is used as context. In contrast, row updates dominate when 8* is the predicted token (Section A.5.2).
3. We then show that in autoregressive training, the expected number of times a token 1* appears as context can differ from the expected number of times predicted, since these two quantities are affected differently by statistical correlations between tokens. In contrast, under bidirectional training, the expected counts always equal, regardless of token correlations (Section A.5.3).
4. Finally, we combine the above results to prove our main theorems: (1) the weight updates to W gk induce column dominance under autoregressive training (Section A.5.4-A.5.5), and (2) the weight updates to W ok induce symmetry under bidirectional training (Section A.5.6).
A.5.1. PROOF OF PROPOSITION 2.2
First, we show that token ta contributes differently to the updates of Wyk depending on whether serves as context for predicting other tokens itself predicted.
Proof. Let U . tx} be sequence of tokens with the embedding of every i-th token be given by x; E R Proposition 2.1 shows that the implicit weight update can be decomposed with two equivalent regrouping of the double summations, as follows,
AW ok
E (F.))EU
X =
Σ 5
Σ iEP,
Biyx
x,
Σ X
B1xx7 \BOx) LiEC
Σ
(S40)
where Pi C U is the set of tokens predicted by given token f,, while C₁ C U is the set of tokens that predict given token lj- We neglect any constant of proportionality such as learning rate for simplicity, and we do not assume any specific structure on P, and C, (autoregressive training, bidirectional training, or others). First, the contribution of t* e U to the weight update when t" used as context to predict a set of tokens P.- CU is
W
Σ IEP,
BLOG
Bioxs X,.
(S41)
gk
The associated weight update of the k-th column is then given by
Aw Aw.g Ae
Σ
Bioxi
[x:+]e
[xr-le
Σ BioKi
(S42)
1
PEP
iEP
while the update of the m-th row is
AWm. m.
Σ PEP
Bin [xi]m
X
(S43)
1==!"
A complementary argument can be made when predicting a given token t" from a set of tokens C.+ Indeed, the contribution to the weight update when t" is predicted by set of tokens C,- C U is given by
W
<
Σ JEC.
Buzxe xJ.
(S44)
gk
The associated weight update of the k-th column is then given by
!
Σ
Bug (xsyle
xr+
(S45)
JEC.
18

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training while the update of the m-th row is
Σ jEC,
A m,
Σ
3
(x
[xx+
Buxx3] jEC,
Σ
(S46)
1,=!"
Therefore, the weight update each column (row) when set of tokens predicts t* is equivalent the weight update of each row (column) when t* is used predict tokens. This concludes the proof.
A.5.2. ASYMMETRIC GROWTH OF ROWS AND COLUMNS DURING WEIGHT UPDATE
Next, we demonstrate that under reasonable assumptions about the statistical distribution of token embeddings, the expected norm of column updates exceeds that of row updates when the token !* is used as context to predict other tokens. Conversely, when td is being predicted by other tokens, the row updates become dominant. To do so, we make the following assumptions:
The token embeddings X; are independent and identically distributed (i.i.d.) random vectors drawn from a probability distribution with zero mean, [[x,] = O, and covariance matrix Cov(x;) = Σ. This assumption holds at initialization for any Transformer model with learnable embeddings.
The covariance matrix is isotropic, i.e., & I²Il.. More specifically, we posit that there is partial alignment between the embeddings x, due to the semantic and predictive relationships between tokens, which typically emerge during training.
Similar to Proposition 2.2, the scenarios where t* is used to predict other tokens and where t" is being predicted by other tokens are complementary. In the following Proposition, we focus solely on the case where t* serves as context to predict a set of tokens. A formal derivation for the opposite case where 1* predicted by other tokens is provided subsequent Corollary.
Proposition A.7. (Asymmetric growth of columns and rows for context). Let U (f1.....tty .,) a sequence of tokens, and let be a token representing the context of every loken l E Let (x1.... .... Xx} such that each x; DD a i.i.d. random vector drawn from probability distribution D with zero mean and non-isotropic covariance Cov(x; Σ. Let AW yk be the weight update from Proposition 2.1. Then the squared norm of the m-th row and k-th column of W gls satisfies
the token embedding associated with U
ED ||
& 13] m. IP
THE) d
Vk E (1....,,d,, Vm s.t. E m
(S47)
E
Aw
Proof. Let U = ,tob...,,t]]. be sequence of tokens, and let t" be the context of every token fz E U. Let {x, be the set of token embeddings associated with U and let y be the token embedding of t". It follows from Proposition 2.2 that the squared norm of the weight update the k-th column is given by
N
N 3 x 12
AW. Σ
/ylx
+
||
(S48)
i=1
while the squared norm of the weight update of the m-th row by
A AWm.-
Σ 3:o [x:]
A m. 11
Bix [x: (mm) 11111112
(S49)
Let {x,} ~ D be a set of i.i.d. random vectors from a probability distribution D such that Ep = 0 and Cov(x;) = Σ E Rdd. Therefore, each k-th coordinate [xe]e is such that Ep [[x]<s] = and Var([x<]<) = Σn,k- We also assume that y is statistically independent from x, Vi with E(y] = and covariance Cov(y) = a L It follows that the expected value of ||Aw 2l2 over given by
Ep [|A = ED
[
1|2
ED [
ED
Σ
B x 1|
(S50)
19

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Given the statistical independence between the entries of X₃, the second term is equal to
ED
3 x 11
ED [3 xy]
3 ED [x] xi] Σ
Br.Be.By [x] xy]
1z1
(S51)
Σ
3/3 Tr Tr [x,x]] x)x + Σ
Bia BiaBBraBou [x]'x]]
1=1
8
N
= Tr()))
32
and therefore
N Σ
Ep
||| Aw x11]]
(S52)
t=1
Similarly, the expected value of m| 113 is given by
Ep
||| m. [1 ED
Bin xx) m 911111 -E
Bea [m)mm)
ED [||y|]]
(S53)
The first term can be decomposed as
Ed
Ba (x)(m))
Σ ED
[xx]]]]]] + Σ
B Bv.IE [1x [x:xmm] Em,m
B2
(S54)
and therefore
N Σ
ED
[|A wwm
12 = Em, Tr(T)
82,
(S55)
1=1
The ratio of the expected value of these squared norms is
Ep
Aw
[2] F Tr(E)
TA Tr(E)
Tr(E)
(S56)
Ep
W m.. 12] mm Tr(T)) Tr((') Em.m d 2m
We assume a non-isotropic covariance structure in Σ, that is, the average variance per dimension is lower than the total variance across all dimensions. This implies Tr(2)= >> d 2mmm for some m. It follows that,
Ep
|| &
IPI
1 TT(E).
1dmm =1 1 Vm s.t. Em,m <
Tr(E)
(S57)
Ep m. [2] d Dm.m
Em.m
d
thus concluding the proof.
Again, a complementary argument can be made about the asymmetric weight update when a set of tokens U = (t1,-.., tx] used to predict a given token t". We formalize this in the following Corollary.
Corollary A.8. (Asymmetric growth of columns and rows for prediction). Let U = (l....,, .N) be a sequence of tokens, and let every token t; E U be the context of given token t". Let the associated embedding be drawn i.i.d. from a probability distribution as Proposition A.7. Let AW gk be the weight update from Proposition 2.1. Then the squared norm of the m-th row and k-th column of Wgk satisfies
Ep Aw & [2
Vm E (1,... d). Vk. s.L. ΣE < Tr(E)
(S58)
Ep Aw m.
d
20

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Proof. Let U Ilo..... . [N] a sequence of tokens, and let 1. be predicted by every token 4 U. Let {x:) be the set of token embeddings associated with U and let he the token embedding of t*. It follows from Proposition 2.2 that the squared norm of the weight update of the k-th column is given by
N Σ
Aw
3,5lx + ||
Σ
Baz (xxxx)le) |yy11,
(S59)
j=1
while the weight update of the m-th row follows
N
! Awm. m.
Σ B,jxsLyim + ||Awm, 112
[y
Σ
(S60)
Therefore, following the same arguments as in Proposition A.7, the ratio of the expected value of the square norms is given by
ED Al 1 Ep Awm. m. [ m. Tr(2:) Tr(2))
ΣR (T(
= d ΣR.
dE
Tr(E)
=1 1
Wk s.t. ME. < d
(S61)
d DR
thus concluding the proof.
A.5.3. EXPECTED CONTRIBUTION FOR CONTEXT AND PREDICTION AND RELATED REMARKS
Additionally, we show that in autoregressive training, the expected number of tokens predicted by given token /* can differ from the number of tokens that predict t", with this ratio influenced by token correlations, while in bidirectional training, the two quantities are always equal regardless of such correlations, yielding a ratio of 1. We formalize this in the following proposition and illustrate, in the subsequent remark, an example from autoregressive training where the expected ratio exceeds
Proposition A.9. (Ratio of expected counts during autoregressive and bidirectional training). Let V (lo:--..- (v) be set of tokens. Let be the sample space of all possible sequences of N tokens, and let the sequence U (t1...., IN E u be a random variable with probability distribution P(U) defined over U. Let Pr[t; t*] be the probability that the token at index j in sequence U E u is given by to E V. Let Ep(pe(t") )] be the expected number of tokens that are predicted by given token t", and R (erl)p((t))) the expected number of tokens that predict a given token t². For autoregressive training, the ratio
Erlpatt" U}] (N- k) Prit =t
(S62)
Eplup(t* (U)] 1)
Pr =t]
depends on Pr(l*- =1 Vl* E V. E (1, (l.....NN .N. while for bidirectional training the same ratio given by
Ep'ue(t",1) U]] (1 1.
(S63)
Proof. Let V = (4.....,4v] be a set of tokens. Let = VN denote the sample space of all possible sequences of N tokens from the vocabulary V. Let F = 24 be the o-algebra given by the power set of U, and let P be the probability distribution over UI that defines the distribution of the random variable sequence U. This defines the probability space P). For each position i E {I..... .V) we let the random variable Tr be Ta : U + V such that (U)) = tg. Let E == (U u:: : 44 [r] the event of all sequences where the k-th token is fixed token 1* The indicator function 1 (U) is a random variable defined as
1 if
1 (EE(U) = 1ftx =()
(S64)
otherwise,
and as such its expected value over is
Ep [1 1114-1000 (()) Σ 1144=0))000 110)01(111101
E Pr[U|=P[[4 (²].
(S65)
VEU
UEU::1=
Let (et(t", U) be the random variable quantifying the number of tokens predicted by t", while (t", U) is the random variable quantifying the number of tokens predicted by t* We analyzed the case of autoregressive and bidirectional training separately.
21

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
During autoregressive training, each time the token 1" appears at position A it is used as context to predict N k tokens, and it is predicted by k- - 1 tokens. It follows that pec(t, U) is given by
1-1 144 = = = Σ IN - -N-1) t"),
Me Ut) =
(S66)
1=
while 1p((",U) is given by
N-11-1
N
(,(1"U) = Σ
Σ
14:=
Σ (k: 1)1 =
(S67)
1=1
Therefore, the expected value of pe(t", UU) over Pis
N Σ
N
Epluc(t", = Ep
- 1)](tt= = =t}
Σ
(N 1) Prite =*],
t",,
(S68)
A=1
k=1
the expected value up((") is
N t3 = 1k-1) = t']
Eplup(t',U) Ep Σ (k - 1)1 1)
(S69)
and their ratio is given by
Epluo(t" , U]]
N ( Pritx E (k- - Pr|tx [*]
(S70)
Eplpp((' U)]
During bidirectional training, each time the token 1" is masked at position k, it is predicted by N - 1 tokens. We assume that the token t" significantly contributes to the context of a masked token only if it is itself not masked (see Appendix assume that the probability p of masking any given position is the same for all positions and does
A.3). Additionally, not depend on the specific token tg that position. As such, we define masking an independent Bernoulli variable my Σn + {0.1} ~ Pm such that Pr[ms, = 1] p, independently of the token and sequence spaces. It follows that Ho(t", m) is given by
N Σ k=1
1tt = =t {1mm= = 0} Σ
1imm = 1},
(S71)
k'tk
while (t", m) is given by
up U, = 11tt== 13 Σ 1m =0..
(S72)
k=1
k'
Therefore, by taking the expectation with respect to both distributions P and Pm. their respective expected values are given by
N
Ep. [pe(t", U, m\] = Ep..p..
Σ
1fth (m = =0} Σ
1m = =}
k/
N
Ep, Pm
litt 111mmm -0 Σ
1 1m = 1}
k/
Σ
Ep, P.. [1lt2x- = t- {m = OHH Σ
[1 [1 1}]
(S73)
N
Pr(ta (r)Pr(m 0) Σ
Pr(me 1)
k=
N Σ
NP - 1)(1 - p)
Pr(ta t*),
km
22

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
and
N 1[ts t- = 11 Σ
P.. (up(t" U, U,m]] Pm..m
1 { = 0}
k k
(S74)
=Np(N 1)(1 p)
Pr(ta = =)),
and their ratio is
Erlueet" U} Eplup (1*, U)
=1, 1,
(S75)
where we omit the dependence on the independent random variable m for simplicity, thus concluding the proof.
The following remark highlights how, during autoregressive training, the ratio of expected token counts can vary depending on the correlations and conditional dependencies among the tokens.
Remark A.10. We separately analyze the following two cases:
No statistical correlation between tokens. Assume that the tokens are statistically independent and identically distributed (i.i.d.) across positions, i.e., the probability of each token is the same at every position and does not depend surrounding tokens. Under this assumption, during autoregressive training, the expected number of times token t is used to predict future tokens is given by,
N
Σ NN k) Pr[('] N(N - 1)
N
Ep(ua((",U))
Σ -N-
kr) = [*]
= - Pr(("]
(S76)
2
k=1
k=1
Similarly, the expected number of times token t" predicted by preceding tokens is,
N Σ (k (k t*] = Pr[t"] Σ (k- 1k- 1 N(N - 1)
(t(mp(t"
(S77)
Therefore, when tokens are independent and identically distributed, the expected number of tokens predicted by t" and the expected number of tokens used to predict t* are equal. In this case, there is no difference between autoregressive and bidirectional training. Statistical correlation between tokens. Let us assume that the probability distribution P encodes statistical dependencies between tokens. Define ju(tt) as the expected position index of token t²,
Σ k= k Pr = Σ Prita t+]
((tt)
(S78)
Using this, the ratio of expected counts under autoregressive training becomes,
Epluc(!", UJ] N- m(()) Er,(up(t",U)
(S79)
n(+*)
This expression shows that if t" tends to appear earlier in the sequence, that is, if ju(²) < (N + 1)/2 (left of center), then the ratio is greater than 1. Conversely, if L" appears later in the sequence (right of center), the ratio is less than 1. More formally, suppose there exists such that
N +1
m(t")
-
(S80)
2
Then the ratio can be rewritten as,
Epluo (t,,))
N + 1 22p(t") - 1
(S81)
EP Hp (1"),
m((²)= -1
where the numerator satisfies N + 1- (1((t)) 2 28, and the denominator is positive for any token that can appear at position 1 (i.e., p((*)= > 1). Hence, the ratio is bounded below by,
Ep (t) ((",))) [U] Er(p, (t+, UU] 2 1 + =(tt), with <(t") 1u(*) -
25
(S82)
23

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
A.5.4. TAIL PROBABILITIES OF DISTRIBUTIONS WITH EQUAL MEAN AND DIFFERENT VARIANCES
Before proving column dominance in the next Theorem, we first establish that probability distribution with higher variance is more likely to produce large values. Specifically, when two distributions share the same mean but differ in variance, the one with higher variance has greater probability of generating samples that exceed given threshold. We formalize this in the following lemma.
Lemma A.11. Let and b be two probability distributions with the same mean ps, but different variances such that o¹ o- Then, for all values >> Vouoh 1s, the probability that random sample from distribution exceeds is greater than or equal to the probability that a sample from distribution b exceeds 2, that is.
Pr[X. > 2] Pr[X,
z| for all 2 > vooob μ.
(S83)
Proof. Our goal is find a such that
Pr [X >
2 Pr [X 2|
(S84)
for which we need the following steps: lower bound for Pr [Xx > z|. an upper bound for [Xs 2]. and a value of that makes the upper bound lower than the lower bound.
We start by using Chebyshev's inequality, to derive an upper bound for Pr [X, >
z].
Pr X6 > 2] = Pr [X. > gaol S q2
=
σbb
(S85)
Now compute a lower bound for Pr [X > =] through the second moment method (which is very similar to the previous one, but inverted),
(E Xxa - =
Pr > 2| Pr [Xa- = 0j
11 2
(S86)
E
(Xa 2)²
aa
Thus, we now only need to find satisfying
2
2 - j 2
Obb
<
(S87)
1- 1
for the case 2 >> μr, the inequality is fulfilled when
>> Vo - μ.
(S88)
A.5.5. PROOF OF THEOREM 2.3
Here, we show that the weight updates of Wyx induce column dominance during autoregressive training, leading the emergence of directionality. To do so, we make the following assumptions:
We adopt the same assumptions stated in Propositions A.7 and A.9.
At initialization, the entries of Wa qk are drawn from a probability distribution P with finite mean μ and variance o². This is satisfied by any standard machine learning initialization scheme. This second assumption implies that each k-th column w.k and m-th row Wm,, have the same mean j and variance o²/ /²/ Furthermore, the squared norm of both rows and columns has equal mean n(o² + 222) and variance n220 + 4/12).
Proof. Let (UI...., }D} be a dataset of sequences of length N drawn i..dd from distribution P. Let k. denote the position of token 1* in sequence U.. and let superscript u refer to tokens in the w-th sequence. Following Proposition 2.2 and A.7, we obtain
!
3 1x3]|N
(S89)
(=
24

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training while the squared norm of the weight update of the m-th row by
n
Ry --
A mm..
Σ
Σ
Pulximm + [y]m
M
M
(S90)
i= +1
j=1
where in both equations the superscript u indicates tokens belonging to the u-th sequence in the dataset, and k2 indicates the position index of t*, follows again from Proposition A.7 that the expected value of the squared norm of these updates is
ED
Aw 11 ED m.
I' Tre) N= Li (4.) + MRA (I D Ef=
(S91)
2mm (T Ex 3%) + mm Tr(2)) D 1..-11 (")
Ec=I
Following Proposition A.9, let po(t", U) be the random variable quantifying the number of tokens predicted by t", while ['p(t, U) is the random variable quantifying the number of tokens predicted by t². When assuming autoregressive training, it follows that the empirical average of pe(l", U) and uy (t", U) over the dataset (U1....,UD: approaches,
Epluc(( (U,,)) Ep uup (t", [U]
( =t t*]
(S92)
Ej= 1) Pr(ly = -M]
Following Remark A.10, assume that the expected position jut*)) of to satisfies pltt) < (N + 11/2 - for some >> 0, then
Erlua((" U)]
> 1 + c(f")
(S93)
E Ep Ep (t U)
for some e(t*) 0. Hence, across the dataset, appears more often context token than as a predicted token. This implies Ep [ky] (N 1)/2, that the first terms in the numerator and denominator of Equation (S91) dominate, leading to,
E Aw
| [.]
>,
(S94)
E
that is, the net increase in column norms exceeds that of row norms. Assume Wok gk is initialized with i.i.d. entries from a distribution with mean j and variance o², then at the beginning of training Var([[w..xll) = Var(Iw,.. while after training we have Var[] &Il) Var( [m, II) Vk, m. Applying Lemma A.11, it follows that for all w % where
7 - Var([[ well- Var[[] (/,] - μ.
Pr[l/w,,all > w| Pr[l[wm,.11 w| Vu > 2
(S95)
thus concluding the proof.
A.5.6. PROOF OF THEOREM 2.4 AND RELATED REMARKS
Finally, we prove that the weight updates of W gk induce symmetry during bidirectional training. Importantly, the column dominance is present only during autoregressive training. Indeed, it follows from Proposition A.9 that, during bidirectional training, the net increase of the norm of the columns is equal to the net increase of the norm of the rows. We formalize this in the following Corollary.
Corollary A.12. (Bidirectional training does not induce directionality) Let V = (lo...., .......,,v]] (v be a vocabulary of tokens, and letld C y denote the sample space of all sequences of length N. Les U = (tI....,tx' E U be a random variable with U ~ PUU). Define Pr[t, = [*] as the marginal probability that the token at position j equals t* E V. Let (x xN be the token embeddings corresponding to the elements of U, where each embedding x; ~ 'D is drawn i.i.d. from a distribution D with zero mean and covariance matrix Cov(x:) E.. Let Wak. be query-key matrix of self-attention mechanism, and let A A.W.. als denote its gradient update defined in Proposition 2.1, computed under an autoregressive objective as in Definition A.3. Then, the variance of the norm of the rows and the colunns at the end of training is equal,
Var (1) (/W-mlll -I)) = Var ([] (llw,,ll) Yk:, m.
(S96)
25

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Proof. Let (U1:.... }D} be dataset of sequences of length N drawn i.i.d. from distribution P. Let ki denote the position of token ' in sequence Un. and let superseript u refer to tokens in the u-th sequence. Following Proposition 2.2 and A.7, we obtain
Ru
y
x
Payl
(S97)
f=
while the squared norm of the weight update of the m-th row by
D
Rv
Awm. m.
Σ
Σ Ru
Pi, x Imy
x
(S98)
where in both equations the superscript indicates tokens belonging to the u-th sequence in the dataset, and k2 indicates the position index of t", It follows again from Proposition A.7 that the expected value of the squared norm of these updates is
TEA Tr)) E Ee Ph) + EES Tr(T) (E -1 ( Ep W m. 112 Emmo Tr(T) (T Disk, +1 (..) - mm Tr(E)) (E E
ED [|] &
(S99)
- (3)
Following Proposition A.9, let (t(tt, U) be the random variable quantifying the number of tokens predicted by t", while Up((", U) is the random variable quantifying the number of tokens predicted by 1", When assuming bidirectional training training. it follows that the empirical average of pe(t", UU) and pp(t", U) over the dataset {UI..... UD} approaches,
Eplua(t (U]] =1
(S100)
Eplpp U)
From Proposition A.7, it follows that the net increase in the norms of the columns and rows of W gt is the same. Assuming the same initialization conditions as in Theorem 2.3, it follows that by the end of training,
Var Wm. = (elll) A k, m,
(S101)
which concludes the proof.
We now prove Theorem 2.4, showing that the bidirectional training objective induces approximate symmetry the gradient contributions of each token pair (i. Specifically, we assume that predicting token 4 from &j is correlated with predicting i, from t,, that is, the terms Big and Bjs from Proposition 2.1 are correlated, leading to similar contributions in both directions.
Proof. Let = (t....., ] be a sequence of tokens. It follows from Proposition 2.1 that the implciit weight update for W gb: following the gradient of L(U) w.r.t. Wak gt is given by
AW ok
Σ
Kij.
(S102)
where we neglect any constant of proportionality (e.g. learning rate) for simplicity. The double summation in Equation (S102) contains N2 elements. Note that K ji k. so can rewrite the double summation as follows,
N
N
N
N
K
B K
+
(B + B,,KK,
(S103)
=1
where the first term includes the diagonal terms, and the second includes the contributions of every pair (i. with j [O,..., Nj. The second term can be written as,
Kiy B,,K))
Σ
(Big + By,Kig T),
(S104)
+==1
26

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training and by decomposing iyy in its symmetric and skew-symmetric parts, such that Kjj -S Nij. obtain,
N
(B, K. + B,, 1) =
[Sy(S,, + +
+ Byl(S + NN
f,f=1
(S105)
Σ
[(3, + Bj.)s, + (Bis - B,,)
=
Let Big and Byx be such that (ig) = sign( (3zz) and 13yl 1133o|1 By defining A AW WWw.leees Mesess = BigKij + 8y.Kis" it follows that
N
AW
Σ
S,
Σ
B.s
AW ok Te,ett
(S106)
ij=
K<j
thus concluding the proof.
Encoder-only models are typically not trained to predict every token in sequence, but rather a random subset of tokens, and the model can attend to tokens bidirectionally. This is usually called Masked Language Modeling (MLM) (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Warner et al., 2024). Therefore, only subset of terms the double summation of Equation (S105) has the symmetric properties described above. We generalize the proof this case in the following.
Remark A.13. Let C; = [0, 1,...,,.. N] and let the summation indexed by to run over a random subset of tokens M C [0, 1, N.. The weight update of W, qb is then given by
N >E KK
AW gh
(S107)
SEM
The double summation contains N [M] elements, where |M is the cardinality of the subset M. We can rewrite the double summation as follows,
ΣΣ 3=1
Σ SEM
K Σ
+
3 K E LEM JEM
(S108)
JEM
i<j
where the first term includes the diagonal terms, the second includes the contributions of the pairs (i,j) with i, E M, and the third includes the remaining terms with = [l...., .N M. The second term can be written as,
K k = ( + B),K!
(S109)
GEEM
iJEM K
Kj
and by decomposing k!-1 in its symmetric and skew-symmetric parts, such that K = 1 =s + N we obtain,
C + 1 Σ
(S + N (S' N 1
ijEM iS
(S110)
= Σ
[[3) + + + (8) - 1
ijEM
with similar structure as in Equation (S105).
Let M = pN with 0 < < 1 being the percentage tokens to be predicted during bidirectional training. The total number of pairs in the second term of Equation (S108) is given by a binomial coefficient, thus the total number of elements in the summation pN(pN - 1). The total number of elements in the third term instead the product pN(N pN).
27

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Therefore, the percentage of symmetric weight updates from the second term over the total number of updates in the third term is given by
( - pN(N PNN (N
P PN)
2
(S111)
in the limit of large N. In practice, p is set to be around 15%-30% (Devlin al., 2019; Liu al., 2019; Lan al., 2020; Warner al., 2024), leading to 25% of symmetric weight updates average.
A.6. Properties of the symmetry score in Definition 3.1 and related proofs
The score introduce in Section 3 indicates the degree of symmetry of matrix M by quantifying the contribution the Frobenious norm of its symmetric and skew-symmetric parts. In particular, equals 1 and -1 for fully symmetric and skew-symmetric matrix. Accordingly, positive (negative) values of indicate the presence of symmetric (skew-symmetrig structures. Here, we provide proof for these properties. First, we show that the Frobenious norm of any square matrix M can be decomposed in the sum of the Frohenious norm of its symmetric and skew-symmetric components, as in the following Lemma,
Lemma A.14. For any square matrix M E Ml, the following equivalence holds
[M| = | + ||M,,
(S112)
Proof. The Frobenius norm of a matrix M can be defined as square matrix M we get
M F V TM(MM and as such we observe that for any
M F Tr(MMT) V Tr((M, (,,,M, + M.)7)
(S113)
(T((M,MT) Tr(M,MT) T(MM)MJ Tr(M,M.))
It follows from the cyclic property of the trace operator that the mixing terms cancel out as follows,
Tr(M,M,)) + Tr(M,M)) -T((MMM.)) + Tr(M,M,)) 0,
(S114)
resulting in
|M]] TT(MMMT) Tr(M,,.).
(S115)
Therefore, as both terms on the right-hand side are semi-positive definite, we conclude the proof follows,
|M] = Tr(M.M]) + (r = |M + |,
(S116)
Next, we formulate the properties of the symmetry score as follows,
Proposition A.15. The symmetry score quantifies the degree of symmetry or skew-symmetry of a given square matrix M. In particular,
1) The symmetry score s is a scalar value bounded in the range [-1,
2) A symmetry score s = =1 indicates a fully symmetric or skew-symmetric matrix, respectively.
3) The symmetry score of random matrix M E M,, with entries M,j p(0, a)) from a probability distribution with zero mean and finite variance lendy to zero 8/n in the limit cc.
Proof. To prove the points (1) and (2), we first show that follows from Lemma A.14 that the squared Frobenious norm of M, and M, in an orthogonal relation
M
M & + 3
(S117)
Therefore, for any given M, the norms | and MM 113 are such that a higher value of the first leads to to a lower value of the second, and vice versa. particular, straightforward to observe that ||M]la ||M]] and |M, if M
28

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
symmetric. Next, derive decomposition of the squared Frobenious of the symmetric and skew-symmetric part of M. From the definition of M,, obtain that
M, 112 Tr(M,M) = TT - M + +M]
[TMMM Tr(MM) TM(M'M + T((MMM)
(S118)
[T(MMM)) Tr(MM)]
= 2 [ + Tr(MM).
Since the upper bound for | is given by [M] the second term the left-hand side has upper bound given by,
Tr(M) S 2 |
1
(S119)
2
A complementary relation holds for the skew-symmetric component of M,
M, 13 = Tr((M - MJ (M - M)] [IMI -Tr(M)))
(S120)
which, following the same logic, defines a lower-bound for Tr(MM)) as follows,
S Tr(M)).
(S121)
Given Definition 3.1 we can write,
M + Tr(MM) [M + Tr(MM) Tr(MM)
(S122)
M
|M
and by combining the bounds derived previously we obtain,
-18851
(S123)
with
8 ==1 if M MT {
(S124)
-1 if M -M
To prove the point (3), let each entry may M be an independent, identically distributed sample from random distribution with mean and a finite variance o², We compute the Frobenius norm of the symmetric and skew-symmetric parts as follows,
MM 12.- = E + +2,)²++ + Σ (2M,)
(S125)
|M,
13. EM,, M,i)
Here, the skew-symmetric part has a zero diagonal term (because of the subtraction), and the symmetric part has twice the diagonal of the original matrix M (because the addition). Since the entries are independent, Mij is independent of My, for all A and thus we can treat the off-diagonal entries of the M, and M, terms as sum and difference of two and the same variance. It follows that the resulting distribution has mean
independent random samples having zero and a variance of 20² in both cases,
(M, +M,) =2
(M, M,,)² s n(n Var Myj + M,s] = n(n - 1)20²
(S126)
n
29

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
where the approximation is due to the central limit theorem. Applying similar logic to the second term on the symmetric norm, each entry is the double of random i.i.d. distribution with
N
(2M,)²
Var [aa] = n4o".
(S127)
N
Finally, we take the Frobenius norm of the random matrix itself and apply the same logic, where there are n²² entries with variance of o²
M n-+00
n²o²
(S128)
follows that the symmetry score is given by
s M 2
M
8 8 n2n² n
(S129)
M
where the symmetry score is zero in the limit n + oo with convergence from the positive side.
A.7. Properties of the directionality score in Definition 3.2 and related proofs
The score d we introduce in Section 3 quantifies the directional bias of a square matrix M by comparing the total norm of the "outliers" rows and columns, that is, that are higher than y times the standard deviations of the norms. A directionality score d of I indicates the presence of rows with high "outlier" norms and the absence of outliers the distribution of the column norms. The opposite true for directionality score of -1. Accordingly, positive (negative) values of d indicate the presence of row (column) dominance in the matrix. Here, we provide a proof for these properties.
Proposition A.16. The symmetry score d provides a quantitative measure of the degree of directional bias in a given square matrix M.
I) The divectionality score d is scalar value that lies within the range [-1,
2) For any given > O, directionality score d = #1 indicates that vectors satisfying the condition defined by ry are exclusively present in the rows or the column distribution, respectively.
3) The directionality score of a random matrix M E M., with entries My ~ p(0, σ) from a probability distribution with zero mean and a variance that scales as O(n 1) tends to zero in the limit n - co.
Proof. To prove that the directionality score bounded the interval [-1, 1]. note that čM: FM > simply because they are sums of norms. As both are positive,
CM FMl < CM < CM + FM
(S130)
and thus
MM FM
<1
(S131)
ÕM + FM
and taking the negative sign for the absolute value,
FM CM
CM FM
>
-1
> -1.
(S132)
CM + FM
CM + FM
In the extremes d +1, the numerator must be equal to the denominator in absolute value, implying that either FM or CM zero and the other is positive. For completeness, we define the score as zero if both are zero.
Finally, we study the case of a random matrix. We start by noting that the values of 7M; ?M are interchanged when we take the transpose, hence
FM EMT
(S133)
Regardless of the scaling of the matrix and the value 7, the key property of random matrix is that all entries are drawn from the same distribution. Hence,
Pr MM, = x] = Pr = = =
M x
(S134)
30

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Configuration
BERT
BERT-Mini
BERT-Large
Hidden Size
768
256
1024
Intermediate Size
3072
1024
4096
Number of Attention Heads
12
4
16
Number of Hidden Layers
12
4
24
Attention Dropout Probability
0.1
0.1
0.1
Hidden Activation Function
gelu
gelu
gelu
Hidden Dropout Probability
0.1
0.1
0.1
Layer Normalization Epsilon
le-12
le-12
le-12
Max Position Embeddings
512
512
512
Position Embedding Type
absolute
absolute
absolute
Vocabulary Size
30522
30522
30522
Table I. Configurations for BERT, BERT-Mini, and BERT-Large models.
for ir and X being any arbitrary value or matrix. As consequence,
Pr [iM = x] = Pr [JM = x]
Pr [iM = x]
(S135)
where the last equality comes from Eq. S133. The main point here is that the probability distribution of both rows and columns is the same. Pushing this forward, the expected value of TM êM is
E [FM [m] = E [FM]] - [E(Mm]
CMPPIMMJMM - /NPPTMM
[NMPr|M]dM dM
(S136)
= / FMPr M dM
- -M
(S137)
Furthermore, the expected value of FM + ōM strictly positive, since both values are positive. Thus, their ratio, the directionality score of random matrix, is zero.
Notice that to be thorough we must show that their variance is bounded scales down. Since weight initialization has been extensively studied, we will just make a general reference to it here. In machine learning, all weights are initialized with zero mean and variances that scale as O(n As M is a product of two matrices with such scaling, each entry would consist of the sum
random variables, where each one has a scaling of O(n -?) since it the product two random variables with an O(n !) scaling. Thus, the entries of M also have scaling of O(n-1) Applying the mean value theorem gives the desired result.
B. Experimental Details
We trained three BERT models (Devlin et al., 2019) to examine the evolution of the symmetry and directionality scores throughout the training process. Detailed information regarding the training procedure provided below.
B.1. Models
We train the standard BERT model (referred to as BERT), a smaller version (referred to as BERT-Mini), and larger version (referred to as BERT-Large), following the implementation by (Devlin al., 2019). Table provides an overview of the model parameters. The standard BERT model has 12 layers, 12 attention heads, and embedding dimensions of 768 for the hidden layers and 3072 for the intermediate layers. In contrast, the smaller BERT-Mini model uses 4 layers, 4 attention heads, and embedding dimensions of 256 and 1024, respectively. The larger BERT-Large model features 24 layers, 16 attention heads, and embedding dimensions of 1024 and 4096 for the hidden and intermediate layers, respectively.
We optionally initialize the models using symmetric attention weights. Specifically, each self-attention block, the query weight matrix Wg is initialized randomly, and the key weight matrix set equal to it, Wx = Wq. This ensures that the key and query matrices are identical initialization, introducing symmetry into the attention mechanism.
Initialization
31

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Configuration
VIT (6 layers) CIFAR-10
ViT (12 layers)
ImageNet-Ik
Hidden Size
512
768
Intermediate Size
2048
3072
Number of Attention Heads
8
12
Number of Hidden Layers
6
12
Attention Dropout Prohability
0.0
0.0
Hidden Activation Function
gelu
gelu
Hidden Dropout Probability
0.0
0.0
Layer Normalization Epsilon
le-6
le-6
Patch Size
4
16
QKV Bias
true
true
Encoder Stride
16
16
Table 2. Configurations for the 6-layer and 12-layer vision transformer models.
B.2. Datasets
The models are trained on three datasets. First, we use the "20220301.ee en" snapshot from the Wikipedia dataset, which consists of 6.46 million samples crawled from Wikipedia. Second, we utilize the Jigsaw dataset with 159K samples, originally collected for a toxic comment classification challenge. Finally, we train the English "2023-14" snapshot of the RedPajama-V V2 dataset, which contains approximately 5.12 billion samples.
B.3. Training Settings
The models are trained for 200, 000 update steps with batch size of 32 and 8 gradient accumulation steps, effectively increasing the batch size to 256 before each parameter update. The optimization is done using the Adam W optimizer (Loshchilov & Hutter, 2019), and the training schedule includes 200 warmup steps to stabilize early training, followed by a linear decay learning rate schedule, starting at an initial learning rate of x 10-5 and weight decay of 0.01. Mixed precision (fp16) training was utilized to maximize training efficiency, which reduces memory consumption and speeds up computation without significant loss of precision. The training data was processed with a masked language modeling (MLM) probability of 15%, ensuring that 15% of tokens were masked during training. The models are trained in the encoder and decoder mode, i... to predict masked tokens and subsequent tokens respectively.
C. Enforcing symmetry at initialization for vision transformers
In parallel to the experiments described in Section 4.2, where symmetric initialization of the query-key weight matrix W is investigated for BERT models, we extend the investigation to the vision domain. Specifically, we evaluate the impac of symmetric initialization in vision transformers (ViTs) (Dosovitskiy et al., 2021), training a 6-layer VIT on CIFAR- 10 (Krizhevsky, 2009) and 12-layer VIT on ImageNet- (Deng et al., 2009), using the same initialization strategy as applied to BERT models (see Appendix B.1 for implementation details). The architectural configurations for both models are summarized in Table 2.
gte
Training is performed using the AdamW optimizer (Loshchilov & Hutter, 2019) with weight decay and a cosine annealing learning rate schedule. A linear warm-up phase is used during the first 30 epochs, with an initial learning rate scaled by a factor of 0.033. The ImageNet- Ik model trained for 200 epochs using a per-device batch size of 256 and gradient accumulation over two steps, while the CIFAR-10 10 model is trained for 500 epochs with batch size of 128 and no gradient accumulation. Both setups use base learning rate of 0.003, optimizer hyperparameters |3 = 0.9 and 33 0.999, and automatic mixed-precision training (fp16). We apply advanced data augmentation tailored to each dataset, including mixup (a = 0.2) (Zhang et al., 2018), cutmix (a 1.0) (Yun et al., 2019), and label smoothing (0. 11 for ImageNet- 1 and 0.01 for 10). An exponential moving average (EMA) of the model weights (Tarvainen & Valpola, 2017) is maintained throughout training (decay rate- = 0.99998, update frequency = 32 steps).
Contrary to the trends observed in our language model experiments, symmetric initialization does not result in faster
32

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Table Final training and evaluation metrics for ViT models CIFAR-I0 (Krizhevsky, 2009) and ImageNet-Il Ik (Deng al., 2009), trained with and without symmetric initialization. Evaluation metrics include final loss and top-1 accuracy. For each dataset, we compare models trained with standard initialization and with symmetric initialization.
DATASET
TRAIN Loss
EVAL Loss
EVAL ACCURACY
CIFAR-I0
VIT 6-LAYERS
1.07
0.40
0.90
VIT 6-LAYERS (+ SYMM.)
1.08
0.42
0.89
IMAGENET-IK
VIT 12-LAYERS
2.49
1.97
0.76
VIT 12-LAYERS (+ SYMM.)
2.49
1.96
0.76
convergence or improved final performance the vision transformer models. shown in Table 3, training and evaluation losses, as well as top-1 accuracy. remained nearly identical between the standard and symmetric initialization conditions across both datasets. Importantly, the results with symmetric initialization at least performance-neutraal in vision setting. While findings suggest that symmetric initialization does not yield immediate benefits for ViT' under the given training regime, we do not rule out the possibility that alternative configurations or optimization strategies could leverage its potential. Further research may uncover conditions under which vision transformer architectures can henefit more substantially from symmetric initialization.
degraded, indicating that such initialization
D. Supplementary Figures
a
vision
audio
b
vision
audio
1.0
1.0
directionality score
symmetry score
0.5
0.5
0.0
0.55
encoder-only decoder-only
0..5
-1.0
10
109
_0
107
10a
10
10'
10
10
107
10
10
# parameters
# parameters
parameters
parameters
Figure S1. a) Median symmetry score of the matrix Wy as function of the total number of parameters for vision and audio models. Each dot corresponds to median and interquartile range across layers a given pre-trained model. Same as median directionality score of the matrix W gk-
E. Supplementary Tables
33

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Table 4. Symmetry score for open-source pretrained language models. All models are available on Hugging Face (Wolf et al., 2020).
Model
Median
Interquartile range
Model
Median
Interquartile range
BERT-tiny
0.77
- [0.07, 0.07]
GPT-Neo1.3B
0.14
[0.03, 0.03]
BERT-mini
0.62
[0.03, 0.05)
GPT-Neo2.7B
0.13
[0.02, 0.04]
BERT-small
0.69
+ [0.10, 0.08]
GPT-J6B
0.11
[0.02, 0.03]
BERT-medium
0.60
+ [0.01, 0.02]
OpenAI-GPT
0.07
t 10.04, 0.03]
BERT-base
0.51
+ [0.09, 0.07]
GPT2-XL
0.12
+ [0.03, 0.05]
BERT-large
0.44
+ [0.03, 0.08)
DistilGPT2
0.19
+ [0.05, 0.05]
DistiBBRTT
0.43
- [0.10, 0.13]
GPT-Neo
0.14
+ [0.09, 0.14]
BERT-2IL-128
0.77
+ [0.07, 0.07]
GPT-Neo1.5
0.14
10.03, 0.03]
BERT-4L-256
0.62
[0.03, 0.05]
GPT-Nco2.7B
0.14
[0.03, 0.02]
BERT-4L-512
0.69
[0.10. 0.08]
GPT-J6B
0.11
- [0.02, 0.03]
BERT-8L-512
0.60
+ [0.01, 0.02]
LLaMA2-7B
0.12
: [0.02, 0.03]
BERT-base
0.51
+ [0.09, 0.07]
LLaMA2-13B
0.17
+ 10.02, 0.02]
BERT-large
0.44
- [0.03, 0.08]
LLaMA3-8B
0.00
+ [0.00, 0.01]
DistilBERT
0.43
+ [0.10, 0.13]
LLaMA3.1-8P
0.00
+ [0.00, 0.01]
BEiT-base
0.40
+ [0.08, 0.02]
LLaMA3.2-8B
0.01
[0.01, 0.01]
large
0.33
+ [0.05, 0.07]
LLaMA3.2-IB
0.01
[0.01, 0.01]
BBiT-
0.39
+ [0.23, 0.07]
LLaMA3.2-3B
0.01
[0.01, 0.01]
BaiTe
0.26
+ [0.17. 0.13]
LLaMA2-7B-ae
0.12
+ [0.02, 0.03]
BEiT-base
0.39
+ [0.23, 0.07]
LLaMA2-70B
0.02
+ [0.01, 0.02]
BEiT-large
0.26
t [0.17, 0.13]
LLaMA2-7B-chat
0.12
+ 10.02,
BEiT-large
0,26
+ [0.17. 0.13]
LLaMA2-13B-od
0.17
+ [0.02, 0.02]
ALBERT-base
0.72
- [0.00, 0.00]
LLaMA3-SB
0.00
+ [0.00, 0.00]
ALBERT-large
0.70
- [0.00, 0.00]
LLaMA3-70B
0.02
[0.01, 0.01]
ALBERT-xlarge
0.59
[0.00, 0.00]
LLaMA3.
0.00
[0.00, 0.011
ALBERT-xxlarge
0.46
+ [0.00. 0.00]
LLaMA3.
0.01
- [0.00, 0.01]
RoBERTa-bar
0.49
+ [0.03, 0.06]
LLaMA3.1-4055
0.03
+ [0.01, 0.03]
RoBERTa-lare
0.47
+ [0.06, 0.06]
LLaMA3.2-1B
0.00
- [0.00, 0.00]
XLM-R-bar -
0.51
+ [0.05, 0.03]
LLaMA3.2-3B
0.01
+ [0.01, 0.01]
XLM-R-large
0.49
+ [0.16, 0.12]
Mistral-7B
0.00
+ [0.00, 0.01]
RoBERTa-mnli
0.47
- [0.06, 0.06]
0.00
- [0.00, 0.00]
DistilRoBERTa
0.53
+ [0.02, 0.06]
MobileLLM125M
0.03
[0.02, 0.03]
ModemnBERT-base
0.18
[0.06, 0.18]
MobileLLM330M
0.01
[0.01, 0.01]
GPTI
0.07
+ [0.04, 0.03]
Phi-1.5
0.09
+ [0.03, 0.03]
GPT2
0.15
+ [0.02. 0.03]
Phi-1
0.14
+ [0.02, 0.01]
GPT2-medium
0.17
+ [0.03, 0.05]
Phi-2
0.07
+ [0.03, 0.06]
34

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
Table 5. Directionality score for open-source pretrained language models. All models are available on Hugging Face (Wolf et al., 2020).
Model
Median
Interquartile range
Model
Median
Interquartile range
BiRT-tiny
-0.79
- [0.11, 0.11]
GPT-Neo1.3B
-0.49
[0.19, 0.13]
BERT-mini
-0.33
[0.03, 0.04]
GPT-Neo2.7B
-0.57
[0.15, 0.16]
BERT-small
-0.22
+ [0.04, 0.03]
GPT-J6B
-0.28
- [0.09, 0.08]
BERT-medium
-0.23
+ [0.06, 0.10]
OpenAI-GPT
-0.18
+ [0.08, 0.07]
BERT-base
-0.08
+ [0.02. 0.03]
GPT2-XL
-0.23
+ [0.11, 0.10]
BERT-large
-0.03
+ [0.02, 0.06]
DistilGPT2
-0.51
+ [0.03, 0.07]
DistiBBERT
-0.13
t [0.00, 0.06]
GPT-Neo125h
-0.56
+ [0.21, 0.08]
BERT-2L-128
-0.79
[0.11, 0.11]
GPT-Nco
-0.49
[0.19, 0.13]
BERT-4L-256
-0.33
- [0.03, 0.04]
GPT-Neo2.7B
-0.49
[0.15, 0.21]
BERT-4L-512
-0.22
- [0.04. 0.03]
GPT-J6B
-0.28
- [0.09, 0.08]
BERT-8L-512
-0.23
+ [0.06, 0.10]
LLaMA2-7B
-0266
+ [0.09, 0.13]
BERT-base
-0.08
- [0.02, 0.03]
LLaMA2-13B
-0.15
- [0.11, 0.03]
BERT-large
-0.03
+ [0.02, 0.06]
LLaMA3-8B
-0.65
+ [0.13, 0.20]
DistilBERT
-0.13
- [0.00. 0.06]
LLaMA3.1
-0.64
+ [0.17, 0.19]
BEiT-base
-0.10
+ [0.06, 0.15]
LLaMA3.2-8B
-0.59
+ [0.18, 0.22]
-argee
-0.15
1 [0.08, 0.07]
LLaMA3.2-IB
-0.59
+ [0.18, 0.22]
BEiT-base
-0.14
+ [0.15. 0.21]
LLaMA3.2-38
-0.77
- [0.08, 0.19]
Barree
-0.14
+ [0.04, 0.14]
LLaMA2-7B-CG
-0.29
+ 10.07, 0.14]
BEiT-base
-0.14
+ [0.15, 0.21]
LLaMA2-708
-0.24
- [0.10, 0.06]
BEiT-large
-0.14
t [0.04, 0.14]
LLaMA2-7B-char
-0.29
+ [0.07, 0.14]
BEiT-large
-0.15
- [0.03, 0,14]
LLaMA2-13B
-0.19
+ [0.12, 0.04]
ALBERT-base
-0.07
- [0.00, 0.00]
LLaMA3-8B
0.01
+ 10.05, 0.05]
ALBERT-large
-0.17
t [0.00, 0.00]
LLaMA3-70B
-0.37
+ 10.09, 0.12]
AL.BERT-xlarge
-0.24
[0.00, 0.00]
LLaMA3.
-0.57
[0.16, 0.13]
ALBERT-xxlarge
-0.15
+ [0.00, 0.00]
LLaMA3.
-0.37
- [0.08, 0.11]
RoBERTa-bass
-0.12
+ [0.11, 0.03]
LLaMA3.
-0.17
- 10.07, 0.07]
RoBeRTaa-lrge
-0.06
+ [0.03, 0.03]
LLAMA3.2-1B
-0.02
+ [0.13, 0.08]
XLM-R-base
-0.02
+ [0.02, 0.02]
LLaMA3.2-3B
-0700
+ [0.07, 0.22]
XLM-R-large
-0.02
+ [0.03, 0.02]
Mistral-7B
-0.88
+ [0.15, 0.13]
RoBERTa-mnli
-0.06
- [0.03, 0.03]
Mixtral Mixtral-8x22B
-0.66
- [0.09, 0.16]
Bisia
-0.14
[0.09, 0.07]
MobileLLMM12M
-0.13
[0.15, 0.10]
ModemmBERT-bass
-0.04
+ [0.05, 0.04]
MobileLLM350M
-0.34
+ [0.13, 0.23]
GPT1
-0.18
+ [0.08, 0.07]
Phi-1.5
-0.8
+ [0.22, 0.19]
GPT2
-0.58
+ [0.06, 0.14]
Phi-1
-0.40
+ [0.03, 0.04]
35

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
a
b
1.0
directionality score
symmetry score
0.5
0.5
0.0
-0.5
-0.5
encoder-unit decoder-unit
-1.0
107
10a
10
-1.0
10/
10a
10
# parameters
# parameters
Figure S2. a) Median symmetry score the matrix Wak as function of the total number of parameters for vision and audio models. Each dot corresponds to the median and the interquartile range across layers of the encoder component (blue) and decoder component (purple) an encoder-decoder Transformer model. The encoder component these models shows high degree of symmetry compared the decoder component. b) Same in for the median directionality score of the matrix W gk. The encoder and decoder components these models do not show significant differences directionality scores.
symmetry score
directionality score
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
0.5
encoder-only decoder-only
-1.0
-1.0
0
1000
20000
1000 epochs
2000
1000 epochs
2000
1000 epochs
2000
epochs
Figure a) Evolution symmetry score during training. Shown are median the interquartile range. Models were trained on the Jigsaw dataset (ejadams et al., 2017) (left) and on the Red Pajama dataset (Computer, 2023) (right). Encoder-only and decoder-only models are color-coded blue and purple, respectively (see legend). b) Same as panel median directionality score.
36

---

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training
a
symmetry
directionality score
1.0
1.0
0.5
0.5
0.0
0.0
-0.5
0.5
-1.0
-1.0 2000
1000 epochs
20000
1000 epochs
1000 epochs
20000
1000 epochs
2000
symmetry score
directionality score
1.0
1.0
0.5
0.0
0.0
-0.5
005
-1.0
11.0
0
1000 epochs
20000
1000 epochs
2000
1000 epochs
20000
1000 epochs
2000
Figure S4. a) Evolution of the median symmetry score across layers of the encoder-only (left) and decoder-only (right) models. Each layer is color-coded shown on the legend of Figure 3. Shown are the median and the interquartile range. Models were trained on the Jigsaw dataset (cjadams et al., 2017). b) Same in panel
median directionality score. e) Same as panel a for models trained on the Red Pajama dataset (Computer, 2023). d) Same as panel for the median directionality score.
a
b
w/out initialization w/ initialization
1.0
1.0
initialization
w/out initialization
symmetry score
0.5
symmetry score
0.5
layers
loss
0.0
-0.5
-0.5
100
1000 epochs
2000
-1.0
0
1000 epochs
2000
-1.0
2000 0
2000
epochs
epochs
Figure S5. a) Loss 4-layer encoder-only model with (dark bluc) and without (light blue) symmetric initialization, respectively. Models trained the Red Pajama dataset (Computer, 2023). b) Median symmetry during training Color code in panel e) Same as in panel for each layer in the model with and without symmetric initialization. Each layer is color-coded as in the illustration on the left. All plots show the median and the interquartile range across the heads of given layer.
37


Reviews:
[{"review_id": "Divm9ftkkc", "reviewer": "Reviewer_HwPP", "summary": "In this paper the authors focus on the structure of the attention matrix used in Transformers and in particular the effect of the training strategy on the overall structure inherited by the same. Showcasing that autoregressive training leads to directional matrices, whereas  bidirectional training induces symmetry, these insights are tested on a wide array of practical models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful review, for going through the proofs in the Appendix, and for the detailed feedback to improve clarity. While we appreciate the concerns raised, we respectfully disagree with the claim that the manuscript lacks the necessary mathematical rigor for its claims and propositions, and we outline our reasoning below.\n\nFirst, we acknowledge the mistake in Equation (S8), but this has no impact on the correctness of our proofs or results. Indeed, like with Equation (S6), (S8) was intended as a direct factorization of the joint distribution under bidirectional training. However, it was included only to motivate Equations (S9) and (S10) (the standard Masked Language Modeling (MLM) objective) which is what we use throughout the paper, and does not depend on (S8). We will remove (S8) and keep only (S9) and (S10).\n\nSecond, we address the weaknesses regarding the explanation of our mathematical results (1-2). We value the reviewer’s feedback and have restructured the presentation of our results to improve clarity. We emphasize that our core results focus on deriving the implicit gradient of self-attention matrices, under minimal assumptions about the data. Importantly, the results in Sections 2.1–2.2 and Proposition A.7 rely solely on properties of self-attention. We are confident that the following revisions will better highlight the intuitions and clarify the theoretical setup:\n\n- Section 2.1: We will keep the standard definition of self-attention and keep Equation (3) to connect it later with (8), (9), and Figure 1. Proposition 2.1 will be removed entirely, and only the essential content needed to link (3) to Proposition 2.2 will be preserved. Given these structural changes, we agree with the reviewer that Proposition 2.1 is not necessary for understanding the subsequent results. We will retain the reference to Section A.1 for relevant definitions and remove the proof of Proposition 2.1.\n\n- Section 2.2: We will present a general definition of the negative log-likelihood in Equation (6) without introducing $C_i$ at that point. Then, in Proposition 2.2, we will define both $C_i$ and $P_j$, followed by a clear explanation of how the gradient of $W_{qk}$ can be derived using these two equivalent summations. This will provide the foundation for introducing the two main theorems. \n\n- Section 2.3: We will enhance the explanation of how a token contributes to the gradient when used as context or prediction by explicitly including Proposition A.7 in this section. Up to this point, all mathematical results are derived purely from self-attention and do not assume anything about the input data. We will move the theorems to a new Section 2.4. \n\n- Section 2.4: We will formally state Theorems 2.3 and 2.4 with their assumptions: (1) There are statistical correlations between tokens, a weak and general assumption that holds in most real-world data. As a result, token embeddings exhibit partial alignment, capturing semantic and predictive structure. Indeed, this alignment either exists in pretrained embeddings or naturally emerges during training in learned embeddings, encoding semantic relationships; (2) Entries of $W_{qk}$ are i.i.d. at initialization with finite mean and variance; (3) Bidirectional training induces approximately symmetric error signals, that is, the error in predicting token $i$ from $j$ is similar to that of predicting $j$ from $i$. This new section will clarify why the theoretical setup broadly applies to real-world data and what predictions it enables.\n\n- Section 3-4: The current versions demonstrate how the predicted structures appear in Transformer models (Fig. 2 for language; Fig. S1 for vision and audio), emerge during training (Fig. 3), and can be leveraged in practical applications (Table 1). These results clearly show how our theoretical insights translate to real-world scenarios.\n\nFinally, we address the questions raised by the reviewer:\n1.  Yes, during symmetric initialization, we randomly initialize $W_q$ and set $W_k = W_q$, ensuring $W_qW_k^T$ is symmetric, as we understand the reviewer suggests. We also experimented with a regularization term to enforce symmetry during training, but it did not improve training speed or final loss compared to the baseline. For details, see point 3 in our response to reviewer jKhN.\n\n2. Our framework shows that, under bidirectional training, each token pair contributes to an approximately symmetric update of $W_{qk}$. However, it does not determine whether a fully symmetric $W_{qk}$ is \"ideal,\" as this would require a precise definition of \"ideal\" and an analysis of attention matrices at convergence. This is an interesting direction for future work. Additionally, Remark A.15 highlights that, in MLM, only a subset of updates is symmetric. As a result, we naturally expect, and have empirically observed, non-fully symmetric $W_{qk}$, though these still exhibit significantly higher symmetry scores than those from autoregressive training.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I felt that the paper lacked the prerequisite mathematical rigor for the claims and propositions in the main paper. In fact I found an important mistake about bi-directional training loss which I am not sure how it will change the main results. That is, the Equation (S8) in the appendix is not true. Take $N=2$ for example. The factorization of $P(t_1, t_2) = P(t_1|t_2) \\cdot P(t_2|t_1)$ is simply not true unless $t_1$ and $t_2$ are independent, which is not stated. \n\nWith regards to lack of mathematical rigor, following are some examples:\n\n1. What's the assumption on the input data for main theorems 2.3 and 2.4 to hold? Does it hold for all inputs? What's the probability over in Equation (13)? \n2. Likewise what's $P_j$  in Proposition 2.2. I only got to know its meaning in the appendix. \n3. Proposition 2.1 is a simple consequence of the fact that the linear attention scores and soft-max attention scores follow the same ordering because the latter is just the exponential of the former up to some scaling. So in the current form it sounded like a bit of fancy result with jargon like projections, subspaces and I didn't really see the need for it nor the main importance of this result, despite the explanation below which didn't fully sound satisfactory to me.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Authors cited all the relevant literature in connection to the paper", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n1. I think the authors study an important problem with regards to understanding attention structures and the final takeaways regarding their symmetry are quite interesting. In particular, trying symmetric initialization strategies and showing speedup is quite nice.\n\n2. I also appreciate the mathematical effort to compute the gradients and reorient them in a suitable way to obtain important insights about the directionality and symmetry of the attention matrices.\n\nWeaknesses:\n\n1. Mathematical rigor part highlighted above. Also the text explanation of the mathematical results is not fully satisfactory. It requires lot more polishing in translating the importance of intuition behind these results into a coherent set of paragraphs.\n2. Likewise, I feel the paper could benefit a lot from significant rewriting by clearly stating what's the setup for theoretical results is. And how and why these insights translate to practical experiments. Currently, it's unclear for what input data or scenarios, the results Theorems 2.3 and 2.4 hold and why they should translate to real data. Improving upon these points can help make the paper more crisp and direct.", "other_comments_or_suggestions": "See above", "questions_for_authors": "1. If I understand correctly, in Section 4.2 either you initialize randomly or with symmetry and let them free to train right? Is there any reason why you didn't try keeping it symmetric throughout the training? What happens in this scenario? For example, you can initialize $W_k = W_q$.\n\n2. Is symmetry score =1 the ideal scenario for bidirectional training? Or is there a reason why this is not always the best?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnVDiVNkNk", "reviewer": "Reviewer_JKhN", "summary": "This work investigates the training process of Transformers, revealing a structured pattern of the attention weights' update as a linear combination of rank-1 matrices. Based on this, it is demonstrated that bidirectional training (encoder-only) induces symmetry in weight matrices, while autoregressive training (decoder-only) results in directionality and column dominance. These phenomena are both mathematically proved and numerically verified.\n\n## Update after rebuttal\n\nAuthors have updated further experiments on large-scale language modeling tasks to demonstrate the effectiveness of symmetry initializations. However, this technique is probably not that valid on image datasets, which limits the applicability of the derived (core) insights of this work. Given the theoretical and (partial) practical contributions, I maintained the recommendation to be not against acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive feedback provided. \n\nBelow, we address the reviewer’s questions, along with the related concerns highlighted as potential weaknesses.\n\n1. Proposition 2.1 aimed to show that accurate token prediction depends on learning an effective bilinear form in the embedding space, represented by the implicit matrix product $\\mathbf{W}_{qk}$. However, we agree with the reviewer that this proposition is not essential. We will revise Section 2 accordingly by: (a) removing Proposition 2.1 and integrating its key insights into the main text, and (b) expanding Section 2.3 and adding a new Section 2.4 to present Theorems 2.3 and 2.4 in greater detail, making better use of the available space.\n\n2. We thank the reviewer for raising these points. The comments led to valuable additional experiments and analyses that helped strengthen the empirical support for our findings. To provide a clear response, we address the two questions separately:\n\n- 2.1 We conducted additional experiments using larger models and observed a consistent training speed-up, in line with our previous results. Specifically, we trained a BERT-large model (24 layers, 3x more parameters than Bert-Base) on the same three datasets used for BERT-mini and BERT-base, that is, Jigsaw, Wikipedia, and Red Pajama. Although full training runs are set to 200k steps, the current results are based on intermediate checkpoints. For Wikipedia, after 123k steps, the loss decreased from 0.2151 (without symmetric initialization) to 0.1874 (with symmetric initialization), yielding a 56.5% speed-up. For Jigsaw, after 184k steps, the loss dropped from 0.8113 to 0.7612, with an 11.0% speed-up. Similarly, for Red Pajama, after 140k steps, the loss improved from 0.2261 to 0.2058, with a 37.8% speed-up. We will include the final results in Table 1 of the revised manuscript. Furthermore, we found a clear positive correlation between increased symmetry from symmetric initialization and both faster training and lower final loss. For details, please refer to: https://drive.google.com/file/d/1nuAGfyjVAp9suVL0NydVPOTQK56_AtvP/view?usp=sharing. Finally, while the reviewer correctly notes a smaller relative gain in speed-up from BERT-mini to BERT-base, this is expected due to neural scaling laws (e.g., Kaplan et al., 2020) that make improvements harder at larger scales.\n\n- 2.2 We conducted preliminary experiments on training Vision Transformers on the CIFAR-10 and ImageNet-1k datasets. We did not observe a significant speed-up with these specific experiments. Nonetheless, we hypothesize that further analysis is necessary to check if symmetric initialization can speed up the training of vision Transformers. \n\n3. We have conducted experiments to enforce symmetry across layers by adding a regularization loss term, as follows:\n\n$L_{reg} = \\frac{\\| \\mathbf{W}^l_{qk}\\|}{\\| {\\mathbf{W}^l_{qk}}_s\\|} \\quad \\forall l \\in [0,L] ,$\n\nwhere the denominator is the symmetric component of the $\\mathbf{W}^l_{qk}$ matrix. However, this did not lead to noticeable improvements in training speed or final loss compared to the baseline. While we see training constraints that promote symmetry as a promising direction for future work, the specific regularization method we tested was not effective. For details on our results, please refer to: https://drive.google.com/file/d/1Paa7z6KxD11MdXyJyhU-xOUFqg6tJkSf/view?usp=share_link \n\n4. We thank the reviewer for raising this important point. In our current work, we have successfully explored methods to exploit symmetry. We are investigating several approaches to leverage column dominance to improve autoregressive training. These ongoing experiments will be fully addressed and presented in a dedicated future study.  \n\nOn the other comments and suggestions, we thank the reviewer for pointing out 3 typos in the manuscript. We will revise the manuscript accordingly.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed initialization strategy is inspired by the weights' symmetry pattern for bidirectional training, and hence quite reasonable.", "theoretical_claims": "I did not check every detail of mathematical proofs, but the theoretical derivations seem sound.", "experimental_designs_or_analyses": "The experimental designs/analyses in this submission align with and validate corresponding theoretical claims.", "relation_to_broader_scientific_literature": "This work generalizes prior results (e.g. Trockman & Kolter (2023)) to initialize query-key matrices as identities based on uncovered diagonal  patterns.", "essential_references_not_discussed": "To the best of my knowledge, I am not aware of any essential references not discussed in this submission.", "other_strengths_and_weaknesses": "Strengths:  \n\n1. This paper is well-written and easy to follow. \n2. There are rigorous mathematical analysis and consistent numerical verifications. \n\nWeaknesses: \n\n1. The paper length is around 7 pages, allowing to present more details in the main texts. \n2. Some of the current statements/discussions are repetitive and can be reduced (see details in the following \"Questions For Authors\" section).", "other_comments_or_suggestions": "Minor issues: \n\n1. Eq. (12): $[\\textbf{x}_i]_k$ -> $[\\textbf{x}_i]_m$. \n2. Section 6: $\\textbf{W}qk$ -> $\\textbf{W}_{qk}$. \n3. Line 1286: \"smaller\" -> \"larger\".", "questions_for_authors": "1. Proposition 2.1: It seems that Eq. (3) is just an enrollment of residual (single-head) self-attention, and Eq. (5) trivially holds due to the monotonicity of softmax operations. What is the significance of this proposition? \n2. Table 1: Note that the performance enhancements of symmetry initializations significantly degrade as the model depth increases. It would be more convincing to test for larger models. In addition, current experiments are all conducted on language datasets. Do similar enhancements appear for image datasets? \n3. Besides symmetry initialization, is it possible to also explore (possibly layer-wise) symmetry regularizations for further enhancements? \n4. How do we exploit the directionality pattern to improve the autoregressive training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8puFD7oK7d", "reviewer": "Reviewer_jZNG", "summary": "The paper investigates the inherent structures within self-attention mechanisms, focusing on symmetry, directionality, and emergent dynamics in Transformer training. The authors provide a mathematical framework for analyzing self-attention weight matrices and examine how different training objectives, namely bidirectional and autoregressive training, impact these structures. They argue that bidirectional training induces symmetric weight matrices, while autoregressive training results in directionality and column dominance. These findings are supported through theoretical derivations and empirical validation across multiple Transformer models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful, detailed, and valuable input. First, we address the weaknesses raised by the reviewer:\n\n1. We acknowledge that we did not evaluate real-world deployments, as our goal was to analyze the structures that emerge in self-attention matrices during pretraining. However, our findings suggest potential downstream benefits worth exploring in future work. Indeed, symmetric initialization speeds up training and leads to lower final loss, which often correlates with better downstream performance. \n\n2. We thank the reviewer for highlighting this point, and we agree that the clarity of some notations could be improved. We will revise our proofs accordingly, specifically Proposition A.10.\n\nNext, we address the questions raised by the reviewer:\n\n1. We have conducted symmetric initialization experiments on autoregressive models. Our observations indicate that these models quickly lose their initial symmetry, eventually converging to $W_{qk}$ matrices with low symmetry scores. Furthermore, we observed no significant improvement in training speed compared to standard initialization. For details, please refer to the plots of the loss curve and symmetry available here (12 layers model): https://drive.google.com/file/d/14Y8huSc7EajiLWiGjjQNM3-BPvRVCw-G/view?usp=sharing \n In contrast, as shown in the manuscript, encoder-only models clearly benefited from symmetric initialization, exhibiting faster training and higher overall symmetry scores.\n\n2. We thank the reviewer for raising this interesting point. We have not yet explored how to extend our mathematical framework to cross-attention mechanisms. Nonetheless, our empirical results show that the encoder components of encoder-decoder models consistently exhibit higher symmetry scores than the decoder components, and this difference holds across models of varying sizes (see Figure S2). This empirical evidence suggests that Theorem 2.4 can potentially be extended to cross-attention.\n\n3. Our results highlight that the implicit weight updates of the $W_{qk}$ matrix encode essential structures for Transformer training. We hypothesize that architectural designs that better preserve and reinforce these structures in both the column and row spaces of $W_{qk}$ could further enhance training efficiency.\n\nOn the other comments and suggestions:\n\n1. We respectfully ask the reviewer for clarification regarding which additional layers should be visualized. Figures 3 and S4 already show symmetry and directionality scores across all layers of several models trained from scratch on three distinct datasets. \n\n2. The main goal of our study was to characterize the structures that emerge in self-attention matrices during pretraining. As discussed above, future work should explore how symmetric initialization can be leveraged to potentially benefit from fine-tuning on downstream tasks.", "claims_and_evidence": "The main claims of the paper include:\n\n- Self-attention matrices exhibit structural properties influenced by the training objective.\n\n- Bidirectional training promotes symmetry in weight matrices, whereas autoregressive training enforces directionality.\n\n- These structural differences emerge naturally and can be leveraged to improve model performance.\n\nThe paper supports these claims through formal mathematical proofs and extensive experiments, showing consistent patterns across different model families and datasets.", "methods_and_evaluation_criteria": "The authors utilize a combination of theoretical analysis and empirical evaluation. They propose symmetry and directionality scores to quantify the structural properties of self-attention weight matrices. These metrics are applied to pretrained models, with comparisons across architectures trained on different datasets. The evaluation criteria are appropriate, as they align well with the research questions posed.", "theoretical_claims": "The paper presents rigorous mathematical derivations supporting its claims. Key results include:\n\n- Proofs showing how gradient updates reinforce symmetric or directional structures depending on the training objective.\n\n- Theorems explaining the emergence of column dominance in autoregressive models.\n\nThese theoretical insights are well-motivated and correctly formulated. The paper provides clear logical progressions from assumptions to conclusions, making the theoretical claims compelling.", "experimental_designs_or_analyses": "The experimental design is sound, involving multiple Transformer architectures trained under different conditions. The authors analyze pretrained models to confirm the presence of symmetry and directionality trends. The statistical significance of results is demonstrated through interquartile range analyses. However, the study could be strengthened by including additional ablation studies to further isolate the effects of training objectives from other hyperparameters.", "relation_to_broader_scientific_literature": "This work builds on prior research in Transformer interpretability and self-attention mechanisms.", "essential_references_not_discussed": "The paper does not discuss some recent works on mechanistic interpretability of self-attention, such as studies on learned feature representations in attention layers. Including a discussion of these works could provide additional context and comparisons.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper provides a novel theoretical perspective on self-attention structures.\n\n- The empirical validation is thorough, covering multiple model families and training settings.\n\n- The findings have potential applications in improving Transformer interpretability and efficiency.\n\nWeaknesses:\n\n- The paper does not explore practical implementations of the proposed insights, such as real-world deployment benefits.\n\n- Some of the notation in the proofs is dense and may be challenging for readers unfamiliar with advanced linear algebra.", "other_comments_or_suggestions": "- Consider adding an appendix section with additional visualizations for symmetry and directionality scores across more layers.\n\n- Clarify the potential implications of these findings for fine-tuning large-scale models.", "questions_for_authors": "1. Have you tested the impact of symmetric initialization on autoregressive models? If so, how does it compare to encoder-only models?\n\n2. Could your framework be extended to analyze cross-attention mechanisms in encoder-decoder models?\n\n3. Do your findings suggest any potential modifications to existing Transformer architectures for efficiency improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CjwPY5BYkn", "reviewer": "Reviewer_ydUg", "summary": "The paper studies how bidirectional and autoregressive training objectives influence the structure of the query-key matrix $W_{qk}$ in self-attention. The results show that bidirectional training induces symmetric structures in $W_{qk}$, whereas autoregressive training\nresults in matrices characterized by directionality and column dominance. The findings are then verified empirically, and inspired a symmetric initialization to speedup training of encoder-only models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and positive review, as well as the valuable suggestions and questions. \n\nFirst, we agree with the reviewer that we could have utilized the available manuscript space more effectively. In response to suggestions from Reviewers HwPP and JKhN, we will revise Section 2 by explicitly presenting Theorems 2.3 and 2.4, clearly stating their assumptions, conditions, and implications, integrating key points into the main text. We appreciate the reviewer’s feedback, as these adjustments will enhance both the rigor and clarity of our theoretical contributions.\n\n\nSecond, we address the questions raised by the reviewer:\n\n1. The analysis presented in Section 2 is not specific to single-layer self-attention, but applies generally to any layer within a Transformer model. Empirically, however, we observe that early layers can deviate from our theoretical predictions compared to deeper layers. This is expected, as backpropagation causes early layers to receive noisier updates, leading to greater divergence from theoretical predictions. On the other hand, the connections between tokens $x_i$ and $x_j$ and the corresponding prediction errors encoded by $\\beta_{ij}$ are accurately reflected in the weight updates of deeper layers (which are closer to the output). The degree of deviation depends on the model and dataset and is often minimal. Figure S4 (complementing Figure 3) shows examples of models where such deviations are negligible. Based on these observations, we hypothesize that symmetric initialization of early layers is unlikely to be beneficial. Indeed, we tested symmetric initialization of all layers of decoder-only models, and we did not observe any significant improvement in training speed compared to standard initialization. We refer to our answer to question 1 of reviewer jZNG for further details. \n\n2. We thank the reviewer for pointing this out. Since Figure 2 presents the symmetry and directionality scores for pretrained models, we assume this comment refers to Figure 3. We will update the caption of Figure 3 to explicitly mention the model (`Bert-Base-Uncased`) and the number of layers (12).", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "i did not check the proofs closely.", "experimental_designs_or_analyses": "The experiments seem solid to me.", "relation_to_broader_scientific_literature": "Attention mechanism is a very sophisticated system with many moving parts, making it very opaque. The paper analyzes how training objective affects attention, which I think is a very important problem to study, and the results are very interesting.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strength: How training objective affects attention is an interesting yet underexplored question. The results in this paper are very interesting and provide both theory and practical values. I also like the intuition described in line 171-186.\n\nWeakness: Given there is still about one-page space left in the draft, I think it would be better to give a formal theorem statement for Theorem 2.3 and 2.4 since they are the **main** results of the paper. The current way of presenting in my opinion, hurts the results's significance and rigor.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. Is the analysis in Section 2 particular to one-layer attention? Do you have any idea why certain layers of a model do not follow the findings of paper? (for decoder-only models, for example,  early layers are very symmetric). Would symmetric initialization of initial layers also speedup training of decoder-only models?\n\n2. Could you specify the name of the model and the layer number in Figure 2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies how bidirectional and autoregressive training objectives influence the structure of the query-key matrix in self-attention. The results show that bidirectional training induces symmetric structures, whereas autoregressive training results in matrices characterized by directionality and column dominance. The findings are then verified empirically, and inspired a symmetric initialization to speedup training of encoder-only models.

With scores 2,3,4,4, and after carefully reading the comments and authors' rebuttal, I recommend acceptance. 

There is strong support from three reviewers. The reviewer recommending rejection had raised concerns about incorrectness of one equation and more importantly the lack of mathematical rigor as manifested in the absence of explicit statement of assumptions and full version of theorems deferred to appendix or very simple results stated as propositions.
The authors have:
(i) clarified that the incorrect equation is not affecting any of the conclusions and was in a non-critical equation meant to give intuition which will be removed
(ii) responded with a clear plan to restructure Section 2 and formalize statements

Since other reviewers who eventually recommend acceptance also discussed the issues about formalizing statements and bringing all details from appendix to main body and appear content with the authors' promise to do that, I recommend acceptance.

However, I do urge the authors to include all the required modifications in the camera ready. These include:
(i) Give formal theorem statements for Theorem 2.3 and 2.4 in main body (ii) Make other promised changes to structure of Section 2 (including clarifying Prop. 2.1, expanding Section 2.3) (iii) Add additional experiments on large-scale datasets as stated in rebuttal (including those on vision models despite not revealing gains)
Decision: Accept (poster)