Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Year: 2023

Abstract:
The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.

heading: 1 Introduction & Related Work
text: Transformer models have become the backbone of natural language processing applications (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2019). Within the transformer architecture, there are two main categories: 1) bidirectional models, such as BERT (Devlin et al., 2019), that are trained using the masked language modeling objective, and 2) (causal) language models, such as GPT (Radford et al., 2019), that are trained using the traditional language modeling objective. Both of these categories share the common feature of using positional embeddings for encoding token distance.
Whether positional embeddings are truly essential has been a subject of ongoing research. While they have been considered necessary for bidirectional transformer models (Lee et al., 2019; Luo et al., 2021; Sinha et al., 2021; Haviv et al., 2022), the situation is different for transformer language models (Irie et al., 2019; Yang et al., 2019; Tsai
†Correspondence to: tachungc@andrew.cmu.edu
et al., 2019; Scao et al., 2022; Haviv et al., 2022). In transformer language models, the removal of positional embeddings results in only a marginal decline in performance, while enabling more efficient training (Haviv et al., 2022). In addition to empirical evidence, it has been proven (Bhattamishra et al., 2020) that transformer language models without positional embeddings are Turingcomplete and able to model sequences akin to recurrent neural networks (Rumelhart and McClelland, 1987; Jordan, 1986). Despite this, it remains an open question where positional information is stored in the absence of positional embeddings. This motivates further investigation into individual operations within a transformer layer.
The example architecture of a pre-LN (Xiong et al., 2020) multi-layer transformer language model with no positional embeddings used in this
1183
work is shown in Figure 1.1 We hereinafter refer to this configuration as TLM. Our primary focus is on the multi-head attention (MHA) module of a randomly initialized TLM, as it is the only module that allows inter-token information exchange. To gain a deeper understanding, we compute the mean and variance of MHA outputs. To our surprise, we discover that the variance already encodes latent positional information, with later tokens in a sequence displaying smaller variance. This motivates us to quantify the variance by deriving the output distribution after MHA operations. Finally, through empirical validation using a fully pre-trained TLM, we confirm thatthe same variance shrinkage effect persists after extensive gradient updates.
To the best of our knowledge, we are the first to identify and quantify the latent positional information in TLMs. Our results provide theoretical insights into the removal of positional embeddings, enabling more efficient pretraining of future TLMs.

heading: 2 Probing Experiments
text: Given BERT and TLM (GPT) with positional embeddings removed, prior work (Haviv et al., 2022) shows that only TLM is able to maintain the same language modeling performance as its original version with positional embeddings. The discrepancy might be explained by the fact that only TLM encodes positional information within its layers, as shown by the position probing experiment in Haviv et al. (2022). Since both BERT and TLM have access to the same semantic input and the only difference is the use of causal attention masks in TLM, we hypothesize that the positional informa-
1Post-LN places layer norm at different positions. It is the configuration used in BERT (Devlin et al., 2019).
tion may be attributed to the interaction between causal attention masks and the TLM architecture.
To further explore this hypothesis, we use a randomly initialized and frozen TLM to eliminate any semantic influence and focus solely on the architectural design. Additionally, to prevent the model from memorizing the order of input sequences, we do not perform embedding lookups and feed the model with randomly sampled input vectors. A trainable two-layer linear classifier with ReLU activation in between was appended to the TLM to probe the position of each token (further details can be found in Appendix B). We plot the mean absolute error (MAE) w.r.t the number of transformer layers in Figure 2. The plot indicates a randomly initialized and frozen TLM with randomly sampled input vectors inherently provides positional information, with an increase in the number of layers resulting in higher probing performance. This surprising outcome prompts further investigation into the encoding of latent positional information inside the TLM architecture.

heading: 3 Theoretical Analysis
text: We dissect the inner workings of a TLM by deriving the distribution of TLM operations in the hope that they elucidate where the latent positional information is stored. The derivation is made possible thanks to the usage of a randomly initialized and frozen TLM. We adopt the initialization settings in accordance with those employed in GPT (Radford et al., 2019). WLOG, our derivation is limited to the operations of the first layer in a TLM and the FFN component is omitted (justified in §3.4). The hyperparameters utilized in the simulations are: hidden dimension d = 768, number of attention heads H = 12, head dimension d/H = 64, sequence length L = 512, standard deviation for initialization σ = 0.02. All proofs of lemmas are deferred to Appendix A.
Given a sequence of randomly sampled input embeddings {xm}Lm=1, where each element of xm ∈ Rd is sampled i.i.d from N(0, σ2), a TLM consists of the following operations:

heading: 3.1 Layer Normalization
text: For each input embedding xm, it computes the sample mean and (biased) sample variance:
xm,: =
∑d i=1 xmi
d , S(xm,:) =
∑d i=1(xmi − xm,:)2
d
Then each entry i of xm, denoted as xmi, is normalized by mean and variance to emi:
emi = xmi − xm,:√ S(xm,:) ∗ γ + β
(∗) ≈ xmi − E[xmi]√
V[xmi] ∼ N(0, 1),
where V[x] denotes the variance of x. Since the initialization scheme sets γ = 1 and β = 0, (∗) holds with sufficiently large d by the Law of large numbers and the continuous mapping theorem.

heading: 3.2 Self Attention
text: Each attention head computes query, key, and value vectors in R d H :
qm = Wqem, km = Wkem, vm = Wvem,
where Wq, Wk, Wv ∈ R d H ×d are matrices with each element sampled i.i.d from N(0, σ2). To be precise, most matrices (W (h)q , W (h) k , W (h) v ), vectors (q (h) m , k (h) m , v (h) m ), and scalars (l (h) mn, a (h) mn) are associated with a head number h. For notation simplicity, we only show the dependency on h when we need it.
Lemma 1. qm, km, and vm have zero mean and (dσ2) · I covariance matrix. The resulting vectors are processed by the selfattention module for pre-Softmax logits:
lmn = { 〈qm,kn〉, if m ≥ n − inf, otherwise
followed by the scaled softmax normalization:
amn = exp
( lmn/ √ d/H )
∑L i=1 exp ( lmi/ √ d/H )
Lemma 2. lmn has zero mean and d 3σ4
H2 variance.
lmn/ √ d/H has d 2σ4
H variance.
The numerical variance of lmn/ √ d/H in our case is 768 2·0.024 12 ≈ 0.0079. Lemma 2 suggests the following approximation:
Property 1. When σ4 H d2 , lm,: has small variance, making the attention weights am,: almost evenly distributed among all positions.2
In Figure 3, we verify Property 1 by showing that amn is almost evenly distributed in simulation.
Observe that the output vector om at position m is:
om = Wo ( ⊕Hh=1 L∑
n=1
a(h)mnv (h) n
) ,
where ⊕ denotes the concatenation of vectors from all H attention heads. Assume that Property 1 is valid and that Wo ∈ Rd×d has elements i.i.d sampled from N(0, σ2), we derive the distribution of om below. Lemma 3. om has zero mean and d 2σ4
m I covariance matrix.
2This approximation was also used in Xiong et al. (2020) except that they made a stronger assumption that Wq and Wk have to be initialized as zero matrices.
Figure 4 is a simulation that verifies Lemma 3 under the assumption of Property 1. We can see that the variance of om already encodes the positional information m.

heading: 3.3 Residual Connection
text: As denoted by the Addition block of Figure 1, the residual connection sets the output as ym = xm + om. It allows the model to pass the first MHA output to later MHA modules as well as the final classifier. As the positional information has been passed by the residual connection, we omit the FFN part in our analysis.

heading: 3.4 The Final Layer Normalization
text: Layer normalization is an operation that might eliminate the positional information derived in Lemma 3, which happens before the MHA modules and position classifier. As mentioned in §3.1, LN(ym) gives:
y′mi ≈ ymi − E[ymi]√ V[ymi] ≈ xmi +WoWv
∑m n eni m√
σ2 + d 2σ4
m
,
E[ymi] = 0, V[ymi] = V[xmi] + V[omi]
= σ2 + d2σ4
m
Lemma 4. The variance of the j-th dimension of ym is:
mσ2 + ∑
i(Wo,j:Wv,:i) 2
mσ2 + d2σ4 ,
where Wo,j: ∈ R1×d is the j-th row of Wo. Wv,:i ∈ Rd×1 is the i-th column of Wv. As long as ∑
i(Wo,j:Wv,:i) 2 6= d2σ4, the classifier should
be able to exploit the discrepancy to derive m. Readers might wonder why Wo,j: and Wv,:i in the numerator cannot be treated as random variables. The reason is that we only focus on one dimension (j-th) at a time. This means we cannot use the law of large numbers to approximate the sample variance of ymj as we did for the denominator.

heading: 3.5 Relaxing the Assumptions
text: We discuss possible relaxation of the assumptions used in §3.2.
What if Property 1 does not hold? Or equivalently, σ4 6 H
d2 . This prompts us to vary the value
of σ. In Figure 5, we see that smaller σ better aligns Lemma 3 with the simulations, which is unsurprising as Lemma 3 assumes small σ. Even when σ is not too small (i.e., σ = 0.2, 0.02), the variance still encodes the positional information as the variance of om is negatively correlated with its position m.
Other Initialization Schemes So far we assume the weight matrices (Wq, Wk, Wv, Wo) are initialized i.i.d fromN(0, σ2). However, we can relax the assumption to i.i.d. samples from a distribution with zero mean and finite variance. This is because the proof in Appendix A calculates the covariance. The variance calculation relies on E[rir>i ] = σ2I where r>i is the i-th row vector of a weight matrix. This property holds for any distribution with zero
mean and σ2 variance.

heading: 4 Discussions
text: Why are the positions of later tokens in a sequence harder to be predicted in Figure 3 of Haviv et al. (2022)? Lemma 3 states the variance is inversely proportional to the position m, so the variance of later tokens (large m) plateaus, resulting in a harder numerical optimization problem. This also suggests a potential downside of removing positional embeddings: It might be challenging for the model to infer positional information of the later tokens in extremely long input sequences.
Why do lower layers (closer to input) give worse probing performances in both Figure 2 and Haviv et al. (2022)? This can be explained by Figure 4. Most of the positions at the 0th layer have tiny variance (exp(−10) = 4.5e−5), which again poses a difficult numerical optimization problem.
Why does BERT fail to converge without positional embeddings? In a BERT model (Devlin et al., 2019), each token has access to all the other tokens, making the variance at all positions d 2σ4
L . Therefore, a BERT model cannot utilize variance differences as its positional indicator.

heading: 5 Post-Training Results
text: Our derivations only apply to the initial stage where the TLM and input embeddings are randomly initialized, which may not hold true after gradient updates. It is essential to verify the existence of variance properties and lemmas on a fully pre-trained TLM on OpenWebText2 (details in Appendix C).
We expect that the properties of lower layers of a pre-trained TLM should align more closely with the theoretical results for two reasons: 1) There are more steps between the lower layers and the final language modeling loss, resulting in smaller gradients and thereby fewer parameter updates, and 2) Lower layers typically encode more lowlevel information dependent on positional information (Vulić et al., 2020; de Vries et al., 2020). Figures 6 and 7 demonstrate that the 0th (lowest) layer exhibits highly similar cumulative attention probability and decay-with-position variance as the theoretical results. In contrast, higher layers deviate from the analyses in § 3. We posit that the model learns to rely more heavily on semantic rather than positional information. This also explains why
predicting positions using outputs of higher transformer layers is more challenging as demonstrated in Figure 2 of Haviv et al. (2022).

heading: 6 Conclusion
text: We mathematically analyzed a randomly initialized transformer language model without positional embeddings. We showed that the variance of the selfattention output decreases as the position increases, which serves as an indicator for positional information. We validated that, after extensive gradient updates, the low layers of a pretrained language model still exhibit highly similar variance reduction behaviors. Our results pave the way for the pretraining of more efficient and positional embedding-free transformer language models.
Limitations
The limitations of this work mostly come from our assumptions: 1) A randomly initialized and frozen TLM, and 2) Input tokens are all different and randomly sampled. These two assumptions obviously do not hold true for human languages and pre-trained TLMs. Therefore, we attempted to empirically verify the existence of lemmas and properties on a pre-trained TLM without positional embeddings in §5.
That being said, several methods could be attempted to remove these assumptions. Firstly, we can analyze the training dynamics of a TLM to shed light on the model parameter distribution after pretraining. Secondly, Zipf’s law or a simple n-gram language model could be used to quantify the degree of input token duplication in human languages. This might give us a more accurate estimate of the variance at different positions. We leave these ideas as future work.
Ethics Statement
Our work provides a deeper understanding of why a transformer language model can still perform well without positional embeddings, potentially enabling the application of developing future transformers that are greener and more cost-efficient. Inappropriate usage of our technique might have negative societal impacts though. These include the ethical challenges of improper text generation and privacy issues inherent in the data collection process. These implications apply to any natural language processing research and are not unique to this specific work.

heading: Acknowledgment
text: The authors acknowledge the support from Boeing (2019-STU-PA-259), Amazon (CC ADV 00474341 2021 TR), NSF MRI Award 1919452, and Princeton Research Computing.

heading: A Proofs
text: The proof of Lemma 1 and 2 are head-dependent while that of Lemma 3 is head-independent. For notation simplicity, at Lemma 1 and 2, we drop the head dependency on matrices (W (h)q , W (h) k , W (h) v ), vectors (q (h) m , k (h) m , v (h) m ), and scalars (l (h) mn, a (h) mn).
Proof of Lemma 1 Here, we use r>i to denote the i-th row vector of Wv.
cov(vm,vn) = E[vmv>n ] = E[Wveme>nW>v ]
= E     r>1 em
... r>d
H
em
  [ e>n r1 . . . e > n r d
H
]  
= [ E[r>i eme>n rj ] ] d H
i,j=1
= [ E[Tr(rjr>i eme>n )] ] d H
i,j=1
= [ Tr(E[rjr>i ]E[eme>n ]) ] d H
i,j=1
(∗) = [ Tr((1i=jσ2) · Id · 1m=n · Id) ] d H
i,j=1
= [ 1i=j1m=ndσ 2 ] d H
i,j=1
= (1m=ndσ 2) · Id/H
(∗) holds because ri and rj are independent when i 6= j (similarly for em and en) and the covariance of a Gaussian random vector is an identity matrix. Id and Id/H denote d × d and dH × dH identity matrices.
Proof of Lemma 2 Here, we use r>i to denote the i-th row vector of Wq and Wk.
cov(lmn, lmp)
= E[(e>mW>q Wken)(e>mW>q Wkep)>] = E[Tr(e>mW>q Wkene>p W>k Wqem)] = E[Tr(eme>mW>q Wkene>p W>k Wq)] = Tr(E[eme>m]E[W>q Wkene>p W>k Wq]) = E[Tr(ene>p W>k WqW>q Wk)] = Tr(E[ene>p ]E[W>k WqW>q Wk)]) = (1n=p)Tr(E[WqW>q ]E[WkW>k ]) (∗) = (1n=p)Tr(( d
H σ2 · I)( d H σ2 · I))
= (1n=p) d3σ4
H2
(∗) holds since:
E[WqW>q ] = E     r>1 ...
r>d H
  [ r1 . . . r d
H
]  
= [ E[r>i rj ] ] d H
i,j=1 =
d
H σ2 · I
Proof of Lemma 3 Because Wo ∈ Rd×d is applied on a concatenation of vectors at all heads, we take vi = ⊕Hh=1v (h) i . vi here is head-independent while vi at Lemma 1 is head-dependent. Here, we use r>i to denote the i-th row vector of Wo.
cov(om,om)
Property 1≈ E [ Wo ∑m i=1 vi m ∑m j=1 v > j m W>o ]
= 1
m2
m∑
i,j=1
E[Woviv>j W>o ]
= 1
m2
m∑
i,j=1
E     r>1 vi
... r>d vi
  [ v>j r1 . . . v > j rd ]  
= 1
m2
m∑
i,j=1
[ E[r>k viv>j rl] ]d k,l=1
= 1
m2
m∑
i,j=1
[ E[Tr(rlr>k viv>j )] ]d k,l=1
= 1
m2
m∑
i,j=1
[ Tr(E[rlr>k ]E[viv>j ]) ]d k,l=1
(∗) = 1
m2
m∑
i,j=1
[ Tr((1k=lσ2) · I
·(1i=jdσ2) · I) ]d k,l=1
= d2σ4
m I
(∗) follows from Lemma 1: because cov(v(h)i ,v (h) j ) = (1i=jdσ
2) · Id/H , a concatenation for all h ∈ H gives E[viv>j ] = (1i=jdσ2) · Id.

heading: B Probing Experiment Details
text: We train a randomly initialized and frozen TLM with 12 layers, d = 768, H = 12, L = 512, and σ = 0.02. We use the Adam optimizer (Kingma and Ba, 2014) with learning rate 1e− 3 and 5000 gradient updates. The batch size is set to 32. We implement our model using PyTorch (Paszke et al., 2019).

heading: C Pre-trained Transformer Language Model Details
text: We use the gpt-neox library (Andonian et al., 2021) to train a TLM with no positional embeddings. Detailed hyperparameters are listed in Table 1. The pretraining takes 5 hours on one NVIDIA A10040GB.

heading: D Scientific Artifacts
text: We use the gpt-neox library (Andonian et al., 2021) under Apache-2.0 license. OpenWebText2 (Gao et al., 2020) is released by the authors of gpt-neox. The codebase and dataset are publicly released for research purposes. The steps taken to protect privacy and anonymization are discussed in Section 6 and 7 of Gao et al. (2020). The distribution and statistics of OpenWebext2 are also discussed in Gao et al. (2020).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
3 A2. Did you discuss any potential risks of your work? Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract and section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 2 and 5
3 B1. Did you cite the creators of artifacts you used? appendix D
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? appendix D
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? appendix D
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? appendix D
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? appendix D
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. appendix D
C 3 Did you run computational experiments? section 2, 3, and 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? appendix C
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In figure captions scattered across all sections
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave these ideas as future work