Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Authors: zican Dong
Junyi Li
Xin Men
Xin Zhao
Bingning Wang
Zhen Tian
weipeng chen
Ji-Rong Wen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Positional Vector
Context Window
Large Language Model
Length Extrapolation
Context Window Extension
Primary area: natural_language_processing

Abstract: Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Zican Dong';June Junyi Li Xin Men', Wayne Xin Zhao'! Bingning Wang'
Zhen Tian', Weipeng Chen'. Ji-Rong Wen
Gaoling School of Artificial Intelligence, Renmin University of China School of Information, Renmin University of China
Department of Computer Science, National University of Singapore Baichuan Inc.
dongzican@ruc edu. cn, junyi_ca@nus ca@nus edu. sg batmanfly@gmail com, cariee@bbichhan com
Abstract
Transformer-basee large language models (LLMs) typically have limited context window, resulting in signifieant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using mean- wesan decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct findings, design
extrapolation and context window extension. Based training-fres free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods effectively extend the context window length.
Introduction
Recently, Transformer based large language models (LLMs) have demonstrated excellent capabilities on downstream tasks [T,]) in which positional encodings (e.g.. absolute or relative) widely used
in Transformers to better capture positional information within input sequences [4] [5] However, LLMs typically suffer from limited input length (called context window), which is constrained by the maximum length of training data. Beyond the context window, the positional encodings at larger position indices are -iitt----itiiittionn (OOD), not encountered during the training phase. Therefore, when the input sequence exceeds the context window length, there would often be a significant degradation model performances, as evidenced by surge in perplexity (PPL) score (6)
Prior work has primarily focused on extending the context window existing LLMs by manipulating positional encodings. Owing to its excellent performance and long-term decay nature, RoPE 7 has been widely used to learn positional encodings for existing LLMs 9 To circumvent the OOD positional encodings in RoPE, various methods have been proposed to modify the base [10H12] or positional indices (63 addition, special relative positional encodings that apply larger negative biases to attention based the relative distance have achieved promising length extrapolation, which
'Equal Contribution. Cooreesponiing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

can effectively stabilize the model performance beyond the context window 18] Furthermore, decoder-only Transformers without positional encodings (NoPE) have been found to be capable of learning implicit positional information [9] and their context window size can be extended via the adjustment of temperature
(0) However, the above extension methods solely focus on adapting positional encodings or attention scores, lacking detailed analysis of the underlying
mechanisms hidden states in LLMs.
In this work, we aim to investigate the inner working mechanism of LLMs within and beyond the context window to interpret these context window extension approaches. As the basis, our work is developed by analyzing the positional information implicitly encoded in the hidden states of LLMs across various layers and positions, both within and outside the context window. Inspired by previous work [1 we use a mean-based decomposition approach to disentangle positional vectors from the hidden states, which captures the information independent of semantics but related to positions.
Specifically, we first investigate how positional information is formed and examine its impact on the attention mechanism within the context window. Second, for inputs beyond the context window, we analyze the change of positional vectors in two settings, i.e, direct extrapolation and context window extension. Our key findings include: (1) After the first layer, initial tokens can form distinct positional vectors, serving as anchors for shaping positional vectors in subsequent tokens; (2) Positional vectors play a critical role in modulating the long-tern decay and establishing attention sinks: (3) When exceeding the context window, OOD positional vector is the major factor contributing to performance degradation, while length extrapolation can effectively keep the consistency of positional vectors both within and beyond the context window; (4) Context window extension methods enable interpolation of positional vectors by adjusting the information flow from initial tokens to subsequent tokens.
Based on the empirical findings, we further propose two training-free context window extension methods from the perspective of interpolating positional vectors: positional vector replacement and attention window extension. For LLMs with NoPE, the former method replaces the positional vectors in critical layers with interpolated ones; while for LLMs with window attention and NoPE, apper-paramttr. We
the latter method directly scales the window size and adjusts the temperature evaluate the length generalization capacities of the proposed methods on PG-19 [22]. Experimental results demonstrate that our methods can effectively generalize to longer texts without fine-tuning, achieving comparable performance to previous methods.
Our main contributions are summarized as follows:
We explicitly delineate the formation process and the effect of positional vectors, highlighting the anchoring role of initial tokens in shaping different positional vectors across tokens and their importance in achieving long-term decay and attention sinks.
We are the first to unify length extrapolation and context window extension from the perspective of positional vectors, identifying that preventing OOD positional vectors is crucial for avoiding performance degradation.
We propose two training-free free context window extension methods via the lens of adjusting posi- tional vectors, i... positional vector replacement and attention window extension. Experimental results show that our methods can effectively generalize to longer texts without fine-tuning.
2 Background
Transformer Decoder-only Transformer 4 has become the foundational architecture for LLMs [4] ا. For a Transformer with L layers and a context window size C, given an input sequence s of T tokens, f.e., {I1,.... ... .Fi.....TT. J. denote the output of the I-th layer as H = (h....... h At each layer, the output H; is obtained through multi-head attention (MHA) and feed-forward network (FFN) with residual connections applied both components as follows:
H MHA(H;,)) - - H = FFN + H.
Finally, the output of the last layer Hi is then projected into the logits, which will be used to generate the prediction probability for each token in the vocabulary.
Positional Vector Previous work has found that positional information can be learned and encoded in the hidden states of Transformers [19]. Drawing inspiration from prior work (21), we hypothesize

---

that each hidden state (e.g., query, key, value, output of each layer) within Transformer can be decomposed into two parts, i.e. a positional vector that captures positional information and a semantic vector that captures the contextual information. Taking the output bf, of the /-th layer at t-th position as an example, it can be decomposed into positional vector Pt, and a semantic vector :
hi, - Pt.e + ci,--
Such decomposition can disentangle two primary factors, namely positional and semantic vectors, for interpreting the internal mechanism of LLMs. Notably, since positional vectors are globally shared across different inputs, there is no superscript for pt,-- Further, the positional vector Pi, can be decomposed into a mean vector u and a positional basis mt,
Pt, U1 + m1,6
where the mean vector u, denotes the mean of the distribution of positional vectors and the positional basis m denotes the offset of (-th position from the mean vector within the context window size C. Following previous work [2I], we adopt a mean-based decomposition method to obtain the above three vectors based on N samples from the training corpus as follows:
N
Pr
1 hi mI Pt, C
Σ PLe f==1
hi -P.t-
With this decomposition, offers an explicit way to analyze and explore the positional information encoded in the hidden states of Transformer models. For example, we can use similarity measurements compare the positional vectors of different positions and also can visualize them in fow-dimensional embedding space. In the following sections, we will mainly focus on studying the formation and impact of the positional vector pt,r, and conduct the analysis experiments.
3 Empirical Analysis
3.1 Experimental Settings
To better analyze positional information, we consider model variants with different positional encod- ings (PH) and attention mechanisms: variants without positional encodings (NoPH) [T9] as well as variants with two different positional encodings: RoPE (7] and ALiBi (6]. We continually pre-train the TinyLlama-I,
checkpoint [23] on 50B tokens from RedPajama (24) with a context window C' 2048, resulting in set of comparison models with different positional encodings and attention mechanisms, as shown the Table Full attention means that each token can attend to all previous tokens, while window attention restricts each token to attend only to previous tokens within a window size W. The training details are described in Appendix AA] We also evaluate common LLMs (e.g., Llama-3-8B and LLMs without positional encodings trained from scratch, and the evaluated results are listed in Appendix F
Specifically, we subsample 32K samples with the same number of tokens from RedPajama. We perform the inference on these data to obtain hidden states of L.MMs. By using the mean-based decomposition method (Eq, 4). we can obtain the positional vectors Pt. of tokens in these sample texts.
Table The compared model variants. Full attention is denoted as Full and window attention with a window size of W tokens is denoted Window (W). We abbreviate TinyLLaMA as TL.
Model
To-NoPE
TL-RoPE
TL-ALiBi
TL-Window
Windoww TL-Window-RoPE
PE Attention
NoPE
RoPE
ALiBi
NoPE
NoPE Window (80)
RoPE Window (512)
Full
Full
Full
Window (512)
3.2 Formation and Effect of Positional Vectors within Context Window
In existing LLMs, the bottom (first) layer typically takes input token embeddings that lack inherent
positional information; while interestingly, the hidden states from top layers can implicitly capture positional information, even without explicit positional encodings [19 21 14). In order to have deep understanding of implicit positional information, we next investigate the formation and effect of positional vectors in Transformers, with both full attention and window attention.
3

---

TL-NoPE (1-st Layar)
TL-NoPE (7-th Layer)
1200
Initall lostens
Positional Vectors Across Layers of TL-Window
000
1300
2000
1750
205
Last token
Diitiint Wictor
1500
206
400
50
05
1250
FC
TL-RoPE (1-st Layer)
TL-RoPE (7-th Layer)
1000
1250
750
Fnimal taiens
100-
750
Distinct positional vectors Maximum
500
230
806
Layer
Figure 1: PCA visualization of positional vectors from the 1-st and 7-th layers.
Figure 2:
Comparison of distinct positional
vectors and theoretical receptive field.
3.2.1 Formation of Positional Vectors with Full Attention
Positional Vector After the First Layer To study how positional information is distributed over different positions, we first visualize the positional vectors P1. (Eq. decomposed from the outputs of the first layer using principal component analysis (PCA). As shown in Figure (left column), initial tokens (e.g.. 4 tokens) exhibit significantly distinct positional vectors, while the positional vectors of subsequent tokens are similar to each other. As a comparison, we also present the PCA results of all positional vectors the 7-th layer. Interestingly, position vectors are evenly distributed across all the positions in Figure [ column). Such a finding indicates that position vectors have captured the corresponding positional information since these vectors are distinct from each other across positions. In other words, being distinct can be considered as kind of positional evidence. By comparing the left and right columns of Figure [ it seems that only initial tokens are different from the rest tokens after the first layer, which might suggest that after the first layer, initial tokens have already formed distinct positional information but subsequent tokens have not yet established such information. To investigate the reasons behind this phenomenon, we select the first attention head in the first layer (similar other heads) analyze attention scores, as detailed Appendix [ We can prove that the positional vector 11, for the first token is different from the following tokens and the attention scores affect the formation of positional information. Thus, through several layers, the tokens after the first token will gradually form distinct positional vectors (Figure I rright column).
Positional Information Flow From Initial Tokens By applying positional vectors top layers (PCA visualized in Appendix G) we find that after forwarding several layers, tokens at all positions can also exhibit distinct positional vectors, and similar findings are also found in previous work [19]. To trace back to the source of positional information, a reasonable speculation is that initial tokens play key role in the formation of positional information for the rest tokens since only initial tokens capture positional information after the first layer. To validate this, we select two groups of reference tokens: initial tokens (I~4) and secondary tokens (4~256), and further analyze what information of these tokens is critical for the formation of positional information in subsequent tokens (>256). Thus, based on a top-down strategy, we conduct an ablation study for each group by respectively deleting the value vi, of attention module (w/o value), the semantic vector cτ,, (w/o semantic vector), positional vector Pi. (w/o positional vector), and positional basis mi, (w/o positional basis). Then, for each variant, we average the outputs of all layers and decompose new positional vectors based on Eq. Finally, we compute the average cosine similarity between the original and new positional vectors for those subsequent tokens (>256) and also report PPL on samples in RedPajama. From Table we can see that removing the positional vector and basis of 1 1~44 tokens largely affect the positional vectors at later positions (low similarity). Conversely, removing the semantic vector or altering secondary tokens has slight effects on both similarity and PPL. From these findings, we conclude that the positional vectors of initial tokens seem to serve as the role of anchors, largely contributing to the formation of positional information in subsequent tokens.

---

Table 2: Results of removing different components attention. Sim denotes the cosine similarity between original and new positional vectors, and PPL denotes perplexity on RedPajama.
original
w/o value
w/o positional vector
w/o positional basis
w/o semantic voctor
1-4
4-256
1~4
4~255
1~4
4~256
1~4
4--266
Sim 1
-0.1558
0.9797
-0,1810
0,9086
-0.1817
0,9046
0,9985
0,9514
TL.-NoPE
PPI. 7.56
>1000
8.97
>1000
13.36
>1000
10.23
8.20
10.55
Sim 1
0.8394
0.9902
0.8505
0,9874
0.1711
0.9944
0,9970
0.9596
TL-RoPE
PPL
6.06
11.98
6.44
12.28
6.24
>100
6.11
6.63
6.85
TL. RoPE original
RoPE semantic vactor
TL-RoPE positional
TL-RoPE pasitional bashs
-2
Attention Eriks
Adtricon Sinks
-2
-3
13
TL-NoPE original
TL-NoPE semantic vector
TL-NoPE positionaa vector
TL-NoPE positional basis
Attentoo Sinks
Attertion Sinks
-8
-2
Figure 3: Logarithmic attention maps of TL-RoPE, and TL-NoPE.
3.2.2 Formation of Positional Vectors with Window Attention
Unlike full attention, LLMs employing window attention restrict each token to attend only to tokens within window size. Previous work has shown that the maximum theoretical receptive field (TRF) in window attention is equal to the product of the window size W and the layer index [8]
To analyze how positional vectors change across layers, we compute the number of distinct positional vectors within the maximum TRF. Notably, those tokens beyond the maximum TRF share the same positional vectors due to translation invariance [8]. Specifically, we first randomly select positional vector outside the maximum TRF and then compute the cosine similarity between positional vectors within the maximum TRF and the selected vector. We consider the positional vector with a similarity score lower than a threshold (i.e., 0.99) as distinct. Figure 2 presents the number of distinct positional vectors and TRF at each layer. We can that after the first layer, only initial tokens show distinct As the layer increases, more tokens
positional vectors, further verifying the findings in Section 3.2.1 3.22.on3.22. display different positional information and the number of distinct positional vectors increases by 512 (a window size W) with each additional layer. The reason is that due to the constraint of window attention, each token at the preceding layer can only influence tokens within the window size the next layer. As being distinct indicates the formation of position information, similar positional information flow from initial tokens to subsequent tokens also occurs for window attention, but gradually propagating across both windows and layers.
3.2.3 Effect of Positional Vectors on Attention
After discussing the formation of positional vectors, we explore their impact on the attention module, mainly focusing on the attention scores. We first extract queries and keys from each head in all layers, and then compute the average attention scores in the following four settings, including (1) original: the original model, (2) w/o semantic vector: removing the semantic vectors of keys and queries, (3) w/o positional vector: removing the positional vectors of keys and queries, (4) w/o positional basis: the logarithmic attention scores
removing the positional basis of keys and queries. for the first 100 tokens in the first head and fifth layer (similar results in many other heads and layers).
5

---

PPL During Direct Extrapolation
Maximum Similarity During Direct Extrapolation
TL-NsPE
Maximum cosine similaniy
0.9
TL-LLBi
TLWirdww
0.8
3.7
TLW/indseeet
TL-MIPE
2.5
0.4
A.Aindow
0.3
[WWindow-Rutt
TL TLWindoww 80
2020
Position
Position
Figure 4: Left: The average PPL across positions during direct extrapolation. Right: The maximum
cosine similarity between positional vectors within and beyond context window during extrapolation.
Effect of Positional Vectors Attention Sinks Previous work has found that the initial tokens will be assigned high attention scores, called "attention sinks" [15], which can be clearly observed Figure 3 However, once the positional vector or positional basis is removed from the keys and queries, the attention scores between initial tokens and other tokens drop significantly for TL-NoPE and TL-RoPE. This finding suggests that the presence of attention sinks is likely attributed to the inherent positional information in the positional vectors of initial tokens.
Effect of Positional Vectors on Long-term Decay For long texts, the attention scores of LLMs often exhibit a long-term decay pattern, which means that the score decreases as the relative distance between tokens increases [7. [6.. However, as shown in Figure 3 when removing the positional vector or positional basis, TL-NoPE fails exhibit long-term decay. Even with explicit relative positional encoding, the distribution of attention scores in TL-RoPE tends to be smooth after removing decomposed positional vectors. Therefore, positional vectors also play a crucial role in the long- term decay property of attention scores.
3.3 Effect of Positional Vectors beyond Context Window
Typically, when dealing with texts that exceed the context window, there are two lines of research, ie., direct extrapolation and context window extension. In this section, aim to investigate the change of positional vectors in these two methods for revealing their effectiveness.
3.3.1 Direct Extrapolation
Relationship Between Positional Vectors and Length Extrapolation Ability To examine the impact of positional vectors in direct extrapolation, we reuse the trained model variants in Table T to perform inference on samples consisting of 8192 tokens. Further, we analyze the change in PPL score and the maximum cosine similarity between positional vectors within and beyond the context window. As shown in Figure Left, only TL-Window-Rooh length extrapolation ability, maintaining stable PPL across longer texts. These models can preserve the consistency of positional vectors both within and beyond the context window (high similarity in Figure 4 Right). Conversely, the rest models, including those with extrapolated positional encodings or window attention (e.g., TL-ALiBi), struggle to generalize to longer contexts. Notably, these models exhibit rapid changes positional vectors (beyond 2048), diverging from the distributions observed within the context window. Thus, our findings underscore the critical role of the stability of positional vectors in enhancing the capability for length extrapolation.
and TL-Window-88 demonstrate the
Efffet OOD Positional Vectors Beyond the context window, position vectors are not encountered during training and are whether OOD positional vector key factor in performance degradation, we select TL-NoPE for evaluation, which does not use explicit positional encodings. First, we compare the attention distribution within and beyond the context window. Figure shows the attention map and scores between initial and rest tokens by averaging all heads of the 5-th layer (similar results in other layers). Once exceeding the context window (T 2048), the attention distribution in these positions changes sharply, losing the characteristics of attention sinks and long-term decay. Since these properties
from those vectors within the context window. To explore

---

highly depend on the positional vectors within the context window, we speculate that OOD positional vectors disrupt the original attention distribution. Besides, we feed the positional vectors of the last layer into the linear projection of the softmax layer to get the logits at different positions. Figure (Right) presents that the logits within the context window are tightly similar while others show different distributions. Thus, the OOD positional vectors can damage the token prediction probability distribution, thereby leading to performance degradation.
Legarithmic Attention
Attention Scores Initial Taken
Similarity Logits
1000
1003
1500
1549
2589
300
2033
10kc
2003
X00
400
3000
1000
2004
200
Posttoond id
Pas Id
positton
Figure 5: Left: Attention map of TL-NoPE. Middle: Attention Scores between initial token and
others in TL-NoPE. Right: Similarity of logits of positional vectors across positions in TL-NoPE.
Table 3: The interpolation results of positional vectors, where Factor (= Target Length/C") is the expansion factor of the context window, Ratio is the effective interpolation ratio of positional vectors (detailed in Appendix G. and Similarity is the average cosine similarity between the scaled positional vector and the original most similar positional vector by averaging all layers.
Model
Method
Target Length
Factor
Ratio
Similarity
PPL/APPL
Attention Scaling (A 1.2)
4096
2.56
0.98
8.95/+1.42
Attention Scaling (A 1.3)
8192
4.30
0.94
17.87/+10.34
TL-NoPE
Initial Scaling (A 1.2)
4096
2.38
0.97
9.82/42.29
Initial Scaling (A = 1.3)
8192
4
4.10
0.91
32.78/425.25
Dynamic NTK
4096
2.05
0.99
6.00/-0.02
TL-RoPE
Dynamic NTK
8192
4
3.75
0.96
6.78/40.76
3.3.2 Context Window Extension
Change Positional Vectors When Extending Context Windows To investigate why context window extension can prevent performance degradation, we analyze the change of positional vectors
two training-free context window extension methods, including dynamic-NTK (I] for TL-RoPE
and attention scaling (q.k, multiplied by scaling factor X) [20] for TL-NoPE. From Figure 6
we can see that after context window extension, positional vectors have undergone interpolation
compared to the original ones. Comparing the Factor and Ratio metrics in Table 3 conclude that
the effective interpolation ratio is close to the expansion factoor (e.g., 2 vs 2.56). Besides, as the expansion factor increases, there is decrease in Similarity and increase in PPL. Therefore, suspect that imperfect interpolation may a major reason for the decline in model performance.
TL-RoPE Dynamic NTK
TL-NoPE Attention Scaling
500
500
3000 1
1500
2000
2000
1000
2000
3203
1000
6000
1000
1063
5000
7000
0.800
-02
Figure 6: The average cosine similarity between the scaled and original positional vectors.
7

---

Effect of Initial Tokens on Context Window Extension Since the initial tokens serve as the anchor for the formation of subsequent positional vectors, we evaluate whether changing the information flow from the initial tokens to the rest tokens can achieve the interpolation effect. To avoid the effect of OOD positional encodings, we follow the attention scaling method on TL.-NoPE but only scale the attention logits between the initial tokens and others, denoted as Initial Scaling. As shown in Table 3 it can achieve comparable performance and interpolation ratios closer than scaling all attention logits in Attention Scaling (e.g., 2.38 vs 2.56), further underscoring that the interpolation of positional vectors is mainly achieved by adjusting the information flow of anchor tokens.
Extending Context Window via Positional Vectors
Inspired by our analysis of the formation of positional vectors and the interpolation of positional vectors when extending the context window, we propose two training-free context window extension methods, ie.. positional vector replacement and attention window extension, The pseudocode of these methods can presented in Appendix
4.1 Positional Vector Replacement
In Section 3.2.3, we show that when exceeding the context window, the OOD positional vectors tend to cause the collapse of attention distribution. Further, we observe that context window extension methods can achieve length interpolation of positional vectors and the effective interpolation ratio is close the expansion factor of the context window. Thus, we propose replace all the implicitly learned positional vectors with the interpolated ones, called positional vector replacement, to avoid the OOD issue in LLMs without positional encodings (NoPE).
Specifically, we linearly interpolate the positional vectors within the context window with an in- terpolation ratio and multiply the interpolated ones with times a (2 1). In practice, find that properly increasing the interpolation ratio r and times a can achieve better effectiveness of interpolation (details are discussed in Appendix D). Owing the critical role of initial tokes, the positional vectors of the first four tokens remain unchanged, while those of subsequent tokens are replaced with the interpolated vectors. The replaced output h1, for each layer can be formulated as:
hr b Pt. + apr.
(5)
(Pr...-.. (1)5()))
Interpolation (Pi,,).... Pi,)),,
where C, I, and s represent the original context window size, replaced layer, and interpolation ratio. Since replacing positional vectors for all layers requires heavy recalculation efforts and the positional information is passed across layers, we only apply the replacement strategy to a single early layer. We find that the 4-th layer is the optimal layer for replacement in TL-NoPE, as shown in Figure g
4.2 Attention Window Extension
As discussed in Section 3.2.2 the positional vectors shaped across layers and windows by the distinct positional information of initial tokens. Inspired by these observations, we propose attention window extension, the first training-free length interpolation method for window attention-based LLMs without positional encodings. The core idea is to extend the attention window size to control the formation of positional vectors. When scaling the context window by ratio, the window size also needs to be extended by the same interpolation ratio r. However, for positions in the extended first window {W + 1...., rW), ). their position vectors are OOD. To avoid this, we follow the attention scaling method (20) and scale the attention logits with scaling factor X, achieving better interpolation of positional vectors. We define the attention score azy between query q, R and key k, E R Pxx for any heads and layers as:
exp(Aq.ks/VDrr
w exp(Aq:k// (DH)
4.3 Results on Language Modeling
To assess the effectiveness of our proposed methods, we evaluate language modeling performance on the test set of PG-19 (22). In line with previous work (6), we measure PPL across various input
8

---

Table 4: Results of language modeling in PG-19, The context window size C 2048.
Model
Interpolation Method
Factor
2K
4K
6K
8K
10.17
10a
10"
> 10
TL-RoPE
Dynamic NTK
10.17
10.45
11.28
28.58
11.92
10
10a
> 10
À 1.2
17.03
17.05
54.26
>. 10
Attention Scaling
TL-NoPE
1.3
32.07
43.84
51.50
46,59
Positional Vector Replacement (ours)
r= 2, = 1.1
13.54
15.58
r= 5,0 = 1.3
28.15
47.65
49.79
73.79
12.86
713.51
660.30
660.51
TL- TL-Window
Attention Window Extension (ours)
2, A 1.1
13.70
14.10
4, 1.2
17.23
31.66
29.27
29.30
lengths (from 2K to 8K) using a sliding window approach. We apply positional vector replacement to TL--NPE and attention window extension to TL.-Window. All the hyper-parameters are selected according to the PPL. and the change positional vectors across layers. For compared baselines, we select Dynamic-NTK [ for TL-RoPE and Attention Scaling (20] for TL-NoPE.
The results are shown in Table 4 First, without interpolation, the PPL increases extremely after beyond the context window (e.g., 10a). When using the positional vector replacement or attention window extension methods, we observe that PPL decreases substantially, showing the effectiveness of our proposed methods. Compared to attention scaling, our attention window extension method successfully extends the context window to 8K tokens with lower PPL. Moreover, our positional vector replacement method achieves similar performance to attention scaling within 6K tokens but shows increased PPI. 8K. We attribute this phenomenon to the decreasing effective interpolation ratio across layers, as shown in Figure Additionally, an increase in PPI. with the rising interpolation ratio r is also observed in both our methods, likely due to imperfect interpolation of positional vectors.
Related Work
Position Information in Transformers Positional information was crucial in Transformer-basee LLMs, to enhance the sequence modeling abilities. The vanilla Transformer introduced absolute positional encodings, using a unique embedding to each position and adding it to the corresponding input embedding (4. In contrast, relative positional encodings introduced biases based on the relative distance between tokens within attention modules 22 6 7 Besides explicit positional encodings, some work investigated the implicit positional information within hidden states of Transformers. Even without positional encodings, positional information was found in hidden states of Transformer decoders [19 28, 2 Besides, prior work decoupled positional basis from hidden states in Trans- formers and analyzed geometric properties [21). Our work mainly explores positional information embedded in the hidden states of L.LMs, examining the formation and impact of positional vectors, and using it to analyze the mechanism of context window for LLMs.
Extending Context Window LLMs were often constrained by pre-defined context windows. When processing inputs that exceed these windows, models typically encountered OOD issues, leading significant performance degradation. To meet the growing demands of long context tasks [30] 31). various methods were proposed to address this limitation and model longer texts, which can be roughly categorized into length extrapolation and context window extension [32] Length extrapolation techniques aimed to maintain stable PPL regardless of text length by designing specialized positional encodings or window attention mechanisms 6 context window extension methods focused on extending the context window of existing models by adapting positional encodings or temperature hyper-pane parameters, thereby enlarging the context window with minimal performance loss 13
15 [4] Conversely,
12
16 20
10) This paper bridges the concepts of
length extrapolation and context window extension through the lens of positional vectors, enhancing the interpretability of context windows in LLMs.

---

6 Conclusion
In this work, we explored the inner working mechanism of LLMs within and beyond the context window via decomposed positional vectors. We found that the initial tokens initially present different positional information and serve as anchors for shaping the positional vectors of subsequent tokens. Besides, after exceeding the context window, length extrapolation methods maintain the stability of positional vectors, while context window extension methods achieve the interpolation positional vectors. Based on our observations, we proposed two methods: positional vector replacement and attention window extension, which achieve training-free context window extension for specific LLMs. We believe that positional vectors will serve effective tool for analyzing the context window of LLMs and promote the design of better algorithms for extending the context windows of LLMs.
Limitation
Our work provides extensive discussion and analysis of the context window through the lens of positional vectors. However, our study is mainly constrained by the use of small-scale LLMS that we trained ourselves, due the unavailability of existing LLMs with the specific positional encodings and attention patterns required for our experiments. Though some mainstream LLMs are evaluated, these models are all based RoPE. Furthermore, we have demonstrated the effectiveness of our proposed methods solely on our own models, again limited by the absence of suitable external models. In future work, we aim to seek a broader range of models to validate our findings more comprchensively.
Acknowledgement
This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Municipal Science and Technology Project under Grant No. Z23 100010323009, and Beijing Natural Science Foundation under Grant No. L233008. Xin Zhao is the corresponding author.
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Merbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[2] OpenAl. GPT-4 technical report. CoRR, abs/2303.08 087744 2023.
[3] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[5] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Lc, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. In ACL, 2019.
[6] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
10

---

[7] Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Ro- former: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063. 2024.
[8] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[9] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, hwtp22377.0988. 2023.
[10] bloc97. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023.
[1]] emozilla. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning. 2023.
[12] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. CoRR, abs/2300.00071, 2023.
[13] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306. 15595, 2023.
[14] Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm- Simple on-the-fly length generalization for large language models. CoRR, abs/2308.16137, 2023.
[15] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. CoRR, abs/2309. 17453, 2023.
[16) Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. L.LM maybe longlm: Self-extend L.M context window without tuning. CoRR, abs/2401 100125,, 2024.
[17] Ta-Chung Chi, Ting- Han Fan, Peter J. Ramadge, and Alexander Rudnicky. KERPLE: kernelized relative positional embedding for length extrapolation. In Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022.
[18] Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter J. Ramadge. Dissecting trans- former length extrapolation via the lens of receptive field analysis. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14. 2023, pages 13522- 13537. Association for Computational Linguistics, 2023.
[19] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1382-1390. Association for Computational Linguistics, 2022.

---

[20] Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaol- ing Wang. Length generalization of causal transformers without position encoding. CoRR, abs/2404. 12224, 2024.
[21] Jiajun Song and Yiqiao Zhong. Uncovering hidden geometry in transformers via disentangling position and context. CoRR, abs/2310.04866 2023.
[22] Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. In 8th International Confer- ence on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[23] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
[24] Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023.
[25] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Short Pupers), pages 464 468. Association for Computational Linguistics, 2018
[26] Zihang Dai, Zhilin Yang. Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhut- dinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 7778-2988.. Association for Computational Linguistics, 2019.
[27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:6; 2020.
[28] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[29] Ta-Chung Chi, Ting-Han Fan, Li-Wei Chen, Alexander Rudnicky, and Peter J. Ramadge. Latent positional information is in the self-attention variance of transformer language models without positional embeddings. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1183- 1193-193. Association for Computational Linguistics, 2023.
[30] Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Lànguistics, Language Resources and Evaluation, IREE//OLING 2024, 20-25 May, 2024, Torino, Italy, pages 2086-2099. ELRA and ICCL, 2024.
[31] Yuhao Wang. Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji- Ji-Rong Wen. Rear: A relevance-aware sarieval-auugenter framework for open-domain question answering. CORR, abs/2402. 2024.
[32] Saurav Pawar, S. M. Towhidul Islam Tonmoy, S. M. Mehedi Zaman, Vinija Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques in large language models A detailed survey. CoRR, abs/2401.07872. 2024.
[33] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR, 0bs/2307.08691 2023.
12

---

[34] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal. Anthony Hartshomn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.21788 2024.
[35] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. CoRR, (2//2403.04652.. 2024.
[36] Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309. 1 6609, 2023.
13

---

A Training and Experimental Details
We continue pre-training all our models from the Tiay [3] checkpoint on the RedPajama 24 dataset. All models undergo the same training process, with differences only in their positional encoding and attention patterns. Each model is trained on 16 A800 GPUs over two days. Detailed training parameters are provided in Table 5
Table 5: Training Details of Models.
Training Data
RedPajama [24
Tokens
50B
Parameters
1.3B
Context Window Size
2048
Decay style
cosine
Learning Rate
2c-5
Min Learning Rate
le-6
Optimizer
Adam Adam5(0..550099)
Warmup Steps
3000
Batch size
48
Gradient clipping
1.0
In addition, all the subsequent experiments are computed in 8 A800 GPUs.
B The Positional Vectors After the First Layer
Though previous work [29 28 have proven that implicit positional information can be encoded in hidden states after one attention module, they only set the attention logits are equal regardless of queries and keys, which does not hold in actual Transformers. In this section, we demonstrate the preference in attention scores promotes the formation of different positional information in the initial tokens.
B.1 Details
For the s-th sample in the corpus, we denote vi,i. hi; as the value and output of the first attention head and the first layer at position and af, the attention score between position i and position We also denote P1.: the positional vector of the attention output, and u, as the vector of values. The positional vector can be represented as the formula:
11,
vi
u
(8)
N
N
N
P1,
Σ N
Σ
Vi.j
(9)
For the first token, the output is equal to the value of itself, so the positional vector is equal to the mean of values. In addition, When a² = 1/6, positional information of all positions equal as follows:
N
PI.
Σ 1=1
u u.
(10)
However, due to the preferences attention scores, values that can be decomposed into more vector Pi.; u, will be assigned larger weights, making the positional information of the following tokens differ from the beginning token. In summary. the preferences of attention scores force the formation of different positional vectors of the following tokens with the initial ones.
http:///wugllgggooooo////////mip otg. 1431k-
intermediate-step
14

---

B.2 Proof of Preference in Attention Scores
Following previous work 28 we only focus on a single attention head in the first layer with specific weights. In our parameterization, we only consider the first two dimensions. We demonstrate the capacity of causal Transformers learn this capacity with either NoPE RoPE.
In our settings, the Transformer has H attention heads in each layer and the word embedding matrix is WE R RXV where D is the dimension of the hidden states and V is the number of vocabulary. We set that the first dimension in word embedding conforms to normal distribution (00, 1), (i.e. e1,t (O, 1), Vt E {I,..., V)), while the second dimension is Other dimensions arbitrary values. Thus, the word embedding matrix W is:
61,1
1122
1.3
WE E
63.1
63,2
63,3
V
(11)
CDD11
CD22
CD,3
CD.VJ DxV
Next, we set the projection matrix of query, key, and value as W Q. WK. Wy of the first layer and first head of Transformers.
0
1
0
[ 0
0
[ 0
07
0
1
1
0
1 0
W
W K
W V
DDD
XD
P (12)
WK k make sure that only the first dimension will be considered attention scores. Wo transfers the second dimension in the input to the first dimension. For Transformers without positional vectors (NoPE), the attention logits for each key at position i is of e,,,, Thus, the vectors with large first dimensions will be assigned with large attention logits, which proves that the model without positional vectors can learn to preference some values regardless of the query.
For Transformer with RoPE, assume that the first two dimensions correspond to the basis of the rope with a value of 1/ 1/10000 and the maximum context window size is 2048. Thus, for the largest relative distance as af,, = 61,s, cos ((i - j)/10000). Thus, the attention logits will be larger than zero only if the first dimension of the key is larger than zero. In addition, for the same relative distance, keys with larger first dimensions have large attention logits. Thus, keys with positive values in the first dimension will be assigned greater attention weights.
j. the rotation angle is smaller than 5. Thus, the attention score can be represent
Transformers with NoPE
Transfomers with RoPE
1.0
attention
Pin waceent of wich holdee Mtlle alsention
0.8
0.6
Rnd wich hitden
0.4
0.4
0.0
1000
1500
2000
súo
1000
1500
2000
Figure The values of first elements of the output single head attention due attention preferences in Transformers with NoPE and RoPE.
We also experimentally examine the first element of the output different positions. With the above settings, we generate 10000 sequences with length of 2048 from this distribution. Then,
15

---

Moditying Different Layers
Effective Interpolation Layers
5m
Effective Interpolation Ratlo
14,0==3
[-4,0=1
Logarthhnic
r=5, d-1.3
Original (8K)
Original (2K)
25
Layer
Layer
Figure 8: Changes of PPL with replacement at different layers.
Figure 9: Effective scaling factors of posi-
tional vectors each layer.
we compute the averaged first element of each hidden state after attention in the first layer and head, which represents the positional vectors that position for both NoPE and RoPE. As shown in Figure 7 the first element of the output increases fast at initial positions. Here, we can observe that the attention preferences make the first element of the output increase fast the initial positions and tend to stabilize at a later position, further proving the different positional vectors of following tokens from the initial ones.
In summary, Transformers with either NoPE or RoPE can learn preference in attention score. In addition, the preference in attention scores forces the different positional vectors of the subsequent tokens with the initial tokens.
C Defination of Effective Interpolation Ratio
Given Transformer with context window size C. For samples with length CC > C), disen- langle positional vectors {Pr.,...., ...l....PP,.r. Subsequently, we employ context window extension method and re-compute the position vectors {p' .i.....,'ii,,'))) We first define the corresponding position indices f'p'r.z (.)) for positional vector after extension P'i,)) the indices of the most similar positional vectors before extension:
PÍ.Pí. | IPL.
f(p' 1,) = arg max ISiS
(13)
Then, the effective interpolation ratio r' can be represented as the ratio between the maximum indices of position vectors after extension whose corresponding position indices are the context window size C, as shown in Equation 14
arg max (f(P' C)
(14)
D Analysis of Positional Vector Replacement
D.1 Optimal Replacement Layer
Replacing positional vectors for all layers requires heavy recalculation efforts, so we only select critical layer apply the replacement strategy. We evaluate the performance (logarithmic PPL score) of our replacement strategy at each layer in TL-NoPE, using different interpolation ratios and expansion factors, (re., 4, a 1) and (r 5, 1.2), samples with 8K tokens from RedPajama. As shown Figure B the PPL is the lowest when replacing the 4-th layer TL-NNPE. Thus, choose the 4-th layer for replacement in TL-NoPE in other experiments.
D.2 Effective Interpolation Ratio
We further examine the effective interpolation ratio of positional vectors with different settings. In setting, we only replace the positional vectors in the 4-th layer of TL-NoPE with interpolated
16

---

positional vectors in four settings on samples with 8K tokens from RedPajama: (1) = 4, 1, (2) 1.3, (4), 5, 1.3.
r
(3) r
Figure presents the effective interpolation ratio each layer. As the layer increases, the effective interpolation ratio decreases the layer increases. When the interpolation ratio is equal to the expansion factor of the context window, i.e., r r= 4, the effective interpolation ratio is unavoidably smaller than the expansion factor, leading to degraded performance. In addition, multiplying the interpolated positional vectors with a larger times e.g, a = 1.3, alleviates the decrease of effective interpolation ratio across layers. Thus, we suggest to properly increase the interpolation ratio and times a.
E
Pseudo Code
E.1 Positional Vector Replacement
For Positional Vector Replacement, we give the implementation with Pytorch code in Algorithm | which can be inserted after the output of the selected layer.
Algorithm Py Torch-sttyll Pseudocode of Positional Vector Replacement
h, # hidden states, positional vectors layer, s, alpha context window size, interpolation ratio, selected layer, scaling factor of positional vectors.
h[:4::] = pllayer, 4:h.shape| 1 unsqueeze(()) removing original positional vectors.
interpolated = fww.atiooaa..iicroooteeeetoloiei
Jrr.rrrposO.....uuunpppopp unsqqueeze(O), size
int(T*s), mode = 'linear', aligaalliinniaaa commess-TTrueerrrasssss(((
Linear interpolation of positional vectors.
h[:44:] += # Replacing with new positional vectors.
E.2 Attention Window Extension
For attention window extension, we give the implementation with Flash-Attention-2 33 in Algo-
rithm 2
Algorithm 2 PyTorch-style Pseudocode of Attention Window Extension
query, key, value, attn output # queries, keys, values, and output in the attention model W, lambda, s original window size, scaling factor of attention logits, window extension factor flash flash_attn_ ait__att___aaatt farle__arle___uuu func attention function in flash attention 2.
nendwindow W*s extended window size
attn catn_ooututt-- flash attn varlen func( query, leydaambda, value, window size (new window, new new_wiiwee www_window)
F Results of Additional LLMs
To ensure a fair comparison of positional vectors across various attention mechanisms and positional encodings, we conducted continual pre-training using TanyL.ama under consistent settings. However, Lama is relatively small language model with suboptimal performance and continual pre- training may result in positional vectors exhibiting properties distinct from those obtained through
17

---

PCA Liama-3 Layer #1
PCA Qwen1.5 Layer #1
600
401
200
1000
4000
200e
53
2000
200
430
660
993
200
410
600
BGC
1090
PCA Yi Layer #1
PCA TL-NoPE Layer #1
1303
300
2000
1000
100
1000
500
200
400
030
000
1000
-100
0
300
200
300
400
500
Figure 10: PCA visualization positional vectors from the 1-st layer of 1llam--3-8B Qwenl Yi-9B, and TL-NoPE-new.
training from scratch. Therefore, we selected three mainstream LLMs: Llama-3-8B 34
Yi-9B 35
and (Dwel1.5-7) (36), for comparison. Additionally, we trained a new LLM, TL-NoPE-new, from scratch under the same conditions as TL-NoPE. similar vein, we extracted positional vectors using 32K samples from the RedPajama dataset.
F.1 Formation Positional Vectors within Context Window
Through principal component analysis (PCA), first visualize the positional vectors from the initial layer of these LLMs, as illustrated in Figure [0] Consistent with our expectations, the initial tokens exhibit distinct positional information, while the subsequent tokens display a high degree of similarity. This observation supports the conclusion that the first-layer attention mechanism makes the initial tokens form unique positional information, diseussed in Section 3.2.1
Furthermore, we remove different components the value vectors different positions across all attention heads after the first layer. We then evaluate the impact of these modifications on both the positional vectors and the perplexity (PPL). As shown Table removing the positional basis the initial tokens significantly degrades the model's performance. Conversely, removing the components from subsequent tokens has a relatively smaller effect, highlighting the pivotal role initial token positioning in influencing later tokens. However, observe that larger LLMs are more attuned to semantic information and less affected by the removal of positional vectors compared to smaller LLMs.
Table Results of removing different components in attention.
original
value
positional vector
w/o positional basis
wlo semantic basis
0-.4
32-256
0-4
32-2556
0-4
32-256
0-4
32-256
simi
0.75
0.9995
0.75
0,9583
0.2059
0.9997
0.9259
0,8666
Lluma-3
ppl
6.74
16.27
6.63
17.20
8.4
>100
6.60
17.6
15.18
Yi-9B
simi
0.98
0,9999
0.92
0.9998
0.5368
1
0.91
0.9996
ppl 7.08
8.03
6.56
37.92
6.62
>100
42.271
7.08
Qwen-1.5-7B
simi 0.98
0.9997
0,9847
0.9986
0.7382
0.9993
0.9998
0.9951
ppl
7.97
9.51
8.03
9.51
8.04
217.13
7.98
8.09
8.68
simi
0.70
0.95
0.69
0.95
0.41
0.93
0.99
1.0
TL-NoPE-new
11.03
224.74
22.36
263,53
20.91
>100
21.78
11.66
12.699
F.2
Effect of Positional Vectors Attention
In line with the experiments conducted in Section 3.2.3 we extract various components from the keys and queries to assess their impact on attention scores. The attention maps for the first 50 tokens are illustrated in Figure II When both the positional vectors and positional basis are removed, attention
18

---

Llama-3 original
Llama-3 we semantic vector
Llama-3 w/o positional vector
Jama-3 we positionall basis
-1
-14
20
14
44
Owen1 5 original
Qwen1.5 5 w/o semantic vector
Qwen w/o positional vector
Qwen1. w/o positional besis
66
12
origginal
positional
TL-NoPE original
TL-NoPE wio semantic vector
TL-NoPE w/o positional vector
TL-NoPE wlo positional basis
Figure 11: Logarithmic attention maps Llama-3-8B, Qwenl.5-7B. Yi-9B, and new TL-NoPE.
sinks disappear, and long-term decay is observed, which is consistent with the behavior seen in TinyL
F.3
Effect of Positional Vectors Beyond Context Window
Table Resuls of PPL and change of positional vectors during direct extrapolation.
model
context window(C)
PPL(C)
PPL.(2C)
Simi(2C)
Llama-3-8B
8192
6.74
>1000
0.30
Yi-9B
4096
7.08
102.58
0.24
TL-NoPE-new
2048
11.75
351.49
0.71
Table 8: Change of attention sinks and output logits beyond context window.
model
context window (C)
property
0~C
1~1.5C
1.5C~2C
attention sink
0.467
0.1
0.005
Llama-3-8B
8192
logits similarity
0.9
0.88
attention sink
0.68
0.344
0.056
Yi-9B
4096
logits similarity
0.98
0.97
attention sink
0.17
0.02
0.0006
TL-NoPE-new
2048
logits similarity
0.97
0.9
We investigate the behavior of positional vectors under direct extrapolation scenarios. Specifically, we evaluate I.IMS on sequences twice the length of their maximum context window, comparing the variations in PPL and positional vectors. The results, shown in Table 7 indicate sharp
19

---

increase in PPL once the context window is exceeded. Additionally, the similarity between positional vectors within and beyond the context window decreases, highlighting the relationship between length extrapolation and the stability of positional representations.
Furthermore, assess the impact of OOD positional vectors. We analyze the attention scores of initial tokens and the positional vectors of output logits within and beyond the context window. As demonstrated Table 8 exceeding the context window results the loss of the attention sinking property, and the positional vectors of the logits exhibit different distribution compared to those within the context window. This suggests that OOD positional vectors influence the model's distribution. Moreover, OOD positional encodings play a crucial role in large language models (LLMs) that leverage RoPE [5
F,4 Experiments of Positional Vector Replacement
To assess whether our positional vector replacement method is specific TL-NoPE, we applied it to TL-NoPE-new, which was trained from scratch. We evaluated its language modeling performance on the PG-19 dataset, using the same experimental setup as TL-NoPE. As shown in Table 9 our approach effectively extends the context window to 8K tokens and achieves performance comparable attention scaling.
Table 9: Results of language modeling in PG-19 with TL-NoPE-new
Interpolation Method
Factor
2048
4096
6144
8192
29.0
110.7
634.8
1078.5
1.2
29.0
27.3
Attention Scaling
1.5
36.8
41.6
44.4
46.2
r=2.5,a=0.8
32.8
29.5
Positional Vector Replacement(ours)
r=5, 0=8.
57.4
50.8
40.0
44.2
G Visualization of Positional Vectors
We visualize all models listed Table I with PCA. The subsequent figures present the visualization of TL-NoPE, TL-RoPE, TL-ALiBi, TL- Window, TL-Window-Rel and TL-Win S0-
Lyyrr
Layer
Layer
Lavere
Layer
Layer
Laysr
Layer
Laya
Laper
Lassr
10
Low
Lpper 33
Lose
Figure 12: PCA visualization of positional vectors at different layers of TL-NoPE.
20

---

Loper
Layer
Dayer
11
Oayer
Dawer
Layer
Figure 13: PCA visualization positional vectors at different layers of TL-RoPE.
Dayrr?
30 10
Laper
3 30c 10
1 20 Laper
10 100
15 10 Oppr11
199 15
S Laye
Layar 29
Laye 10
26 Layo
Figure 14: PCA visualization of positional vectors at different layers of TL-ALiBi.
Liyorig
Gret 10
Loyerts
Lavir
Laya
Laper:3
Layer
Layer37
Dyyer1
Layer
Layar
Figure 15: PCA visualization of positional vectors at different layers of TL-Windows
21

---

Layor
Layer
Loyer?
Loycr
Layer
Liye
Layer 15
Layer 16
Layer
Layer
Luper
Layer
Layer
Figure 16: PCA visualization of positional vectors at different layers of TL. TL.-Widow-RoP.
Layera
Iaper'
Liver 10
Liver 11
Paper
Layer 13
Layer 12
Tayer
Layer
Laper
Layer
Tayer
Figure 17: PCA visualization of positional vectors at different layers of TL- TL--indow-80.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide the main claims abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations, A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
22

---

Justification:
We discuss the limitation in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We have provided proof in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provide the information for reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We only provide the Pytorch implementation of our methods in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guideeee http://oeeuuiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the training details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not provide this.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/nneurppe http:/uwww........ooo...aaaaaaaa
Answer: [Yes]
Justification: The paper respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite existing papers and url.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The models trained by us can not be launched. However, we give the details of training these models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
27

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "MWlqHVPBbE", "reviewer": "Reviewer_R24u", "summary": "This paper dives into the inner workings of how transformer-based language models handle positional information. By decomposing hidden states into semantic and positional vectors, the authors give a series of analysis about how the positional information are encoded and propagated through layers. I believe this work offers valuable insights for understanding the positional information within the transformer architecture.", "questions": "see the above weakness section", "limitations": "see the above weakness section", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Very detailed and clear analysis about how the positional information is encoded and propagated within the transformer architecture, and to the best of my knowledge, I have not seen similar work before. I particularly enjoyed reading Figure 2 and 3, which shows how positional information is propagated through layers and goes beyond the window size, and shows how the manipulation of the positional embedding causally influence the attention patterns, particularly removing the attention sink.", "weaknesses": "There are few points that I would like to suggest here to make the paper even stronger.\n\n- Section 4 feels weak and unnecessary. The performance of replacing the positional vector, if my understanding is correct, seems to be much worse than Dynamic NTK. Given the current mainstream approach is modifying the base of Rope (like YaRN), which is much easier than the approach proposed by this work, I do not think this work’s proposed context extension will be accepted by mainstream model builder.\n- That being said, I think the in-depth analysis of the positional embeddings are strong enough for me to give an acceptance (I learned a lot from it), so **I would strongly suggest removing the content of section 4, and use its space for more experimental analysis of the positional vectors**\n\nThere are a few important problems that I believe will receive the communities’ attention and worth being addressed:\n\n- Although this paper shows the positional information can propagate through layers (Figure 2), in practice, many work found that models with window attention cannot pass the needle in a haystack test, and this is why Mistral 1.5 changed its attention back to full attention. It would be insightful if the authors can discuss the relationships between positional information and needle-in-a-haystack performance (because needle in haystack is what makes long-context models useful), i.e., why window attention cannot pass needle in haystack even it does have the correct positional information?\n- This paper’s analysis is restricted on TinyLLaMA, but TinyLLaMA is not a widely used open-source model, thus casting the doubt whether this discovery of this paper will hold for other model families, particularly mainstream open-weight models like LLaMA 3, Mistral, QWen or Yi. I would strongly suggest the authors verify the behavior of positional embedding on either LLaMA 3, Mistral, QWen, or Yi.\n\nCurrently I’m given a borderline accept, and I will consider increasing my scores if the authors could either (1) discuss the relationship between positional vectors v.s. needle-in-a-haystack or (2) verify that the properties of positional vectors hold for LLaMA 3, Mistral, QWen or Yi (any 2 out of the 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments!\n\n# W1: Unnecessaries of Section 4\n\nIn Section 4, the proposed methods are significant evidence for our analysis of the relationship between positional vectors and the context window. Our experiments substantiate our previous viewpoints. For instance, interpolating positional vectors can extend the context window to some extent. Although it has poorer performance than Dynamic NTK, it provides a new approach to solving the problem of context window extension and can be applied to models without RoPE. In the final version with one additional page, we will adopt your suggestions by supplementing more experiments and condensing the content of this section to some extent.\n\n# W2: Failure of Window Attentions in Needle in the Haystack\n\nWhy can window attention hardly solve the needle in the haystack problem?\n\n* Some work has reported that models with full attention can utilize \"retrieval heads\" to directly retrieve critical information from the needle [1], while window attention cannot directly attend to these tokens.\n\n* Models with window attention acquire knowledge only through the indirect transmission of information between windows. However, previous work has highlighted that there is severe attenuation in the information passing between windows, making it difficult to leverage semantic information from distant tokens [2].\n\n* Positional information tends to be a kind of global information at a given position, ensuring the model can produce coherent output, and is orthogonal to the semantic information of the input.\n\nTo validate this, we first compare the hidden states of tokens when using the full input sequence versus using only the window-sized input at a time. We test TL-RoPE-Window and Mistral-7B-v0.1 with input lengths of 16K and 32K, respectively. We find that the similarity of the last layer's hidden states under both settings was above 0.98, suggesting that information from tokens outside the window has little impact.\n\nAdditionally, we test the scenario where the \"needle\" is placed at the beginning but is modified into random tokens from the vocabulary. The hidden states of the outputs remain largely unchanged, further supporting our hypothesis.\n\n[1] Retrieval Head Mechanistically Explains Long-Context Factuality\n\n[2] Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis\n# W3: Absence of Some Mainstream LLMs\nConsidering resource constraints, we only continuously pre-trained TinyLlama with modified positional encoding under identical settings for a fair comparison. The mainstream models are all based on RoPE, and continuing pre-training is prohibitively expensive; hence, we only examined the positional vectors of the original Llama3-8B, Yi-9B, and Qwen1.5-7B.\n\n* Formation of positional vectors: After the first layer, the initial positions have significantly different positional vectors compared to other tokens, as shown in Figure 1 of the PDF. Furthermore, we remove different components from values at different positions and observe changes in the positional vectors and PPL, as shown in the table below. The findings are consistent with the original paper, indicating that the initial tokens play a critical role in shaping the positional vectors and influencing the model’s PPL. When the positional information of the initial tokens is lost, the model loses its ability to produce coherent output. Moreover, Llama-3-8B and Yi-9B are more sensitive to semantic information than smaller models.\n|  |  | original | w/o value || w/o positional vector | | w/o positional basis  | | w/o semantic basis | |\n|---|---|---|---|---|---|---|---|---|---|---|\n| position | - | - | 0-4 | 32-256 | 0-4 | 32-256 | 0-4  | 32-256 | 0-4 | 32-256 |\n| Llama-3 | simi | 1 | 0.75 | 1.0 | 0.75 | 0.96 | 0.21 | 1.0 | 0.93 | 0.87 |\n| Llama-3 | ppl | 6.74 | 16.27 | 6.63   | 17.20 | 8.4 | >1000 | 6.60 | 17.6 | 15.18 |\n| Yi-9B | simi |1| 0.98 | 1.0 | 0.92  | 1.0 | 0.54 | 1 | 0.91 | 1.0 |\n| Yi-9B | ppl | 6.51 | 8.03 | 6.56 | 37.92 | 6.62 | >1000  | 6.52 | 42.27 | 7.08 |\n| Qwen1.5-7B | simi | 1 | 0.98 | 1.0 | 0.98 | 1.0 | 0.74 | 1.0 | 1.0  | 0.99 |\n| Qwen1.5-7B | ppl | 7.97| 9.51| 8.03 | 9.51 | 8.04 | 217.13 | 7.98 | 8.09 | 8.68 |\n\n* Impact of Positional Vectors on Attention: We examine the changes in attention after removing the positional vectors, as illustrated in Figure 2 of the PDF. We observe that attention sinks and long-term decay are eliminated after removing the positional vectors or basis.\n\n* Effect of Positional Vectors Beyond the Context Window: As shown in the following table, after exceeding the context window, models without an extended context window experience a sharp change in positional vectors and PPL, similar to TL-RoPE. By using dynamic NTK, we achieve interpolation of the positional vectors, maintaining high similarity to the original positional vectors.\n| model | context window(W) | PPL(W) | PPL(2W) | Simi(2W) | PPL(NTK, 2W) | Simi(NTK, 2W) |\n|---|---|---|---|---|---|---|\n| Llama-3-8B | 8192 | 6.74 | >1000 | 0.30 |7.84 | 0.96 |\n| Yi-9B | 4096 | 6.51 | 102.58 | 0.24 | 6.70 | 0.99 |\n* Effect of OOD Positional Vectors: After exceeding the context window, the properties of high attention scores on the first token (attention sinks) in Llama-3 and Yi-9B rapidly disappear, and the logits of the positional vectors differ from those within the window, as shown in the table below. However, the change in logits for Yi is not obvious, although it has formed the same pattern as shown in Figure 5 (right) of the paper.\n| model | context window (W) | property | 0~W   | W~1.5W | 1.5W~2W |\n|---|---|---|---|---|---|\n| Llama-3-8B | 8192 | attention sink    | 0.47 | 0.1 | 0.005 |\n| Llama-3-8B | 8192 | logits similarity | 1 | 0.9 | 0.88 |\n| Yi-9B | 4096 | attention sink | 0.68 | 0.34 | 0.06 |\n| Yi-9B | 4096 | logits similarity | 1 | 0.98 | 0.97 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ky79P1tkV6", "reviewer": "Reviewer_1sVW", "summary": "This paper proposes a mean-based decomposition technique to analyze the formation and effect of positional encodings in LLMs. It then uses these results to propose methods to extend the context window, resulting in models that generalize better to longer texts.", "questions": "1. The legends and graphs in Figure 4 overlap.", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is very well-written, and the main findings are properly highlighted.\n\n2. This paper not only explains how positional vectors are formed, but also introduces methods to interpolate them based on the findings.\n\n3. Experiments are performed to show that the new methods result in better perplexity scores beyond the context window.", "weaknesses": "I believe this contribution is novel and insightful enough, and there is no apparent weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments! We will revise Figure 4 in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2EEgqJtRt4", "reviewer": "Reviewer_yDA4", "summary": "This paper disentangles positional vectors from the hidden states of a pretrained Transformer language model to facilitate the understanding of length extrapolation. After a series of analyses, this paper proposes two context extending techniques. Experiments show that the proposed methods lower the perplexity on the task of language modeling.", "questions": "1. Instead of continue training from the tinyllama model, I think training models from scratch using the 50B tokens budget will make the results in this paper more convincing. This is because you can get rid of the ripple effect of the originally used rope positional embeddings. Maybe your models were trying to unlearn rope during the continue training stage?\n2. Apart from testing perplexity scores on the task of language modeling, I highly recommend the authors adding the experiment of needle in a haystack, otherwise I do not know if the models are really using all the tokens.\n3. How do you decide the values of alpha and lambda in section 4.1 and 4.2? In addition, the temperature scaling technique was also used in several other places [1, 2] with explanations of how they did temperature selection.\n\n[1] YaRN: Efficient Context Window Extension of Large Language Models\n[2] Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "It's always good to have a mechanistic interpretability view of the hidden states of language models. The findings presented in this paper might inspire follow-up work along this direction.", "weaknesses": "The experiments presented in the current draft are not convincing enough to me. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments!\n# Q1: Training from Scratch\n\nWe initially performed from-scratch pretraining on smaller models and found that the properties of the positional vectors were largely similar to continually-trained models, but the models trained from scratch had inferior performance. To address your question, we pretrained TinyLlama from scratch with the same configuration, but using different positional encodings and attention patterns. Due to time and resource constraints, we only trained TL-NoPE-new for 50B tokens, while the other configurations were trained for only 5B tokens.\n\n* Formation of positional vectors for TL-NoPE: For the TL-NoPE-new trained from scratch, the formation of its positional vectors is similar to the continually trained one: the initial tokens exhibit distinct positional information after the first layer (as shown in Figure 1 in the PDF). As the table below shows, removing the positional information of the initial tokens significantly harms the performance, indicating that the flow of this positional information facilitates the formation of subsequent tokens' positional information.\n\n| |   | original | w/o value | | w/o positional vector |       | w/o positional basis  |       | w/o semantic basis |        |\n|----|----|----|----|-----|--|-----|-----|-----|--------|-----|\n| position | - | - | 0-4 | 32-256 | 0-4 | 32-256 | 0-4  | 32-256 | 0-4 | 32-256 |\n| TL-NoPE-new | simi |     1    |    0.70    |  0.95 | 0.69         |  0.95 |          0.41         |  0.93 |        0.99        |   1.0  |\n| TL-NoPE-new |  ppl |   11.03  |   224.74  | 22.36 |         263.53        | 20.91 |         >1000         | 21.78 |        11.66       | 12.699 |\n\n* Formation of positional vector for TL-NoPE-Window-new: For window attention with NoPE, the positional information flow from initial tokens to subsequent tokens also occurs, and the distinct positional vectors gradually propagate across both windows and layers. The distinct positional vectors of the initial layers are shown in the table below.\n\n| layer |  1 |   2 |   3 |    4 |    5 |    6 |    7 |\n|--|---:|----:|----:|-----:|-----:|-----:|-----:|\n| Distinct Positional Vectors | 79 | 561 | 922 | 1303 | 1682 | 2038 | 2048 |\n\n* Effect of positional vectors on attention: we subsequently remove different components from the query and keys (shown in Figure 2 in PDF), and observe that after removing the positional vectors or positional basis, the long-term decay and attention sink all disappear. \n* Positional vectors change beyond the context window:  as shown in the following table, with inputs exceeding the context window (2K), only models with consistent positional vectors exhibit length extrapolation capacities. In addition, when we employ attention scaling on TL-NoPE, its PPL score on 8K length is 14.05, slightly larger than 11.75 in the 2K length of the original model. The similarity between the original and extended models is 0.97, indicating the interpolation of positional vectors.\n\n| models                | PPL(2K) | PPL(8K) | similarity(8K) |\n|---|---|---|---|\n| TL-NoPE-new  |   11.75 |  351.49 |     0.71 |\n| TL-NoPE-Window-new    | 70.05 |  105.42 |   0.80 |\n| TL-RoPE-Window-new    |  24.95 |   23.43 |           0.98 |\n| TL-NoPE-Window-80-new |  69.91 |   68.12 |           0.99 |\n| TL-ALiBi-new          |        37.63 |   36.17 |           0.99 |\n\n* Effect of OOD Positional Vectors: similarly, the attention sinks and logits of positional vectors of TL-NoPE-new also change after exceeding the context window.\n\n| model       | context window | property  | 2048 | 4096 |   8192 |\n|--|---|----|-----:|-----:|----:|\n| TL-NoPE-new |           2048 | attention sink    | 0.17 | 0.02 | 0.0006 |\n| TL-NoPE-new |           2048 | logits similarity |    1 | 0.97 |    0.9 |\n\n* Proposed Methods: Finally, we test our proposed methods with the from-scratch trained TL-NoPE-new and TL-NoPE-window-new, and observe similar phenomenons. Both positional vector replacement and attention window extension can achieve context window extension. However, due to the properties of these models, the hyper-parameters and the original performance are different. \n\n| Model              | Interpolation Method                | Factor            |  2048 |  4096 |  6144 |   8192 |\n|---|---|---|---:|---:|---:|---:|\n| TL-NoPE-new | -  | -  |  29.0 | 110.7 | 634.8 | 1078.5 |\n| | Attention Scaling   |  lambda = 1.2 |  29.0 |  27.3 | -     | -      |\n| |   |lambda=1.5 |  36.8 |  41.6 |  44.4 |   46.2 |\n| | Positional Vector Replacement(ours) | r=2.5, alpha=0.8 |  32.8 |  29.5 | -     | -      |\n| |  | r=5, alpha=0.8   |  57.4 |  50.8 |  40.0 |   44.2 |\n| TL-NoPE-Window-new | - | -  | 112.7 | 141.5 | 149.7 |  149.7 |\n|  | Attention Window Extension(ours)    | r=2,lambda = 1.1 | 116.5 | 114.7 | 139.6 |  151.7 |\n|   |  | r=4,lambda = 1.2 | 122.3 | 124.4 | 123.3 |  126.0 |\n\n# Q2: Absence of Needle in the Haystack\n\nWe initially explored the \"needle in a haystack\" task. However, due to the poor performance of TinyLlama, the models we continued to pre-train exhibited diminished retrieval capabilities. In our tests, even with an input text length of 200 tokens, the models failed to generate accurate responses, regardless of the positional embeddings and attention patterns employed. Given the difficulty of the \"needle in a haystack\" task, we did not include the outputs in our paper.\n\n# Q3: How to Decide Values of Alpha and Lambda\n\nFor the alpha value in positional vector replacement, we conducted experiments in Appendix E according to the effective interpolation ratio and PPL. We tested different combinations and identified the optimal interpolation ratio, replacement layer, and interpolation times (alpha). For lambda in the attention window extension, we similarly used a PPL-based search to determine the scaling factor. We will further point out how to decide these hyper-parameters and add citations in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for your insightful comments! The supplementary PDF includes figures that support our rebuttals.

Meta-review: This paper presents a novel approach to understanding how positional information is encoded and propagated within transformer-based language models. With a mean-based decomposition, they decompose hidden states into semantic and positional vectors, the authors provide detailed insights into the role and behavior of positional embeddings. Also, the work proposes methods for extending the context window based on these findings, aiming to improve model performance on longer sequences.

Pros:

The reviewers mostly recognize the paper's in-depth analysis of how positional information is encoded and propagated within transformer layers. This is a novel contribution that can benefit both research and industry aspects of LLMs.

The paper is well-written and organized, with the main findings and contributions clearly highlighted. The figures are praised for their clarity and the insights they provide into the positional information.

Cons:

Generalization Across Models: Both Reviewer R24u and yDA4 highlighted concerns regarding the generalizability of the findings. The experiments were conducted on TinyLLaMA, which is a less powerful model, raising doubts about whether the results would hold for more mainstream models.
Decision: Accept (spotlight)