Exploring Context Window of Large Language Models via Decomposed Positional Vectors

Authors: zican  dong
junyi  li
xin  men
wayne xin zhao
bingning  wang
zhen  tian
weipeng  chen
ji-rong  wen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:40:52

Abstract: Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

Main paper: 
 introduction: 
 Recently, Transformer-based large language models (LLMs) have demonstrated excellent capabilities on downstream tasks [1] [2] [3] , in which positional encodings (e.g., absolute or relative) are widely used in Transformers to better capture positional information within input sequences [4, 5] . However, LLMs typically suffer from a limited input length (called context window), which is constrained by the maximum length of training data. Beyond the context window, the positional encodings at larger position indices are out-of-distribution (OOD), not encountered during the training phase. Therefore, when the input sequence exceeds the context window length, there would often be a significant degradation in model performances, as evidenced by a surge in perplexity (PPL) score [6] .Prior work has primarily focused on extending the context window of existing LLMs by manipulating positional encodings. Owing to its excellent performance and long-term decay nature, RoPE [7] has been widely used to learn positional encodings for existing LLMs [8, 9] . To circumvent the OOD positional encodings in RoPE, various methods have been proposed to modify the base [10] [11] [12] or positional indices [13] [14] [15] [16] . In addition, special relative positional encodings that apply larger negative biases to attention based on the relative distance have achieved promising length extrapolation, which can effectively stabilize the model performance beyond the context window [6, 17, 18] . Furthermore, decoder-only Transformers without positional encodings (NoPE) have been found to be capable of learning implicit positional information [19] , and their context window size can be extended via the adjustment of temperature hyper-parameters [20] . However, the above extension methods solely focus on adapting positional encodings or attention scores, lacking a detailed analysis of the underlying mechanisms of hidden states in LLMs.In this work, we aim to investigate the inner working mechanism of LLMs within and beyond the context window to interpret these context window extension approaches. As the basis, our work is developed by analyzing the positional information implicitly encoded in the hidden states of LLMs across various layers and positions, both within and outside the context window. Inspired by previous work [21] , we use a mean-based decomposition approach to disentangle positional vectors from the hidden states, which captures the information independent of semantics but related to positions. Specifically, we first investigate how positional information is formed and examine its impact on the attention mechanism within the context window. Second, for inputs beyond the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Our key findings include: (1) After the first layer, initial tokens can form distinct positional vectors, serving as anchors for shaping positional vectors in subsequent tokens; (2) Positional vectors play a critical role in modulating the long-term decay and establishing attention sinks; (3) When exceeding the context window, OOD positional vector is the major factor contributing to performance degradation, while length extrapolation can effectively keep the consistency of positional vectors both within and beyond the context window; (4) Context window extension methods enable interpolation of positional vectors by adjusting the information flow from initial tokens to subsequent tokens.Based on the empirical findings, we further propose two training-free context window extension methods from the perspective of interpolating positional vectors: positional vector replacement and attention window extension. For LLMs with NoPE, the former method replaces the positional vectors in critical layers with interpolated ones; while for LLMs with window attention and NoPE, the latter method directly scales the window size and adjusts the temperature hyper-parameter. We evaluate the length generalization capacities of the proposed methods on PG-19 [22] . Experimental results demonstrate that our methods can effectively generalize to longer texts without fine-tuning, achieving comparable performance to previous methods.Our main contributions are summarized as follows:• We explicitly delineate the formation process and the effect of positional vectors, highlighting the anchoring role of initial tokens in shaping different positional vectors across tokens and their importance in achieving long-term decay and attention sinks. • We are the first to unify length extrapolation and context window extension from the perspective of positional vectors, identifying that preventing OOD positional vectors is crucial for avoiding performance degradation. • We propose two training-free context window extension methods via the lens of adjusting positional vectors, i.e.,positional vector replacement and attention window extension. Experimental results show that our methods can effectively generalize to longer texts without fine-tuning. 
background: 
 Transformer Decoder-only Transformer [4] has become the foundational architecture for LLMs [4, 8, 1] . For a Transformer with L layers and a context window size C, given an input sequence s of T tokens, i.e., {x 1 , . . . , x T }, we denote the output of the l-th layer l as H s l = {h s l,1 , . . . , h s l,T }. At each layer, the output H s l is obtained through multi-head attention (MHA) and feed-forward network (FFN) with residual connections applied to both components as follows:EQUATIONFinally, the output of the last layer H s L is then projected into the logits, which will be used to generate the prediction probability for each token in the vocabulary.Positional Vector Previous work has found that positional information can be learned and encoded in the hidden states of Transformers [19] . Drawing inspiration from prior work [21] , we hypothesize that each hidden state (e.g., query, key, value, output of each layer) within Transformer can be decomposed into two parts, i.e., a positional vector that captures positional information and a semantic vector that captures the contextual information. Taking the output h s l,t of the l-th layer at t-th position as an example, it can be decomposed into a positional vector p l,t and a semantic vector c s l,t : h s l,t = p l,t + c s l,t .(2) Such a decomposition can disentangle two primary factors, namely positional and semantic vectors, for interpreting the internal mechanism of LLMs. Notably, since positional vectors are globally shared across different inputs, there is no superscript s for p l,t . Further, the positional vector p l,t can be decomposed into a mean vector u l and a positional basis m l,t :EQUATION) where the mean vector u l denotes the mean of the distribution of positional vectors and the positional basis m l,t denotes the offset of t-th position from the mean vector within the context window size C. Following previous work [21] , we adopt a mean-based decomposition method to obtain the above three vectors based on N samples from the training corpus as follows:EQUATIONWith this decomposition, it offers an explicit way to analyze and explore the positional information encoded in the hidden states of Transformer models. For example, we can use similarity measurements to compare the positional vectors of different positions and also can visualize them in low-dimensional embedding space. In the following sections, we will mainly focus on studying the formation and impact of the positional vector p l,t , and conduct the analysis experiments.3 Empirical Analysis 
experimental settings: 
 To better analyze positional information, we consider model variants with different positional encodings (PE) and attention mechanisms: variants without positional encodings (NoPE) [19] as well as variants with two different positional encodings: RoPE [7] and ALiBi [6] . We continually pre-train the TinyLlama-1.1B checkpoint [23] on 50B tokens from RedPajama [24] with a context window C = 2048, resulting in a set of comparison models with different positional encodings and attention mechanisms, as shown in the Table 1 . Full attention means that each token can attend to all previous tokens, while window attention restricts each token to attend only to previous tokens within a window size W . The training details are described in Appendix A. We also evaluate common LLMs (e.g., Llama-3-8B) and LLMs without positional encodings trained from scratch, and the evaluated results are listed in Appendix F.Specifically, we subsample 32K samples with the same number of tokens from RedPajama. We perform the inference on these data to obtain hidden states of LLMs. By using the mean-based decomposition method (Eq. 4), we can obtain the positional vectors p l,t of tokens in these sample texts. In existing LLMs, the bottom (first) layer typically takes as input token embeddings that lack inherent positional information; while interestingly, the hidden states from top layers can implicitly capture positional information, even without explicit positional encodings [19, 21, 14] . In order to have a deep understanding of implicit positional information, we next investigate the formation and effect of positional vectors in Transformers, with both full attention and window attention. We can prove that the positional vector p 1,1 for the first token is different from the following tokens and the attention scores affect the formation of positional information. Thus, through several layers, the tokens after the first token will gradually form distinct positional vectors (Figure 1 right column).Positional Information Flow From Initial Tokens By applying positional vectors at top layers (PCA visualized in Appendix G), we find that after forwarding several layers, tokens at all positions can also exhibit distinct positional vectors, and similar findings are also found in previous work [19] .To trace back to the source of positional information, a reasonable speculation is that initial tokens play a key role in the formation of positional information for the rest tokens since only initial tokens capture positional information after the first layer. To validate this, we select two groups of reference tokens: initial tokens (1∼4) and secondary tokens (4∼256), and further analyze what information of these tokens is critical for the formation of positional information in subsequent tokens (>256). Thus, based on a top-down strategy, we conduct an ablation study for each group by respectively deleting the value v s l,t of attention module (w/o value), the semantic vector c s l,t (w/o semantic vector), positional vector p l,t (w/o positional vector), and positional basis m l,t (w/o positional basis). Then, for each variant, we average the outputs of all layers and decompose new positional vectors based on Eq. 4. Finally, we compute the average cosine similarity between the original and new positional vectors for those subsequent tokens (>256) and also report PPL on samples in RedPajama. From Table 2 , we can see that removing the positional vector and basis of 1∼4 tokens largely affect the positional vectors at later positions (low similarity). Conversely, removing the semantic vector or altering secondary tokens has slight effects on both similarity and PPL. From these findings, we conclude that the positional vectors of initial tokens seem to serve as the role of anchors, largely contributing to the formation of positional information in subsequent tokens. Unlike full attention, LLMs employing window attention restrict each token to attend only to tokens within a window size. Previous work has shown that the maximum theoretical receptive field (TRF) in window attention is equal to the product of the window size W and the layer index l [18] .To analyze how positional vectors change across layers, we compute the number of distinct positional vectors within the maximum TRF. Notably, those tokens beyond the maximum TRF share the same positional vectors due to translation invariance [18] . Specifically, we first randomly select a positional vector outside the maximum TRF and then compute the cosine similarity between positional vectors within the maximum TRF and the selected vector. We consider the positional vector with a similarity score lower than a threshold (i.e., 0.99) as distinct. Figure 2 presents the number of distinct positional vectors and TRF at each layer. We can see that after the first layer, only initial tokens show distinct positional vectors, further verifying the findings in Section 3.2.1. As the layer increases, more tokens display different positional information and the number of distinct positional vectors increases by 512 (a window size W ) with each additional layer. The reason is that due to the constraint of window attention, each token at the preceding layer can only influence tokens within the window size at the next layer. As being distinct indicates the formation of position information, similar positional information flow from initial tokens to subsequent tokens also occurs for window attention, but gradually propagating across both windows and layers.After discussing the formation of positional vectors, we explore their impact on the attention module, mainly focusing on the attention scores. We first extract queries and keys from each head in all layers, and then compute the average attention scores in the following four settings, including (1) original: the original model, (2) w/o semantic vector: removing the semantic vectors of keys and queries, (3) w/o positional vector: removing the positional vectors of keys and queries, (4) w/o positional basis: removing the positional basis of keys and queries. Figure 3 presents the logarithmic attention scores for the first 100 tokens in the first head and fifth layer (similar results in many other heads and layers). Effect of Positional Vectors on Attention Sinks Previous work has found that the initial tokens will be assigned high attention scores, called "attention sinks" [15] , which can be clearly observed in Figure 3 . However, once the positional vector or positional basis is removed from the keys and queries, the attention scores between initial tokens and other tokens drop significantly for TL-NoPE and TL-RoPE. This finding suggests that the presence of attention sinks is likely attributed to the inherent positional information in the positional vectors of initial tokens.Effect of Positional Vectors on Long-term Decay For long texts, the attention scores of LLMs often exhibit a long-term decay pattern, which means that the score decreases as the relative distance between tokens increases [7, 6] . However, as shown in Figure 3 , when removing the positional vector or positional basis, TL-NoPE fails to exhibit long-term decay. Even with explicit relative positional encoding, the distribution of attention scores in TL-RoPE tends to be smooth after removing decomposed positional vectors. Therefore, positional vectors also play a crucial role in the longterm decay property of attention scores.Typically, when dealing with texts that exceed the context window, there are two lines of research, i.e., direct extrapolation and context window extension. In this section, we aim to investigate the change of positional vectors in these two methods for revealing their effectiveness.Relationship Between Positional Vectors and Length Extrapolation Ability To examine the impact of positional vectors in direct extrapolation, we reuse the trained model variants in Table 1 to perform inference on samples consisting of 8192 tokens. Further, we analyze the change in PPL score and the maximum cosine similarity between positional vectors within and beyond the context window. As shown in Figure 4 Left, only TL-Window-RoPE and TL-Window-80 demonstrate the length extrapolation ability, maintaining stable PPL across longer texts. These models can preserve the consistency of positional vectors both within and beyond the context window (high similarity in Figure 4 Right). Conversely, the rest models, including those with extrapolated positional encodings or window attention (e.g., TL-ALiBi), struggle to generalize to longer contexts. Notably, these models exhibit rapid changes in positional vectors (beyond 2048), diverging from the distributions observed within the context window. Thus, our findings underscore the critical role of the stability of positional vectors in enhancing the capability for length extrapolation.Effect of OOD Positional Vectors Beyond the context window, position vectors are not encountered during training and are out-of-distribution from those vectors within the context window. To explore whether OOD positional vector is a key factor in performance degradation, we select TL-NoPE for evaluation, which does not use explicit positional encodings. First, we compare the attention distribution within and beyond the context window. Figure 5 shows the attention map and scores between initial and rest tokens by averaging all heads of the 5-th layer (similar results in other layers).Once exceeding the context window (T = 2048), the attention distribution in these positions changes sharply, losing the characteristics of attention sinks and long-term decay. Since these properties highly depend on the positional vectors within the context window, we speculate that OOD positional vectors disrupt the original attention distribution. Besides, we feed the positional vectors of the last layer into the linear projection of the softmax layer to get the logits at different positions. Figure 5 (Right) presents that the logits within the context window are tightly similar while others show different distributions. Thus, the OOD positional vectors can damage the token prediction probability distribution, thereby leading to performance degradation. [11] for TL-RoPE and attention scaling (q i k j multiplied by a scaling factor λ) [20] for TL-NoPE. From Figure 6 , we can see that after context window extension, positional vectors have undergone interpolation compared to the original ones. Comparing the Factor and Ratio metrics in Table 3 , we conclude that the effective interpolation ratio is close to the expansion factor (e.g., 2 vs 2.56). Besides, as the expansion factor increases, there is a decrease in Similarity and an increase in PPL. Therefore, we suspect that imperfect interpolation may be a major reason for the decline in model performance. Since the initial tokens serve as the anchor for the formation of subsequent positional vectors, we evaluate whether changing the information flow from the initial tokens to the rest tokens can achieve the interpolation effect. To avoid the effect of OOD positional encodings, we follow the attention scaling method on TL-NoPE but only scale the attention logits between the initial tokens and others, denoted as Initial Scaling. As shown in Table 3 , it can achieve comparable performance and interpolation ratios closer than scaling all attention logits in Attention Scaling (e.g., 2.38 vs 2.56), further underscoring that the interpolation of positional vectors is mainly achieved by adjusting the information flow of anchor tokens. 
extending context window via positional vectors: 
 Inspired by our analysis of the formation of positional vectors and the interpolation of positional vectors when extending the context window, we propose two training-free context window extension methods, i.e., positional vector replacement and attention window extension. The pseudocode of these methods can presented in Appendix E.In Section 3.2.3, we show that when exceeding the context window, the OOD positional vectors tend to cause the collapse of attention distribution. Further, we observe that context window extension methods can achieve length interpolation of positional vectors and the effective interpolation ratio is close to the expansion factor of the context window. Thus, we propose to replace all the implicitly learned positional vectors with the interpolated ones, called positional vector replacement, to avoid the OOD issue in LLMs without positional encodings (NoPE).Specifically, we linearly interpolate the positional vectors within the context window with an interpolation ratio r and multiply the interpolated ones with a times α (≥ 1). In practice, we find that properly increasing the interpolation ratio r and times α can achieve better effectiveness of interpolation (details are discussed in Appendix D). Owing to the critical role of initial tokes, the positional vectors of the first four tokens remain unchanged, while those of subsequent tokens are replaced with the interpolated vectors. The replaced output ĥl,t for each layer can be formulated as:EQUATION) {p l,5 , . . . , pl,r(C-4)+5 } = Interpolation({p l,5 , . . . , p l,C }),where C, l, and s represent the original context window size, replaced layer, and interpolation ratio. Since replacing positional vectors for all layers requires heavy recalculation efforts and the positional information is passed across layers, we only apply the replacement strategy to a single early layer. We find that the 4-th layer is the optimal layer for replacement in TL-NoPE, as shown in Figure 8 .As discussed in Section 3.2.2, the positional vectors are shaped across layers and windows by the distinct positional information of initial tokens. Inspired by these observations, we propose attention window extension, the first training-free length interpolation method for window attention-based LLMs without positional encodings. The core idea is to extend the attention window size to control the formation of positional vectors. When scaling the context window by a ratio, the window size also needs to be extended by the same interpolation ratio r. However, for positions in the extended first window {W + 1, . . . , rW }, their position vectors are OOD. To avoid this, we follow the attention scaling method [20] and scale the attention logits with a scaling factor λ, achieving better interpolation of positional vectors. We define the attention score a ij between query q i ∈ R D H and key k j ∈ R D H for any heads and layers as:EQUATION)To assess the effectiveness of our proposed methods, we evaluate language modeling performance on the test set of PG-19 [22] . In line with previous work [6] , we measure PPL across various input lengths (from 2K to 8K) using a sliding window approach. We apply positional vector replacement to TL-NoPE and attention window extension to TL-Window. All the hyper-parameters are selected according to the PPL and the change of positional vectors across layers. For compared baselines, we select Dynamic-NTK [11] for TL-RoPE and Attention Scaling [20] for TL-NoPE.The results are shown in Table 4 . First, without interpolation, the PPL increases extremely after beyond the context window (e.g., > 10 3 ). When using the positional vector replacement or attention window extension methods, we observe that PPL decreases substantially, showing the effectiveness of our proposed methods. Compared to attention scaling, our attention window extension method successfully extends the context window to 8K tokens with lower PPL. Moreover, our positional vector replacement method achieves similar performance to attention scaling within 6K tokens but shows increased PPL at 8K. We attribute this phenomenon to the decreasing effective interpolation ratio across layers, as shown in Figure 9 . Additionally, an increase in PPL with the rising interpolation ratio r is also observed in both our methods, likely due to imperfect interpolation of positional vectors. 
related work: 
 Position Information in Transformers Positional information was crucial in Transformer-based LLMs, to enhance the sequence modeling abilities. The vanilla Transformer introduced absolute positional encodings, using a unique embedding to each position and adding it to the corresponding input embedding [4] . In contrast, relative positional encodings introduced biases based on the relative distance between tokens within attention modules [25-27, 6, 7] . Besides explicit positional encodings, some work investigated the implicit positional information within hidden states of Transformers. Even without positional encodings, positional information was found in hidden states of Transformer decoders [19, 28, 29] . Besides, prior work decoupled positional basis from hidden states in Transformers and analyzed geometric properties [21] . Our work mainly explores positional information embedded in the hidden states of LLMs, examining the formation and impact of positional vectors, and using it to analyze the mechanism of context window for LLMs.Extending Context Window LLMs were often constrained by pre-defined context windows. When processing inputs that exceed these windows, models typically encountered OOD issues, leading to significant performance degradation. To meet the growing demands of long context tasks [30, 31] , various methods were proposed to address this limitation and model longer texts, which can be roughly categorized into length extrapolation and context window extension [32] .Length extrapolation techniques aimed to maintain stable PPL regardless of text length by designing specialized positional encodings or window attention mechanisms [6, 17, 18, 15, 14] . Conversely, context window extension methods focused on extending the context window of existing models by adapting positional encodings or temperature hyper-parameters, thereby enlarging the context window with minimal performance loss [13, 12, 16, 20, 11, 10] . This paper bridges the concepts of length extrapolation and context window extension through the lens of positional vectors, enhancing the interpretability of context windows in LLMs.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We only provide the Pytorch implementation of our methods in the Appendix.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this work, we explored the inner working mechanism of LLMs within and beyond the context window via decomposed positional vectors. We found that the initial tokens initially present different positional information and serve as anchors for shaping the positional vectors of subsequent tokens. Besides, after exceeding the context window, length extrapolation methods maintain the stability of positional vectors, while context window extension methods achieve the interpolation of positional vectors. Based on our observations, we proposed two methods: positional vector replacement and attention window extension, which achieve training-free context window extension for specific LLMs. We believe that positional vectors will serve as an effective tool for analyzing the context window of LLMs and promote the design of better algorithms for extending the context windows of LLMs.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the training details in the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitation: 
 Our work provides an extensive discussion and analysis of the context window through the lens of positional vectors. However, our study is mainly constrained by the use of small-scale LLMs that we trained ourselves, due to the unavailability of existing LLMs with the specific positional encodings and attention patterns required for our experiments. Though some mainstream LLMs are evaluated, these models are all based on RoPE. Furthermore, we have demonstrated the effectiveness of our proposed methods solely on our own models, again limited by the absence of suitable external models. In future work, we aim to seek a broader range of models to validate our findings more comprehensively.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not provide this.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite existing papers and url. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [No] Justification: The models trained by us can not be launched. However, we give the details of training these models. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We continue pre-training all our models from the TinyLlama3 [23] checkpoint on the RedPajama [24] dataset. All models undergo the same training process, with differences only in their positional encoding and attention patterns. Each model is trained on 16 A800 GPUs over two days. Detailed training parameters are provided in Table 5 . In addition, all the subsequent experiments are computed in 8 A800 GPUs.Though previous work [29, 28] have proven that implicit positional information can be encoded in hidden states after one attention module, they only set the attention logits are equal regardless of queries and keys, which does not hold in actual Transformers. In this section, we demonstrate the preference in attention scores promotes the formation of different positional information in the initial tokens.For the s-th sample in the corpus, we denote v s 1,i , ĥs 1,i as the value and output of the first attention head and the first layer at position i, and a s i,j as the attention score between position i and position j. We also denote p1,i as the positional vector of the attention output, and u v as the mean vector of values. The positional vector can be represented as the formula:EQUATIONFor the first token, the output is equal to the value of itself, so the positional vector is equal to the mean of values. In addition, When a s i,j = 1/i, positional information of all positions is equal as follows:EQUATIONHowever, due to the preferences in attention scores, values that can be decomposed into more vector p1,iu v will be assigned larger weights, making the positional information of the following tokens differ from the beginning token. In summary, the preferences of attention scores force the formation of different positional vectors of the following tokens with the initial ones.Following previous work [28] , we only focus on a single attention head in the first layer with specific weights. In our parameterization, we only consider the first two dimensions. We demonstrate the capacity of causal Transformers to learn this capacity with either NoPE or RoPE.In our settings, the Transformer has H attention heads in each layer and the word embedding matrix is W E ∈ R D×V , where D is the dimension of the hidden states and V is the number of vocabulary. We set that the first dimension in word embedding conforms to normal distribution N (0, 1), (i.e., e 1,t ∼ N (0, 1), ∀t ∈ {1, . . . , V }), while the second dimension is 1. Other dimensions are arbitrary values. Thus, the word embedding matrix W E is: EQUATIONNext, we set the projection matrix of query, key, and value as W Q , W K , W V of the first layer and first head of Transformers.EQUATIONW K make sure that only the first dimension will be considered in attention scores. W Q transfers the second dimension in the input to the first dimension. For Transformers without positional vectors (NoPE), the attention logits for each key at position i is a s i,j = e 1,si . Thus, the vectors with large first dimensions will be assigned with large attention logits, which proves that the model without positional vectors can learn to preference some values regardless of the query.For Transformer with RoPE, we assume that the first two dimensions correspond to the basis of the rope with a value of 1/10000 and the maximum context window size is 2048. Thus, for the largest relative distance i -j, the rotation angle is smaller than π 2 . Thus, the attention score can be represent as a s i,j = e 1,si cos ((i -j)/10000). Thus, the attention logits will be larger than zero only if the first dimension of the key is larger than zero. In addition, for the same relative distance, keys with larger first dimensions have large attention logits. Thus, keys with positive values in the first dimension will be assigned greater attention weights. We also experimentally examine the first element of the output at different positions. With the above settings, we generate 10000 sequences with a length of 2048 from this distribution. Then, we compute the averaged first element of each hidden state after attention in the first layer and head, which represents the positional vectors at that position for both NoPE and RoPE. As shown in Figure 7 , the first element of the output increases fast at initial positions. Here, we can observe that the attention preferences make the first element of the output increase fast at the initial positions and tend to stabilize at a later position, further proving the different positional vectors of following tokens from the initial ones.In summary, Transformers with either NoPE or RoPE can learn preference in attention score. In addition, the preference in attention scores forces the different positional vectors of the subsequent tokens with the initial tokens.Given a Transformer with a context window size C. For samples with length C ′ (C ′ > C), we disentangle positional vectors {p l,1 , . . . , p l,C ′ }. Subsequently, we employ a context window extension method and re-compute the position vectors {p ′ l,1 , . . . , p ′ l,C ′ }. We first define the corresponding position indices f (p ′ l,t ) for positional vector after extension p ′ l,t ) as the indices of the most similar positional vectors before extension:EQUATION)Then, the effective interpolation ratio r ′ can be represented as the ratio between the maximum indices of position vectors after extension whose corresponding position indices are the context window size C, as shown in Equation 14.EQUATIONReplacing positional vectors for all layers requires heavy recalculation efforts, so we only select one critical layer to apply the replacement strategy. We evaluate the performance (logarithmic PPL score) of our replacement strategy at each layer in TL-NoPE, using different interpolation ratios and expansion factors, i.e.,(r = 4, α = 1) and (r = 5, α = 1.2), on samples with 8K tokens from RedPajama. As shown in Figure 8 , the PPL is the lowest when replacing the 4-th layer of TL-NoPE. Thus, we choose the 4-th layer for replacement in TL-NoPE in other experiments.We further examine the effective interpolation ratio of positional vectors with different settings. In our setting, we only replace the positional vectors in the 4-th layer of TL-NoPE with interpolated positional vectors in four settings on samples with 8K tokens from RedPajama: (1) r = 4, α = 1, (2) r = 5, α = 1, (3) r = 4, α = 1.3, (4), r = 5, α = 1.3.Figure 9 presents the effective interpolation ratio in each layer. As the layer increases, the effective interpolation ratio decreases as the layer increases. When the interpolation ratio is equal to the expansion factor of the context window, i.e., r = 4, the effective interpolation ratio is unavoidably smaller than the expansion factor, leading to degraded performance. In addition, multiplying the interpolated positional vectors with a larger times e.g., α = 1.3, alleviates the decrease of effective interpolation ratio across layers. Thus, we suggest to properly increase the interpolation ratio r and times α.For Positional Vector Replacement, we give the implementation with Pytorch code in Algorithm 1, which can be inserted after the output of the selected layer. For attention window extension, we give the implementation with Flash-Attention-2 [33] in Algorithm 2. To ensure a fair comparison of positional vectors across various attention mechanisms and positional encodings, we conducted continual pre-training using TinyLlama under consistent settings. However, TinyLlama is a relatively small language model with suboptimal performance and continual pretraining may result in positional vectors exhibiting properties distinct from those obtained through training from scratch. Therefore, we selected three mainstream LLMs: Llama-3-8B [34] , Yi-9B [35] , and Qwen1.5-7B [36] , for comparison. Additionally, we trained a new LLM, TL-NoPE-new, from scratch under the same conditions as TL-NoPE. In a similar vein, we extracted positional vectors using 32K samples from the RedPajama dataset.Through principal component analysis (PCA), we first visualize the positional vectors from the initial layer of these LLMs, as illustrated in Figure 10 . Consistent with our expectations, the initial tokens exhibit distinct positional information, while the subsequent tokens display a high degree of similarity. This observation supports the conclusion that the first-layer attention mechanism makes the initial tokens form unique positional information, as discussed in Section 3.2.1.Furthermore, we remove different components of the value vectors at different positions across all attention heads after the first layer. We then evaluate the impact of these modifications on both the positional vectors and the perplexity (PPL). As shown in Table 6 , removing the positional basis of the initial tokens significantly degrades the model's performance. Conversely, removing the components from subsequent tokens has a relatively smaller effect, highlighting the pivotal role of initial token positioning in influencing later tokens. However, we observe that larger LLMs are more attuned to semantic information and less affected by the removal of positional vectors compared to smaller LLMs. In line with the experiments conducted in Section 3.2.3, we extract various components from the keys and queries to assess their impact on attention scores. The attention maps for the first 50 tokens are illustrated in Figure 11 . When both the positional vectors and positional basis are removed, attention sinks disappear, and long-term decay is observed, which is consistent with the behavior seen in TinyLlama. We investigate the behavior of positional vectors under direct extrapolation scenarios. Specifically, we evaluate LLMs on sequences twice the length of their maximum context window, comparing the variations in PPL and positional vectors. The results, as shown in Table 7 , indicate a sharp increase in PPL once the context window is exceeded. Additionally, the similarity between positional vectors within and beyond the context window decreases, highlighting the relationship between length extrapolation and the stability of positional representations.Furthermore, we assess the impact of OOD positional vectors. We analyze the attention scores of initial tokens and the positional vectors of output logits within and beyond the context window.As demonstrated in Table 8 , exceeding the context window results in the loss of the attention sinking property, and the positional vectors of the logits exhibit a different distribution compared to those within the context window. This suggests that OOD positional vectors influence the model's distribution. Moreover, OOD positional encodings play a crucial role in large language models (LLMs) that leverage RoPE [15] .To assess whether our positional vector replacement method is specific to TL-NoPE, we applied it to TL-NoPE-new, which was trained from scratch. We evaluated its language modeling performance on the PG-19 dataset, using the same experimental setup as TL-NoPE. As shown in Table 9 , our approach effectively extends the context window to 8K tokens and achieves performance comparable to attention scaling. We visualize all models listed in Table 1 with PCA. The subsequent figures present the visualization of TL-NoPE, TL-RoPE, TL-ALiBi, TL-Window, TL-Window-RoPE, and TL-Window-80. NeurIPS Paper Checklist , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T 


Decision: This paper presents a novel approach to understanding how positional information is encoded and propagated within transformer-based language models. With a mean-based decomposition, they decompose hidden states into semantic and positional vectors, the authors provide detailed insights into the role and behavior of positional embeddings. Also, the work proposes methods for extending the context window based on these findings, aiming to improve model performance on longer sequences.

Pros:

The reviewers mostly recognize the paper's in-depth analysis of how positional information is encoded and propagated within transformer layers. This is a novel contribution that can benefit both research and industry aspects of LLMs.

The paper is well-written and organized, with the main findings and contributions clearly highlighted. The figures are praised for their clarity and the insights they provide into the positional information.

Cons:

Generalization Across Models: Both Reviewer R24u and yDA4 highlighted concerns regarding the generalizability of the findings. The experiments were conducted on TinyLLaMA, which is a less powerful model, raising doubts about whether the results would hold for more mainstream models.

Reviews: review: paper_summary: paper_summary: This paper dives into the inner workings of how transformer-based language models handle positional information. By decomposing hidden states into semantic and positional vectors, the authors give a series of analysis about how the positional information are encoded and propagated through layers. I believe this work offers valuable insights for understanding the positional information within the transformer architecture.
strength_weakness: strength_weakness: There are few points that I would like to suggest here to make the paper even stronger.

- Section 4 feels weak and unnecessary. The performance of replacing the positional vector, if my understanding is correct, seems to be much worse than Dynamic NTK. Given the current mainstream approach is modifying the base of Rope (like YaRN), which is much easier than the approach proposed by this work, I do not think this work’s proposed context extension will be accepted by mainstream model builder.
- That being said, I think the in-depth analysis of the positional embeddings are strong enough for me to give an acceptance (I learned a lot from it), so **I would strongly suggest removing the content of section 4, and use its space for more experimental analysis of the positional vectors**

There are a few important problems that I believe will receive the communities’ attention and worth being addressed:

- Although this paper shows the positional information can propagate through layers (Figure 2), in practice, many work found that models with window attention cannot pass the needle in a haystack test, and this is why Mistral 1.5 changed its attention back to full attention. It would be insightful if the authors can discuss the relationships between positional information and needle-in-a-haystack performance (because needle in haystack is what makes long-context models useful), i.e., why window attention cannot pass needle in haystack even it does have the correct positional information?
- This paper’s analysis is restricted on TinyLLaMA, but TinyLLaMA is not a widely used open-source model, thus casting the doubt whether this discovery of this paper will hold for other model families, particularly mainstream open-weight models like LLaMA 3, Mistral, QWen or Yi. I would strongly suggest the authors verify the behavior of positional embedding on either LLaMA 3, Mistral, QWen, or Yi.

Currently I’m given a borderline accept, and I will consider increasing my scores if the authors could either (1) discuss the relationship between positional vectors v.s. needle-in-a-haystack or (2) verify that the properties of positional vectors hold for LLaMA 3, Mistral, QWen or Yi (any 2 out of the 4).
questions: questions: see the above weakness section
limitations: limitations: see the above weakness section
paper_summary: paper_summary: This paper proposes a mean-based decomposition technique to analyze the formation and effect of positional encodings in LLMs. It then uses these results to propose methods to extend the context window, resulting in models that generalize better to longer texts.
strength_weakness: strength_weakness: I believe this contribution is novel and insightful enough, and there is no apparent weakness.
questions: questions: 1. The legends and graphs in Figure 4 overlap.
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: This paper disentangles positional vectors from the hidden states of a pretrained Transformer language model to facilitate the understanding of length extrapolation. After a series of analyses, this paper proposes two context extending techniques. Experiments show that the proposed methods lower the perplexity on the task of language modeling.
strength_weakness: strength_weakness: The experiments presented in the current draft are not convincing enough to me. See questions below.
questions: questions: 1. Instead of continue training from the tinyllama model, I think training models from scratch using the 50B tokens budget will make the results in this paper more convincing. This is because you can get rid of the ripple effect of the originally used rope positional embeddings. Maybe your models were trying to unlearn rope during the continue training stage?
2. Apart from testing perplexity scores on the task of language modeling, I highly recommend the authors adding the experiment of needle in a haystack, otherwise I do not know if the models are really using all the tokens.
3. How do you decide the values of alpha and lambda in section 4.1 and 4.2? In addition, the temperature scaling technique was also used in several other places [1, 2] with explanations of how they did temperature selection.

[1] YaRN: Efficient Context Window Extension of Large Language Models
[2] Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
limitations: limitations: See above.

Comments: title: Thank you for the additional experiments
Experiments with TL-NoPE.
Thank you. I have one more question.
Our methods can keep the long context utilization capactity of LLMs to some extent.
Thank you for the rebuttal
comment: Thanks for the response!
Thank you for the additional results. It would be great if you can add them to the final revision.

In addition, please also add the missing references [1, 2] to the final revision so that the readers are aware of the similar temperature scaling technique explored in the context of other positional embeddings.

I increased the score to 6.
Thank you for your response! We have evaluated the continually trained TL-NoPE and observed similar phenomena, with the hyperparameters listed in the second column of the table below. As demonstrated in the table, TL-NoPE with our method can utilize the long context to some extent to repeat the first token. Moreover, longer substrings (200 tokens) enhance the model's performance in in-context learning (ICL) with long contexts, which underscores the effectiveness of our methods in leveraging long contexts. Finally, further expanding the interpolation ratio and times can yield better performance on longer inputs, albeit with a slight degradation in performance on shorter inputs. 

| Total Length(L*N) | -                                             |      | 1K   |      |      | 2K   |      |      | 4K   |      |      | 8K   |      |
|-------------------|-----------------------------------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| Text Length(L)    | -                                             |   20 |   50 |  200 |   20 |   50 |  200 |   20 |   50 |  200 |   20 |   50 |  200 |
|      TL-NoPE      |                    original                   |   1  | 0.99 | 0.94 |   1  | 0.99 | 0.92 | 0.11 | 0.05 | 0.06 | 0.06 | 0.06 | 0.06 |
|                   | positional vector replacement(r=2, alpha=1.1) | 0.97 | 0.97 | 0.91 | 0.95 | 0.95 | 0.93 | 0.67 | 0.74 | 0.68 |   -  |   -  |   -  |
|                   | positional vector replacement(r=5, alpha=1.3) | 0.65 | 0.85 | 0.86 | 0.56 |  0.8 | 0.87 |  0.5 | 0.72 | 0.84 | 0.14 | 0.28 | 0.68 |
|                   | positional vector replacement(r=6, alpha=1.4) | 0.61 | 0.85 | 0.83 | 0.46 | 0.68 | 0.86 | 0.43 | 0.59 | 0.77 | 0.23 | 0.43 | 0.73 |
Thank you for the additional experiments. One question: Could you also report the numbers of TL_NoPE with your proposed positional vector replacement method?
Thank you for your response and valuable suggestion!

To assess the long-context utilization capability, we have adopted your recommendation and evaluated the LLMs using ICL. Specifically, we randomly sampled consecutive substrings from the RedPajama dataset, with lengths of 20, 50, and 200 tokens, respectively. These substrings were then repeated to achieve varying lengths, and we evaluated the accuracy of the models' ability to repeat the first tokens of substrings correctly. The results are presented in the following table.
| Total Length(L*N) | -  |      | 1K   |      |      | 2K   |      |      | 4K   |      |      | 8K   |      |
|-------|----|------|------|------|------|------|------|------|------|------|------|------|------|
| Text Length(L)    | -                                               |   20 |   50 |  200 |   20 |   50 |  200 |   20 |   50 |  200 |   20 |   50 |  200 |
| TL-NoPE-Window    | original                                        | 0.98 | 0.98 | 0.92 | 0.95 | 0.97 | 0.93 | 0.08 | 0.07 | 0.05 | 0.08 | 0.07 | 0.05 |
|                   | attention window extension(r=2, lambda=1.1)     | 0.99 | 0.96 |  0.9 | 0.89 | 0.89 | 0.92 | 0.66 | 0.89 | 0.92 | 0.06 | 0.07 | 0.06 |
|                   | attention window extension(r=4, lambda=1.2)     | 0.88 | 0.74 |  0.9 | 0.75 |  0.7 |  0.9 | 0.48 |  0.7 | 0.83 | 0.44 | 0.59 | 0.74 |
| TL-NoPE-new           | original                                        |    1 |  0.9 | 0.86 |    1 |  0.9 | 0.86 | 0.11 | 0.49 | 0.13 | 0.06 | 0.07 | 0.01 |
|                   | positional vector replacement(r=2.5, alpha=0.8) | 0.51 | 0.66 | 0.74 | 0.46 | 0.68 | 0.73 | 0.69 | 0.82 | 0.74 | -    | -    | -    |
|                   | positional vector replacement(r=5, alpha=0.8    |  0.3 | 0.26 | 0.23 | 0.25 | 0.21 | 0.28 | 0.24 | 0.22 | 0.29 | 0.21 | 0.49 | 0.64 |

After exceeding the context window, the performance of models deteriorates significantly, whereas our methods maintain a degree of capability to utilize longer contexts effectively. Specifically, for TL-Window-NoPE, as the substring length increases, the model exhibits slower degradation of performances. For TL-NoPE, with much longer substrings (for example, 200 tokens), the model even demonstrates superior performance at 8K tokens compared to 2K tokens. These enhancements underscore that our models can leverage longer contexts for enhanced performance.
Q1 and Q3: Thank you for addressing the concerns. We are good now.

Q2: I believe there should be other methods to test whether the model is truly utilizing long-context information. I would assume TinyLLama still has a decent in-context learning ability given its size. If so, one experiment I can suggest is:

Take a segment (S) of natural text with length L.
Repeat it N times and concatenate them to form an artificial sequence (A) with a length of L*N.
Feed A into your length-extended model and observe the output.
If the model learns to use long-context information, it should output the first token of the original short sequence (S[0]). Note that S[0] is not a natural continuation of A; the model will only output S[0] if it leverages the long artificial in-context examples provided. You can then experiment with different values of L and N to demonstrate the model's ability to process long-context information.

The experiment above is just one example. Feel free to devise other methods that can effectively demonstrate the long-context processing capabilities of your model.
Hi Reviewers,

Can I please know whether our response addresses your questions? We are curious if there are any particular limitations or weaknesses that prevent you from raising your score. If so, we'd like to have the opportunity to address those during this discussion period.