Why Does the Effective Context Length of LLMs Fall Short?
Authors: Chenxin An
Jun Zhang
Ming Zhong
Lei Li
Shansan Gong
Yao Luo
Jingjing Xu
Lingpeng Kong
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Long-Context Modeling
Position Embedding
Primary area: foundation or frontier models, including LLMs

Abstract: Advancements in distributed training and efficient attention mechanisms have significantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of relative positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information.
To address this challenge, we introduce Shifted Rotray Position Embedding (STRING).  STRING  shifts well-trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. 
Experimental results show that without additional training, STRING dramatically improves the performance of the latest large-scale models, such as Llama3.1 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, establishing new state-of-the-art results for open-source LLMs. Compared to commercial models, Llama 3.1 70B with STRING  even achieves better performance than GPT-4-128K and clearly surpasses Claude 2 and Kimi-chat.

Published as a conference paper at ICLR 2025
WHY DOES THE EFFECTIVE CONTEXT LENGTH OF LLMS FALL SHORT?
Chenxin An' Jun Zhang?
Ming Zhong Lingpeng Kong'
Lei Li'
Shansan Gong'
Yao Luo² Jingjing Xu²
The University of Hong Kong "ByteDance Inc.
University of Illinois Urbana- Chhapa-Chammpain
ABSTRACT
Advancements in distributed training and efficient attention mechanisms have sig- nificantly expanded the context window sizes of large language models (LLMs). However, recent work reveals that the effective context lengths of open-source LLMs often fall short, typically not exceeding half of their training lengths. In this work, we attribute this limitation to the left-skewed frequency distribution of rela- tive positions formed in LLMs pretraining and post-training stages, which impedes their ability to effectively gather distant information. To address this challenge, we introduce ShifTed Rotray position embeddING (STRING). STRING shifts well- trained positions to overwrite the original ineffective positions during inference, enhancing performance within their existing training lengths. Experimental results show that without additional training, STRING dramatically improves the perfor- mance of the latest large-scale models, such as Llama3. 70B and Qwen2 72B, by over 10 points on popular long-context benchmarks RULER and InfiniteBench, es- tablishing new state-of the-art results for open-source LLMs. Compared to commer- cial models, Llama 3.1 70B with STRING even achieves better performance than GP SPT-4-11 12 8K and clearly surpasses Claude 2 and Kimi-chat. All code and data used in this work are released at https: //te:///gtttttt com/ HKUNLP ITRIING
INTRODUCTION
The increase in context length for large language models (LLMs; OpenAl 2023; Anthropic 2023; Bai al. 2023; Xiong et al. 2023; Llama Team 2024) has facilitated the development of wide range of applications (Pang et al., 2022; Bairi et al., 2023), substantially expanding the capabilities of AI systems. Recent advancements in efficient training and attention calculation (Li et al., 2024a; Dao, 2023; Liu et al., 2023) have made it feasible to train LLMs with exceptionally long context windows. For instance, Llama3. 1 (Llama Team, 2024) features a context length of 128K tokens, which is 64x longer than that of its initial release (Touvron et al., 2023a).
This trend towards longer context lengths in LLMs promises enhanced capabilities. Previous work has primarily focused on extending the context length of LLMs, with significant efforts devoted to improving data engineering techniques (Fu et al., 2024b; Hu et al., 2024; Bai et al., 2024; Zhao et al., 2024). High-quality natural long-context data are scarce in real-world settings, limiting the availability of such data for training purposes. To address this challenge, recent methods aim to generate synthetic training data that better capture the nuances of naturally occurring long-con long-context information, despite inherent challenges such as time consumption in continual training and potential biases (Zhao et al., 2024; An al., 2024b; Lv et al., 2024). Researchers have also focused addressing specific architectural limitations. Efforts have been made to correct the improper adjustment of the base frequency in Rotary Position Embedding (RoPE) (Su al., 2022; Peng et al., 2023; Chen et al., 2023; Lin et al., 2024b; Chen et al., 2024).
However, recent studies (An al., 2023; Zhang et al., 2024d: Li al., 2024b; Wang et al., 2024a) reveal a notable discrepancy between these theoretical improvements and observed performance. In practice, the effective context utilization of these models often falls substantially below their claimed or training context lengths. For example, on the widely used RULER benchmark (Hsieh et al., 2024),
Work done during internship ByteDance Inc.

---

Published as a conference paper at ICLR 2025
the effective context length of the latest Llama 3.1 70B model is only 64K, despite employing scaled RoPE base firquency (Peng et al., 2023) and having sufficient training data (Llama Team, 2024). In fact, most open-souree models demonstrate effective context length less than 50% of their training length (Hsieh et al., 2024). A key research question emerges from these observations: Why does the effective context length of L.LMs fall short of their training context lengths?
In this study, instead of further extending the context window size of current LLMs, we take fresh perspective understand and address this gap. Our core insight revolves around a phenomenon we term the left-skewed position frequency distribution a pattern of severe undertraining of long- distance position indices during pretraining and post-training stages. This skewed distribution significantly contributes to the model's 's suboptimal performance in long-range modeling tasks. In SlimPajama-6271 (Cerebras, 2023), a widely used pretraining corpus (Geng & Liu, 2023; Zhang et al., 2024b), we clearly observe this left-skewed phenomenon. As illustrated in Figure 1a, even with presumably adequate unn-sequence data, the frequeney of position indices decreases dramatically as distances increase. For instance, when training model with 2048 context length on SlimPajama, the frequency of position indices used to model relationships between distant tokens (distances 2 1024) is less than 20%, and for even longer distances (2 1536), it drops below 5%. Probing experiments conducted during pretraining reveal that the frequency of exposure to specific position indices has a crucial impact on the training context utilization. Capturing long-range dependencies is inherently more challenging (Zhu et al., 2023; Wu et al., 2024), and this challenge is exacerbated when the frequency of position indices allocated gather distant information exceedingly low, as observed in Figure 1. In other words, the difficulty in modeling long-term dependencies, coupled with the undertraining of the positions responsible for them, provides a compelling explanation for the discrepancy between the theoretical and practical context lengths in LLMs.
Building on these findings, we investigate whether well-trained positions can be leveraged to capture information from distant inputs during inference. To address this, we propose a training- free approach called ShifTed Rotray position embeddING (STRING). This method eschews the use of positions at the tail of the frequency distribution during inference. Specifically, STRING shifts position indices from the main diagonal of the position matrix toward its bottom-left corner. This adjustment enables the model to represent Long-range dependencies using frrequentl encountered position indices, effectively approximating the undertrained ones. STRING can be efficiently implemented using Flash Attention (Dao, 2023) by combining two key components: (1) sliding window attention (Beltagy et al., 2020; Ding et al., 2023; Xiao et al., 2023; 2024) around the diagonal, and (2) self-attention at the bottom-left corner using shifted position indices (Algorithm 1). This implementation incurs no additional computational costs and causes no obvious slowdowns during inference.
By strategically overwriting position indices in the upper range of the training length, we achieve substantial performance enhancements across seven open-source LLMs with context lengths ranging from 2K 128K on the Needle-in-a-Haystade (4-needle) test, resulting in an average score increase of 18 points. STRING requires no additional training, enabling seamless scaling up with powerful large-scale models such as Llama3.1 70B (Llama Team, 2024) and Qwen2 72B (Bai et al., 2023). This integration not only establishes new state-of-the-an performance for open-source LLMs on long-context benchmarks RULER (Hsieh et al., 2024) and InfiniteBench (Zhang et al., 2024d) but also enables Llama3.1 to outperform leading commercial models, including GPT-4-1 28K (OpenAI, 2023), Claude-2 (Anthropic, 2023), and Kimi-chat (Moonshot AI, 2023), across a wide range of synthetic and practical tasks. The substantial improvements achieved by STRING provide strong evidence for our hypothesis: underrepresented position indices at the tail of the position frequency distribution, strongly constrain the long-context capabilities of current LLMs. We hope our findings will inspire new approaches to overcome these limitations and lead to more effective long-context processing in future LLM designs.
2
LEFT-SKEWED POSITION FREQUENCY DISTRIBUTION
2.1 POSITION EMBEDDINGS IN LLMS
Self-attention mechanisms (Vaswani et al., 2017: Radford et al., 2018; Dai et al., 2019) inherently lack positional information (Liu et al., 2021; Su et al., 2022; Sun et al., 2022). To introduce positional information, a common approach is to design a function p. For an input at position i, we inject positional information using the following method: h, = p(h, i) where h is the hidden representation
2

---

Published as a conference paper at ICLR 2025
wngh
LSCM
Megt
102V
Peatten Theqgemo
237
(a) Natural data distribution
(b) Uniform data distribution (c) Concatcnated data distribution
Figure 1: Position frequency distribution exhibits a pronounced left-skewed pattern across training data of varying lengths. Figure Ia illustrates the natural data length distribution of SlimPajama-6277 where oversized data is truncated into multiple 2K sequences. Figure 1b presents the case with uniform length distribution and the position frequency decline quadratically. Figure Ic demonstrates that when all data are concatenated into 2K sequence, the position frequency decreases linearly with increasing position indices. The X-axis represents data length (shown in orange) and position indices (shown blue). The left Y-axis indicates the frequency of each position, while the right Y-axis represents the number of data for each length.
of the input token. Another approach involves relative positional encodings (Bao et al., 2020), such as T5-bias (Raffel et al., 2023) and ALiBi (Press et al., 2022), which injects relative positional information by incorporating the relative distance (i (i-)) when computing the attention score between the j-th token and the i-th token.
To achieve better training stability and lower perplexity, mainstream large models like Qwen (Hui et al., 2024) and Llama (Llama Team, 2024) employ Rotary Position Embedding (RoPE) (Su et al., 2022) their positional encoding method. RoPE directly injects positional information into the query and key vectors, enabling the inner product to encode the relative position information between the query and key. We adopt the notation p for the embedding function of RoPE. Considering the i-th query and the j-th key, we have: q p(q. i) and k, = j), When computing attention, the inner product q. k, contains only the relative positional information (i - j), which means for any pair (m, n) such that m n = - j. it holds that q k,, = q, ky.
2.2 RELATIVE POSITION MATRIX AND POSITION FREQUENCY
Using relative positional encodings implies that, given training length L, the resulting relative position matrix P after computing Q K is defined by:
0
1
0
P
1.
1.
L L-2
2
1 0
L L
0/
where the Toeplitz matrix P captures the relative positional relationships between tokens, with each element P(m][r]] m encoding the relative distance between the m-th and n-th tokens in sequence. Based on Eq. 1. we define the frequency of relative position by f(i) L i, which is the number of occurrences of relative position Throughout the remainder of this paper, the term "position" refers to relative position. The structure of matrix P is linearly skewed toward smaller positions, which inherently favors performance on shorter sequences. For example, when using training context window of L = 2048 tokens, the relative position 2047 occurs only once in P.
The frequency of relative positions in also depends on the data length distribution of the pretraining corpus C. We can obtain the frequency of relative position by the following equation:
f(i)
max([s] -i, 0), 0Si<L
SEC
We observe that the position frequency distribution is usually highly left-skewed, indicating that the model is frequently exposed to small positions, while larger positions account for only small pro- portion. To illustrate this phenomenon, we examine the position distribution when using SlimPajama- 627B (Cerebras, 2023) the training corpus. The blue bars in Figure/ 1 illustrate the position
3

---

Published as a conference paper at ICLR 2025
2.5
postion 5143
Tralame-1.Bb66 Trofame-1..b-
effactive langet
1938
1.5K
1928
1.00
0.5
1024
2540
consumed tokens
effective length
(a) Effective length vs. consumed tokens
(b) Effective length vs. position frequency
Figure 2: Analyzing effective context length of LLMs pretrained on SlimPajama with respect to training length, token consumption, and position frequency. Figure 2b, we use the model effective length as the X-axis, and the Y-axis indicates the number of times the model was exposed to that specific position during training.
frequency distribution based on the natural data length distribution of SlimPajama. Specially, when the training length is 2048, the position indices i < 1024 account for more than 80% of all indices, whereas those with 2 1536 constitute less than %. In addition the biased relative position matrix P. the real-world data length distribution is also biased. Given training context length of 2048 tokens, the data length distribution is shown in Figure Ia (orange bars): about 20% of the data consists of sequences around 256-512 tokens, and approximately 20% of the samples are around 2048 tokens. This latter percentage arises because long sequences are segmented into multiple sequences of length 2048, following popular open-source pretraining projects (Geng & Liu, 2023; Zhang et al., 2024b). Due to the combined effect of the data distribution and the relative position matrix, the frequency of positions decreases following polynomial trend as the position indices increase.
Despite capturing local dependencies is often effective for LLMs, the imbalance in position frequency distribution when modeling both local and long-range dependencies is more pronounced than expected. This may result in a substantial underrepresentation long-range dependencies.
A PROBING EXPERIMENT ON POSITION FREQUENCY AND MODEL EFFECTIVE LENGTH
In this section, we empirically investigate the impact of the left-skewed position frequency distribution on the effective context length of LLMs. Since the training data distributions for most open-source LLMs are opaque and cannot be directly analyzed by researchers, this study represents the first exploration of the impact of position frequency during the pretraining stage.
Evaluation To measure the effective context length, we adopt the popular Beedle-in-a-Hassteee task (gkamradt, 2023). We the 4-needle setting, the same described in the Llama 3.1 report (Llama Team, 2024), which involves inserting four needles (6-digit numbers (Hsieh et al., 2024; Mohtashami & Jaggi, 2023)) into the context various positions. The model should perfectly retrieve at least two of them. The input examples used in this experiment can be found in Table 5 of the Appendix. The evaluation context length increases in 128-token steps until the model fails correctly find 2 of 4 inserted needles. We perform 500 tests at each length.
Experimental Setup We pretrain two 1.3B- 1B--arrmeter models (referred to as TinyLlama- 1.3B) from scratch on the natural data distribution of the SlimPajama dataset to observe changes in the model's effective length. The total training tokens are IT and we evaluate the model's 's effective context length for every 10B tokens during training. Both models begin to exhibit needle-retrieval ability after about 50B tokens of training. Since position frequency is difficult to control directly, we perform controlled experiments by adjusting two factors: (1) consumed tokens, and (2) the training context window size. The first factor is straightforward. For the second factor, we illustrate the position frequency distribution after training with IT tokens using different training lengths (2K and 4K) in Figure 3. The configuration of our pretraining codebase and models is detailed in Section A.2.
4

---

Published as a conference paper at ICLR 2025
Findings Following previous work (Kaplan et al., 2020), we demonstrate how the models' effective length grows with increasing training tokens for two different training lengths (Finding (1)), while our further analysis reveals that the position frequency is the underlying factor (Findings (2) and (3)).
(1) Larger training context window consumes fewer tokens to achieve the same effective context length: In Figure 2a, notable observation is that training with longer sequences results in a greater effective context length when the same number of tokens is consumed. Specifically, the model trained with a sequence length of 4K tokens achieves an effective context length of 1.4K after consuming 400B tokens. In contrast, the model with a 2K training length needs around IT tokens to attain the same effective context length. But it does not mean larger context training window can save more computation.
(2) Models can achieve similar effective context lengths if they have been exposed to similar fre- quencies of position indices, even if their maxi- mum training lengths differ: By directly plotting the effective context length against the frequency of position indices used to model that length (Fig- ure 2b), we observe that the growth trends of ef- fective lengths for different models align when the Y-axis represents the frequency of indices at that length. For instance, when the effective context length reaches 1,280 tokens, both models exhibit position frequency f(1280) of 100B. This indi- cates that models can attain comparable effective context lengths when they have been trained on similar frequencies of position indices, regardless of differences in their maximum training lengths.
Training Longth Trainingg Length 2K
Position Frequency
48T
32T
16T
OT
0.5%
1.0K
1.5
25k
3.5k
Figure 3: Position frequency distribution for models trained with different training lengths after consuming IT tokens. With the same num- ber of tokens, training length has little effect on small relative positions. For example, the relative position 0 appears 4K times in both single 4K sequence and two 2K sequences with the same total token count of 4K in each case.
(3) The growth trend of the model's effective length aligns with the position frequency distribution: In Figure 3, we observe that models with different training lengths have close position frequencies when the position index S 1024. As continues to increase, the frequency gap between models trained with 4K and 2K context lengths becomes increasingly larger. The growth rates of these two models' effective lengths also align with this trend (Figure 2). Both models consume roughly the same number of tokens (around 300B) when reaching an effective length of 1024. However, as the effective length increases further, the growth rate of the model pretrained with 2K context window becomes significantly slower.
Limitations in Gathering Distant Inputs We visualize the performance of infrequent positions with the Needle-in-a- Haystack (4-needle) test (gkamradt, 2023). The distance between the query and the needles increases as the depth becomes smaller and the testing context length becomes longer. The results indicate that when the needle and query far apart, both TinyLlama 1.3B and the latest Llama3.1 1 8B model fail to retrieve the needle effectively. In Figure 4, when we place the query at the end of the document, we find that models fail at retrieving information from the beginning of the document. Concretely, in Llama3. 1, performance significantly degrades when position indices exceed 90K. TinyLlama struggles to gather information when the distance exceeds 1,536 tokens. We also evaluate 13 models from the open-source community, as shown in Table 4, and find that most failure cases occur within the first & of the document. This may indicate that the last //d positions of current LLMs all fall in the tail of the position frequency distribution.
SHIFTED ROTARY POSITION EMBEDDING
In Figure Ic, we demonstrate that even when all data are concatenated to fill the training context window, positions at the tail remain infrequent. In this section, we introduce ShifTed Rotray position embeddING (STRING), STRING shifts position indices from the diagonal of P towards its bottom-left corner, allowing the model to gather distant information with frequent position indices.
5

---

Published as a conference paper at ICLR 2025
251
164
603
915
S0 ww yor of
Itaylloma
(9) Lloma.31 BB 125K
Figure 4: NIAH results for our pretrained model TinyLlama-L.3 1.3B (2K) and Llama3. (128K) where the X-axis means imput context length and the Y-axis represents the document depth. In this figure, we clearly observe that for TinyLlama 2K and Llama3. 128K, most poor-pertie ppor-performing cases are concentrated the lower-left triangle, indicating that the models are unable to gather distant needles.
2
0
0:102
Io-rr 0-3r2
1-900-992
5
1+ 2
2
4-3r
B+A 0-1F2
laftward ofhet=
(a) Dropping infrequent positions
(b) Shifting frequent positions
(c) Recovering locality
Figure 5: A illustrative example of STRING for sequence length of L 9. (a) Position indices 6, 7, and are removed from the matrix. (b) Indices 0, 1, 2, 3, 4, and are shifted from the main diagonal to the lower-left triangle with an offset of 3. (c) A small constant W is added all diagonals where m 2 π 3, thereby restoring emphasis on the neighboring W tokens. The position matrix of 1- using STRING shown in Figure 8 Appendix.
4.1 MANIPULATING THE POSITION MATRIX
STRING is implemented by manipulating the position matrix P. The three main procedure of STRING
shown in Figure 5:
(1) Dropping Infrequent Positions: We begin by assuming that position indices greater than a threshold N falls into the infrequent area. Consequently, STRING initially drops all position indices 2 N. As depicted in Figure 5a, we set N = and I, = 9, resulting in the removal position indices 6, 7, and from the matrix and leaving an empty area.
(2) Shifting Frequent Positions:
Next, we shift the remaining position indices from the main diagonal (the high-frequency area) to fill the empty triangle in the bottom-left corner of P. The shift offset is
defined as S = L - N. In our example, S 9 6 = 3, as shown in Figure 5b. For instance, let's consider the last row the matrix P. The position indices after dropping are [- To fill the 3 empty slots, we shift the positions leftwards with a stride of 3, and they become [5, 4, 3, 1, Formally, the updated position matrix is defined as:
5, 4, 3, 2, 1, O).
P[m][n] - S mm 2 n P[m][n]
P[m] [n] =
33)
otherwise.
Here, m, n is the row/column index, m = n indicates that the element located on diagonal of S away from the main diagonal, and m S signifies that the element is in the lower-left region relative to this diagonal. The resulting position matrix after this operation is shown in Figure 5b.
(3) Restoring Locality with Small Window: Applying Eq. 3 disrupts the model's ability to capture local relationships because alters the relative positions between neighboring tokens (Su, 2023; Jin al., 2024; An et al., 2024a). Specifically, the relative positions the S-th diagonal set zero. Since neighboring tokens are crucial for generating fluent content, we introduce small local window value W < for elements where > S, illustrated in Figure Sc. This adjustment maintains

---

Published as a conference paper at ICLR 2025
emphasis on the closest W neighboring tokens. The final position matrix is defined as:
P|m][n]] - S m 2 n S; [[
P[m][n] =
(4)
otherwise.
In Eq.4, S is the shift offset, and W is used to ensure the neighboring W tokens remain the closest in
terms of positional encoding. Notably, does not rely on I, whereas S heavily depends on L. We suggest setting the local window W 2 32 and the offset 5/2 S S 1.2 We set S 3 and W 128 for all models across downstream tasks. An ablation study is shown in Figure 7.
FlashA ttention Implementation We implement STRING using FlashAttention (Dao et al., 2022), which is essential for verifying the method on modern large language models (LLMs) that typically have long context windows (c.g.. 128K tokens). STRING can be efficiently implemented by modifying the position indices used in RoPE and combining two attention patterns. The pseudocode for STRING is provided in Algorithm 1. Our implementation splits the standard self-attention mechanism into two components:
Sliding Window Attention (lines 11-13): This approach calculates the attention outputs
around the main diagonal by considering positions where computing the sliding window attention, there is no need to modify the position indices for either queries (1ine 6) or keys (line 7).
- (line 13). When
2. Shifted Self-Attention (lines 15-19): This method computes the attention outputs in the bottom-left triangle, specifically for positions where 2 n - S, utilizing causal self-attention (line 19). In this process, the position indices for queries are replaced with shifted position indices (line 1).. STRING controls the relative distance by only modifying the position indices for queries and there is no influence on caching keys and values.
Finally, we merge the attention outputs from the sliding window around the main diagonal and the left-bottom triangle to produce the final output. An example of shown in Section $A.1 and the efficiency test of STRING is shown in Figure
applying STRING on Llama3.1 is
Algorithm Pseudocode of STRING with FlashAttention
1 Q K, V: sensors with shape [L, d]
2 9: the local window value (scalaz)
3 4 S: the siding window size (scalar)
left botton triangle height (scalar)
5
standard position ids for keys 7 pida_ = 10, 18,1,22... # atandard positior. ida for queries
6 pida_query 10,
8 & Apply rotary position embeddings to K
9 apply_rotary_per pce_eettey__ss_ebik. pids_key)
10
attention d_diag hoppyy_rttrr_.pe_emb/ pids_ quezy! O_diag, alta_rap_diag alash_atta K, V, sliding window-S)
Caloulating sliding
the diagonal
15 $ <--- calculating self-attention at the laft-boctom triangle ---> pida_a_shifred pida_query
position ida queries 17 a_siffred = appll_rotarry potarrypos_emblb, sids_g_ahifted
obtain v, in the bettom-lest corner calculete élash-attn O_shif:ed, attn_mep_shifted
(Q_
K[:N], V[:N])
20
diagenal and left-botton triangle 22 output - merge_diag aiaseeddieggeaiieedo_dddo 0_stilled, alta_nap_diag, altinmag_shitted)
Merge stteation outputs
Figure 6: Detailed pseudocode of STRING incorporating FlashAttention Dao et al. (2022). The implementation of me merge__d aig_ssh fted can be found in Algorithm 2 in the Appendix.
4.2 MAIN RESULTS OF STRING
In this section, we evaluate the effectiveness of STRING across three widely recognized long-context benchmarks: Needle-in-2 -neelle-n-a-HHyystcck (NIAH) (gkamradt, 2023), RULER (Hsich et al., 2024), and InfiniteBench (Zhang et al., 2024d). These tasks enable us to assess STRING'S performance across a broad spectrum of practical scenarios.
7

---

Published as a conference paper at ICLR 2025
Baselines We primarily compare STRING with the original position embedding RoPE used in mainstream Large Language Models. Additionally, we evaluate RoPE against several effective extrap- olation baselines. Specifically, compare STRING with the following training- -ree extrapolation methods: NTK-Aware RoPE LooallLLaMa 2023b;a), YaRN (Peng et al., 2023), ReRoPE (Su, 2023), Self-Extend (Jin al., 2024), and DCA (An al., 2024a). Extrapolation refers to testing L.MSs on sequence lengths beyond their training lengths while STRING focus on improving the performance within the training context size. NTK-Aware RoPE and YaRN implement extrapolation by increasing the base frequency of RoPE. Meanwhile, ReRoPE, Self-Extend, and DCA modify the position matrix to aviod unseen positions. We reproduced their results using scripts from their official repositories. When testing these extrapolation baselines, we modify the training length of the model to 3 of the original length and set the extrapolation scaling factor to Lin 3. meaning the test sequence length Linin
1.5 times the training length. All other configurations remain the same in their paper. Our findings indicate that although extrapolation methods can extend the model's 's capability to handle longer sequences, the performance improvements are still limited within the original training length.
Table Ncedle-in-a-hhystacc (4 needles) results of 7 base models across various methods (columns reordered from smallest to largest average) where Livain means the size of the training context window. All the models were tested using their training length. The number of test cases is 500.
Model
Tapain
ReRoPE
NTK
RoPPaeggo
Self-Fxtend
YaRN
DCA
STRING
TinyLiama- 1.30a-1-.333 1.3B (ours)
2k
62.8
62.0
56.6
60.2
68.6
74.4
84.6
TinyLlama-1n IB-3TT
77.2
79.8
69.8
83.2
88.0
80.2
97.2
Llama-2-7B
4k
98.6
98.6
98.0
95.4
98.0
91.6
100.0
Llama-3-8B
8k
99.6
100.0
99.8
99.8
100.0
99.9
99.6
LWM-7B-basc
32k
25.2
19.4
31.8
29.0
22.2
28.8
50.4
Mistral-7B-baat
32k
54.5
42.2
52.8
54.2
48.2
64.2
73.0
Llama-3.1-8B
128k
53.6
71.2
66.0
65.8
68.8
72.8
95.2
Average
67.3
67.6
67.8
69.6
70.5
73.1
85.7
reedle-in-a-Haystted Needle-in-a-Haysttcc (gkamradt, 2023) (NIAH) is the most popular long- context task, extensively utilized in recent studies (Zheng et al., 2024; Liu et al., 2024b). As reported by Hsich et al. (2024); Wang et al. (2024a), single needle retrieval is no longer challenging task for current L.Mss, and we adopt the multi-needle setting following Llama 3.1 (I.lama Team, 2024) and the input example can be found in Table 5. We verify the effectiveness of our method on seven community models with training lengths ranging from 2K to 128K. Across all seven models, LargeWorldModel (LWM-7B-brie --base) (Liu et al., 2024a), Mistral 7B (Mistral.Al, 2024), and Llama 3.1 8B (Llama Team, 2024) are continually trained on longer contexts. On models with various training context lengths, STRING consistently outperforms other methods, achieving the highest scores each model. Notably, STRING improves the average performance by a significant margin, reaching 85.7% compared the next best method, DCA, at 73.1 %, and the original RoPE only 67.8%.
The RULER benchmark (Hsieh et al., 2024) encompasses variety of synthetic tasks, including eight variants of Aeedle-in-a-Haysttal (NIAH), as well as tasks involving variable tracking, counting, and Long-context question answering (QA). The evaluation code and metrics are from their official repository'. The primary results are presented in Table 2. The results on Llama3. 1-8B reveal that, except for our proposed method (STRING), all other extrapolation-basee approaches fail to achieve performance improvements. Since our method does not require additional training, we are able to validate its effectiveness on 70B-level models. Applying our method to larger models yields remarkable enhancements: 15-point improvement on Llama3.1 1 70B and over 30-point improvement on Qwen2 72B compared to the baseline. Furthermore, our approach achieved state- of-the-art performance on the RULER benchmark for open-source models. Notably, after applying STRING, both Llama3.1 70B and Qwen2 72B surpass GPT 128K average performance. The remarkable performance gain large models demonstrates that the frequent positions in large models may possess a stronger potential for modeling long-range dependencies. Additionally, we also demonstrate that both Llama3.1 1 and Qwen2 can be effectively boosted to effective sequence length of 100K on RULER by STRING (the last block in Table 2).
RULER
'https://// thub. com hos weeeeehhaaccchcce RULER
8

---

Published as a conference paper at ICLR 2025
Table Performance various models and methods RULER with tested sequence length of 128K. The RULER benchmark consists of 13 tasks (500 test cases for each task) categorized into Needle-in-a-Hand Haystaak (NIAH), Variable Tracing (VT), Aggregation, and Question Answering (QA). We report the average scores for each category as well the overall average across all 13 tasks. Effective denotes the actual effective sequence length as defined RULER, indicating whether model surpasses the performance of Llama2 (Touvron et al., 2023b), and Claimed represents the
sequence length reported by the model.
Models
Effective/Claimed
NIAH
VT
Aggregation
QA
Avg. (13 tasks)
Llama2-chat
4K/4K
96.9
89.7
84.8
49.7
85.6
GPT-4-1 1774--106-preview
64K 128K
84.8
99.6
79.7
GLM4 (Open-sous) source best)
64K 1M
94.4
97.7
49.7
63.6
83.1
LWM (7B)
4K / 128K
83.4
15.2
29.1
52.6
65.0
Phi3-medium (14B)
8K / 128K
51.3
26.0
43.5
38.0
46.1
I (8B)
32K 128K
92.6
70.4
36.2
77.0
+ YaRN
32K 128K
94.7
39.8
38.2
58.8
76.3
DCA
32K 128K
89.5
62.5
39.2
55.2
74.4
+ Self-Extend
32K 128K
94.9
65.0
37.3
49.8
76.8
+ ReRoPE
32K 128K
90.0
56.3
38.7
56.9
74.4
-TTRING
32K 128K
94.0
88.1
37.6
62.7
80.0
(34B)
32K 200K
90.2
76.8
43,4
59,9
77.3
GradientAV/LLamme (70B)
16K IM
84.9
56.2
41.4
59.8
72.1
Mixtral (8x22B)
32K 64K
23.8
0.0
69.7
40.8
31.7
Command-R R-pul (104B)
32K 128K
65.7
97.2
39.2
Llama3.1 1 (70B)
64K 128K
78.9
59.2
39,8
47.6
66.6
STRING
100K 128K
92.7
95.6
50.0
63.0
81.7
Qwen2 (72B)
64K 128K
48.0
79.0
70.3
47.2
53.7
+ STRING (new SOTA)
100K / 128K
91.2
98.4
83.7
52.2
84.6
Test Length: 100K
Llama3. --STRING (70B)
100K 128K
94.6
97.8
72.1
67.3
87.2
Qwen2-STRING (72B)
100K 128K
93.9
97.7
88.1
57.8
87.8
Table 3: Comparison of STRING with three leading commercial long-contenn models on InfiniteBench. Each model is evaluated using a maximum context length of 128K.
Commercial Models
Llama3.1 8B
Llama3.1 70B
Tasks
GPT-4
Claude2
Kimi-chat
RoPBavigi
STRING
RoPEneiaa
STRING
En.Sum
14.73
14.45
17.93
26.00
28.22
26.89
27.66 64
En.QA
22.22
11.97
16.52
10.05
10.20
13.68
16.73
Er.MC
67.25
62.88
72.49
65.50
70.30
76.41
81.98
Dia
8.50
46.50
11.50
20.00
19.50
18.00
30.50
Retr. Retr.Passke PassKey
100.00
97.80
98.14
100.00
100.00
100.00
100.00
Retr.Number
100.00
98.14
94.42
99.32
99.89
100.00
100.00
Retr.KV
89.00
65.40
53.60
42.00
83,00
2,22
76.07
Code.debug
39.59
2.28
18.02
22.84
26.90
29.20
32.80
Math.find
60.00
32.29
12.57
32.18
34.87
40.92
46.28
Avg.
55.69
47.96
43.91
46.43
52.54
45.25
56.88
InfiniteBench InfiniteBench (Zhang et al., 2024d) encompasses variety of real-world tasks, including long-context question answering (QA), multiple-choice QA, mathematical problem-solving, long-dialogue QA, long-contex, summarization, retrieval tasks, and code debugging.
The evaluation code and metrics are sourced from the official repository The results for commercial models are from Zhang et al. (2024d). We compare our method, STRING, with the original position
Phtpps://gi thub. http::/tt..bbccco///////////////
teltench

---

Published as a conference paper at ICLR 2025
100
Performance
Performancce
80
Tinyl
40
TeySJama-2K
Llmma-2-K
2
LAN--78-32X
RoPR
122
256
512
RoPE
1/5
L/4
L/5
6/2
Local window W
Shifted offset
(a) Ablation local window W (S
(h) Ablation shifted offset S (W 128)
Figure 7: Ablation study the local window W and shifted offset S where L. is the training length.
embedding, RoPE, across two scales of Llama3.1: 8B and 70B parameters. The results presented in Table 3. STRING demonstrates significant improvements for both models; for instance, we enhance the performance of 1 70B by over 10 points, establishing new state-of-the-art for open- source models. On InfiniteBench, our method also surpasses the performance of strong baseline GPT-4-128K and significantly outperforms Claude-2 and Kimi-chat.
Ablation Study We conduct an ablation study on the Needle-in-a-Haasttack (4 needles) task to examine the impact of two main hyperparameters in our STRING: the local window size W and the shifted offset size S. The experimental results are shown in Figure 7. We increase the local window size from to 512 and find that when W 2 32, the model achieves significant improvement compared to the original RoPE method. Furthermore, as long as W < S, further increasing W does not cause performance drop. For the offset size S, we experiment with values ranging from L to 1/- As S increases, more position indices are discarded. We observe that within this range, the performance increased with the growth of S. However, the trend slowed down when S exceeded L 3
5
indicating that at least the last 33% 50% of the position can be overwritted.
RELATED WORK
Long-Context Scaling of LLMs Modeling long text has always been a challenging problem. With the development of large language models (LLMs), researchers have begun to explore ways to extend these models to handle longer contexts from various perspectives. (1) Efficient Architectures: Jiang al. (2024); Fu et al. (2024a); Ding et al. (2023); Song et al. (2023); Yang al. (2024); Zhu et al. (2024b) demonstrate that the training and inference overhead long-context LLMs can be substantially optimized by sparse attention patterns. Another crucial architecture is state space models (Gu & Dao, 2023; Yuan et al., 2024; Lieber et al., 2024). (2) Continual Training with Long Data: Efforts have been made to continually train models by collecting high-quality long sequences (Fu et al., 2024b; Zhu et al., 2024a; Wu et al., 2024; Gao et al., 2024). (3) LLMS with Infinite Contexts: Recent work has shown that the context length LLMs can be scaled to infinite, as evidenced by models such as StreamingLLM and InfLLM (Xiao et al., 2023; 2024; Han et al., 2023; Zhang al., 2024a; Cai et al., 2024; Lin et al., 2024a; Dong al., 2024). However, these methods typically cannot maintain full KV cache, resulting in weakened long- long-context capabilities.
Length Extrapolation Training to extend the model context length incurs significant overhead. Recent works focus on length extrapolation, training on short sequences to infer longer ones, as means address this issue (Press et al., 2022; Raffel et al., 2023; Han et al., 2024). An et al. (2024a); Jin et al. (2024); Su (2023); Ma et al. (2024); Zhang et al. (2024e) believe that the model's inability to generalize to longer contexts caused by positions being out-of-distribution. They achieved effective extrapolation by repeating trained positions, thereby maintaining low perplexity in exceedingly long contexts. On the other hand, Zhu et al. (2023) randomly places large position indices within the training window in the training and infer longer sequences. For RoPE-based LLMs, Peng al. (2023); Men et al. (2024); Zhong et al. (2024); Wang et al. (2024b) reduce the long-range attenuation effect of RoPE by amplifying the base frequency, thereby bringing the remote token closer.
10

---

Published as a conference paper at ICLR 2025
6 CONCLUSION
This work uncovers the limitations of current open-source large language models in effectively utilizing their extended training context windows. We show that using positions at the tail of the left- skewed position frequency distributions strongly hinders models' long-range dependency modeling ability. We introduce STRING, novel approach that shifts well-trained positions replace ineffective ones during inference, thereby enhancing the model's 's ability to capture distant contextual information without requiring additional training. Our experiments demonstrate that S STRING significantly boosts the performance of strong baselines like Llama 3.1 70B and Qwen-2 72B on prominent long-context benchmarks, setting new state-of-the of-the--rttttttt results for open-source LLMs.
ACKNOWLEDGMENTS
We sincerely appreciate the assistance provided by various individuals and teams. The pretraining part of this work is built upon the TinyLlama Project. We would also like to express our gratitude to the Llama3 and Qwen2 teams for providing the robust base models that have been essential for this work. Additionally, we thank Yushi Bai for his help and valuable discussions. This research was supported in part by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N N_HKU714/21.
REFERENCES
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L--eal:: Instituting standardized evaluation for long context language models. arXiv preprint urXiv:2307.11088 2023.
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training- free long-context scaling of large language models, 2024a.
Shengnan An, Zexiong Ma, Zeqi Lin, Namning Zheng, and Jian Jian-Guang Guang Lou. Make your I1m fully utilize the context, 2024b. URL https: / /awpiv.ivvv/////////////////aa org/ abs/ 222444444...111
Anthropic. Introducing 100K Context Windows, 2023. URL https: http://wwww/ttttt www.attthroic.com
110kk contexxt - w idows.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan. Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models, 2024. URL https:// /ttp///aaxvvvvv/vv///// org/ abs 2401. 18058.
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. Codeplan: Repository-level coding using llms 2309. 12499.
and planning, 2023. URL https: wwtpwxiwww....a.a.aeoteeee/bbeae
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang. Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training, 2020. URL https:
/aar//////////// abs/ 2202..00.... 12804.
Iz. Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https::t/// //tp:s//////vvi/ wrwiv.iiivv....m www.oorggaoooo abs 2000000000000 05150.
Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. ArXiv, 0as/2400.02006 2024. URL https: //api. semant cschol ar. www.oo///ooraeee :7022:::77222

---

Published as a conference paper at ICLR 2025
Cerebras.
Slimpajama: A 627b of slimpa redpajama, 2023.
token, URL hettps:/cerrbraastt
cleaned and
deduplicated
sion
Guanzheng Chen, Xin Li, Zaiqiao Meng. Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models, 2023.
Yuhan Chen, Ang Lv, Ting-En En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. Fortify the shortest stave attention: Enhancing context awareness of large language models for effective tool use, 2024. URL https: http:ps////xvvvorr///oo
wtt..orr..bbb//./aaaa.//// 04455.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiett exact attention with io-awareness. In NeurIPS, 2022.
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023.
Harry Dong, Xinyu Yang. Zhenyu (Allen) Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient Ilm inference. ArXiv, abs/2402.09399 2024. URL httph:tt//// /api. semanticscholas Crgg/CorpuIDD. 267657553.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang, Zixiao Huang, Shiyao Li, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Moa: Mixture of sparse attention for automatic large language model compression. ArXiv, abs/2406.149099 2024a.
URL //stts://appe semanticsss wemanticscholaa.. org/ 22:.puSI::2708822
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models 128k context, 2024b. URL https:
//arxiv. wwt. abs/ 2200224021... 1002.10117 10171.
Chaochen Gao, Xing Wu, Qingfang Fu, and Songlin Hu. Quest: Query-centric data synthesis approach for long-context scaling of large language model. ArXiv, 0bs/24005.1946, 2024. URL https: wew.nticsccooii........c
Arg/CorpusIDD 2 270123337.
Xinyang Geng and Hao Liu. Openllama: An open reproduction Ilama, May 2023. URL https: //github. hwtp//ppenmmmmmm weapeennn-eeseecc/ooooaeeee opama 11ama.
gkamradt. L needleinahhysttak:: Doing simple retrieval from from Ilm models. http://:tttttttttttttttt 2023, [Online; accessed
com/ gkamradt httpaa/dddd.atttteeeeeeeeeeaaaae
29-Deccembrr20233.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752. 2023.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models, 2023.
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3991-4008, 2024.
Cheng- Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https: /ttss///aaxvv..// .rg. abs 2 4 06654.
12

---

Published as a conference paper at ICLR 2025
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang, Yan Wang. Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng. Zhiwei Jiang, and Bryan Hooi. Longrecipe: Recipe for efficient long context generalization in large language models, 2024. URL https: wttpvv.iiiivv//..//...//aaaaa..a wttpa/awww....////////0000/// 00509.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2/86, 2024.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context Ilms via dynamic sparse attention, 2024. URL https: / arxiv. org/ aa///bb/////00 02490.
Hongye Jin, Xiaotian Han, Jingfeng Yang. Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Lim maybe longlm: Self-extend IIm context window without tuning, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: wwwaaaiii.ggaaaaaaaaaaaaaaea /200. 0801.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedr, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https: /ts///gtttt github..bb....g..g............ com/ http:///aaboookeeeeeceeeee wwarrch/fformeee xformers, 2022.
Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez, and Hao Zhang. Distflashattn: Distributed memory-efficient attention for long-context Ilms training, 2024a. htthtts: //axxiv org/ /bsp/b///////a/////////////aaaet 2310. 03294.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context ilms struggle with long in-context learning, 2024b. URL https: http/aaxvvvvv... abs 02060.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, al. Jamba: A hybrid transformer- mamba language model. arXiv preprint 2:XXx::400./9880 2024.
Bin Lin, Tao Peng. Chen Zhang, Minmin Sun, Lanbo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. Infinite-llm: Efficient Ilm service for long context with distattention and distributed kvcache. ArXiv, abs/2401.0266 2024a. URL http//CrrussDDIC 22I.II:::666800
https://api semanticscholar
Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, and Rui Yan. Mixture of in-context experts enhance IIms' long context awareness, ArXiv, abs/2406. 19598, 2024b. URL /etps://apte semant icscchlarcrccccco Corg/orrusIDD 270845965.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context, 2023. URL https: / arxiv. org/ abs//////////aaaaaaa/aa///aaaaea abs/ 2210. 011100018000 01889.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint, 2024a.
Xiaoran Liu, Qipeng Guo, Yuerong Song, Zhigeng Liu, Kai Lv, Hang Yan, Linlin Li, Qun Liu, and Xipeng Qiu. Farewell to length extrapolation, training-free infinite context with finite attention scope. ArXiv, abs/2407. 2024b. URL https:://////..... //aps://pp.... semant ieraanttscchhe icscholar. org/Corpus 2
Ze Liu, Yutong Lin. Yue Cao. Han Hu. Yixuan Wei. Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL https:/ //tp///wvvv.////////e//eee//.wvv wtg/abs.////eoooooooo/oooo///oor /tws///...... 14030.
Llama Team. The llama 3 herd of models. CoRR, abs/2407.21783, 2024. doi: 10.48550/ARXIV, 2407.21783. URL https: //w.. .oroo.oorg/0000 18550/arxiss 240......2...1111 21783.
13

---

Published as a conference paper at ICLR 2025
LocaILLaMA. Dynamically scaled rope further increases performance of long context Ilama with zero com./ntoommtttttmmmmm fine-tuning, July 2023a. URL httpt://wwww reddit wtt/://o///cc/e codddi...o/////LaaaLLaaaaa
rope wwwtterrrinrreaaeseeee
LocaILLaMA. Nik-aware scaled rope allows Ilama models have extended (8k+) con- text size without any fine-tuning and minimal perplexity degradation,, June 2023b. URL https:
/ommentss//411177////aaaaaa scaled_rope__ lows_ ama_modeeessa | ta s_to_have/_
/ttp://wwwwwo reddit com/I/LocalLLMa, http:/ooaaaaaaa/aooo////t////aa/
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019, URL https:
///axxiv.org/abb....... (tizg//bb/171111 05101.
Kai Lv, Lv, Xiaoran Liu, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwanjuan: Towards systematic measurement for long text quality, 2024.
Xindian Ma, Wenyuan Liu, Peng Zhang, and Nan Xu. 3d-rpe: Enhancing long-context modeling through 3d rotary position encoding. ArXiv, Abs/2406.09897, 2024. URL https://api.
semanticscho wts.cholla..or/CorruuttD
:
Xin Men, Mingyu Xu, Bingning Wang. Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. ArXiv, abs/2405. 14591, 2024. URL https: /ttts://app... /api.
semant cettiiischhlar... org/ ID:
Mistral.AI. La plateforme, 2024. URL httph:://istttaat ai aateews/la--ppaaeeommm
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context
length for transformers. arXiv preprint arXiv:2305. 2023.
Moonshot AL. Kimi chat, https: /kimiii...... www.iii.oooooot.oommmomm
OpenAl. Gpt-4 technical report, 2023.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long imput texts, yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1336-53553 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 12.18866/11//22.2.2 main..-9ii.391...... URL https:ttp//.o htt/aaaaannntooogoooo org/ 2022. www.naannnannnn main. 391.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language under- standing by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL
wwtp:/iwwwwrrras/aa//eeaa 9100. 10683.
Kaiqiang Song. Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, and Dong Yu. Zebra: Extending context window with layerwise grouped local-global attention, 2023.
Jianlin Su. Rectified rotary position embeddings. https: http:///tt.aatccoooooooootrrer. 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2022.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishray Chaudhary, Xia Song, and Furu Wei. A length-extrapplltable transformer, 2022.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. Leave no document behind: Benchmarking long-context Ilms with extended multi-doc qa. 2024a. URL https: /atp/aivvvvv////// wrg.orrgg//s...o 2406. 17419.
Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, and Bang Liu. Resonance rope: Improving context length generalization of large language models. In Annual Meeting of the Association for Computational Linguistics, 2024b. URL https: /appi.pp..//////// semanticschoo
org/CorpusID: : 268201728.
Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. Long context alignment with short instructions and synthesized positions, 2024. URL https: :ttp:s///aaiiie /ttp/aawvvvv////////////////aavl org/ abs/ 24C5. 03939.
Chaojun Xiao, Pengle Zhang. Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Inflim: Unveiling the intrinsic capacity of iims for understanding extremely long sequences with training-free memory, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog. Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. CoRR, abs/2309.160N 2023. doi: 10.48500AAXXIV.300.66333 URL https::/doi.orgg/ 48550/arX 2309.16039. 16099.
Shuo Yang, Ying Sheng. Joseph E. Gonzalez, Ion Stoica, and Lianmin Zheng. Post-training sparse attention with double sparsity. ArXiv, abs/2408.0700 070092, 2024. URL bttps://apt semant wscholar.....ooo.....oooocc.. org/ wwwp:rrprr//rouurrreo/
Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, and Dongyan Zhao. Remamba: Equip mamba with effective long-sequ sequence modeling, arXiv preprint arXiv:2408. 15496. 2024.
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending Ilm's 's context with activation beacon. ArXiv, abs/2401.03462, 2024a. URL https:// /upps://ap. api. semant iesemanticccollaa.
org/Corpuse : 26844488.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprint arXiv:2406. /685,, 2024c. URL htttps://arxiiss wwwpxiiivoor///aaoooo
htrgg/ab///// / 2406... 6852.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang. Zhiyuan Liu, and Maosong Sun. ocbench: Extending long context evaluation beyond 100k tokens, 2024d. URL https: /arxivv
www.ii..oor/aaao.o.. www.obsgg//.......... /202 13718.
15

---

Published as a conference paper at ICLR 2025
Zhenyu (Allen) Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. Found in the middle: How language models use long contexts better via -laya positional encoding. ArXiv, has/2403.04797, 2024e. URL htttss://api
semante escholalll ar . wtrg//orpussiii IIPpUsSD:: 68296885.
Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang. Peng Cheng, Lijie Wang. Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou, Longskywork: A training recipe for efficiently extending context length in large language models, 2024. URL htwps:tt/////. /tass:///riiii org/ abs 6400. 00605.
Chuanyang Zheng. Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng. Xin Jiang, Zhenguo Li, and Yu Li. Dape: Data-adaptive positional encoding for length extrapolation, 2024. URL https: / www.i.orgg///aaaaaaa...t...tt.tt hbc//bb//22 2405 14722.
Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu, Yan Gao, Yao Hu, Kehai Chen, and Min Zhang. Understanding the rope extensions of long-context Ilms: An attention perspective. ArXiv, abs/2406.13222 2024. URL httpt://appi 270620800.
org/CorpusID:
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of ilms via positional skip-wise training, 2023.
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song. Wenhao Wu, Furu Wei, and Sujian Li. Longembed: Extending embedding models for long context retrieval. ArXiv, 1is/2404.12096 2024a. URL https: //api. semant commantccccoor icscholar.......ccccooccccccaa.e wtg/CorpusID 299214659.
Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng. Xin Lv, Huanqi Cao, Xiao Chuanfu, Xingcheng Zhang, Dahua Lin, and Chao Yang. Sampleattention: Near-lossless acceleration of long context Ilm inference with adaptive structured sparse attention, 2024b. URL https://arxis org/abs/////oooooooogggoosa.ssrr 6406 15486.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 APPLYING STRING ON LLAMA3.1 128K
In this section, we demonstrate the application of STRING on Llama3.1 128K. We present the utilization of STRING to drop position indices greater than 3/2 L 2 42K and L = 64K, where L=128K represents the training length of Llama3.1. The result- ing position matrix is illustrated in Figure 8. In Figure 8a, let us consider the last row of the matrix. The original position indices [128K 1 : 2, After drop- ping position indices 2 86K, they become 3 86K 2,,0-,1..,1100
To fill the
42K empty sloss
86K indices
empty slots, we shift the positions leftwards with a stride of S
42K, resulting in [86K 1...., 2, 1, 42K 1, ....., 2, 10,,0, 0). After adding local window W of 128, we obtain the shifted position indices: [86K + 127 129, 128, 42K - 1,,,.... 2, 1. OJ. Applying STRING with an offset S 64K shown (Figure 8b). The procedure is the same. We also illustrate the changes in the last row of the position matrix. After dropping position indices 2 64K, the row is converted
64K .,...:.2............. 2, 1,. Then, the well-trained positions are shifted from
64K ,,...,2,..........,. 2, - 164K - 1, 1,0, 64K - 1,... 1,0]. Finally, the empty
the diagonal:
64K empty slots
position indices after adding local window of 128 are [64K + 127, 129, 128, 64K - 1, 1, 1,0].
Projecting Local Relationships The local value W is essential for projecting the relationships among neighboring tokens, ensuring fluent generation. To demonstrate the importance of setting local value W, consider the relative matrix of Llama3.1 128K, where the value on the 42K-th diagonal is to W instead of 42K (using ROPE) to achieve shifting effect. This means each token treats the token 42K positions away as if it's at distance of W. In an extreme case, if W = 0, the current token would treat the token 42K positions away as itself. As we know, L.LMs rely heavily on neighboring tokens to maintain fluency. If W is too small, the model can not correctly focus these nearby tokens. By setting = 128, we ensure that relative positions 0-127 are unique and assigned to neighboring tokens. This maintains fluency while enhancing long-context performance.
Handling Short-Context Input STRING will produce the same results as RoPE on short-context benchmarks. When the input length / S where S is the shifting offset, the sliding window attention which calculates attention within in neighboring S tokens. For Llama3. we have S=42K which significantly exceeds the input length of short context tasks. The sliding window attention in line 13 of Algorithm 6 calculates full attention for the short sequence without shifting positions, and the shifted self-attention mechanism not invoked.
0
428--
I
0
64K-7
42X-7
128
64K-
129
72N
42K-1
0
129
128
64K-
6
F28
47K-1
66K-116
129
128
1
64K+726
129
138
64K-1
1
A6K-122
826
129
F2N
42K-1
64X+127
645-126
(a) Shifted offset
(b) Shifted offset
Figure The resulted position matrix Llama3.1 128K after shifting. In Figure (a), we use shifted offset of L 42K and the local window W is 128. In Figure (b), we overwrite more infrequent positions and the shifted offset is S 1/2 64K.
17

---

Published as a conference paper at ICLR 2025
A.2 PRETRAINING SETUP
We pretrain two 1.3B models with maximum context window sizes of 2048 and 4096 to observe how the models gain the effective context length. The model architecture aligns with TinyLlama I.IB". We utilize a hidden size of 2,048, the size of the feed-forward layers inside each transformer block is set to 5632. The model employs 32 attention heads and comprises 22 layers. The only difference is the use of the Ilama3 tokenizer (Llama Team, 2024), which has a larger vocabulary size of 128,256 tokens compared to the 32, 200 tokens in TinyL 1.1B. This difference results in a larger embedding matrix. We used the SlimPajama-66 (227 (Cerebras, 2023) dataset as our pretraining corpus and total training tokens for each model is IT tokens.
Our pretraining codebase is primarily built on the TinyLJama project', popular codebase for reproducing Llama at the IB scale. The main speed optimization libraries employed in this project are Fully Sharded Data Parallel (FSDP)', FlashAttention-2 (Dao, 2023) and xFormers (Lefaudeux ct al., 2022)'. The entire project is based on PyTorch Lightning 8. We use the cross entropy loss as the pretraining objective and the A optimizer (Loshchilov & Hutter, 2019). Additionally, we employed a cosine learning rate schedule with a maximum learning rate of 4 10-A, starting from minimum learning rate of 4 10-5 The warmup steps are 2,000. The batch size is set to 4M tokens for different training context lengths. For the model pretrained with a 4K context length, the gradient accumulation is the sequences in a mini- batch into long sequence and used the variable-length version of Flash Attention to calculate casual self self-attention on packed sequences. A gradient clipping threshold of 1.0 is used to stablize the gradient.
twice that of the model trained with a 2K context length. We pack
We utilized 16 NVIDIA 80G A100 GPUs on 2 nodes. Training 1.3B model with a 2K context length and IT tokens took approximately 28 days, while expanding the context length to a 4K context length took around 32 days.
A.3 EFFICIENCY TEST OF STRING
In this section, we demonstrate that STRING can be implemented with negligible additional overhead compared to flash attention by comparing the inference time and GPU memory consumption. We test the baseline and STRING inputs are sourced from the summarization task in InfiniteBench (Zhang et al., 2024d). We test the model 50 times and report the average results. The results of inference time shown in Figure 9a, where we test the model with context lengths ranging from 64K to 128K. STRING maintains the average time consumed per token within 0.3 seconds of the standard Flash Attention. Figure 9b shows the consumption of GPU memory, with the growth of input context lengths, STRING exhibiting only a less than 5GB increase.
single NVIDIA 80G A100 GPU based Llama3.1 8B. The long
A.4 LIMITATIONS
One limitation of this work is that it only investigates pretraining lengths smaller than 4K tokens, while the question of how to effectively implement long-context training remains an open open. The open-source community's ' approaches this problem remains diverse (Hu al., 2024; Fu et al., 2024b; An et al., 2024a; Jin et al., 2024). For companies, Llama3.1 (Llama Team, 2024) reported using a 6-stage training approach gradually implement long-context training, but this makes difficult to analyze position frequencies because the data distribution used in each stage is unknown.
STRING achieves surprising results by only using frequent position during inference. It is clear that there are many ways to adjust the distribution of frequent positions during training, but this may
/https://hugginggfat mlob/main/config...... json
http://wwgggn.aa..ooooooooaaaaaa wot.Inyyllaa/aainllllallaa 1. 1t-11B-iintt
-43pk 3T/
Taash_atn__nnerracee.p//
18

---

Published as a conference paper at ICLR 2025
B000 Flub-tro
Avemp tite pa sazi
3.71
91-
42.00 48.00
1.4 2.07
#1-
133
10-
Saput 11238
995.
1256
lapul laugth
(a) Inference time
(b) GPU memory consumption
Figure 9: Efficiency Test of STRING and the standard Flash Attention based on Llama3.1 8B. All experiments are run on single NVIDIA 80G A100 GPU.
require data with distribution similar the Llama training corpus to avoid the model losing its reasoning ability. A key feature of STRING that it can be casily applied to all existing models without requiring the collection of high-quality data for training. We leave the problem of addressing the left-skewed distribution from a training perspective as a future work.
Algorithm 2 Pseudocode of mer gerrgeddiagg.. fted
def ded
0_shifted, attn_map_diag, aten_ma__shifted)
Mezge the attention outputs Izom the diagonal and left botton triangle.
4
Pacemeters:
6 O_dlag (Tenaor: [b, d] 1: Output tensor from diagonal attention. o_shiffed (Tensor: [M, d]) putput
from left-hottom triangle attention. attn_map_dia diag (Tensor: [L, Li): Actention map from ciagonal attention.
actn_map_ahitee
fted fted ("ensor: [9, N)): Actention nap from left-nottom triangle attension.
11
Returns:
cutput (Tansor: (L, c] ): Merged output tenser.
13
the softmax normalizer of the aliding window attention
16
S-L-8 + S is the siding window size, and 8 is the triangle height
diag_norm attn_map_diagg # the softmax normalizar of the aelf-attention
shapes [L,]
18
shifted_nerm
petn_map_shifted. www.fttedsuummil..... $ shape: [N.]
o_drag_head 0_dtag[:5] shape: 18, d], no o_diag_toil = o_diag(S:: $ d] diag_norm_tail diag_ise[S:]
for changing the first pokena
21
[8,]
23
dieg_rate tlag__ozm_tail (disg_ners_tail) shifted [X,]
shifted_rate shifted_norm. ldiog_norm_taal shifted_normi [8,]
25
Q_merged_tail disg_rate d_diag shifted O_shifted [1,4]
26
output torch.cat ([o_diag_head, t_merged_tall]) d [b, d]
return output
19

---

Published as a conference paper at ICLR 2025
Table 4: Performance of GPT-4 and 13 community models on the Needle-in-4 Haystack aeeele-in-a-Hayyttckk task at various document depths. The document is split into three equal segments: 0-33% % depth, 33-66% depth, and 66- 100% depth. Peak Failure Depth indicates the document depth at which the most test cases failed for each model. Results reported at the training length for each model.
Model
Lirai
HF PATH
Peak Failure Depth
Acc
GPT-4-128K
0-33.3%
100.0
Trained on open-source data
TinyLlama- 1.3h (ouurs))
0.33.3%
56.6
TinyL Ib
2k
TinyL.ama/Tinn Jawa//aaaaaaaaaaaaaaaaaaaa Llmma-1 1.1 -B-iitermedinte-see--ee -880--
0:33.3%
38.0
TinyLlama-I.Bel
0-33.3%
69.8
1,4h
2k
FleutherA//pythia- 1,4h
0-33.3%
22.5
OpenLlama-3B
2k
opcolm-reseurrchopeenn __ama_aaaa
0-33.3%
85.0
Llama2-7B
meta-llama/llamm-222
0-33.3%
98.0
LJama3 8B
meta Ilama// 3-7b
0.33.3%
99.8
Together-buse
32k
togethercomppter/laaaae -am---7--3
0-33.3%
63.0
LWM-base
32k
0-33.3%
31.8
Mistral-base
32k
alpindal/Miittal-7B--000
0-33.3%
52.8
Llama3.1-88
meta laallllaaaaaaaaaaaaaaaa
0.33.3%
66.0
Yarn-hase
128k
0.33,3%
32,4
Yi-6b-2000 200k
200k
0-33.3%
20.8
Gradient- Llama3-88
262k
gradientaill 1 Saniiiintttet-GGddieeete
0.33.3%
46.0
Table 5: The input format of the Needle-in-a-Haye Hayasaac (4-Needle) test where the needles are 6-digit
numbers and the haystack is Paul Graham Essays (gkamradt, 2023). The needles in this work are numbers to exclude the influence by inner-knowledge following previous work (Zhang et al., 2024c; Mohtashami & Jaggi, 2023; Hsieh et al., 2024; Zhang al., 2024d)
Haystack Needles Query
There is an important info hidden inside lot of irrelevant text. Find it and memorize them. I will quiz you about the important information there An'a July 20061 ve discovered a handy test for figuring out what you' re addicted to. Imagine you were going to spend the weekend at a friend's house on a little island off the coast of Maine. There are no shops on the island and you won't be ablc to leave while you're there. Also, you'l ve never been to this house before, you can't 't assume it will have more than any house might. What, besides clothes and toiletries, do you make a point of packing? That's what you're re addicted to... One of the magic numbers is 144231. they 're going to issue public report tracking how this specific tranche of money is spent, NFTs are I'm excited about its potential. And I'm excited see what happens with this particular auction, hecause unlike an NFT representing something that has already happened, this NET gets better as the price gets higher. The reserve price was about $2.5 million, because that's 's what takes for the name to be accurate: that's what it costs to... One of the magic numbers is 543171. you can't 't expect an idea to be novel to everyone. Any insight that you have will probably have already been had by at least one of the world's billion people. But it's sufficient if an idea is novel to a lot of readers. Ditto for correctness, importance, and strength. In effect the four components One of the magic numbers is 264468. And we were always ahead of them in features Sometimes, in desperation, competitors would try to introduce features that we didn't 't have. But with Lisp our development cycle was so fast that we could sometimes duplicate new feature within a day two of a competitor announcing it in press release. By the time journalists covering the press relcase got round One of the magic numbers 423103. nThere a founder community just as there's 's a VC community. They all know one another, and techniques spread rapidly between them. If one tries a new programming language or a new hosting provider and gets good results, 6 months later half of... In'n What are the magic numbers mentioned in the provided text?in The numbers are
territory, and this way of using them is especially new, but
Answer [144231, 543171, 264468, 423103]
20


Reviews:
[{"review_id": "BgJcfyk96e", "reviewer": "Reviewer_SwNh", "summary": "The authors present STRING (Shifted Rotary position embedding) which proposes to shift the position relevance of tokens during inference. With STRING, the authors are able to demonstrate improved performance of 10+ points on RULER, and IB for LLAMA 70B. The authors further train a tiny-llama 1.3B model to demonstrate the effectiveness and compare STRING with SOTA models. The authors also discuss the implementation of STRING with modification to flash attention.", "questions": "The results in the paper for the 1.3B Tiny-llama and the commercial models are impressive on the long-context benchmarks. However, assuming the \"system\" to mean the LLM and STRING combination will be used together, I'd also like to see benchmarks for short-context lengths - especially around coding (MBPP, Humaneval), tool-use (BFCL), and Reasoning (GSM-8K, MATH). If not for all, atleast for 1/2 models to reason if there are any trade-offs. I'm especially curious for coding and tooling benchmarks, since they need to adhere to syntax, which might be compromised by the re-positioning. \n\nSince you have implemented STRING, what is the latency overheads (and impact on throughput) vis-a-vis vanilla RoPE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thank you for submitting your paper to ICLR. I thoroughly enjoyed reading your paper, especially it is very well written. \n1. The paper observes \"left-skewed position frequency distribution\" - a bias in the position indices required to answer a relevant question being towards the shorter end. For example, on SlimPajama, the authors observe that the frequency of position indices used to model relationships between distant tokens (distances\n≥1024) is less than 20%, and for even longer distances (≥1536), it drops below 5%.\n2. The authors also demonstrate interesting findings, such as \"larger training context window consumes fewer tokens to achieve the same context length\". Although it's only been tested on tinyllama (1.3B) so weather scaling laws hold is an open problem.", "weaknesses": "1. The authors acknowledge in part (3) of 4.1 that \"Applying Eq. 3 disrupts the model’s ability to capture\nlocal relationships because it alters the relative positions between neighboring tokens\" but I'm not fully convinced if the heuristic of \"W\" is sufficient to allevate this concern. While emperically the experiments suggest so, I'd have liked to see a slightly more detailed study on the impact of altering the relative positions between neighboring tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fv5GThOCNb", "reviewer": "Reviewer_v29A", "summary": "This paper reveals the possible reasons why current LLM can't maintain good performance with long-context task even the training has involved such corpus. According to the analysis on the left-skewed frequency-distribution, the paper proposes a training-free method (STRING) to mitigate the position bias. By replacing infrequent long-range position index to frequently trained positions during inference, their method enhances the long-context capability without requiring additional training. The experiment results on multiple long-context benchmarks demonstrate the effectiveness of STRING.", "questions": "It is shown that the most distant position S indexes can be ignored during inference to obtain a better performance, have you conducted investigation whether it is the same case for training? Can decreasing the longest context window in training directly yields a better performance in inference?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The exploratory analysis about the left-skewed frequency-distribution of position index is insightful, and can help the community understand the potential reasons why the long-context problem exists in current methods.\n\n2. The proposed method is intuitive and simple to implement, which seems effective simultaneously.\n\n3. The presentation of the paper is very clear,  and the analysis together with the method description are easy to follow,", "weaknesses": "1. The parameter S and W needs to be set manually, which maybe improved by an automatic algorithm to obtain a better performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FKqislbzki", "reviewer": "Reviewer_rtHL", "summary": "This paper suggests that large language models (LLMs) struggle to handle dependencies over long distances due to the skewed occurrence of relative distance towards shorter ranges while training. The authors propose STRING, a simple method that re-uses more frequently occuring positions for distant tokens, avoiding the use of under-utilized long-distance positions. Through extensive evaluations on long-context benchmarks, they demonstrate that STRING can improve the performance of SOTA models within their pre-trained context windows.", "questions": "1. Can this method also be useful for context extrapolation? I understand that the primary focus of the paper is to improve performance within the pre-trained context window, but I believe this is also a large understated benefit of your method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. The proposed method is simple and effective. In particular, it improves the performance of existing LLMs within their pre-trained context window.\n3. The proposed method can be easily deployed. STRING does not require any additional training and can therefore be easily applied to existing LLMs. The authors also provide a way to make the method compatible with FlashAttention, enabling fast and efficient inference.", "weaknesses": "1. Observation (1) in Section 3 (line 216) is overstated, as training with longer context involves larger computation demands due to the quadratic nature of self-attention. For fair comparison, training efficiency should be compared with the same amount of computation, instead of the number of consumed tokens.\n2. STRING looks more like an empirical workaround to avoid using less-frequent positions, rather than a fundamental solution for the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "obtc3a7pqb", "reviewer": "Reviewer_QJGa", "summary": "This paper discusses how current LLMs use only a portion of their maximum context length due to issues in handling distant information. The authors identify the root cause as a left-skewed distribution in the model's relative positions during pretraining and post-training. \n\nTo address this, they introduce a new method called Shifted Rotary Position Embedding (STRING), which adjusts well-trained positions during inference to improve performance without extra training. \nExperimental results show that STRING enhances the context length usage of open-source models like Llama 3.1 70B and Qwen2 72B by over 10 points on benchmarks like RULER and InfiniteBench. STRING's effectiveness even surpasses commercial models like GPT-4-128K, Claude 2, and Kimi-chat.", "questions": "1. Where can we find your implementation of STRING?\n\n2. Are error bars available for the tables in the experiments? If space is limited, could they be included in the Appendix to give readers a better understanding of the variability in the results?\n\n3. Regarding the improved results shown in the experiments, could you provide more insights into why and how STRING performs effectively? Additionally, are there any scenarios where STRING might not perform as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-structured, with clear organization, understandable tables, and figures, effectively addressing an important topic in LLM development.\n\nThe empirical analysis of position frequency on the training data is thorough (However, I recommend providing additional explanation regarding the meaning of the plot in Figure 1 in the Appendix for further clarity) .\n\nThe method has been evaluated on/against quite a few LLMs.", "weaknesses": "1. The code is missing; the provided GitHub link leads to a 404 error.\n\n2. The analysis in Section 3 is intriguing, but the results are limited to small-sized LLMs. It remains uncertain whether the same findings apply to medium- and large-sized LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)