Title: An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model

Long Context

Position Encoding
Abstract: Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle ($\texttt{CREAM}$), which interpolates positional encodings by manipulating position indices. Apart from being simple, $\texttt{CREAM}$ is training-efficient: it only requires fine-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs to a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the ''Lost-in-the-Middle'' problem faced by long-context LLMs. Experimental results show that $\texttt{CREAM}$ successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with ``Never Miss A Beat''. Our code is publicly available at https://github.com/bigai-nlco/cream.

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding

Abstract

Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ( ≫ 4 K ) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose C ontinuityR elativity ind E xing with g A ussian M iddle ( CREAM ), which interpolates positional encodings by manipulating position indices. Apart from being simple, CREAM is training-efficient: it only requires fine-tuning at the pre-trained context window ( e.g ., Llama 2-4K) and can extend LLMs to a much longer target context length ( e.g ., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the 'Lost-in-the-Middle' problem faced by long-context LLMs. Experimental results show that CREAM successfully extends LLMs to the target length for both Base and Chat versions of Llama2-7B with 'Never Miss A Beat'. Our code is publicly available at https://github.com/bigai-nlco/cream .

1th

19th

38th

55th

75th

Position of Key to Retrieve

40

60

80

100

PoSE

Ours

(a) Linear Interpolation

1th

19th

38th

55th

75th

Position of Key to Retrieve

20

40

60

80

100

PoSE

Ours

1 Introduction

Transformer-based Large Language Models (LLMs) are typically pre-trained with a fixed context window size, e.g ., 4K tokens in Touvron et al. [2023a]. However, many downstream applications, including in-context learning [Huang et al., 2023, Li et al., 2023a] and LLM agents [Qian et al., 2023, Zheng et al., 2023] necessitate the processing of significantly longer contexts, e.g ., up to 256K tokens. Recent works have proposed promising approaches that efficiently extend the context window of

pre-trained LLMs by interpolating Positional Encodings (PEs) [Chen et al., 2023, Peng and Quesnelle, 2023, Peng et al., 2023, Xiong et al., 2023, Zhang et al., 2024] with a short period of fine-tuning. Unlike other techniques such as efficient transformer [Tworkowski et al., 2024, Munkhdalai et al., 2024] and memory augmentation [Tan et al., 2024], PE-based methods do not necessitate alterations to the model's architecture or the incorporation of supplementary modules. Consequently, PE-based methods offer the advantages of straightforward implementation and rapid adaptation, making them a practical solution for extending the operational range of LLMs in tasks involving larger context windows.

Despite the simplicity and effectiveness, existing PE-based methods exhibit two significant limitations. First, prior approaches, such as positional interpolation [Chen et al., 2023], still require fine-tuning on the target context window size, which imposes a substantial computational overhead [Zhu et al., 2023]. Secondly, though some PE methods have demonstrated potential in handling extremely long sequences, as evidenced by low sliding window perplexity scores, their performance deteriorates notably in 'in-the-middle' scenarios [Liu et al., 2024]. Specifically, when the model is required to accurately retrieve and process content located in the middle of an extended context, there is a marked drop in performance on the extended window size (Figure 1 and Figure 3).

These observations and insights underscore a fundamental question: Can we extend the context window size of pre-trained LLMs efficiently while simultaneously optimizing their effectiveness in processing "in-the-middle" content?

To answer the above question, we propose CREAM , namely C ontinuityR elativity ind E xing with g A ussian M iddle. CREAM is a novel PE-based fine-tuning recipe that shows both efficiency in fine-tuning and effectiveness in enhanced middle content understanding. Our key insights lie in manipulating the positional indices of long target sequences to produce shorter ones within the pre-trained context window size (Figure 2).

In Section 2.1, we summarize two crucial ingredients of effective positional indices: continuity that produces densely connected positional indices and relativity that reveals the long-range dependencies between fragments. CREAM is a recipe designed with the best of both worlds by introducing two indexing strategies for continuity and relativity, respectively (Section 2.2). Besides, to alleviate the 'Lost-in-the-Middle' challenge, we introduce truncated Gaussian distribution for middle segment sampling, enabling the LLM to prioritize the information in the middle positions, even when performing positional interpolation within the pre-trained context window size.

In Section 3, we conduct comprehensive experiments to demonstrate the efficiency and effectiveness of CREAM . We continually pre-trained on Llama 2-7B with CREAM for a short period and extend the context window size from 4K to up to 256K. Furthermore, we instruction tuning on Llama 2-7B-Chat with CREAM for 100 steps and obtain promising results. We highlight our empirical advantages as:

CREAM can not only fine-tune within the pre-training context window size, but also alleviate the issue of the model easily getting lost in the middle. e.g ., CREAM -YaRN outperforms PoSE-YaRN [Zhu et al., 2023] by over 20% on average in the 'Lost in the Middle' [Liu et al., 2024] task.

CREAM can further be enhanced by integrating novel designs on positional interpolation frequencies (such as Linear [Chen et al., 2023], NTK [Peng and Quesnelle, 2023], Yarn [Peng et al., 2023], etc .), and can be extended to context window sizes of up to 256K or beyond.

CREAM -Chat model requires only 100 steps of instruction-tuning to achieve nearly perfect performance on the Needle-in-a-Haystack pressure test, and it outperforms existing strong baselines on LongBench [Bai et al., 2023].

2 Methodology

2.1 Preliminaries

Problem Formulation. Given an LLM with a pre-trained context window size N , our goal is to unlock the inference capacity of the LLM on the testing data D test with an extended context window size L (where L > N ) by efficiently learning from a small-scale training data D train with a maximum sequence length N . We expect the extended model to perform reasonably well in long-context evaluation.

Middle

0

L-1

0 < 𝑘 ≪ 𝑁

Head

𝑁/3

Relativity

𝐿ℎ

= 𝐿t = 𝑁/3

Tail

𝑁/3

𝑷𝒔

𝑷𝒆

1

L-2

𝑁

Pretrained Context Window

…

…

Full Extended Context Window

Continuity

0

1

8192

…

32K

absolute position

relative position

Fine-tuning Position Indices:

8193

…

Middle

Head

Tail

𝑘

𝑷𝒔

𝑷𝒆

𝑘

Continuity in Positional Encoding. Transformer-based language models typically encode positional indices sequentially as { 0 , 1 , · · · , N -1 } . Traditional length extension methods [Chen et al., 2023, Peng and Quesnelle, 2023, Peng et al., 2023] directly fine-tune on the target length L with an updated positional index. This approach preserves the continuity of all absolute positions and learns all position indices within [0 , L -1] , thereby successfully extending to the target length. Furthermore, PoSE [Zhu et al., 2023] attributed their superior performance over RandPos [Ruoss et al., 2023] to the ensured continuity of segments during fine-tuning.

Relativity in Positional Encoding. Relative positional encoding (RPE) [Shaw et al., 2018] has been proposed as an effective positional encoding method, where only the relative positions between two tokens are considered. Similar to prior works [Ruoss et al., 2023, Zhu et al., 2023, Wu et al., 2024], our work focuses on rotary positional encoding (RoPE) [Su et al., 2024], which is one of the most prominent RPE methods and has been widely applied to LLMs including the recent Llama family [Touvron et al., 2023b,a, AI@Meta, 2024]. In RoPE, only the relative distances between position pairs ( | j -i | ; 0 ≤ i < j ≤ L -1) are learned during fine-tuning (Appendix A). Due to this property, we can manipulate the position indices such that all relative positions between [0 , L -1] are learnable within the pre-trained window size.

2.2 Proposed Recipe: Continuity-Relativity indExing with gAussian Middle ( CREAM )

In the following section, we start by introducing our design of dividing the context window N to learn relative positional information. Then, we propose two strategies that target continuity and relativity, respectively. Lastly, we propose a novel truncated Gaussian sampling method to enhance the middle part of the long context. The overall framework is depicted in Figure 2.

Context division. We first discuss the motivations behind our design of the context length. First, prior works [Han et al., 2023, Xiao et al., 2023] observed that a significant amount of attention score is allocated to the beginning tokens of a sequence, which can potentially encode absolute positional information even without explicit positional encoding [Kazemnejad et al., 2024]. Secondly, the starting and ending tokens of long contexts can be treated as two pointers that localize the middle indices with the help of relative encodings. Therefore, we divide the pre-trained context window into three segments. The detailed ablation results are shown in Section 3.6.

Definition 2.1. Given the pre-trained context window size N and target extended length L , the position set of { Head,Middle,Tail } is defined as follows:

where L h and L t denote the length of the head and tail segments, P s and P e denote the start and end position index of the middle segment.

The relative positions among the three segments in each sample are calculated in pairs, i.e ., {| j -i | ; ∀ i, j ∈ { Head,Middle,Tail }} .

The formed relative distance union D r learned by the model is given by:

Given that not all samples possess the same values for L h , P s , P e , and L t , as fine-tuning progresses, the union D r in Equation (2) can encompass the entire range [0 , L -1] , facilitating the model to learn all relative positions within the target length L .

Two segmentation strategies. For the sake of continuity , we set the L h and L t to a very small value k , where 0 < k ≪ N . Specifically, we use k = 32 in our experiments. This choice allows the middle segment to closely approximate the pre-trained context window. To maintain relativity , we divide N equally into three parts and fix the L h and L t to N/ 3 , enabling the model to learn as many relative positions as possible. In our fine-tuning process, both types of examples are sampled with equal probability to maintain balance.

Truncated Gaussian Middle Sampling To better focus the training process on the middle part of the long context, we introduce a truncated Gaussian function. This approach reduces the interval overlap in Equation (2) and directs the model's attention toward the middle section of the long context. In Appendix B, we provide theoretical justifications of our truncated Gaussian design, indicating that the maximization of | D r | holds for middle positions in [ N,L/ 2) ∪ ( L/ 2 , L -N ] .

Formally, given the probability density function (PDF) of a Gaussian distribution:

where µ is the mean and σ is the standard deviation. The corresponding cumulative distribution function (CDF) is:

where E ( · ) is the error function. To calculate the CDF value within the truncated interval, we use a sufficiently large number ( e.g . 1000) of equally spaced x values from the given interval [1 , L/N ] :

By substituting Equation (4) into Equation (3), the cumulative distribution function (CDF) curve is derived within the truncated interval. For sampling from this truncated Gaussian distribution, the inverse transform method is employed, as demonstrated in Equation (5):

where u ∼ Uniform (0 , 1) , round ( · ) represents rounding to the nearest integer. Finally, we can get:

where L m denotes the length of the middle segments. In summary, the overall sampling flow of our algorithm is presented in Algorithm 1.

3 Experiments

3.1 Experimental Setup

Extended Models We use Llama-2-7B and Llama-2-7B-Chat [Touvron et al., 2023a] as the base models and extend their pre-trained context window size of 4K to a target context length of 32K. The extended models are referred to as CREAM -Base and CREAM -Chat, respectively. Note that, though the target context length is 32K, we do not have to fine-tune CREAM on 32K token long text (see Section 2.2).

Algorithm 1 CREAM sampling algorithm

Require: Pre-trained context window size N , extended context window size L , training sample size S , mean µ , variance σ and hyperparameter k .

1: Generate enough x equally spaced according to Equation (4).

2: Substitute x into Equation (3) to derive the truncated Gaussian CDF F ( x ) .

3: for i = 0 to S -1 do

4: Sample L h ∼ DiscreteUniform ( { k, N/ 3 } ) , and let L t = L h , L m = N -L h -L t .

5: Sample u ∼ Uniform (0 , 1) , and substitute it into Equation (5) to get α .

6: Calculate the start and end position ids P s , P e of the middle part according to Equation (6).

7: Get position set P i = { 0 , 1 , . . . , L h , P s , . . . , P e , L -L t . . . , L -1 } , where | P | = N .

8: end for

9: return P = { P 0 , P 1 , . . . , P S -2 , P S -1 } .

Benchmarks We conduct long-context LLM evaluation of CREAM -Base on LongChat-Lines [Pal et al., 2023] and Lost-in-the-Middle [Liu et al., 2024]. Ideally, fine-tuning should not disrupt what the base model has learned, so we further evaluate CREAM -Base on the language modeling task and the evaluation benchmark [Beeching et al., 2023] adopted by Llama2. Additionally, we assess the CREAM -Chat model with Needle-in-a-Haystack 1 and LongBench[Bai et al., 2023]. Unless otherwise specified, we use linear interpolation to adapt LLMs to a longer context length.

Baselines As far as we know, RandPos [Ruoss et al., 2023] and PoSE [Zhu et al., 2023] are similar to our approach in that they manipulate position indices to enable fine-tuning on the pre-trained length for context expansion. Therefore, these two methods serve as the baselines for our primary comparisons. More details about the experimental setup can be found in the Appendix C.

3.2 Effective Context Window Size Evaluation on CREAM -Base

We evaluate the long-context understanding capabilities of the CREAM -Base model on two tasks: LongChat-Lines 2 [Pal et al., 2023] (Figure 3) and 'Lost in the Middle' [Liu et al., 2024] (Table 1).

2500

3600

4200

4800

6000

7100

9400

11800

14000

16000

17500

20000

22000

26000

28000

30000

32000

Token Length

0

20

40

60

80

100

96.0

82.0

92.0

94.0

92.0

86.0

84.0

78.0

76.0

56.0

62.0

52.0

62.0

46.0

52.0

38.0

40.0

Original

RandPos

PoSE (Avg:54.0)

Ours (Avg:69.9)

CREAM -Base performs best in retrieving information from long contexts of varying lengths. We extend the context window size up to 32K and compare CREAM with the Llama 2-7B [Touvron et al., 2023a], RandPos [Ruoss et al., 2023], and PoSE [Zhu et al., 2023]. As the context window size increases, the performance of all models drops, but CREAM always performs best except for the window size of 3.6K (see Figure 3). In terms of the average performance over all context window sizes, CREAM outperforms PoSE by 16%, demonstrating its good long-context understanding ability.

CREAM -Base alleviates the Lost-in-the-Middle issue. Lost-in-the-Middle is an observation that LLMs are generally good at retrieving relevant information appearing at the beginning/end of the input context [Liu et al., 2024]. To validate the effectiveness of our middle-focused truncated Gaussian

sampling, we evaluate CREAM and compare it with PoSE on the key-value retrieval task proposed by Liu et al. [2024]. We present results in Table 1, where the cyan shading indicates middle segments. We find that: regardless of the chosen interpolation method, CREAM always outperforms PoSE by a large margin. e.g ., CREAM -Linear surpasses PoSE-Linear by 21.2% when the relevant information is placed at 18.

3.3 Long Context Understanding Evaluation on CREAM -Chat

We conduct long-context evaluations of CREAM -Chat on two tasks:

Needle in A Haystack (Figure 10) This task is a test that places an answer ( i.e ., Needle) at any position of a long context window ( i.e ., Haystack) and requires a model to retrieve the correct answer given a question-answer pair. We follow Wu et al. [2024] and use the GPT ( GPT-3.5-Turbo-0125 ) score as the evaluation metric.

LongBench (Table 2) Bai et al. [2023] is a more realistic benchmark because it covers real-world application scenarios like long-context QA and summarization. Moreover, it is specifically designed for Chat models.

(a) SkipAlign

†

1k

4k

8k

12k

15k

18k

22k

25k

29k

32k

0%

20%

40%

60%

80%

2

4

6

8

10

gpt score

(b)

CREAM

0%

20%

40%

60%

80%

CREAM -Chat outperforms SkipAlign in context window expansion. We visualize the results of CREAM -Chat and the recent SkipAlign in Figure 10. Clearly, CREAM -Chat beats SkipAlign because the performance of SkipAlign [Wu et al., 2024] decreases from the window size of 18K while CREAM -Chat displays a perfect performance everywhere until from the window size of 29K. Notably, CREAM -Chat is only fine-tuned for 100 steps.

CREAM -Chat makes best use of the extended context window size. We present results on LongBench in Table 2. CREAM -Chat again surpasses strong baseline models, demonstrating its better use of extended context size. In terms of the average performance over all tasks, it outperforms the second

best model, i.e ., LongChat-v1.5-7B-32k [Li et al., 2023b], by 1.6%, though it is only tuned on a very small amount of data and for only 100 steps.

3.4 Effectiveness of PEFT Integration

To demonstrate that CREAM can be directly combined with PEFT techniques (such as LoRA [Hu et al., 2022] and QLoRA [Dettmers et al., 2023]), requiring no additional modifications. We conducted experiments on LLaMa-2-7B-Chat using the identical dataset and settings. The experimental results are presented in Table 3. The results indicate that models fine-tuned using LoRA and QLoRA achieve performance nearly equivalent to those fine-tuned with full parameter.

3.5 Language Modeling and Standard Benchmark

Following Chen et al. [2023], Zhu et al. [2023], Peng et al. [2023], we perform the classic language modeling evaluation, i.e ., perplexity evaluation, on GovReport [Huang et al., 2021] and Proofpile [Zhangir Azerbayev, 2022]. Since a lower perplexity does not necessarily imply better model performance on downstream tasks [Zhang et al., 2024, Hu et al., 2024, Arora et al., 2024, Park et al., 2024], we further conduct evaluation on the standard natural-language-understanding (NLU) benchmark [Beeching et al., 2023]. This also lets us know whether fine-tuning hurts the NLU ability of the pre-trained base model.

Both CREAM and PoSE demonstrate the lowest perplexity. We apply different positional interpolation methods to RandPos [Ruoss et al., 2023], PoSE [Zhu et al., 2023], and CREAM and report their perplexities in Table 4. We find that: CREAM and PoSE have a similar perplexity in different settings and both outperform RandPos. This occurs primarily because the position indices used during RandPos fine-tuning are discontinuous, which creates an inconsistency with the pre-training stage.

CREAM has nearly the same NLU abilities as the pre-trained base model. Ideally, fine-tuning should not adversely affect the original capabilities of the pre-trained base model. Our evaluation of CREAM confirms this, i.e ., CREAM nearly retains all NLU abilities of the base Llama2-7B (see Table 5). Interestingly, CREAM improves over Llama2-7B on ARC-C and HellaSwag. This is because

these two tasks are few-shot tasks with longer prompts, necessitating the assistance of long-context understanding.

Extending the context length to 256K. We push the limit and extend the context length of Llama-2-7B up to 256K. Following Zhu et al. [2023], we evaluate the extended model by calculating the average perplexity over 20 samples from PG-19 [Rae et al., 2019] and Book3 [Presser, 2020]. 3 Since the PG-19 test set does have enough samples that are longer than 256K, we select a subset of samples from the PG-19 training set.

We experiment with target context lengths 64K, 96K, 128K, 192K, and 256K and apply different positional interpolation methods to the extended model (see Table 6). The results of PoSE [Zhu et al., 2023] in Table 6 are based on fine-tuning LLaMa 1-7B with 2K data length, and are provided for reference only. Surprisingly, the increase of the target context length brings little to no perplexity increase, demonstrating the stability of CREAM across different target context lengths, even when the target context is extremely long.

3.6 Ablation Study

To validate the effectiveness of our modeling choices, we further conduct an ablation study of three main components of CREAM : truncated Gaussian sampling, fixed start and end segments, and the trade-off between continuity and relativity.

Truncated Gaussian sampling versus Uniform sampling. We use truncated Gaussian sampling to encourage CREAM to make better use of the middle part of the context. As a comparison, we replace

4K

9K

16K

20K

28K

32K

Token Length

40

50

60

70

80

90

92.0

84.0

56.0

52.0

52.0

40.0

Uniform (Avg:60.7)

vs

. Uniform.

4K

9K

16K

20K

28K

32K

Token Length

0

20

40

60

80

92.0

84.0

56.0

52.0

52.0

40.0

None (Avg:44.0)

Only Head (Avg:49.9)

Only Tail (Avg:52.7)

Head & Tail (Avg:69.9)

(b) Head

vs

. Tail.

4K

9K

16K

20K

28K

32K

Token Length

20

40

60

80

92.0

84.0

56.0

52.0

52.0

40.0

Head=Tail=k (Avg:55.2)

Head=Tail=N/3 (Avg:44.2)

Random (Avg:69.9)

(c) Relativity

vs

. Continuity.

it with the Uniform sampling (see Figure 5(a)). We observe that the Uniform sampling always leads to worse retrieval performance, suggesting the effectiveness of the truncated Gaussian sampling.

Fixing the head and tail segments is crucial for good retrieval performance. We compare our choice of fixing the head and tail segments with three alternatives: (i) removing both the head and tail segment, (ii) fixing only the head segment, and (iii) fixing only the tail segment (see Figure 5(b)). We find that: removing the head and tail segments leads to the worst performance; it results in a complete failure ( i.e ., zero score) for the context size 32K. Keeping either head or tail segments performs slightly better than removing both but underperforms our default choice of fixing both. We suppose that this is because fixing both gives rise to better relativity information, a finding that is consistent with that of Han et al. [2023].

Maintaining a good balance between continuity and relativity is necessary. We encourage continuity by setting the head and tail segment lengths to k = 32 and elicit relativity by letting k = N/ 3 (see Section 2.2). To balance the two desired properties, we randomly choose k = 32 and k = N/ 3 with an equal probability during fine-tuning. Here we compare three scenarios: (1) enforce only continuity, (2) enforce only relativity, and (3) balance continuity and relativity (see Figure 5(c)). We find that balancing continuity and relativity gives rise to the best performance, thus justifying our modeling choice.

Ablation of Hyperparameters In our implementation of truncated Gaussian sampling, as illustrated in Equation (3), the only hyperparameters are the mean µ and the variance σ . The mean µ is determined by the expansion factor. The variance σ is adaptable based on data, we conducted experiments with five different values of σ . The results, as presented in Figure 6, indicate that the current selection ( σ = 3 ) yields optimal performance.

2500

3600

4200

4800

6000

7100

9400

11800

14000

16000

17500

20000

22000

26000

28000

30000

32000

Token Length

20

40

60

80

100

sigma=2.0 (Avg:50.6)

sigma=2.5 (Avg:58.5)

sigma=3.0 (Avg:69.9)

sigma=3.5 (Avg:59.3)

sigma=4.0 (Avg:54.0)

4 Related Works

Efficient Transformers and Extra Memory FoT [Tworkowski et al., 2024] addresses the limitations of local attention in transformers by integrating memory attention layers, which enable large models to learn from a wide context while reducing interference. Infini-attention [Munkhdalai et al., 2024] incorporates compressed memory into the standard attention mechanism and integrates masked local attention and long-term linear attention mechanisms within a single Transformer block. LLoCO [Tan et al., 2024] employs LoRA in conjunction with context compression, retrieval, and parameter-efficient fine-tuning to learn context offline. Although these methods can successfully extend the long context window of LLMs, they either require modifications to the attention mechanism or the addition of extra modules for assistance. In contrast, CREAM does not require these operations and can be directly applied to a pre-trained model.

Positional Interpolation Chen et al. [2023] first proposed extending the context window through positional interpolation, which linearly reduces the input position indices to match the original context window size, thereby preventing catastrophic high attention scores from completely disrupting the self-attention mechanism. Subsequently, various methods (such as NTK [Peng and Quesnelle, 2023], ABF [Xiong et al., 2023], and EABF [Zhang et al., 2024]) emerged that modify the base frequency of rotary positional encoding to achieve positional interpolation. YaRN [Peng et al., 2023] introduced a segmented interpolation method, applying different positional interpolations to different dimensions. LongRoPE [Ding et al., 2024] identifies and utilizes two forms of non-uniformity in positional interpolation through search, and introduces a progressive expansion strategy for positiona interpolation. Moreover, CREAM can be combined with any positional interpolation method.

Positional Encoding RandPos [Ruoss et al., 2023] first modified position indices so that the model leverages the relativity of positions, enabling it to extend to the target length with fine-tuning over shorter lengths. PoSE [Zhu et al., 2023] then emphasized the importance of continuous segments, dividing the training length into two parts to further enhance the interpolation effect. CREAM utilizes both relativity and continuity, and it also better enables the model to focus on the middle part of the context.

5 Conclusion

We proposed C ontinuityR elativity ind E xing with g A ussian M iddle ( CREAM ), a simple yet effective method to extend the context of large language models. CREAM achieves a trade-off between continuity and relativity, enabling the model to exploit positional relativity ( i.e ., fine-tuning within the pre-trained length), while preserving text continuity ( i.e ., remaining as close as possible to the pre-trained state). Furthermore, by employing truncated Gaussian sampling, the model can concentrate more on the middle positions during fine-tuning. Experimental results demonstrate that CREAM outperforms other methods on both Base and Chat models and effectively mitigates the issue of 'lost in the middle'.

References

Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise training. In The Twelfth International Conference on Learning Representations , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023a.

Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, and Mao Yang. Boosting llm reasoning: Push the limits of few-shot learning with reinforced in-context pruning. arXiv preprint arXiv:2312.08901 , 2023.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939 , 2023a.

Chen Qian, Xin Cong, Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023.

Zilong Zheng, Zixia Jia, Mengmeng Wang, Wentao Ding, Baichen Tong, and Songchun Zhu. Langsuit · e: Controlling, planning, and interacting with large language models in embodied text environments, 2023. URL https://github.com/bigai-nlco/langsuite .

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 , 2023.

Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation., 2023. URL https://redd.it/14lz7j5 .

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations , 2023.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039 , 2023.

Yikai Zhang, Junlong Li, and Pengfei Liu. Extending llms' context window with 100 samples. arXiv preprint arXiv:2401.07004 , 2024.

Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miło´ s. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems , 36, 2024.

Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143 , 2024.

Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E Gonzalez, and Raluca Ada Popa. Lloco: Learning long contexts offline. arXiv preprint arXiv:2404.07979 , 2024.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 12, 2024.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 , 2023.

Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 1889-1903, 2023.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 464-468, 2018.

Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. Long context alignment with short instructions and synthesized positions. arXiv preprint arXiv:2405.03939 , 2024.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023b.

AI@Meta. Llama 3 model card. 2024. URL https://github.com/meta-llama/llama3/blob/main/ MODEL_CARD.md .

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137 , 2023.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 , 2023.

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems , 36, 2024.

Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308.10882 , 2023.

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard. https://huggingface.co/spaces/ HuggingFaceH4/open_llm_leaderboard , 2023.

Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Models@ ICML2023 , 2023.

Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source LLMs truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following , 2023b. URL https://openreview.net/forum?id= LywifFNXV5 .

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=OUIFPHEgJU .

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021 , pages 1419-1436. Association for Computational Linguistics (ACL), 2021.

Bartosz Piotrowski Zhangir Azerbayev, Edward Ayers. Proof-pile. 2022. URL https://github.com/ zhangir-azerbayev/proof-pile .

Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? In The Second Tiny Papers Track at ICLR 2024 , 2024.

Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=LY3ukUANko .

Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2024. URL https://openreview.net/forum?id=xvr0Hctddy .

Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations , 2019.

Shawn Presser. 2020. URL https://twitter.com/theshawwn/status/1320282149329784833 .

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753 , 2024.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations , 2023.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

A Relative Positional Encoding in RoPE

We provide a simple background proof on the relative positional encoding performed by Rotary Position Embedding (RoPE) Su et al. [2024]. Given two embedding vectors x q , x k ∈ R d corresponds to query and key at positions ( m,n ) ∈ [0 , L ) , where d is embedding dimension, their encoding counterparts can be defined as:

where is the rotary matrix, Θ = { θ i = 10000 -2( i -1) /d , i = [1 , 2 , . . . , d/ 2] } is pre-defined rotation angles. Then the self attention score can be obtained with:

where x ∗ represents the conjugate complex of x , g is the derived attention function of RoPE. As seen, RoPE only depends on the relative distances between and encodes the relative position information.

B Theoretical findings of CREAM design

Theorem B.1. If N ≪ L , the spanning size | D r | of the relative position union in Equation (2) reaches its maximum iff. one of the following groups of inequalities satisfies:

or

Proof. Denote four intervals in Equation (2) as S i , i = 1 , . . . , 4 . According to the inequality of inclusion-exclusion principle for the cardinality of the union of n sets:

where the equality holds iff. all sets are pairwise disjoint. That is

̸

Given intervals as in Equation (2), we have

where MAX = max( L h -1 , P e -P s , L t -1) . The above inequalities can be simplified to Equations (10) and (11).

LemmaB.2. Under mild assumptions that L -L t ≈ L , L + L h ≈ L , the maximization in Theorem B.1 holds for all ( P s , P e ) ∈ [ N,L/ 2) ∪ ( L/ 2 , L -N ] .

Proof. Given that

the inequalities in Equations (10) and (11) turns into [ N,L/ 2) ∪ ( L/ 2 , L -N ] .

Theorem B.3. If N ≪ L , when the spanning size | D r | of the relative position union in Equation (2) reaches its maximum, we denote the coverage area of the middle segment as:

thus, we have:

Furthermore, as N L → 0 , we have:

C Experimental Details

Model Hyperparameters We fine-tune all models by optimizing the causal language modeling objective. A learning rate of 2 × 10 -5 with a linear scheduler is adopted, incorporating 10 warmup steps. We use the AdamW Loshchilov and Hutter [2018] optimizer with the hyperparameter configurations specified by PyTorch Paszke et al. [2019]. To speed up fine-tuning, we resort to DeepSpeed 4 ZeRO stage 1 and Flash Attention-2 Dao [2023]. We perform fine-tuning on two A100-80G GPUs with a total batch size of 32 and run inference on a single A100-80G GPU. For CREAM -Base, we fine-tune it for 1,000 steps on a dataset derived from Pile Gao et al. [2020]; for CREAM -Chat, we fine-tune it for 100 steps on ShareGPT Zheng et al. [2024]. To ensure fair comparison, we follow the fine-tuning and inference configurations established by Zhu et al. [2023].

Datasets and Training Cost For training the Base model, we directly utilize The Pile data provided by Zhu et al. [2023], and select samples with token lengths exceeding 4K. For training the Chat model, we filter the ShareGPT data from public datasets 5 . Specifically, we used the Vicuna prompt template to sequentially concatenate the ShareGPT data until each data point comprises at least 4K tokens. Then, we select 3.2K data points to train for 100 steps. Particularly, during the instruction tuning process, we mask the USER part and allow the model to calculate the loss only on the ASSISTANT part. We utilize two A100-80G machines with a global batch size of 32, fully utilizing the available memory. Running 1,000 steps for the Base model takes approximately 6 hours, while running 100 steps for the Chat model takes approximately 2 hours.

D Robustness Across LLMs

Our proposed method exhibits strong generalization capabilities and can be applied to other large language models (LLMs) without the need for parameter modification. To validate this, we conducted experiments on Baichuan2-7B , with the corresponding results presented in Table 7.

Furthermore, we fine-tuned LLaMa3-8B using a context window size of 4 K tokens, with the experimental outcomes shown in Table 8.

The results in Tables 7 and 8 clearly demonstrate the transferability of our method to different models, underscoring its robustness. Of particular note is that despite LLaMa3-8B having a native context length of 8 K tokens, fine-tuning on training data with a 4 K context window yielded unexpectedly strong performance.

E LongChat Lines Results

The interpolation methods using NTK and Yarn are presented in Figures 7 and 8. As can be seen, CREAM performs the same as the Linear method for interpolation, still outperforming other methods. The result of NTK at 26K-32K is zero, which is due to the inherent properties of NTK, a finding that is aligns with Zhu et al. [2023].

2500

3600

4200

4800

6000

7100

9400

11800

14000

16000

17500

20000

22000

26000

28000

30000

32000

Token Length

0

20

40

60

80

100

96.0

86.0

88.0

84.0

60.0

58.0

38.0

36.0

26.0

26.0

16.0

12.0

12.0

0.0

0.0

0.0

0.0

Original

RandPos

PoSE (Avg:33.2)

Ours (Avg:37.5)

F LongBench Subtasks Results

The results of each subtask in Tables 2 are shown in Tables 10 and 11.

It is noteworthy that, to provide further evidence of the efficacy of our model, we have specifically chosen 12 tasks from the four categories outlined in Zhang et al. [2024] for comparison purposes. As delineated in Table 9, we are able to attain superior performance on LongBench in comparison to EABF Zhang et al. [2024], even with shorter training lengths and less data.

G Limitations

When extending the context beyond the pre-trained length, there is an inevitable loss of information due to position interpolation, particularly when fine-tuning is restricted to the pre-trained length. However, in comparison to previous methods such as RandPos Ruoss et al. [2023] and PoSE Zhu et al. [2023], CREAM has effectively mitigated the issue of 'Lost-in-the-Middle' by introducing truncated Gaussian sampling. Additionally, as discussed in reference Liu et al. [2024], decoder-only models are prone to inherently exhibiting a U-shaped performance curve on this task. Therefore, completely solving this problem remains challenging.

2500

3600

4200

4800

6000

7100

9400

11800

14000

16000

17500

20000

22000

26000

28000

30000

32000

0

20

40

60

80

90.0

94.0

92.0

88.0

78.0

72.0

62.0

52.0

34.0

32.0

32.0

26.0

24.0

14.0

12.0

6.0

14.0

Original

RandPos

PoSE (Avg:37.2)

Ours (Avg:48.4)

Token Length

H Loss Curve

(a)

CREAM

-192K Training Loss

(b)

CREAM

-192K Validation Loss

(c)

CREAM

(d)

CREAM

-256K Validation Loss

2.05

1.6

1.55

1.5

1.95

1.45

4

1.9

001

200

300

400

00S

600

700

800

006

1000X

100

200

300

500

500

700

800

006

1000×

1.65

2.1

1.6

2.05

1.55

1.45

1.95

1.4

1.9

100

200

300

400

500

600

700

800

900

1000×

100

200

300

400

500

009

00

008

006

1000 X

(a) RandPos Training Loss

(b) RandPos Validation Loss

(c) PoSE Training Loss

(d) PoSE Validation Loss

(e)

CREAM

Training Loss

(f)

CREAM

Validation Loss

.45

1.35

.35

006

100

UL

600

700

1000 X

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work performed by us.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theoretical result, we provide the full set of assumptions and a complete (and correct) proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: All our experiments are seeded, the results are unique, there are no error conditions, and therefore they are not needed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For each experiment, we provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no potential positive or negative social impacts involved in our work, so there is no need for that.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We use all open source datasets and models, and do not involve data or models with a higher risk of abuse, so there is no need to do so.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators and original owners of assets (e.g., code, data, models), used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We're not introducing new assets, so they're not needed.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't have crowdsourced experiments and research with human subjects, so we don't need to.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We don't involve any subjects, there are no risks, so there's no need for it.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CREAM (Continuity-Relativity indExing with gAussian Middle), an efficient method for extending the context window of large language models (LLMs) to handle longer contexts without the need for extensive fine-tuning. CREAM manipulates position indices for shorter sequences within the pre-trained context window, using two key strategies: continuity, and relativity. Additionally, CREAM employs a truncated Gaussian distribution to enhance the model's focus on the middle part of the context, mitigating the \"Lost-in-the-Middle\" problem. Comprehensive experiments demonstrate CREAM's efficiency and effectiveness, showing improved performance over baselines like RandPos and PoSE on various tasks.\n\nSTRENGTHS:\nCREAM achieves superior performance across a spectrum of long-context tasks. The empirical results substantiate CREAM's ability to effectively mitigate the \"Lost-in-the-Middle\" problem.\n\nWEAKNESSES:\nI think the presentation of this paper is a significant problem. Actually, I cannot understand the method itself nor the intuitions behind it. First, I do not understand the concept of continuity in positional encoding. The authors use a short paragraph to explain continuity (lines 69-74) without any formulation, which is very abstract. The same goes for relativity, though I understood it from another paper. After explaining these two concepts, you still need to explain why they are important (I did not see this in the paper). Second, I do not understand why splitting the pre-trained context into three segments with different sizes can achieve continuity and relativity. Third, I do not understand why truncated Gaussian middle sampling mitigates the \"Lost-in-the-Middle\" problem. Line 113 states that it reduces the interval overlap in Eq2. But what is interval overlap and why does it result in the \"Lost-in-the-Middle\" problem?\n\nI may not be the ideal reader for this paper, but if other reviewers feel the same way, this paper may have been hastily completed.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents CREAM (Continuity-Relativity indExing with gAussian Middle), an innovative approach to extend the context window of Large Language Models (LLMs) without the need for extensive fine-tuning at the target length. The authors address the \"Lost in the Middle\" problem, which plagues long-context LLMs by causing a performance drop when retrieving information from the middle of the context. CREAM achieves this by manipulating position indices and introducing a truncated Gaussian sampling method to focus on the middle part of the context during fine-tuning.\n\nSTRENGTHS:\n1. **Novel Approach**: CREAM offers a novel positional encoding strategy that efficiently extends the context window of LLMs, which is a significant contribution to the field.\n   \n2. **Empirical Evidence**: The paper provides strong empirical evidence through comprehensive experiments, demonstrating CREAM's superiority over existing methods like PoSE and RandPos, especially in handling middle context information.\n\n3. **Training Efficiency**: The method requires fine-tuning at the pre-trained context window size, which is computationally efficient compared to fine-tuning at the target length.\n\n4. **Theoretical Foundation**: The paper includes theoretical justifications for the use of truncated Gaussian distribution, adding rigor to the proposed method.\n\nWEAKNESSES:\n1. **Generalizability**: While the paper shows impressive results, it is not clear how generalizable these findings are to other LLMs beyond the tested Llama2-7B model. And I'm wondering whether the method can work with PEFT tuning techniques, such as QLoRA. \n\n2. **Lack of Comparative Analysis**: The paper could benefit from a more detailed comparative analysis with other contemporary methods. For example, in Table 2, the paper only compares one model (LongChat-v1.5-7B-32k) with the same 32k context window. I suggest that in such comparision, peer models should at least take the same length of input, otherwise, it cannot prove the effectiveness of the proposed model over other Long LLMs with 32k+ context windows. I suggest the author compare with models like Mistral-7B-Instruct-v0.2, LongLoRA etc.\n\nQUESTIONS:\n1. What are the computational overheads associated with implementing CREAM, and how does it scale with larger context sizes?\n2. Are there any specific hyperparameter settings in CREAM that are particularly sensitive, and how were these chosen?\n\nLIMITATIONS:\nThe discussion in Appendix F regarding the limitations of the CREAM method could be misleading. The author implies that CREAM provides enhanced performance compared to other methods concerning the \"Lost in the Middle\" issue but acknowledges that the problem is not fully solved due to the nature of decoder-only models. For a genuine limitations section, it is essential to delve deeper into the specific constraints of the CREAM approach and how these limitations might impact its application and effectiveness. A more explicit acknowledgment of the method's shortcomings and a clear explanation of why these issues cannot be overcome with the current model design would significantly improve the section's clarity and usefulness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method, CREAM, that better enables extrapolation to longer contexts via finetuning at the base context length. The method uses positional embedding interpolation with the embeddings divided into three areas of interest and uses a truncated Gaussian  to sample the position used for the middle segment, to increase training of the middle context. They show that this is at least as effective as prior length extrapolation methods that use finetuning at the base context length, and that CREAM-trained models suffer less from the lost in the middle problem.\n\nSTRENGTHS:\nS1. The method shows clear improvement on the targeted issue (the lost in the middle phenomenon)-- e.g. in Table 1. \n\nS2. The method is well-motivated both intuitively and theoretically, and the explanation in section 2.2 is well-written and easy to follow. \n\nS3. The authors evaluate against the two most reasonable baselines (to my knowledge) and across a good selection of synthetic, long-context, and short-context tasks. (It would be an added benefit to evaluate against finetuning with position interpolation and *long* context at finetuning-time, to see the efficiency-performance tradeoff; however, I do not think this is strictly necessary to the paper's claims.)\n\nWEAKNESSES:\nW1. The method is fairly similar to POSE in concept and performance. While I think the idea of emphasizing middle training is reasonable, well-explored, and clearly effective (e.g. in Figure 1), this has also been explored as a post-training or data selection correction in other work. Given the slight performance degradation from POSE on short-context tasks (i.e. in Table 4), I'm concerned that this might not be worth the tradeoff practically to do at training time. \n\nW2. Evaluation settings can differ subtly across papers, and so it would be better for the authors to reproduce baseline results rather than citing them from literature, where possible. This is most critical, I think, in Table 4, where the differences between methods are relatively small and all results are compared to the same baseline method, which is not an infeasibly expensive or unavailable model to run.\n\nQUESTIONS:\nQ1. In Table 5: why does the 256k-extrapolated version of CREAM underperform the 128k-extrapolated version on short contexts?\n\nQ2. Say you had an even more limited amount of compute, to the point where finetuning on the base context length was prohibitively expensive (e.g., maybe you want to finetuned Llama3-8b but can't fit 8192 context at training time). Do you believe CREAM would be useful in this setting? Do you have any results (or speculation!) that suggest when the method may begin to break down with decreasing maximum training length?\n\nQ3. (Minor) What is the context length used in Figure 1? The x axis only reports the position of keys.\n\nSuggestions/comments:\n*  Please explain in the text what the occasional highlighting of numbers represents (e.g. in Table 2). It would be helpful for readability to bold the best number in each column across tables.\n\nLIMITATIONS:\nYes, no concerns here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for extending the context window of pretrain large language models. The approach, CREAM, relies on modifying position indices to interpolate the positional encodings. Despite the often computationally expensive nature of such work, CREAM can extend to very long context windows while only needing to train at the original pretrained context window. Additionally, in their approach to the context length problem, the authors propose a solution that focuses explicitly on learning at the middle of the context — a span that often under-performs in long context models. The experimental results cover a wide variety of problems, both in terms of tasks, as well as context length challenges. Overall the results look very promising, and show a strong method for addressing a challenging task of extending the context length of LLMs.\n\nSTRENGTHS:\n- The authors tackle two problems that go hand in hand, namely extending the context length during fine-tuning - but doing so in a way ensures consistent performance. \n- The approach divides the desired context length into three segments, which then results in relative positional distances that vary and consequently learning all relative positions within the target length $L$. The technique is simple, but clever and an effective way to efficiently expose the model to a broader range of relative positional distances during training.\n- The results are very strong, particularly with the performances listed on Long Bench, which encompasses a broad range of tasks.\n\nWEAKNESSES:\n- No error bars shown on results. In most cases the results are quite strong, but the error bars would be helpful — particularly in some of the closer comparisons with PoSE.\n- The solution of using a truncated Gaussian approach lacks motivation. The \"lost in the middle\" problem is clear, however the solution of using a truncated Gaussian to force more focus on the middle seems brittle. Could the parameters of the truncated Gaussian be learned from data? The approach works well based on the results shown in Figure 5a, however.\n\nQUESTIONS:\nThe authors mention that CREAM only needed to be trained for 100 in some cases. Can the authors provide more insight on why the results are very good despite so little training? This seems like a significant achievement, but was not discussed in detail.\n\nLIMITATIONS:\nThe amount of focus on the middle context seems fixed in this approach. While the results for Gaussian truncation appear promising, it's not clear if this direct approach to solving the \"lost in the middle\" problem is universal. In short, how do we know how much to reweight the importance of the middle context, and does it vary by dataset or task?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}