An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
Authors: Tong Wu
Yanpeng Zhao
Zilong Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Long Context
Position Encoding
Primary area: natural_language_processing

Abstract: Recently, many methods have been developed to extend the context length of pre-trained large language models (LLMs), but they often require fine-tuning at the target length ($\gg4K$) and struggle to effectively utilize information from the middle part of the context. To address these issues, we propose $\textbf{C}$ontinuity-$\textbf{R}$elativity ind$\textbf{E}$xing with g$\textbf{A}$ussian $\textbf{M}$iddle ($\texttt{CREAM}$), which interpolates positional encodings by manipulating position indices. Apart from being simple, $\texttt{CREAM}$ is training-efficient: it only requires fine-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs to a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, we introduce a truncated Gaussian to encourage sampling from the middle part of the context during fine-tuning, thus alleviating the ''Lost-in-the-Middle'' problem faced by long-context LLMs. Experimental results show that $\texttt{CREAM}$ successfully extends LLMs to the target length for both Base and Chat versions of $\texttt{Llama2-7B}$ with ``Never Miss A Beat''. Our code is publicly available at https://github.com/bigai-nlco/cream.

An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding
Tong Wu aatongl@bigai.. ai
Yanpeng Zhao zhaoyanpong@bigai. ai
Zilong Zheng [
zlzheng@bigai. ai
State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
t Corresponding author.
Abstract
Recently, many methods have been developed extend the context length pre-trained large language models (LLMs),, but they often require fine-tuning at the target length (> 4K) and struggle to effectively utilize information from the middle part of the context. To address these issues, propose Continuity- Relativity indExing with gAussian Middle (CREAM), which interpolates positional encodings by manipulating position indices. Apart from being simple, CREAM is training-efficien it only requires mee-tuning at the pre-trained context window (e.g., Llama 2-4K) and can extend LLMs a much longer target context length (e.g., 256K). To ensure that the model focuses more on the information in the middle, introduce truncated Gaussian encourage sampling from the middle part of the context during fine-tuming, thus alleviating the "Lost-in-the -tid-ll--h--Md problem faced by long-context LLMs. Experimental results show that CREAM successfully extends LLMs to the target length for both Base and Chat versions of Llama2-7B with "Never Miss Beat". Our code is publicly available https:/ githhu..c.. cww/bigai nlco/ cream.
100
PoSE
Accuracy
80
60
-
PoSE
20
1th
19th
38th
55th
75th
1th
19th
38th
55th
75th
of Key
Key Retrieve
(a) Linear Interpolation
(b) YaRN Interpolation
Figure 1: Results of applying different position interpolation methods to the "Lost-in boosstinntth---e the-Middle" task CREAM and PoSE [Zhu al., 2023]. We can that CREAM outperforms PoSE [Zhu al., 2023] at every position, with a particularly improvement in the middle.
Introduction
Transformer-basee Large Language Models (LLMs) are typically pre-trained with a fixed context window size, e,g., 4K tokens Touvron et al. [2023a]. However, many downstream applications, including in-context learning [Huang et al., 2023, Li et al., 2023a] and LLM agents [Qian et al., 2023, Zheng al., 2023] necessitate the processing of significantly longer contexts. e.g., up to 256K tokens. Recent works have proposed promising approaches that efficiently extend the context window of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

pre-trained LLMs by interpolating Positional Encodings (PEs) [Chen et al., 2023, Peng and Quesnelle, 2023, Peng al., 2023, Xiong et al., 2023, Zhang et al.. 2024] with a short period of fine-tuning Unlike other techniques such as efficient transformer [Twoorkoski et al., 2024, Munkhdalai et al., 2024] and memory augmentation [TTn al., 2024], PE--hased methods do not necessitate alterations to the model's s architecture or the incorporation of supplementary modules. Consequently, PE-based methods offer the advantages of straightforward implementation and rapid adaptation, making them practical solution for extending the operational range of LLMs in tasks involving larger context windows.
Despite the simplicity and effectiveness, existing PE- methods exhibit two significant limitations. First, prior approaches. such as positional interpolation [Chen et al., 2023], still require fine-tuning on the target context window size, which imposes a substantial computational overhead [Zhu et al., 2023]. Secondly, though some PE methods have demonstrated potential in handling extremely long sequences, as evidenced by low sliding window perplexity scores, their performance deteriorates notably in "in-the-middle" scenarios [Liu al., 2024]. Specifically, when the model is required to accurately retrieve and process content located in the middle of an extended context, there is marked drop in performance on the extended window size (Figure and Figure 3).
These observations and insights underscore a fundamental question: Can we extend the context window size of pre-trained LLMs efficiently while simultaneously optimizing their effectiveness in processing "in-the-middle" content?
To answer the above question, we propose CREAM, namely Continuity- Relatiiviy indExing with gAussian Middle. CREAM a novel PE-based fine-tuning recipe that shows both efficiency in fine-tuning and effectiveness in enhanced middle content understanding. Our key insights lie in manipulating the positional indices of long target sequences to produce shorter ones within the pre-trained context window size (Figure 2).
In Section 2.1, we summarize two crucial ingredients of effective positional indices: continuity that produces densely connected positional indices and relativity that reveals the long-range dependencies between fragments. CREAM is a recipe designed with the best of both worlds by introducing two indexing strategies for continuity and relativity, respectively (Section 2.2). Besides, to alleviate the Lost-in-the-Middlee challenge, we introduce truncated Gaussian distribution for middle segment sam- pling, enabling the LLM to prioritize the information in the middle positions, when performing positional interpolation within the pre-trained context window size.
In Section 3,
we conduct comprehensive experiments to demonstrate the efficiency and effectiveness of CREAM. We continually pre-trained on Llama 2-7B with CREAM for a short period and extend the context window size from 4K to up to 256K. Furthermore, we instruction tuning on Llama 2-7B-Chat with CREAM for 100 steps and obtain promising results. We highlight our empirical advantages
1. CREAM can not only fine-tune within the pre-training context window size, but also alleviate the issue of the model easily getting lost in the middle. egg., CREAM YaRN outperforms PoSE- YaRN [Zhu et al., 2023] by over 20% on average in the "Lost in the Middle" [Liu et al., 2024] task. positional interpolation frequen-
2. CREAM can further be enhanced by integrating novel designs cies (such as Linear [Chen et al., 2023], NTK [Peng and Quesnelle, 2023], Yarn [Peng et al., 2023], etc.), and can be extended to context window sizes of up to 256K or beyond.
3. CREAM-Chai model requires only 100 steps of instruction-tuning to achieve nearly perfect performance the Needle-in-a-Haye Haystack pressure test, and it outperforms existing strong baselines on LongBench [Bai et al., 2023].
2 Methodology
2.1 Preliminaries
Problem Formulation. Given an LLM with pre-trained context window size N, our goal is to unlock the inference capacity of the LLM on the testing data Drest. with an extended context window size L (where I. > N) by efficiently learning from small-scale training data Dirain with a maximum sequence length N. We expect the extended model to perform reasonably well long-context evaluation.

---

Full Extended Context Window
L-2
L-1
Fine-tuning Position Indices:
k- Head
Truncated Gaussian
k k
Continuity
absolute position
Middle
Tail
<<
N/3
N/3
8192
8193 ...
32K
Relativity
Head
Middle
P.
Tail
LA 4= N/3
P Pretrained Context Window
relative position
Figure 2: Illustration of CREAM position interpolation. The pre-trained context window is divided
into three segments: the head, middle, and tail. To ensure continuity, we fix the lengths of the head and tail to a small value k. To maintain relativity, we the lengths of the head and tail NV/3. For the middle part, the start and end position indices are determined via truncated Gaussian sampling, thereby encouraging the model pay more attention to the information in the middle part.
Continuity in Positional Encoding. Trrnsformer- language models typically encode posi- tional indices sequentially as {0, ,, N.NN-1 1}. Traditional length extension methods [Chen et al., 2023, Peng and Quesnelle, 2023, Peng et al., 2023] directly fine-tune on the target length L with an updated positional index. This approach preserves the continuity of all absolute positions and learns all position indices within [0, L-11, 1].
thereby successfully extending the target length, Furthermore, PoSE [Zhu et al., 2023] attributed their superior performance over RandPos [Ruoss et al., 2023] to the ensured continuity of segments during fine-tuning.
Relativity in Positional Encoding. Relative positional encoding (RPE) [Shaw et al., 2018] has been proposed as effective positional encoding method, where only the relative positions between two tokens are considered. Similar to prior works [Ruoss et al., 2023, Zhu et al., 2023, Wu et al., 2024], work focuses on rotary positional encoding (RoPE) [Su et al., 2024], which is one of the most prominent RPE methods and has been widely applied to LLMs including the recent Llama family [Touvron al., 2023b,a, AI@ Meta, 2024]. In ROPE, only the relative distances between position pairs (lj - il; 0 < < < L- 1) are learned during fine-tuning (Appendix A). Due to this property, we can manipulate the position indices such that all relative positions between [0, L 1] are learnable within the pre- pre-trained window size.
2.2 Proposed Recipe:
indExing with gAussian Middle (CREAM)
In the following section, start by introducing design of dividing the context window N to learn relative positional information. Then, we propose two strategies that target continuity and relativity, respectively. Lastly, propose novel truncated Gaussian sampling method to enhance the middle part of the long context. The overall framework is depicted in Figure
Context division. We first discuss the motivations behind our design of the context length. First, prior works [Han et al., 2023, Xiao et al., 2023] observed that significant amount of attention score is allocated to the beginning tokens of a sequence, which can potentially encode absolute positional information even without explicit positional encoding [Kazemnejad et al., 2024]. Secondly, the starting and ending tokens of long contexts can be treated as two pointers that localize the middle indices with the help of relative encodings. Therefore, we divide the pre-trained context window into three segments. The detailed ablation results are shown in Section 3.6.
Definition 2.1. Given the pre-trained context window size N and target extended length L, the position set of (Head, Middle, Tail} is defined as follows:
Head (0, 1,.,, La 1}.
Middle = (P,, P. + 1, ...., P. - 1, Pc)
(I)
1), Tail (L Lr..., L - 1-2,11 1-L-1 1),
s.t. Ln + (Pe P.) I N,
where T. and T. denote the length of the head and tail segments, P. and P. denote the start and end position index of the middle segment.
3

---

The relative positions among the three segments in each sample are calculated in pairs, i.e., {i Vi, (Head, Middle, ].
The formed relative distance union D, learned by the model is given by:
[0, max P. P, L - [),
P.]U[L- L, Pe, L- PoJU[L- +1, (2)
Given that not all samples possess the same values for Ln. P.. Pe, and L,, as fine-tuning progresses, the union D, in Equation (2) can encompass the entire range [0, I. 1]. facilitating the model to learn all relative positions within the target length T..
Two segmentation strategies. For the sake of continuity, we set the LA and a very small value k, where 0 < k < N. Specifically, we use A = 32 in our experiments. This choice allows the middle segment to closely approximate the pre-trained context window. To maintain relativity, we divide N equally into three parts and fix the L, and Le to N/3, enabling the model to learn as many relative positions as possible. In our ine-tuning process, both types of examples are sampled with equal probability to maintain balance.
Truncated Gaussian Middle Sampling To better focus the training process the middle part of the long context, we introduce a truncated Gaussian function. This approach reduces the interval overlap in Equation (2) and directs the model's attention toward the middle section of the long context. In Appendix B, we provide theoretical justifications of our truncated Gaussian design, indicating that the maximization of [D,| holds for middle positions [AN, L/2) (L/2, L - .).
Formally, given the probability density function (PDF) of a Gaussian distribution:
[xr (-
20² (-(r-1)) (1)3
f(x)
exp
where j the mean and the standard deviation. The corresponding cumulative distribution function (CDF) is:
F(x) - f f(t) 0.5 (1+ (++x((r))))
f de,
E(z)
(3)
V J
where FE-) is the error function. To calculate the CDF value within the truncated interval, use sufficiently large number (e.g. 1000) of equally spaced x values from the given interval [1, L/N]:
(1 (L/N)) ( 1i
1,2, ... 1000,
(4)
999
By substituting Equation (4) into Equation (3), the cumulative distribution function (CDF) curve is derived within the truncated interval. For sampling from this truncated Gaussian distribution, the inverse transform method is employed, as demonstrated in Equation (5):
round [1-1 (2, Ft (u (x)-1)))
(5)
F(x:) F(
where u ~ Uniform(O, 1), round(.) represents rounding to the nearest integer. Finally, we can get: P. ~ Uniform(Ly, a x Lmm, (a 1) It),
(6)
P. P. Lm + 1,
where I. denotes the length of the middle segments. summary, the overall sampling flow of our algorithm is presented in Algorithm 1.
Experiments
3.1
Experimental Setup
Extended Models We use -lama-2-7B and Llama-2-11 1-ama-2-7B-Cha CB-tt [Touvron et al., 2023a] as the
base models and extend their pre-trained context window size of 4K to a target context length of 32K. The extended models are referred to as CREAM-Bast and CREAM-Chat. respectively. Note that, though the target context length 32K, we do not have to fine-tune CREAM on 32K token long text (see Section 2.2).

---

Algorithm/ 1 CREAM sampling algorithm
Require: Pre-trained context window size N, extended context window size L, training sample size S, mean JA,
variance a and hyperparameter k.
1: Generate enough equally spaced according to Equation (4).
2: Substitute into Equation (3) derive the truncated Gaussian CDF F(x).
3: for i to S do
4: Sample La ~ DiscreteUniifomm(( {k, N/33), and let L. Li, Li = N - Li. Lt.
5: Sample u ~ Uniform(0, (0, 1). and substitute it into Equation (5) to get a.
6: Calculate the start and end position ids P.. Pe of the middle part according to Equation (6).
7: Get position set P. {0, 1...., The Pa....; Pe, I. L Ir...,, 1}, where |P| N.
end for
9: return P = {Po, Pi...., Ps-2 28-2, Ps-1)-
Benchmarks We conduct long-contex: LLM evaluation of CREAM-Baa on LongChat-Lines [Pal et al., 2023] and Lost-in-the-Miide thd-Middle [Liu et al., 2024]. Idcally, fine-tuning should not disrupt what the base model has learned, so we further evaluate CREAM-Base on the language modeling task and the evaluation benchmark [Beeching al., 2023] adopted by Llama2. Additionally, we assess the CREAM-Chat model with Needle-in-a-Haystaee
and LongBench(Bal [Bai et al., 20233. Unless otherwise specified, use linear interpolation to adapt LLMs longer context length.
Baselines As far as we know, RandPos [Ruoss et al., 2023] and PoSE [Zhu et al., 2023] are similar to our approach in that they manipulate position indices to enable fine-tuning on the pre-trained length for context expansion. Therefore, these two methods serve as the baselines for our primary comparisons, More details about the experimental setup can be found in the Appendix C.
3.2 Effective Context Window Size Evaluation CREAM-Base
We evaluate the long-context understanding capabilities of the CREAM-Base model on two tasks: LongChat-Lines [Pal et al., 2023] (Figure 3) and "Lost in the Middle" [Liu et al., 2024] (Table 1).
Drigial
Fansthon
Fuse (Av 54.21
Cus Heg 63.50
420
02
42-
2500
3000
<250
3800
6000
7100
6000
11850
30050
16000
17535
20003
22003
24000
24000
33000
30000
Tooer Length
Figure 3: Results (%) on LongChat-Lines. Each length consists of 50 samples. All results are fine- tuned on Llama-2-77 with 4K length data through linear position interpolation. Refer to Appendix E for ablated results using NTK [Peng and Quesnelle, 2023] and Yarn [Peng et al., 2023].
CREAM-Base performs best in retrieving information from long contexts of varying lengths. We extend the context window size up to 32K and compare CREAM with the Llama 2-7B [Touvron et al., 2023a], RandPos [Ruoss et al., 2023], and PoSE [Zhu et al., 2023]. As the context window size increases, the performance of all models drops, but CREAM always performs best except for the window size of 3.6K (see Figure 3). In terms of the average performance over all context window sizes, CREAM outperforms PoSE by 16%, demonstrating its good long-conted understanding ability.
CREAM-Base alleviates the Lost-in-the-Middlee issue. Lost-in-the-MMe Middle an observation that LLMS are generally good at retrieving relevant information appearing at the beginning/end of the input context [Liu et al., 2024]. To validate the effectiveness of our middle-focused truncated Gaussian
hhttps: h/tp:::///ttain thub. com/ tttpg/aaaaa/////ee /atttaatttLLLeessttee LLtM/estMMtetteeeeeeeeeeeets t_eesd__eeeeeeeeeeeeeee enAlfaystack
e Passkeyy retrieval [Mohtashami and Jaggi, 2023] is another similar task for evaluating long conteconext LLMs, but it is too simplistic to reflect model performance at different context window sizes, so we use the dataset provided by Pal et al. [2023]. which closely aligns with the task described in Li et al. [2023b]
5

---

Table Results (%) on "Lost in the Middle". "Position" indicates the correct answers' index, and each index comprises 500 samples. All results are fine-tuned on Llama- 2-BB with 4K length data.
Position (75 keys, 5SK tokens)
Position (140 keys, 100 tokens)
Model
AVG
AVG
18
37
54
74
0
34
69
104
139
PoSF--Linear
99.4
24.4
37.4
47.2
46.2
50.9
95.2
8.2
7.6
13.8
18.6
28.7
BREAM-Liness
99.6
45.6
56.0
67.0
58.0
65.2
96.6
19.8
23.4
31.0
26.2
39,4
PoSE-NTK
98.6
49.6
44.6
40.2
41.4
54.9
97.6
3.4
0
27.6
25.7
CREAX-NTK
96.2
53.8
52.6
72.8
42.0
63.5
78.6
6.0
23.4
41.8
29.9
PoSE-YaRN
99.6
32.6
12.2
57.2
48.4
50.0
91.8
0.6
2.8
8.2
18.8
24.4
CREAK-YaRN
100.0
49.6
47.6
77.4
92.6
73.4
99.4
8.0
5.8
43.8
69.2
45.2
sampling, evaluate CREAM and compare it with PoSE the key-value retrieval task proposed by Liu et al. [2024]. We present results Table 1, where the cyan shading indicates middle segments. We find that: regardless of the chosen interpolation method, CREAM always outperforms PoSE by large margin. e.g., CREAM-L surpasses PoSE-Lineat by 21 21.2% when the relevant information is placed at 18.
3.3 Long Context Understanding Evaluation on CREAM-Chat
conduct long-context evaluations of CREAM-Chat on two tasks:
Needle in A Haystack (Figure 10) This task is test that places an answer (i.e., Needle) at any position of a long context window (i.e,, Haystack) and requires model to retrieve the correct answer given rrestion-answer pair. We follow Wu et al. [2024] and use the GPT (GPT-3. 5-Turbo-001 0125)) score as the evaluation metric.
LongBench (Table 2) Bai et al. [2023] is more realistic benchmark because it covers application scenarios like long-context QA and summarization. Moreover, specifically designed for Chat models.
6%
0%
20%
30%
40%
40%
gpt score
60%
80%
90%
14
8k
11k
13k
21k
20k
31k
11
12k
15k
18k
22k
25k
29k
32k
(a) SkipAlign
(b) CREAM
Figure 4: Results on Needle-in-a-Haystee Hayystack. indicates the results excerpted from Wu et al. [2024].
Both results are instruction-tume on LLaMa2-7BB Miiiaa--B--ha with 4K length data. The color gradually changes from deep green deep red, indicating the Recall performance decreases from 10
CREAM-Chat outperforms SkipAlign context window expansion. We visualize the results of CREAM-Chat and the recent SkipAlign in Figure 10. Clearly, CREAM-Chat beats SkipAlign because the performance of SkipAlign [Wu et al., 2024] decreases from the window size of 18K while CREAM-Chat displays perfect performance everywhere until from the window size of 29K. Notably, CREAM-Chat is only fine-tuned for 100 steps.
CREAM-Chat makes best use of the extended context window size. We present results Long- Bench Table CREAM-Chal again surpasses strong baseline models, demonstrating better use of extended context size. In terms of the average performance over all tasks, it outperforms the second

---

Table 2: Results (%) on LongBench. iidiicates results reported by Bai et al. [2023]. CREAM-7B-324
is instruction-tumed for 100 steps using 4K length data on LLaMa2-78 SatMMa2-7B-Chae Chat.
Single- Doc QA
Multi- Doc QA
Summari- zation
Few-shot
Code Completion
Synthetic Tasks
Model
AVG
Learning
24.9
22.6
24.7
60.0
48.1
31.0
XGen-7B-8k
24.6
20.4
24.7
56.2
38.6
28.3
Mistral-7B- -BInnttruct--OO.
29.5
20.7
26.4
13.6
29.6
21.8
Dostral-BBInnsruut--o
28.5
21.5
26.1
50.1
33.8
13.9
29.0
Distra-7B-Insttucr-ae
33.2
30.6
26.8
56.4
15.3
10.4
28.8
Internl Internl.M-7B-88 M-7788k"
17.4
20.2
16.1
50.3
36.4
24.2
Vicuna-vI -5-7B-16k"
28.0
18.6
26.0
66.2
47.3
5.5
31,9
LongChat-v1.5-78
28.7
20.6
26.7
60.0
54.1
15.8
34.3
CREAM-7B-32K
34.8
31.1
27.2
65.1
50.4
7.0
35.9
al., 2023b], by 1.6%, though only tuned on very small amount of data and for only 100 steps.
best model, i.e., LongChat-vl 55-7B-32k
[Li ----a----11--7-34
3.4 Effectiveness of PEFT Integration
To demonstrate that CREAM can be directly combined with PEFT techniques (such as LoRA [Hu et al., 2022] and QLoRA [Dettmers al., 2023]), requiring additional modifications. We conducted experiments on LLaMa-22 2aaa--2----haa using the identical dataset and settings. The experimental results presented in Table 3. The results indicate that models fine-tuned using LoRA and QLoRA achieve performance nearly equivalent to those fine-tuned with full parameter.
Table 3: Results (%) LongBench. indicates results reported by Bai et al. [2023]. CREAM-7B-324 instruction-tumed for 100 steps using 4K length data on LLaMa2- Ma-Maa--B-Chat
Single- Doe QA
Multi- Doe QA
Summari- zation
Few-shot Learning
Code Completion
Synthetic Tasks
Model
Macro
Llama2-7B*
24.9
22.6
24.7
60.0
48.1
5.9
31.0
LoRA
28.9
28.6
27.8
62.2
54.6
10.8
35.5
QLoRA
28.1
27.6
28.1
61.7
54.6
10.1
35.0
CREAM-7B-32k
34.8
31.1
27.2
65.1
50.4
7.0
35.9
3.5 Language Modeling and Standard Benchmark
Following Chen et al. [2023], Zhu et al. [2023], Peng et al. [2023], we perform the classic language modeling evaluation, Le., perplexity evaluation, GovReport [Huang et al., 2021] and Proof- pile [Zhangir Azerbayev, 2022]. Since lower perplexity does not necessarily imply better model performance downstream tasks [Zhang et al., 2024, Hu et al., 2024, Arora et al., 2024, Park et al., 2024], we further conduct evaluation on the standard saauraa-anngagg-nddestatadd benchmark [Beeching al., 2023]. This also lets know whether fine-tuning hurts the NLU ability of the pre-trained base model.
(NLU)
Both CREAM and PoSE demonstrate the lowest perplexity. We apply different positional inter- polation methods to RandPos [Ruoss et al., 2023], PoSE [Zhu et al., 2023], and CREAM and report their perplexities in Table 4. We find that: CREAM and PoSE have similar perplexity in different settings and both outperform RandPos. This occurs primarily because the position indices used during RandPos fine-tuning
discontinuous, which creates inconsistency with the pre-training stage.
CREAM has nearly the same NLU abilities as the pre-trained base model. Ideally, fine-tuning should not adversely affect the original capabilities of the pre-trained base model. Our evaluation of CREAM confirms this, i.e, CREAM nearly retains all NLU abilities of the base Llama2-7B (see Table 5). Interestingly, CREAM improves over Llama2-7B on ARC-C and HellaSwag. This is because

---

Table 4: Perplexity results of GovReport and Proof-pile. Each experiment is the average perplexity of samples, and all results are based on LLaMa2-7B fine-tuned on 4K data length.
GovReport
Proof-pile
Model
4K
8K
16K
32K
4K
16K
32K
Original
3.6
4.6
RandPos-Linear
8.9
7.4
6.2
5.8
12.1
11.9
11.9
12.9
PoSE-Linear
3.8
3.2
2.5
4.7
4.6
4.4
CREAM-I.inea
3.8
3.2
2.7
2.5
4.7
4.6
4.5
4.4
RandPos-NTK
4.6
4.0
4.0
5.8
5.8
7.3
PoSE-NTK
3.7
3.2
2.7
2.6
4.7
4.6
4.7
CREAM-NTK
3.8
3.2
2.7
2.7
4.7
4.6
4.5
4.7
RandPos- YaRN
5.0
4.4
4.0
4.6
6,4
6.5
6.8
9.1
PoSE-Yate
3.7
3.2
2.7
2.5
4,6
4,6
4.5
4.4
CREAM-YaRN
3.7
3.2
2.7
2.5
4.6
4.6
4.5
4.4
Table 5: Experimental results standard benchmarks. indicates results cited from Touvron et al. [2023a], and all results are based on LLaMa2-7B fine-tuned on 4K data length.
Zero-Shot
Few-Shot
Model
WinoGrande
TruthfulQA(me2))
PIQA
BoolQ
ARC-C
HellaSwag
LLaMa-2-7b-hr"
69.2
39.5
78.8
77.4
45.9
77.2
RandPos-Liness
63.3
39.3
76.5
66.6
32.0
48.5
PoSE-Linear
68.8
38.6
77.8
76.2
47.7
77.1
CREAM-Linear
67.5
37.4
78.5
75.4
46.8
76.9
RandPos-NTK
68.7
35.9
78.6
74.8
45.5
74.4
PoSE-NTK
68.8
38.6
77.8
76.2
47.7
77.1
CREAM-NTK
67.5
37.4
78.5
75.4
46.8
76.9
RandPos- MaRN
69.3
36.6
78.3
72.5
43.4
69.2
YaRN
69.4
39.6
78.1
76.7
49.0
78.0
CREAM- YaRN
68.7
38.5
78.0
76.4
49.0
78.0
these two tasks are few-shot tasks with longer prompts, necessitating the assistance of long-context understanding.
Extending the context length to 256K. We push the limit and extend the context length of Ll 1lama-2-7B up 256K. Following Zhu al. [2023], evaluate the extended model by calculating the average perplexity over 20 samples from PG-19 [Rae et al., 2019] and Book3 [Presser, 2020]." Since the PG-19 test set does have enough samples that are longer than 256K, select subset of samples from the PG-19 training set.
We experiment with target context lengths 64K, 96K, 128K, 192K, and 256K and apply different positional interpolation methods to extended model (see Table 6). The results of PoSE [Zhu et al., 2023] in Table are based on fine-tuning LLaMa 1-7B with 2K data length, and are provided for reference only. Surprisingly, the increase the target context length brings little to no perplexity increase, demonstrating the stability of CREAM across different target context lengths, even when the target context is extremely long.
3.6 Ablation Study
To validate the effectiveness of our modeling choices, we further conduct an ablation study of three main components of CREAM: truncated Gaussian sampling, fixed start and end segments, and the trade-off between continuity and relativity.
Truncated Gaussian sampling versus Uniform sampling. We truncated Gaussian sampling to encourage CREAM to make better use of the middle part of the context. As a comparison, we replace
We use sliding window for calculation, with window size of 32,768 and sliding step size of 4,096.

---

Table Perplexity results of PG-19 and Book3. indicates results copied from Zhu al. [2023].
and CREAM is based on LLaMa2- 7B fine-tuned on 4K data length.
PG-19
Book3
Model
64K
96K
128K
192K
256K
64K
96K
128K
192K
256K
PoSF:-Linaat 128K*
22.47
26.77
31.18
43.62
57.08
70.87
PoSE-NTK-1280 --2KK
14.84
29.48
34.80
16.04
31.42
37.00
PoSE- YaRN-128K'
10.36
10.77
11.33
12.30
13.07
13.81
CREAM-Lincar- 192K
5.9
6.0
6.1
6.1
7.6
7.7
7.8
7.8
CREAM-NTK-1920 192K
5.0
5.1
5.2
5.2
6.9
7.0
7.0
7.1
CREAM- MEEM-YYaRN119K
5.0
5.2
5.2
5.3
7.0
7.1
7.1
7.1
CREAM-Linear-256K
7.8
8.0
8.0
8.1
8.2
10.2
10.3
10.5
10.7
10.8
CREAM-NTK-256K
5.1
5.3
5.3
5,4
5.4
7.2
7.3
7.3
7.3
7.4
CREAM- YaRN-256K
5.2
5.3
5.4
5.5
7.1
7.2
7.3
7.3
122
Headd Haad Tallmaa s0gge44
Cusssian Magg:5
Hexd Ma
50-
Onty (lagd2.7)
tread Megptt
-X- Rendom (Auge2.0
40-
30
20
20-
216
Tharr Length
Tocan Langen
Tokoa Longe?
Gaussian vs. Uniform.
(b) Head vs. Tail.
(c) Relativity vs. Continuity.
Figure 5: Ablation study of CREAM on LongChat-Lines. The result at each length is estimated using 50 samples.
with the Uniform sampling (see Figure 5(a)). We observe that the Uniform sampling always leads worse retrieval performance, suggesting the effectiveness of the truncated Gaussian sampling.
Fixing the head and tail segments crucial for good retrieval performance. We compare our choice of fixing the head and tail segments with three alternatives: (i) removing both the head and tail segment, (ii) fixing only the head segment, and (ii) fixing only the tail segment (see Figure 5(b)). We find that: removing the head and tail segments leads to the worst performance; it results in complete failure (ie., zero score) for the context size 32K. Keeping either head or tail segments performs slightly better than removing both but underperforms our default choice of fixing both. We suppose that this is because fixing both gives rise to better relativity information, a finding that consistent with that of Han et al. [2023].
Maintaining good balance between continuity and relativity is necessary. We encourage continuity by setting the head and tail segment lengths to k 32 and elicit relativity by letting NN/3 (see Section 2.2). To balance the two desired properties, we randomly choose 32 and k N/3 with an equal probability during fine-tuning, Here compare three scenarios: (1) enforce only continuity, (2) enforce only relativity, and (3) balance continuity and relativity (see Figure 5(C)). We find that balancing continuity and relativity gives rise to the best performance, thus justifying modeling choice.
Ablation of Hy Hyperparameters In our implementation of truncated Gaussian sampling, as illustrated Equation (3), the only hyperparameters are the mean and the variance a. The mean μ determined by the expansion factor. The variance is adaptable based data, we conducted experiments with five different values of a. The results, as presented Figure 6, indicate that the current selection (a 3) yields optimal performance.

---

cmn 25
22
pm 55legg
28
Irio Lenge?
Figure 6: Ablation Results (%) on LongChat-Lines. Each length consisting of 50 samples. The above are the results of using Linear interpolation on the Llama 2-7B model. Related Works
Efficient Transformers and Extra Memory FoT [Tworkowski al., 2024] addresses the lim- itations of local attention in transformers by integrating memory attention layers, which enable large models to learn from a wide context while reducing interference. Infini-attention [Munkhdalai et al., 2024] incorporates compressed memory into the standard attention mechanism and integrates masked local attention and long-term linear attention mechanisms within single Transformer block. LLoCO [Tan et al., 2024] employs LoRA in conjunction with context compression, retrieval, and parameter-efficient fine-tuning to learn context offline. Although these methods can successfully ex- tend the long context window of LLMs, they either require modifications to the attention mechanism or the addition extra modules for assistance. In contrast, CREAM does not require these operations and can be directly applied to a pre-trained model.
Positional Interpolation Chen et [2023] first proposed extending the context window through positional interpolation, which linearly reduces the input position indices to match the original context window size, thereby preventing catastrophic high attention scores from completely disrupting the self-attention mechanism. Subsequently. various methods (such as NTK [Peng and Quesnelle, 2023], ABF [Xiong et al., 2023], and EABF [Zhang et al., 2024]) emerged that modify the base frequency of rotary positional encoding to achieve positional interpolation. YaRN [Peng et al., 2023] introduced segmented interpolation method, applying different positional interpolations to different dimensions. LongRoPE [Ding et al., 2024] identifies and utilizes two forms of non-uniformity in positional interpolation through search, and introduces a progressive expansion strategy for positiona interpolation. Moreover, CREAM can be combined with any positional interpolation method.
Positional Encoding RandPos [Ruoss al., 2023] first modified position indices so that the model leverages the relativity of positions, enabling it to extend to the target length with fine-tuning over shorter lengths. PoSE [Zhu et al., 2023] then emphasized the importance of continuous segments, dividing the training length into two parts to further enhance the interpolation effect. CREAM utilizes both relativity and continuity, and also better enables the model to focus on the middle part of the context.
5 Conclusion
We proposed Continuity- ContinuityRRllatvvity indExing with gAussian Middle (CREAM), a simple yet effective method to extend the context of large language models. CREAM achieves trade-off between continuity and relativity. enabling the model to exploit positional relativity (i.e., fine-tuning within the pre-traitse rre-triined length), while preserving text continuity (i.e., remaining as close as possible the pre-trained state). Furthermore, by employing truncated Gaussian sampling, the model can concentrate more on the middle positions during fine-tuning. Experimental results demonstrate that CREAM outperforms other methods on both Base and Chat models and effectively mitigates the issue of "lost in the middle".
Acknowledgement
The authors thank the reviewers for their insightful suggestions to improve the manuscript. This work presented herein supported by the National Natural Science Foundation of China (62376031).
10

---

References
Dawei Zhu, Nan Yang. Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension Ilms via positional skip-wise training. The Twelfth International Conference on Learning Representations, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023a.
Xijie Huang, Li Lyna Zhang, Kwang- Kwang-Ting Cheng, and Mao Yang. Boosting Ilm reasoning: Push the limits of few-shot learning with reinforced in-context pruning. arXiv preprint urXiv:23/2.08901 2023.
Jiaqi Li, Mengmeng Wang. Zilong Zheng. and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311. 04499, 2023a.
Chen Qian, Xin Cong, Wei Liu, Cheng Yang. Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu,
Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023.
Zilong Zheng, Zixia Jia, Mengmeng Wang, Wentao Ding, Baichen Tong, and Songchun Zhu. Langsuit-e: Controlling, planning, and interacting with large language models in embodied text environments, 2023. URL https: gi thub. com/bi manm/bigaiiaaa maainncoo//iiaaaaaiaee / llaguuiin..e..
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023.
Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows Ilama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.. 2023. URL https://reet it/ //141z7 j5.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarı: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023.
Wenhan Xiong. Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
Yikai Zhang, Junlong Li, and Pengfei Liu. Extending llms' context window with 100 samples. arXiv preprint arXiv:240..70004 2024.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miloš. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with antni-attention. arXiv preprint arXiv:2404.07143, 2024.
Sijun Tan, Xiuyu Ii, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E Gonzalez, and Raluca Ada Popa. Lloco: I Learning long contexts offline. arXiv preprint arXtv:2404.07979, 2024.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mchdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. In Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1889-1903, 2023.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages
2018.
Wenhao Wu, Yizhong Wang, Yao Fu, Xiang Yue, Dawei Zhu, and Sujian Li. Long context alignment with short instructions and synthesized positions. arXiv preprint 2xXx:24055009333 2024.
11

---

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023b.
Al@Meta. Llama 3 model card. 2024. URL https::/ggithhi com/meta conmmetta-lammae httpa/aaaaaa///oo////aaaa MODEL CARD_ mdd
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16133 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 10009.17444 17453, 2023.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308. 10882, 2023.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. httpp::/huggiingfaee ca/cco//aa// spaces/ Satgginngacee////oo /otpl/////ee////// 1 lalllmmmeeee__eeeeeeeeeeeeeeeeee leaderboard, 2023.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023.
Dacheng Li, Rulin Shao, Auze Xie, Ying Sheng, Liammin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-soures LLMs truly promisc? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023b. URL https: / /ttp///peeeeeeeee eveen. wwt/tfeuuurii LywifFNXV5.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http///pppeeeeeeeee/eeeeeeeeeeee nwt www.eeeeee/eineeeeeeeeeeee id=nnZeKeeFFFFF
Tim Dettmers, Artidoro Pagmoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In seirth- Conference on Neural Information Processing Systems, 2023. URL
bttaps://operreviii wwt.rriwwe..eeeeeeeeeeeeeeeeeeee wwtpeeeeeteteeeeeeeeeeeeeeeeeeee idd-rPPPPEd-
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 1419 1436. Association for Computa- tional Linguistics (ACL), 2021.
Bartosz Piotrowski Zhangir Azerbayev, Edward Ayers. Proof-pile. 2022. URL https://github...
zhangir- www.aaaeeeebbroooooeaaee proll..ie....
Yutong Hu, Quzhe Huang. Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? The Second Tiny Pupers Track at ICLR 2024, 2024.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ttpss///pperrvveew.tt net/foruntttrrr id=LY3ukUANko.
Jongho Park, Jacscung Park, Zheyang Xiong, Nayoung Lec, Jacwoong Cho, Samet Oymak, Kangwook Lec, and Dimitris Papailiopoulos. Can mamba learn how to leam? a comparative study on in-context learning tasks. In ICLR 2024 Workshop Mathematical and Empirical Understanding of Foundation Models, 2024. URL ttttpa://openreveee wwwearevieennee/forreeeeeeeeeeee net/forum? iid.rrroctdx.r
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2019.
Shawn Presser. 2020.
12

---

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond 2 million tokens. arXiv preprint 2XXv:2442..1355. 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Pytorch: imperative style, high sech-pefformance deep learning library, Advances in neural information processing systems, 32, 2019.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In Twelfth International Conference on Learning Representations, 2023.
Leo Gao, Stella Biderman, Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 00027, 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm as-a- with mt mecceecc and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
13

---

A Relative Positional Encoding in RoPE
We provide simple background proof on the relative positional encoding performed by Rotary Position Embedding (RoPE) Su et al. [2024]. Given two embedding vectors 9. E RR corresponds to query and key at positions (m,n) [0, L), where d is embedding dimension, their encoding counterparts can be defined as:
4m fg m)) R m (2gg m)
(7)
hn = fa(22 ni) Rgg, (o0k; n)
where
m0)
sin m0 cos m&1
sin min
Rg.m
(8)
cos m0a/2
sin m0./2
0
0
sin mBa/2
cos m&a/2
the rotary matrix, θ (0; = 10000 20000-20011/4 [1, ... ../_] is pre-defined rotation angles.
Then the self attention score can be obtained with:
qm ki
(F m), f& (2),
d/2-1
I 10 (m-
Re
g(mm: En n2)
where x" represents the conjugate complex of æ, is the derived attention function of RoPE. As seen, RoPE only depends on the relative distances between and encodes the relative position information.
B Theoretical findings of CREAM design
Theorem B.1. N & L, the spanning size \Drl of the relative position union in Equation (2)
reaches its maximum iff, one of the following groups of inequalities satisfies:
max Thh - P. Ps, I1 -) 1) T, < (I. T.)/2,
(10)
or
(L+ T2)/2 - < P. I, (L, P. Ps, T, 1);
(11)
where max D, max (L, P. P.. L, 1) 2
Proof. Denote four intervals in Equation (2) as S,, = 1.... 4. According to the inequality of inclusion-exclusion principle for the cardinality of the union of sets:
DJ L S
Σ
S.
(12)
where the equality holds iff. all sets pairwise disjoint. That is
S,n Ø, vi Vi
(13)
Given intervals as Equation (2), we have
MAX < P. - La +
MAX < L - - L P +1
Ps < L L, Pr 1 Ps < L
or
(14)
Ln +
where MAX max (In 1, Pa
Ps, Iu 1). The above inequalities can be simplified to Equa-
tions (10) and (11).
14

---

Lemma B.2. Under mild assumptions that L --L,, L, = L, L- Ln ze L, the maximization in Theorem B.
holds for all (Ps.Pal E (N. N).
Proof. Given that
max Ln 1, Pe - P.. Li 1) LA - 1 < max(2Ln, N - Lt, N - Lmm < N
(15)
L- L. max Li. I, P. Pe. L, - 1)
max(N Lm N Lh, 2L N,
the inequalities in Equations (10) and (11) turns into LN,L/2) N..
Theorem B.3. N L, when the spanning size |Dy of the relative position union in Equation (2)
reaches its maximum, we denote the coverage area of the middle segment as:
Sm == fal E (P
P.)
arg max [rr] (P.
(16)
thus, we have:
L2 Sm + Ln + L N/2
(17)
Furthermore, as 0, we have:
Lp + Sm Lu - L
(18)
C Experimental Details
Model Hyperparameters We fine-tune all models by optimizing the causal language modeling objective. A learning rate of 2 x 10-5 with a linear scheduler adopted, incorporating 10 warm- up steps. We use the AdamW Loshchilov and Hutter [2018] optimizer with the hyperparameter configurations specified by PyTorch Paszke et al. [2019]. To speed up fine-tuning, we resort to DeepSpeed 4 ZeRO stage 1 and Flash Attention-2 Dao [2023]. We perform fine-tuning two A100-80G GPUs with a total batch size of 32 and run inference on a single A100-80G GPU. For CREAM-Base, we fine-tune it for 1,000 steps dataset derived from Pile Gao et al. [2020]; for CREAM-Chat, we fine-tune for 100 steps on ShareGPT Zheng et al. [2024]. To ensure fair comparison, follow the fine-tuning and inference configurations established by Zhu al. [2023].
Datasets and Training Cost For training the Base model, we directly utilize The Pile data provided by Zhu et al. [2023], and select samples with token lengths exceeding 4K. For training the Chat model, we filter the ShareGPT data from public datasets Specifically, we used the Vicuna prompt template to sequentially concatenate the ShareGPT data until each data point comprises at least 4K tokens. Then, we select 3.2K data points to train for 100 steps. Particularly, during the instruction tuning process, mask the USER part and allow the model to calculate the loss only the ASSISTANT part. We utilize two A A100-80G machines with a global batch size of 32, fully utilizing the available memory. Running 1,000 steps for the Base model takes approximately 6 hours, while running 100 steps for the Chat model takes approximately hours.
D Robustness Across LLMs
Our proposed method exhibits strong generalization capabilities and can be applied to other large language models (LLMs) without the need for parameter modification. To validate this, we conducted experiments on Bai Baichuun2-77 with the corresponding results presented in Table 7.
Furthermore, we fine-tuned LLaMa3-88 using a context window size of 4K tokens, with the experi- mental outcomes shown Table 8.
The results in Tables 7 and clearly demonstrate the transferability of our method to different models, underscoring its robustness. Of particular note is that despite LLaMa3-8B having native context length of 8K tokens, fine-tuning on training data with a 4K context window yielded unexpectedly strong performance.
'nttps://hugginggaace wttp:///.....tttttttaeeeeaaoaeee
hwwpeeettseeaaaaaaaeaaeeeee Vicuna, unf tered
15

---

Table 7: Perplexity results of GovReport and Proof-pile. Each experiment is the average perplexity of 50 samples, and all results are based on Bai chuan2 7B fine-tuned on 4K data length.
GovReport
Proof-pile
Model
4K
8K
16K
32K
4K
8K
16K
32K
Original
3.3
5.8
CREAM-Linear
3.6
2.9
2.5
2.2
6.2
6.1
6.0
5.8
Table 8: Results (%) on LongChat-LLine Each length consists of 50 samples. All results are fine-tuned on Llama-3-8B with 4K length data through linear position interpolation.
AVG Length
2000
4000
7800
8800
9700
11000
12000
14000
17000
19000
24000
28000
32000
CREAM CREAM-Linear
0.98
1.00
0.96
0.94
0.86
0.92
0.92
0.92
0.86
0.84
0.70
0.60
0.48
E
LongChat Lines Results
The interpolation methods using NTK and Yarn are presented in Figures and 8. As can be seen, CREAM performs the same the Linear method for interpolation, still outperforming other methods. The result of NTK at 26K-32K zero, which is due to the inherent properties of NTK, a finding that is aligns with Zhu et al. [2023].
Orginnl
22
Megg23.22
62
2500
3500
4200
4500
6000
7100
2002
11509
14020
28000
37500
20002
22003
25000
25000
30000
52000
Length
Figure 7: Results (%) on LongChat-Lines. Each length consisting of 50 samples. The above are the results of using NTK interpolation on the Llama 2-7B model.
F LongBench Subtasks Results
The results of each subtask in Tables 2 are shown in Tables 10 and 11.
It is noteworthy that, provide further evidence of the efficacy of our model, we have specifically chosen 12 tasks from the four categories outlined in Zhang et al. [2024] for comparison purposes. As delineated in Table 9, are able to attain superior performance LongBench in comparison to EABF Zhang et al. [2024], even with shorter training lengths and less data.
G Limitations
When extending the context beyond the pre-trained length, there is an inevitable loss of information due position interpolation, particularly when fine-tuning restricted the pre- pre-trained length. However, in comparison to previous methods such as RandPos Ruoss et al. [2023] and PoSE Zhu et al. [2023], CREAM has effectively mitigated the issue of Wost-in-the-Middee by introducing truncated Gaussian sampling. Additionally, as discussed in reference Liu et al. [2024], decoder-only models are prone inherently exhibiting U-shaped performance curve on this task. Therefore, completely solving this problem remains challenging.
16

---

Origial
Fanshoo
80
FoSE 37.2)
Cus (ag 2agg-48.00
AuAr 8
10
24.
20
Figure 8: Results ( (%) on LongChat-Liines Each length consisting of 50 samples. The above are the results of using Yarn interpolation on the Llama 2-7B model.
Table 9: Experimental results (%) of the LongBench subtasks selected in Zhang al. [2024]. indicates results quoted from Zhang et al. [2024]. Len represents the context length during fine-tuning. All results are based on Llama 2-7B.
Multi-Doe
Learning
Num
QAPR
MNWS
TRVQA
20.1
30.4
28.1
36.6
68.0
NYK--R-Pand
15.0
31.1
40.1
25.4
76.5
26.7
22.4
76.9
68.5
82.8
42.9
34.7
Yara!
3.5/ 16K
20.3
25.9
42.5
27.8
30.7
7.2
27.4
22.5
26.8
66.0
85.6
42.6
35.7
ABP
24.6
/2K
2300
36.8
Table 10: Experimental results ( (%) of the LongBench subtasks.
Singl-Doc QA
Multi-Doc QA
Summarization
Model
NQA
QAPR
MFQA_en
HPQA
WMQA
MSQ
GR
QMSM
MNWS
JJama2-7B-chat
18.7
19.2
36.8
25.4
32.8
27.3
20.8
25.8
XGen-7B-8k
18.0
18.1
37.7
29.7
21.1
10.3
27.3
20.5
26.2
InternLM-7B-S9
12.1
16.7
23.4
28.7
22.8
9.0
9.7
15.9
22.8
Vicuma-vl.5-7B... 16k
19.4
26.1
38.5
25.3
20.8
9.8
27.9
22.8
27.2
LongChat-v..--7B-222
16.9
27.7
41.4
31.5
20.6
9.7
30.8
22.7
26.4
CREAM
23.0
34.6
46.8
42.2
33.7
17.4
30.4
24.3
26.8
Table 11: Experimental results (%) of the LongBench subtasks.
Few Learning
Code Completion
Synthetic Tasks
Model
TREC
TRVQA
SMSM
PC
PR_en
LCC
RBP
Jlama2-7B-chat-44
61.5
77.8
40.7
2.1
9.8
52.4
43.8
XGen-7B-8k
65.5
77.8
25.3
2.1
8.5
38.6
38.6
InternL.M-7B-8K-
$2.0
77.8
21.2
3.0
6.0
44.1
28.8
Vicuna-v1.5-7B-1o-
71.5
86.2
40.8
6.5
51.0
LongChat-vl.5-788
63.5
82.3
34.2
1.0
30.5
53.0
55.3
CREAM
69.5
84.0
41.9
3.0
11.0
52.0
48.7
17

---

H
Loss Curve
(a) CREAM- 192K Training Loss
(b) CREAM 192K Validation Loss
(c) CREAM-256K Training Loss
(d) CREAM-256K Validation Loss
Figure 9: Fine-tuning loss curve based on Llama 2-7B. black line represents Linear interpola-
tion, the pink line represents NTK interpolation, and the cyan line represents YaRN interpolation.
(a) RandPos Training Loss
(b) RandPos Validation Loss
(c) PoSE Training Loss
(d) PoSE Validation Loss
CREAM Training
(f CREAM Validation Loss
Figure 10: Fine-tuning loss curve based on Llama 2-7B. The black line represents Linear interpo- lation, the pink line represents NTK interpolation, and the cyan line represents YaRN interpolation.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work performed by us.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

For each theoretical result, we provide the full set of assumptions and complete (and correct) proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
onoollooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. cctpublic//acccccccccccccccc.lct //wp//uuss/////ee /Codd/ooddeemiii cestsuummissioi ssisnnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: All our experiments are seeded, the results are unique, there are no error conditions, and therefore they are not needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: For each experiment, we provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /neurpps....p// cc htt.ai/c/thiiisuiieeeeee
Answer: [Yes]
Justification: We conducted the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no potential positive or negative social impacts involved our work, there is need for that.
22

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. dep.lymell of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We use all open source datasets and models, and do not involve data or models with a higher risk of abuse, so there no need do so.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators and original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We're not introducing new assets, so they's re not needed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't have crowdsourced experiments and research with human subjects, so we don't need to.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We don't involve any subjects, there are no risks, there's no need for it.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "aqMETYgnVI", "reviewer": "Reviewer_fWsk", "summary": "The paper proposes a method for extending the context window of pretrain large language models. The approach, CREAM, relies on modifying position indices to interpolate the positional encodings. Despite the often computationally expensive nature of such work, CREAM can extend to very long context windows while only needing to train at the original pretrained context window. Additionally, in their approach to the context length problem, the authors propose a solution that focuses explicitly on learning at the middle of the context — a span that often under-performs in long context models. The experimental results cover a wide variety of problems, both in terms of tasks, as well as context length challenges. Overall the results look very promising, and show a strong method for addressing a challenging task of extending the context length of LLMs.", "questions": "The authors mention that CREAM only needed to be trained for 100 in some cases. Can the authors provide more insight on why the results are very good despite so little training? This seems like a significant achievement, but was not discussed in detail.", "limitations": "The amount of focus on the middle context seems fixed in this approach. While the results for Gaussian truncation appear promising, it's not clear if this direct approach to solving the \"lost in the middle\" problem is universal. In short, how do we know how much to reweight the importance of the middle context, and does it vary by dataset or task?", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "- The authors tackle two problems that go hand in hand, namely extending the context length during fine-tuning - but doing so in a way ensures consistent performance. \n- The approach divides the desired context length into three segments, which then results in relative positional distances that vary and consequently learning all relative positions within the target length $L$. The technique is simple, but clever and an effective way to efficiently expose the model to a broader range of relative positional distances during training.\n- The results are very strong, particularly with the performances listed on Long Bench, which encompasses a broad range of tasks.", "weaknesses": "- No error bars shown on results. In most cases the results are quite strong, but the error bars would be helpful — particularly in some of the closer comparisons with PoSE.\n- The solution of using a truncated Gaussian approach lacks motivation. The \"lost in the middle\" problem is clear, however the solution of using a truncated Gaussian to force more focus on the middle seems brittle. Could the parameters of the truncated Gaussian be learned from data? The approach works well based on the results shown in Figure 5a, however.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments ``simple, but clever, effective and effcient`` and acknowledgement of our approach with ``very strong performances that encompass a broad range of tasks``. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.\n\n>**W1**：No error bars shown on results. In most cases the results are quite strong, but the error bars would be helpful — particularly in some of the closer comparisons with PoSE.\n\n**A1:** Thank you for the helpful suggestion. In our manuscript, we fixed all random seeds of all experiments to guarantee the reproducibility of our reported results. Per your advice, we have further conducted repeated experiments with five randomly generated seeds on the linear extension experiment. The detailed results with mean$\\pm$std are reported in the table below.  \n\n| Model           | Zero-Shot  |                 |           |           | Few-Shot  |           |\n| :-------------- | :--------- | :-------------- | :-------- | :-------- | :-------- | :-------- |\n|                 | WinoGrande | TruthfulQA(mc2) | PIQA      | BoolQ     | ARC-C     | HellaSwag |\n| PoSE-Linear  | 68.7±1.30  | 38.6±1.36       | 77.9±0.97 | 76.3±0.74 | 47.4±1.46 | 76.9±0.42 |\n| CREAM-Linear | 68.8±1.30  | 38.5±1.35       | 78.1±0.97 | 76.3±0.75 | 47.5±1.46 | 77.0±0.42 |\n\nBy comparing the mean and variance in the table, it can be observed that CREAM and PoSE have comparable capabilities in handling short texts. We will add the results in our revision.\n\n>**W2:** The solution of using a truncated Gaussian approach lacks motivation. The \"lost in the middle\" problem is clear, however the solution of using a truncated Gaussian to force more focus on the middle seems brittle. Could the parameters of the truncated Gaussian be learned from data? The approach works well based on the results shown in Figure 5a, however.\n\n**A2:** Thank you for pointing this out. Our motivations for truncated Gaussian are three-fold:\n\n1. **Intuitive Explanation.** The issue of \"lost in the middle\" highlights that the performance of LLMs is often strong at the beginning and end, but weak in the middle[1].  As discussed in **L111-114**, a straightforward idea to address this is to **guide LLMs to focus more on the middle part relative to the beginning and end** of the context. This approach results in a \"reverse U\" shape curve, similar to a truncated Gaussian distribution curve.\n2. **Theoretical support.** In Appendix B and L114-115, we provide theoretical justification that sampling the middle part of a long context with a high importance rate yields a maximization of learned relative position intervals as identified in Equations (2). \n3. **Empirical observation.** As you mentioned, it has proven to be quite effective for all demonstrated long context tasks, encompassing retrieval, lost-in-the-middle, LongBench, etc.\n\n**Could the parameters of the truncated Gaussian be learned from data?**\n\nGreat question. In our truncated Gaussian sampling, as shown in Equation (3), the only hyperparameters are the mean $\\mu$ and the variance $\\sigma$.\n\n1. For $\\mu$, it depends on the expansion factor. For example, when expanding from 4K to 32K, the expansion factor is 8, so $\\mu = (1+8)/2$. \n\n2. For $\\sigma$, learning from the data is flexible, but **since the sampling process is discrete, the gradient cannot be back-propagated, making the learning cost high.** Additionally, we conducted experiments with five different $\\sigma$ values, and the results in the table show that the current choice ($\\sigma=3$) indeed performs the best.\n\n| Length    | 2500 | 3600 | 4200 | 4800 | 6000 | 7100 | 9400 | 11800 | 14000 | 16000 | 17500 | 20000 | 22000 | 26000 | 28000 | 30000 | 32000 | AVG   |\n| :-------- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| sigma=2   | 0.68 | 0.56 | 0.7  | 0.66 | 0.7  | 0.68 | 0.68 | 0.56  | 0.52  | 0.4   | 0.42  | 0.38  | 0.48  | 0.38  | 0.4   | 0.28  | 0.12  | 0.506 |\n| sigma=2.5 | 0.9  | 0.72 | 0.78 | 0.86 | 0.78 | 0.76 | 0.7  | 0.56  | 0.64  | 0.38  | 0.52  | 0.4   | 0.48  | 0.36  | 0.46  | 0.34  | 0.3   | 0.585 |\n| sigma=3   | 0.96 | 0.82 | 0.92 | 0.94 | 0.92 | 0.86 | 0.84 | 0.78  | 0.76  | 0.56  | 0.62  | 0.52  | 0.62  | 0.46  | 0.52  | 0.38  | 0.4   | 0.699 |\n| sigma=3.5 | 0.9  | 0.72 | 0.84 | 0.8  | 0.84 | 0.78 | 0.74 | 0.66  | 0.58  | 0.4   | 0.54  | 0.4   | 0.5   | 0.38  | 0.38  | 0.36  | 0.26  | 0.593 |\n| sigma=4   | 0.9  | 0.8  | 0.86 | 0.84 | 0.78 | 0.72 | 0.72 | 0.5   | 0.5   | 0.42  | 0.44  | 0.26  | 0.36  | 0.22  | 0.3   | 0.3   | 0.26  | 0.540 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SVpfzVpJpS", "reviewer": "Reviewer_srX3", "summary": "The paper proposes a new method, CREAM, that better enables extrapolation to longer contexts via finetuning at the base context length. The method uses positional embedding interpolation with the embeddings divided into three areas of interest and uses a truncated Gaussian  to sample the position used for the middle segment, to increase training of the middle context. They show that this is at least as effective as prior length extrapolation methods that use finetuning at the base context length, and that CREAM-trained models suffer less from the lost in the middle problem.", "questions": "Q1. In Table 5: why does the 256k-extrapolated version of CREAM underperform the 128k-extrapolated version on short contexts?\n\nQ2. Say you had an even more limited amount of compute, to the point where finetuning on the base context length was prohibitively expensive (e.g., maybe you want to finetuned Llama3-8b but can't fit 8192 context at training time). Do you believe CREAM would be useful in this setting? Do you have any results (or speculation!) that suggest when the method may begin to break down with decreasing maximum training length?\n\nQ3. (Minor) What is the context length used in Figure 1? The x axis only reports the position of keys.\n\nSuggestions/comments:\n*  Please explain in the text what the occasional highlighting of numbers represents (e.g. in Table 2). It would be helpful for readability to bold the best number in each column across tables.", "limitations": "Yes, no concerns here.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The method shows clear improvement on the targeted issue (the lost in the middle phenomenon)-- e.g. in Table 1. \n\nS2. The method is well-motivated both intuitively and theoretically, and the explanation in section 2.2 is well-written and easy to follow. \n\nS3. The authors evaluate against the two most reasonable baselines (to my knowledge) and across a good selection of synthetic, long-context, and short-context tasks. (It would be an added benefit to evaluate against finetuning with position interpolation and *long* context at finetuning-time, to see the efficiency-performance tradeoff; however, I do not think this is strictly necessary to the paper's claims.)", "weaknesses": "W1. The method is fairly similar to POSE in concept and performance. While I think the idea of emphasizing middle training is reasonable, well-explored, and clearly effective (e.g. in Figure 1), this has also been explored as a post-training or data selection correction in other work. Given the slight performance degradation from POSE on short-context tasks (i.e. in Table 4), I'm concerned that this might not be worth the tradeoff practically to do at training time. \n\nW2. Evaluation settings can differ subtly across papers, and so it would be better for the authors to reproduce baseline results rather than citing them from literature, where possible. This is most critical, I think, in Table 4, where the differences between methods are relatively small and all results are compared to the same baseline method, which is not an infeasibly expensive or unavailable model to run.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your valuable suggestions and acknowledgment of our well-motivated method. Below we address your questions point-by-point and hope we can resolve your major concerns.\n\n>**W1:** The method is fairly similar to POSE in concept and performance. While I think the idea of emphasizing middle training is reasonable, well-explored, and clearly effective (e.g. in Figure 1), this has also been explored as a post-training or data selection correction in other work. Given the slight performance degradation from POSE on short-context tasks (i.e. in Table 4), I'm concerned that this might not be worth the tradeoff practically to do at training time.\n\n**A1:** We would like to highlight the major contributions and differences compared with PoSE, other post-training, and data selection correction as follows.\n\n1. For the concept, PoSE does not simultaneously leverage the benefits of continuity and relativity, nor does it emphasize the importance of intermediate context. These aspects are our core contributions.\n2. For the performance, our method **significantly outperforms PoSE across various tasks**. For example, it achieves an average improvement of 15.9% on LongChat Lines compared to PoSE. In the Lost in the Middle task, our different interpolation methods (Linear, NTK, Yarn) also show an average improvement of about 10%.\n3. In post-training, previous methods usually **require a large amount of data and computational resources, with the training context length being much greater than the context length during pre-training**. For instance, Yi-1.5[1] uses 10 billion tokens and requires upsampling of long sequences. Qwen-2[2] and InternLM2[3] extend their training context length from 4K to 32K in the final stages of pre-training. Therefore, we can conclude that CREAM **not only requires less data and shorter training context lengths but also demonstrates impressive performance**.\n4. For data selection correction, such as [3] using data engineering techniques to expand context. However, their method requires fine-tuning on at least 500M tokens to achieve satisfactory performance in the Needle-in-a-Haystack test (Figure 3). In contrast, our method only requires $seqlen(4096) \\times bsz(32) \\times steps(1000) = 125M$ tokens. Therefore, **their training data requirement is not only four times larger than ours but also requires meticulous data selection.** Additionally, **our method can be applied to instruction fine-tuning**, which they did not mention.\n\nIt is important to emphasize that on short-context tasks (i.e. Table 4), **our method performs just as well as PoSE**. In our manuscript, we overlooked the minor differences between ours and PoSE in Table 4 (<1%) since our focus and superiority lie in addressing the challenges of long contexts.  The minor performance turbulence results from the sampling of training data (a de facto strategy as in PoSE and others). To further validate this, we tested model performance with five different random seeds used to sample the training data. The mean$\\pm$std results are shown in the table below.\n\n| Model           | Zero-Shot  |                 |           |           | Few-Shot  |           |\n| :-------------- | :--------- | :-------------- | :-------- | :-------- | :-------- | :-------- |\n|                 | WinoGrande | TruthfulQA(mc2) | PIQA      | BoolQ     | ARC-C     | HellaSwag |\n| PoSE-Linear  | 68.7±1.30  | 38.6±1.36       | 77.9±0.97 | 76.3±0.74 | 47.4±1.46 | 76.9±0.42 |\n| CREAM-Linear | 68.8±1.30  | 38.5±1.35       | 78.1±0.97 | 76.3±0.75 | 47.5±1.46 | 77.0±0.42 |\n\nBased on the experimental results in the table, we can be more confident that the performance of CREAM and PoSE on short contexts is actually comparable. Thank you for the careful review. We would update and add the discussion in our revision.\n\n>**W2:** Evaluation settings can differ subtly across papers, and so it would be better for the authors to reproduce baseline results rather than citing them from literature, where possible. This is most critical, I think, in Table 4, where the differences between methods are relatively small and all results are compared to the same baseline method, which is not an infeasibly expensive or unavailable model to run.\n\n**A2:** Thank you for your suggestion. We've followed your advice and reproduced the results of LLaMa-2-7b-hf, as presented in Table 4. The results are detailed in the table below:\n\n| Model             | Zero-Shot  |                 |          |          | Few-Shot |           |\n| :---------------- | :--------- | :-------------- | :------- | :------- | :------- | :-------- |\n|                   | WinoGrande | TruthfulQA(mc2) | PIQA     | BoolQ    | ARC-C    | HellaSwag |\n| LLaMa-2-7b-hf*    | 69.2       | 39.5            | 78.8     | 77.4     | 45.9     | 77.2      |\n| **LLaMa-2-7b-hf** | **68.5**   | **38.9**        | **78.8** | **78.0** | **48.5** | **78.1**  |\n\nThe bolded results in the table are our reproductions, and they are very close to the original results. Therefore, we believe our final conclusions and claims still stand across all tasks. Thank you for the advice. We will add the above results to improve the soundness of our work.\n\n>**Q1**: In Table 5: why does the 256k-extrapolated version of CREAM underperform the 128k-extrapolated version on short contexts?\n\n**A1:** In Table 5, the 128K results are not ours but are cited from PoSE. Since **a lower perplexity is better**, the results in Table 5 indicate that our method has the potential to extend to 192K or even 256K. Furthermore, compared to PoSE, our performance is significantly superior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0TKm4UAB2k", "reviewer": "Reviewer_XDBh", "summary": "The paper presents CREAM (Continuity-Relativity indExing with gAussian Middle), an innovative approach to extend the context window of Large Language Models (LLMs) without the need for extensive fine-tuning at the target length. The authors address the \"Lost in the Middle\" problem, which plagues long-context LLMs by causing a performance drop when retrieving information from the middle of the context. CREAM achieves this by manipulating position indices and introducing a truncated Gaussian sampling method to focus on the middle part of the context during fine-tuning.", "questions": "1. What are the computational overheads associated with implementing CREAM, and how does it scale with larger context sizes?\n2. Are there any specific hyperparameter settings in CREAM that are particularly sensitive, and how were these chosen?", "limitations": "The discussion in Appendix F regarding the limitations of the CREAM method could be misleading. The author implies that CREAM provides enhanced performance compared to other methods concerning the \"Lost in the Middle\" issue but acknowledges that the problem is not fully solved due to the nature of decoder-only models. For a genuine limitations section, it is essential to delve deeper into the specific constraints of the CREAM approach and how these limitations might impact its application and effectiveness. A more explicit acknowledgment of the method's shortcomings and a clear explanation of why these issues cannot be overcome with the current model design would significantly improve the section's clarity and usefulness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Approach**: CREAM offers a novel positional encoding strategy that efficiently extends the context window of LLMs, which is a significant contribution to the field.\n   \n2. **Empirical Evidence**: The paper provides strong empirical evidence through comprehensive experiments, demonstrating CREAM's superiority over existing methods like PoSE and RandPos, especially in handling middle context information.\n\n3. **Training Efficiency**: The method requires fine-tuning at the pre-trained context window size, which is computationally efficient compared to fine-tuning at the target length.\n\n4. **Theoretical Foundation**: The paper includes theoretical justifications for the use of truncated Gaussian distribution, adding rigor to the proposed method.", "weaknesses": "1. **Generalizability**: While the paper shows impressive results, it is not clear how generalizable these findings are to other LLMs beyond the tested Llama2-7B model. And I'm wondering whether the method can work with PEFT tuning techniques, such as QLoRA. \n\n2. **Lack of Comparative Analysis**: The paper could benefit from a more detailed comparative analysis with other contemporary methods. For example, in Table 2, the paper only compares one model (LongChat-v1.5-7B-32k) with the same 32k context window. I suggest that in such comparision, peer models should at least take the same length of input, otherwise, it cannot prove the effectiveness of the proposed model over other Long LLMs with 32k+ context windows. I suggest the author compare with models like Mistral-7B-Instruct-v0.2, LongLoRA etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper and acknowledging the novelty and empirical superiority of our approach. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.\n\n>**W1:** **Generalizability**: While the paper shows impressive results, it is not clear how generalizable these findings are to other LLMs beyond the tested Llama2-7B model. And I'm wondering whether the method can work with PEFT tuning techniques, such as QLoRA.\n\n**A1:** Thank you for the constructive comments. Our method has strong generalization capabilities and **can be applied to other LLMs without modifying any parameters**. To verify this, we conducted experiments on Baichuan2-7B. The experimental results are shown in the table below. (Incidentally, as shown in the table in A2 of reviewer srX3's response, the experimental results on LLaMa3-8B also support this conclusion.)\n\n|            | GovReport |      |      |      | Proof-pile |      |      |      |\n| :---------------- | :-------- | :--- | :--- | :--- | :--------- | :--- | :--- | :--- |\n|                   | 4K        | 8K   | 16K  | 32K  | 4K         | 8K   | 16K  | 32K  |\n| Baichuan2-7B-Base | 3.3       | -    | -    | -    | 5.8        | -    | -    | -    |\n| CREAM-Linear      | 3.6       | 2.9  | 2.5  | 2.2  | 6.2        | 6.1  | 6.0  | 5.8  |\n\nThrough two sets of experiments, we demonstrate that **CREAM can work seamlessly with PEFT techniques like LoRA [1] and QLoRA [2] without requiring additional modifications**. This compatibility arises because CREAM does not alter any data formats or model structures during fine-tuning; it only adjusts position indices. Detailed experimental results are presented in the following two tables.\n\n| Model              | Single-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning | Code Completion | Synthetic Tasks | Macro |\n| :----------------- | :------------ | :----------- | :------------ | :---------------- | :-------------- | :-------------- | :---- |\n| Llama2-7B-chat-4k* | 24.9          | 22.6         | 24.7          | 60                | 48.1            | 5.9             | 31.0  |\n| LoRA-step-800      | 28.7          | 28.5         | 27.7          | 62.3              |54             |10.3              | 35.3  |\n| QLoRA-step-400     | 20.9          | 19.0         | 26.8          | 54.1              | 47.4            | 4.0             | 28.7  |\n| Ours               | 34.8          | 31.1         | 27.2          | 65.1              | 50.4            | 7               | 35.9  |\n\nWe will include the training curves of these two methods in the revised version.\n\n>**W2:** **Lack of Comparative Analysis**: The paper could benefit from a more detailed comparative analysis with other contemporary methods. For example, in Table 2, the paper only compares one model (LongChat-v1.5-7B-32k) with the same 32k context window. I suggest that in such comparision, peer models should at least take the same length of input, otherwise, it cannot prove the effectiveness of the proposed model over other Long LLMs with 32k+ context windows. I suggest the author compare with models like Mistral-7B-Instruct-v0.2, LongLoRA etc.\n\n**A2:** Thank you for the advice for improving our work. Per your suggestion, we have added experimental results of all three versions of Mistral-7B-Instruct (v0.1, v0.2, v0.3) on Longbench. Since the 7B-Instruct-32K model of LongLoRA is not publicly available, we could not add corresponding results and are more than willing to add the comparison once it's open-sourced. The detailed results of Mistral are presented in the table below.\n\n| Model                    | Single-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning | Code Completion | Synthetic Tasks | Macro    |\n| :----------------------- | :------------ | :----------- | :------------ | :---------------- | :-------------- | :-------------- | :------- |\n| Mistral-7B-Instruct-v0.1 | 29.5          | 20.7         | 26.4          | 13.6              | 29.6            | 10.8            | 21.8     |\n| Mistral-7B-Instruct-v0.2 | 28.5          | 21.5         | 26.1          | 50.1              | 33.8            | 13.9            | 29.0     |\n| Mistral-7B-Instruct-v0.3 | 33.2          | 30.6         | 26.8          | 56.4              | 15.3            | 10.4            | 28.8     |\n| LongChat-v1.5-7B-32k*    | 28.7          | 20.6         | 26.7          | 60.0              | 54.1            | 15.8            | 34.3     |\n| **Ours**                 | 34.8          | 31.1         | 27.2          | 65.1              | 50.4            | 7.0             | **35.9** |\n\nAs shown, it is evident that **our method outperforms all three versions of Mistral-7B-Instruct**. This further confirms the effectiveness of CREAM.\n\n>**Q1:** What are the computational overheads associated with implementing CREAM, and how does it scale with larger context sizes?\n\n**A1:** Our method **does not incur any additional computational overhead beyond standard fine-tuning**. This is because CREAM does not modify the model structure itself; the only change is to the positional indices. Additionally, as the context window length increases, the computational overhead will increase similarly to standard fine-tuning.\n\nHowever, even with a larger context window, it is still possible to fine-tune with a 4K context length, significantly reducing computational overhead and memory usage. As as shown in the table in A2 of reviewer srX3's response, fine-tuning LLaMA3-8B with a 4K context length (despite its pre-training context length of 8K) remains remarkably effective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIiMOCeU43", "reviewer": "Reviewer_qJ1n", "summary": "The paper introduces CREAM (Continuity-Relativity indExing with gAussian Middle), an efficient method for extending the context window of large language models (LLMs) to handle longer contexts without the need for extensive fine-tuning. CREAM manipulates position indices for shorter sequences within the pre-trained context window, using two key strategies: continuity, and relativity. Additionally, CREAM employs a truncated Gaussian distribution to enhance the model's focus on the middle part of the context, mitigating the \"Lost-in-the-Middle\" problem. Comprehensive experiments demonstrate CREAM's efficiency and effectiveness, showing improved performance over baselines like RandPos and PoSE on various tasks.", "questions": "See Weaknesses", "limitations": "None", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "CREAM achieves superior performance across a spectrum of long-context tasks. The empirical results substantiate CREAM's ability to effectively mitigate the \"Lost-in-the-Middle\" problem.", "weaknesses": "I think the presentation of this paper is a significant problem. Actually, I cannot understand the method itself nor the intuitions behind it. First, I do not understand the concept of continuity in positional encoding. The authors use a short paragraph to explain continuity (lines 69-74) without any formulation, which is very abstract. The same goes for relativity, though I understood it from another paper. After explaining these two concepts, you still need to explain why they are important (I did not see this in the paper). Second, I do not understand why splitting the pre-trained context into three segments with different sizes can achieve continuity and relativity. Third, I do not understand why truncated Gaussian middle sampling mitigates the \"Lost-in-the-Middle\" problem. Line 113 states that it reduces the interval overlap in Eq2. But what is interval overlap and why does it result in the \"Lost-in-the-Middle\" problem?\n\nI may not be the ideal reader for this paper, but if other reviewers feel the same way, this paper may have been hastily completed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our paper. First and foremost, we would like to express our sincere apologies for any confusion that our work brings to you. As you may have noticed, our presentation reached an average of 3.33 / 4 among three other reviewers. Specifically, reviewer srX3 appreciates the clear writing in methods and fWsk confirms the excellent presentation. Nevertheless, we deeply respect your criticism and would like to further clarify all your confusion and misunderstandings **point-by-point** as follows.\n\n>**Q1:** First, I do not understand the concept of continuity in positional encoding. The authors use a short paragraph to explain continuity (lines 69-74) without any formulation, which is very abstract. The same goes for relativity, though I understood it from another paper. After explaining these two concepts, you still need to explain why they are important (I did not see this in the paper). \n\n**A1:** In short,  the concept of continuity in positional encoding lies in the importance of ensuring the consistency of position indices between fine-tuning and pre-training. Specifically,\n\n1. In **lines 69-74,** we've discussed the continuity of positional encoding in detail  and provided relevant references. PoSE also highlights the importance of continuity in positional encoding during training. \n2. In **lines 75-82**, we elaborate on the relativity of positional encoding and provide a theoretical proof in **Appendix B**.\n3. In **lines 38-44**, we explain the roles of continuity and relativity. Our ablation experiments (Figure 5(c)) experimentally validate the importance of continuity and relativity.\n\n>**Q2:** Second, I do not understand why splitting the pre-trained context into three segments with different sizes can achieve continuity and relativity.\n\n**A2:** In lines 105-110 of our paper, we've provided a detailed explanation of this matter. The continuity design aims to ``allow the middle segment to closely approximate the pre-trained context window.`` The relativity design aims to ``enable the model to learn as many relative positions as possible.`` \n\n>**Q3**: Third, I do not understand why truncated Gaussian middle sampling mitigates the \"Lost-in-the-Middle\" problem. Line 113 states that it reduces the interval overlap in Eq2. But what is interval overlap and why does it result in the \"Lost-in-the-Middle\" problem?\n\n**A3:** In **lines 111-115** of our paper, we note that truncated Gaussian sampling can alleviate the \"lost in the middle\" problem by ``directing the model's attention towards the middle section of the long context``. The overlap of intervals refers to the repeated position indices in Equation (2). The \"lost in the middle\" issue **is not caused by interval overlap but by a bias inherent in the model, as noted in [1]**. In Appendix B and L114-115, we further provide theoretical justification that sampling the middle part of a long context with a high importance rate yields a maximization of learned relative position intervals as identified in Equations (2). \n\n\nThis work indeed has gone through extensive experiments, careful writing, and figure polishing. We tried to make all contents illustrative and provided detailed supplementary materials in appendices. We hope our answers together with appendices have provided sufficient explanations and clarified your confusion. We are more than willing to elaborate more details if you have further questions. \n\nReference:\n\n[1] Lost in the middle: How language models use long contexts. TACL, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers for their hard work, and we will re-emphasize a few strengths of our work:

1. We propose a ``simple, but clever and an effective way to efficiently expose the model to a broader range of relative positional distances during training`` (Reviewer fWsk). This ``novel positional encoding strategy that efficiently extends the context window of LLMs, which is a significant contribution to the field`` (Reviewer XDBh).
2. Our method ``provides strong empirical evidence through comprehensive experiments, demonstrating CREAM's superiority over existing methods like PoSE and RandPos, especially in handling middle context information``  (All Reviewers) . Additionally, our method ``is computationally efficient compared to fine-tuning at the target length`` (Reviewer XDBh).
3. Notably, our ``method is well-motivated both intuitively and theoretically, and the explanation in section 2.2 is well-written and easy to follow`` (Reviewer srX3,XDBh).

In the subsequent revisions, we address all the reviewers' comments by making the following modifications to our paper:

1. We add the surprisingly good experimental results of fine-tuning LLaMa3-8B using CREAM on 4K data length. Additionally, we include the experimental results of applying CREAM on Baichuan2-7B to demonstrate the generality of our approach across different scenarios and LLMs. (Reviewers srX3, XDBh)
2. We include the experimental results of combining CREAM with different PEFT techniques, such as LoRA and QLoRA, to further prove the versatility of our method. (Reviewer XDBh)
3. We add the ablation results of different $\sigma$ values. (Reviewers fWsk, XDBh)
4. We include error bars in the results of Table 4 and add the reproduced results of LLaMa2-7B. (Reviewers fWsk, srX3)
5. We add the experimental results of Mistral-7B-Instruct (v0.1, v0.2, v0.3) on Longbench in Table 2 to further validate the effectiveness of our approach. (Reviewer XDBh)
6. We further discuss the reasons for the limitations of our method in the limitations section. (Reviewer XDBh)
   
We hope our responses will address the reviewers' concerns. If there are any questions, we look forward to further discussions.

Meta-review: The paper introduces CREAM, a method for extending the context window of large language models that emphasizes middle context enhancement, showing promising results with efficient training and broad applicability.  CREAM manipulates position indices and employs a truncated Gaussian distribution to focus on the middle part of the context, effectively addressing the "Lost-in-the-Middle" problem.  The proposed method demonstrates superior performance over existing methods like PoSE and RandPos across various tasks. The method shows robustness and has the potential to significantly enhance LLMs' ability to process long contexts.  The authors well addressed the reviewers' concerns in the rebuttal.  Reviewers are overall positive about the paper except for one who suggests to reject but seems not very confident.
Decision: Accept (poster)