Make Your LLM Fully Utilize the Context
Authors: Shengnan An
Zexiong Ma
Zeqi Lin
Nanning Zheng
Jian-Guang Lou
Weizhu Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Long-Context Large Language Model
Long-Context Training Data Construction
Long-Context Probing
Lost-in-the-Middle
Primary area: natural_language_processing
TL;DR: This work introduces information-intensive training to overcome the lost-in-the-middle problem of long-context LLMs.

Abstract: While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the *lost-in-the-middle* challenge.
We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information.
Based on this intuition, our study presents **information-intensive (IN2) training**, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) **fine-grained information awareness** on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the **integration and reasoning** of information from two or more short segments.
Through applying this information-intensive training on Mistral-7B, we present **FILM-7B** (FIll-in-the-Middle).
To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).
The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window.
Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU).

Make Your LLM Fully Utilize the Context
Shengman An Zexiong Ma" Ma* Zeqi Lin Nanning Zheng JJinn-Guang Lou Weizhu Chen
O National Key Laboratory Sluman-Maching Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi' Jiaotong University * Microsoft Peking University
(an1006634493estu mwwpaeeeeetttttmmegnnnaneaaaaaaa 10 edu. cn, mazeeiong@stu pku. edu. cn, * (Zeqi Lin, jlou, wzchen (encheenemicrooootooooo
Abstract
While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middlle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails emphasize that any position long context can hold crucial information. Based on this intuition, study presents INformation-INtensii (IN2) training. purely data-driven solution to overcome lost-in-the-nie the-middee. Specifically, IN2 training leverages synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on short segment 128 tokens) within synthesized long context (4K-32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying coormation-inteniive this information intensive training on Mistral-7B, we present FILM-7B (FILI- in-the- MhddMe. To thoroughly assess the ability of FII.-7BB for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5- Fl score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3 +59.2 accuracy MMLU).
Introduction
To a great mind, nothing is little.
-Artur Conan Doyle
Long-context large language models (LLMs) have recently received significant attention within the open-source community (Jiang et al.
2023 Du et al. [022 Li et al., 2023a Shi et 2023 Team
al. 2023 Team 2023 Chen al.
2023a Song et al. 2023 | Liu al. [023]] Peng et al.) 2023b
Chen et al. 2023b Xiong al. 2023 Tworkowski et al. 2024 Al et al. 2024 Ding et al. 2024
Mohtashami & Jaggi]
2024 et al. 2024) Cai et al. 2024 Bai al. 2024 Lv et al.) 2024). The
training context windows of many contemporary LLMs have been expanded to tens of thousands of
Work done during the internship at Microsoft. Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Document Sentence Retrieval (Bi- Directirc)
Code Function Retrieval (Backward)
Database Entity Retrieval (Forward)
Porfomance (56)
18.
0.2
07
15. FIXGSS 1O
CIFATuns
50
14
30
Relative Posttions BOO Sentences
Relative Positions 900 Funch
Relative Positions Entities
Figure I: Performance of FILM-7B, Mistral- TB-Inntruct- and GPT4-Turbo on our three probing tasks. FILM-7B significantly overcomes the problem of information loss in the middle of the context.
tokens, thereby enabling these models to process extensive context as input. This extended training context window can enhance many world downstream tasks such as long-context question answering (Kočiský et al. 2018 Dasigi et al. 2021: Bai et al., 2023) and summarization (Fabbri
al. 2019
Huang et al.) 2021: hoogg al.] 2021p.
However, recent studies have revealed that these long-context LLMs struggle to effectively and robustly utilize all the information provided in the context, known the lost-in-the-micee chal- lenge (Liu et al., 2024b; Xu et al.; 2023). It implies that while the LLM can comprehend the information at the beginning and end of the long context, it often overlooks the information in the middle. This challenge could significantly hinder the development of long-context LLMs, as they even often fail pass simple probing tasks such as Credle-in-the-Hayeee trieval (Mohtashami & Jaggil 2024). Consequently. a pressing research question arises: how can we make long-context LLMs fally utilize the information in the long context?
and passkey re-
We hypothesize that the root cause of lost-in-ther tidde stems from the unintentional bias hidden in the general training data. In auto-regressive pre- pre-training, the loss predicting the next token is more likely to be influenced by a few nearby pre-tokens rather than long-distance tokens (Sharan et al. 2018 Sun al., 2021). 1. For supervised fine-tuning and alignment, the system message, which strongly influences the generation of the response, is typically presented at the beginning of the context (Touvron et al., 2023; Cai et al. (024).. As result, the general training process may inadvertently introduce a position blas, suggesting that important information is always located at the beginning and end of the context.
Based on this hypothesis, our work introduces INformation- INtensive (IN2) training to explicitly teach the model that the crucial information can be intensively present throughout the context, not just at the beginning and end. IN2 training is a purely data-driven solution that utilizes a synthesized long-context question-answe answer dataset. The long context (ranging from 4K to 32K tokens) is concatenated from many short segments (~~128 tokens), and the question-answa (QA) pairs ask for the information contained in one or more segments which are randomly placed in the long context. Specifically, we generate two types of questions, requiring (1) fine-grained information awareness on exactly one short segment, and (2) the integration and reasoning of information from two or more segments. These QA pairs are generated by prompting GPT-44Turbo (OpenAl 2023b) with the designed instructions and the raw segments.
By applying this information-intesiia training on Mistral-7B (Jiang et al. 2023 we present FILM- 7B MeLl-in-the-Middee To thoroughly assess the cong-context information awareness of FILM-7B, we design three probing tasks encompassing various context styles (document, code, and structured- data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results (Figure [T demonstrate that IN2 training significantly overcomes the lost-in- the- middle problem for the backbone model. Moreover, it can enhance the open-source model to achieve comparable or even more robust performance compared with proprietary LLMs such as GPT-4- Turbo.
Beyond these probing tasks, the performance of FILM-7B on real-world long-context tasks also exhibits significant improvements (e-g.. 23.5 23.5-+26.9 Fl score on NarrativeQA (Kočiský et al. 2018)). This demonstrates that the post-training synthesized long-context data can be generalized to

---

Rew
Single Segment
Local Information QA
Long Context
Spitt by 128 Tokens
Prompting GPT-4
41
E
S1
a;
Concaterate with Randoon Segmonts
Fine-Grained Information Awareness
Raw Text
Multiple Segments
Multi-Hop QA
Long Context
s
Split by 128 Tokers
Prompting GPT-4
qi
C₁
s
a;
si
2
Concateruate with Kandorn Segmantss
3
Integration and Reasoning of Information
Figure 2: The data construction process for IN2 training, aimed at enhancing the fine-grained information awareness (upper), and the integration and reasoning information (lower).
real-world scenarios. Moreover, FILM-7B maintains comparable performance short-context tasks compared with the vanilla backhone model (e.g., 59.3-+59.2 accuracy on MMLU (Hendrycks) al. 2020). This indicates that the short-context capability of Fm.M-7B compromised during training. Our further analysis explores how the sliding window strategy and the choice of RoPE base 0 influence the performance of IN2 training.
2
Information--nteesive Training
This section introduces the construction of the dataset for IN2 training and the detailed training process of our model FILM-7B.
2.1 Training Data Construction
Overview. The IN2 training aims to explicitly teach the model that any position in a long context can contain crucial information. To achieve this goal, we construct long-context question-an answer training dataset DD = {C,, J where the answer to the question requires the information contained in some short segments that are randomly placed the whole long context L.
Figure illustrates an overview of the data construction process. Specifically, the training data ID constructed based on general natural language corpus C. Given generate question-answer pair (g;, a:) using powerful LLM, then synthesize a long context C; that includes the necessary information from and other randomly sampled texts from C. We generate two types of question-answer pairs that require (1) the awareness of fine-grained information in the long context, and (2) the integration and reasoning of information appearing at different positions in the long context. We take the realnewslike ke subset from the C4 corpus (Raffel et al. 2020) as C, and take GPT-4-Turbo (OpenAl 2023b) the LLM to generate QA pairs.
text C, E C, first
Fine-grained information awareness. We consider 128-token segment as the minimum informa- tion unit of the contexf" Given raw text C,, we first randomly extract 128-token segment s; from it, then generate the qis ag and C; accordingly,
(gs, a4) ~ Prompting(s;, Ty: LLM), L; = e (Shuffless; (rs))
Appendix contains the implementation and our considerations for this design choice.

---

Document Sentence Retrieval (ai-Direction)
Code Funxtion Ketrieval (Badswerrd)
Database Entry Retriesa (Foeward)
ABB Contwi::
ARB Conteat:
MR* Contect:
Th's erueally regiehes our a rigenthmm from the Specificlly fy, our moda fiy resiing owart prompee cat These resuts demoneraat there ane large engineering in
del gar, plaurecty iy tan reuulll dissse "Init
<:l 02 036022 libe haw York Stote Foute IR1,
def tppiatebata las selitrain nonn of inllocll seff
080829066, Mottl Tronseese se aidooo inision
commentts commanti commenteo
070869:34, Monitonin plaem
design better optinioent,
plor pari
subplonit passo
Progresion
stage
pempreeeet
063052215 Online
Edeediiv expartments conducted validate the eefectiveness our proposed methed, siliesing ittte-otthee oi-te--rt performane on all four bendhmark with notuabee gain.
sance mptiy plogie- Hijorshhopeh wwwpsooppoettiiiico the def creatderipp Ar story fntdvigrtcoorfggee
<ut GHSSREEER, best Alphasd S-c-------hila--o -deaaannaj-- rinycin as phar hag.aaallggaa chaperone for Barcher
It Rspae alfaeteen to androldio eccepe Exception om
disone, descrption scientffc artide published on
del brees cheder yy odfl mutatio tfftt Ri
August 2009>
Notably ach aced the top highty compatiiive...
del gat, mosule deper bys Faramatarryy depti, mage
libe Wary wern nicht eie deseriot
this, shown how approsimate F904x2 GEMMM ner
run, Jevottin setting is None serting...
Unilitpp meshar tamn of
shvceeging
regiter
by proetiin
muthed
G25835225, Micurologie
The realts shete (jemphiSikMteee digittovetyy Our opparinno apporiinnnaall insulls on cumminn bendhnan
with aps Stopwateh) nacon for
02-4554111, bibel Change, deser pricn:
def corigg stroanell by besictiaal e duoco ctii....
<i. 024110047, kabll inpotiiciaal proticns SMG Lu
Instruction:
A4 Instruction:
444 Intrustion:
aboce conteet, which sentencs connain placs "imleoking
asove cortect which funation contain the code un
above content, shat and disic option query 910041888
perforacce
"Especia
Figure 3: Three tasks in VAL Probing. The retrieval patterns are determined by the relative positions between the retrieval keywords and the information to be retrieved.
where (gi, a:) is sampled by prompting the powerful LLM with the segment and the instruction 1 ( represents the concatenation of the contained segments, and [ry] are randomly sampled from 128-token segments in C. Note that y instructs the LLM to make the question -anserr pair highly specific to the information provided in sy.
Integration and reasoning of information. Beyond utilizing each single segment, consider to generate question-ans -answerr pairs for information contained two or more segments. Following the setting of the minimum information unit above, we split full text C, into set of 128-token segments [s:]. then generate the qs. Og and L; accordingly,
(q: a.) ~ Prompting((ss] 4,; LLM), C. = @(Shuffee[ss)) [ry])),
where I, instructs the LLM to generate a multi-hop question- answer pair that requires the information within at least two segments [s:]. All segments [ss] and [r;]] are jointly shuffled, so the required segments may appear far apart in the context.
Context length balance and data mixture. To prevent length bias during IN2 training. we ensure the length of the long context L; is evenly distributed from 4K to 32K tokens. Such length balance strategy can be implemented with restricted sampling on [ry], according to Equation and 2 To alleviate catastrophic forgetting on short-context capabilities, we retain ~ 100 question-answer pairs with the original texts C2 instead of converting them into a longer context, and add some general instraction-tuning data from the OpenOrca (Lian et al. 2023) dataset.
Overall, our dataset for IN2 training contains 1.1M long-context data for the fine tne-grained information awareness (~~3%)), 300K long-context data for the integration and reasoming of information (~~ (~-1%). 150K short-context question-answe data (~9%), and 200K general instruction-tuni Appendix | contains the handcraft instructions for data generation. Appendix HH] illustrates some examples of our constructed long-context QA data, Appendix [A dejerribs the tiltering strategy to avoid data contamination for evaluation.
data (~11 (~11%).
2.2 Training Details
Using the training data constructed above, we further fine-tune the Mistral-7B-Instte In-trrttuct-- Jiang et al. 2023) to get our FILM-7B f'LL--i-the-MMiee
We perform IN2 training in the instruction- tuning paradigm: the long contexts and questions are used as instructions, and the loss on the answer parts are used to update the model. Appendix [] contains the system template used for formatting the training data. For parpe--arrmeters we set the global batch size as 128 and conduct one-epoch training with ~ 144 training steps. We use the cosine learning rate decay with le-6 maximum
http:://buggiigirssrcooommmaaait I-nstrructvtt

---

pocument Sentence Retrieval (Bi- Direction)
Code Function Retrieval diwaard)
Database Enfity Retrieval (Forward)
10
10-
25.
0.4
Mitre -24mltooo22
Milte 734rdtrow0
12.
P o
/ $
Relative Positions in 800 Sentences
Relalive Positions 800 Functions
Relative Positions in 750 Entilies
(a) Performance of FILM-7BB 7B, Dhnstall7B-Isttuutt--... 1, and -aairal-7B--nsttutt--O0.22
Document Sentence Retrieval (i-Direction)
Code Function Retrieval (Be ckward)
Dalabase Enti Retrieval (Forward)
14.
Longhion 76
33.58 Positions Entities
Relative
(b) Performance of FILM-7B, Alig--7i-- and LongAlign-13B-66
Decument Retrieval (Bi- Dirriiection)
Retrieval
Entity Retrieval
6.
15.
1.4
FLM-7910U5
Relative Positions 800 Sertencess
Relative Posttions in 800 Functions
Relative Positions 750 Entities
(c) Performance of Fr..M-B,, M2-chat7- and InternI. M22-chat-20P.
Figure 4: Performance of FILM-7B on VAL Probing and the comparisons with (a) Mistral, (b) Align, and (c) I22ernLM2. The X-axis is the relative position the context (~>2K tokens).
learning rate and 3% warm-up steps. The training process is conducted on 16 nodes of 8x80G A100 GPUs with the full sharding strategy and cpu offload strategy implemented by pytorch FSDP (Zaao One entire training process (for single FILM-7B model) consumes ~300 GPU days.
al. 2023
Long-Context Probing
In this section, we first show the preliminary evaluation of FILM-7B on the Needle-in -hale-iiintt---ystaade and discuss about the inadequacies of this probing task. Subsequently, to comprehensively evaluate the long-context information awareness of FmLM-7B, we introduce VArious Long-context (VAL) Probing. This includes three tasks that cover various context styles (document, code, and structured- data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).

---

Table 1: Quantified performances of various models VAL Probing.
Document Avg
Code
Database
All
Model
Gapl
Avg
Gapl
Avg
Gapl
Avg
Gapl
Mistral-7B-Imstruue-o001 Jiang et al.
2023
44.8
29.9
6.8
53.2
8.8
74.5
20.1
52.5
Mistral-7B-Instrucee Jiang el al.
2023
74.2
32.1
20.3
59.5
47.5
77.0
47.3
56.2
Long/Align-7B-64K (Bai et al.
2024
65.3
39.3
56.0
55.0
36.2
36,4
LongAlign-133B-64K (Bai et al.
2024
71.7
13.4
50.8
40.8
82.9
27.0
68.5
27.1
IntemL.M2-ccht-77 (Cai al. 2024
68.8
18.7
50.2
44.1
61.2
57.1
60.1
40.0
IntemL.M2-chat2205 CCai al. 2024
27.2
63.4
45.5
74.9
57.2
43.3
GPT-4-Turho (OpenAl 2023b
81.3
31.7
66.1
46.5
89.6
18.0
79.0
32.1
FtLM-7B (ours)
85.4
6.1
83.3
18.7
89.0
16.8
85.9
13.9
3.1 Near-Perfect Performance on Needle-ii-tthe--ayystaee Are We There Yet?
The Needle-in sennthhe-Haystar (lygi et al. 2023 Liu et al. 2024b) is widely used to assess how robustly model utilizes information positioned the long context. It reveals that even some powerful proprietary LLMs, such as GPT-4 and Claude 2.1 (Anthropic 2023), struggle to fully exploit the information within the long context.
We use the -heee-iiintt---ystaade task to preliminarily evaluate the long-context capability of FII.M-BB. Appendix [ demonstraaees that FII.M-BB has achieved near-perfect performance this task. This result is not surprising as recent open-source LLMs, such as LongAlign (Bai et al. 2024) and InternLM2 (Cai al.
2024 have also shown near-perfect performance on this task.
However, the near-perfect performance on Needle-in-the-lee Hheell-in-the-Haystack may overestimate the long-context capabilities of LLMs (Lei et al. 2024 Hsich et al. 2024). Specifically, we have the following two concerns:
Needle-in-the-Haystee employs a document-style context, which LLMs could be quite familiar with due to the pre-training on natural language corpora. may simplify the difficulty of information
The forward retrieval pattern in seeking the long context.
The "forward retrieval" means that the information being retrieved directly follows the retrieval keyword in long context. For example, the default question used in Beeel--in-the-Haystee is "What the best thing to do in San Francisco?" and the answer is contained "The best thing to do in San Francisco eat sandwich and in Dolores Park on a sunny day." The retrieved information "eat sandwich and just follows the retrieval keywords "best thing to do in San Francisco". According to the mechanism of induction head (Olsson an easily learned pattern for LLMs, thus less challenging for evaluating long context utilization (just like the observation of "reversal curse" (Berglund
2022). such following-up copying is
2024
Given these considerations, we suggest that performances on Needle-in-ther taalliin-th--HHyytakkk may not adequately reflect the long-context capabilities of LLMMs. Therefore, we propose VAL. Probing for more comprehensive evaluation involving various context styles and retrieval patterns.
3.2 VAL Probing
Our setrieval-based VAL Probing considers three context styles (document, code, and structured- context) and three retrieval patterns (forward, backward, and bi-directional retrieval). Each context in VAL Probing contains ~32K tokens. and each task contains ~3K examples. Figure briefly illustrates the contexts and retrieval instructions in VAL. Probing.
Document Sentence Retrieval (Bi-Direction). The contexts consist of numerous natural language sentences, and the instruction aims to retrieve single sentence containing given piece. The sentences are sampled from the abstracts of papers on arXiv This task follows the bi-directional
httpp://gghan..oom./aaaaddLLLL/ Hawpaa/eeeeooooooooteeeeeaaaaaaa NoedleInAHaystackk

---

Dpcument Sentence Retrieval (Bi Directiren)
Code Function Retrieval (Backw ard)
Database Entity Retrieval (Forward)
10
2.0
Potorancc
2.-
2..
0.2
Mate
64
14
30 2
Sentences
Relative Positions Funch
Relative Positions
Figure 5: Compare the performance of IN2 training and general instruction tuning (IT). Both two training process takes the same number of training instances (20% of the full data size, 300K examples). Compare with Mistral + IN2 training, the gains from normal instruction tuning are marginal and unstable.
Table 2: Quantified comparison between IN2 training and normal instruction tuning.
Document Avg
Code
Databuse
All
Model
Gapl
Avg
Gapl
Avg
Gapl
Avg
Gapl
Mistral-7B Instruct-vO.22 Jiang et al. 2023
74.2
20.3
59.5
47.5
47.3
56.2
Normal Instruction Tuning (Lian al. 2023
69.0
25.9
30.2
76.5
53,4
54.4
50.9
52.3
Information-Intene Training (ours)
82.9
11.5
74.5
27.7
83.5
31.6
80.3
23.6
retrieval pattern, the expected retrieval results contain words both before and after the given piece in the context. The evaluation metric is the word-level recall score.
Code Function Retrieval (Backward). The contexts consist Python functions, and the instruc- tion aims to retrieve the function name for given line of code within the function definition. The raw code functions are sampled from the StarCoder (Li et al. 2023c) datase P We randomly select three lines of definitions for each function. This task follows the backward retrieval pattern, the function name always precedes the definition. The evaluation metric the exact-match accuracy.
Database Entity Retrieval (Forward). The contexts contain lists of structured entities, each with three fields: ID, label, and description. The query aims to retrieve the label and description for a given ID. The entities are sampled from Wikidata [ This task follows the forward retrieval pattern, as the label and description follow the ID. We take relaxed exact-match accuracy as the metric: I score given if either the label or the description is exactly matched in the response, otherwise a score.
Experiments and Analysis
We assess the long-context capability of FILM-7B both probing tasks and real-world long-context tasks. Moreover, we investigate if the performance in short-context scenarios is affected.
4.1 Experimental Setup
Models. We mainly compare FILM-7B with long-conter open open-source models that have been trained with >32K context windows, including the Mistral (Jiang et al.) 2023), LongChat (Li et al.
2023ab. ChatGLM (Du al. 2022), LongAlign (Bai et al., 2024), Long Wanjuan (Lv al. 2024), Yi Yi et al. 2024) and Internl.M L.2 (Cai et al. 2024). We utilize the instruct/chat versions of these models as most our evaluation tasks are under the zero-shot mnstruction-follovin paradigm. We
also draw comparisons with popular proprietary LLMs such as GPT-3.5-Turbo (OpenAl 2023a and
http://wwgnnnntteeeodddeeeeeeees http://wwww.iiid....gwgwzizaaaa.

---

Table 3: Performances of various models real-world long-context tasks. Results of models with are reported in Bai et al. (2023) and Lv et al. (2024).
NatratieQAA Queper MaliFQA HotporQA 2WikiMQA MuSiQue GovReport QMSum MultiNews Avg
Close-Sarce
GPT-4-Turbo OpesAl
68.5
33.9
24.9
GPT.335 OpenAl 1023a
23,6
52.3
51.6
37.7
26.9
29.5
23.4
26.7
35.0
Open Source
L.ngCha-v1.--7..322
27.7
31.5
20,6
30.8
22.7
26,4
25.3
ChaGLLM2-6B 6B-32K"
et 2023
21.1
31.5
46.2
25.3
20.8
9.8
32.4
21.0
26.5
26.4
LongAlign-7B-66H [Ban ct 20M4
18.7
49.1
28.6
23,4
12.5
30.6
23.7
27.5
27.5
Hikrrl-7B-lnstrus--1 Jiang
2023
38.8
42,9
31.2
27.5
22.4
26.5
Mistral-7B 7B
Danraal---Inttuug--o0 Instruct-n0.22
Jiang
23.5
33.8
45.9
42.4
24.3
20.8
33.3
24.8
26.8
30.6
Yi-6B-20UK*
at
X024
12.4
26.4
36.8
46.6
40.4
25.8
29.3
20.7
27.1
29.5
ChatGi M3-661-12K
2023
$5.3
I2sern_.22chhr
Caretal
A0M
24,4
35.4
50.2
52.4
48.2
30.5
33.6
25.3
29.0
36.5
Internl. Insernl.M-77B. M2-7B-Loog Wanjurn
et
2024
29.9
39.6
50.2
53.7
42.3
32.1
33.0
25.5
27.8
37.1
FILM-7B
Table 4: Model performances on few-shot learning tasks.
TriviaQA SAMSum Average
GPT4-Turke (OpenAT
2023h
77.0
91.7
39.7
69.5
Mistral-7B (esaral-7--nnsmmet--e Instract-vO.
Jiang al.
2023
71.0
84.5
35.8
63.8
FILM- 7B (ours)
76.0
90.0
39,5
68.5
GPT-4-Tuuro (OpenAl
2023b). All models and tasks employ greedy decoding. For probing tasks, primarily compare FILM with LongAlign and InternLM2 series, these models have shown near-perfect performances on Hoolle-intth--Haystee
Real-world long-context tasks. We take 9 tasks from the LongBench (Bai al. 2023) collection to
evaluate the long-context capability on real-world scenarios. These tasks encompass long-document
question answering (NarrativeQA (Kočiský al., 2018). Qasper (Dasigi et al. 2021) and Multi-
FieldQA (MultiFQA) (Bai et al. 2003)), multi-document multi-hop reasoning (HotpotQA YYngg et al.
2018). 2WikiMultihopAA (2WikiMQA) (Ho al. (2020) and MuSiQue (Trivedi et al., 2022 and
long-context summarization (GovReport (Huang et al. 2021), QMSum (Zhong et al. 1021] and MultiNews (Fabbri et al. 2019)). We employ the middle truncation strategy in LongBench to limit the input within 32K tokens. We report ROUGE-L Lin 2004) for summarization tasks and F1 scores for other tasks. The evaluation metrics are computed using the official evaluation scripts
Despite these QA and summarization tasks, we also conduct evaluations on few-shot learning tasks, in which the contexts could also be extremely lengthy if there many "few-shot" examples presented in the context. We take three in-context learning tasks from LongBench, including TREC (Li & Roth 2002) for few-shot classification, TriviaQA (Joshi al. 2017) for few-shot QA, and SAMSum (Gliwa et al.
2019) for few-shot summarization.
Short-context tasks. We select 8 short-context tasks commonly used for evaluating the general capabilities of models. These include MMLU (Hendrycks et al. 2020). BoolQ (Clark et al. 2019). RACE-High (RACE-H) (Lai et al. 2017). Commonsense(A (CSQA) (Talmor et al. 2019 D. ARC-
Challenge (ARC-C) (Clark et al.J 2018). HellaSwag (Zellers et al. 2019), GSMSK (Cobbe et al. 202T). and MATH (Hendrycks for MATH, and O-shot for other tasks. We utilize the lm_eval (Gao et al. 2024) for the evaluations on MMLU, BoolQ, RACE-H, ARC-C and HellaSwag, and use the evaluation scripts from An et al. (2024) for other tasks.
202TT. We use 5-shot for MMLU, 8-shot for GSM8K, 4-shot
4.2 Main Results and Analysis
FILM-7B significantly mitigates the lost-in-the-middle results for both FILM-7B and the backbone model, problem. Figure 4a presents the probing In all three probing tasks within VAL Probing, the vanilla Mistral model experiences substantial information loss the middle positions in the long contexts. In contrast, our FILM-7B model consistently exhibits robust
hutgs//gthhbboom///UUUUU/LLnggg
8

---

Mistral78-Inntrrue--..
FILM-78
Perfomancc (%1
87.7
70.0
75.3
83.6
79.1
39.3
29.2
$5.9
52.5
40.4
44.5
8.7
11.3
MMLU
BoolQ
RACE-H
CSQA
ARC-C
HellaSwag
GSMBK
MATH
Figure 6: Performances of FILM-7B and the backbone model on short-context tasks.
performance across different positions within the whole context. This stark comparison illustrates that the lost-intthe-middle problem can be effectively addressed using our IN2 training.
FTLM--BB achieves performance comparable to, or even outperforming, that GPT-4-Turbo. Figure 1 illustrates the comparison between FILM-7B and GPT-4-Turbo on our probing tasks. Beyond a qualitative comparison between the performance curves of two models, we quantify the long-context performances VAL Probing using two metries:
Average score (Avg). We compute the average performances across the entire context length, reflecting the overall long-context utilization.
Min-max gap (Gap). We calculate the differences between the maximum and minimum per- formances Figure A smaller performance gap signifies greater robustness across different
positions.
Table I presents the quantified performances VAL Probing. It reveals that FILM-7B has compara- ble performance with GPT-4-Tum Turbo on the database probing task, and exhibits better robustness in document and code probing tasks. These results indicate great potential for the development of open-source long-context models to close the gap with proprietary models.
IN2 training can effectively alleviate the lostiie-the-middle problem, while the normal instruc- tion tuning cannot. Considering that FILM-7B additionally uses instruction-tuning-eo effectiveness of IN2 training, here we present more
data
for post-training, to further demonstrate controlled experiments to compare IN2 training and normal instruction tuning under the same train- ing data size. Specifically, for both IN2 training and normal instruction tuning, we take the same backbone model (i.c., Mistral-7B. Bosdral-7B-Instructtaa Instruct-v0.2) and the same number of pust-training instances (20% of our full data size, 300K examples). The data for normal instruction tuning are randomly sampled from OpenOrca (Lian et al. 2023). Comparisons shown in Figure 5 shows the performance curves on VAL Probing after two training processes, and Table 2 contains the quantified results. These comparisons clearly demonstrate that IN2 training effectively alleviate the lost-in-the-mice problem while the normal instruction tuning cannot.
VAL Probing presents a more challenging test suite for long-context models. Figure 4b and 4c show the probing results of LongAlign and InternI two state-of-the-an long-context models. Despite their extended training context windows. these models still encounter the lost-in-the tet--nithh-middlle middle problem. This is particularly noteworthy given their near-perfect performance on the Needle-in -hethe-intthe- Haystack task. This comparison suggests that VAL Probing provides more challenging evaluation for long-context models.
In particular, the results on document and database tasks in VAL Probing demonstrate clear com- parisons with Needle-in-ther thyy-----t Compared to Needle-in-the -ialeeinntth--Hystcck which uses forward retrieval on natural language context, the document task employs natural language context but with bi-directional retrieval, and the database task uses forward retrieval but with structured-dat context. These comparisons highlight that both context styles and retrieval patterns significantly contribute to the hardness of the probing tasks.

---

Training on synthesized long-context data effectively generalizes to real-world scenarios. Ta- bleB] and I contain the results on various real-world long-context tasks. shows that FiLM-7B also significantly improves the performance of the backbone model in real-world long-context scenarios. Moreover, also achieves SOTA-leve performances on these tasks among ~7B size open-source models. Notably, the long contexts used in IN2 training are all synthesized from short segments. These improvements suggest that the long-context capabilities learned from the synthesized data can be successfully applied to real-world tasks.
FILM-7B maintains the performance on short-context tasks. Figure 6 illustratee the perfor- mances of FILM-79 7B and the vanilla backbone model on short-context tasks. It reveals that the overall performances on short-context tasks are almost comparable with minor variances. These results confirm that FILM-7B does not compromise the short-context capabilities of the backbone model.
Analysis training strategies. We specifically interested in investigating the impact of the following two training strategies: applying the sliding window and adjusting the position encoding. Due to the page limitation, we provide these further ablations and analysis in Appendix C
5 Related Work
Long-context LLMs. Recent research has significantly contributed to the exploration of training
large models with extended context windows (Jiang et al. 2023 Du al. 2022 Li et al. 2023a
Team et al. 2023; Team, 2023; Xiong et al.) [003] Song et al. [023 Tworkowski et al. [0244] Al
al. 2024 Cai et 2024). There are primarily two directions the development of of long-context
LLMs. (1) Data engineering, which emphasizes the construction of long-context data for training
the LLMs. This includes data balancing (Fu al. 2024), ) data order arrangement (Shi al. 2023),
instruction data collection (Bai et al.) 2024), and data quality measurement (Lv et al. 22024). Our
IN2 training can be categorized into this field. (2) Effective and efficient training, which investigates methods optimize the training of long-context model. This encompasses the design of position
encoding (Chen et al.] 2023a;; Liu et al. 2023 Peng et al., 2023b Ding et al.) (2024), batching
strategy (Bai et al., 2024), parameter-efficient training (Chen et al. 2023b), and the development of
new model architectures (Peng et al. 2023a Gu & Dao 2023).
Long-context evaluations. Existing benchmarks for evaluating long-come long-context models can be divided into two categories. (1) Real-world benchmarks that assess general long-context capabilities (e.g.. long-context QA, summarization, and language modeling). such as Narrative (Kočiský et al.
2018). LongBench (Bai
2023). ZeroSCROLLS (Shaham al. 2023). L-Eval (An al.
2023). Loogle (Li et al.] 2023b), co Bench (Zhang et al. (024). and a series of work on perplexity
evaluation (Beltagy et 2020 Roy et al. 2021 Press et al. 202T Chen al. 2023a Liu et al.)
2023, Peng et al. 2023b Chen et al. [023b Ding et al. 2024 Mohtashami & Jaggi 2024). (2)
Probing tasks that provide a more concise reflection of the long-context utilization across different
context lengths and positions. These include Heedle-in-the-HHasstack passkey retrieval (Mohtashami
Jaggil (204)), synthesized document QA (Liu et al. (2024)), S3Eval (Lei et al. (024)), Discovery (Li
al. (2024), RULER (Hsich et al. 2024). and the VAL Probing proposed in this study. Among these
probing tasks, our VAL Probing is the first to explicitly incorporate a variety of retrieval patterns.
Conclusion
This work introduces IN2 training overcome the sot-in-the-middle problem. By applying IN2 training on the open-source model, our FILM-7B exhibits significant improvements on probing tasks and real-world long-context tasks while does not compromise the short-context performance.
The bold numbers in Table 3 are SOTA-level results among 7B open-source models. Specifically, for each task, we bold the highest result and the results within margin to the highest one (0.5 for summarization tasks and 2.0 for others).
10

---

Acknowledgments
We thank all the anonymous reviewers for their valuable comments. Shengnan An and Nanning Zheng were supported in part by NSFC under grant No. 62088102.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Beeker. Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang. Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang. Heng Li, Jianghheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307. 1088, 2023.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian MaanGGang Lou, and Weizhu Chen. Learning from mistakes makes Ilm better reasoner, 2024.
Anthropic. Model card and evaluations for claude models, 2023, URL https://ww-files
anthropic. wtt./oom//ooooee httmm/ooouutooo/mnnaaao httpo//eeoooooomeeee/e/eeeeeeeee
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng. Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401. 18058, 2024.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint a0x.::2004005150, 2020.
Lukas Berglund, Meg Tong. Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a", 2024. URL
https://arxiv org. abwaabbs....e 12288
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Intern1m2 technical report. arXiv preprint arXiv:2403. 2024.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 2023a.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- glora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2023b.

---

Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 1924-2936, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: wwtp::10330050 180.05547, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Pradeep Dasigi. Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-ssekking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 4599 4610, 2021.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending I1m context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074. Association for Computational Linguistics, 2019.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot wttp:///eeoo//o///ore./e.
language model evaluation, 07 2024. URL
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset for abstractive summarization. F.MNLP-JJCNIPP 2019, pp. 70, 2019.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Xanh Ho, Anh-Khoo Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, 2020.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler; What's the real context size of your long-contex, language models?, 2024.
12

---

Luyang Huang. Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In 2021 Conference of the North American Chapter of the Association for Computational I Linguisties: Human Language Technologies, NAACI-HLT 2021, pp. 1419-1436. Association for Computational Linguistics (ACI.), 2021.
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284-299, 2023. doi: 10.1162/tacl_ a a_00577. h httpss://aclaatthllllg///o/ http:///c.aattollooooo////////// wrg./223..
http://oooooooo..r////.......11. tac1-1.17
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0683 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601- 1611, 2017.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguisties. 6:317-328, 2018.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.
Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. S3eval: A synthetic, scalable, systematic evaluation suite for large language models. 2024.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source LLMs truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. URL https
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint 1tt.::311.049933 2023b.
Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023c.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long- Ilms struggle with long in-context learning, 2024.
Xin Li and Dan Roth. Learning question classifiers. COLING 2002: The 19th International Conference on Computational Linguistics, 2002.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:// huggingface cor0ppen-Orra/ppenorrall 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summarics. In Text summarization branches out, pp. 74-81, 2004.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2024a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.
Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, 2023.
13

---

Kai Lv. Xiaoran Liu, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwanjuan: Towards systematic measurement for long text quality. arXiv preprint arXiv:2402.1358. 2024.
Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers.
Advances in Neural Information Processing Systems, 36, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
OpenAL Gpt-3.5 turbo fine-tuning and api updates, 2023a. URL https://openaa.coo//bl..g 3pt-3-5-ture S----turbo-fine-tumee tuning-anddd and- andd-aap--uns updates
OpenAI. Gpt-4 technical report, 2023b.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 14048-14077, 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2023b.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama Open foundation models for code. arXiv preprint 2r0vv:2388. 19500, 2023.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot bench- mark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, 2023.
Vatsal Sharan, Sham Kakade, Percy Liang, and Gregory Valiant. Prediction with short memory. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1074-1087, 2018.
Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations, 2023.
Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jachyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained llms. In The Twelfth International Conference Learning Representations, 2023.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Tyyer. Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 807-822, 2021.
14

---

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ. A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-44 -458, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N19914933 URL https V
MosaicML NLP Team et al. Introducing mpt-30b: Raising the bar for open open-source foundation models, 2023.
Together Team. Together 32k, 2023. URL http::///wusiiggffeee..oooooooep
LLaMA-2- 7B-32K
Together.AL. Llama-2-7b-233 12r/--3-2--nnstree and fine tuning for Ilama-2 models with together api, 2023. URL https:///www together. ai/blog/ 1aalooglllaaa---eeeee 2-kb-32k-innn 32k-instruct)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (tX.ixv2277.0228265 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 00:539-554, 2022.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mitoš. Focused transformer: Contrastive training for context scaling. Advances Neural Information Processing Systems, 36, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang. Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, 2023.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4791-4800, 2019.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extending long context evaluation beyond 100k tokens, 2024.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint ar 1277, 2023.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based based multi- domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 5905-5921, 2021.
15

---

This is the Appendix of the paper. Make Your LLM Fully Utilize the Context.
A Data Filtering Strategy
To avoid data contamination for the evaluation stage in Section we apply pre-filtering strategy during sampling the raw texts for constructing the dataset of TN2 training. Specifically, during sampling C, for generating data, if the sampled C, has a 10-gram overlap with any example in all of our evaluation data (including probing tasks, real-world tasks and short-context tasks), it will not be used for neither generating question-anse answer pairs nor serving as the random segments [ry]-
B Performance on Neelle-n-thh--Hastade
Needle In HaySlack
Tolen Linit
Figure 7: Performances of FILM-7B on Needle in-the- the-Haystack.
Figure 7 shows the performance of FILM-7B on Needle-in-ther theel--i-thh-HHyytackkk It shows that FILM-7B has achieved near-perfect performance on Needle-in-the Meedle-in-the-Hayee Haystack
tellliinttt--Haystcck within its 32K context window.
C Training Strategy Analysis
Experimental results in Section 4.2 demonstrate the feasibility of IN2 training. We aim to explore fur- ther into enhancing the effectiveness and efficiency of IN2 training, partieularly from the perspective of training strategies. We are specifically interested investigating the impact the following two training strategies: applying the sliding window and adjusting the position encoding. Considering the high cost of training, the following experiments use 20% of all training examples.
Models using sliding windows cannot effectively capture the long distance information. Our experiments involving Mistral models, as shown in Figure 4a| reveal that the performance Mistral- 7B IB-Inntruct-vO.1 is awful when the information is positioned that Mistral-7B--Insrruce 1 employs the sliding window strategy while Mistral-7B--Instructt does not. Consequently, the pestin-the-miidlee problem under the sliding window strategy. We conduct the following two experiments with 4K sliding window during training:
long distance. It's worth noting
interested in determining whether IN2 training can still alleviate
Apply the sliding window in both pre-training and IN2 training. We take the Mistral-7B- Instruct-vO.1 the backbone model and conduct IN2 training with the same window size (4K).
Apply the sliding window only during the IN2 training. We take the the backbone model and additionally apply 4K sliding window during IN2 training.
as
Figure g illustrates the performances models with sliding windows. shows that in both two settings with sliding windows, the performances drop dramatically when the distance between the
16

---

Dpcument Sentence Retrieval (Bi- D-recirrcin)
Code Function Retrieval (Backw and)
Database Entily Retrieval (forward)
10
29.
Potormarco
0.7
FRM (280)
11.0-75 12% 5W (7402
022
$ 2 Relative Posttions BO0 Sentences
3.9 Relative Positions Entitee
Relstive Posttions 900 Funch
Figure 8: Performance of FILM-7B with a 4K sliding window (SW). PT-IN2: apply the sliding window in both pre-training and IN2 training. IN2: apply the sliding window only in IN2 training.
Table 5: Performance of FILM-7B with different RoPE base during IN2 training.
Document
Code
Database Avg
All
Model
RoPE Base 0
Avg
Gapl
Avg
Gapl
Gapl
Avg
Gapl
1.0 x 10 (default)
82.9
11.5
74.5
27.7
83.5
31.6
80.3
23.6
2.0 x 10
83.9
9.3
79.8
27.1
87.7
13.2
83.8
16.5
FILM-7B (20%)
1.0 x 107
83.7
7.6
81.7
18.4
89.4
16.8
84.9
14.3
1.0 x 10
84.6
6.6
81.4
22.3
87.7
13.2
84.6
14.0
retrieval question and information longer than the sliding window size. reveals that the sliding window strategy greatly hurts the long-context capability of models.
Training with higher information intensity requires a larger RoPE base 0. The training stage in Section 2 follows the RoPE settings configured for the backbone model. Previous studies context extension suggest that training with an extended context length necessitates a larger RoPE base 8 (Roziere al. 2023: Xiong et al., 2023 Cai et al. 2024). In the case of our IN2 training, the context length remains unchanged, but the information density is significantly changed to be more uniform. There high-level similarity between context-extension training and our IN2 training: both aim to enhance the model's capability to perceive information from a wider range of input positions compared to the previous training stages. Consequently, carry out experiments to investigate whether the experience from context-extension training could also benefit our IN2 training. 5 the results with increasing the RoPE base from 1.0 10 to 1.0 x 10 It shows that increasing the default RoPE base 0 of the backbone model leads to better performances on VAL Probing. We suggest to use a 10 times of the default RoPE base & to conduct IN2 training.
D Implementation and Reasons for Segmentation
Algorithm Iillutrrates how we segment a raw text into 128-token segments. We set the ~ 1-28ttoken segment as the minimum information unit due to the following consideration:
If the segment contains too few tokens (e.g., 16 tokens or 32 tokens), it might not contain enough information for asking a meaningful question.
If we set a large threshold for segmentation (e.g.. 1024 tokens or 4096 tokens), most raw texts will just contain one segmen [I] thus affecting the construction of QA pairs that require the integration and reasoning of information.
Moreover, we do not just use full raw text to generate a question with GPT-4, as we are concerned about whether GPT-4 will also more focus on the head and the tail of the text. It could result in a local bias for training: the answers are always placed on the boundaries between two segments containing consecutive information text.
The raw texts in realnewslike ike have average length of ~600 tokens with the Mistral tokenizer.
17

---

Algorithm Implementation of Raw Text Segmentation
Given:
C;: The raw text;
Tokjenizer()): The tokenizer of the model;
1 = 128: The minimal length of each segment;
Return:
S. ( & A set of segments;
1: S.
2: P. = =Split(,, 'in'
3: temp_list =[]
temp_ hmmplength=00
5: for p/ P: do
6: length Len -en(((oeeeeeeeeee(())))) temp_list.appppp cmmp_iist.pppend()) temp_lengte += length
7: temp_length >= then
8: lemp_segment n\n' join(temp_lixg
S.appennd(emmp_seemen))
10:
temp_list=
11: temp length = 0
12: else
continue
14: end if
15: end for
16: temp_list not empty then
17:
= temp_e_ie list
temp_ cemp_segment= "\n' join(temp_lin
19: w.append((emp__eggeent
20: end if'
21: return S.
Table 6: Performance of Fr.M-7BB with different training data sizes for IN2 training.
Document Avg
Code
Database
Model
Data Size
Gap.
Avg
Gap
Avg
Gap.
Avg
Gapl
100%
85.4
6.1
83.3
18.7
89.0
16.8
85.9
13.9
50%
84.2
13.3
80.5
89.7
15.3
84.8
16.7
FILM-7B
20%
82.9
11.5
74.5
27.7
83.5
31.6
80.3
23.6
10%
84.3
11.3
75.2
31.8
82.3
32.7
80.6
25.3
1%
76.2
18.8
63.3
48.0
70.9
36.7
70.1
34.5
Based on the above considerations, we use the ~~128-token segment. Due to the high cost on data generation, do not conduct further ablations this design choice.
E
Data Scaling Trend
Table 6 shows the performance of FII.-7BB with different training data sizes for IN2 training. Generally, with the data size increasing, the average performance increases and the performance variance in different positions decreases. Such trends more significant code and document probing tasks. Note that training data for IN2 training are almost from natural language corpus. It indicates that increasing the training data size can better help the generalization of long-contess capability on different context styles.
Performance on RULER Benchmark
RULER (Hsieh et al. 2024) is a synthetic benchmark that evaluates the effective context length of the long-context LLMs. It revealed that while all existing long-context models with <7B sizes claim context size of 32k tokens or greater (except for Llama3), none of them can effectively handle sequence length of 32K by exceeding qualitative threshold, Llama2-7b performance at 4K (85.6%).
18

---

Table 7: Performances (%) of <BB models RULER benchmark. The performance exceeding the threshold (i.e., Llama2-7B on 4K length) is underlined.
Model
Claimed
Effective
4K
8K
16K
32K
64K
128K
Avg.
Llama2-7B Touvron et al.
2023
4K
85.6
LongChat-7B (Li al.
2023a
32K
<4K
84.7
79.9
70.8
59.3
0.0
49.1
Together-7B TTgether.Al.
2023
32K
4K
88.2
81.1
69.4
63.0
0.0
0.0
50.3
Phi3-3B Abdin al.
2024
128K
4K
86.7
78.1
75.6
70.3
58.9
43.3
68.8
LWM-7B (Luu al.
2024a
IM
<4K
82.3
73.7
68.1
72.8
ChatGLM3-615 (Du al
2022
128K
4K
87.8
83.4
78.6
69.9
56.0
42,0
69.6
Mistral-7B
Jiang el al.
2023
32K
16K
93.6
91.2
87.2
75.4
49.0
13.8
68.4
FILM-7B (ours)
32K
32K
92.8
88
86.9
70.1
27.1
75.5
Document Sentence Retrieval (Bi- Difeeirrctinn, -64K)
Performance (86)
8.5
(Gust
0.4
1450
1500
1685
1790
1200
Relative Positions in 1800 Sentences
Figure 9: Performance of FILM-7B 7B on 64K context length. The position embeddings are extended
through YaRN.
Table shoows that FILM-7B is the first <7B size model that achieves 32K effective context length. Note that for the evaluation >32K lengths (i.e., 64K and 128K), YaRN (Peng et al. 2023b) to extend the position embeddings without further fine-tuning.
G Position Extension Suffers Lost-in-the-Middlee
Some existing studies focused on how to extend the context window without further training, such as YaRN (Peng et al.) (2023)). We tried to extend the context window of FILM-7B from 32K 64K with Yam and evaluated whether the model can still overcome in-the--middi problem under the extended context length. Figure shows that although FILM-7B outperforms vanilla Mistral-7B on the extended 64K context window, both two models severely suffer the lost-in-the-midan problem. Such an observation indicates that simply extending the position embeddings of the model may not make the model fully utilize the extended context window.
19

---

H Training Examples for IN2 Training
Example 1: Fine-Grained Information Awareness (1)
Segment:
Last summer T took cross-countryy road trip with group of people driving different types of electric vehicles. It was amazing experience, and our trip, adventures, and misadventures were portrayed in the documentary Kick Gas. That experience gave the idea take my road trip experience the freedom traveling alone on electric motorcycle. My goals were simple: ride the great motorcycle roads America, meet other electric motorcycle riders, and visit friends...
Question:
What were the primary goals of the individual's solo road trip electric motorcycle?
Answer:
The individual's 's primary goals for the solo road trip on an electric motorcycle were to ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends.
Example Fine-Grained Information Awareness (2)
Segment:
Another addition the kernel is the ar5523 driver, which started over five years ago for the Atheros USB chipset same name. Extensions supporting more chips and WLAN adapters were added a number of other drivers; the bremsmac WLAN driver, for example, now supports the BCM43224 Broadcom chip. while the r12800usb RaLink driver supports the Sweex LW323 USB WLAN adapter... Question:
What new addition to kernel supports BCM43224 Broadcom chip?
Answer:
The new addition to the kernel that supports the BCM43224 Broadcom chip is the bremsmac WLAN driver.
Example 3: Fine-Grained Information Awareness (3)
Segment:
The Arlington County Board plans to Saturday afternoon on giving Amazon $23 million and other incentives to build a headquarters campus Crystal City, but only after hearing scores of northern Virginia residents and advocates testify for or against the project. The five-member board is expected to support the plan, which was announced amid much hoopla on Nov. 13. The proposed county incentives are part of an agreement which Amazon would occupy significant office space and bring least 25,000 high-paying jobs to Arlington coming years. Opponents hope to postpone the vote until after additional public hearings, where they want representatives of the online retail giant to answer questions directly from anyone in the community.
Question:
What is the Arlington County Board's plan regarding Amazon's proposed headquarters in Crystal City?
Answer:
The Arlington County Board is planning to vote on providing Amazon with $23 million and other incentives to construct headquarters campus in Crystal City. However. this vote will only take place after listening the opinions of numerous residents and advocates from northern Virginia. The board, which consists of five members, is anticipated endorse this plan. The proposal includes Amazon occupying a significant amount office space and introducing at least 25, ,000 high-paying jobs to Arlington the forthcoming years.
20

---

Example 4: Integration and Reasoning of Information (1)
Segment 1:
Cloud Cruiser's 's utilization analyties application enables users to manage and optimize public, private and hybrid cloud usage and spending
Segment
This reduces the risk of organizations investing too much or too little in IT, eliminates unused capacity and frees up valuable ITT' resources for new value-adding projects
Segment 3:
As Cloud Cruiser customer, have seen firsthand the value that Cloud Cruiser's technology creates by enabling HPE Flexible Capacity meter bill for usage of on-premise infrastructure a pay-as-you-go model...
Question:
What benefits does the acquisition of Cloud Cruiser bring to HPE and its Flexible Capacity business?
Answer:
The acquisition of Cloud Cruiser enriches HPE's Flexible Capacity business by providing a utilization analytics application that allows users to manage and optimize their cloud usage and spending. Cloud Cruiser's & software also facilitates the metering and billing for customers' IT consumption. This enables HPE offer its customers an on-premise physical IT infrastructure at the cost of cloud services, effectively rental agreement. Moreover, reduces risk organizations either under or over investing IT, eliminates unused capacity, and frees up IT resources for new projects...
Example Integration and Reasoning of Information (2)
Segment 1:
Julie Annette Wasden, 45, of 38 Roles St., Haines City, was charged with retail theft Beall's 5998 Cypress Gardens Blvd Segment
Katherine D. Collazo Santos, 18, of 108 Hilltop Drive, Winter Haven, was charged with retail theft at Macy s at 700 Third St., S.W.
Question:
Who was charged with retail theft Beall's and who was charged with the same crime Macy's?
Answer:
Julic Annette Wasden and Katherine D. Collazo Santos
Example 6: Integration and Reasoning of Information (3)
Segment
A court presided by Magistrate Joseph Mifsud explained its judgement of 17 August, 2016 in The Police v Eebis Getu, that the charges of false criminal report and calumnious accusations may be factually similar, however, from legal point of view they are two separate and distinct charges. The accused, Echis Getu was charged with having filed false police report and with obeying legitimate order, She admitted these charges
Segment 2:
In passing judgement the court took into consideration that the accused admitted to the charges immediately and also that she did what she did in order to be with her husband in Malta. Magistrate Mifsud referred what Pope Francis said last June, where today2019s information technology brings suffering others instantly, but we also become immune to tragedics and sufferings..
Question:
What were the two charges Eebis Getu admitted to, and what was her reason for committing these actions according the court's judgement?
Answer:
Filing a false police report and not obeying legitimate order; to be with her husband in Malta.
21

---

I Prompts For Data Generation and Training
Example 7: Prompt For Equation 1
Generate one question and the answer from the given context. The question should be highly specific to the information provided in the context. It should not be general question that suits any context.
Rules follow when generate the question:
The question should fully answerable
information present given context.
2. Make sure the question is clear and unambiguous.
3. Phrases like 'based on the provided context', "acordding to the context', ctc, are not allowed to appear in the question.
Rules follow when generate the answer:
1. The answer must use the information provided in context.
2. Do not just copy words from the context. Answer the question in your own words.
### Context #H#l:
### Question ####: (complction)
Example 8: Prompt For Equation 2
Generate question and the answer from the given context. The context contains several pieces. Answering the question should require reader to make multiple logical connections inferences using **at least two pieces**
Rules to follow when generate the question:
1. The question should be fully answerable from information present in given context.
Make sure the question is clear and unambiguous.
Phrases like based on the provided context', "according to the context', etc, are not allowed to appear the question.
Rules to follow when generate the answer:
1. The answer must the information provided in the context.
Do not just copy words from the context. Answer question your own words.
### Context ###::
# Piece 1: 8)
# Piece 8
### Question #ill: [complction)
Example 9: Training Template
Input:

 Below is a context and an instruction. Based on the information provided the context, write a response for the instruction.
### Context:
C.
### Instruction: 

Output:
Limitations
The main limitation of this work lies in the insufficient analysis on the choices of training hyper- parameters (e.g., learning rate, batch size, training steps and warm-up rate), data construction settings (c.g., data size and data mixture rate) and backbone models (e.g., model sizes and model architectures).
22

---

We just take the commonly used settings for IN2 training without further searching and analyzing. Intuitively, we believe the change of these settings will not affect the feasibility of IN2 training.
K Broader Impacts
This work used pre-trained large language models (i.e., GPT-4 and Mistral-7B) during data construc- tion and training. Therefore, our model may inherit the potential risks these pre-trained large language models in terms of ethical and safety issues.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected generalize to other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images taken in low lighting. Or a speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: Section 2 experiments.
3 and 4
fully disclose all the information needed to reproduce
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models. it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: The model and evaluation data will be released. Due to internal data release policies, the training data will not be released soon.
25

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cccppbbll
Saiides/CoeSubmmsssioooo..
for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines pattts://nippe cc/public/ guides/Cooeeuumi ssionPolicy) for more details.
reproduce
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detaill
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 2 and provide all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We show the error bars for the results on probing tasks. For other tasks, there is only one final performance for each task as we use greedy decoding for evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
26

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
See Section 2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurips.cc http:///wrppp..ccooooooccccceeet
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section K
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
27

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for moni- toring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We provide the usage guidelines and disclaimers in our github repo (not released now).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper and the github repo (not released now) include all necessary citations, URLs, acknowledgements and licenses.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ com/datasets datasets has curated licenses for
some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The model and probing tasks (to be released) are incorporated with detailed docu- mentations.
Justification:
Guidelines:
28

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were
obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
29


Reviews:
[{"review_id": "HOti9KYONG", "reviewer": "Reviewer_HMgi", "summary": "This work tackles an essential problem in LLMs, called lost-in-the-middle, that causes the lack of information in the middle of lengthy input. As a data-driven solution, the authors propose a new training approach, INformation-INtensive (IN2). IN2 aims to enhance the use of context information in an unbiased position by shuffling the clue part of the data during training. When creating the shuffled data, the authors use GPT-4-Turbo to maintain its grammatical and semantic appropriateness. Furthermore, the authors propose a new model, FILM-7B, additionally trained Mistral-7B on a dataset created by their proposed IN2. To comprehensively evaluate FILM-7B's long-context handling ability, the author introduces VArious Long-context (VAL) Probing. VAL Probing results show that FILM-7B can outperform even GPT-4-Turbo by avoiding being trapped by lost-in-the-middle. Experimental results on long-context tasks from the LongBench collection and short-context tasks, MMLU, BoolQ, RACE-High, CommonsenseQA, ARC-Challenge, HellaSwag, GSM8K, and MATH show that FILM-7B outperforms Mistral-7B models.", "questions": "Currently, the evaluation is conducted in a zero-shot setting. Is there any possibility that FILM-7B outperforms proprietary models like GPT-4-Turbo on a few-shot setting by using its ability to handle long contexts equally?", "limitations": "Even though IN2 mitigates the lost-in-the-middle problem, experimental results show that this problem is still left in some tasks.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a new training method, INformation-INtensive (IN2), that can make trained LLMs handle long-context with mitigating the influence of lost-in-the-middle.\n- By using IN2, the authors train Mistral-7B to make a new model, FILM-7B, than can handle long-context.\n- The authors provide a newly constructed evaluation task, VArious Long-context (VAL) Probing, that covers various long-context tasks.\n- The evaluation result on VAL Probing shows the effectiveness of IN2 based on the improved performance of FILM-7B.\n- The evaluation results on long-context and short-context tasks also show that FILM-7B outperforms FILM-7B.", "weaknesses": "- From the viewpoint of Scaling Laws, comparing Mistral-7B and its further trained model FILM-7B is unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback on our work. We hope the following response could further address your concerns.\n\n> W1: From the viewpoint of Scaling Laws, comparing Mistral-7B and its further trained model FILM-7B is unfair.\n> \n> R1: We present further experimental results to compare IN2 training and normal instruction tuning (IT) **under the same training data size**. These results demonstrate that **IN2 training can effectively alleviate the lost-in-the-middle problem while the normal instruction tuning cannot**. In the following table, both Mistral+IN2 and Mistral+IT have the same training data size (20% of our full data size, ~300K examples). Furthermore, the PDF file in our general response illustrates the performance curves of Mistral+IT. Compared with Mistral+IN2, the gains from normal instruction tuning are marginal and unstable. \n> \n>   |  | Doc| | Code| | DB| | All| |\n>   |---|---|---|---|---|---|---|---|---|\n>   |Model|Avg|Gap&darr;| Avg|Gap&darr;| Avg|Gap&darr;| Avg|Gap&darr;|\n>   |Mistral|74.2|32.1|20.3|59.5|47.5|77.0|47.3|56.2|\n>   |Mistral+IT|69.0|25.9|30.2|76.5|53.4|54.4|50.9|52.3|\n>   |**Mistral+IN2 (ours)**|**82.9**|**11.5**|**74.5**|**27.7**|**83.5**|**31.6**|**80.3**|**23.6**|\n\n\n> Q1: How well does FILM-7B perform on few-shot setting?\n> \n> A1: We conduct further evaluations on the few-shot learning tasks in LongBench, including TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). Although FILM-7B does not outperform GPT-4-Turbo, it shows that **IN2 training can significantly close the gap with proprietary models on few-shot settings**. These results demonstrate that even though the surface forms differ across various long-context scenarios, we believe that **the underlying capability to comprehend long-context information is transferable**.\n>\n> |Model|TREC|TriviaQA|SAMSum|Avg.|\n> |---|---|---|---|---|\n> |GPT4-Turbo|77.0|91.7|39.7|69.5|\n> |Mistral|71.0|84.5|35.8|63.8|\n> |**FILM (ours)**|**76.0**|**90.0**|**39.5**|**68.5**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FwziqGqzDE", "reviewer": "Reviewer_UZJA", "summary": "In this paper, the authors propose two types of training to mitigate the lose-in-the-middle issue of Large Language Models (LLMs). The first one creates some QA pairs. They first pick 128-token segments with which they use LLM to generate QA pairs and mix these 128 tokens with other contexts to form long context QA pairs. The second one extends this approach with multiple segments. The authors also propose three VAL Probing tasks that ask LLMs to print the context of certain phrases. Results show that the model fine-tuned with proposed methods can significantly mitigate the lose-in-the-middle issue on challenging long context tasks and do well on Val Probing while keeping good performance on short context tasks.", "questions": "See weakness.\n\nAlso, what is bold in table 2?", "limitations": "They have limitations and broader impact sections. I think more discussion on synthetic data usage will be helpful.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Lost-in-the-middle is an important and challenging tasks that need to be solved to fully utilize the capability of LLMs.\n2. The proposed fine-tuning approaches demonstrate effective performance on many real-world problems, showing the potential usage of the proposed method on Large Language Models (Perhaps GPT-4). Also, it can also keep the good performance on short tasks, showing the fine-tuning won't hurt the unrelated tasks \n3. The paper is well-organized and easy to follow. I like figure one which shows the core idea of whole paper.", "weaknesses": "1. The comparison with natural dataset. What is the difference between fine-tuning on existing long context tasks, such as L-EVAL, LongBench, and SCOLLS. Why do you use synthetic data rather than these data? Do you think your synthetic data can work better? This comparison should be helpful in understanding the effectiveness of proposed approaches. \n\n2. Many choices are not justified.  What is the context you use to mix 128-token segments with gold answers? Are they the best choices compared with using similar contexts like LongBench does? Do you have any criteria for choosing segments with gold answers? \n\n3. Intermediate results are not evaluated. What is the quality of the synthetic QA?  Can we ensure the multihop question is valid?  Is the split of two or more segments natural (not breaking the context)? Which fine-tuning approach helps more on the final results? Some quality evaluation and ablation should be important for the analysis of the proposed approaches.\n\n4. Lacks detail analysis. For instance, the fine-tuning is only conducted on QA tasks. How could it improve the performance of Summarization? Will you include more content in the middle of the source as well? \n\nOverall, the strong improvement in small LLM demonstrates the effectiveness of the proposed method. Thus, I would assume that the proposed approaches can obtain positive answers to most of the questions above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful questions about our work. The following are our responses to these questions.\n\n> Q1: What are the reasons for using synthetic data rather than existing natural long context datasets during fine-tuning?\n> \n> A1: There are mainly two reasons.\n> - First, many existing natural long context datasets are often purposed for evaluation rather than training, including datasets such as L-EVAL [1], LongBench [2], and SCOLLS [3]. Therefore, by utilizing synthetic data exclusively, we can **circumvent potential data contamination** issues between training and evaluation.\n> - Second, the use of synthetic data allows our methodology to be **scalable in terms of accessing larger training data sizes**. Moreover, it offers **flexibility in training a long-context model in a specific domain** where there may be a scarcity of manually annotated QA pairs. Leveraging synthetic training data has recently become a prevalent data construction strategy for training LLMs [4,5].\n\n> Q2: What is the context we use to mix 128-token segments with the gold answer?\n> \n> A2: When selecting the filler segments $[r_j]$ to create a lengthy context, we want that $[r_j]$ bears no connection to the QA pair. This way, to respond to the given question, **the model will be trained to carefully find out and utilize the related $s_i$ (or $[s_i]$) from a large amount of redundant information**. For implementation, we randomly sample $[r_j]$ from a large candidate pool where $s_i$ (or $[s_i]$) are excluded. We have manually inspected 40 synthetic cases to verify that this random selection generally meets our expectations of unrelatedness.\n> \n> We did not use the similarity-based selection strategy due to the following concerns:\n> - Similar segments could have supporting information for the QA pairs, **creating a shortcut during training**. Given that $[r_j]$ could also provide useful information to answer the question, the model may just use the $[r_j]$ at the beginning and end of the context, without thoroughly searching for the $s_i$ in the middle.\n> - Similar segments might contain information that could lead to **ambiguity in answering the question** (for instance, different individuals with the same name). This ambiguity could potentially affect our training.\n> \n> It is a valuable research question to explore how to properly incorporate the similarity-based selection with our IN2 training. Mitigating the above issues requires further designs and ablations, such as carefully controlling the mixture proportions, designing and adjusting the rule-based/embedding-based filters, and also incorporating human check if necessary. Given the high training cost, we consider the use and ablation of similarity-based selection for future work.\n\n> Q3: Require further analysis on data quality and fine-tuning approach.\n> \n> A3: \n> - For the data quality, **a manual assessment of 40 synthetic examples was conducted prior to large-scale data generation** (20 for local information QA and 20 for multi-hop QA). Some of these examples are included in Appendix H. We find that GPT-4 is highly capable of generating QA pairs that are specifically related to the information provided in the given contexts. To generate multi-hop QA pairs, a simple prompting strategy was employed that utilizes ** to emphasize the need for \"using at least two segments\" to generate the QA pair (refer to Appendix I). Using this prompt, all 20 sampled multi-hop QA pairs have successfully incorporated information from at least two segments.\n> \n> - For the fine-tuning approach, we take the **standard instruction-tuning approach** for fine-tuning, which involves updating the model parameters only based on the loss on the output side. We do not include the input-side loss on the synthesized long contexts because we are concerned that it will dominate the training process and interfere with natural language modeling. For hyper-parameters, we initially tried four learning rates (1e-5, 5e-6, 1e-6 and 5e-7) in the first 100 training steps. We chose the 1e-6 learning rate as it exhibited the best loss trend in the first 100 training steps. Other hyper-parameters in Section 2.2 follow the general settings for post-training. \n\n> Q4: Why fine-tuning on QA-style data can enhance performance in other long-context tasks such as summarization?\n> \n> A4: Although the surface forms differ across various tasks, we believe that **the underlying capability to comprehend long-context information is transferable**. Our further evaluation results on few-shot learning tasks can also demonstrate this transferability. These tasks include TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). Although the surface form of few-shot learning is quite different from our zero-shot training data, these results further support that an improved capability to perceive long-context information can be generalized beyond surface forms.\n>\n> |Model|TREC|TriviaQA|SAMSum|Avg.|\n> |---|---|---|---|---|\n> |GPT4-Turbo|77.0|91.7|39.7|69.5|\n> |Mistral|71.0|84.5|35.8|63.8|\n> |**FILM (ours)**|**76.0**|**90.0**|**39.5**|**68.5**|\n\n> Q5: What are bold in Table 2?\n> \n> A5: The bold numbers are SOTA-level results among ~7B open-source models. Specifically, for each task, we bold the highest result and the results within a margin to the highest one (0.5 for summarization tasks and 2.0 for others).\n\n[1] L-Eval: Instituting Standardized Evaluation for Long Context Language Models\n\n[2] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding\n\n[3] SCROLLS: Standardized CompaRison Over Long Language Sequences\n\n[4] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone\n\n[5] The Llama 3 Herd of Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2o0C115PYJ", "reviewer": "Reviewer_d47H", "summary": "The paper argues that long-context models suffer from a “lost in the middle” phenomenon due to the infrequency of important data at any one position in the middle of the window during training. To correct for this, the authors propose IN2, a method of training on synthetic long-context QA data to artificially control for when important information is at each point in the context window during training. The model trained with this method, FILM-7B, demonstrates stronger performance than its base model at the 32k context scale.", "questions": "Q1. There are existing QA datasets for single-hop and multi-hop reasoning. Why generate data instead of using an existing corpus and mixing in other pretraining data as the filler material?\n\nQ2. The paper focuses on improving performance within the pretrained context length. Do you think IN2 training would be beneficial when finetuning to *extend* the context length as well? \n\nQ3. What is information intensity (line 257), and why would training with higher information intensity require a larger RoPE base? \n\nQ4. Can you provide any evidence from your experiments that your method's gains come from the synthetic QA data used rather than the benefits of additional (instruction tuning) data and training time?\n\nminor suggestions/typos: \n\n- line 205: `lm_eval` provides a preferred citation format; please cite them instead of using a footnote here.\n- line 554: “illustrates how do we segment” → “illustrates how we segment”\n- line 556: capitalized “We” in the middle of the sentence\n- line 127: was it 300 GPU days for all experiments or for training a single model? It seems from Table 3 that there were multiple models (at least partially) trained; it would be helpful to have numbers for total compute used and compute used on the final FILM-7b.\n- Needle in the Haystack is often attributed jointly to [Ivgi et al 2023](https://aclanthology.org/2023.tacl-1.17/) and [Liu et al 2023](https://arxiv.org/abs/2307.03172).\n- Table 1: why are the FiLM numbers for database bolded when GPT-4-Turbo is outperforming it?\n- It would be nice to reference what kinds of experiments are in the appendices somewhere in the main text, to make them more discoverable (e.g. I don’t think Appendix G is mentioned anywhere in the main body).\n- In the second critiquing needle in the haystack, should cite work on the reversal curse and other critiques of needle in the haystack (there have been several, including critique specifically of the single-needle case). Some of this work is already discussed in the related work, but should be cited here to show that there is community discussion about these critiques of needle in the haystack.\n- The claim that sliding window strategies greatly hurt long-context capability— I think this is a bit too broadly stated for the experimental results provided.\n- Line 70: “This demonstrates that training on synthesized long-context data can be generalized to real-world scenarios.” I think care should be taken here to emphasize that this is *additional* training on synthesized long-context data, after already training on real long-context documents at that same length (during Mistral’s pretraining).", "limitations": "The authors identify insufficient ablations as a limitation, which I agree with. However, the concern of the limitations section seems to be that the method might either 1) be brittle to the choice of hyperparameters here (which I agree is unlikely) or 2) perform much better with careful tuning. My concern is really 3) the method may not outperform existing baselines of normal instruction tuning (possibly plus RoPE base scaling).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. The paper clearly outlines a problem (the lost in the middle phenomenon), a proposed root cause (the insufficient supervision during training for important info in the middle context), and a proposed solution (IN2 training). The work is well-motivated and the proposed root cause, while not empirically demonstrated, seems reasonable. \n\nS2. The method clearly improves the performance of the base model on both synthetic (probing) and realistic long-context tasks. It’s exciting that this improvement comes with the use of very little genuinely-long-context data, and I find it very interesting that increasing the RoPE base is helpful here.\n\nS3. The design of the three probing tasks to focus around directionality of information is well-justified, and I believe the choice of different domains for source text for each task is appropriate, given that no general claims are being made about relative performance between “forward” and “backward” retrieval. The authors also correctly identify that these probing tasks are not sufficient for evaluation and present results on a reasonable set of existing long-context datasets. \n\nS4. I appreciate the concern taken to avoid data contamination and the explicit description of the steps taken (in Appendix A).", "weaknesses": "W1. My biggest concern is that I’m not really convinced that this improvement is due to IN2 rather than the general benefits to downstream tasks of finetuning on QA and instruction tuning data, maybe coupled with some additional benefit from changing the RoPE base that would hold independently of IN2 training’s data intervention. Compared to Mistral-7b-Instruct-v0.2, FILM is trained on ~1.75M additional tokens of data, including quality instruction tuning data and a large amount of QA-style tasks. Comparing this model to Mistral-7b-Instruct-v0.2 directly does not seem to be a fair comparison for evaluating IN2’s qualities as a data selection strategy for finetuning. The real comparison here would be comparing FILM to a Mistral model finetuned on the same quantity and ordering of data but with the IN2-generated synthetic question/answer pairs removed. I understand that this is very computationally costly, but even a checkpoint at 20% of the data mixture (in the same style as the RoPE ablations) would be very helpful to understand how much benefit the method is actually providing. Ideally you could also compare finetuning on non-IN2 instruction data with and without modifying the RoPE base.\n\nW2. The insight that increasing the RoPE base improves performance here is really cool, but I think the reasons why are underexplained and underexplored. The paper attributes this to “higher information intensity” in the context window, but this phrase is not defined in the text. I think it’s okay if the improvements from scaling theta are a purely empirical finding, but any claims about the cause of the improvement require much more justification. \n\nW3.  Notation in 2.1. I think the level of notational formality used here for the synthetic data generation isn’t strictly necessary, but if it’s included, more specificity is necessary. In particular, is [r_j] restricted to never include s_i / [s_i]? What number of segments might be included in [s_i]? In equation 2, it may be helpful to remind the reader of what [r_i] is, as it is not referenced in the text explaining equation 2 before the equation. And when, on line 108, you write that the length balancing is achieved “by reject[ion?] sampling on [r_j]” — does this simply mean you reduce the number of segments you sample? I don’t quite understand how this is rejection sampling, especially given that all segments are the same length.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and constructive suggestions. We hope the following response and our additional experiments have addressed your concerns.\n\n> W1 and Q4: Require experimental comparison with normal instruction tuning.\n>\n> R1: We present further experimental results to demonstrate that **normal instruction tuning (IT) cannot alleviate the lost-in-the-middle problem**. We conduct a comparison between the effectiveness of IN2 training and normal instruction tuning, with the results shown in the following table. Specifically, **both Mistral+IN2 and Mistral+IT have the same training data size** (20% of our full data size, ~300K examples) and do not incorporate adjustment of RoPE. The training data for Mistral+IT are sampled from a large-scale instruction-following dataset OpenOrca [1]. Compared with Mistral+IN2, the gains from normal instruction tuning are marginal and unstable.\n>\n> |  | Doc| | Code| | DB| | All| |\n> |---|---|---|---|---|---|---|---|---|\n> |Model|Avg|Gap&darr;| Avg|Gap&darr;| Avg|Gap&darr;| Avg|Gap&darr;|\n> |Mistral|74.2|32.1|20.3|59.5|47.5|77.0|47.3|56.2|\n> |Mistral+IT|69.0|25.9|30.2|76.5|53.4|54.4|50.9|52.3|\n> |**Mistral+IN2 (ours)**|**82.9**|**11.5**|**74.5**|**27.7**|**83.5**|**31.6**|**80.3**|**23.6**|\n>\n> Furthermore, the PDF file in our general response illustrates the performance curves of Mistral+IT. These comparisons demonstrate that IN2 training can effectively alleviate the lost-in-the-middle problem while the normal instruction tuning cannot.\n\n> W2 and Q3: Require further explanations for “higher information intensity” and the insights behind adjusting RoPE.\n> \n> R2: The term \"higher information intensity\" is not a quantitative measure but a conceptual phrase we utilize to characterize our IN2 training. In our QA-style training data, **the essential information for different questions are spread evenly across all positions, instead of being concentrated only at the beginning and the end (which would indicate high sparsity)**.\n> \n> The rationale behind our adjustment of RoPE is derived from practical experiences with context-extension training. Studies [2,3] show that when the context window was extended from 4K to 32K during training, the RoPE base $\\theta$ was increased from 10,000/50,000 to 1,000,000. There is a high-level similarity between context-extension training and our IN2 training: **both aim to enhance the model’s capability to perceive information from a wider range of input positions** compared to the previous training stages. Consequently, we carry out experiments to investigate whether the experience from context-extension training could also benefit our IN2 training.\n\n\n\n> W3: Require further clarification on notations and sampling process.\n> \n> R3: We will revise our Section 2 to make it clearer for readers.\n> - About $[r_j]$ and $s_i$/$[s_i]$: It is satisfied that $[r_j]$ will not contain $s_i$/$[s_i]$. Moreover, we make a restriction that 2 <= len($[s_i]$) <= 16. We set an upper bound here considering the time cost of calling GPT-4 for an excessively long input sequence.\n> - About the sampling process for length balance: There is a typo in line 108: the “reject sampling on $[r_j]$” should be “restricted sampling on $[r_j]$”. During sampling $[r_j]$, we restrict the distribution of len($[r_j]$) in order to manage the distribution of the final context length. Thanks for pointing out this error!\n\n> Q1: Why synthesize QA data instead of using an existing QA dataset?\n> \n> A1: There are mainly two reasons. \n> - First, the use of synthetic data allows our methodology to be **scalable in terms of accessing larger training data sizes**. Moreover, it offers **flexibility in training a long-context model in a specific domain** where there may be a scarcity of manually annotated QA pairs. Leveraging synthetic training data has recently become a prevalent data construction strategy for training LLMs [4,5].\n> - Second, as we have taken many existing QA datasets in our evaluations (including both long-context and short-context datasets), by utilizing synthetic data exclusively, we can **circumvent potential data contamination** issues between training and evaluation.\n\n> Q2: Would IN2 training be beneficial to extend the context length?\n> \n> A2: That's an insightful question. We think the answer is Yes, and we have some preliminary results and observations: applying IN2 training on a 32K context window and subsequently extending it to 64K with YaRN (without fine-tuning) still yields improvements (as illustrated in Figure 8 of our Appendix). These results may not be as robust as within the 32K context, but we believe that incorporating fine-tuning on the 64K context window could further enhance the performance.\n\n[1] https://huggingface.co/datasets/polinaeterna/OpenOrca. \n\n[2] InternLM2 Technical Report\n\n[3] Qwen2 Technical Report\n\n[4] Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone\n\n[5] The Llama 3 Herd of Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ej1O8cKb3U", "reviewer": "Reviewer_oKze", "summary": "This paper introduces INformation-INtensive (IN2) training, a data-driven approach to address the \"lost-in-the-middle\" problem in large language models (LLMs) with long context windows. The authors hypothesize that this issue stems from insufficient explicit supervision during long-context training. They create a synthetic long-context question-answering dataset that requires models to utilize information from various positions within long contexts. By applying IN2 training to the Mistral-7B model, they develop FILM-7B, which demonstrates significantly improved performance on probing tasks designed to assess long-context utilization across different context styles and retrieval patterns. FILM-7B also shows enhanced performance on real-world long-context tasks while maintaining comparable performance on short-context tasks. The authors provide detailed analyses of their training strategies, including the impact of sliding windows and position encoding adjustments.", "questions": "- Will the training data be released to the community?\n- I saw a related paper a few months ago that developed a multi-hop qa style needly-in-the-haystack benchmark. I tried to find the paper during the review process but unfortunately, I didn't find it. The basic idea is similar to the training data construction method described in the paper. If the authors also came across this paper, it would be great if you could test your model on the benchmark. If not, it is totally fine. I will revise my comment and add the paper if I can find it later.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces IN2 training, an innovative data-driven solution to address the \"lost-in-the-middle\" problem in long-context language models. This tackles a critical issue in the field of long-context LLMs, potentially improving their ability to utilize information from extended contexts.\n- The authors develop a thorough evaluation framework called VAL Probing, which assesses long-context capabilities across various context styles (document, code, and structured-data) and retrieval patterns (forward, backward, and bi-directional). This provides a more rigorous and diverse testing suite compared to existing benchmarks like Needle-in-the-Haystack.\n- FILM-7B, the model developed using IN2 training, demonstrates significant improvements not only on the probing tasks but also on real-world long-context tasks. Importantly, it maintains performance on short-context tasks, showing that the improvements in long-context handling do not come at the expense of other capabilities. The model even achieves performance comparable to or better than GPT-4-Turbo on some tasks.", "weaknesses": "- I do not see a major flaw in the paper. But one potential concern is: since the data is synthesized and the chunk is randomly injected into the document, would the model memorize such synthetic data but lose the generalizability on other long-context tasks beyond such scenarios? \n- The authors may add more details about their construction of the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback on our work. We hope the following response could further address your concerns.\n\n> W1: Since the data is synthesized and the chunk is randomly injected into the Doc, would the model memorize such synthetic data but lose the generalizability on other long-context tasks beyond such scenarios?\n> \n> R1: Our evaluation results on real-world long-context tasks (Table 2) could address this concern. For instance, the long contexts in NarrativeQA are **entire books or movie scripts**, and the long contexts in Qasper are **full academic papers**. These natural long contexts are quite different from our synthetic data, and the improvements on these tasks demonstrate the generalizability beyond the synthetic pattern. \n> \n> We also further evaluate on the following few-shot learning tasks which also have different surface forms with our synthetic data. These tasks are taken from LongBench, including TREC (few-shot classification), TriviaQA (few-shot QA) and SAMSum (few-shot summarization). The improvements on these tasks further demonstrate the generalizability on general long-context tasks.\n>\n> |Model|TREC|TriviaQA|SAMSum|Avg.|\n> |---|---|---|---|---|\n> |GPT4-Turbo|77.0|91.7|39.7|69.5|\n> |Mistral|71.0|84.5|35.8|63.8|\n> |**FILM (ours)**|**76.0**|**90.0**|**39.5**|**68.5**|\n\n\n> W2: The authors may add more details about their construction of the training data.\n> \n> R2: We will add further details about the process of constructing our training data, including the selection strategy on filling segments $[r_j]$ and the data quality analysis before generating data on a large scale. Moreover, Appendix D contains details about the considerations and implementation for our segmentation process.\n> - **Selection strategy on filling segments $[r_j]$**. We randomly sample $[r_j]$ from a large candidate pool where $s_i$ and also $[s_i]$ are excluded. During sampling $[r_j]$, we restrict the distribution of len($[r_j]$) in order to manage the distribution of the final context length.\n> - **Data quality analysis**. We manually accessed 40 synthetic examples before generating data on a large scale. Some of examples are contained in our Appendix H. We find that GPT-4 is highly capable of generating QA pairs that are specifically related to the information provided in the given contexts. \n\n> Q1: Will the training data be released to the community?\n> \n> A1: Our data will be released upon completion of the internal data review process.\n\n> Q2: A related paper about multi-hop QA style Needly-in-the-Haystack benchmark.\n> \n> A2: Are you referring to the RULER benchmark [1]? This benchmark purports to access \"multi-hop tracing and aggregation\". Our Appendix F includes the evaluation process and results on the RULER benchmark. It shows that our FILM-7B is the first ~7B model that achieves 32K effective context length.\n> \n> |Models|Claimed Length|Effective Length|4K|8K|16K|32K|\n> |---|---|---|---|---|---|---|\n> |Llama2-7B|4K| |85.6|-|-|-|\n> |LongChat-7B|32K|<4K|84.7|79.9|70.8|59.3|\n> |Together-7B|32K|4K|$\\underline{88.2}$|81.1|69.4|63.0|\n> |LWM-7B|1M|<4K|82.3|78.4|73.7|69.1|\n> |ChatGLM3-6B|128K|4K|$\\underline{87.8}$|83.4|78.6|69.9|\n> |Mistral-7B|32K|16K| $\\underline{93.6}$| $\\underline{91.2}$|$\\underline{87.2}$|75.4|\n> |**FILM-7B**|32K|**32K**|$\\underline{92.8}$|$\\underline{88.2}$|$\\underline{88.1}$|$\\underline{86.9}$|\n\n[1] RULER: What's the Real Context Size of Your Long-Context Language Models?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments and questions on our paper. 
We appreciate the recognition of the important and challenging research question we explored, the innovative idea and the effectiveness method we proposed, and the thorough evaluations we conducted. 

We summarize our responses to the main concerns as follows and will include all other minor points in our revised version.

- **More experimental results and comparisons.** We present additional results to further demonstrate the effectiveness of IN2 training.
    - Comparison with normal instruction tuning under the same data size (R1 to Reviewer HMgi, R1 to Reviewer d47H and Figure 1 in PDF file).
    - Performance of FILM-7B on few-shot scenarios (A1 to Reviewer HMgi, A4 to Reviewer UZJA and R1 to Reviewer oKze).
    - Performance of FILM-7B on RULER benchmark (A2 to Reviewer oKze).

- **Why do we use synthetic data?** There are mainly two reasons.
    - First, the use of synthetic data allows our methodology to be **scalable in terms of accessing larger training data sizes**. Moreover, it offers **flexibility in training a long-context model in a specific domain** where there may be a scarcity of manually annotated QA pairs. Leveraging synthetic training data has recently become a prevalent data construction strategy for training LLMs (e.g., Phi-3 and Llama-3.1).
    - Second, exclusively utilizing synthetic data for training can **circumvent potential data contamination** issues in our evaluations on natural language datasets, including both long-context and short-context datasets.

- **More details about data construction and data quality analysis.** We will add further details about the process of constructing our training data, including the selection strategy on filling segments $[r_j]$ (A2 to Reviewer UZJA, R3 to Reviewer d47H and R2 to Reviewer oKze) and the data quality analysis before generating data on a large scale (A3 to Reviewer UZJA and R2 to Reviewer oKze). Moreover, Appendix D contains details about the considerations and implementation for our segmentation process.

Meta-review: The paper identifies an issue with LLMs, commonly referred to as lost-in-the-middle problem, where they exhibit positional bias when integrating information from in-context document, often ignoring information presented in the middle. The core idea is to generate large-scale synthetic question answering dataset (1.1M long-context data) where important information is evenly spread throughout the context to continue training LLMs. In addition, they also propose improved evaluation setting (forward, backward, bi-directional information retrieval) and evaluate their approach extensively. The proposed method shows performance gains on a variety benchmark, both long-context focused and regular text understanding/reasoning benchmark. 

The reviewers have pointed out some weaknesses with the initial draft, such as (1) unfair with base model which were trained on fewer tokens (reviewer HMgi, d47H) and (2) the assessment of generated QA data quality (Reviewer UZJA). The authors have provided additional experimental results and addressed most, if not all, concerns of the reviewers. 

For camera-ready, if time/compute allows, it would be great to have an experiment showing results on performing continued fine-tuning on another base model to show the generality of the proposed method. Overall the paper presents a convincing approach to address well-known issue in LLM.
Decision: Accept (poster)