Title: Make Your LLM Fully Utilize the Context
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model

Long-Context Large Language Model

Long-Context Training Data Construction

Long-Context Probing

Lost-in-the-Middle
Abstract: While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the *lost-in-the-middle* challenge.
We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information.
Based on this intuition, our study presents **information-intensive (IN2) training**, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) **fine-grained information awareness** on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the **integration and reasoning** of information from two or more short segments.
Through applying this information-intensive training on Mistral-7B, we present **FILM-7B** (FIll-in-the-Middle).
To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).
The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window.
Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU).

Make Your LLM Fully Utilize the Context

Abstract

While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the lost-in-the-middle challenge. We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information. Based on this intuition, our study presents INformation-INtensive (IN2) training , a purely data-driven solution to overcome lost-in-the-middle. Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) fine-grained information awareness on a short segment ( ∼ 128 tokens) within a synthesized long context (4K -32K tokens), and (2) the integration and reasoning of information from two or more short segments. Through applying this information-intensive training on Mistral-7B, we present FILM-7B ( FIL lin-theM iddle). To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window. Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5 → 26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3 → 59.2 accuracy on MMLU).

1 Introduction

To a great mind, nothing is little.

-Arthur Conan Doyle

Long-context large language models (LLMs) have recently received significant attention within the open-source community (Jiang et al., 2023; Du et al., 2022; Li et al., 2023a; Shi et al., 2023; Team et al., 2023; Team, 2023; Chen et al., 2023a; Song et al., 2023; Liu et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Xiong et al., 2023; Tworkowski et al., 2024; AI et al., 2024; Ding et al., 2024; Mohtashami & Jaggi, 2024; Fu et al., 2024; Cai et al., 2024; Bai et al., 2024; Lv et al., 2024). The training context windows of many contemporary LLMs have been expanded to tens of thousands of

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

FILM-7B (Ours)

GPT4-Turbo

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

tokens, thereby enabling these models to process extensive context as input. This extended training context window can enhance many real-world downstream tasks such as long-context question answering (Koˇ cisk` y et al., 2018; Dasigi et al., 2021; Bai et al., 2023) and summarization (Fabbri et al., 2019; Huang et al., 2021; Zhong et al., 2021).

However, recent studies have revealed that these long-context LLMs struggle to effectively and robustly utilize all the information provided in the context, known as the lost-in-the-middle challenge (Liu et al., 2024b; Xu et al., 2023). It implies that while the LLM can comprehend the information at the beginning and end of the long context, it often overlooks the information in the middle. This challenge could significantly hinder the development of long-context LLMs, as they even often fail to pass simple probing tasks such as Needle-in-the-Haystack and passkey retrieval (Mohtashami & Jaggi, 2024). Consequently, a pressing research question arises: how can we make long-context LLMs fully utilize the information in the long context?

We hypothesize that the root cause of lost-in-the-middle stems from the unintentional bias hidden in the general training data. In auto-regressive pre-training, the loss on predicting the next token is more likely to be influenced by a few nearby pre-tokens rather than long-distance tokens (Sharan et al., 2018; Sun et al., 2021). For supervised fine-tuning and alignment, the system message, which strongly influences the generation of the response, is typically presented at the beginning of the context (Touvron et al., 2023; Cai et al., 2024). As a result, the general training process may inadvertently introduce a position bias, suggesting that important information is always located at the beginning and end of the context.

Based on this hypothesis, our work introduces INformation-INtensive (IN2) training to explicitly teach the model that the crucial information can be intensively present throughout the context , not just at the beginning and end. IN2 training is a purely data-driven solution that utilizes a synthesized long-context question-answer dataset. The long context (ranging from 4K to 32K tokens) is concatenated from many short segments ( ∼ 128 tokens), and the question-answer (QA) pairs ask for the information contained in one or more segments which are randomly placed in the long context. Specifically, we generate two types of questions, requiring (1) fine-grained information awareness on exactly one short segment, and (2) the integration and reasoning of information from two or more segments. These QA pairs are generated by prompting GPT-4-Turbo (OpenAI, 2023b) with the designed instructions and the raw segments.

By applying this information-intensive training on Mistral-7B (Jiang et al., 2023), we present FILM7B ( FIL l-in-theM iddle). To thoroughly assess the long-context information awareness of FILM-7B, we design three probing tasks encompassing various context styles (document, code, and structureddata context) and information retrieval patterns (forward, backward, and bi-directional retrieval). The probing results (Figure 1) demonstrate that IN2 training significantly overcomes the lost-in-themiddle problem for the backbone model. Moreover, it can enhance the open-source model to achieve comparable or even more robust performance compared with proprietary LLMs such as GPT-4-Turbo.

Beyond these probing tasks, the performance of FILM-7B on real-world long-context tasks also exhibits significant improvements (e.g., 23.5 → 26.9 F1 score on NarrativeQA (Koˇ cisk` y et al., 2018)). This demonstrates that the post-training on synthesized long-context data can be generalized to

Raw Text

𝓒𝒊

Split by 128 Tokens

𝒔𝒊

Single Segment

𝒔𝒊

Long Context

…

…

Prompting GPT-4

𝒒𝒊

𝒂𝒊

Local Information QA

Concatenate with Random Segments

Fine-Grained Information Awareness

Raw Text

𝓒𝒊

𝒔 𝒊

𝟏

Multiple Segments

Prompting GPT-4

Multi-Hop QA

Integration and Reasoning of Information

𝒔 𝒊

𝟐

𝒔 𝒊

𝟑

𝒔 𝒊

𝟏

𝒔 𝒊

𝟐

𝒔 𝒊

𝟑

Long Context

…

…

Concatenate with Random Segments

Split by 128 Tokens

𝒒𝒊

𝒂𝒊

real-world scenarios. Moreover, FILM-7B maintains a comparable performance on short-context tasks compared with the vanilla backbone model (e.g., 59.3 → 59.2 accuracy on MMLU (Hendrycks et al., 2020)). This indicates that the short-context capability of FILM-7B is not compromised during training. Our further analysis explores how the sliding window strategy and the choice of RoPE base θ influence the performance of IN2 training.

2 Information-Intensive Training

This section introduces the construction of the dataset for IN2 training and the detailed training process of our model FILM-7B.

2.1 Training Data Construction

Overview. The IN2 training aims to explicitly teach the model that any position in a long context can contain crucial information. To achieve this goal, we construct a long-context question-answer training dataset D = {L i , q i , a i } , where the answer a i to the question q i requires the information contained in some short segments that are randomly placed in the whole long context L i .

Figure 2 illustrates an overview of the data construction process. Specifically, the training data D is constructed based on a general natural language corpus C . Given a raw text C i ∈ C , we first generate a question-answer pair ( q i , a i ) using a powerful LLM, then synthesize a long context L i that includes the necessary information from C i and other randomly sampled texts from C . We generate two types of question-answer pairs that require (1) the awareness of fine-grained information in the long context, and (2) the integration and reasoning of information appearing at different positions in the long context. We take the realnewslike subset from the C4 corpus (Raffel et al., 2020) as C , and take GPT-4-Turbo (OpenAI, 2023b) as the LLM to generate QA pairs.

Fine-grained information awareness. We consider a 128-token segment as the minimum information unit of the context 3 . Given a raw text C i , we first randomly extract a 128-token segment s i from it, then generate the q i , a i and L i accordingly,

Document Sentence Retrieval (Bi-Direction)

Code Function Retrieval (Backward)

Database Entity Retrieval (Forward)

### Context:

…

This crucially distinguishes our algorithms from the

…

Specifically, our modality-missing-aware prompts can

…

These results demonstrate that there are still a large ...

We

design better optimizers, a crucial engineering …

We present a study of modern architectures

applied …

This scalability issue is to use

of consensus algorithms …

Extensive experiments are conducted to validate the

effectiveness of our proposed method, achieving new

state-of-the-art performance on all four benchmarks

with a notable gain.

Notably, we achieved the top in highly

competitive …

With this, it is shown how approximate FP64x2 GEMM

…

It is challenging to address widespread and

…

To verify the effectiveness of the proposed method

…

The results show that \\

emph{GCMiner} significantly …

Our experimental results on all common benchmark

…

…

### Instruction:

In above context, which sentence contains the piece

"achieving new state-of-the-art performance on all four"?

### Context:

…

def get_clause:\n llen = len(lineup)\n clause = ''\n if

…

def updateData:\n if self.train:\n if self.inplace:\n self.

…

def save_comments:\n for comment in comments:\n

…

def plot_patio:\n ax = plt.subplot(111)\n passo_x = 1 /

…

def encode_label:\n Label record format:\n Total: 5

…

def _parse_array:\n array = []\n for child in node.

…

def serve_rpc:\n plugins = [QuarkAsyncPlugin()]\n rpc

=…

def

createStrip

:\n story = fetchVign(config)\n

if specialPlatform == 'android':\n except Exception as err:

def breed_childern:\n self.mutation(first_child)\n self

. …

def get_module_depth:\n Parameters\n depth_image

: …

def run_layout:\n if settings is None:\n if settings.

…

def register:\n user = None\n if user_id:\n if

request …

= list( …

def with_laps:\n with Stopwatch() as sw:\n for i in

…

def config_iq_stream:\n bwActual = c_double(0)\n

…

…

### Instruction:

In above context, which function contains the code snip

"if specialPlatform == 'android':" ?

### Context:

…

<id: Q2486402, label: New York State Route 191,

…

>

<id: Q80329096, label: Transverse abdominal incision

…

>

<id: Q70559114, label: Monitoring plasma level of

…

>

<id: Q91568218, label: Progression of the first stage

…

>

<id: Q84088820, label: Historical perspective of low-

…

>

<id: Q63952215, label: Online action-to-perception

… >

<id: Q40241868, label:

Alpha-1-C-octyl-1-deoxynoji-

rimycin as a pharmacological chaperone for Gaucher

disease

, description:

scientific article published on 21

August 2006

>

<id: Q5651247, label: Wer, wenn nicht wir, descript

…

>

<id: Q42133313, label: UnZIPping mechanisms of

…

>

<id: Q74650195, label: Pursued by genetics: an auto

… >

<id: Q38835253, label: Neurological Aspects of

…

>

<id: Q64358411, label: Unity for Change, description:

…

>

<id: Q24110047, label: Hypothetical protein SM_b20

… >

…

### Instruction:

In above context , what is the label and description for

the query where the id is Q40241868 ?

where ( q i , a i ) is sampled by prompting the powerful LLM with the segment s i and the instruction I f , ⊕{·} represents the concatenation of the contained segments, and [ r j ] are randomly sampled from 128-token segments in C . Note that I f instructs the LLM to make the question-answer pair highly specific to the information provided in s i .

Integration and reasoning of information. Beyond utilizing each single segment, we consider to generate question-answer pairs for information contained in two or more segments. Following the setting of the minimum information unit above, we split a full text C i into a set of 128-token segments [ s i ] , then generate the q i , a i and L i accordingly,

where I r instructs the LLM to generate a multi-hop question-answer pair that requires the information within at least two segments in [ s i ] . All segments in [ s i ] and [ r j ] are jointly shuffled, so the required segments may appear far apart in the context.

Context length balance and data mixture. To prevent length bias during IN2 training, we ensure the length of the long context L i is evenly distributed from 4K to 32K tokens. Such a length balance strategy can be implemented with restricted sampling on [ r j ] , according to Equation 1 and 2. To alleviate catastrophic forgetting on short-context capabilities, we retain ∼ 10% question-answer pairs with the original texts C i instead of converting them into a longer context, and add some general instruction-tuning data from the OpenOrca (Lian et al., 2023) dataset.

Overall, our dataset for IN2 training contains 1.1M long-context data for the fine-grained information awareness ( ∼ 63%), 300K long-context data for the integration and reasoning of information ( ∼ 17%), 150K short-context question-answer data ( ∼ 9%), and 200K general instruction-tuning data ( ∼ 11%). Appendix I contains the handcraft instructions for data generation. Appendix H illustrates some examples of our constructed long-context QA data. Appendix A describes the filtering strategy to avoid data contamination for evaluation.

2.2 Training Details

Using the training data constructed above, we further fine-tune the Mistral-7B-Instruct-v0.2 4 (Jiang et al., 2023) to get our FILM-7B ( FIL l-in-theM iddle). We perform IN2 training in the instructiontuning paradigm: the long contexts and questions are used as instructions, and the loss on the answer parts are used to update the model. Appendix I contains the system template used for formatting the training data. For hyper-parameters, we set the global batch size as 128 and conduct one-epoch training with ∼ 14K training steps. We use the cosine learning rate decay with a 1e-6 maximum

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

FILM-7B (Ours)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

(a) Performance of FILM-7B, Mistral-7B-Instruct-v0.1, and Mistral-7B-Instruct-v0.2.

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

FILM-7B (Ours)

LongAlign-13B-64K

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

(b) Performance of FILM-7B, LongAlign-7B-64K, and LongAlign-13B-64K.

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

FILM-7B (Ours)

InternLM2-chat-20B

InternLM2-chat-7B

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

(c) Performance of FILM-7B, InternLM2-chat-7B, and InternLM2-chat-20B.

learning rate and 3% warm-up steps. The training process is conducted on 16 nodes of 8x80G A100 GPUs with the full sharding strategy and cpu offload strategy implemented by pytorch FSDP (Zhao et al., 2023). One entire training process (for a single FILM-7B model) consumes ∼ 300 GPU days.

3 Long-Context Probing

In this section, we first show the preliminary evaluation of FILM-7B on the Needle-in-the-Haystack and discuss about the inadequacies of this probing task. Subsequently, to comprehensively evaluate the long-context information awareness of FILM-7B, we introduce VArious Long-context (VAL) Probing . This includes three tasks that cover various context styles (document, code, and structureddata context) and information retrieval patterns (forward, backward, and bi-directional retrieval).

3.1 Near-Perfect Performance on Needle-in-the-Haystack: Are We There Yet?

The Needle-in-the-Haystack (Ivgi et al., 2023; Liu et al., 2024b) is widely used to assess how robustly a model utilizes information positioned in the long context. It reveals that even some powerful proprietary LLMs, such as GPT-4 and Claude 2.1 (Anthropic, 2023), struggle to fully exploit the information within the long context.

We use the Needle-in-the-Haystack task 5 to preliminarily evaluate the long-context capability of FILM-7B. Appendix B demonstrates that FILM-7B has achieved near-perfect performance on this task. This result is not surprising as recent open-source LLMs, such as LongAlign (Bai et al., 2024) and InternLM2 (Cai et al., 2024), have also shown near-perfect performance on this task.

However, the near-perfect performance on Needle-in-the-Haystack may overestimate the long-context capabilities of LLMs (Lei et al., 2024; Hsieh et al., 2024). Specifically, we have the following two concerns:

Needle-in-the-Haystack employs a document-style context, which LLMs could be quite familiar with due to the pre-training on natural language corpora.

The forward retrieval pattern in Needle-in-the-Haystack may simplify the difficulty of information seeking in the long context.

The 'forward retrieval' means that the information being retrieved directly follows the retrieval keyword in a long context. For example, the default question used in Needle-in-the-Haystack is "What is the best thing to do in San Francisco?" and the answer is contained in "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day." The retrieved information "eat a sandwich and ..." just follows the retrieval keywords "best thing to do in San Francisco". According to the mechanism of induction head (Olsson et al., 2022), such a following-up copying is an easily learned pattern for LLMs, thus less challenging for evaluating long context utilization (just like the observation of 'reversal curse' (Berglund et al., 2024)).

Given these considerations, we suggest that performances on Needle-in-the-Haystack may not adequately reflect the long-context capabilities of LLMs. Therefore, we propose VAL Probing for a more comprehensive evaluation involving various context styles and retrieval patterns.

3.2 VAL Probing

Our retrieval-based VAL Probing considers three context styles (document, code, and structured-data context) and three retrieval patterns (forward, backward, and bi-directional retrieval). Each context in VAL Probing contains ∼ 32K tokens, and each task contains ∼ 3K examples. Figure 3 briefly illustrates the contexts and retrieval instructions in VAL Probing.

Document Sentence Retrieval (Bi-Direction). The contexts consist of numerous natural language sentences, and the instruction aims to retrieve a single sentence containing a given piece. The sentences are sampled from the abstracts of papers on arXiv 6 . This task follows the bi-directional

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

Mistral + IN2 (Ours)

Mistral + IT

Mistral

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

retrieval pattern, as the expected retrieval results contain words both before and after the given piece in the context. The evaluation metric is the word-level recall score.

Code Function Retrieval (Backward). The contexts consist of Python functions, and the instruction aims to retrieve the function name for a given line of code within the function definition. The raw code functions are sampled from the StarCoder (Li et al., 2023c) dataset 7 . We randomly select three lines of definitions for each function. This task follows the backward retrieval pattern, as the function name always precedes the definition. The evaluation metric is the exact-match accuracy.

Database Entity Retrieval (Forward). The contexts contain lists of structured entities, each with three fields: ID, label, and description. The query aims to retrieve the label and description for a given ID. The entities are sampled from Wikidata 8 . This task follows the forward retrieval pattern, as the label and description follow the ID. We take a relaxed exact-match accuracy as the metric: a 1 score is given if either the label or the description is exactly matched in the response, otherwise a 0 score.

4 Experiments and Analysis

We assess the long-context capability of FILM-7B on both probing tasks and real-world long-context tasks. Moreover, we investigate if the performance in short-context scenarios is affected.

4.1 Experimental Setup

Models. We mainly compare FILM-7B with long-context open-source models that have been trained with ≥ 32K context windows, including the Mistral (Jiang et al., 2023), LongChat (Li et al., 2023a), ChatGLM (Du et al., 2022), LongAlign (Bai et al., 2024), LongWanjuan (Lv et al., 2024), Yi (AI et al., 2024) and InternLM2 (Cai et al., 2024). We utilize the instruct/chat versions of these models as most of our evaluation tasks are under the zero-shot instruction-following paradigm. We also draw comparisons with popular proprietary LLMs such as GPT-3.5-Turbo (OpenAI, 2023a) and

GPT-4-Turbo (OpenAI, 2023b). All models and tasks employ greedy decoding. For probing tasks, we primarily compare FILM-7B with LongAlign and InternLM2 series, as these models have shown near-perfect performances on Needle-in-the-Haystack.

Real-world long-context tasks. We take 9 tasks from the LongBench (Bai et al., 2023) collection to evaluate the long-context capability on real-world scenarios. These tasks encompass long-document question answering (NarrativeQA (Koˇ cisk` y et al., 2018), Qasper (Dasigi et al., 2021) and MultiFieldQA (MultiFQA) (Bai et al., 2023), multi-document multi-hop reasoning (HotpotQA (Yang et al., 2018), 2WikiMultihopQA (2WikiMQA) (Ho et al., 2020) and MuSiQue (Trivedi et al., 2022)), and long-context summarization (GovReport (Huang et al., 2021), QMSum (Zhong et al., 2021) and MultiNews (Fabbri et al., 2019)). We employ the middle truncation strategy in LongBench to limit the input within 32K tokens. We report ROUGE-L (Lin, 2004) for summarization tasks and F1 scores for other tasks. The evaluation metrics are computed using the official evaluation scripts 9 .

Despite these QA and summarization tasks, we also conduct evaluations on few-shot learning tasks, in which the contexts could also be extremely lengthy if there are many "few-shot" examples presented in the context. We take three in-context learning tasks from LongBench, including TREC (Li & Roth, 2002) for few-shot classification, TriviaQA (Joshi et al., 2017) for few-shot QA, and SAMSum (Gliwa et al., 2019) for few-shot summarization.

Short-context tasks. We select 8 short-context tasks commonly used for evaluating the general capabilities of models. These include MMLU (Hendrycks et al., 2020), BoolQ (Clark et al., 2019), RACE-High (RACE-H) (Lai et al., 2017), CommonsenseQA (CSQA) (Talmor et al., 2019), ARCChallenge (ARC-C) (Clark et al., 2018), HellaSwag (Zellers et al., 2019), GSM8K (Cobbe et al., 2021), and MATH (Hendrycks et al., 2021). We use 5-shot for MMLU, 8-shot for GSM8K, 4-shot for MATH, and 0-shot for other tasks. We utilize the lm_eval (Gao et al., 2024) for the evaluations on MMLU, BoolQ, RACE-H, ARC-C and HellaSwag, and use the evaluation scripts from An et al. (2024) for other tasks.

4.2 Main Results and Analysis

FILM-7B significantly mitigates the lost-in-the-middle problem. Figure 4a presents the probing results for both FILM-7B and the backbone model, Mistral-7B-Instruct-v0.2. In all three probing tasks within VAL Probing, the vanilla Mistral model experiences substantial information loss at the middle positions in the long contexts. In contrast, our FILM-7B model consistently exhibits robust

59.3

85.4

46.0

70.0

55.9

83.6

40.4

8.7

59.2

87.7

45.6

75.3

52.5

79.1

44.5

11.3

0

20

40

60

80

100

Performance (%)

MMLU

FILM-7B

BoolQ

RACE-H

CSQA

ARC-C

MATH

performance across different positions within the whole context. This stark comparison illustrates that the lost-in-the-middle problem can be effectively addressed using our IN2 training.

FILM-7B achieves performance comparable to, or even outperforming, that of GPT-4-Turbo. Figure 1 illustrates the comparison between FILM-7B and GPT-4-Turbo on our probing tasks. Beyond a qualitative comparison between the performance curves of two models, we quantify the long-context performances on VAL Probing using two metrics:

Average score (Avg). We compute the average performances across the entire context length, reflecting the overall long-context utilization.

Min-max gap (Gap). We calculate the differences between the maximum and minimum performances in Figure 3. A smaller performance gap signifies greater robustness across different positions.

Table 1 presents the quantified performances on VAL Probing. It reveals that FILM-7B has comparable performance with GPT-4-Turbo on the database probing task, and exhibits better robustness in document and code probing tasks. These results indicate a great potential for the development of open-source long-context models to close the gap with proprietary models.

IN2 training can effectively alleviate the lost-in-the-middle problem, while the normal instruction tuning cannot. Considering that FILM-7B additionally uses instruction-tuning-style data for post-training, to further demonstrate the effectiveness of IN2 training, here we present more controlled experiments to compare IN2 training and normal instruction tuning under the same training data size. Specifically, for both IN2 training and normal instruction tuning, we take the same backbone model (i.e., Mistral-7B-Instruct-v0.2) and the same number of post-training instances (20% of our full data size, ∼ 300K examples). The data for normal instruction tuning are randomly sampled from OpenOrca (Lian et al., 2023). Comparisons shown in Figure 5 shows the performance curves on VAL Probing after two training processes, and Table 2 contains the quantified results. These comparisons clearly demonstrate that IN2 training can effectively alleviate the lost-in-the-middle problem while the normal instruction tuning cannot.

VAL Probing presents a more challenging test suite for long-context models. Figure 4b and 4c show the probing results of LongAlign and InternLM2, two state-of-the-art long-context models. Despite their extended training context windows, these models still encounter the lost-in-the-middle problem. This is particularly noteworthy given their near-perfect performance on the Needle-in-theHaystack task. This comparison suggests that VAL Probing provides a more challenging evaluation for long-context models.

In particular, the results on document and database tasks in VAL Probing demonstrate clear comparisons with Needle-in-the-Haystack. Compared to Needle-in-the-Haystack which uses forward retrieval on natural language context, the document task employs natural language context but with bi-directional retrieval, and the database task uses forward retrieval but with structured-data context. These comparisons highlight that both context styles and retrieval patterns significantly contribute to the hardness of the probing tasks.

Training on synthesized long-context data effectively generalizes to real-world scenarios. Table 3 and 4 contain the results on various real-world long-context tasks. It shows that FILM-7B also significantly improves the performance of the backbone model in real-world long-context scenarios. Moreover, it also achieves SOTA-level 10 performances on these tasks among ∼ 7B size open-source models. Notably, the long contexts used in IN2 training are all synthesized from short segments. These improvements suggest that the long-context capabilities learned from the synthesized data can be successfully applied to real-world tasks.

FILM-7B maintains the performance on short-context tasks. Figure 6 illustrates the performances of FILM-7B and the vanilla backbone model on short-context tasks. It reveals that the overall performances on short-context tasks are almost comparable with minor variances. These results confirm that FILM-7B does not compromise the short-context capabilities of the backbone model.

Analysis on training strategies. We are specifically interested in investigating the impact of the following two training strategies: applying the sliding window and adjusting the position encoding. Due to the page limitation, we provide these further ablations and analysis in Appendix C.

5 Related Work

Long-context LLMs. Recent research has significantly contributed to the exploration of training large models with extended context windows (Jiang et al., 2023; Du et al., 2022; Li et al., 2023a; Team et al., 2023; Team, 2023; Xiong et al., 2023; Song et al., 2023; Tworkowski et al., 2024; AI et al., 2024; Cai et al., 2024). There are primarily two directions in the development of long-context LLMs. (1) Data engineering, which emphasizes the construction of long-context data for training the LLMs. This includes data balancing (Fu et al., 2024), data order arrangement (Shi et al., 2023), instruction data collection (Bai et al., 2024), and data quality measurement (Lv et al., 2024). Our IN2 training can be categorized into this field. (2) Effective and efficient training, which investigates methods to optimize the training of a long-context model. This encompasses the design of position encoding (Chen et al., 2023a; Liu et al., 2023; Peng et al., 2023b; Ding et al., 2024), batching strategy (Bai et al., 2024), parameter-efficient training (Chen et al., 2023b), and the development of new model architectures (Peng et al., 2023a; Gu & Dao, 2023).

Long-context evaluations. Existing benchmarks for evaluating long-context models can be divided into two categories. (1) Real-world benchmarks that assess general long-context capabilities (e.g., long-context QA, summarization, and language modeling), such as NarrativeQA (Koˇ cisk` y et al., 2018), LongBench (Bai et al., 2023), ZeroSCROLLS (Shaham et al., 2023), L-Eval (An et al., 2023), Loogle (Li et al., 2023b), ∞ Bench (Zhang et al., 2024), and a series of work on perplexity evaluation (Beltagy et al., 2020; Roy et al., 2021; Press et al., 2021; Chen et al., 2023a; Liu et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Ding et al., 2024; Mohtashami & Jaggi, 2024). (2) Probing tasks that provide a more concise reflection of the long-context utilization across different context lengths and positions. These include Needle-in-the-Haystack, passkey retrieval (Mohtashami &Jaggi, 2024), synthesized document QA (Liu et al., 2024b), S3Eval (Lei et al., 2024), Discovery (Li et al., 2024), RULER (Hsieh et al., 2024), and the VAL Probing proposed in this study. Among these probing tasks, our VAL Probing is the first to explicitly incorporate a variety of retrieval patterns.

6 Conclusion

This work introduces IN2 training to overcome the lost-in-the-middle problem. By applying IN2 training on the open-source model, our FILM-7B exhibits significant improvements on probing tasks and real-world long-context tasks while does not compromise the short-context performance.

Acknowledgments

We thank all the anonymous reviewers for their valuable comments. Shengnan An and Nanning Zheng were supported in part by NSFC under grant No. 62088102.

References

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024.

AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088, 2023.

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Learning from mistakes makes llm better reasoner, 2024.

Anthropic. Model card and evaluations for claude models, 2023. URL https://www-files. anthropic.com/production/images/Model-Card-Claude-2.pdf .

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508, 2023.

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058, 2024.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.

Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a", 2024. URL https://arxiv.org/abs/2309.12288 .

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023a.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2023b.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936, 2019.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610, 2021.

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.

Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074. Association for Computational Linguistics, 2019.

Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context, 2024.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.org/records/12608602 .

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A humanannotated dialogue dataset for abstractive summarization. EMNLP-IJCNLP 2019, pp. 70, 2019.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, 2020.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024.

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pp. 1419-1436. Association for Computational Linguistics (ACL), 2021.

Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284-299, 2023. doi: 10.1162/tacl_a_00547. URL https://aclanthology.org/2023.tacl-1.17 .

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16011611, 2017.

Tomáš Koˇ cisk` y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, 2017.

Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. S3eval: A synthetic, scalable, systematic evaluation suite for large language models, 2024.

Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source LLMs truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. URL https: //openreview.net/forum?id=LywifFNXV5 .

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939, 2023b.

Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023c.

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning, 2024.

Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002.

Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https:// huggingface.co/Open-Orca/OpenOrca , 2023.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention, 2024a.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.

Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. In The Twelfth International Conference on Learning Representations, 2023.

Kai Lv, Xiaoran Liu, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. Longwanjuan: Towards systematic measurement for long text quality. arXiv preprint arXiv:2402.13583, 2024.

Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. Advances in Neural Information Processing Systems, 36, 2024.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.

OpenAI. Gpt-3.5 turbo fine-tuning and api updates, 2023a. URL https://openai.com/blog/ gpt-3-5-turbo-fine-tuning-and-\api-updates .

OpenAI. Gpt-4 technical report, 2023b.

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077, 2023a.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2023b.

Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, 2023.

Vatsal Sharan, Sham Kakade, Percy Liang, and Gregory Valiant. Prediction with a short memory. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pp. 1074-1087, 2018.

Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundaries. In The Twelfth International Conference on Learning Representations, 2023.

Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained llms. In The Twelfth International Conference on Learning Representations, 2023.

Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 807-822, 2021.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https: //aclanthology.org/N19-1421 .

MosaicML NLP Team et al. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023.

Together Team. Together 32k, 2023. URL https://huggingface.co/togethercomputer/ LLaMA-2-7B-32K .

Together.AI. Llama-2-7b-32k-instruct - and fine-tuning for llama-2 models with together api, 2023. URL https://www.together.ai/blog/llama-2-7b-32k-instruct .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.

Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miło´ s. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models, 2023.

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, 2023.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. ∞ bench: Extending long context evaluation beyond 100k tokens, 2024.

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multidomain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5905-5921, 2021.

This is the Appendix of the paper: Make Your LLM Fully Utilize the Context .

A Data Filtering Strategy

To avoid data contamination for the evaluation stage in Section 4, we apply a pre-filtering strategy during sampling the raw texts for constructing the dataset of IN2 training. Specifically, during sampling C i for generating data, if the sampled C i has a 10-gram overlap with any example in all of our evaluation data (including probing tasks, real-world tasks and short-context tasks), it will not be used for neither generating question-answer pairs nor serving as the random segments [ r j ] .

B Performance on Needle-in-the-Haystack

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

22000

24000

26000

28000

30000

Token Limit

0

10

20

30

40

50

60

70

80

90

100

Needle In A HayStack

0

2

4

6

8

10

Score

Figure 7 shows the performance of FILM-7B on Needle-in-the-Haystack. It shows that FILM-7B has achieved near-perfect performance on Needle-in-the-Haystack within its 32K context window.

C Training Strategy Analysis

Experimental results in Section 4.2 demonstrate the feasibility of IN2 training. We aim to explore further into enhancing the effectiveness and efficiency of IN2 training, particularly from the perspective of training strategies. We are specifically interested in investigating the impact of the following two training strategies: applying the sliding window and adjusting the position encoding. Considering the high cost of training, the following experiments use 20% of all training examples.

Models using sliding windows cannot effectively capture the long distance information. Our experiments involving Mistral models, as shown in Figure 4a, reveal that the performance of Mistral7B-Instruct-v0.1 is awful when the information is positioned at a long distance. It's worth noting that Mistral-7B-Instruct-v0.1 employs the sliding window strategy while Mistral-7B-Instruct-v0.2 does not. Consequently, we are interested in determining whether our IN2 training can still alleviate the lost-in-the-middle problem under the sliding window strategy. We conduct the following two experiments with a 4K sliding window during training:

Apply the sliding window in both pre-training and IN2 training. We take the Mistral-7BInstruct-v0.1 as the backbone model and conduct IN2 training with the same window size (4K).

Apply the sliding window only during the IN2 training. We take the Mistral-7B-Instruct-v0.2 as the backbone model and additionally apply a 4K sliding window during IN2 training.

Figure 8 illustrates the performances of models with sliding windows. It shows that in both two settings with sliding windows, the performances drop dramatically when the distance between the

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction)

FILM-7B (20%)

FILM-7B (20%) + SW (IN2)

FILM-7B (20%) + SW (PT-IN2)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

Relative Positions in 800 Functions

0.0

0.2

0.4

0.6

0.8

1.0

Code Function Retrieval (Backward)

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

Relative Positions in 750 Entities

0.0

0.2

0.4

0.6

0.8

1.0

Database Entity Retrieval (Forward)

retrieval question and information is longer than the sliding window size. It reveals that the sliding window strategy greatly hurts the long-context capability of models.

Training with higher information intensity requires a larger RoPE base θ . The training stage in Section 2 follows the RoPE settings configured for the backbone model. Previous studies on context extension suggest that training with an extended context length necessitates a larger RoPE base θ (Roziere et al., 2023; Xiong et al., 2023; Cai et al., 2024). In the case of our IN2 training, the context length remains unchanged, but the information density is significantly changed to be more uniform. There is a high-level similarity between context-extension training and our IN2 training: both aim to enhance the model's capability to perceive information from a wider range of input positions compared to the previous training stages. Consequently, we carry out experiments to investigate whether the experience from context-extension training could also benefit our IN2 training. Table 5 shows the results with increasing the RoPE base θ from 1 . 0 × 10 6 to 1 . 0 × 10 8 . It shows that increasing the default RoPE base θ of the backbone model leads to better performances on VAL Probing. We suggest to use a 10 times of the default RoPE base θ to conduct IN2 training.

D Implementation and Reasons for Segmentation

Algorithm 1 illustrates how we segment a raw text into ∼ 128-token segments. We set the ∼ 128-token segment as the minimum information unit due to the following consideration:

If the segment contains too few tokens (e.g., 16 tokens or 32 tokens), it might not contain enough information for asking a meaningful question.

If we set a large threshold for segmentation (e.g., 1024 tokens or 4096 tokens), most raw texts will just contain one segment 11 , thus affecting the construction of QA pairs that require the integration and reasoning of information.

Moreover, we do not just use a full raw text to generate a question with GPT-4, as we are concerned about whether GPT-4 will also more focus on the head and the tail of the text. It could result in a local bias for training: the answers are always placed on the boundaries between two segments containing consecutive information text.

Algorithm 1 Implementation of Raw Text Segmentation

Given:

C i : The raw text; Tokjenizer( · ) : The tokenizer of the model;

l = 128 : The minimal length of each segment;

Return:

Based on the above considerations, we use the ∼ 128-token segment. Due to the high cost on data generation, we do not conduct further ablations on this design choice.

E Data Scaling Trend

Table 6 shows the performance of FILM-7B with different training data sizes for IN2 training. Generally, with the data size increasing, the average performance increases and the performance variance in different positions decreases. Such trends are more significant on code and document probing tasks. Note that the training data for IN2 training are almost from natural language corpus. It indicates that increasing the training data size can better help the generalization of long-context capability on different context styles.

F Performance on RULER Benchmark

RULER (Hsieh et al., 2024) is a synthetic benchmark that evaluates the effective context length of the long-context LLMs. It revealed that while all existing long-context models with ≤ 7B sizes claim context size of 32k tokens or greater (except for Llama3), none of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%).

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

Relative Positions in 1800 Sentences

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Performance (%)

Document Sentence Retrieval (Bi-Direction, ~64K)

FILM-7B (Ours)

Table 7 shows that FILM-7B is the first ≤ 7B size model that achieves 32K effective context length. Note that for the evaluation on >32K lengths (i.e., 64K and 128K), we use YaRN (Peng et al., 2023b) to extend the position embeddings without further fine-tuning.

G Position Extension Suffers Lost-in-the-Middle

Some existing studies focused on how to extend the context window without further training, such as YaRN (Peng et al., 2023b). We tried to extend the context window of FILM-7B from 32K to 64K with Yarn and evaluated whether the model can still overcome lost-in-the-middle problem under the extended context length. Figure 9 shows that although FILM-7B outperforms vanilla Mistral-7B on the extended 64K context window, both two models severely suffer the lost-in-the-middle problem. Such an observation indicates that simply extending the position embeddings of the model may not make the model fully utilize the extended context window.

H Training Examples for IN2 Training

Example 1: Fine-Grained Information Awareness (1)

Segment:

Last summer I took a cross-country road trip with a group of people driving different types of electric vehicles. It was an amazing experience, and our trip, our adventures, and our misadventures were portrayed in the documentary Kick Gas. That experience gave me the idea to take my own road trip to experience the freedom of traveling alone on an electric motorcycle. My goals were simple: ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends ...

Question:

What were the primary goals of the individual's solo road trip on an electric motorcycle?

Answer:

The individual's primary goals for the solo road trip on an electric motorcycle were to ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends.

Example 2: Fine-Grained Information Awareness (2)

Segment:

Another addition to the kernel is the ar5523 driver, which was started over five years ago for the Atheros USB chipset of the same name. Extensions for supporting more chips and WLAN adapters were added to a number of other drivers; the brcmsmac WLAN driver, for example, now supports the BCM43224 Broadcom chip, while the rt2800usb RaLink driver supports the Sweex LW323 USB WLAN adapter ...

Question:

What new addition to the kernel supports the BCM43224 Broadcom chip?

Answer:

The new addition to the kernel that supports the BCM43224 Broadcom chip is the brcmsmac WLAN driver.

Example 3: Fine-Grained Information Awareness (3)

Segment:

The Arlington County Board plans to vote Saturday afternoon on giving Amazon $23 million and other incentives to build a headquarters campus in Crystal City, but only after hearing scores of northern Virginia residents and advocates testify for or against the project. The five-member board is expected to support the plan, which was announced amid much hoopla on Nov. 13. The proposed county incentives are part of an agreement in which Amazon would occupy significant office space and bring at least 25,000 high-paying jobs to Arlington in coming years. Opponents hope to postpone the vote until after additional public hearings, where they want representatives of the online retail giant to answer questions directly from anyone in the community.

Question:

What is the Arlington County Board's plan regarding Amazon's proposed headquarters in Crystal City?

Answer:

The Arlington County Board is planning to vote on providing Amazon with $23 million and other incentives to construct a headquarters campus in Crystal City. However, this vote will only take place after listening to the opinions of numerous residents and advocates from northern Virginia. The board, which consists of five members, is anticipated to endorse this plan. The proposal includes Amazon occupying a significant amount of office space and introducing at least 25,000 high-paying jobs to Arlington in the forthcoming years.

Example 4: Integration and Reasoning of Information (1)

Segment 1:

Cloud Cruiser's utilization analytics application enables users to manage and optimize public, private and hybrid cloud usage and spending ...

Segment 2:

This reduces the risk of organizations investing too much or too little in IT, eliminates unused capacity and frees up valuable IT resources for new value-adding projects ...

Segment 3:

As a Cloud Cruiser customer, we have seen firsthand the value that Cloud Cruiser's technology creates by enabling HPE Flexible Capacity to meter and bill for usage of on-premise IT infrastructure in a pay-as-you-go model ...

Question:

What benefits does the acquisition of Cloud Cruiser bring to HPE and its Flexible Capacity business?

Answer:

The acquisition of Cloud Cruiser enriches HPE's Flexible Capacity business by providing a utilization analytics application that allows users to manage and optimize their cloud usage and spending. Cloud Cruiser's software also facilitates the metering and billing for customers' IT consumption. This enables HPE to offer its customers an on-premise physical IT infrastructure at the cost of cloud services, effectively a rental agreement. Moreover, it reduces the risk of organizations either under or over investing in IT, eliminates unused capacity, and frees up IT resources for new projects ...

Example 5: Integration and Reasoning of Information (2)

Segment 1:

Julie Annette Wasden, 45, of 38 Roles St., Haines City, was charged with retail theft at Beall's at 5998 Cypress Gardens Blvd ...

Segment 2:

Katherine D. Collazo Santos, 18, of 108 Hilltop Drive, Winter Haven, was charged with retail theft at Macy's at 700 Third St., S.W. ...

Question:

Who was charged with retail theft at Beall's and who was charged with the same crime at Macy's?

Answer:

Example 6: Integration and Reasoning of Information (3)

Segment 1:

A court presided by Magistrate Joseph Mifsud explained in its judgement of 17 August, 2016 in The Police v Eebis Getu, that the charges of a false criminal report and calumnious accusations may be factually similar, however, from a legal point of view they are two separate and distinct charges. The accused, Eebis Getu was charged with having filed a false police report and with not obeying a legitimate order. She admitted these charges ...

Segment 2:

In passing judgement the court took into consideration that the accused admitted to the charges immediately and also that she did what she did in order to be with her husband in Malta. Magistrate Mifsud referred to what Pope Francis said last June, where today ˘ 2019s information technology brings suffering of others instantly, but we also become immune to tragedies and sufferings ...

Question:

What were the two charges Eebis Getu admitted to, and what was her reason for committing these actions according to the court's judgement?

Answer:

Filing a false police report and not obeying a legitimate order; to be with her husband in Malta.

I Prompts For Data Generation and Training

Example 7: Prompt For Equation 1

Generate one question and the answer from the given context. The question should be highly specific to the information provided in the context. It should not be a general question that suits any context.

Rules to follow when generate the question:

The question should be fully answerable from information present in given context.

Make sure the question is clear and unambiguous.

Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.

Rules to follow when generate the answer:

The answer must use the information provided in the context.

Do not just copy words from the context. Answer the question in your own words.

Example 8: Prompt For Equation 2

Generate one question and the answer from the given context. The context contains several pieces. Answering the question should require the reader to make multiple logical connections or inferences using **at least two pieces**.

Rules to follow when generate the question:

The question should be fully answerable from information present in given context.

Make sure the question is clear and unambiguous.

Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.

Rules to follow when generate the answer:

The answer must use the information provided in the context.

Do not just copy words from the context. Answer the question in your own words.

Example 9: Training Template

Input:


 Below is a context and an instruction. Based on the information provided in the context, write a response for the instruction.

J Limitations

The main limitation of this work lies in the insufficient analysis on the choices of training hyperparameters (e.g., learning rate, batch size, training steps and warm-up rate), data construction settings (e.g., data size and data mixture rate) and backbone models (e.g., model sizes and model architectures).

We just take the commonly used settings for IN2 training without further searching and analyzing. Intuitively, we believe the change of these settings will not affect the feasibility of IN2 training.

K Broader Impacts

This work used pre-trained large language models (i.e., GPT-4 and Mistral-7B) during data construction and training. Therefore, our model may inherit the potential risks of these pre-trained large language models in terms of ethical and safety issues.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Appendix J.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 2, 3 and 4 fully disclose all the information needed to reproduce our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The model and evaluation data will be released. Due to internal data release policies, the training data will not be released soon.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 2 and 4 provide all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show the error bars for the results on probing tasks. For other tasks, there is only one final performance for each task as we use greedy decoding for evaluation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Section K.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We provide the usage guidelines and disclaimers in our github repo (not released now).

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper and the github repo (not released now) include all necessary citations, URLs, acknowledgements and licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The model and probing tasks (to be released) are incorporated with detailed documentations.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces INformation-INtensive (IN2) training, a data-driven approach to address the \"lost-in-the-middle\" problem in large language models (LLMs) with long context windows. The authors hypothesize that this issue stems from insufficient explicit supervision during long-context training. They create a synthetic long-context question-answering dataset that requires models to utilize information from various positions within long contexts. By applying IN2 training to the Mistral-7B model, they develop FILM-7B, which demonstrates significantly improved performance on probing tasks designed to assess long-context utilization across different context styles and retrieval patterns. FILM-7B also shows enhanced performance on real-world long-context tasks while maintaining comparable performance on short-context tasks. The authors provide detailed analyses of their training strategies, including the impact of sliding windows and position encoding adjustments.\n\nSTRENGTHS:\n- The paper introduces IN2 training, an innovative data-driven solution to address the \"lost-in-the-middle\" problem in long-context language models. This tackles a critical issue in the field of long-context LLMs, potentially improving their ability to utilize information from extended contexts.\n- The authors develop a thorough evaluation framework called VAL Probing, which assesses long-context capabilities across various context styles (document, code, and structured-data) and retrieval patterns (forward, backward, and bi-directional). This provides a more rigorous and diverse testing suite compared to existing benchmarks like Needle-in-the-Haystack.\n- FILM-7B, the model developed using IN2 training, demonstrates significant improvements not only on the probing tasks but also on real-world long-context tasks. Importantly, it maintains performance on short-context tasks, showing that the improvements in long-context handling do not come at the expense of other capabilities. The model even achieves performance comparable to or better than GPT-4-Turbo on some tasks.\n\nWEAKNESSES:\n- I do not see a major flaw in the paper. But one potential concern is: since the data is synthesized and the chunk is randomly injected into the document, would the model memorize such synthetic data but lose the generalizability on other long-context tasks beyond such scenarios? \n- The authors may add more details about their construction of the training data.\n\nQUESTIONS:\n- Will the training data be released to the community?\n- I saw a related paper a few months ago that developed a multi-hop qa style needly-in-the-haystack benchmark. I tried to find the paper during the review process but unfortunately, I didn't find it. The basic idea is similar to the training data construction method described in the paper. If the authors also came across this paper, it would be great if you could test your model on the benchmark. If not, it is totally fine. I will revise my comment and add the paper if I can find it later.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues that long-context models suffer from a “lost in the middle” phenomenon due to the infrequency of important data at any one position in the middle of the window during training. To correct for this, the authors propose IN2, a method of training on synthetic long-context QA data to artificially control for when important information is at each point in the context window during training. The model trained with this method, FILM-7B, demonstrates stronger performance than its base model at the 32k context scale.\n\nSTRENGTHS:\nS1. The paper clearly outlines a problem (the lost in the middle phenomenon), a proposed root cause (the insufficient supervision during training for important info in the middle context), and a proposed solution (IN2 training). The work is well-motivated and the proposed root cause, while not empirically demonstrated, seems reasonable. \n\nS2. The method clearly improves the performance of the base model on both synthetic (probing) and realistic long-context tasks. It’s exciting that this improvement comes with the use of very little genuinely-long-context data, and I find it very interesting that increasing the RoPE base is helpful here.\n\nS3. The design of the three probing tasks to focus around directionality of information is well-justified, and I believe the choice of different domains for source text for each task is appropriate, given that no general claims are being made about relative performance between “forward” and “backward” retrieval. The authors also correctly identify that these probing tasks are not sufficient for evaluation and present results on a reasonable set of existing long-context datasets. \n\nS4. I appreciate the concern taken to avoid data contamination and the explicit description of the steps taken (in Appendix A).\n\nWEAKNESSES:\nW1. My biggest concern is that I’m not really convinced that this improvement is due to IN2 rather than the general benefits to downstream tasks of finetuning on QA and instruction tuning data, maybe coupled with some additional benefit from changing the RoPE base that would hold independently of IN2 training’s data intervention. Compared to Mistral-7b-Instruct-v0.2, FILM is trained on ~1.75M additional tokens of data, including quality instruction tuning data and a large amount of QA-style tasks. Comparing this model to Mistral-7b-Instruct-v0.2 directly does not seem to be a fair comparison for evaluating IN2’s qualities as a data selection strategy for finetuning. The real comparison here would be comparing FILM to a Mistral model finetuned on the same quantity and ordering of data but with the IN2-generated synthetic question/answer pairs removed. I understand that this is very computationally costly, but even a checkpoint at 20% of the data mixture (in the same style as the RoPE ablations) would be very helpful to understand how much benefit the method is actually providing. Ideally you could also compare finetuning on non-IN2 instruction data with and without modifying the RoPE base.\n\nW2. The insight that increasing the RoPE base improves performance here is really cool, but I think the reasons why are underexplained and underexplored. The paper attributes this to “higher information intensity” in the context window, but this phrase is not defined in the text. I think it’s okay if the improvements from scaling theta are a purely empirical finding, but any claims about the cause of the improvement require much more justification. \n\nW3.  Notation in 2.1. I think the level of notational formality used here for the synthetic data generation isn’t strictly necessary, but if it’s included, more specificity is necessary. In particular, is [r_j] restricted to never include s_i / [s_i]? What number of segments might be included in [s_i]? In equation 2, it may be helpful to remind the reader of what [r_i] is, as it is not referenced in the text explaining equation 2 before the equation. And when, on line 108, you write that the length balancing is achieved “by reject[ion?] sampling on [r_j]” — does this simply mean you reduce the number of segments you sample? I don’t quite understand how this is rejection sampling, especially given that all segments are the same length.\n\nQUESTIONS:\nQ1. There are existing QA datasets for single-hop and multi-hop reasoning. Why generate data instead of using an existing corpus and mixing in other pretraining data as the filler material?\n\nQ2. The paper focuses on improving performance within the pretrained context length. Do you think IN2 training would be beneficial when finetuning to *extend* the context length as well? \n\nQ3. What is information intensity (line 257), and why would training with higher information intensity require a larger RoPE base? \n\nQ4. Can you provide any evidence from your experiments that your method's gains come from the synthetic QA data used rather than the benefits of additional (instruction tuning) data and training time?\n\nminor suggestions/typos: \n\n- line 205: `lm_eval` provides a preferred citation format; please cite them instead of using a footnote here.\n- line 554: “illustrates how do we segment” → “illustrates how we segment”\n- line 556: capitalized “We” in the middle of the sentence\n- line 127: was it 300 GPU days for all experiments or for training a single model? It seems from Table 3 that there were multiple models (at least partially) trained; it would be helpful to have numbers for total compute used and compute used on the final FILM-7b.\n- Needle in the Haystack is often attributed jointly to [Ivgi et al 2023](https://aclanthology.org/2023.tacl-1.17/) and [Liu et al 2023](https://arxiv.org/abs/2307.03172).\n- Table 1: why are the FiLM numbers for database bolded when GPT-4-Turbo is outperforming it?\n- It would be nice to reference what kinds of experiments are in the appendices somewhere in the main text, to make them more discoverable (e.g. I don’t think Appendix G is mentioned anywhere in the main body).\n- In the second critiquing needle in the haystack, should cite work on the reversal curse and other critiques of needle in the haystack (there have been several, including critique specifically of the single-needle case). Some of this work is already discussed in the related work, but should be cited here to show that there is community discussion about these critiques of needle in the haystack.\n- The claim that sliding window strategies greatly hurt long-context capability— I think this is a bit too broadly stated for the experimental results provided.\n- Line 70: “This demonstrates that training on synthesized long-context data can be generalized to real-world scenarios.” I think care should be taken here to emphasize that this is *additional* training on synthesized long-context data, after already training on real long-context documents at that same length (during Mistral’s pretraining).\n\nLIMITATIONS:\nThe authors identify insufficient ablations as a limitation, which I agree with. However, the concern of the limitations section seems to be that the method might either 1) be brittle to the choice of hyperparameters here (which I agree is unlikely) or 2) perform much better with careful tuning. My concern is really 3) the method may not outperform existing baselines of normal instruction tuning (possibly plus RoPE base scaling).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose two types of training to mitigate the lose-in-the-middle issue of Large Language Models (LLMs). The first one creates some QA pairs. They first pick 128-token segments with which they use LLM to generate QA pairs and mix these 128 tokens with other contexts to form long context QA pairs. The second one extends this approach with multiple segments. The authors also propose three VAL Probing tasks that ask LLMs to print the context of certain phrases. Results show that the model fine-tuned with proposed methods can significantly mitigate the lose-in-the-middle issue on challenging long context tasks and do well on Val Probing while keeping good performance on short context tasks.\n\nSTRENGTHS:\n1. Lost-in-the-middle is an important and challenging tasks that need to be solved to fully utilize the capability of LLMs.\n2. The proposed fine-tuning approaches demonstrate effective performance on many real-world problems, showing the potential usage of the proposed method on Large Language Models (Perhaps GPT-4). Also, it can also keep the good performance on short tasks, showing the fine-tuning won't hurt the unrelated tasks \n3. The paper is well-organized and easy to follow. I like figure one which shows the core idea of whole paper.\n\nWEAKNESSES:\n1. The comparison with natural dataset. What is the difference between fine-tuning on existing long context tasks, such as L-EVAL, LongBench, and SCOLLS. Why do you use synthetic data rather than these data? Do you think your synthetic data can work better? This comparison should be helpful in understanding the effectiveness of proposed approaches. \n\n2. Many choices are not justified.  What is the context you use to mix 128-token segments with gold answers? Are they the best choices compared with using similar contexts like LongBench does? Do you have any criteria for choosing segments with gold answers? \n\n3. Intermediate results are not evaluated. What is the quality of the synthetic QA?  Can we ensure the multihop question is valid?  Is the split of two or more segments natural (not breaking the context)? Which fine-tuning approach helps more on the final results? Some quality evaluation and ablation should be important for the analysis of the proposed approaches.\n\n4. Lacks detail analysis. For instance, the fine-tuning is only conducted on QA tasks. How could it improve the performance of Summarization? Will you include more content in the middle of the source as well? \n\nOverall, the strong improvement in small LLM demonstrates the effectiveness of the proposed method. Thus, I would assume that the proposed approaches can obtain positive answers to most of the questions above.\n\nQUESTIONS:\nSee weakness.\n\nAlso, what is bold in table 2?\n\nLIMITATIONS:\nThey have limitations and broader impact sections. I think more discussion on synthetic data usage will be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles an essential problem in LLMs, called lost-in-the-middle, that causes the lack of information in the middle of lengthy input. As a data-driven solution, the authors propose a new training approach, INformation-INtensive (IN2). IN2 aims to enhance the use of context information in an unbiased position by shuffling the clue part of the data during training. When creating the shuffled data, the authors use GPT-4-Turbo to maintain its grammatical and semantic appropriateness. Furthermore, the authors propose a new model, FILM-7B, additionally trained Mistral-7B on a dataset created by their proposed IN2. To comprehensively evaluate FILM-7B's long-context handling ability, the author introduces VArious Long-context (VAL) Probing. VAL Probing results show that FILM-7B can outperform even GPT-4-Turbo by avoiding being trapped by lost-in-the-middle. Experimental results on long-context tasks from the LongBench collection and short-context tasks, MMLU, BoolQ, RACE-High, CommonsenseQA, ARC-Challenge, HellaSwag, GSM8K, and MATH show that FILM-7B outperforms Mistral-7B models.\n\nSTRENGTHS:\n- This paper proposes a new training method, INformation-INtensive (IN2), that can make trained LLMs handle long-context with mitigating the influence of lost-in-the-middle.\n- By using IN2, the authors train Mistral-7B to make a new model, FILM-7B, than can handle long-context.\n- The authors provide a newly constructed evaluation task, VArious Long-context (VAL) Probing, that covers various long-context tasks.\n- The evaluation result on VAL Probing shows the effectiveness of IN2 based on the improved performance of FILM-7B.\n- The evaluation results on long-context and short-context tasks also show that FILM-7B outperforms FILM-7B.\n\nWEAKNESSES:\n- From the viewpoint of Scaling Laws, comparing Mistral-7B and its further trained model FILM-7B is unfair.\n\nQUESTIONS:\nCurrently, the evaluation is conducted in a zero-shot setting. Is there any possibility that FILM-7B outperforms proprietary models like GPT-4-Turbo on a few-shot setting by using its ability to handle long contexts equally?\n\nLIMITATIONS:\nEven though IN2 mitigates the lost-in-the-middle problem, experimental results show that this problem is still left in some tasks.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}