Make Your LLM Fully Utilize the Context

Authors: shengnan  an
zexiong  ma
zeqi  lin
nanning zheng †♢
jian-guang  lou
weizhu  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 06:21:57

Abstract: While many contemporary large language models (LLMs) can process lengthy input, they still struggle to fully utilize information within the long context, known as the *lost-in-the-middle* challenge.
We hypothesize that it stems from insufficient explicit supervision during the long-context training, which fails to emphasize that any position in a long context can hold crucial information.
Based on this intuition, our study presents **information-intensive (IN2) training**, a purely data-driven solution to overcome lost-in-the-middle.
Specifically, IN2 training leverages a synthesized long-context question-answer dataset, where the answer requires (1) **fine-grained information awareness** on a short segment (~128 tokens) within a synthesized long context (4K-32K tokens), and (2) the **integration and reasoning** of information from two or more short segments.
Through applying this information-intensive training on Mistral-7B, we present **FILM-7B** (FIll-in-the-Middle).
To thoroughly assess the ability of FILM-7B for utilizing long contexts, we design three probing tasks that encompass various context styles (document, code, and structured-data context) and information retrieval patterns (forward, backward, and bi-directional retrieval).
The probing results demonstrate that FILM-7B can robustly retrieve information from different positions in its 32K context window.
Beyond these probing tasks, FILM-7B significantly improves the performance on real-world long-context tasks (e.g., 23.5->26.9 F1 score on NarrativeQA), while maintaining a comparable performance on short-context tasks (e.g., 59.3->59.2 accuracy on MMLU).

Main paper: 
 introduction: 
 To a great mind, nothing is little.-Arthur Conan DoyleLong-context large language models (LLMs) have recently received significant attention within the open-source community (Jiang et al., 2023; Du et al., 2022; Li et al., 2023a; Shi et al., 2023; Team et al., 2023; Team, 2023; Chen et al., 2023a; Song et al., 2023; Liu et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Xiong et al., 2023; Tworkowski et al., 2024; AI et al., 2024; Ding et al., 2024; Mohtashami & Jaggi, 2024; Fu et al., 2024; Cai et al., 2024; Bai et al., 2024; Lv et al., 2024) . The training context windows of many contemporary LLMs have been expanded to tens of thousands of Figure 1 : Performance of FILM-7B, Mistral-7B-Instruct-v0.2, and GPT4-Turbo on our three probing tasks. FILM-7B significantly overcomes the problem of information loss in the middle of the context.tokens, thereby enabling these models to process extensive context as input. This extended training context window can enhance many real-world downstream tasks such as long-context question answering (Kočiskỳ et al., 2018; Dasigi et al., 2021; Bai et al., 2023) and summarization (Fabbri et al., 2019; Huang et al., 2021; Zhong et al., 2021) .However, recent studies have revealed that these long-context LLMs struggle to effectively and robustly utilize all the information provided in the context, known as the lost-in-the-middle challenge (Liu et al., 2024b; Xu et al., 2023) . It implies that while the LLM can comprehend the information at the beginning and end of the long context, it often overlooks the information in the middle. This challenge could significantly hinder the development of long-context LLMs, as they even often fail to pass simple probing tasks such as Needle-in-the-Haystack and passkey retrieval (Mohtashami & Jaggi, 2024) . Consequently, a pressing research question arises: how can we make long-context LLMs fully utilize the information in the long context?We hypothesize that the root cause of lost-in-the-middle stems from the unintentional bias hidden in the general training data. In auto-regressive pre-training, the loss on predicting the next token is more likely to be influenced by a few nearby pre-tokens rather than long-distance tokens (Sharan et al., 2018; Sun et al., 2021) . For supervised fine-tuning and alignment, the system message, which strongly influences the generation of the response, is typically presented at the beginning of the context (Touvron et al., 2023; Cai et al., 2024) . As a result, the general training process may inadvertently introduce a position bias, suggesting that important information is always located at the beginning and end of the context.Based on this hypothesis, our work introduces training to explicitly teach the model that the crucial information can be intensively present throughout the context, not just at the beginning and end. IN2 training is a purely data-driven solution that utilizes a synthesized long-context question-answer dataset. The long context (ranging from 4K to 32K tokens) is concatenated from many short segments (∼128 tokens), and the question-answer (QA) pairs ask for the information contained in one or more segments which are randomly placed in the long context. Specifically, we generate two types of questions, requiring (1) fine-grained information awareness on exactly one short segment, and (2) the integration and reasoning of information from two or more segments. These QA pairs are generated by prompting GPT-4-Turbo (OpenAI, 2023b) with the designed instructions and the raw segments.By applying this information-intensive training on Mistral-7B (Jiang et al., 2023) , we present FILM-7B (FILl-in-the-Middle). To thoroughly assess the long-context information awareness of FILM-7B, we design three probing tasks encompassing various context styles (document, code, and structureddata context) and information retrieval patterns (forward, backward, and bi-directional retrieval).The probing results (Figure 1 ) demonstrate that IN2 training significantly overcomes the lost-in-themiddle problem for the backbone model. Moreover, it can enhance the open-source model to achieve comparable or even more robust performance compared with proprietary LLMs such as GPT-4-Turbo.Beyond these probing tasks, the performance of FILM-7B on real-world long-context tasks also exhibits significant improvements (e.g., 23.5→26.9 F1 score on NarrativeQA (Kočiskỳ et al., 2018) ). This demonstrates that the post-training on synthesized long-context data can be generalized to Prompting Multi-Hop QA real-world scenarios. Moreover, FILM-7B maintains a comparable performance on short-context tasks compared with the vanilla backbone model (e.g., 59.3→59.2 accuracy on MMLU (Hendrycks et al., 2020) ). This indicates that the short-context capability of FILM-7B is not compromised during training. Our further analysis explores how the sliding window strategy and the choice of RoPE base θ influence the performance of IN2 training. 
information-intensive training: 
 This section introduces the construction of the dataset for IN2 training and the detailed training process of our model FILM-7B.Overview. The IN2 training aims to explicitly teach the model that any position in a long context can contain crucial information. To achieve this goal, we construct a long-context question-answer training dataset D = {L i , q i , a i }, where the answer a i to the question q i requires the information contained in some short segments that are randomly placed in the whole long context L i .Figure 2 illustrates an overview of the data construction process. Specifically, the training data D is constructed based on a general natural language corpus C. Given a raw text C i ∈ C, we first generate a question-answer pair (q i , a i ) using a powerful LLM, then synthesize a long context L i that includes the necessary information from C i and other randomly sampled texts from C. We generate two types of question-answer pairs that require (1) the awareness of fine-grained information in the long context, and (2) the integration and reasoning of information appearing at different positions in the long context. We take the realnewslike subset from the C4 corpus (Raffel et al., 2020) as C, and take GPT-4-Turbo (OpenAI, 2023b) as the LLM to generate QA pairs.Fine-grained information awareness. We consider a 128-token segment as the minimum information unit of the context3 . Given a raw text C i , we first randomly extract a 128-token segment s i from it, then generate the q i , a i and L i accordingly, Figure 3 : Three tasks in VAL Probing. The retrieval patterns are determined by the relative positions between the retrieval keywords and the information to be retrieved.EQUATIONwhere (q i , a i ) is sampled by prompting the powerful LLM with the segment s i and the instruction I f , ⊕{•} represents the concatenation of the contained segments, and [r j ] are randomly sampled from 128-token segments in C. Note that I f instructs the LLM to make the question-answer pair highly specific to the information provided in s i .Integration and reasoning of information. Beyond utilizing each single segment, we consider to generate question-answer pairs for information contained in two or more segments. Following the setting of the minimum information unit above, we split a full text C i into a set of 128-token segments [s i ], then generate the q i , a i and L i accordingly,EQUATIONwhere I r instructs the LLM to generate a multi-hop question-answer pair that requires the information within at least two segments in learning rate and 3% warm-up steps. The training process is conducted on 16 nodes of 8x80G A100 GPUs with the full sharding strategy and cpu offload strategy implemented by pytorch FSDP (Zhao et al., 2023) . One entire training process (for a single FILM-7B model) consumes ∼300 GPU days. 
long-context probing: 
 In this section, we first show the preliminary evaluation of FILM-7B on the Needle-in-the-Haystack and discuss about the inadequacies of this probing task. Subsequently, to comprehensively evaluate the long-context information awareness of FILM-7B, we introduce VArious Long-context (VAL) Probing. This includes three tasks that cover various context styles (document, code, and structureddata context) and information retrieval patterns (forward, backward, and bi-directional retrieval). 2 32.1 20.3 59.5 47.5 77.0 47.3 56.2 LongAlign-7B-64K (Bai et al., 2024) 65.3 16.9 39.3 56.0 55.0 36.2 53.2 36.4 LongAlign-13B-64K (Bai et al., 2024) 71.7 13.4 50.8 40.8 82.9 27.0 68.5 27.1 InternLM2-chat-7B (Cai et al., 2024) 68.8 18.7 50.2 44.1 61.2 57.1 60.1 40.0 InternLM2-chat-20B (Cai et al., 2024) 66.4 27.2 63.4 45.5 74.9 57.2 68.2 43.3 GPT-4-Turbo (OpenAI, 2023b) 81.3 31.7 66.1 46.5 89.6 18.0 79.0 32.1 FILM-7B (ours)85.4 6.1 83.3 18.7 89.0 16.8 85.9 13.93.1 Near-Perfect Performance on Needle-in-the-Haystack: Are We There Yet?The Needle-in-the-Haystack (Ivgi et al., 2023; Liu et al., 2024b) is widely used to assess how robustly a model utilizes information positioned in the long context. It reveals that even some powerful proprietary LLMs, such as GPT-4 and Claude 2.1 (Anthropic, 2023) , struggle to fully exploit the information within the long context.We use the Needle-in-the-Haystack task5 to preliminarily evaluate the long-context capability of FILM-7B. Appendix B demonstrates that FILM-7B has achieved near-perfect performance on this task. This result is not surprising as recent open-source LLMs, such as LongAlign (Bai et al., 2024) and InternLM2 (Cai et al., 2024) , have also shown near-perfect performance on this task.However, the near-perfect performance on Needle-in-the-Haystack may overestimate the long-context capabilities of LLMs (Lei et al., 2024; Hsieh et al., 2024) . Specifically, we have the following two concerns:• Needle-in-the-Haystack employs a document-style context, which LLMs could be quite familiar with due to the pre-training on natural language corpora. • The forward retrieval pattern in Needle-in-the-Haystack may simplify the difficulty of information seeking in the long context.The "forward retrieval" means that the information being retrieved directly follows the retrieval keyword in a long context. For example, the default question used in Needle-in-the-Haystack is "What is the best thing to do in San Francisco?" and the answer is contained in "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day." The retrieved information "eat a sandwich and ..." just follows the retrieval keywords "best thing to do in San Francisco".According to the mechanism of induction head (Olsson et al., 2022) , such a following-up copying is an easily learned pattern for LLMs, thus less challenging for evaluating long context utilization (just like the observation of "reversal curse" (Berglund et al., 2024) ).Given these considerations, we suggest that performances on Needle-in-the-Haystack may not adequately reflect the long-context capabilities of LLMs. Therefore, we propose VAL Probing for a more comprehensive evaluation involving various context styles and retrieval patterns.Our retrieval-based VAL Probing considers three context styles (document, code, and structured-data context) and three retrieval patterns (forward, backward, and bi-directional retrieval). Each context in VAL Probing contains ∼32K tokens, and each task contains ∼3K examples. Figure 3 briefly illustrates the contexts and retrieval instructions in VAL Probing.Document Sentence Retrieval (Bi-Direction). The contexts consist of numerous natural language sentences, and the instruction aims to retrieve a single sentence containing a given piece. The sentences are sampled from the abstracts of papers on arXiv6 . This task follows the bi-directional 5 0  1 0 0  1 5 0  2 0 0  2 5 0  3 0 0  3 5 0  4 0 0  4 5 0  5 0 0  5 5 0  6 0 0  6 5 0  7 0 0  7 5 0  8 0 0 Relative Positions in 800 Sentences 5 0  1 0 0  1 5 0  2 0 0  2 5 0  3 0 0  3 5 0  4 0 0  4 5 0  5 0 0  5 5 0  6 0 0  6 5 0  7 0 0  7 5 0  8 0 0 Relative Positions in 800 Functions 5 0  1 0 0  1 5 0  2 0 0  2 5 0  3 0 0  3 5 0  4 0 0  4 5 0  5 0 0  5 5 0  6 0 0  6 5 0  7 0 0  7 5 retrieval pattern, as the expected retrieval results contain words both before and after the given piece in the context. The evaluation metric is the word-level recall score.Code Function Retrieval (Backward). The contexts consist of Python functions, and the instruction aims to retrieve the function name for a given line of code within the function definition. The raw code functions are sampled from the StarCoder (Li et al., 2023c ) dataset7 . We randomly select three lines of definitions for each function. This task follows the backward retrieval pattern, as the function name always precedes the definition. The evaluation metric is the exact-match accuracy.Database Entity Retrieval (Forward). The contexts contain lists of structured entities, each with three fields: ID, label, and description. The query aims to retrieve the label and description for a given ID. The entities are sampled from Wikidata8 . This task follows the forward retrieval pattern, as the label and description follow the ID. We take a relaxed exact-match accuracy as the metric: a 1 score is given if either the label or the description is exactly matched in the response, otherwise a 0 score. 
experiments and analysis: 
 We assess the long-context capability of FILM-7B on both probing tasks and real-world long-context tasks. Moreover, we investigate if the performance in short-context scenarios is affected.Models. We mainly compare FILM-7B with long-context open-source models that have been trained with ≥32K context windows, including the Mistral (Jiang et al., 2023), LongChat (Li et al., 2023a) , ChatGLM (Du et al., 2022) , LongAlign (Bai et al., 2024) , LongWanjuan (Lv et al., 2024) , Yi (AI et al., 2024) and InternLM2 (Cai et al., 2024) . We utilize the instruct/chat versions of these models as most of our evaluation tasks are under the zero-shot instruction-following paradigm. We also draw comparisons with popular proprietary LLMs such as GPT-3.5-Turbo (OpenAI, 2023a) and GPT-4-Turbo (OpenAI, 2023b). All models and tasks employ greedy decoding. For probing tasks, we primarily compare FILM-7B with LongAlign and InternLM2 series, as these models have shown near-perfect performances on Needle-in-the-Haystack.Real-world long-context tasks. We take 9 tasks from the LongBench (Bai et al., 2023) collection to evaluate the long-context capability on real-world scenarios. These tasks encompass long-document question answering (NarrativeQA (Kočiskỳ et al., 2018) , Qasper (Dasigi et al., 2021) and Multi-FieldQA (MultiFQA) (Bai et al., 2023) , multi-document multi-hop reasoning (HotpotQA (Yang et al., 2018) , 2WikiMultihopQA (2WikiMQA) (Ho et al., 2020) and MuSiQue (Trivedi et al., 2022) ), and long-context summarization (GovReport (Huang et al., 2021) , QMSum (Zhong et al., 2021) and MultiNews (Fabbri et al., 2019) ). We employ the middle truncation strategy in LongBench to limit the input within 32K tokens. We report ROUGE-L (Lin, 2004) for summarization tasks and F1 scores for other tasks. The evaluation metrics are computed using the official evaluation scripts9 .Despite these QA and summarization tasks, we also conduct evaluations on few-shot learning tasks, in which the contexts could also be extremely lengthy if there are many "few-shot" examples presented in the context. We take three in-context learning tasks from LongBench, including TREC (Li & Roth, 2002) for few-shot classification, TriviaQA (Joshi et al., 2017) for few-shot QA, and SAMSum (Gliwa et al., 2019) for few-shot summarization.Short-context tasks. We select 8 short-context tasks commonly used for evaluating the general capabilities of models. These include MMLU (Hendrycks et al., 2020) , BoolQ (Clark et al., 2019) , RACE-High (RACE-H) (Lai et al., 2017) , CommonsenseQA (CSQA) (Talmor et al., 2019) , ARC-Challenge (ARC-C) (Clark et al., 2018) , HellaSwag (Zellers et al., 2019) , GSM8K (Cobbe et al., 2021) , and MATH (Hendrycks et al., 2021) . We use 5-shot for MMLU, 8-shot for GSM8K, 4-shot for MATH, and 0-shot for other tasks. We utilize the lm_eval (Gao et al., 2024) for the evaluations on MMLU, BoolQ, RACE-H, ARC-C and HellaSwag, and use the evaluation scripts from An et al. (2024) for other tasks.FILM-7B significantly mitigates the lost-in-the-middle problem. Figure 4a presents the probing results for both FILM-7B and the backbone model, Mistral-7B-Instruct-v0.2. In all three probing tasks within VAL Probing, the vanilla Mistral model experiences substantial information loss at the middle positions in the long contexts. In contrast, our FILM-7B model consistently exhibits robust FILM-7B achieves performance comparable to, or even outperforming, that of GPT-4-Turbo.Figure 1 illustrates the comparison between FILM-7B and GPT-4-Turbo on our probing tasks.Beyond a qualitative comparison between the performance curves of two models, we quantify the long-context performances on VAL Probing using two metrics:• Average score (Avg). We compute the average performances across the entire context length, reflecting the overall long-context utilization.• Min-max gap (Gap). We calculate the differences between the maximum and minimum performances in Figure 3 . A smaller performance gap signifies greater robustness across different positions. VAL Probing presents a more challenging test suite for long-context models. Figure 4b and 4c show the probing results of LongAlign and InternLM2, two state-of-the-art long-context models.Despite their extended training context windows, these models still encounter the lost-in-the-middle problem. This is particularly noteworthy given their near-perfect performance on the Needle-in-the-Haystack task. This comparison suggests that VAL Probing provides a more challenging evaluation for long-context models.In particular, the results on document and database tasks in VAL Probing demonstrate clear comparisons with Needle-in-the-Haystack. Compared to Needle-in-the-Haystack which uses forward retrieval on natural language context, the document task employs natural language context but with bi-directional retrieval, and the database task uses forward retrieval but with structured-data context. These comparisons highlight that both context styles and retrieval patterns significantly contribute to the hardness of the probing tasks.Training on synthesized long-context data effectively generalizes to real-world scenarios. Table 3 and 4 contain the results on various real-world long-context tasks. It shows that FILM-7B also significantly improves the performance of the backbone model in real-world long-context scenarios. Moreover, it also achieves SOTA-level10 performances on these tasks among ∼7B size open-source models. Notably, the long contexts used in IN2 training are all synthesized from short segments. These improvements suggest that the long-context capabilities learned from the synthesized data can be successfully applied to real-world tasks.FILM-7B maintains the performance on short-context tasks. Figure 6 illustrates the performances of FILM-7B and the vanilla backbone model on short-context tasks. It reveals that the overall performances on short-context tasks are almost comparable with minor variances. These results confirm that FILM-7B does not compromise the short-context capabilities of the backbone model.Analysis on training strategies. We are specifically interested in investigating the impact of the following two training strategies: applying the sliding window and adjusting the position encoding.Due to the page limitation, we provide these further ablations and analysis in Appendix C.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Section 2, 3 and 4 fully disclose all the information needed to reproduce our experiments.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
related work: 
 Long-context LLMs. Recent research has significantly contributed to the exploration of training large models with extended context windows (Jiang et al., 2023; Du et al., 2022; Li et al., 2023a; Team et al., 2023; Team, 2023; Xiong et al., 2023; Song et al., 2023; Tworkowski et al., 2024; AI et al., 2024; Cai et al., 2024) . There are primarily two directions in the development of long-context LLMs.(1) Data engineering, which emphasizes the construction of long-context data for training the LLMs. This includes data balancing (Fu et al., 2024) , data order arrangement (Shi et al., 2023) , instruction data collection (Bai et al., 2024) , and data quality measurement (Lv et al., 2024) . Our IN2 training can be categorized into this field.(2) Effective and efficient training, which investigates methods to optimize the training of a long-context model. This encompasses the design of position encoding (Chen et al., 2023a; Liu et al., 2023; Peng et al., 2023b; Ding et al., 2024) , batching strategy (Bai et al., 2024) , parameter-efficient training (Chen et al., 2023b) , and the development of new model architectures (Peng et al., 2023a; Gu & Dao, 2023) .Long-context evaluations. Existing benchmarks for evaluating long-context models can be divided into two categories. (1) Real-world benchmarks that assess general long-context capabilities (e.g., long-context QA, summarization, and language modeling), such as NarrativeQA (Kočiskỳ et al., 2018) , LongBench (Bai et al., 2023) , ZeroSCROLLS (Shaham et al., 2023) , L-Eval (An et al., 2023) , Loogle (Li et al., 2023b) , ∞Bench (Zhang et al., 2024) , and a series of work on perplexity evaluation (Beltagy et al., 2020; Roy et al., 2021; Press et al., 2021; Chen et al., 2023a; Liu et al., 2023; Peng et al., 2023b; Chen et al., 2023b; Ding et al., 2024; Mohtashami & Jaggi, 2024) . ( 2) Probing tasks that provide a more concise reflection of the long-context utilization across different context lengths and positions. These include Needle-in-the-Haystack, passkey retrieval (Mohtashami & Jaggi, 2024) , synthesized document QA (Liu et al., 2024b) , S3Eval (Lei et al., 2024) , Discovery (Li et al., 2024) , RULER (Hsieh et al., 2024) , and the VAL Probing proposed in this study. Among these probing tasks, our VAL Probing is the first to explicitly incorporate a variety of retrieval patterns.training on the open-source model, our FILM-7B exhibits significant improvements on probing tasks and real-world long-context tasks while does not compromise the short-context performance. This is the Appendix of the paper: Make Your LLM Fully Utilize the Context.To avoid data contamination for the evaluation stage in Section 4, we apply a pre-filtering strategy during sampling the raw texts for constructing the dataset of IN2 training. Specifically, during sampling C i for generating data, if the sampled C i has a 10-gram overlap with any example in all of our evaluation data (including probing tasks, real-world tasks and short-context tasks), it will not be used for neither generating question-answer pairs nor serving as the random segments [r j ].B Performance on Needle-in-the-Haystack 0 2 0 0 0 4 0 0 0 6 0 0 0 8 0 0 0 1 0 0 0 0 1 2 0 0 0 1 4 0 0 0 1 6 0 0 0 1 8 0 0 0 2 0 0 0 0 2 2 0 0 0 2 4 0 0 0 2 6 0 0 0 2 8 0 0 0 3 0 0 0 0 Token Limit Figure 8 illustrates the performances of models with sliding windows. It shows that in both two settings with sliding windows, the performances drop dramatically when the distance between the 5 0 1 0 0 1 5 0 2 0 0 2 5 0 3 0 0 3 5 0 4 0 0 4 5 0 5 0 0 5 5 0 6 0 0 6 5 0 7 0 0 7 5 0 8 0 0Relative Positions in 800 Sentences Algorithm 1 illustrates how we segment a raw text into ∼128-token segments. We set the ∼128-token segment as the minimum information unit due to the following consideration:• If the segment contains too few tokens (e.g., 16 tokens or 32 tokens), it might not contain enough information for asking a meaningful question.• If we set a large threshold for segmentation (e.g., 1024 tokens or 4096 tokens), most raw texts will just contain one segment11 , thus affecting the construction of QA pairs that require the integration and reasoning of information.• Moreover, we do not just use a full raw text to generate a question with GPT-4, as we are concerned about whether GPT-4 will also more focus on the head and the tail of the text. It could result in a local bias for training: the answers are always placed on the boundaries between two segments containing consecutive information text. Based on the above considerations, we use the ∼128-token segment. Due to the high cost on data generation, we do not conduct further ablations on this design choice. RULER (Hsieh et al., 2024) is a synthetic benchmark that evaluates the effective context length of the long-context LLMs. It revealed that while all existing long-context models with ≤7B sizes claim context size of 32k tokens or greater (except for Llama3), none of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0 9 0 0 1 0 0 0 1 1 0 0 1 2 0 0 1 3 0 0 1 4 0 0 1 5 0 0 1 6 0 0 1 7 0 0 1 8 0 0Relative Positions in 1800 Sentences Table 7 shows that FILM-7B is the first ≤7B size model that achieves 32K effective context length.Note that for the evaluation on >32K lengths (i.e., 64K and 128K), we use YaRN (Peng et al., 2023b) to extend the position embeddings without further fine-tuning.Some existing studies focused on how to extend the context window without further training, such as YaRN (Peng et al., 2023b) . We tried to extend the context window of FILM-7B from 32K to 64K with Yarn and evaluated whether the model can still overcome lost-in-the-middle problem under the extended context length. Figure 9 shows that although FILM-7B outperforms vanilla Mistral-7B on the extended 64K context window, both two models severely suffer the lost-in-the-middle problem. Such an observation indicates that simply extending the position embeddings of the model may not make the model fully utilize the extended context window.Example 1: Fine-Grained Information Awareness (1)Segment:Last summer I took a cross-country road trip with a group of people driving different types of electric vehicles.It was an amazing experience, and our trip, our adventures, and our misadventures were portrayed in the documentary Kick Gas. That experience gave me the idea to take my own road trip to experience the freedom of traveling alone on an electric motorcycle. My goals were simple: ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends ...What were the primary goals of the individual's solo road trip on an electric motorcycle? Answer:The individual's primary goals for the solo road trip on an electric motorcycle were to ride the great motorcycle roads of America, meet other electric motorcycle riders, and visit friends.Example 2: Fine-Grained Information Awareness (2)Segment:Another addition to the kernel is the ar5523 driver, which was started over five years ago for the Atheros USB chipset of the same name. Extensions for supporting more chips and WLAN adapters were added to a number of other drivers; the brcmsmac WLAN driver, for example, now supports the BCM43224 Broadcom chip, while the rt2800usb RaLink driver supports the Sweex LW323 USB WLAN adapter ...What new addition to the kernel supports the BCM43224 Broadcom chip? Answer:The new addition to the kernel that supports the BCM43224 Broadcom chip is the brcmsmac WLAN driver.Example 3: Fine-Grained Information Awareness (3)Segment:The Arlington County Board plans to vote Saturday afternoon on giving Amazon $23 million and other incentives to build a headquarters campus in Crystal City, but only after hearing scores of northern Virginia residents and advocates testify for or against the project. The five-member board is expected to support the plan, which was announced amid much hoopla on Nov. 13. The proposed county incentives are part of an agreement in which Amazon would occupy significant office space and bring at least 25,000 high-paying jobs to Arlington in coming years. Opponents hope to postpone the vote until after additional public hearings, where they want representatives of the online retail giant to answer questions directly from anyone in the community.What is the Arlington County Board's plan regarding Amazon's proposed headquarters in Crystal City? Answer:The Arlington County Board is planning to vote on providing Amazon with $23 million and other incentives to construct a headquarters campus in Crystal City. However, this vote will only take place after listening to the opinions of numerous residents and advocates from northern Virginia. The board, which consists of five members, is anticipated to endorse this plan. The proposal includes Amazon occupying a significant amount of office space and introducing at least 25,000 high-paying jobs to Arlington in the forthcoming years.Example 4: Integration and Reasoning of Information (1)Segment 1: Cloud Cruiser's utilization analytics application enables users to manage and optimize public, private and hybrid cloud usage and spending ...This reduces the risk of organizations investing too much or too little in IT, eliminates unused capacity and frees up valuable IT resources for new value-adding projects ... Segment 3: As a Cloud Cruiser customer, we have seen firsthand the value that Cloud Cruiser's technology creates by enabling HPE Flexible Capacity to meter and bill for usage of on-premise IT infrastructure in a pay-as-you-go model ...What benefits does the acquisition of Cloud Cruiser bring to HPE and its Flexible Capacity business? Answer:The acquisition of Cloud Cruiser enriches HPE's Flexible Capacity business by providing a utilization analytics application that allows users to manage and optimize their cloud usage and spending. Cloud Cruiser's software also facilitates the metering and billing for customers' IT consumption. This enables HPE to offer its customers an on-premise physical IT infrastructure at the cost of cloud services, effectively a rental agreement. Moreover, it reduces the risk of organizations either under or over investing in IT, eliminates unused capacity, and frees up IT resources for new projects ... Filing a false police report and not obeying a legitimate order; to be with her husband in Malta.Example 7: Prompt For Equation 1Generate one question and the answer from the given context. The question should be highly specific to the information provided in the context. It should not be a general question that suits any context. Rules to follow when generate the question: 1. The question should be fully answerable from information present in given context. 2. Make sure the question is clear and unambiguous.3. Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.Rules to follow when generate the answer:1. The answer must use the information provided in the context. 2. Do not just copy words from the context. Answer the question in your own words.### Context ###: si ### Question ###: {completion}Example 8: Prompt For Equation 2Generate one question and the answer from the given context. The context contains several pieces. Answering the question should require the reader to make multiple logical connections or inferences using **at least two pieces**.Rules to follow when generate the question: 1. The question should be fully answerable from information present in given context. 2. Make sure the question is clear and unambiguous.3. Phrases like 'based on the provided context', 'according to the context', etc, are not allowed to appear in the question.Rules to follow when generate the answer:1. The answer must use the information provided in the context. Output: aiThe main limitation of this work lies in the insufficient analysis on the choices of training hyperparameters (e.g., learning rate, batch size, training steps and warm-up rate), data construction settings (e.g., data size and data mixture rate) and backbone models (e.g., model sizes and model architectures).Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The model and evaluation data will be released. Due to internal data release policies, the training data will not be released soon.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Section 2 and 4 provide all the training and test details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We show the error bars for the results on probing tasks. For other tasks, there is only one final performance for each task as we use greedy decoding for evaluation.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We provide the usage guidelines and disclaimers in our github repo (not released now).Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The paper and the github repo (not released now) include all necessary citations, URLs, acknowledgements and licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The model and probing tasks (to be released) are incorporated with detailed documentations.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Appendix D contains the implementation and our considerations for this design choice.https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2.https://github.com/gkamradt/LLMTest_NeedleInAHaystack.https://info.arxiv.org/help/api/basics.html.https://huggingface.co/datasets/bigcode/starcoderdata.https://www.wikidata.org/wiki/Wikidata:Data_access.https://github.com/THUDM/LongBench.The bold numbers in Table3are SOTA-level results among 7B open-source models. Specifically, for each task, we bold the highest result and the results within a margin to the highest one (0.5 for summarization tasks and 2.0 for others).The raw texts in realnewslike have an average length of ∼600 tokens with the Mistral tokenizer. 
 
 Appendix:  


Decision: The paper identifies an issue with LLMs, commonly referred to as lost-in-the-middle problem, where they exhibit positional bias when integrating information from in-context document, often ignoring information presented in the middle. The core idea is to generate large-scale synthetic question answering dataset (1.1M long-context data) where important information is evenly spread throughout the context to continue training LLMs. In addition, they also propose improved evaluation setting (forward, backward, bi-directional information retrieval) and evaluate their approach extensively. The proposed method shows performance gains on a variety benchmark, both long-context focused and regular text understanding/reasoning benchmark. 

The reviewers have pointed out some weaknesses with the initial draft, such as (1) unfair with base model which were trained on fewer tokens (reviewer HMgi, d47H) and (2) the assessment of generated QA data quality (Reviewer UZJA). The authors have provided additional experimental results and addressed most, if not all, concerns of the reviewers. 

For camera-ready, if time/compute allows, it would be great to have an experiment showing results on performing continued fine-tuning on another base model to show the generality of the proposed method. Overall the paper presents a convincing approach to address well-known issue in LLM.

Reviews: review: paper_summary: paper_summary: This work tackles an essential problem in LLMs, called lost-in-the-middle, that causes the lack of information in the middle of lengthy input. As a data-driven solution, the authors propose a new training approach, INformation-INtensive (IN2). IN2 aims to enhance the use of context information in an unbiased position by shuffling the clue part of the data during training. When creating the shuffled data, the authors use GPT-4-Turbo to maintain its grammatical and semantic appropriateness. Furthermore, the authors propose a new model, FILM-7B, additionally trained Mistral-7B on a dataset created by their proposed IN2. To comprehensively evaluate FILM-7B's long-context handling ability, the author introduces VArious Long-context (VAL) Probing. VAL Probing results show that FILM-7B can outperform even GPT-4-Turbo by avoiding being trapped by lost-in-the-middle. Experimental results on long-context tasks from the LongBench collection and short-context tasks, MMLU, BoolQ, RACE-High, CommonsenseQA, ARC-Challenge, HellaSwag, GSM8K, and MATH show that FILM-7B outperforms Mistral-7B models.
strength_weakness: strength_weakness: - From the viewpoint of Scaling Laws, comparing Mistral-7B and its further trained model FILM-7B is unfair.
questions: questions: Currently, the evaluation is conducted in a zero-shot setting. Is there any possibility that FILM-7B outperforms proprietary models like GPT-4-Turbo on a few-shot setting by using its ability to handle long contexts equally?
limitations: limitations: Even though IN2 mitigates the lost-in-the-middle problem, experimental results show that this problem is still left in some tasks.
paper_summary: paper_summary: In this paper, the authors propose two types of training to mitigate the lose-in-the-middle issue of Large Language Models (LLMs). The first one creates some QA pairs. They first pick 128-token segments with which they use LLM to generate QA pairs and mix these 128 tokens with other contexts to form long context QA pairs. The second one extends this approach with multiple segments. The authors also propose three VAL Probing tasks that ask LLMs to print the context of certain phrases. Results show that the model fine-tuned with proposed methods can significantly mitigate the lose-in-the-middle issue on challenging long context tasks and do well on Val Probing while keeping good performance on short context tasks.
strength_weakness: strength_weakness: 1. The comparison with natural dataset. What is the difference between fine-tuning on existing long context tasks, such as L-EVAL, LongBench, and SCOLLS. Why do you use synthetic data rather than these data? Do you think your synthetic data can work better? This comparison should be helpful in understanding the effectiveness of proposed approaches. 

2. Many choices are not justified.  What is the context you use to mix 128-token segments with gold answers? Are they the best choices compared with using similar contexts like LongBench does? Do you have any criteria for choosing segments with gold answers? 

3. Intermediate results are not evaluated. What is the quality of the synthetic QA?  Can we ensure the multihop question is valid?  Is the split of two or more segments natural (not breaking the context)? Which fine-tuning approach helps more on the final results? Some quality evaluation and ablation should be important for the analysis of the proposed approaches.

4. Lacks detail analysis. For instance, the fine-tuning is only conducted on QA tasks. How could it improve the performance of Summarization? Will you include more content in the middle of the source as well? 

Overall, the strong improvement in small LLM demonstrates the effectiveness of the proposed method. Thus, I would assume that the proposed approaches can obtain positive answers to most of the questions above.
questions: questions: See weakness.

Also, what is bold in table 2?
limitations: limitations: They have limitations and broader impact sections. I think more discussion on synthetic data usage will be helpful.
paper_summary: paper_summary: The paper argues that long-context models suffer from a “lost in the middle” phenomenon due to the infrequency of important data at any one position in the middle of the window during training. To correct for this, the authors propose IN2, a method of training on synthetic long-context QA data to artificially control for when important information is at each point in the context window during training. The model trained with this method, FILM-7B, demonstrates stronger performance than its base model at the 32k context scale.
strength_weakness: strength_weakness: W1. My biggest concern is that I’m not really convinced that this improvement is due to IN2 rather than the general benefits to downstream tasks of finetuning on QA and instruction tuning data, maybe coupled with some additional benefit from changing the RoPE base that would hold independently of IN2 training’s data intervention. Compared to Mistral-7b-Instruct-v0.2, FILM is trained on ~1.75M additional tokens of data, including quality instruction tuning data and a large amount of QA-style tasks. Comparing this model to Mistral-7b-Instruct-v0.2 directly does not seem to be a fair comparison for evaluating IN2’s qualities as a data selection strategy for finetuning. The real comparison here would be comparing FILM to a Mistral model finetuned on the same quantity and ordering of data but with the IN2-generated synthetic question/answer pairs removed. I understand that this is very computationally costly, but even a checkpoint at 20% of the data mixture (in the same style as the RoPE ablations) would be very helpful to understand how much benefit the method is actually providing. Ideally you could also compare finetuning on non-IN2 instruction data with and without modifying the RoPE base.

W2. The insight that increasing the RoPE base improves performance here is really cool, but I think the reasons why are underexplained and underexplored. The paper attributes this to “higher information intensity” in the context window, but this phrase is not defined in the text. I think it’s okay if the improvements from scaling theta are a purely empirical finding, but any claims about the cause of the improvement require much more justification. 

W3.  Notation in 2.1. I think the level of notational formality used here for the synthetic data generation isn’t strictly necessary, but if it’s included, more specificity is necessary. In particular, is [r_j] restricted to never include s_i / [s_i]? What number of segments might be included in [s_i]? In equation 2, it may be helpful to remind the reader of what [r_i] is, as it is not referenced in the text explaining equation 2 before the equation. And when, on line 108, you write that the length balancing is achieved “by reject[ion?] sampling on [r_j]” — does this simply mean you reduce the number of segments you sample? I don’t quite understand how this is rejection sampling, especially given that all segments are the same length.
questions: questions: Q1. There are existing QA datasets for single-hop and multi-hop reasoning. Why generate data instead of using an existing corpus and mixing in other pretraining data as the filler material?

Q2. The paper focuses on improving performance within the pretrained context length. Do you think IN2 training would be beneficial when finetuning to *extend* the context length as well? 

Q3. What is information intensity (line 257), and why would training with higher information intensity require a larger RoPE base? 

Q4. Can you provide any evidence from your experiments that your method's gains come from the synthetic QA data used rather than the benefits of additional (instruction tuning) data and training time?

minor suggestions/typos: 

- line 205: `lm_eval` provides a preferred citation format; please cite them instead of using a footnote here.
- line 554: “illustrates how do we segment” → “illustrates how we segment”
- line 556: capitalized “We” in the middle of the sentence
- line 127: was it 300 GPU days for all experiments or for training a single model? It seems from Table 3 that there were multiple models (at least partially) trained; it would be helpful to have numbers for total compute used and compute used on the final FILM-7b.
- Needle in the Haystack is often attributed jointly to [Ivgi et al 2023](https://aclanthology.org/2023.tacl-1.17/) and [Liu et al 2023](https://arxiv.org/abs/2307.03172).
- Table 1: why are the FiLM numbers for database bolded when GPT-4-Turbo is outperforming it?
- It would be nice to reference what kinds of experiments are in the appendices somewhere in the main text, to make them more discoverable (e.g. I don’t think Appendix G is mentioned anywhere in the main body).
- In the second critiquing needle in the haystack, should cite work on the reversal curse and other critiques of needle in the haystack (there have been several, including critique specifically of the single-needle case). Some of this work is already discussed in the related work, but should be cited here to show that there is community discussion about these critiques of needle in the haystack.
- The claim that sliding window strategies greatly hurt long-context capability— I think this is a bit too broadly stated for the experimental results provided.
- Line 70: “This demonstrates that training on synthesized long-context data can be generalized to real-world scenarios.” I think care should be taken here to emphasize that this is *additional* training on synthesized long-context data, after already training on real long-context documents at that same length (during Mistral’s pretraining).
limitations: limitations: The authors identify insufficient ablations as a limitation, which I agree with. However, the concern of the limitations section seems to be that the method might either 1) be brittle to the choice of hyperparameters here (which I agree is unlikely) or 2) perform much better with careful tuning. My concern is really 3) the method may not outperform existing baselines of normal instruction tuning (possibly plus RoPE base scaling).
paper_summary: paper_summary: This paper introduces INformation-INtensive (IN2) training, a data-driven approach to address the "lost-in-the-middle" problem in large language models (LLMs) with long context windows. The authors hypothesize that this issue stems from insufficient explicit supervision during long-context training. They create a synthetic long-context question-answering dataset that requires models to utilize information from various positions within long contexts. By applying IN2 training to the Mistral-7B model, they develop FILM-7B, which demonstrates significantly improved performance on probing tasks designed to assess long-context utilization across different context styles and retrieval patterns. FILM-7B also shows enhanced performance on real-world long-context tasks while maintaining comparable performance on short-context tasks. The authors provide detailed analyses of their training strategies, including the impact of sliding windows and position encoding adjustments.
strength_weakness: strength_weakness: - I do not see a major flaw in the paper. But one potential concern is: since the data is synthesized and the chunk is randomly injected into the document, would the model memorize such synthetic data but lose the generalizability on other long-context tasks beyond such scenarios? 
- The authors may add more details about their construction of the training data.
questions: questions: - Will the training data be released to the community?
- I saw a related paper a few months ago that developed a multi-hop qa style needly-in-the-haystack benchmark. I tried to find the paper during the review process but unfortunately, I didn't find it. The basic idea is similar to the training data construction method described in the paper. If the authors also came across this paper, it would be great if you could test your model on the benchmark. If not, it is totally fine. I will revise my comment and add the paper if I can find it later.
limitations: limitations: Yes.

Comments: title: The response meets my expectations
comment: Thanks for the in-depth analysis and reply the concerns one by one! I hope the authors can also include the results and analysis in the final version. Thanks!
Thanks for the additional results and the response! 

>  demonstrate that normal instruction tuning (IT) cannot alleviate the lost-in-the-middle problem

Thanks for these results! This addresses my primary concern.

> "higher information intensity" is not a quantitative measure but a conceptual phrase

I think if you're going to use this term, you should define it with some degree of rigor. It sounds like this concept could be described also as "more uniform information density," which is a more commonly used term. 

> rationale behind our adjustment of RoPE 

This intuition here is interesting, and would be a good addition to the paper!

> as we have taken many existing QA datasets in our evaluations (including both long-context and short-context datasets), by utilizing synthetic data exclusively, we can circumvent potential data contamination issues 

It's worth noting that this doesn't completely avoid issues of data contamination-- synthetic data generated from GPT4 could leak some information from datasets that GPT4 saw during pretraining-- but data contamination from pretraining is complex and this is a good-faith attempt to avoid it.


While I think the paper could still benefit from some revisions before camera-ready, my primary concern about the method is addressed and I think the work is sound, so I'll raise my score 4 -> 6.
Thanks for the response! Yeah, it might be the RULER benchmark. Thanks for your efforts in building this. Really hope your training data will be released to the community!
I appreciate your in-depth responses based on the actual evaluation results. The result addressed every concern raised by me. I am really thankful to the authors for doing such high-quality research.