Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Authors: zhenyu  zhang
runjin  chen
shiwei  liu
zhewei  yao
olatunji  ruwase
beidi  chen
xiaoxia  wu
zhangyang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:07:31

Abstract: This paper aims to overcome the ``lost-in-the-middle'' challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code will be made public upon acceptence.

Main paper: 
 introduction: 
 Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of applications [1, 2] , from understanding extensive texts [3, 4] and managing day-long conversations [5, 6] to code generation [7, 8] and science discoveries [9, 10] . Recent system support advancements [11, 12] have enabled training transformers for any L sequence length even with O(L 2 ) computational complexity. This is exemplified by models such as MPT [13] and Mistral [14] pre-trained with sequence lengths 16k and 32k respectively.Nevertheless, emerging research reveals the constrained efficacy of LLMs in managing tasks requiring long contextual understanding. Particularly, [15] demonstrated a substantial degradation in LLMs' performance when crucial information is positioned amidst a lengthy context, a phenomenon they refer to as "lost-in-the-middle". One explanation is about the use of rotary positional embedding (RoPE) [16] , a prevalent positional encoding technique used in open-source LLMs. As a relative position embedding, RoPE incorporates a long-term decay property, predisposing the model to prioritize current/nearby tokens while paying less attention to further ones. [17] identified a surprising trend attributed to the Softmax operation where attention scores are disproportionately allocated into initial tokens, irrespective of their relevance to the language modeling task. Despite the presence of considerable redundancy in long-context inputs [18] , crucial information may be located across different positions. The inclination of LLMs to overlook the middle section presents a challenge for their applications, particularly in the context of long-context reasoning. Several approaches successfully extend pre-trained LLMs with context up to extreme token length, either through sparse selection of crucial tokens during generation [17, 18, 19] or by modifying positional encoding [20, 21] . Nevertheless, these approaches primarily aim to extend the context length of LLMs and, consequently, fall short in addressing the "lost-in-the-middle" problem when applied out-of-the-box.Figure 1 : The x-axis illustrates the placement of essential information within the prompt, ranging from start to end. The green bar serves as a standard baseline, illustrating the "lost-in-the-middle" phenomenon. We introduce our method, Multi-scale Position Encoding (Ms-PoE), which requires neither additional fine-tuning nor increased memory usage. Instead, it involves a simple remapping of the position embedding depicted in Figure 2 , which enables the important information in the middle to be detected effectively (brown bars). For more details, see Section 4.2 and Figure 5 .Efforts have been made to enhance LLMs' capacity to capture vital information located within the middle of the context. These include extra memory bank [22] , reordering the input context based on relevance [23, 24] , enhancing the information searching and reflection ability via attention strengthening tasks [25, 26] , splitting the input into short segments and applying short-text models [27] . For example, [23] empirically discovered that LLMs tend to emphasize more on the current window while still paying more attention to the relevant text than distracting content. They subsequently introduced "attention sorting" where the main idea is iteratively sorting documents based on their attention scores, such that critical information will likely be placed at the end, to fit the position-biased nature of RoPE. [24] conducted parallel runs of LLMs with different RoPE angles, thereby mitigating the risk of overlooking crucial information through a weighted sum of the outputs. These approaches usually require additional memory or multiple inference runs, which can be expensive for LLMs.In this paper, we aim to address the "lost-in-the-middle" problem by reintroducing the concept of multi-scale features from computer vision into the context of Transformer-based LLMs. Multiscale features, well-established in Inception-style models [28, 29, 30] , utilize parallel employment of kernels with different sizes to fuse multi-scale information, spanning short to long distances. Introducing multi-scale operations into LLMs intuitively can help compensate for crucial information located in the middle, which might be easily overlooked by full attention operation. Unlike modifying the attention module to form multi-scale attention, we choose to re-scale the indices of positional encoding. This decision is grounded not only in its effectiveness in easily adjusting the scale of the context window by simply changing the position indices [20] but also in the potential of down-scaling the position indices to relieve the long-term decay property introduced by RoPE. However, this approach was initially introduced to extend context windows, and its performance regarding the "lostin-the-middle" problem remains uncertain for several reasons: (i) Indice re-scaling forces position embeddings of original context window to reside in a narrower region, leading to performance degradation in the original context window as shown in [20] . (ii) Uniformly applying the same scaling ratio throughout the entire model might be sub-optimal to preserve essential knowledge learned during pre-training; (ii) Fine-tuning is necessary for the original approach, albeit minimal. The impact without fine-tuning remains unknown.To this end, we systematically visit the position indices scaling regarding the "lost-in-the-middle" problem and counter-intuitively discover that it is possible to slightly mitigate the "lost-in-the-middle" issue if we carefully choose the scaling ratio to be around 1.5-2. Additionally, we observe that different attention heads exhibit varying sensitivity to the position shift of the relevant document. Some attention heads are "position-aware", consistently capturing relevant information even with position shifts, while others may occasionally capture position changes, and some heads are completely insensitive to position changes. This highlights the need to treat attention heads separately when re-scaling position indices.Contribution. Inspired by the above observations, we introduce Multi-scale Positional Encoding (Ms-PoE), a simple yet effective plug-and-play approach that can enhance the long-context reasoning capability of pre-trained LLMs without requiring fine-tuning or introducing any additional overhead. Ms-PoE meticulously assigns distinct scaling ratios to different attention heads, with the scaling factor monotonically increasing from "position-aware" heads to "position-unaware" heads. This enables us to improve long-context ability by re-scaling position indices to shorter values while preserving essential knowledge acquired during the pre-training phase. The efficacy of Ms-PoE is substantiated through extensive experiments. By simply re-scaling the indices of positional encoding, Ms-PoE consistently enhances the performance of various LLMs including Llama-2 [31] , StableBeluga [32] and Vicuna [33] on the ZeroSCROLLS [34] , achieving a notable accuracy gain of up to 3.8. 
background and related works: 
 In this section, we provide a concise overview of the background knowledge and recent literature about the generative inference process of Large Language Models (LLMs), their abilities for long-context reasoning, and details of positional encoding.The generative inference process in LLMs can be categorized into two distinct phases: ① Prefilling Stage: In this initial phase, LLMs receive an input sequence containing detailed instructions that define a specific generation goal. Throughout this stage, intermediate Key and Value embeddings are generated at each layer and stored in memory, commonly referred to as the KV cache. ② Decoding Stage: This phase involves retrieving embeddings from the KV cache to generate new tokens. The decoding process is inherently iterative, where each newly generated token serves as input for the subsequent token generation. In real-world LLM deployment, the cumulative length of input sequences and the subsequently generated text can reach several thousand or even millions of tokens, presenting significant challenges for the LLMs' long-context reasoning capability.Two challenges for LLMs in handling long-context reasoning tasks. One is to extend the context window to process sentences that exceed the pre-trained window length. Another is the "lost-in-thewindow" issue where LLMs likely overlook the information located in the middle of the sentences.The reason for the former challenge is that open-source LLMs are usually pre-trained with fixed sequence lengths, such as 4096 for Llama-2 [31] . When the sequence length surpasses the predefined context length used in pre-training, LLMs often suffer from performance collapses and thus generate incoherent or fragmented text. Recent efforts to address this issue can be broadly categorized into two streams. Recently, several works have been proposed to address this issue, which can be broadly categorized into two streams. The first one explores from the expansion of positional encoding, with notable contributions including PI [20] , CLEX [35] , YaRN [36] , Self-Extend [21] . On the other hand, some works modify the attention mechanism, such as StreamingLLM [17] , LM-Inifinite [19] , H 2 O [18] , TOVA [37] , Zebra [38] , and Activation Beacon [39] . These approaches have successfully expanded the contextual window with minimal or no additional training overhead.Despite the extended context window, LLMs still face a significant challenge in long-context inference due to the uneven utilization of lengthy inputs. [15] conducted a pivotal investigation, revealing that LLMs tend to overlook the middle portion of the input. This bias compromises the practical application of LLMs, as critical information may be located in the middle part of the input, leading to unreliable outputs. To tackle this issue, [23] introduced 'attention sorting' to reorder inputs, placing critical information at the end. However, this method's reliance on potentially biased attention scores to identify crucial content may compromise its reliability, and the prerequisite knowledge of document count in inputs may affect its effectiveness. [24] utilize Attention Buckets, an ensemble approach that combines multiple forward processes with positional modifications. However, this technique necessitates a considerably higher computational cost. Other general approaches for enhancing long-context reasoning include prompt compression [40] , retrieval augmentation [26] , and inference refinement by constructing memory trees [41] while these approaches typically necessitate extra LLMs' assistance or bring extra computational cost.For effective processing of long contexts, LLMs necessitate the explicit encoding of positional information. Common techniques include absolute positional embedding and relative positional encoding. Absolute positional embedding integrates word embeddings with an additional positional vector based on the token's absolute position, which can be either fixed [42] or learnable [43, 44, 45, 46, 47] . In contrast, relative positional encoding, increasingly popular in contemporary LLMs, encodes the relative distances between tokens instead of their absolute positions. Notable among these are Rotary Position Embedding (RoPE) [16] that widely implemented in models like Llama [31] , Falcon [48] , Mistral [49] , and ALiBi [50] , which used in MPT [13] .RoPE. The primary goal of RoPE [16] is to encode positional information such that the inner product of the query and key embeddings inherently contains the relative position information:f (q m , m) T f (k n , n) = g(q m , k n , m -n)Here, f is the positional encoding function applied to the query and key embeddings at positions m and n, respectively. To satisfy this condition, the function f is defined as a vector-valued complex function, as follows:f (x, m) = xe imθ = [(x 1 + ix 2 )e imθ1 , (x 3 + ix 4 )e imθ2 , ..., (x l-1 + ix l )e imθ l/2 ] TIn this equation, l represents the dimension of the embeddings, θ k = 10000 -2k/l , and i is the imaginary unit. For calculating the attention score, RoPE considers the real part of the product, specifically Re(f (q m , m) T f (k n , n)). This approach allows RoPE to effectively integrate relative positional information into the attention mechanism of transformer models. 
methodology: 
 In this section, we present the details of our Multi-Scale Positional Encoding (Ms-PoE) approach. Section 3.1 demonstrates that the context utilization of LLMs can be directly enhanced by re-scaling the positional information without incurring extra training costs. Then, Section 3.2 analyzes the properties of various attention heads in LLMs. Section 3.3 outlines the detailed pipeline of Ms-PoE. Current LLMs tend to neglect information located in the middle of the context, despite its potential relevance. This "lost in the middle" phenomenon likely arises from two contributing factors: (i) Casual Attention, where preceding tokens undergo a higher number of attention processes, leading LLMs to disproportionately favor initial tokens. This phenomenon has been demonstrated in recent research which highlights the pivotal role of the initial tokens in model generation [19, 17] , with these starting tokens consistently accumulating higher attention scores [18] . (ii) The utilization of RoPE [16] introduces a longterm decay effect, diminishing the attention score of distantly positioned yet semantically meaningful tokens. The combination of these factors contributes to LLMs neglecting the context in the middle part. To tackle this issue and improve the context utilization of LLMs, a seemingly unreasonable yet remarkably effective strategy is to down-scale positional information [38] . Formally, RoPE encodes the position as f (x, m) = xe imθ . By substituting the position m with m r , we can force the long-distance tokens to reside in the shorted range, which can potentially alleviate the long-term decay effects by a factor of r. In the following sections, we conduct experiments to evaluate how LLMs' context utilization responds to varying re-scaling ratios r.Details. Experiments are conducted using Llama-2-7B-Chat [31] and Vicuna-7B [33] on the Multi-Document Question Answering (MDQA) task [15] . Each question includes ten documents, with only one relevant to the question. By varying the position of the relevant document, we can evaluate LLMs' context utilization properties. For each position of the key document, we calculate the accuracy over 500 samples. And results show in Figure 3 include both the Average accuracy over the 10 documents as well as Gap accuracy, i.e., the difference between the best and worst accuracy when varying the positions of the relevant document.Llama-2-7B-Chat Vicuna-7BAccuracy (%) Results. Figure 3 demonstrates that the gap accuracy can be alleviated via appropriate positional re-scaling.Particularly, we see that the Gap between the best and the worst accuracy is greatly reduced when increasing the rescaling ratio. An enhanced average accuracy can be observed with a scaling ratio equals near 1.5. Additionally, changing the scaling ratio also affects the favored zone of LLMs. With a small scaling ratio (e.g., 0.5), LLMs tend to focus more on the most recent part of the context, while with a large ratio (e.g., 2.5), LLMs favour the beginning part.Improving context reasoning via positional re-scaling. Building upon this, we introduce a plugand-play treatment for RoPE by re-scaling the position of each token. This approach seamlessly enhances the context utilization of LLMs without requiring additional training or inference overhead. However, there is a trade-off in terms of LLMs favoring certain context regions. For instance, when r = 0.5, LLMs achieve peak accuracy when the relevant document is located at the end of the input, while at the beginning for r = 1.5. It remains challenging to decide which re-scaling ratio to use, given that we lack prior knowledge of the location of relevant information in real-world applications. Moreover, as the re-scaling ratio increases, LLMs may face the positional out-of-distribution (O.O.D) issue [21, 20] , where many position values do not directly exist during pretraining (e.g., using 0.1, 0.2, ..., 0.9 for position when LLMs only recognize 1, 2, ..., 9 during pretraining), potentially reducing their average reasoning ability. To tackle these challenges, we investigate the head-wise properties of LLMs and propose a multi-scale positional encoding approach.Inspired by recent works that leverage attention patterns to identify most crucial tokens and optimize inference efficiency [37, 18, 51] , we carry out a preliminary study to investigate the interaction between attention patterns and token positions.Details. We visualize the attention patterns of the most recent query with results collected from Vicuna-7B on the MDQA task, following [37] . In the same input sample, we manually switch the position of the relevant document from the beginning to the end and illustrate the attention scores across different positions.Observation. We observe the presence of "position-aware" attention heads capable of capturing relevant information even when its position is shifted. As an example, we select the eighth attention head in the fifteenth layer, depicted in the bottom of Figure 4 , while consistent observations can be drawn across different layers and input samples. Firstly, most attention scores are near zero and can be ignored, consistent with other studies highlighting high sparsity in attention blocks [18, 52, 53] .For the remaining positions, these "position-aware" attention heads can capture important information across positions, with attention patterns shifting as the position of relevant tokens changes. However, for other attention heads (upper subfigure in Figure 4 ), they fail to capture relevant tokens and only attend to the beginning and end words, contributing to the "lost-in-the-middle" issue.Based on this observation, we devise a position-aware strategy to adaptively determine the re-scaling ratio via the inherent properties of different attention heads. For the "position-aware" attention heads, we assign a re-scaling ratio close to one to avoid changing their functionality significantly, as altering them too much could degrade performance due to the positional O.O.D issue. On the other heads, we condense their position indices to a higher degree, providing more opportunity to alleviate the persistent bias toward the beginning and recent tokens. To identify the properties of n h attention heads, we introduce a Position-Awareness Score S P ∈ R n h formulated as:EQUATIONIn Equation 1, A represents the attention score vector of the most recent query, and α is a hyperparameter determining the threshold of effective attention scores. In all experiments, we default to using α = 3, and the corresponding important tokens are highlighted in Figure 4 , which are shown in red. In the spirit of numerous studies that investigate the outlier properties in LLMs [17, 54, 55] , we utilize S P to evaluate the ratio of effective attention tokens, where a larger S P value implies better positional awareness.The pipeline for utilizing Multi-Scale Positional Encoding (Ms-PoE) in LLM inference is: Given a pre-trained LLM, we initially replace the original rotary positional encoding with Ms-PoE. As illustrated in Figure 2 , Ms-PoE condenses the positional indices of RoPE and employs different re-scaling ratios for each attention head. The re-scaling ratios are assigned during the prefilling stage, where we first calculate the distribution of attention scores for the most recent query and obtain the corresponding position-awareness score for each attention head. Larger re-scaling ratios are subsequently allocated to attention heads exhibiting smaller position-awareness scores. And the set of re-scaling ratios r defaults to a linear range from R min to R max . For example, the ith sorted-head would be using re-scaling ratioEQUATIONOnce the re-scaling ratios are assigned, they remain fixed in the subsequent decoding stage. We consistently using R min = 1.2 and R max = 1.8 is our experiments.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 The goal of this section is to demonstrate Ms-PoE, a plug-and-play positional encoding capable of enhancing the context utilization of LLMs, and consequently improving the quality of generation across diverse models and downstream reasoning tasks. Our main results can be summarized below. In Section 4.1, we demonstrate that Ms-PoE consistently enhances reasoning over long contexts for a range of tasks in the ZeroSCROLLS benchmarks [34] , all without the need for additional training. Additionally, Ms-PoE exhibits superior performance when compared to other methods in the field, including PI [20] and Self-Extend [21] . Detailed comparison results are shown in Tables 1 and 2 .In section 4.2, we highlight that Ms-PoE improves the context utilization and achieves consistent improvement when varying the position of critical information, as shown in Figure 1 & 5 .In Section 4.3, we conduct multiple ablation studies to assess the effectiveness of Ms-PoE under different scaling ratios and selection strategies. Results are reported in Table 3 & 4.We empirically validate the ability of Ms-PoE to enhance long-context reasoning with a noteworthy improvement up to 13.4 without additional training overhead. Notably, our approach surpasses other competitive baselines, demonstrating improvements from 2.64 to 43.72.Experimental Setup. In our experiments, we select seven representative LLMs, including Llama-2chat-7B and 13B [31] , StableBeluga-7B and 13B [32] , and Vicuna-7B [33] , along with its longercontext version (Vicuna-7B-16K). To comprehensively evaluate the long-context reasoning abilities of LLMs, we choose seven tasks from ZeroSCROLLS [34] , spanning all four task categories: ① Document Summarization (Government and SummScreenFD), ② Query-Based Summarization (QMSum and SQuALITY), ③ Question Answering (Qasper and NarrativeQA), and ④ Information Aggregation (BookSumSort). We also compare Ms-PoE with other competitive methods on additional generation tasks, including Multi-document Question Answering (MDQA) and Key-Value Retrieval [15] .. Table 1 summarizes the main results, yielding several key observations: (i) By simply substituting the original positional encoding module with our Ms-PoE, the performance of LLMs consistently improves across all tasks without additional training, resulting in an average performance enhancement ranging from 0.6 to 3.8; (ii) These improvements hold consistently across different model sizes of 7 billion and 13 billion parameters; (iii) The efficacy extends to LLMs with varying sequence lengths, such as Vicuna-7B and its extended version, Vicuna-7B-16K, both showing improvements from 3.3 to 3.8.Outperform other competitive methods. We conduct a thorough comparison between Ms-PoE and other competitive methods, including Positional Interpolation (PI) [20] and Self-Extend [21] , both of which modify position indices without utilizing head-wise properties. For PI, we employ the scaling ratio as the average value of our method while for Self-Extend, we set the group size as 2 with the local window size as 1024. The results presented in Table 2 consistently showcase the superiority of our approach over other baselines, demonstrating improvements of up to 3.92 and 43.72 for MDQA and Key-Value Retrieval, respectively. Such improvements might come from two primary factors. Firstly, the incorporation of head-wise properties offers a more adaptive strategy for positional modification. Secondly, our approach enhances the general context utilization ability. Notably, our approach demonstrates superiority even when the core document or key is positioned at the end of the input, surpassing other baselines with improvements ranging from 2.4 to 27.8.This performance surpasses the recent work [23] , which addresses the "lost-in-the-middle" effect by reordering key documents and placing them at the end of the input. When the identified core document is already located at the recent area, such method can not gain further improvements, while our approach offers a fine-grained strategy to improve context utilization. We assess the context utilization ability of our approaches on two tasks, including multi-document question answering (MDQA) and key-value retrieval (KV retrieval) tasks from [15] . Such tasks provide a good input structure and offers the flexibility to switch the position of crucial information, thus evaluate the context utilization ability of LLMs.Experimental Setup. In the MDQA task, each input sample comprises ten documents and one question, with only one document being relevant to the question. For the KV retrieval tasks, there are 50 key-value pairs with one question querying the value of the chosen key. In both tasks, we systematically switch the important document or key-value pair from the beginning to the end and report the accuracy of the generated context. All results are averaged across 500 samples. The Gap accuracy metric is employed to assess the context utilization ability of LLMs, defined as the gap between the best and worst accuracy when varying the position of important information.Main Results. As depicted in Figure 5 and 1, Ms-PoE demonstrates consistent improvement across different models, tasks and critical positions. Even when the important information exists in the sweet region (beginning and end) of the input, Ms-PoE achieves significant performance improvements ranging from 3% to 6%, highlighting its efficacy in enhancing generation quality. Moreover, the "lost-in-the-middle" issue is notably alleviated, with Ms-PoE quantitatively reducing the gap accuracy by approximately 2% to 4%, showcasing improved context utilization.This section conducts a further evaluation of the effectiveness of Ms-PoE by addressing the following questions: Q1: How does the effectiveness of Ms-PoE relate to the head-wise selection strategy of the scaling ratio? Q2: How does the model perform with different scaling ratios? A1: Positional awareness metrics achieve superior performance compared to other strategies. For a set of scaling ratios r ∈ R n h , where n h is the number of attention heads, and using scaling ratios linearly ranging from 1.2 to 1.8, we evaluate various strategies for assigning these ratios to different attention heads. These strategies include: ① Random, which randomly assigns the scaling ratios to each head within each layer; ② Sequential, performing the assignment based on the original head order; ③ Entropy, where we follow metrics measuring the sparsity level of attention scores [56] . Larger entropy implies less sparse attention scores, indicating the model attends to more tokens rather than just the beginning and end words, so we assign a scaling ratio near to 1, and vice versa for larger ratios. Results in Table 3 demonstrate that the proposed position-awareness effectively captures the head-wise properties of LLMs, enhancing performance when critical information is located at various positions-beginning, middle, or end. This leads to an average accuracy gain of 3.2 (65.3 v.s. 62.1). A2: Ablation study of the scaling ratios.We first examined the effect of uniform scaling ratios across all heads on model performance. Our findings, outlined in Table 4 , indicate that adjusting the scaling ratio between 0.5 and 2.5 can significantly enhance generative performance and mitigate the "lost-in-themiddle" effect by 1.0% (63.1% v.s. 62.1%), particularly with a ratio of 1.5. Further testing with an average ratio of 1.5 across all heads revealed that an optimal range exists between 1.2 and 1.8, leading to an additional 2.2% (65.3% v.s. 63.1%) accuracy improvement with our approach, Ms-PoE. Based on these results, we established these ratios as our experimental standard. 
conclusion: 
 In this paper, we present a plug-and-play strategy designed to address the "lost-in-the-middle" challenge observed in LLMs. This challenge stems from the persistent bias exhibited by LLMs towards the beginning and local content within the input, leading to the neglect of crucial information in the middle. Our investigation reveals the effects of position indice rescaling and the head-wise position-awareness property, leading to the introduction of Multi-scale Positional Encoding (Ms-PoE). This approach enhances the capability of LLMs to effectively capture information in the middle of the context without the need for additional fine-tuning. Comprehensive experiments conducted on Zero-SCROLLS benchmarks, multi-document question-answering tasks, and key-value retrieval tasks confirm the effectiveness of Ms-PoE. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: All our experiments are conducted using 1 × A6000 GPU.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The paper doesn't have potential positive societal impacts and negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: A.1 Position-Aware Attention Heads Figure 6 illustrates the attention patterns of "position-aware" heads. Each row represents the attention pattern of the same head. As the key document is positioned from the beginning to the end, the attention peak gradually shifts, indicating robust positional awareness. It's important to note that we randomly selected 9 attention heads with these "position-aware" properties, and these results were validated with different input samples and layers. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: NeurIPS Code of Ethics is followed.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no risk involved.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: All assets are in public domain.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 


Decision: This paper proposes a plug-and-play solution for the the "lost-in-the-middle" problem in large language models (LLMs) by introducing Multi-scale Positional Encoding (Ms-PoE). Ms-PoE uses position index rescaling and distinct scaling ratios for different attention heads to maintain essential knowledge. Experiments show Ms-PoE improves LLM accuracy on the several benchmarks.

Reviewers agree that this paper addresses an important problem and praise the simplicity and effectiveness of the proposed approach and the fact that it does not require any training. However they point out several weaknesses, such as the modest improvement in some tasks, the lack of analysis of performance in terms of context length, the lack of justification for the scaling ratio allocation heuristic (via an arithmetic sequence), and the process of tuning hyperparameters. Most of these issues have been convincingly addressed by the authors during the rebuttal, where they also added several other experiments (attention patterns before and after rescaling, new experiments in LongBench, and additional evaluations of the proposed method across varying input lengths). I am leaning towards acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the 'lost-in-the-middle' issue in large language models (LLMs) by introducing Multi-scale Positional Encoding (Ms-PoE). This approach enhances LLMs' ability to handle relevant information in the middle of the context without fine-tuning or added overhead. Ms-PoE uses position index rescaling and distinct scaling ratios for different attention heads to maintain essential knowledge. Experiments show Ms-PoE improves LLM accuracy, achieving up to a 3.8% gain on the Zero-SCROLLS benchmark.
strength_weakness: strength_weakness: 1．	The improvement is actually limited, despite the method’s efficiency and ease of use.

2．	The heuristic of the scaling ratio allocation process is a little arbitrary. There could be better solutions than just using an arithmetic sequence to achieve that. It is not strictly proved that a larger S_P means that the model should have a less rescaling ratio and vice versa. There could be possibilities that different heads could use the same scaling ratio or the other way around. 

3．	Moreover, there is no evidence that the rescaled position embedding could make the attention heads like the “Top in Figure 4” better at catching relevant information except for an empirical marginal task performance improvement and the comparison with other strategies in Section 4.3. An attention map of the attention heads after the rescaling could better strengthen this main claim of the paper. The authors could also partially show this by examining the S_P variation before and after the rescaling. 

4．	In Section 3.1, directly adjusting the hyperparameter and seeing the results on the test set is not very appropriate since it means the test set information is leaked. The authors should analyze the scaling ratio on the validation set and see if it is consistent with that in the test set. Moreover, the selection of the hypermeters R_min and R_max is also probably acquired by the test set performance, reflected by the ablation experiments in Table 4.
questions: questions: 1.	L212-215: “With a small scaling ratio LLMs tend to focus more on the most recent part of the context while with a large ratio, LLMs favor the beginning part” Are there any supporting results for this claim?

2.	Did you choose the hyperparameters of R_min and R_max directly from their performance on the test set?

3.	I’m happy to raise the scores if all the concerns are well addressed.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a plug-and-play method named Ms-PoE to mitigate the lost-in-middle challenge of LLMs. Specifically, Ms-PoE leverages multi-scale position embeddings to enhance information awareness in different parts of the context. Without fine-tuning the model,  Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs.
strength_weakness: strength_weakness: 1. It seems that the performance improvement on MDAQ and ZeroSCROLLS benchmark is small compared to the "Self-Extend" method, which can be seen as a special type of "Multi-Scale Positional Encoding", i.e., "Single-Scale Positional Encoding".

2. Although this paper suggests that Ms-PoE is suitable for "Long Contexts," I believe it is suitable for any context situation. What exactly defines a "Long Context"? Is it a context length above 12K? Additionally, I have not come across any analysis of the impact of context length and how Ms-PoE can help the model understand the middle of long contexts. It might be beneficial to conduct experiments to determine whether Ms-PoE can enhance the model's understanding of the intermediate content of the context. This could involve analyzing different context lengths, such as inserting a specific length of context and evaluating the model's predictions.
questions: questions: 1. What's the performance of MS-PoE(with LLama-2-7B-Chat as the backbone model) on LongBench-EN benchmark ?
2. Why chose 1.2 and 1.8 for R_{min} and R_{max} in Equation 2 ?
3. See Weakness 2
limitations: limitations: See Question and Weakness above.
paper_summary: paper_summary: Paper addresses the lost-in-the-middle effect, observed in the past for some LLMs, with a method called Multi-scale Positional Encoding (Ms-PoE) where position is encoded using different scales for each attention head. More precisely, for each head a re-scaling ratio is substituting the position m of a token with m/r (r can be 1.5 for instance) and different values of r are used for each attention head. This multi-scale view is evaluated experimentally and authors claim that this mitigates the ‘lost-in-the-middle’ effect.
strength_weakness: strength_weakness: -The "lost-in-the-middle" effect is often assumed to impact all LLMs. But  recent research indicates that this effect does not uniformly affect all LLMs, though the effect was observed on models such as Vicuna7B. 
	see for instance https://arxiv.org/abs/2403.20262  which shows that not all LLMs display the lost-in-the-middle effect on a long context benchmark

-For evaluation,  the ZeroSCROLLS benchmark was used but authors could have considered  incorporating other benchmarks designed for long context scenarios, such as "needle in the haystack" tasks. There are also many other relevant benchmarks available, including: NarrativeQA; LongEval; LongBench; LongBench-Chat; Loogle;  ∞Bench and Long Range Arena (LRA) among others
      => on that aspect paper could have been stronger experimentally if evaluating the approach on more long context benchmarks
questions: questions: -eq(2): how are Rmin & Rmax fixed ?

-fig 5: why only vicuna model evaluated, does Llama-2-7B-Chat also display a lost-in-the-middle effect ?

-tab.3 are the differences between begin/middle/end results really significant ? A statistical test (for instance one-tailed Welch’s t-test) would have been welcome here
limitations: limitations: No 'limitations' section was provided

Comments: title: We are keen to discuss further with you
Response to Reviewer LVQT
Thanks for the response
Thanks for the responses
Re-Respond to Authors
Respond to further questions
Respond to authors
comment: Dear Reviewer LVQT,

We sincerely appreciate your valuable time and constructive feedback. We have carefully addressed each of your concerns. As the discussion period deadline approaches, we would be grateful if you could inform us if there are any further questions. Thank you!

Best,

Authors
Dear Reviewer LVQT,

We sincerely thank you for your time and effort in reviewing our work. Your constructive and insightful feedback has been invaluable in helping us improve the quality of our manuscript. We have carefully addressed each of your comments and hope that we have successfully resolved all of your concerns. We are open to further discussion if you have any additional questions and look forward to your response.

Best regards,  

Authors
We sincerely appreciate all the constructive feedback and positive evaluations from reviewer xsrg. Thanks for your time and support!
Thanks for the responses. We are glad that our responses resolved your concerns and we will include additional results in the updated draft. Also, many thanks for raising the score.
Thanks for informing me about this point. I have raised the score from 5 to 6.
Thanks for the responses. They resolved most of my concerns and I've raised my scores accordingly. Hope you all the best!
Thanks for the good question. We conducted an ablation study to examine the impact of $R_{min}$ and $R_{max}$ on the final results. The findings are presented in Table 4 and discussed in Section 4.3 A2. Our study indicates that there is a sweet point for scaling ratios that enhances performance ($R_{min} = 1.2$, $R_{max} = 1.8$). However, when the scaling ratios are either too small (e.g., 0.5) or too large (e.g., greater than 2), performance tends to degrade. Additionally, such sweet point shows good generalization, as verified by downstream tasks such as key-value retrieval, Longbench, and Zeroscrolls.
Thanks for your response. 

Although Ms-PoE is an efficient and plug-and-play method, it may require some parameter-selection processes and the hyper-parameter may vary for different situations.

I want to understand the impact of R_min and R_max. Do these two values significantly affect the final results?