Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
Authors: Zhenyu Zhang
Runjin Chen
Shiwei Liu
Zhewei Yao
Olatunji Ruwase
Beidi Chen
Xiaoxia Wu
Zhangyang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Positional encoding; Large language model; long context reasoning
Primary area: natural_language_processing

Abstract: This paper aims to overcome the ``lost-in-the-middle'' challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code will be made public upon acceptence.

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
Zhenyu Zhang'", Runjin Chen', Shiwei Liu", Zhewei Yao", Olatunji Ruwase",
Beidi Chen', Xiaoxia Wu Zhangyang Wang" University of Texas at Austin, University of Oxford, " "Microsoft, *Camegic Mellon University Work done during internship Microsoft, Equal advising
Abstract
This paper aims overcome the "fost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is simple yet effective salu-andd-layy approach to enhance the capacity LLMs to handle the relevant information located in the middle of the context, without fine-tuning introducing any additional overhead. Ms-PoE leverages the position indice rescal- ing to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming multi-scale context fu- sion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of approach. Notably, MssPoE achieves average accuracy gain of up to 3.8 on the ZcRo-SCROLLS benchmark over the original LLMs. Code available at httpt:::///. //tith///ttttttt/ /VIT/ www.////AAG////////OOC PoE
Introduction
Effective long-sequence reasoning in large language models (LLMs) is crucial for a wide range of
applications (122) from understanding extensive texts 3 and managing day-long conversations [5]
6 to code generation 7 8 and science discoveries 2 10 Recent system support advancements [I]
12 have enabled training transformers for any L sequence length even with (LL)) computational
complexity. This is exemplified by models such MPT [3] and Mistral 14] pre-trained with
sequence lengths 16k and 32k respectively.
Nevertheless, emerging research reveals the constrained efficacy of LLMs in managing tasks requiring long contextual understanding. Particularly, [15] demonstrated substantial degradation LLMs' performance when crucial information is positioned amidst lengthy context, phenomenon they refer to as seess-i-tthe-mideee
One explanation is about the use of rotary positional embedding (RoPE) [66, a prevalent positional encoding technique used in open-source LLMs. As a relative position embedding, RoPE incorporates a long-term decay property, predisposing the model to prioritize current/nearby tokens while paying less attention to further ones. [17) identified surprising trend attributed to the Softmax operation where attention scores are disproportionately allocated into initial tokens, irrespective of their relevance to the language modeling task. Despite the presence of considerable redundancy in long-context inputs [18), crucial information may be located across different positions. The inclination of LLMs to overlook the middle section presents challenge for their applications, particularly in the context of long-context reasoning. Several approaches successfully extend pre-trained LLMs with context up extreme token length, either through sparse
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

selection of crucial tokens during generation 17 Nevertheless, these approaches primarily aim to extend the context length of LLMs and, consequently, fall short in addressing the most-in-the-middtee problem when applied out-of-the-bon
or by modifying positional encoding 20 21
Efforts have been made to enhance LLMS'
capacity to capture vital information lo- cated within the middle of the context. These include extra memory bank [22], re- ordering the input context based on rele- vance [23 244,, enhancing the information searching and reflection ability via atten- tion strengthening tasks [25] 26]. splitting the input into short segments and applying short-text models [27] For example, [23 empirically discovered that LLMS tend to emphasize more on the current window while still paying more attention to the rel- evant text than distracting content. They subsequently introduced "attention sorting" where the main idea is iteratively sorting documents based on their attention scores, such that critical information will likely be placed at the end, to fit the position-biased nature of RoPE. (24) conducted parallel runs of LLMs with different RoPE angles, thereby mitigating the risk of overlooking crucial information through a weighted sum of the outputs. These approaches usually require additional memory or multiple inference runs, which can be expensive for LLMs.
Vicrm--ll MIVA
Viceneiill Koy-lalze Hetrvvl
Baclit
Bexdiar
Ours
Duts
Docoreent Dodox
Kry fnces
Figure 1: tial information within the prompt, ranging from start to end. The green bar serves as a standard baseline, illustrating the "lost-in-the-middlee phenomenon. We introduce our method, Multi-scale Position Encoding (Ms (Ms-PoE), which requires neither additional fine-tuning nor increased memory usage. Instead, it involves a sim- ple remapping of the position embedding depicted in Figure [] which enables the important information in the middle to be detected effectively (brown bars). For more details, see Section 4.2 and Figure 5
The x-axis illustrates the placement of essen-
In this paper, we aim to address the "lost-in-the the-hhe-meet tee-tthe-miiiee middle" problem by reintroducing the concept of multi-scale features from computer vision into the context of Transformer- baadsfeeerrbbeeddd LLMs. Multi- scale features, well-ostablished in Inception-style models [28, 29 30,, utilize parallel employment of kemells with different sizes to fuse multi-scale information, spanning short to long distances. Introducing multi-scale operations into LLMs intuitively can help compensate for crucial information located in the middle, which might be easily overlooked by full attention operation. Unlike modifying the attention module to form multi-scale attention, we choose to re-scale the indices of positional encoding. This decision is grounded not only in its effectiveness in easily adjusting the scale of the context window by simply changing the position indices 20 but also in the potential of down-scaling the position indices to relieve the long-term decay property introduced by RoPE. However, this approach was initially introduced to extend context windows, and its performance regarding the "lost- thi-diddle" problem remains uncertain for several reasons: (i) Indice re-scaling forces position embeddings of original context window to reside in a narrower region, leading to performance degradation in the original context window as shown in (0). (ii) Uniformly applying the same scaling ratio throughout the entire model might be sub-optimal to preserve essential knowledge learned during pre- -raininnn (ii) Fine-tuning necessary for the original approach, albeit minimal.
The impact without fine-tuning remains unknown.
To this end, we systematically visit the position indices scaling regarding the "Yost-in-themmidle problem and counter-intuitively discover that it is possible to slightly mitigate the "lost-in-the-middle issue we carefully choose the scaling ratio to be around 1.5-2. Additionally, we observe that different attention heads exhibit varying sensitivity to the position shift of the relevant document. Some attention heads are "position-aware" consistently capturing relevant information even with position shifts, while others may occasionally capture position changes, and some heads are completely insensitive to position changes. This highlights the need to treat attention heads separately when re- re-scaling position indices.
Contribution. Inspired by the above observations, we introduce Multi-scale Positional Encoding (Ms-PoE), a simple yet effective plug-and-play approach that can enhance the long-context reasoning capability of pre-trained LLMs without requiring fine-tuning or introducing any additional overhead. Ms-PoE meticulously assigns distinct scaling ratios to different attention heads, with the scaling factor monotonically increasing from "position-aws awarre" heads to "position-unawee heads. This enables
2

---

us to improve context ability by re-scaling position indices to shorter values while preserving essential knowledge acquired during the pre-training phase. The efficacy Ms-PoE is substantiated through extensive experiments. By simply re-scaling the indices of positional encoding, Ms-PoR consistently enhances the performance of various LLMSs including llama-2 [31], StableBeluga [2]
and Vicuna 3
on the ZeroSCROLLS (4), achieving notable accuracy gain of up to 3.8.
2 Background and Related Works
In this section, we provide concise overview of the background knowledge and recent literature about the generative inference process of Large Language Models (LLMs), their abilities for long-context reasoning, and details of positional encoding.
2.1 Generative Inference of LLMs
The generative inference process in LLMs can be categorized into two distinct phases: @ Prefilling Stage: In this initial phase. LLMs receive an input sequence containing detailed instructions that define a specific generation goal. Throughout this stage, intermediate Key and Value embeddings are generated at each layer and stored in memory, commonly referred to as the KV cache. @ Decoding Stage: This phase involves retrieving embeddings from the KV cache to generate new tokens. The decoding process is inherently iterative, where each newly generated token serves as input for the subsequent token generation. In real-world LLM deployment, the cumulative length of input sequences and the subsequently generated text can reach several thousand or even millions of tokens, presenting significant challenges for the LLMs' long-context reasoning capability.
2.2 Long Context Reasoning
Two challenges for LLMs in handling long-com long-context reasoning tasks. One is to extend the context window to process sentences that exceed the pre-trained window length. Another the "lost-in-the window" issue where LLMs likely overlook the information located in the middle of the sentences.
The reason for the former challenge is that open-source LLMs usually pre-trained with fixed sequence lengths, such as 4096 for Llama-2 31. When the sequence length surpasses the predefined context length used in pre-training. LLMs often suffer from performance collapses and thus generate incoherent or fragmented text. Recent efforts to address this issue can be broadly categorized into two streams. Recently, several works have been proposed address this issue, which can be broadly categorized into two streams. The first one explores from the expansion of positional encoding, with notable contributions including PI 220 CLEX 335 YaRN [36,, Belf-Extend 21. On the other hand, some works modify the attention mechanism, such as StreamingLLLM [17, LM-Inifinite [19], H2O [8] TOVA (37, Zebra (38, and Activation Beacon [39. These approaches have successfully expanded the contextual window with minimal or no additional training overhead.
Despite the extended context window, LLMs still face significant challenge in long-context inference due to the uneven utilization of lengthy inputs. [15] conducted a pivotal investigation, revealing that LLMs tend to overlook the middle portion of the input. This bias compromises the practical application of LLMs. as critical information may be located in the middle part of the input, leading to unreliable outputs. To tackle this issue. [23] introduced 'attention sorting' to reorder inputs, placing critical information at the end. However, this method's 's reliance on potentially biased attention scores identify crucial content may compromise its reliability, and the prerequisite knowledge of document count in inputs may affect its effectiveness. [24] utilize Attention Buckets, an ensemble approach that combines multiple forward processes with positional modifications. However, this technique necessitates a considerably higher computational cost. Other general approaches for enhancing long-context reasoning include prompt compression 40, retrieval augmentation (26), and inference refinement by constructing memory trees [4]] while these approaches typically necessitate extra LLMs' assistance or bring extra computational cost.
2.3 Positional Encoding
For effective processing of long contexts, LLMs necessitate the explicit encoding of positional information. Common techniques include absolute positional embedding and relative positional encoding. Absolute positional embedding integrates word embeddings with an additional positional
3

---

vector based on the token's absolute position, which can be either fixed [42] or learnable 4 44 45
contrast, relative positional encoding. increasingly popular contemporary LLMs. encodes the relative distances between tokens instead of their absolute positions. Notable among these are Rotary Position Emhedding (RoPE) [16] that widely implemented in models like Llama 31 Falcon 48
Mistral 49 and ALiBi [50] which used in MPT 13
RoPE.
The primary goal of RoPE [6 is encode positional information such that the inner product of the query and key embeddings inherently contains the relative position information:
m) n) = g 9m, k,. m n)
Here, f is the positional encoding function applied to the query and key embeddings at positions m and respectively. To satisfy this condition, the function is defined as a valued complex function, as follows:
f(x, xeim@
[[x1 + ixz))e (03 + it de
... (x1-1 + ixi)e m
In this equation, 7 represents the dimension of the embeddings, 0,. = 10000 2k/1 and is the imaginary unit. For calculating the attention score, RoPE considers the real part of the product, specifically Re(f(gm: m)" f(,,, n)). This approach allows RoPE to effectively integrate relative positional information into the attention mechanism transformer models.
Methodology
In this section, present the details of our Multi-Scale Positional Encoding (Ms-PoE) approach. Section demonntratsss that the context utilization LLMs can be directly enhanced by re-scaling the positional information without incurring extra training costs. Then, Section 3.2 analyzes the properties various attention heads in LLMs. Section 3.3 outlines the detailed pipeline Ms-PoE.
3.1 Positional Re-scaling Improves Context Utilization
Current LLMs tend to neglect information located in the middle of the context, de- spite its potential relevance. This "lost in middle" phenomenon likely arises from two contributing factors: (i) Casual Atten- tion, where preceding tokens undergo a higher number of attention processes, lead- ing LLMs to disproportionately favor ini- tial tokens. This phenomenon has been demonstrated in recent research which highlights the pivotal role of the initial to- kens in model generation 19 17], with these starting tokens consistently accumu- lating higher attention scores [8]. (ii) The utilization of RoPE [16] introduces a long- term decay effect, diminishing the attention score of distantly positioned yet semanti- cally meaningful tokens. The combination LLMs ne-
Attention
Attention
Head
Head
Machil
Oenr
Km
Vakeo
Quar
Key
Vilk
Basy
Ker
(Mr.PoF) Multi-Scale Povitional Enonding
Figure 2: Illustration of our Multi-scale Positional En- coding (Ms-PoE) framework. The top figure demon- strates the implementation of Ms-PoE with various scal- ing ratios in different attention heads, marked with dif- ferent colors. The bottom figure shows the position details of each head, in which the first matrix (r; = 1) represents the original RoPE.
of these factors contributes glecting the context in the middle part. To tackle this issue and improve the context utilization of LLMs, seemingly unreason- able yet remarkably effective strategy to down-scale positional information (38]. Formally, RoPE encodes the position as fxx, m) im8 By substituting the position m with + can force the long-distance tokens to

---

reside in the shorted range, which can potentially alleviate the long-tern term decay effects by a factor r. In the following sections, we conduct experiments to evaluate how LLMs' context utilization responds to varying re-scaling ratios r.
the Multi- Document Question Answering (MDQA) task [1]. Each question includes ten documents, with only one relevant the question. By varying the position of the relevant document, we can evaluate LLMs' context utilizaaion properties. For each position of the key document, we calculate the accuracy over 500 samples. And results show in F include both the Average accuracy over the 10 documents as well as Gap accuracy, i.e., the difference between the best and worst accuracy when varying the positions of the relevant document.
Details. Experiments are conducted using Llama-2-7B--Che [3 and Vicuna-7B 33
Results. Figure 3 demon- strates that the gap accuracy can be alleviated via appro- priate positional re-scaling. Particularly, we see that the
Llaata2-7B-Chhtt
Vicuns-78
60-
60-
Accuracy P%)
50-
Avorge
Avrrgi
Gap between the best and the
40
40
worst accuracy is greatly re- duced when increasing the re- scaling ratio. An enhanced average accuracy can be ob- served with a scaling ratio equals near 1.5. Additionally, changing the scaling ratio also affects the favored zone of LLMs. With a small scaling ratio (e.g.. 0.5), LLMs tend to focus more on the most recent part of the context, while with large ratio (e.g., 2.5), LLMs favour the beginning part.
20-
Gap
20-
Gap
10-
10-
0.5
1.5
2.5
0.5
1.5
2.5
Scaling Ratio
Scaling Ratio
Figure 3: Results of the relationship between positional re-scaling and context utilization. The upper curve illustrates the average ac- curacy when placing the key document in various positions. The bottom curve indicates the gap between the best and worst accuracy.
Improving context reasoning via positional re-scaling, Building upon this, we introduce plug- and-play treatment for RoPE by re-scaling the position of each token. This approach seamlessly enhances the context utilization of LLMs without requiring additional training or inference overhead. However, there is trade-off in terms of LLMs favoring certain context regions. For instance, when r= 0.5, LLMs achieve peak accuracy when the relevant document is located at the end of the input, while at the beginning for r = 1.5. remains challenging to decide which re-scaling ratio to use, given that we lack prior knowledge of the location of relevant information in real-world applications. Moreover, as the re-scaling ratio increases, LLMS may face the positional out-of-distribution (O.0.D) issue [21 20), where many position values do not directly exist during pretraining (e.g.. using 0.1, 0.2, ..., 0.9 for position when LLMs only recognize J..,... 9 during pretraining), potentially reducing their average reasoning ability. To tackle these challenges, investigate the head-wise properties of LLMs and propose a multi-scale positional encoding approach.
3.2 Position-Aware Head-Wise Re-scaling Ratio
Inspired by recent works that leverage attention patterns to identify most crucial tokens and optimize inference efficiency 37 18, 51). we carry out a preliminary study to investigate the interaction between attention patterns and token positions.
Details. We visualize the attention patterns of the most recent query with results collected from Vicuna-7B on the MDQA task, following (37]. In the same input sample, we manually switch the position of the relevant document from the beginning to the end and illustrate the attention scores across different positions.
Observation. We observe the presence of "position-aware" attention heads capable of capturing relevant information even when its position is shifted. As an example, we select the eighth attention head in the fifteenth layer, depicted in the bottom of Figure while consistent observations can be drawn across different layers and input samples. Firstly, most attention scores are near zero and can be ignored, consistent with other studies highlighting high sparsity in attention blocks [8 52 53 For the remaining positions, these "position-aware" attention heads can capture important information across positions, with attention patterns shifting the position of relevant tokens changes. However,
5

---

Dicunnrt Iolo::
Decunori Iolou:
folox:
1290
1991
1890
1590
Deeureett
Insko.
Docament
0.10
<60"
1469
1960
1000
1260
Poattin Irdes
Prsticn Indox
Postiin 2ndes
Faxtica falox
Figure 4: Visualization of attention pattern of the most recent query within two different attention heads. Top: Results of the 12th attention head in the 15th layer. Bottom: Results of the 8th attention head in the 15th layer. The most recent query remains unchanged while varying the position of the crucial document, More examples are reported in Figure in the appendix.
for other attention heads (upper subfigure in Figure F. they fail to capture relevant tokens and only attend to the beginning and end words, contributing to the
issue.
Based on this observation, we devise a position-awart strategy to adaptively determine the re-scaling ratio via the inherent properties of different attention heads. For the "position-aware" attention heads, we assign a re-scaling ratio close to one to avoid changing their functionality significantly, as altering them too much could degrade performance due to the positional O.O.D issue. On the other heads, we condense their position indices to a higher degree, providing more opportunity to alleviate the persistent bias toward the beginning and recent tokens. To identify the properties of ny attention heads, we introduce a Position-Awann Awareness Score Sp R formulated as:
1 Sp
=p=7
(A. M =11
Σ AA) 1=1
In Equation| 1 A represents the attention score vector of the most recent query, and a is a hyper- parameter determining the threshold of effective attention scores. all experiments, we default to using 3, and the corresponding important tokens highlighted in Figure 4 which are shown in red. In the spirit of numerous studies that investigate the outlier properties in LLMs [17 54 (55,, we utilize Sp evaluate the ratio of effective attention tokens, where a larger Sp value implies better positional awareness.
3.3 Inference with Multi-Scale Positional Encoding
The pipeline for utilizing Multi-Scale Positional Encoding (Ms-PoE) in LLM inference is: Given a pre-trained LLM, we initially replace the original rotary positional encoding with Ms-PoE. As illustrated in Figure 2 Ms-PoE condenses the positional indices of RoPE and employs different re-scaling ratios for each attention head. The re-scaling ratios are assigned during the prefilling stage, where we first calculate the distribution of attention scores for the most recent query and obtain the corresponding position- position-awareness awareness score for each attention head. Larger re-scaling ratios are subsequently allocated to attention heads exhibiting smaller position-awareness scores. And the set of re-scaling ratios defaults to a linear range from Rmin to Rmaz- For example, the ith corted-head would be using re-scaling ratio Rmin (i Rm Rmmin/((r)
(2)
Once the re-scaling ratios assigned, they remain fixed in the subsequent decoding stage. We consistently using Rmin = 1.2 and Rmaz 1.8 our experiments.
Experiments
The goal of this section is to demonstrate Ms-PoE, a plug-and-play positional encoding capable of enhancing the context utilization of L.LMs, and consequently improving the quality of generation across diverse models and downstream reasoning tasks. Our main results can be summarized below.

---

Table 1:
Comparsion results on ZeroSCROLLS 33 benchmarks. The evaluation metrics for various tasks are tailored as follows: GovReport, SummScreenFD, QMSum, and SQuALITY utilize the geometric mean of Rouge-1/22 1/2/L scores. Qasper and NarrativeQA are assessed through the Fl score, while BuooSt employs the concordance index.
Models
Methods
GovRoport
SummScrrenDD
QMSum
SQUALITY
Qusper
NarativoQA
BoooSt
Avergee
LJoma-2-770 2-78h-hh
Boseline
168
14.1
15.2
19.5
21,9
14.4
3.1
15.0
Llams: 2 CBa
Ours
17.7 (+0.9)
14.2 14.2 (+0.1)
15.8 (+0.6)
(+0.4)
25.1 (+3.2)
17.7 (+3.3)
5.8 (+2.)
16.6 (+1.6)
Llama: 2-13 Chat
Baseline
15.4
12.3
15.1
18.9
19.0
15.0
5,7
14,5
Llama: 2- 2-13B Chai
Ours
16.5 (+LI)
13.1 (+0.5)
15.5 (+0.4)
(-0.3)
20.8 (+1.8)
17.0 (+2.0)
5,9 (+0.2)
1544 (+0.9)
BlableBeluga StableBeluga-7B
14.9
1688
16.5
Owss
(+1.7
(+0.4)
15.2 (40.5)
18.7 (+0.8)
(+8.8)
(+1.2)
(45.0)
19.1 (+2.6)
StableBelugo-13B
Baseline
5.7
7.1
12.9
13.3
19.2
13.4
4,8
10.9
StableBelugo- 13B
Ours
7.4 (+1.7)
7.4 (+0.3)
12.8 (.0.1)
(-0.1)
20.8 (+1.6)
(3.4
5.6 (+0.8)
11.5 (+0.6)
Vicuna 7B Vicumn-88
Baseline
16.2
13.7
15.1
18.9
24.3
13.7
3,3
15.0
Ours
20.2
(+1.8)
15.4
19.8 (+0.9)
34.7 (+13.4)
16.2 (+2.5)
188
Vicuns-88: 7B-0--78-666K
Reseline
20.2
13,9
16.2
20.1
18.8
29.9
21.6
Vicums-78-166
Ours
21.4 (+1.2)
14.3 (+0.4)
16.2 (40)
20.2 (+0.1)
37.8 (45.5)
21.0 (+2.3)
(413.4)
24.9 (43.3)
In Section 4.1
we demonstrate that Ms-PoR consistently enhances reasoning over long contexts for a
range of tasks in the ZeroSCROLLS benchmarks [34], all without the need for additional training. Additionally, Ms-PoE exhibits superior performance when compared to other methods in the field,
including PI [20] and Self-Extend (21). Detailed comparison results are shown in Tables [ 2
In section we highlight that Ms-PoE improves the context utilization and achieves consistent improvement when varying the position of critical information, as shown in Figure [ &
In Section 4.3 we conduct multiple ablation studies assess the effectiveness of Ms-PoE under different scaling ratios and selection strategies. Results are reported in Table 3 &
4.1 Enhanced Generation Quality
We empirically validate the ability of Ms-PoE to enhance long-context reasoning with noteworthy improvement up to 13.4 without additional training overhead. Notably, our approach surpasses other competitive baselines, demonstrating improvements from 2.64 to 43.72.
Experimental Setup. In our experiments, we select seven representative LLMs, including Llama-2- chat-7B and 13B B1, StableBeluga-7B and 13B (32, and Vicuna-7B B3 along with its longer- context version (Vicuna- 7B-KK). To comprehensively evaluate the long- reasoning abilities of LLMs, we choose seven tasks from ZeroSCROLLS [34), spanning all four task categories: ①D Doc- ument Summarization (Government and SummScreenFD), 2 Query-Based Summarization (QMSum and SQuALITY), @ Question Answering (Qasper and NarrativeQA). and @ Information Aggregation (BookSumSort). We also compare Ms-PoE with other competitive methods on additional generation tasks, including Multi-document Question Answering (MDQA) and Key-Valur Retrieval (15).
Main Results. Table [ summarizes the main results, yielding several key observations: (i) By simply substituting the original positional encoding module with our Ms-PoE, the performance of LLMs consistently improves across all tasks without additional training, resulting in an average performance enhancement ranging from 0.6 to 3.8; (ii) These improvements hold consistently across different model sizes of billion and 13 billion parameters; (iii) The efficacy extends to LLMs with varying sequence lengths, such as Vicuna-7B and its extended version, Vicuna-7B-16K, both showing improvements from 3.3 to 3.8.
Outperform other competitive methods. We conduct a thorough comparison between Ms-PoF and other competitive methods, including Positional Interpolation (PI) 20 and Self-Extend (2I). both of which modify position indices without utilizing head-wise properties. For PI, we employ the scaling ratio as the average value of our method while for Self-Extend, we set the group size as 2 with the local window size as 1024. The results presented in Table 2 consistently showcase the superiority of our approach over other baselines, demonstrating improvements of up to 3.92 and 43.72 for MDQA and Key-Value Value Retrieval, respectively. Such improvements might come from two primary factors. Firstly, the incorporation of head-wist properties offers a more adaptive strategy for positional modification. Secondly, our approach enhances the general context utilization ability. Notably, our approach demonstrates superiority even when the core document or key is positioned at the end of the input, surpassing other baselines with improvements ranging from 2.4 to 27.8.
7

---

This performance surpasses the recent work [23], which addresses the "lost-in flost-n-the--mee tee-tthe-mie theemiddle" effect by reordering key documents and placing them at the end of the input. When the identified core document is already located at the recent area, such method can not gain further improvements, while our approach offers a fine-graieed strategy to improve context utilizationn.
Table 2:
Comparsion results with other competitive methods on MDQA and Key-Value Retrival.
Results are reported in accuracy.
MDQA
Models
Methods
1
3
5
7
10
Average
Baseline
64.0)
61.0
57.4
58.4
64.8
61.12
Pl
65.2
62.4
60.0
60.4
64.0
62,40
Vicuna-7B
Self-Extend
64.7
63.7
61.4
59.8
62.0
62.32
Ms-PoE
65.6
64.2
63.0
65.2
67.2
65.04
Key-Vane Value Retrival
Models
Methods
1
15
40
50
Average
Baseline
92.0
25.8
8.0
25.4
30.0
36.24
Pl
96.4
76.4
61.4
64.6
57.8
67.60
Vicuna-7B
Self-Extend
88.6
63.8
76.2
59.4
42.0
66.00
Ms-PoE
97.0
83.4
75.0
86.6
57.8
79.96
4.2 Superior Context Utilization
We assess the context utilization ability of our approaches on two tasks, includ- ing multi-document question answering (MDQA) and key-value retrieval (KV retrieval) tasks from [13]. Such tasks provide a good input structure and of- fers the flexibility to switch the position of crucial information, thus evaluate the context utilization ability of LLMs.
MDQA
Vioua-138 Key-Whie Whe Retrival
Bascine
Bauciiee
Chrrs
50
65-
A00m
Experimental Setup. In the MDQA task, each input sample comprises ten documents and one question, with only document being relevant to the ques- tion. For the KV retrieval tasks, there 50 vayuv pairs with question querying the value of the chosen key. In both tasks, systematically switch the important document or key-value pair from the beginning the end and report the accuracy of the generated context. All results are averaged across 500 samples. The Gap accuracy metric is employed to assess the context utilization ability of LLMs, defined the gap between the best and worst accuracy when varying the position of important information.
Dovumeet
Kry
Figure Comparison results for the multi-documen ques- tion answering (MDQA) and key-value retrieval (KV re- trieval) tasks. Each subfigure depicts the comparison when varying the position of critical information from the begin- ning to the end. For Vicuna-77 -B,, please refer to Figure
Main Results. As depicted in Figure and Ms-PoE demonstrates consistent improvement across different models, tasks and critical positions. Even when the important information exists in the sweet region (beginning and end) of the input, Ms-PoE achieves significant performance improvements ranging from 3% to 6%, highlighting its efficacy in enhancing generation quality. Moreover, the toas-in-the-middle issue is notably alleviated, with Ms-PoE quantitatively reducing the gap accuracy by approximately 2% to 4%, showcasing improved context utilization.
4.3 Ablation Study and More Investigation
This section conducts further evaluation of the effectiveness of Ms-PoE by addressing the following questions: Q1: How does the effectiveness Ms-PoE relate to the head-wise selection strategy of the scaling ratio? Q2: How does the model perform with different scaling ratios?

---

AI: Positional awareness met- ries achieve superior perfor- mance compared to other strategies. For set of scaling ratios E h where n, is the number of attention heads, and using scaling ratios linearly rang- ing from 1.2 to 1.8, we evalu- ate various strategies for assign- ing these ratios to different at- tention heads. These strategies include: ① Random, which ran- domly assigns the scaling ratios to each head within each layer; @ Sequential, performing the assignment based the original head order; @ Entropy, where we follow metrics measuring the sparsity level of attention scores [56). Larger entropy implies less sparse attention scores, indicating the model attends to more tokens rather than just the beginning and end words, so we assign a scaling ratio near to I, and vice versa for larger ratios. Results in Table 3 demonstrate that the proposed position- position-awareness awareness effectively captures the head wise properties of LLMs, enhancing performance when critical information is located at various positions beginning, middle, or end. This leads to an average accuracy gain of 3.2 (65.3 v.s. 62.1).
Table Ablation results of different ordering metrics. Experi- ments are conducted on Multi-Documents Question Answering lask with the Vicuna-7B model.
Methods
Begin
Middle
End
Average
Baseline
64.0
57.4
64.8
62.1
Random
64.5
55.0
65.5
61.7
Sequential
60.5
54.5
58.5
57.8
Entropy
63.5
59.5
64.0
62.3
Position-Awareness
65.6
63.0
67.2
65.3
A2: Ablation study of the scaling ratios. We first examined the effect of uniform seal- ing ratios across all heads on model perfor- mance. Our findings, outlined in Table 4 in- dicate that adjusting the scaling ratio between 0.5 and 2.5 can significantly enhance genera- tive performance and mitigate the "lost-in-the- middle" effect by 1.0% (63. (63.1% v.s. 62.1%).
Table 4: Ablation results of the condensing ratios. Experiments are conducted on Multi-Documents
Question Answering task with the Vicuna-7B model.
Scaling Ratio
Begin
Middle
End
Average
64.0
57.4
64.8
62.1
0.5
56.0
51.0
68.0
58.3
particularly with a ratio of 1.5. Further test- ing with an average ratio of 1.5 across all heads revealed that an optimal range exists between 1.2 and 1.8, leading to an additional 2.2% (65.3%
1.5
65.2
60.0
64.0
63.1
2
61.5
59.0
62.5
61.0
2.5
59.5
57.5
57.0
58.0
0.8 2.2
53.5
59.5
67.5
60.2
63.1%) accuracy improve- ment with our approach, Ms-PoE. Based on these results, we established these ratios our experimental standard.
1 1
61.0
57.0
63.0
60.3
1.2+
65.6
63.0
67.2
65.3
1.4 -
65.5
59.0
63.0
62.5
Conclusion
In this paper, we present a plug-and-play strategy designed to address the thottin-theemmiddee" challenge observed in LLMs. This challenge stems from the persistent bias exhibited by LLMs towards the beginning and local content within the input, leading to the neglect of crucial information in the middle. Our investigation reveals the effects of position indice rescaling and the head-wise rnsition-awarenee property, leading to the introduction of Multi-scale Positional Encoding (Ms-PoE). This approach enhances the capability of LLMs effectively capture information in the middle of the context without the need for additional fine-tuning. Comprehensive experiments conducted on SCROLSS benchmarks, multi-document question-answwring tasks, and key-value retrieval tasks confirm the effectiveness of Ms-PoE.
Acknowledgements
We thank Dr. Yuandong Tian for interesting discussions this work. Z. Zhang and Z. Wang were in part supported by an Intel Gift Funding.

---

References
[1] Chris Ré, Tri Dao, Dan Fu, and Karan Goel. Can longer sequences help take the next leap in ai?, June 2022. Accessed: 2024-01-29.
[2] Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311 1ete2331114499,, 2023.
[3] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In International Conference on Learning Representations, 2020.
[4] Wojeiech Kryšciñíski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08200 08200, 2021.
[5] Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed H Awadallah, Dragomir Radev, and Rui Zhang. Summ n: A multi-stage summarization framework for long input dialogues and documents. arXiv preprint arXiv:2110.10150 2021.
[6] Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Dialoglm: Pre- trained model for long dialogue understanding and summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11765-11773, 2022.
[7] Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually-crafted benchmark for evaluating Iims on class-level code generation. arXiv preprint arXiv:2308. 01861, 2023.
[8] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang. and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. In KDD, 2023.
[9] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D43- D444, 2022.
[10] Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Cheng- ming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, al. Deep- speed4science initiative: Enabling large-scale scientific discovery through sophisticated ai system technologies. arXiv preprint arXiv:2310.046/ 2023.
[11] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023.
[12] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309.14509 2023.
[13] MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms, 2023. Accessed: 2023-05-05.
[14] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024.
[15] Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2337.03372..22 2023.
[16] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomputing, 568:127063, 2024.
[17] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1743 2023.
10

---

[18] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H _ o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306. 14048, 2023.
[19] Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint xrivv2330..61377 2023.
[20] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.1559) 2023.
[21] Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Chang, Huiyuan Chen, and Xia Hu, Llm maybe longlm: Self-extend Ilm context window without tuning. arXiv preprint arXiv:2401.013: 013.0,,,, 2024.
[22] Weizhi Wang, Li Dong. Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory, arXiv preprint arXiv:2306.0711 2023,
[23] Alexander Peysakhovich and Adam Lerer. Attention sorting combats recency bias in long context language models. arXiv preprint arXi::231000142,2000 2023.
[24] Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455, 2023.
[25] He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qianguo, Zhang Songxin, Xie Zejian, al. Never lost in the middle: Improving large lan- guage models via attention strengthening question answering. arXiv preprint arXiv:2311.09) 2023.
[26] Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subrama- nian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.000 03025, 2023.
[27] Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11:284-299, 2023.
[28] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1-9, 2015.
[29] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 28 18-666, 2016.
[30] Meng-Hao Guo, Cheng Ze Lu, Qibin Hou, Zhengming Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 15:1140-17 1156, 2022.
[31] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0921 2023.
[32] Dakota Mahan, Ryan Carlow, Louis Castricato, Nathan Cooper, and Christian Laforte. Stable beluga models.
[33] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[34] Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305. 14190....11...... 2023.
11

---

[35] Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450, 2023.
[36] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023.
[37] Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. Transformers multi-state mas. arXiv preprint arXiv:2401.061000 2024.
[38] Kaiqiang Song, Xiaoyang Wang. Sangwoo Cho, Xiaoman Pan, and Dong Yu. Zebra: Extending context window with layerwise grouped local-giobal attention. arXiv preprint 2tt.:::312.006.... 2023.
[39] Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soar- ing from 4k to 400k: Extending Ilm's context with activation beacon. arXiv preprint arXiv:2401.03462. 2024.
[40] Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839 2023.
[41] Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading, arXiv preprint arXiv:2310.05029, 2023.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[43] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:1810.04805. 2018.
[44] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint 1909v 11942, 2019.
[45] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003. 10555,, 2020.
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[47] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[48] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Reb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306. 01 116, 2023.
[49] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arX.v:2310.06825, 2023.
[50] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
[51] Suyu Ge. Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive ky cache compression for 1ms.. arXiv preprint arXiv:2310.01801, 2023.
[52] Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. On the expressive power of self-attention matrices. arXiv preprint arXiv:2106.03377 2021.
12

---

[53] Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and vari- able creation in self-attention mechanisms. In International Conference on Machine Learning. pages 5793-5831. PMLR, 2022.
[54] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-awary weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0097 2023.
[55] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pech- enizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms high sparsity, 2023.
[56] Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du Joma: De- mystifying multilayer transformers via joint dynamics of mlp and attention. arXiv preprint arXiv:2310.00535. 2023.
[57] Yushi Bai, Xin Lv, Jiajie Zhang, Hongehang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023.
13

---

A More Experiment Results
A.1 Position-Awars Attention Heads
Dacased
Thu neert
Ducaser
Desccua
50x
falos
Potle
Pudion
Dacemene Iodie:
Decirunt brior
Decence Tokks
Docamurt balec
Decemeet Indere
180
190
280
180
190
100
Docament Todin:
Docarves
Dicanen Pades
Docanner baloc
Dicamee Iadac:
1008
1009
1300
1900
158
V-a
109
900
180
1800
110
1008
02
Peetem kide
Nea
Docaueel lades!
Indes
Docazer Inkes
lades?
Feal
Dacemet Inden:
Docree Indes
Decenee Inden
Diceren Indent?
Dacemee Indos
Dooioea Indon
Decrnve booo)
Devnuet Indon
Jouanee bakos
Devuxne Inders
219
180
140
200
180
1900
100
12
Tacarrst Tablec
Dicauen Todes
Decenet Lode
Dacared frllee?
Docaseet ladic
20
100
20
500
Vide
25
bas
02
C18
Figure 6: Visualization of "position-aware" attention heads. Each row contains the attention pattern for the same heads when varying the key documents within the inputs.
Figure Coaure/lilian iillustrates the attention patterns of "position-aware" heads. Each row represents the attention pattern of the same head. the key document is positioned from the beginning the end, the attention peak gradually shifts, indicating robust positional awareness. It's important to note that we randomly selected 9 attention heads with these "position-aware" properties, and these results were validated with different input samples and layers.
14

---

Table 5: Comparison results of Ms-PoE on LongBench-EN benchmark with Llama-2-7BB BB-Cht.
MegpagA
Paoage
PaogpBereealve
Repelieads
Awrage
Basciiiee
31.11
$6.77
27.0
30.00
3.58
38.7
36.35
38.45
01.81
10.32
$1.33
24.14
2012
A.2 Results on LongBench-EN Benchmark
We further evaluate Ms-PoE the LongBench-EN benchmark [57] that contains 13 tasks aims for long context understanding. Results are reported Table We can observe that Ms-PoE achieves consistent performance improvement without any finetuning.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
We describe sufficient details about the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: The primary limitation of our work remains limited exploration for only RoPE based models.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the necessary ratios for MS-PoE settings and made the relevant code publicly available in GitHub repository.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: All the models and datasets used in this paper are openly accessible on Huggingface, We made the relevant code publicly available GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips. //nips.cc/ppl wcc/pbblicgguuiiccceeec tttgggidddeeeoeebbbiiinnnn
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All primary hyperparameters presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
The majority of our results are significantly outperforming the baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All our experiments are conducted using x A6000 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////.aaro...ooo//oo///aaaa
Answer: [Yes]
Justification: NeurlPS Code of Ethics is followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper doesn't 't have potential positive societal impacts and negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no risk involved.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All assets are in public domain.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "HnmwpBCdZD", "reviewer": "Reviewer_qA6u", "summary": "This paper addresses the 'lost-in-the-middle' issue in large language models (LLMs) by introducing Multi-scale Positional Encoding (Ms-PoE). This approach enhances LLMs' ability to handle relevant information in the middle of the context without fine-tuning or added overhead. Ms-PoE uses position index rescaling and distinct scaling ratios for different attention heads to maintain essential knowledge. Experiments show Ms-PoE improves LLM accuracy, achieving up to a 3.8% gain on the Zero-SCROLLS benchmark.", "questions": "1.\tL212-215: “With a small scaling ratio LLMs tend to focus more on the most recent part of the context while with a large ratio, LLMs favor the beginning part” Are there any supporting results for this claim?\n\n2.\tDid you choose the hyperparameters of R_min and R_max directly from their performance on the test set?\n\n3.\tI’m happy to raise the scores if all the concerns are well addressed.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.\tThis paper is well-written and very easy to follow.\n\n2.\tThe analysis of the position embedding sensitivity of different attention heads is very interesting and intriguing.", "weaknesses": "1．\tThe improvement is actually limited, despite the method’s efficiency and ease of use.\n\n2．\tThe heuristic of the scaling ratio allocation process is a little arbitrary. There could be better solutions than just using an arithmetic sequence to achieve that. It is not strictly proved that a larger S_P means that the model should have a less rescaling ratio and vice versa. There could be possibilities that different heads could use the same scaling ratio or the other way around. \n\n3．\tMoreover, there is no evidence that the rescaled position embedding could make the attention heads like the “Top in Figure 4” better at catching relevant information except for an empirical marginal task performance improvement and the comparison with other strategies in Section 4.3. An attention map of the attention heads after the rescaling could better strengthen this main claim of the paper. The authors could also partially show this by examining the S_P variation before and after the rescaling. \n\n4．\tIn Section 3.1, directly adjusting the hyperparameter and seeing the results on the test set is not very appropriate since it means the test set information is leaked. The authors should analyze the scaling ratio on the validation set and see if it is consistent with that in the test set. Moreover, the selection of the hypermeters R_min and R_max is also probably acquired by the test set performance, reflected by the ablation experiments in Table 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer qA6u for acknowledging our work as “interesting and intriguing”. We provide pointwise responses in the following.\n\n**[Q1: Limited Improvements]** We respectfully disagree with the claim that our improvements are limited. Our method offers a plug-and-play solution to enhance current open-source LLMs without any fine-tuning. And the enhancements are consistent across multiple benchmarks, including a 3.8-point average improvement on ZeroSCROLLS (Table 1), a 3.92-point increase in Multi-Document QA tasks, and a 43.72-point gain in KV Retrieval tasks (Table 2). Additionally, as shown in Table R1 in the uploaded PDF, our evaluations on the LongBench benchmark further demonstrate a consistent improvement of up to 6.67 points. Therefore, we believe our improvements are both consistent and significant.\n\n**[Q2: Studies of Scaling Ratio Allocation]** Thank you for the insightful comments. In our experiments, we chose the heuristic linear (arithmetic sequence) rescaling ratio due to its simplicity and input-independent nature. We can also rescale each attention head using other strategies, such as directly via the position-aware score ($S_P$) while it introduces additional computational costs. Since when implementing a scaling ratio based on the position-aware score, each input would have distinct rescaling ratio values. This would necessitate recalculating the $\\theta$ value in RoPE on the fly, leading to additional computational overhead. Additionally, we can also allow different heads to share the same scaling ratio.\n\nTo further address Reviewer qA6u’s concerns, we explored multiple scaling ratio allocation strategies and reported the results in Table R2. For the exponential and cosine strategies, we aimed to examine whether non-linear allocation strategies perform well. For the stepwise solution, we allowed different attention heads to share the same scaling ratio, and the position-aware score provided evidence for rescaling directly based on its raw scores. We observed that the position-aware strategy achieves similar results to the linear strategy but requires extra computational overhead. Similarly, allowing different attention heads to share the same scaling ratio yields comparable results. Therefore, based on the superior performance and computational efficiency, we chose the linear assignment method.\n\n**Details**: We compare several strategies in our experiments, including (i) Exponential: $r_i = 1.8 - (1.8-1.2) \\cdot 0.9^{i}$, where $r_i$ is the i-th scaling ratio and 0.9 is the multiplication factor for scaling ratio decay. (ii) Cosine: $r_i = 1.8 - (1.8-1.2)cos(\\frac{i}{n_h-1}\\phi)$, where $n_h$ is the number of attention heads. (iii) Stepwise: $r_i = 1.2 + \\frac{1.8-1.2}{3} round (\\frac{i*4}{n_h})$, (iv) Position-Aware score: $r = 1.2 + \\frac{(1.8-1.2)\\cdot(S_P-min(S_P))}{max(S_P)-min(S_P)}$, as well as the Linear strategy in our current implementation. \n\nTable R2. Results of different scaling ratio strategies on the MDQA task with Vicuna-7B.\n\n|Key-Document-Index|1|3|5|7|10|Average|\n|---|---|---|---|---|---|---|\n|Baseline|64.0|61.0|57.4|58.4|64.8|61.12|\n|Ms-PoE (Linear)|65.6|64.2|63.0|65.2|67.2|65.04|\n|Exponential|64.0|60.0|63.6 | 63.6 | 63.6 |62.96|\n|Cosine|66.4| 63.0| 63.2 | 65.6 |66.4 |64.92|\n|Stepwise| 62.6|63.6 |63.6 |67.6 |68.0 | 64.88|\n|Position-Aware score|65.2|64.0|63.6 | 65.6 | 66.3 |64.94|\n\n\n**[Q3: Visualization of Attention Pattern before and after Rescaling]** Thank you for the suggestion. We visualized the attention patterns of different attention heads and reported the variation of $S_P$ before and after rescaling. The results are demonstrated in Figures R1 in the uploaded PDF. We found that the rescaling step effectively enhances the context utilization of LLMs, making some non-\"position-aware\" heads focus on critical information. Quantitatively, $S_P$ consistently increases with an average improvement from 1.79% to 1.93%.\n\n\n**[Q4: Determination of $R_{min}$ and $R_{max}$]** Thanks for pointing it out. We’d like to clarify that we didn’t search the best hyperparameter of $R_{min}$ and $R_{max}$ on the test set. Instead, we randomly selected 500 samples from MDQA tasks as the validation set and did the ablation studies of different scaling ratios on this validation set. We found that selecting a ratio between 1.2 and 1.8 significantly boosts the performance. Then. we apply the same ratios to all other tasks without further adjustment. \n\n**[Q4: Scaling Ratio Affects the Favored Zone]** Thanks. We report the raw accuracy results corresponding to Figure 3 in Table R3 where we position the key documents at the beginning, middle, or end of the sequences. From Table R3, we can observe that “changing the scaling ratio also affects the favored zone of LLMs. With a small scaling ratio (e.g., 0.5), LLMs tend to focus more on the most recent part of the context, while with a large ratio (e.g., 2.5), LLMs favor the beginning part”. We have included this table in the revised version for a clearer understanding of the content discussed in Lines 212-215.\n\nTable R3: Accuracy results for the MDQA task when key documents are positioned at various locations within the sequences. Different rescaling ratios are applied, with all attention heads sharing the same rescaling ratio.\n|Llama-2-7B-Chat | Beginning | Middle| End|\n|---|---|---|---|\n|0.5|36.6|41.8|59.4|\n|1|52.6|53.2|64.0|\n|1.5|59.0|58.4|59.8|\n|2|59.4|59.4|59.8|\n|2.5|62.4|55.6|56.0|\n\n|Vicuna-7B | Beginning | Middle| End|\n|---|---|---|---|\n|0.5|56.0|51.0|68.0|\n|1|64.0|57.4|64.8|\n|1.5|65.2|60.0|64.0|\n|2|61.5|59.0|62.5|\n|2.5|59.5|57.5|57.0|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N6AeWpbO7Z", "reviewer": "Reviewer_xsrg", "summary": "This paper proposes a plug-and-play method named Ms-PoE to mitigate the lost-in-middle challenge of LLMs. Specifically, Ms-PoE leverages multi-scale position embeddings to enhance information awareness in different parts of the context. Without fine-tuning the model,  Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs.", "questions": "1. What's the performance of MS-PoE(with LLama-2-7B-Chat as the backbone model) on LongBench-EN benchmark ?\n2. Why chose 1.2 and 1.8 for R_{min} and R_{max} in Equation 2 ?\n3. See Weakness 2", "limitations": "See Question and Weakness above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. MS-PoE is a plug-and-play, simple yet efficient method, which is training-free and can achieve good performance.\n\n2. The observation of \"position-aware\" attention heads is insightful, and applying re-scaling ratio dynamically to each attention head is reasonable.\n\n3. I strongly agree with the authors' opinion that \"the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled.\"", "weaknesses": "1. It seems that the performance improvement on MDAQ and ZeroSCROLLS benchmark is small compared to the \"Self-Extend\" method, which can be seen as a special type of \"Multi-Scale Positional Encoding\", i.e., \"Single-Scale Positional Encoding\".\n\n2. Although this paper suggests that Ms-PoE is suitable for \"Long Contexts,\" I believe it is suitable for any context situation. What exactly defines a \"Long Context\"? Is it a context length above 12K? Additionally, I have not come across any analysis of the impact of context length and how Ms-PoE can help the model understand the middle of long contexts. It might be beneficial to conduct experiments to determine whether Ms-PoE can enhance the model's understanding of the intermediate content of the context. This could involve analyzing different context lengths, such as inserting a specific length of context and evaluating the model's predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer xsrg for acknowledging our method is “efficient”, and the observation of attention heads is “insightful”. To address Reviewer xsrg’s concerns, we provide pointwise responses in the following.\n\n\n**[Q1: Limited Improvements]** We respectfully disagree that our performance improvement is limited compared to “Self-Extend.” As demonstrated in Table 2, “Self-Extend” achieves similar performance to direct positional interpolation, or “Single-Scale Positional Encoding,” with gains of 1.20 and 29.76 on the MDQA and KV Retrieval tasks, respectively. In contrast, our Multi-Scale Positional Encoding achieves significantly higher gains of 3.92 and 43.72 on the same tasks, respectively, clearly surpassing the improvements achieved by “Self-Extend.” \n\n\n**[Q2: Ablation Studies of Context Lengths]** That’s a good suggestion. We conducted additional evaluations of our method across varying input lengths by changing the number of documents from 3 to 15. In these experiments, the key document is consistently positioned in the middle of the input to assess the LLMs' ability to capture middle context information. As depicted in Table R1, when the input length is short, our approach shows negligible improvement compared to the Baseline, with a 0.2 accuracy gain. However, as we increase the number of documents, thereby lengthening the input, significant improvements are achieved, with gains up to 6.4 in accuracy. \n\nTable R1: Results of the effectiveness of our approach across different input lengths. The relevant key documents are located in the middle of the inputs and experiments are conducted with Vicuna-7B\n|Number-of-Docs|3|5|7|10|15|\n|---|---|---|---|---|---|\n|Baseline|69.2|63.8|62.8|57.4|52.8|\n|Ours|69.4|64.4|67.0|63.0|59.2|\n|Improvements|0.2|0.6|4.2|5.6|6.4|\n\n\n\n**[Q3: Results on LongBench-EN Benchmark]** Thanks for the question. We conduct additional experiments of all 13 tasks from the LongBench-EN benchmark. Results are reported in Table R3. We can observe that Ms-PoE achieves consistent performance improvement without any finetuning.\n\nTable R3. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the LongBench benchmark. We use the same scaling ratios without further turning.\n\n| Methods    | MultiFieldQA-en | LCC   | GovReport | HotpotQA | Passage Count | Qasper | MultiNews | SAMSum | TriviaQA | PassageRetrieval-en | RepoBench-P | TREC  | 2WikiMQA | Average |\n| ---------- | --------------- | ----- | --------- | -------- | ------------- | ------ | --------- | ------ | -------- | ------------------- | ----------- | ----- | -------- | -------- |\n| w.o Ms-PoE | 33.51           | 59.77 | 27.97     | 30.10    | 3.74          | 19.27  | 24.36     | 39.45  | 82.81    | 10.00               | 49.22       | 57.33 | 28.14    | 35.82 |\n| w. Ms-PoE  | 37.33           | 62.03 | 29.87     | 34.08    | 4.60          | 20.96  | 24.69     | 39.79  | 85.28    | 16.67               | 50.11       | 58.67 | 30.19    | 38.02 |\n\n\n\n\n**[Q4: Determination of $R_{min}$ and $R_{max}$]** Thanks for the question, we determine the $R_{min}$ and $R_{max}$ via an ablation study on the MDQA task. Specifically,  we randomly selected 500 samples from MDQA tasks as the validation set and examined the effect of different scaling ratios, the results are reported in Table 4. We then apply the same ratios to other downstream tasks without further adjustment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PPU3GCIYNf", "reviewer": "Reviewer_LVQT", "summary": "Paper addresses the lost-in-the-middle effect, observed in the past for some LLMs, with a method called Multi-scale Positional Encoding (Ms-PoE) where position is encoded using different scales for each attention head. More precisely, for each head a re-scaling ratio is substituting the position m of a token with m/r (r can be 1.5 for instance) and different values of r are used for each attention head. This multi-scale view is evaluated experimentally and authors claim that this mitigates the ‘lost-in-the-middle’ effect.", "questions": "-eq(2): how are Rmin & Rmax fixed ?\n\n-fig 5: why only vicuna model evaluated, does Llama-2-7B-Chat also display a lost-in-the-middle effect ?\n\n-tab.3 are the differences between begin/middle/end results really significant ? A statistical test (for instance one-tailed Welch’s t-test) would have been welcome here", "limitations": "No 'limitations' section was provided", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-a new approach for positional encoding Multi-scale Positional Encoding (Ms-PoE)\n-approach is plug-and-play (no additional fine-tuning needed)", "weaknesses": "-The \"lost-in-the-middle\" effect is often assumed to impact all LLMs. But  recent research indicates that this effect does not uniformly affect all LLMs, though the effect was observed on models such as Vicuna7B. \n\tsee for instance https://arxiv.org/abs/2403.20262  which shows that not all LLMs display the lost-in-the-middle effect on a long context benchmark\n\n-For evaluation,  the ZeroSCROLLS benchmark was used but authors could have considered  incorporating other benchmarks designed for long context scenarios, such as \"needle in the haystack\" tasks. There are also many other relevant benchmarks available, including: NarrativeQA; LongEval; LongBench; LongBench-Chat; Loogle;  ∞Bench and Long Range Arena (LRA) among others\n      => on that aspect paper could have been stronger experimentally if evaluating the approach on more long context benchmarks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer LVQT for supporting our work and providing constructive suggestions. To address Reviewer LVQT’s concerns, we provide point-wise responses below.\n\n\n**[Q1: No Lost-in-the-Middle Effect]** Thank you for the insightful question. We have observed similar findings to those in [1], particularly with **LLaMA-2-7B-Chat**, which does not exhibit the lost-in-the-middle phenomenon but rather shows performance loss at the beginning. The results are reported in Table R1. Additionally, after applying Ms-PoE, consistent improvements are achieved by an average of 1.5 accuracy. This further demonstrates the effectiveness of our approach for enhancing the context utilization of LLM inference. We’ve added the results in the updated draft.\n\n[1] ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models.\n\nTable R1. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the MDQA tasks.\n\n|Key-Document-Index|Beginning|Middle|End|Average|\n|---|---|---|---|---|\n|w.o. Ms-PoE|52.6|53.2|64.0|56.6|\n|w. Ms-PoE|56.0|55.6|64.6|58.7|\n\n\n**[Q2: More Benchmark Results]** Thanks for the suggestions. We conducted additional experiments on all 13 tasks from the LongBench benchmark for further evaluation, and the results are presented in Table R2. Our findings show that Ms-PoE consistently demonstrated effectiveness across all 13 tasks, with improvements reaching up to 6.67 and an average of 2.2. It is important to note that we fixed $R_{min}=1.2$ and $R_{max}=1.8$ for all tasks without tuning on the test set.\n\nTable R2. Comparison results of LLaMA-2-7B-Chat with and without Ms-PoE on the LongBench benchmark. We use the same scaling ratios without further turning.\n\n| Methods    | MultiFieldQA-en | LCC   | GovReport | HotpotQA | Passage Count | Qasper | MultiNews | SAMSum | TriviaQA | PassageRetrieval-en | RepoBench-P | TREC  | 2WikiMQA | Average |\n| ---------- | --------------- | ----- | --------- | -------- | ------------- | ------ | --------- | ------ | -------- | ------------------- | ----------- | ----- | -------- | -------- |\n| w.o Ms-PoE | 33.51           | 59.77 | 27.97     | 30.10    | 3.74          | 19.27  | 24.36     | 39.45  | 82.81    | 10.00               | 49.22       | 57.33 | 28.14    | 35.82 |\n| w. Ms-PoE  | 37.33           | 62.03 | 29.87     | 34.08    | 4.60          | 20.96  | 24.69     | 39.79  | 85.28    | 16.67               | 50.11       | 58.67 | 30.19    | 38.02 |\n\n\n**[Q3: Determination of $R_{min}$ and $R_{max}$]** Thanks for the question, we determine the $R_{min}$ and $R_{max}$ via an ablation study on the MDQA task. Specifically,  we randomly selected 500 samples from MDQA tasks as the validation set and examined the effect of different scaling ratios, the results are reported in Table 4 where $R_{min}=1.2$ and $R_{max}=1.8$ demonstrates superior performance.  We then apply the same ratios to other downstream tasks without further adjustment.\n\n\n**[Q5: Welch’s t-test of Table 3]** Good suggestion. We conducted a further statistical analysis of the results presented in Table R3. As illustrated in Table R3, the baseline method shows a significant difference in performance when critical documents are positioned in the middle versus the beginning of the inputs, as well as between the middle and the end. However, the difference between the beginning and the end is not significant (p-value > 0.05). In contrast, when applying our Ms-PoE method, there is no significant difference in performance between the beginning, middle, and end positions, and it also achieves better average performance. These findings further confirm the effectiveness of our approach, and we have included these results in the updated manuscript.\n\nTable R3. Welch’s t-test of the results when different ordering metrics are applied. The Null hypothesis states that there is no difference between the means of two results.\n\n|      p-value      | Begin v.s. Middle | Begin v.s. End | Middle v.s. End |\n| :----------------: | ------------------ | -------------- | --------------- |\n|      Baseline      | 0.033              | 0.792          | 0.016           |\n|       Random       | 0.002              | 0.740          | 0.001           |\n|     Sequential     | 0.055              | 0.519          | 0.202           |\n|      Entropy      | 0.194              | 0.877          | 0.145           |\n| Position-Awareness | 0.391              | 0.562          | 0.151           |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank Reviewer qA6u, xsrg, and LVQT for their constructive suggestions and valuable questions. Additional supplementary materials are provided in the PDF, including:
- The attention patterns before and after rescaling **[Reviewer qA6u]**
- More results on LongBench. **[Reviewer qA6u, xsrg, LVQT]**

Meta-review: This paper proposes a plug-and-play solution for the the "lost-in-the-middle" problem in large language models (LLMs) by introducing Multi-scale Positional Encoding (Ms-PoE). Ms-PoE uses position index rescaling and distinct scaling ratios for different attention heads to maintain essential knowledge. Experiments show Ms-PoE improves LLM accuracy on the several benchmarks.

Reviewers agree that this paper addresses an important problem and praise the simplicity and effectiveness of the proposed approach and the fact that it does not require any training. However they point out several weaknesses, such as the modest improvement in some tasks, the lack of analysis of performance in terms of context length, the lack of justification for the scaling ratio allocation heuristic (via an arithmetic sequence), and the process of tuning hyperparameters. Most of these issues have been convincingly addressed by the authors during the rebuttal, where they also added several other experiments (attention patterns before and after rescaling, new experiments in LongBench, and additional evaluations of the proposed method across varying input lengths). I am leaning towards acceptance.
Decision: Accept (poster)