Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs
Authors: Runchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng, Qinyu Luo, Cheng Qian, Shuo Wang, Xin Cong, Zhong Zhang, Yesai Wu, Yankai Lin, Huadong Wang, Xiaojiang Liu
Accepted at: acl25

Distance between Relevant Information Pieces
Causes Bias in Long-Context LLMs

Runchu Tian1*, Yanghao Li1*, Yuepeng Fu1, Siyang Deng1, Qinyu Luo1
Cheng Qian1, Shuo Wang1, Xin Cong1†, Zhong Zhang1, Yesai Wu1
Yankai Lin2†, Huadong Wang1, Xiaojiang Liu3
1 Department of Computer Science and Technology, Tsinghua University
2 Gaoling School of Artificial Intelligence, Renmin University of China 3Apple Inc.
trc20@mails.tsinghua.edu.cn, goody1027@gmail.com

Abstract
Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LONGPIBENCH, a benchmark designed to assess positional bias involving multiple pieces of relevant information. It includes various tasks and input lengths. Thorough experiments are conducted with three commercial and six open-source models. These experiments reveal that while most current models are more robust against the "lost in the middle" issue, there also exist noticeable biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases for long-context LLMs. Code and data have been made publicly available.

1 Introduction
Large language models (LLMs) (Zhao et al., 2023; Minnaee et al., 2024) have made significant progress in various natural language processing tasks (Hendrycks et al., 2021; Han et al., 2021). In particular, applications such as code repository analysis (Chen et al., 2021) and information extraction (Kočiský et al., 2018) often require processing long texts, with context lengths reaching up to 200,000 tokens (Li et al., 2024; Zhang et al., 2024a). To address these demands, researchers have focused on enhancing LLMs' ability to handle extended inputs effectively (Chen et al., 2024; Han et al., 2024). As a result, multiple LLMs have been developed (Dubey et al., 2024; Team et al., 2024; OpenAI, 2024) which support context lengths of up to one million tokens.

Recent studies have shown that the position of relevant information significantly affects the performance of long-context LLMs (Liu et al., 2024; Lei et al., 2023; Hsieh et al., 2024). In "needle in a haystack" tasks, models struggle to utilize information located in the middle of the input, which is known as the "lost in the middle" effect (Liu et al., 2024). This evaluation method is commonly used to analyze positional bias (Hengle et al., 2024; Nelson et al., 2024). These analyses (Liu et al., 2024) focused on single relevant information pieces and their positions in the input sequence (front, middle, back), which we refer to as absolute positions.

However, real-world tasks like data analysis (Zhang et al., 2024a) often involve multiple pieces of relevant information. This introduces a new characteristic: the distance between relevant information pieces, or how densely they are distributed, which we term as relative position. Evidence from two types of extreme cases indicates that varying relative position may lead to significant bias, impairing LLMs' long-context performance (Lei et al., 2023). However, this kind of biases have not been systematically studied so far, which highlights the need for thorough investigation.

To bridge the gap, we introduce LONGPIBENCH, a benchmark designed to evaluate positional bias with multiple relevant pieces. It assesses positional bias in two categories: (1) absolute positions, referring to the location of relevant information within the entire context, and (2) relative positions, referring to the distribution and distance between multiple relevant information pieces. It includes diverse tasks of different complexity and spans four input lengths from 32K to 256K tokens. To the best of our knowledge, LONGPIBENCH is the most comprehensive benchmark for isolating and analyzing positional bias in long text models.

![Illustration of absolute position and relative position. Absolute position refers to the location of relevant information within the entire context sequence, while relative position represents the distribution and distance between multiple relevant information pieces.](page_1012_370_323_246.png)

Figure 1: Illustration of absolute position and relative position. Absolute position refers to the location of relevant information within the entire context sequence, while relative position represents the distribution and distance between multiple relevant information pieces.
Figure 2: Construction and task examples of LONGPiBENCH. We manually annotated seed data and varied the positions of relevant information for data augmentation.

We evaluated nine popular LLMs. Our experimental analysis yields several key findings: (1) most current models demonstrate enhanced robustness against "lost in the middle" phenomenon. (2) However, they show biases related to the spacing of relevant information (i.e. relative positions), especially in retrieval tasks. (3) Additionally, we discuss the impact of model size and query-aware contextualization on this issue.

These findings emphasize the importance of evaluating and mitigating positional biases to advance long-context LLM capabilities.

2 Related Works

2.1 Long-Context Benchmarks

Many benchmarks have been proposed to evaluate long-context performance of LLMs by designing a variety of tasks with different context length. This field is relatively saturated at present, with some of the representative benchmarks including Long Range Arena (Tay et al., 2021), Scrolls (Shaham et al., 2022), ZeroScrolls (Shaham et al., 2023), Longbench (Bai et al., 2023), L-Eval (An et al., 2023), LV-Eval (Yuan et al., 2024), Counting-Stars (Song et al., 2024) and ∞Bench (Zhang et al., 2024b). However, these benchmarks tend to provide only a general conclusion regarding which task types are more challenging, without offering in-depth analysis on positional bias like this paper proposes.

2.2 Long Context Data Augmentation

Data augmentation is a technique widely used (Song et al., 2024; Zhang et al., 2024b) in LLM evaluation to expand the datasets for different purposes. Specifically, Levy et al. (2024) explored the impact of input length on reasoning performance using a similar data augmentation approach, adding irrelevant elements to context-relevant elements. While their method shares some similarities with ours, our focus is fundamentally different, leading to entirely distinct conclusions. Their study centers on the overall input length which has nothing to do with positional bias. But we investigate the distance between relevant information pieces, where the input length is fixed.

3 LONGPiBENCH

LONGPiBENCH is a dataset designed to evaluate positional bias with multiple relevant information pieces. As shown in Figure 2, we first manually annotated several seed examples and then augmented them by varying the positions of relevant information. More details can be found in Appendix A.

3.1 Core Statistics

LONGPiBENCH contains 3 different tasks, 4 different input length levels1: (32k, 64k, 128k, and 256k). To analyze the impact of positional bias, we set 16 different absolute and relative location levels respectively. The benchmark is composed of 7,040 instances, each containing around 10 pieces of relevant information. The whole dataset comprises to 845M tokens.

3.2 Seed Data Annotation

We manually labeled 15-20 seed data points for three tasks: Table SQL, Code Completion, and Wiki Retrieval, which represent typical use cases in real-world applications of long-context models (Lei et al., 2023; Jimenez et al., 2024; Ajith et al., 2024).

Table SQL focuses on querying large tables to retrieve entries with specific features, based on experiments from S3Eval (Lei et al., 2023) that explored data with highly variable positions. Code Completion requires solving basic programming problems using long masked API documentation. LLMs must identify and apply relevant API details without relying on prior knowledge. The data comes from Zan et al. (2022). Wiki Retrieval tests LLMs’ ability to find and rerank (Sun et al., 2023) relevant Wikipedia (Wikipedia, 2024) passages in response to a question, simulating typical information retrieval workflows (Zhang et al., 2023).

Each instance contains 10 relevant pieces of information. This selection was based on an examination of long-context application scenarios, where the number of relevant elements typically falls around the order of magnitude of ten, although it varies across different tasks (Bai et al., 2023; Wang et al., 2024; Dong et al., 2024). Detailed task definitions, examples, and other pertinent details are provided in Appendix A.

1 measured with GPT2Tokenizer (Radford et al., 2019)
Figure 3: The impact of relevant information’s absolute and relative position on Geimini-1.5-Flash (Team et al., 2024), Claude-3.5-Haiku (Anthropic, 2024) and Qwen 2.5 model family (Qwen, 2024). A higher absolute position feature level indicates locations closer to the end of input, while a higher relative position feature level indicates a greater distance between relevant pieces of information.

3.3 Data Augmentation

To analyze the positions of relevant information, we augmented the data by altering the absolute and relative positions of the relevant pieces while keeping all other features unchanged.

We broke down the context into elements based on natural information units: table entries for Table SQL, API instances for Code Completion and documents for Wiki Retrieval. We labeled each element as relevant or irrelevant in a reversal way. We select some elements to be relevant, and then form queries around them, and add irrelevant ones to form the context. By introducing varying amounts of irrelevant information, the context lengths are varied at four levels: 32K, 64K, 128K, and 256K. We then shuffled the element positions to introduce positional variations. Notice that changing the order of elements does not compromise the coherence of the context.

Absolute Position. To analyze the impact of absolute position on LLM performance, we manipulated where relevant information appears in the context. Each context was divided into 16 equal segments from start to end. We placed all 10 relevant pieces within a single segment to keep their relative positions consistent. By moving this segment from the first to the last position, we varied the absolute position from the start to the end of the input. The average position of these relevant pieces served as the absolute position metric which is calculated as:

\[
\text{Average Location} = \left( \frac{l - 1}{N - 1} \right) \times L,
\]

where \( l \) is the current level, \( N \) is the total number of levels (16), and \( L \) is the length of the context.

This setup allowed us to assess how model performance changes as relevant information is placed further back in the context.

Relative Position. To examine the effect of spacing between relevant information pieces on LLM performance, we created 16 levels of distribution density. Each level represents a different spacing configuration among the 10 relevant pieces. At the densest level, all relevant pieces are adjacent with no irrelevant information between them. At the sparsest level, they are evenly distributed throughout the context with equal intervals of irrelevant information. Intermediate levels gradually increase spacing from adjacent to evenly spaced. The distance between each relevant piece is calculated as:

\[
\text{Distance} = \left( \frac{L}{n - 1} \right) \times \left( \frac{l - 1}{N - 1} \right),
\]

where \( n \) is the number of relevant pieces (10), \( l \) is the current level ranging from 1 to \( N \), \( N \) is the total number of levels (16), and \( L \) is the length of the context.

To control for absolute position effects, we randomized the starting position of the first relevant piece in each example. This ensures that any observed performance differences are due to relative spacing rather than absolute positions within the context.

4 Experimental Setup

To evaluate the influence of context information positioning on long-text LLMs, we conducted experiments using popular long-context language models.

Models. We assessed a total of nine LLMs, comprising six open-source and three commercial options. The selection of open-source models includes the 70B model from Llama-3.1-Instruct series (Dubey et al., 2024), the 7B, 14B, 32B, 72B models from Qwen-2.5 family (Qwen, 2024), the 8×22B model of WizardLM-2 (Xu et al., 2023). The commercial models we selected are Gemini-1.5-Flash (Team et al., 2024), Claude-3-Haiku (Anthropic, 2024) and GPT-4o-mini (OpenAI, 2024). The selected models are good representatives of popular and top-performance long-context models. Due
to computational limitations, we evaluated the open-source model only on the Table SQL task.

Metric. For both the Table SQL and Wiki Retrieval tasks, performance is measured using recall rate. This metric evaluates the proportion of relevant items included in the output. Formally, given a set of reference items \( D = \{d_1, \ldots, d_n\} \) and a set of retrieved/generated items \( \hat{D} \), the recall rate is:

\[
M_{\text{Recall}} = \frac{|D \cap \hat{D}|}{|D|}.
\]

In Table SQL, \( D \) represents target entries, and \( \hat{D} \) represents the entry present in the output. In Wiki Retrieval, \( D \) represents the set of relevant documents, and \( \hat{D} \) represents the top 10 documents retrieved by the model.

For the Code Completion task, performance is evaluated with the pass rate across 8-12 test cases \( T = \{t_1, \ldots, t_m\} \). The pass rate is computed as:

\[
M_{\text{Code}} = \frac{1}{|T|} \sum_{j=1}^{|T|} 1[G \text{ passes } t_j].
\]

All metrics range from 0.0 to 1.0, where 0.0 means complete failure, and 1.0 means perfect performance.

Context Length. Since 32k tokens is the minimal context length supported by tested LLMs, we standardized the context length to 32k\(^2\) tokens for all experiments.

Detailed discussions on parameter settings and prompt configurations are provided in Appendix B.

5 Results and Discussion

In this section, we analyze the impact of absolute and relative positional bias. And we further analyze these phenomena from two perspectives: the number of parameters and query-aware contextualization. Full experimental results are available in Appendix C.

5.1 Impact of Absolute Position

As illustrated by the blue lines in Figure 3, we progressively shift the interval of relevant information from the beginning to the end.

We observe that (1) some open-source models like Qwen 2.5 (7B) (Qwen, 2024) still suffer heavily from the severe "lost in the middle" phenomenon but (2) commercial models and larger open-source models are more robust to the bias of absolute position. Although absolute position still significantly affects the recall rate in the Code Completion experiments, this bias becomes less severe in the Table SQL and Wiki Retrieval tasks.

5.2 Impact of Relative Position

As illustrated by the orange lines in Figure 3, we progressively increase the distance between relevant pieces of information.

\footnotetext{2The minimal context size is 64k, but some tokenizers expand our 64k inputs to nearly 80k, exceeding the limit.}

We observe that both open-source and commercial models exhibit noticeable biases toward different relative positions. In the case of Code Completion, this bias is prominent. As the relative positions of relevant information pieces shift from being fully adjacent to uniformly distributed across the context, the model's performance fluctuates by 20-30%. For tasks with a stronger retrieval nature, such as Table SQL and Wiki Retrieval, the bias even displays certain patterns. Specifically, performance initially declines sharply and then decreases more gradually.

These findings indicate that the relative positioning among multiple relevant pieces of information is a serious and unresolved issue, which may substantially undermine the effectiveness of long-text language models in practical applications.

5.3 Further Analysis

Effect of Parameter Size. When selecting models for evaluation, we included four variants from the Qwen 2.5 Family (Qwen, 2024) with differing parameter sizes. These models exhibit no significant differences in architecture, training methods, or training data. By analyzing their performance under identical positional information features, we can isolate the impact of parameter size on the robustness to positional bias. We use Table SQL task, where the pattern is most significant

As illustrated in Figure 3, for absolute position bias, we found that simply increasing the model parameters from 7B to 14B—while keeping architecture, training methods, and data constant substantially mitigates the "lost in the middle" (Liu et al., 2024) issue. This suggests that robustness to absolute positions may be an "emergent ability" (Wei et al., 2022) and increasing the number of parameters can significantly enhances it.

In contrast, regarding biases related to relative positional information, augmenting the number of parameters only yielded minor quantitative improvements and did not alter the pronounced bias trend. This trend remains largely unchanged even in commercial models with approximately hundreds of billions of parameters. These findings indicate that merely increasing parameter size is insufficient to develop robustness to relative positions, and new techniques may be necessary.

Effect of Query-Aware Contextualization. Liu et al. (2024) demonstrated that the placement of the query (beginning or end of the context) significantly affects the performance of decoder-only models due to unidirectional attention. When the query is placed after the context, the LLM cannot attend to the query token while processing the context tokens.

As shown in Figure 4, our experiments with GPT-4o-mini (OpenAI, 2024) and Qwen-2.5-14B (Qwen, 2024) on Table SQL corroborate this observation and confirm that it also holds for bias caused by relative position changes. When the query is placed at the end of the context, the model performs much worse than when the query is at the beginning or both at the beginning and
end. However, the difference between placing the query only at the beginning and at both the beginning and end depends on the model. This indicates that for decoder-only long-text models, the position of the query is also crucial in influencing biases related to the absolute and relative positions of relevant information.

6 Conclusion

This study investigates a new category of positional bias involving multiple relevant pieces of information in long-context LLMs through three key contributions.

(1) Benchmark Development: We introduce LONG-PiBENCH, the most comprehensive benchmark for evaluating positional bias in long-text LLMs, assessing both absolute and relative biases.

(2) Comprehensive Evaluation: Using LONG-PiBENCH, we evaluated nine popular LLMs, investigated the "lost in the middle" phenomenon, and identified novel yet significant biases related to the relative positioning of multiple relevant pieces of information.

(3) Findings: Our experiments show that while LLMs have improved robustness against absolute positional biases, they are still sensitive to relative positional biases, especially for retrieval-intensive tasks. We also explore how model size and query-aware contextualization impact these biases.

These findings emphasize the necessity of continuously mitigating positional biases in long-text models.

Limitation

Lack of In-depth Analysis. Our systematic experiments demonstrate that two types of positional bias exist when multiple related pieces of information are present in the context. We also analyzed how these biases relate to the number of parameters and query contextualization. However, we are currently unable to explain the reasons behind these two positional biases.

Focus on Specific Models. The evaluation was conducted on a set of nine popular large language models (LLMs), including both open-source and commercial options. However, the findings are limited to these models. The study does not account for the performance of other emerging or less popular models, which might exhibit different results regarding positional biases.

Ethical Considerations

Human Annotation. Our seed construction process involves manual annotation. This annotation was carried out by some of the authors, who are researchers with substantial knowledge in LLM evaluation. Consent was obtained from the individuals whose data we are using or curating. The data collection protocol was approved.

Data Security. Some data in our Table SQL task may appear to pertain to personal information. However, this data is not actual personal information. Instead, it is generated by us through specific heuristics, eliminating the risk of personal information leakage.

Use of AI assistants We use GPT-4o (OpenAI, 2024) for expression modification and grammar sanity check during the composition process.

Acknowledgment

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62376273), the Postdoctoral Fellowship Program of CPSF (Grant No. GZB20230343 and Grant No. GZC20240831) and the China Postdoctoral Science Foundation (Grant No. 2023M741945).

We thank all collaborators who made their contributions to this project, including Runchu Tian (trc20@mails.tsinghua.edu.cn), Yanghao Li (goody1027@gmail.com), Yuepeng Fu (fuyup22@mails.tsinghua.edu.cn), Siyang Deng (sydeng@bupt.edu.cn), Qinyu Luo (qinyu_luo@163.com), Cheng Qian (qianc20@mails.tsinghua.edu.cn), Shuo Wang (wangshuo.thu@gmail.com), Xin Cong (congxin1995@tsinghua.edu.cn), Zhong Zhang (zhongzhang@tsinghua.edu.cn), Yesai Wu (wuyesai@gmail.com), Yankai Lin (yankailin@ruc.edu.cn), Huadong Wang (huadw2012@163.com), and Xiaojiang Liu (xiaojiang_liu@apple.com). For further information or collaboration, feel free to contact them.

References

Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. Litsearch: A retrieval benchmark for scientific literature search. ArXiv preprint, abs/2407.18940.

Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-Eval: Instituting Standardized Evaluation for Long Context Language Models. arXiv e-prints, pages arXiv–2307.
Anthropic. 2024. Introducing claude 3.5 sonnet. Accessed: 2024-09-15.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A Bilingual, Multitask Benchmark for Long Context Understanding. ArXiv preprint, abs/2308.14508.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. LongLora: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2086–2099, Torino, Italia. ELRA and ICCL.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. LM-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991–4008, Mexico City, Mexico. Association for Computational Linguistics.

Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhang Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021. Pre-trained models: Past, present and future.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models.

Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization.

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328.

Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. 2023. S3eval: A synthetic, scalable, systematic evaluation suite for large language models.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. ArXiv preprint, abs/2402.14848.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024. LooGLE: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16304–16333, Bangkok, Thailand. Association for Computational Linguistics.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlou, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. ArXiv preprint, abs/2402.06196.

Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. 2024. Needle in the haystack for memory based large language models.

OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-06-08.

Qwen. 2024. Qwen2.5: A party of foundation models.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007–12021, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A multi-evidence, position-aware, and scalable benchmark for evaluating long-context large language models. Preprint, arXiv:2403.11802.

Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatpt good at search? investigating large language models as re-ranking agent. ArXiv preprint, abs/2304.09542.

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena : A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Gemini Team, Petko Georgiev, Ying Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

Cuxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Xi- angkun Hu, Zheng Zhang, Qian Wang, et al. 2024. Novelqa: Benchmarking question answering on documents exceeding 200k tokens.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. ArXiv preprint, abs/2206.07682.

Wikipedia. 2024. Wikipedia, The Free Encyclopedia. https://www.wikipedia.org/. [Online; accessed 15-September-2024].

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. ArXiv preprint, abs/2304.12244.

Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024. LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K. ArXiv preprint, abs/2402.05136.

Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2022. When language model meets private library. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 277–288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jianzhang Zhang, Yiyang Chen, Chuang Liu, Nan Niu, and Yinglin Wang. 2023. Empirical evaluation of chatgpt on requirements information retrieval under zero-shot setting. 2023 International Conference on Intelligent Computing and Next Generation Networks (ICNGN), pages 1–6.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024a. Infinitebench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15262–15277.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024b. \( \infty \) Bench: Extending Long Context Evaluation Beyond 100K Tokens. ArXiv preprint, abs/2402.13718.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. ArXiv preprint, abs/2303.18223.
A  Details of LONGPiBENCH

A.1 Task Definitions

Table SQL  This task involves retrieving entries containing specific features from a table with a large number of entries. The prototype of this task is primarily derived from experiments in S3Eval (Lei et al., 2023), specifically those examining information distributions with extreme positional variability.

Code Completion  This task involves performing basic programming assignments based on the definitions, signatures, examples, and other information provided in API documentation. The task is considered more challenging than Table SQL tasks because an LLM must not only identify which parts of the API documentation are relevant but also correctly utilize them during coding. The data we use originates from the Private Coding Dataset introduced by Zan et al. (2022). To ensure that the LLM does not rely on internal knowledge about common Python libraries, both the API documentation and task function names have been masked. This privatization process is crucial for evaluating performance on long-text scenarios, as it compels the LLM to extract relevant information directly from the provided context.

Wiki Retrieval  This task involves identifying relevant passages from Wikipedia (Wikipedia, 2024) pages based on a given question. It is a common scenario in which LLMs are used to rerank relevant passages retrieved through information retrieval systems (Ajith et al., 2024).

A.2 Task Examples

Here are some examples of the three tasks in LONG-PiBENCH. Queries are placed both before and after the context for better query contextualization.

A.2.1 Table SQL

Input  You are given a table of entries with the following columns: Country, Name, Birth Year, Birth Month, Blood Type. Your task is to find all the entry with the following Country: China. You should return all the entries that match the query as a python list. For example, ['| China | Hong Liang | 1991 | August | A |', ...]. You should not generate anything else. Here is the table:
| Country | Name | Birth Year | ... | Blood Type |
| Italy | Ginevra | 2009 | February | O |
| Argentina | Martina | 1966 | March | B |
| Egypt | Salma | 1985 | July | B |
...
| China | Zhang Wei | 2006 | November | O |
...
| China | Wang Wei | 1966 | February | AB |
...
| Australia | Emily | 1983 | December | O |
| Italy | Leonardo | 1985 | November | O |

You are given a table of entries with the following columns: Country, Name, Birth Year, Birth Month, Blood Type. Your task is to find all the entry with the following Country: China. You should return all the entries that match the query as a python list. For example, ['| China | Hong Liang | 1991 | August | A |', ...]. You should not generate anything else.

Ground Truth
[
"| China | Zhu Wei | 1992 | September | B |",
"| China | Zhang Wei | 1955 | March | O |",
"| China | Zhang Wei | 2006 | November | O |",
"| China | Wang Wei | 2001 | September | B |",
"| China | Yang Wei | 2016 | November | AB |",
"| China | Li Na | 1974 | January | B |",
"| China | Liu Wei | 1975 | November | O |",
"| China | Gao Wei | 1954 | August | B |",
"| China | Zhu Wei | 1989 | September | AB |",
"| China | Wang Wei | 1966 | February | AB |"
]

A.2.2 Code Completion

Notice that in the Code Completion task, the ground truth is provided in its unmasked form, while the LLMs generate code based on the masked API documentation, resulting in masked code as output.

Input  Please complete the code snippet above according to the provided code snippet and the api doc.
# Text where substitution will take place
text = 'The lib_2 alib_2 123 apples and 456 oranges.'
# Define pattern and replacement for substitution
sub_pattern = r'd+'
d+ lib_2placement = 'NUM'
# Task 1: Substitute matching text using 'sub_pattern' and 'lib_2placement'
lib_2sult_1 = print(lib_2sult_1)
# Task 2: ...
The following context is a code snippet with the detailed api doc.

{
    "api_path": "lib_2.submodule_26",
    "api_doc": "Returns complex...",
    "api_signature": "",
    "api_parameters": "",
    "api_parameters_number": "=0",
    "api_returns": ""
},
...
(more API instances)
Please complete the code snippet above according to the provided code snippet and the api doc.
# Text where substitution will take place
text = 'Thelib_2 alib_2 123 apples and 456 oranges.'
# Define pattern and replacement for substitution
sub_pattern = r'\d+'
d+ lib_2placement = 'NUM'
# Task 1: Substitute matching text using 'sub_pattern' and 'lib_2placement'
lib_2sult_1 = print(lib_2sult_1)
# Task 2: ...

Ground Truth
import re

text = 'There are 123 apples and 456 oranges.'
sub_pattern = r'\d+'
replacement = 'NUM'

## task 1
result_1 = re.sub(sub_pattern, replacement, text)
print(result_1)
...

A.2.3 Wiki Retrieval

Input Please find the top-10 most helpful Docs that will help answer the question. (You do not need to answer it.)
What are ten easy eco-friendly practices that individuals can adopt in their daily lives?
Here is the context
Doc 1
Gaetano JamesSenese (born 6 January 1945) is an Italian saxophonist, composer, and singer-songwriter. Life and career Senese was born in Naples, the son of Anna Senese and James Smith, an American soldier from North Carolina in Italy because of World War II. Senese’s father moved back to the US eighteen months after Gaetano’s birth and never returned. Senese started playing the saxophone at 12 years old.
Doc 2
He made his professional debut in the 1960s, as a member of the rhythm and blues band The Showmen (later known as Showmen 2), with whom he won the 1968 edition of Cantagiro. In 1974 Senese co-founded and led the critically acclaimed jazz-rock group Napoli Centrale. After the group disbanded in 1978, he started a long collaboration with Pino Daniele, both in studio and on stage. His first solo album was released in 1983 by Polydor Records.
......
Doc 1128
Release and critical reception Generations in Song was first released on Coldwater Records in 2001. It was originally offered as a compact disc and contained 19 tracks in its original release. On February 10, 2004, the album was re-released on Slewfoot Records in a compact disc format again. However, only 12 tracks were included on the re-release. The album cover was also changed for the re-release of the project.
Please find the top-10 most helpful Docs that will help answer the question. (You do not need to answer it.)
What are ten easy eco-friendly practices that individuals can adopt in their daily lives?
You should output a python list of the Doc Index like "[["Doc 1", ...]]" as your answer

Ground Truth
[ "Doc 920", "Doc 927", "Doc 935", "Doc 942", "Doc 949", "Doc 957", "Doc 964", "Doc 971", "Doc 979", "Doc 986" ]

B Details of Experimental Setup

B.1 Inference Parameters
To ensure consistency and reproducibility in our experiments, we standardized the inference parameters across all models during the inference phase. Specifically, we set the temperature parameter (temp) to 0.1 and the top-p sampling parameter (top_p) to 0.9. This unification of inference settings facilitates the replication of experiments and establishes a consistent evaluation standard across different models.

B.2 Prompt Template
For the three tasks, we used the following prompt templates respectively. Notice that we place queries both before and after the context body for better query contextualization.

B.2.1 Table SQL

Input You are given a table of entries with the following columns: Country, Name, Birth Year, Birth Month, Blood Type. Your task is to find all the entry with the following Country: {country}. You should return all the entries that match the query as a python list. For example, ['| China | Hong Liang | 1991 | August | A |', ...]. You should not generate anything else. Here is the table:
{context}
You are given a table of entries with the following columns: Country, Name, Birth Year, Birth Month, Blood Type. Your task is to find all the entry with the following Country: {country}. You should return all the entries that match the query as a python list. For example, ['| China | Hong Liang | 1991 | August | A |', ...]. You should not generate anything else.

B.2.2 Code Completion

Input Please complete the code snippet above according to the provided code snippet and the api doc.
{query}
The following context is a code snippet with the detailed api doc.
{context}
Please complete the code snippet above according to the provided code snippet and the api doc.
{query}

B.2.3 Wiki Retrieval

Input Please find the top-10 most helpful Docs that will help answer the question. (You do not need to answer it.)
{query}
Here is the context
{context}
Please find the top-10 most helpful Docs that will help answer the question. (You do not need to answer it.)
{query}
You should output a python list of the Doc Index like "[Doc 1", ...]" as your answer

C Details of Experimental Results

In the main text, for better readability, we only presented the experimental results of a subset of tested LLMs in the form of line charts. Here we present all the experimental results in both tabular and chart form. This will better facilitate the precise display of the experimental results.

Figure 5 and 6 use line charts to illustrate the performance of all selected closed-source and open-source models across the respective test tasks.

Table 1 and Table 2 summarize the performance of all models on the Table SQL task across different absolute and relative positions. Similarly, Table 3 and Table 4 present the results for the Code Completion task, while Table 5 and Table 6 correspond to the Wiki Retrieval task. Finally, Table 7 and Table 8 show the impact of query contextualization.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>97.5</td><td>97.5</td><td>98.0</td><td>92.0</td><td>94.5</td><td>96.5</td><td>96.0</td><td>91.5</td><td>96.0</td><td>97.0</td><td>99.5</td><td>97.5</td><td>97.0</td><td>96.5</td><td>99.0</td><td>98.0</td>
  </tr>
  <tr>
    <td>Deepseek</td><td>100.0</td><td>99.5</td><td>99.5</td><td>97.0</td><td>98.5</td><td>98.5</td><td>97.5</td><td>99.5</td><td>99.0</td><td>95.5</td><td>97.0</td><td>98.0</td><td>99.0</td><td>99.0</td><td>97.5</td><td>100.0</td>
  </tr>
  <tr>
    <td>Gemini</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td>
  </tr>
  <tr>
    <td>GLM</td><td>100.0</td><td>99.0</td><td>94.0</td><td>93.0</td><td>91.5</td><td>91.0</td><td>95.0</td><td>96.5</td><td>93.5</td><td>91.5</td><td>92.0</td><td>89.0</td><td>91.0</td><td>91.0</td><td>94.0</td><td>86.0</td>
  </tr>
  <tr>
    <td>GPT</td><td>100.0</td><td>100.0</td><td>100.0</td><td>100.0</td><td>99.0</td><td>99.5</td><td>99.5</td><td>100.0</td><td>99.5</td><td>100.0</td><td>99.0</td><td>98.5</td><td>99.5</td><td>100.0</td><td>98.5</td><td>96.0</td>
  </tr>
  <tr>
    <td>Llama</td><td>96.0</td><td>96.0</td><td>93.0</td><td>96.0</td><td>91.0</td><td>88.0</td><td>92.0</td><td>92.0</td><td>94.0</td><td>94.0</td><td>89.0</td><td>94.0</td><td>99.0</td><td>99.0</td><td>98.0</td><td>98.0</td>
  </tr>
  <tr>
    <td>Wizard</td><td>85.5</td><td>42.5</td><td>31.5</td><td>20.5</td><td>3.0</td><td>38.0</td><td>36.5</td><td>23.0</td><td>12.0</td><td>13.5</td><td>3.5</td><td>5.0</td><td>1.5</td><td>8.5</td><td>24.5</td><td>90.0</td>
  </tr>
  <tr>
    <td>Qwen 7b</td><td>85.5</td><td>93.5</td><td>98.0</td><td>99.5</td><td>98.5</td><td>93.0</td><td>98.0</td><td>99.5</td><td>96.0</td><td>70.5</td><td>45.0</td><td>70.5</td><td>64.0</td><td>74.0</td><td>81.0</td><td>87.5</td>
  </tr>
  <tr>
    <td>Qwen 14b</td><td>93.5</td><td>90.5</td><td>93.5</td><td>93.5</td><td>93.5</td><td>90.5</td><td>93.5</td><td>96.0</td><td>94.0</td><td>96.0</td><td>96.0</td><td>97.0</td><td>98.0</td><td>96.0</td><td>98.0</td><td>98.5</td>
  </tr>
  <tr>
    <td>Qwen 32b</td><td>98.0</td><td>98.0</td><td>99.0</td><td>99.0</td><td>99.5</td><td>98.0</td><td>98.0</td><td>97.5</td><td>97.0</td><td>98.5</td><td>95.5</td><td>93.9</td><td>96.0</td><td>94.5</td><td>93.5</td><td>93.5</td>
  </tr>
  <tr>
    <td>Qwen 72b</td><td>99.5</td><td>99.5</td><td>98.0</td><td>96.0</td><td>92.5</td><td>93.5</td><td>96.5</td><td>98.0</td><td>99.0</td><td>99.5</td><td>99.5</td><td>100.0</td><td>99.5</td><td>100.0</td><td>100.0</td><td>100.0</td>
  </tr>
</table>

Table 1: Performance of various models across different absolute position levels in Tabel SQL. The model names are abbreviated for better layout. Full names are listed in Section 4.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>100.0</td><td>74.0</td><td>68.0</td><td>65.5</td><td>72.5</td><td>63.0</td><td>71.0</td><td>60.5</td><td>63.0</td><td>61.0</td><td>66.0</td><td>64.0</td><td>65.0</td><td>67.5</td><td>66.5</td><td>64.0</td>
  </tr>
  <tr>
    <td>Deepseek</td><td>100.0</td><td>78.0</td><td>81.0</td><td>82.0</td><td>82.0</td><td>79.5</td><td>69.0</td><td>81.0</td><td>72.0</td><td>79.0</td><td>70.5</td><td>69.0</td><td>75.0</td><td>78.0</td><td>76.0</td><td>81.5</td>
  </tr>
  <tr>
    <td>Gemini</td><td>100.0</td><td>97.5</td><td>89.0</td><td>81.0</td><td>78.0</td><td>78.5</td><td>84.5</td><td>79.0</td><td>79.5</td><td>78.5</td><td>79.5</td><td>74.0</td><td>77.5</td><td>74.0</td><td>75.0</td><td>82.5</td>
  </tr>
  <tr>
    <td>GLM</td><td>90.0</td><td>69.5</td><td>68.5</td><td>67.5</td><td>63.0</td><td>58.0</td><td>65.0</td><td>48.5</td><td>62.0</td><td>50.5</td><td>60.0</td><td>57.5</td><td>61.5</td><td>52.0</td><td>51.5</td><td>44.0</td>
  </tr>
  <tr>
    <td>GPT</td><td>100.0</td><td>74.0</td><td>82.0</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td><td>83.5</td>
  </tr>
  <tr>
    <td>Llama</td><td>100.0</td><td>77.0</td><td>70.0</td><td>75.0</td><td>88.0</td><td>75.0</td><td>79.0</td><td>74.0</td><td>80.0</td><td>80.0</td><td>68.0</td><td>66.0</td><td>75.0</td><td>72.0</td><td>79.0</td><td>69.0</td>
  </tr>
  <tr>
    <td>Wizard</td><td>74.0</td><td>28.5</td><td>23.5</td><td>22.5</td><td>47.5</td><td>47.0</td><td>61.5</td><td>51.5</td><td>54.0</td><td>56.5</td><td>65.5</td><td>61.0</td><td>61.5</td><td>60.0</td><td>61.0</td><td>59.5</td>
  </tr>
  <tr>
    <td>Qwen 7b</td><td>95.0</td><td>39.0</td><td>42.5</td><td>53.0</td><td>61.0</td><td>56.0</td><td>39.0</td><td>48.5</td><td>42.0</td><td>51.5</td><td>36.0</td><td>48.0</td><td>36.0</td><td>40.0</td><td>45.5</td><td>42.5</td>
  </tr>
  <tr>
    <td>Qwen 14b</td><td>99.5</td><td>59.0</td><td>59.0</td><td>63.0</td><td>68.5</td><td>56.0</td><td>58.5</td><td>55.0</td><td>59.5</td><td>59.0</td><td>58.0</td><td>62.5</td><td>59.5</td><td>62.5</td><td>54.0</td><td>63.0</td>
  </tr>
  <tr>
    <td>Qwen 32b</td><td>99.5</td><td>72.0</td><td>69.0</td><td>65.5</td><td>75.5</td><td>68.0</td><td>60.5</td><td>71.0</td><td>64.0</td><td>61.5</td><td>66.0</td><td>64.0</td><td>67.5</td><td>64.0</td><td>64.0</td><td>69.0</td>
  </tr>
  <tr>
    <td>Qwen 72b</td><td>100.0</td><td>81.0</td><td>77.0</td><td>85.0</td><td>87.5</td><td>72.5</td><td>67.5</td><td>63.5</td><td>67.0</td><td>66.0</td><td>67.5</td><td>73.5</td><td>70.0</td><td>75.5</td><td>75.5</td><td>83.0</td>
  </tr>
</table>

Table 2: Performance of various models across different relative position levels in Tabel SQL. The model names are abbreviated for better layout. Full names are listed in Section 4.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Absolute Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>54.28</td><td>55.19</td><td>52.55</td><td>51.44</td><td>62.81</td><td>65.84</td><td>56.86</td><td>46.59</td><td>60.76</td><td>65.84</td><td>56.86</td><td>65.84</td><td>60.41</td><td>62.81</td><td>60.81</td><td>56.40</td>
  </tr>
  <tr>
    <td>Gemini</td><td>83.17</td><td>76.37</td><td>75.59</td><td>73.87</td><td>83.34</td><td>81.81</td><td>83.46</td><td>83.95</td><td>84.55</td><td>83.95</td><td>84.15</td><td>84.78</td><td>82.81</td><td>84.55</td><td>81.94</td><td>80.31</td>
  </tr>
  <tr>
    <td>GPT</td><td>47.64</td><td>57.34</td><td>44.99</td><td>43.00</td><td>44.57</td><td>53.65</td><td>48.37</td><td>50.20</td><td>50.78</td><td>48.66</td><td>47.45</td><td>49.87</td><td>49.37</td><td>54.72</td><td>60.61</td><td>45.37</td>
  </tr>
</table>

Table 3: Performance of various models across different absolute levels in Code Completion. The data includes absolute scores for the Claude, Gemini, and GPT models.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Relative Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>67.50</td><td>57.27</td><td>56.64</td><td>52.69</td><td>53.18</td><td>60.16</td><td>47.24</td><td>43.26</td><td>64.58</td><td>52.96</td><td>47.43</td><td>62.41</td><td>47.01</td><td>51.31</td><td>48.79</td><td>52.96</td>
  </tr>
  <tr>
    <td>Gemini</td><td>83.87</td><td>83.65</td><td>74.71</td><td>84.36</td><td>84.36</td><td>81.37</td><td>84.11</td><td>74.29</td><td>83.09</td><td>75.45</td><td>83.09</td><td>75.59</td><td>84.36</td><td>77.47</td><td>76.45</td><td>72.61</td>
  </tr>
  <tr>
    <td>GPT</td><td>54.57</td><td>51.13</td><td>56.03</td><td>58.11</td><td>52.19</td><td>48.57</td><td>41.59</td><td>51.64</td><td>68.87</td><td>62.38</td><td>44.21</td><td>42.66</td><td>51.66</td><td>45.25</td><td>66.06</td><td>54.83</td>
  </tr>
</table>

Table 4: Performance of various models across different relative levels in Code Completion. The data includes relative scores for the Claude, Gemini, and GPT models.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Absolute Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>96.33</td><td>98.33</td><td>99.00</td><td>95.00</td><td>93.33</td><td>95.00</td><td>96.00</td><td>95.67</td><td>97.67</td><td>97.00</td><td>99.00</td><td>94.33</td><td>93.00</td><td>91.33</td><td>92.33</td><td>92.00</td>
  </tr>
  <tr>
    <td>Gemini</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>97.00</td><td>98.00</td><td>97.67</td><td>98.00</td><td>98.00</td><td>97.33</td><td>98.00</td><td>98.00</td><td>95.33</td><td>98.00</td><td>96.00</td>
  </tr>
  <tr>
    <td>GPT</td><td>100.00</td><td>99.00</td><td>100.00</td><td>98.00</td><td>98.00</td><td>100.00</td><td>99.00</td><td>100.00</td><td>96.00</td><td>97.00</td><td>98.00</td><td>99.00</td><td>99.00</td><td>98.00</td><td>96.00</td><td>96.00</td>
  </tr>
</table>

Table 5: Performance of various models across different absolute levels in Wiki Retrieval. The data includes absolute scores for the Claude, Gemini, and GPT models.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="16">Relative Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th><th>Lv. 2</th><th>Lv. 3</th><th>Lv. 4</th><th>Lv. 5</th><th>Lv. 6</th><th>Lv. 7</th><th>Lv. 8</th><th>Lv. 9</th><th>Lv. 10</th><th>Lv. 11</th><th>Lv. 12</th><th>Lv. 13</th><th>Lv. 14</th><th>Lv. 15</th><th>Lv. 16</th>
  </tr>
  <tr>
    <td>Claude</td><td>100.00</td><td>96.00</td><td>99.67</td><td>96.00</td><td>98.00</td><td>96.33</td><td>95.00</td><td>95.67</td><td>93.00</td><td>92.67</td><td>91.00</td><td>93.00</td><td>95.33</td><td>92.00</td><td>96.67</td><td>96.33</td>
  </tr>
  <tr>
    <td>Gemini</td><td>99.67</td><td>98.00</td><td>95.33</td><td>92.00</td><td>93.67</td><td>93.33</td><td>94.33</td><td>85.67</td><td>94.00</td><td>91.00</td><td>93.00</td><td>94.00</td><td>91.67</td><td>95.00</td><td>96.00</td><td>96.00</td>
  </tr>
  <tr>
    <td>GPT</td><td>100.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>100.00</td><td>98.00</td><td>100.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td><td>98.00</td>
  </tr>
</table>

Table 6: Performance of various models across different relative levels in Wiki Retrieval. The data includes relative scores for the Claude, Gemini, and GPT models.
Figure 5: The impact of relevant information’s absolute and relative position for all open-source commercial models. A higher absolute position feature level indicates locations closer to the end of input, while a higher relative position feature level indicates a greater distance between relevant pieces of information.

Figure 6: The impact of relevant information’s absolute and relative position for all tested commercial models. A higher absolute position feature level indicates locations closer to the end of input, while a higher relative position feature level indicates a greater distance between relevant pieces of information.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Query Position</th>
    <th colspan="16">Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th>
    <th>Lv. 2</th>
    <th>Lv. 3</th>
    <th>Lv. 4</th>
    <th>Lv. 5</th>
    <th>Lv. 6</th>
    <th>Lv. 7</th>
    <th>Lv. 8</th>
    <th>Lv. 9</th>
    <th>Lv. 10</th>
    <th>Lv. 11</th>
    <th>Lv. 12</th>
    <th>Lv. 13</th>
    <th>Lv. 14</th>
    <th>Lv. 15</th>
    <th>Lv. 16</th>
  </tr>
  <tr>
    <td rowspan="3">GPT</td>
    <td>Head</td>
    <td>100.0</td><td>90.5</td><td>85.0</td><td>89.5</td><td>98.0</td><td>99.5</td><td>95.0</td><td>100.0</td><td>90.0</td><td>95.0</td><td>89.0</td><td>83.0</td><td>100.0</td><td>83.0</td><td>69.0</td><td>86.5</td>
  </tr>
  <tr>
    <td>Tail</td>
    <td>100.0</td><td>80.5</td><td>36.0</td><td>47.0</td><td>68.0</td><td>81.5</td><td>84.5</td><td>70.5</td><td>81.5</td><td>79.5</td><td>60.0</td><td>68.0</td><td>72.5</td><td>67.0</td><td>83.0</td><td>83.0</td>
  </tr>
  <tr>
    <td>Both</td>
    <td>100.0</td><td>80.0</td><td>100.0</td><td>100.0</td><td>90.5</td><td>95.5</td><td>90.0</td><td>95.5</td><td>90.0</td><td>95.5</td><td>85.0</td><td>85.0</td><td>100.0</td><td>88.5</td><td>86.0</td><td>86.0</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen 14B</td>
    <td>Head</td>
    <td>93.5</td><td>84.5</td><td>91.0</td><td>96.0</td><td>97.5</td><td>90.5</td><td>96.5</td><td>97.5</td><td>95.0</td><td>93.5</td><td>94.0</td><td>95.0</td><td>96.5</td><td>98.5</td><td>98.5</td><td>97.5</td>
  </tr>
  <tr>
    <td>Tail</td>
    <td>82.5</td><td>57.0</td><td>72.5</td><td>88.5</td><td>88.0</td><td>79.0</td><td>86.0</td><td>77.5</td><td>89.5</td><td>90.0</td><td>88.0</td><td>89.5</td><td>92.5</td><td>96.5</td><td>95.0</td><td>97.5</td>
  </tr>
  <tr>
    <td>Both</td>
    <td>93.5</td><td>80.5</td><td>93.0</td><td>93.5</td><td>98.5</td><td>93.0</td><td>93.5</td><td>98.0</td><td>96.0</td><td>94.5</td><td>96.0</td><td>96.0</td><td>97.0</td><td>98.5</td><td>96.0</td><td>98.5</td>
  </tr>
</table>

Table 7: Performance of GPT-4o-mini (OpenAI, 2024) and Qwen-2.5 14B (Qwen, 2024) across different absolute position levels with varying placement of the query. The query position can be at the head, tail, or both positions in the input.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Query Position</th>
    <th colspan="16">Performance / %</th>
  </tr>
  <tr>
    <th>Lv. 1</th>
    <th>Lv. 2</th>
    <th>Lv. 3</th>
    <th>Lv. 4</th>
    <th>Lv. 5</th>
    <th>Lv. 6</th>
    <th>Lv. 7</th>
    <th>Lv. 8</th>
    <th>Lv. 9</th>
    <th>Lv. 10</th>
    <th>Lv. 11</th>
    <th>Lv. 12</th>
    <th>Lv. 13</th>
    <th>Lv. 14</th>
    <th>Lv. 15</th>
    <th>Lv. 16</th>
  </tr>
  <tr>
    <td rowspan="3">GPT</td>
    <td>Head</td>
    <td>95.0</td><td>60.0</td><td>68.5</td><td>69.5</td><td>60.5</td><td>63.0</td><td>64.0</td><td>75.5</td><td>54.5</td><td>63.5</td><td>66.0</td><td>62.0</td><td>67.0</td><td>46.5</td><td>60.0</td><td>79.0</td>
  </tr>
  <tr>
    <td>Tail</td>
    <td>94.0</td><td>67.5</td><td>66.5</td><td>55.5</td><td>40.5</td><td>50.0</td><td>69.5</td><td>58.0</td><td>52.5</td><td>49.0</td><td>55.0</td><td>51.0</td><td>52.5</td><td>58.0</td><td>49.0</td><td>68.0</td>
  </tr>
  <tr>
    <td>Both</td>
    <td>100.0</td><td>84.5</td><td>86.5</td><td>82.5</td><td>70.5</td><td>74.0</td><td>86.5</td><td>80.0</td><td>83.0</td><td>76.5</td><td>81.5</td><td>78.0</td><td>78.5</td><td>77.0</td><td>73.5</td><td>80.0</td>
  </tr>
  <tr>
    <td rowspan="3">Qwen 14b</td>
    <td>Head</td>
    <td>95.0</td><td>60.0</td><td>68.5</td><td>69.5</td><td>60.5</td><td>63.0</td><td>64.0</td><td>75.5</td><td>54.5</td><td>63.5</td><td>66.0</td><td>62.0</td><td>67.0</td><td>46.5</td><td>60.0</td><td>79.0</td>
  </tr>
  <tr>
    <td>Tail</td>
    <td>94.0</td><td>67.5</td><td>60.0</td><td>55.5</td><td>40.5</td><td>50.0</td><td>69.5</td><td>58.0</td><td>52.5</td><td>49.0</td><td>55.0</td><td>51.0</td><td>52.5</td><td>58.0</td><td>49.0</td><td>68.0</td>
  </tr>
  <tr>
    <td>Both</td>
    <td>100.0</td><td>84.5</td><td>86.5</td><td>82.5</td><td>70.5</td><td>74.0</td><td>86.5</td><td>80.0</td><td>83.0</td><td>76.5</td><td>81.5</td><td>78.0</td><td>78.5</td><td>77.0</td><td>73.5</td><td>80.0</td>
  </tr>
</table>

Table 8: Performance of GPT and Qwen 14b across different relative levels with varying placement of the query. The query position can be at the head, tail, or both positions in the input.

Reviewer paper summaries: This paper addresses the position bias in LLMs by proposing a new evaluation benchmark, LongPiBench. It emphasizes the distribution of relevant information in the context, including sparse and dense scenarios. LongPiBench covers tasks such as Table SQL, Code Completion, Wiki Retrieval, with contexts ranging from 32K to 256K tokens.

The benchmark is evaluated on models including Llama-3.1, Qwen2.5, WizardLLM, Gemini-1.5-Flash, Claude-3-Haiku, and GPT-4o-mini. Results indicate a significant performance drop as relative positions increase.

This paper asks the direct and simple question of what is the impact of the spread and location of the atoms of info in the context on the performance, when the model is required to reason over those atoms of info.
The paper introduces a benchmark that is composed of 3 tasks of different types: Coding, info processing and sql.
Summary of strengths: 1. The research problem is intriguing and approached from a unique angle.
2. The work has meaningful implications and practical application value.
3. Extensive experiments validate the hypotheses and reveal new phenomena related to position bias from a novel perspective.

The question is direct, simple and explained well.

The collection of the 3 tasks include a good variety of scenarios that are interesting to test the impact of the locations of info in.

The construction of the placement of the info atoms is clear.

The results are interesting although the tasks dont really require the models to reason over the atoms of info (see weaknesses).
Summary of weaknesses: - The conclusions are straightforward but lack applications to more realistic scenarios.

1. The explanation of the tasks themselves is not clear enough.

2. I think that an important requirement for the formulation of the tasks in such evaluation is that the required info will be gathered from multiple places in the input **at once**. Meaning that to get the right answer the model would need to collect info from multiple places in one forward:

The SQL task does not require this reasoning at all. Moreover, each forward requires just one atom of info from the context.

The wiki require it to only some degree. The task is to order items, to do so the model is required to decide on each forward what is the next item in the ordered list. As I understood from the explanation of the task in the appendix: the paragraphs are collected from several articles, however, it is not mentioned if the correct ranking has multiple ranks beyond relevant and non relevant. For example it might be that the only "computation" needed on each forward is to find any relevant paragraph and not compare between the paragraphs relevance.

and for the coding task I think it is sample dependent. (It is not clear from the explanation of the task)
Comments, suggestions and typos: N/A

I would suggest to place the explanation of the task early in the paper (instead of in the appendix), I would shorten section 2 or place some of it in the appendix to create space.

The title of section 2.3 does not fit its content, id call it "Versions of samples" (or something better).
Overall assessment scores: 3

2.5
Metareview score: 3
Metareview: This paper introduces LongPiBench, a new benchmark designed to assess positional bias in large language models (LLMs) when processing long inputs containing multiple pieces of relevant information. The benchmark includes various tasks (Table SQL, Code Completion, Wiki Retrieval) and input lengths (32K to 256K tokens). Experiments were conducted with both commercial and open-source models, revealing that while most current models are more robust against the "lost in the middle" issue, there are still noticeable biases related to the spacing of relevant information pieces. The work aims to highlight the importance of evaluating and reducing positional biases for long-context LLMs.
Summary of reasons to publish: - The benchmark addresses an important and novel aspect of LLM evaluation by focusing on multiple pieces of relevant information rather than just single pieces.

- The experimental setup is well-designed, with a good variety of tasks that have real-world applications.

- The results provide interesting insights into the performance of current LLMs with respect to positional bias in long contexts.