LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024)

Abstract: Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs’ long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including singledoc QA, multi-doc QA, summarization, fewshot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other opensourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability.

heading: 1 Introduction
text: The field of NLP has long sought to endow machines with the ability to understand and reason
†Corresponding authors
over a long context. Tasks such as summarization and question answering based on books, reports, and documents, and code generation at the repository level demand the ability to model long context sequences that span thousands or even tens of thousands of tokens in length. However, many of today’s large language models can only comprehend and generate texts a few thousand tokens long, leaving room for potential improvements in processing longer contexts. More recently, there has been an increasing effort to improve large language models’ capabilities on long context understanding. These methods include extending the context window (Press et al., 2022; Chen et al., 2023), utilizing recurrent memory (Dai et al., 2019; Bulatov et al., 2023), using sparsed attention (Ding et al., 2023; Mohtashami and Jaggi, 2023), and augmenting with an external memory (Liang et al., 2023; Zhou et al., 2023). However, unlike in short context, where a multitude of multi-task benchmarks are available for a multi-aspect evaluation (Hendrycks et al., 2021; Srivastava et al., 2023), there is no such benchmark on longer context.
To facilitate further research in this direction, we propose LongBench, the first bilingual, multitask benchmark tailored for long context understanding. LongBench is composed of 6 major task categories and 21 different tasks, covering key long-text application scenarios including multidocument QA, single-document QA, summarization, few-shot learning, code completion, and synthetic tasks. In addition, LongBench includes different languages (Chinese and English) to provide a more comprehensive evaluation of the large models’ bilingual capabilities on long contexts. With overview statistics shown in Figure 1, LongBench contains 4,750 test instances, with an average length of 6,711 words and 13,386 characters for English and Chinese instances, respectively.
All 21 datasets in LongBench are standardized into a unified format, among which 6 are directly
3119
extracted from the original datasets provided by previous studies, 10 are built based on the original datasets and processed to be suitable for long context evaluation, and 5 are created and annotated by us. We are fully aware of the potentially high costs involved in the model evaluation process, especially in the context of long context scenarios (such as manual annotation costs or API call costs). Therefore, we adopt a fully automated evaluation method, where we utilize automatic metrics such as ROUGE-L and F1 to measure the similarity of the outputs to the groundtruth answers.
We conduct a comprehensive evaluation of 8 models on LongBench. The empirical results provide insightful conclusions about the multi-task capability of current models in terms of long context comprehension. Additionally, to better disentangle the models’ long context ability from their task ability, we construct LongBench-E that features a more even length distribution, thus suited for gauging each model’s capability across various context lengths. Results on LongBench-E reveal that although some models are trained or fine-tuned on longer contexts, they still experience a significant decline in performance as the context length increases. We also investigate the effect of retrievalbased and summarization-based context compression techniques. Our results demonstrate that these approaches are beneficial only to models that exhibit weaker capability on long contexts.

heading: 2 Related Work
text: Long Context Modeling Techniques. We first discuss some popular lines of methods that aim to tackle long context understanding. These studies are mainly aimed at solving two key challenges
in long text modeling, including the high runtime overhead on longer context, and the catastrophic forgetting phenomenon when processing long sequence. A series of studies focus on how to make Transformers more efficient and unforgetful (Tay et al., 2022), with designs such as sparse and efficient computation (Child et al., 2019; Kitaev et al., 2020; Beltagy et al., 2020; Zaheer et al., 2020; Wang et al., 2020; Fedus et al., 2022; Ding et al., 2023), recurrent and memory modules (Dai et al., 2019; Rae et al., 2020; Wu et al., 2022; Martins et al., 2022; Bulatov et al., 2022; Orvieto et al., 2023; Liang et al., 2023; Zhou et al., 2023). More recently, several methods (Press et al., 2022; Sun et al., 2022; Chen et al., 2023) have been proposed to enable length extrapolation of Transformers, and have been adopted in the training process of long context LLMs such as ChatGLM2-32k (Zeng et al., 2023) and LongChat-32k (Li et al., 2023). Evaluation for Long Context Understanding. Many previous works on long text modeling rely on the perplexity metric for evaluation (Beltagy et al., 2020; Roy et al., 2021; Press et al., 2022). However, as suggested in (Sun et al., 2021), the perplexity metric may not necessarily reflect the model’s performance on sequence-level tasks in real applications Meanwhile, some works assess long text modeling through artificial tasks such as retrieval (Tay et al., 2021; Chen et al., 2023; Li et al., 2023), which may also fall short in mirroring real-world scenarios.
Concurrently, ZeroSCROLLS (Shaham et al., 2022, 2023) and L-Eval (An et al., 2023) are proposed as evaluation benchmarks for long text modeling. Nonetheless, they encompass a restricted range of task types, thereby limiting the diversity of long text modeling patterns required in the bench-
marks, and consequently, the comprehensiveness of the evaluation results. Recently, AgentBench (Liu et al., 2023c) also mentions the challenge of LLMas-Agent’s handling long interaction trajectories, but fails to incorporate it as a dedicated evaluation dimension. In contrast, LongBench includes six major task categories, with each category featuring sequences of varying lengths, languages, and domains. We believe it provides a more holistic evaluation of long text modeling ability of large language models across a spectrum of lengths, distributions, as well as long dependency patterns.

heading: 3 LongBench: Task and Construction

heading: 3.1 Problem Definition
text: We formalize the problem of long context understanding as follows: Given the input and context sequences (I, C), the model is expected to output the answer A. For instance, in a QA task, the input I would be the question, context C refers to the document, and A denotes the answer to the question. Generally, in LongBench, I and A tend to be short, while C represents a long sequence up to thousands of tokens in length. The instantiation of (I, C,A) for each task is listed in Table 7.

heading: 3.2 Dataset Construction
text: In this section, we will provide a detailed introduction to the data collection, annotation, and organization process for each dataset in LongBench (LongBench-E), according to the specific tasks. For the overall data statistics of LongBench, we refer to Table 1.

heading: 3.2.1 Data Collection and Annotation
text: Single-Doc QA. For single-document QA, we focus on instances with longer documents. We extract NarrativeQA from the original dataset in Kočiskỳ et al. (2018), which consists of long stories along with questions posed to test reading comprehension. We also sample from Qasper (Dasigi et al., 2021), which features QA over NLP papers and is annotated by NLP practitioners.
To better test the model’s long context understanding ability across diverse fields, we manually curate the MultiFieldQA datasets in both English and Chinese. We first collect documents and articles from multiple sources, including legal documents, government reports, encyclopedias, academic papers, etc (sources detailed in Appendix). We invite three Ph.D. students to annotate the ques-
tion and answer for each article, with definitive answers as much as possible for ease of automated evaluation. During the annotation, we ensure that the answers can be inferred from the documents, and the placement of evidence is fairly random to avoid biases that might occur if, for instance, the answer-related statements are frequently found at the beginning or the end, as mentioned in (Liu et al., 2023a).
Multi-Doc QA. Multi-document QA requires models to extract and combine information from several documents to obtain the answer, which is usually more challenging than single-doc QA. The English test samples are built from three Wikipedia-based multi-hop QA datasets: HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), and MuSiQue (Trivedi et al., 2022). HotpotQA involves a number of 2-hop questions directly written by native speakers given two related paragraphs. 2WikiMultihopQA consists of up to 5-hop questions that are synthesized through manually designed templates to ensure that they cannot be solved through shortcuts. The questions in MuSiQue are carefully composed from simple questions involving up to 4-hop reasoning, and are then paraphrased by annotators to both avoid shortcuts and ensure linguistic naturalness. Each question in the original datasets is supplemented by 2-4 supporting paragraphs that provide one-step reasoning evidence and several distracting paragraphs.
To tailor the data for long-context evaluation, we utilize complete Wikipedia passages encompassing the supporting or distracting paragraphs as the context. Initially, supporting passages are included within the context, and then as many distracting passages are added until the total length reaches a maximum length. Finally, these passages are randomly ordered to form the multi-document context.
Beyond these three English datasets, we also construct a Chinese dataset based on DuReader (He et al., 2018), which is developed based on Baidu Search and Baidu Zhidao, comprising 200K questions and 1M related documents. To adapt it for assessing long context ability, for each question, we not only provide several documents related to the question but also arbitrarily select several from the total set of documents as distractors, until each question is associated with 20 documents.
Summarization. Compared to QA tasks, which can often be solved using local information within the context, summarization demands a more global
understanding of the whole context. We extract GovReport from the original dataset (Huang et al., 2021). The original GovReport dataset is a largescale collection of detailed reports from the U.S. Government Accountability Office and Congressional Research Service, each accompanied by a human-written summary, spanning a wide variety of national policy issues. We also sampled from QMSum (Zhong et al., 2021), which consists of query-summary pairs annotated over 232 meetings across multiple domains, including product, academic, and committee meetings. We treat the query as input I , the meeting content as context C, and the summary as answer A. MultiNews is derived from the original multi-document summarization dataset in (Fabbri et al., 2019). The MultiNews dataset features clusters of 2-10 news articles discussing the same event or topic, each paired with a human-written summary that summarizes the key information from the multiple source articles. In LongBench, we include “Document i” before the ith news article, and concatenate them into the context C. VCSUM (Wu et al., 2023) is a large-scale Chinese meeting summarization dataset consisting of 239 real-life meetings with over 230 hours of duration, with versatile annotations to support multiple summarization tasks. In LongBench, we select the long segments from VCSUM to compose our evaluation samples. Few-shot Learning. We identify few-shot incontext learning as a practical setting that requires long context understanding, especially when the number of examples increases (Ainslie et al., 2023). To ensure the diversity of the tasks, we incorporate classification, summarization, and reading comprehension tasks within the few-shot learning scenario. We include 2 classification datasets with fine-grained class labels, including TREC (Li
and Roth, 2002), a question classification task involving 50 fine classes, and LSHT (NLPCC, 2014), a Chinese news classification task with 24 classes. For summarization task, we use the SAMSum dataset (Gliwa et al., 2019), which contains messenger-like conversations with humanannotated summaries. TriviaQA (Joshi et al., 2017) contains question-answer pairs labeled with evidence passages, and we use it as a reading comprehension task. We filter the passages in TriviaQA with less than 1,000 words to be examples.
In each of the above datasets adapted for LongBench, for each test data, we first randomly select an integer within a range as the number of examples, then randomly sample the corresponding number of samples from the training set, and concatenate them to form the context C. For TREC, LSHT, SAMSum, and TriviaQA, the ranges are [100, 600], [10, 40], [10, 100], [2, 24], respectively.
Synthetic Task. Unlike standard tasks that are more alike on the required long dependency pattern, synthetic tasks can be meticulously designed to test the model’s ability on specific scenarios and patterns. In LongBench, we design three synthetic tasks. PassageRetrieval-en and PassageRetrievalzh are constructed based on English Wikipedia and the Chinese sections of the C4 dataset (Raffel et al., 2020). For each data entry, we randomly sample 30 passages and select one of them for summarization using GPT-3.5-Turbo. The task asks the model to identify the original paragraph to which the crafted summary corresponds.
PassageCount seeks to create a more demanding situation where the model is required to utilize the full context to resolve the task. For each piece of data, we randomly select several passages from English Wikipedia, repeat each paragraph at random several times, and finally shuffle the paragraphs. The task asks the model to determine the number of unique passages among the given set. Specifically, we randomly select M from [17, 50] as the upper limit for the number of passages. Subsequently, the number N of unique passages is randomly selected from the range [2,M ]. We conduct random sampling with replacement from the set of N unique passages to get the final M passages.
Code Completion. Code completion is a critical task employed by auto-completion systems to assist users by completing code based on previous code input and context (Chen et al., 2021; Zheng et al., 2023b). This task can pose a significant challenge
for models, especially when dealing with lengthy code inputs or even repository-level data. This is mainly because the models need to establish attention across long-range sequences according to relationships within code elements, such as between class and function definitions. Hence we recognize this as a suitable task for evaluating a model’s long context modeling ability.
The LCC dataset is sampled from the original Long Code Completion dataset (Guo et al., 2023). The original dataset is constructed by filtering code within one file from GitHub based on length. This data includes a long piece of preceding lines of code as context, and the next line of code as the answer. We also consider the repository-level code completion setting, which necessitates aggregating information from code across files. For this task, we adapt the RepoBench-P dataset from (Liu et al., 2023b). RepoBench-P is collected from Github repositories, and is constructed by first retrieving relevant code snippets from other files based on module import statements. These snippets are then concatenated with the preceding lines of code within the current file as context, and are used to predict the next line of code. We select the most challenging XF-F (Cross-File-First) setting from the original dataset, where the in-file context gives no prior usage of the module to aid the prediction. For each original piece of data, we shuffle the crossfile code snippets that include the gold cross-file code snippet (manually annotated as the optimal context for prediction), and combine them into context C. The preceding lines of code are taken as input I , and the next line of code as the answer A.

heading: 3.2.2 Data Extraction
text: Since LLMs may have already been trained on the training set of some of our collected public datasets, to avoid test leakage, we extract data from the test sets of these public datasets, with the exception of VCSUM due to its insufficient data in its test set. We employ two extraction strategies: random sampling and uniform sampling. Through random sampling, we maintain a natural length distribution to more accurately mimic real scenarios, and obtain LongBench. Alternatively, we perform uniform sampling based on the length of the data with a focus on studying the model’s capabilities across varying context lengths within each task itself. This approach provides insights into the model’s true ability to understand long contexts independent of task capability. We
choose 13 of the English datasets, including Qasper, MultiFieldQA-en, HotpotQA, 2WikiMultihopQA, GovReport, Multi-news, TREC, TriviaQA, SAMSum, PassageCount, PassageRetrieval-en, LCC, and RepoBench-P, which offer broader coverage on data length. During the uniform sampling process, we use word count as the length and sample a comparable quantity of data from the length ranges of 0-4k, 4k-8k, and 8k+. The resulting data is compiled into LongBench-E (statistics in Table 8).

heading: 4 Experiments

heading: 4.1 Benchmarking Results on LongBench and LongBench-E
text: Experiment Setup. We evaluate 8 popular LLMs that feature long context capability, including GPT3.5-Turbo-16k (OpenAI, 2022a), Llama2-7B-chat4k (Touvron et al., 2023), LongChat-v1.5-7B32k (Li et al., 2023), XGen-7B-8k (Nijkamp et al., 2023), InternLM-7B-8k (Team, 2023), ChatGLM26B, ChatGLM2-6B-32k (Du et al., 2022; Zeng et al., 2023), and Vicuna-v1.5-7B-16k (Zheng et al., 2023a). ChatGLM2-6B-32k is trained based on ChatGLM2-6B, with a 32k context length during alignment and position interpolation (Chen et al., 2023). LongChat-v1.5-7B-32k and Vicuna-v1.5-
7B-16k are fine-tuned from Llama2-7B, with supervised fine-tuning and linear RoPE scaling.
We conduct the assessment in a zero-shot setting, except for the few-shot learning tasks where the few-shot examples are provided as part of the long context. The input format prompt and the maximum output length we used during evaluation can be found in Appendix. When the input length L surpasses the maximum context length M of a model (indicated by the suffix of its name), we truncate the input sequence S from the middle since the front and end of the sequence may contain crucial information such as the instruction or question: S1:L → [S1:⌊M/2⌋;SL−⌊M/2⌋−1:L]. During generation, we use greedy decoding for reproducibility. It’s worth noting that the chat models evaluated typically have specific prompts that induce the models to generate dialogue-like responses. During evaluation, we avoid adding these prompts in fewshot learning and code completion tasks, since the answers to these tasks should be generated in a completion style rather than a chat style.
The metric for each dataset is shown in Table 1. For tasks built based on previous datasets, the metrics we used are consistent with those used in the original work. F1 and ROUGE-L (Lin, 2004) are
two popular N-gram based metrics widely adopted in QA and summarization tasks. Edit Sim (Levenshtein distance) is popularly used in code generation evaluation (Svyatkovskiy et al., 2020). For the few-shot learning tasks, we extract the first line of the response. For the two code completion tasks, we extract the first line of model generation that is not comment. The code and datasets are available at https://github.com/THUDM/ LongBench. Results on LongBench. Table 2, 3 report the performance (%) on all datasets in LongBench. Additionally, Figure 4 presents a radar plot depicting the models’ abilities on the 6 major tasks. For better visualization, we scale the maximum score across all models on each task to 100 in the radar plot. We summarize our key findings from the experiment results: (1) There is still a performance gap on long context tasks between open-sourced models of smaller size and commercial model (GPT-3.5-Turbo-16k). (2) Models benefit from scaled positional embedding and continued training on longer context, as ChatGLM2-6B32k and LongChat-v1.5-7B-32k obtain relative improvements of 62% and 19%, respectively. We further analyze the multi-task property of LongBench by the inter-task correlation among and across each category of tasks in Appendix D. We find higher correlations for performance on tasks of the same category or language.
To study whether the models with a longer maximum length truly benefit from utilizing longer context, we conduct experiments with GPT-Turbo3.5-16k, ChatGLM2-6B-32k, and Vicuna-v1.5-7B16k with truncation sizes of 4k and 8k on LongBench. The macro-average scores across all tasks with varying truncation sizes are depicted in Figure 2. Here, ‘maximum length’ denotes truncation
at the model’s maximum length configuration. We observe that GPT-Turbo-3.5-16k and ChatGLM26B-32k obtain higher scores under a larger truncation size, suggesting they can better make use of a longer context. Furthermore, this confirms that our benchmark indeed necessitates long context modeling — using truncated information alone is insufficient for successfully completing the tasks in LongBench. On the other hand, the performance of LLMs on LongBench can be further improved by enhancing their long context modeling capabilities. Results on LongBench-E. While LongBench facilitates the measurement of an overall multi-task ability on tasks that require long context understanding, LongBench-E focuses more on measuring how the model’s performance changes under different context lengths within the same task. As introduced in Sec 3.2.2, LongBench-E contains a subset of datasets included in LongBench, featuring more evenly distributed context lengths. Figure 3 reports the macro-average scores (%) on data in length ranges of 0-4k, 4k-8k, and 8k+ (See the results on all datasets in Table 9). One can derive a model’s long context ability from the slope of the curve — a significant drop in performance on data of greater length, as indicated by a steeper curve, points to the model’s limitations in effectively handling long text modeling. From the results on LongBench-E, we observe that ChatGLM2-6B-32k and LongChatv1.5-7B-32k are more robust to longer context length, with a relative drop of 4% and 7% from 0-4k to 8k+ respectively. Moreover, despite GPT3.5-Turbo-16k demonstrating impressive overall performance across all tasks, we find that it still struggles on longer contexts (-17% from 0-4k to 8k+), leaving room for future development on long
context modeling.

heading: 4.2 Impact of Context Compression Techniques
text: We further explore the impact of context compression techniques on LongBench, including retrievalbased context compression and summarizationbased context compression. Retrieval is widely used in augmenting language models with external memory (Khandelwal et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022b). This application can be extended to consider longer contexts, such as documents or books, as forms of external memory, from which relevant information can be retrieved using a specific query. Given a long context, we first split it into chunks with a default size of M words (or characters on Chinese datasets), then use
a specific retriever to compute the embedding of the text chunks and the query, and concatenate only the top-N chunks according to the cosine similarity of their embeddings to the query embedding. The topN chunks as the compressed context, together with the query, are then fed into the model to produce an answer. A similar pipeline is also implemented in LangChain. We experiment with three retrievers — OpenAI Embedding (text-embedding-ada002 (OpenAI, 2022b)), Contriever (Izacard et al., 2022a), and BM25 — alongside two chunk sizes of 200 and 500. In order to conduct a fairer comparison under the same context length, we take the top-7 and top-3 chunks respectively when the chunk sizes are 200 and 500. Table 4 reports the results on QA tasks in LongBench. We summarize our findings: (1) text-embedding-ada-002 performs the best among the three retrievers, while the open-sourced Contriever results are closer to text-embedding-ada-002 and superior to BM25. (2) In general, splitting the long context into shorter chunks and retrieving more chunks results in better performance. (3) Under the best retrieval method, the improvements for the three models are -2%, 21%, and -5%, respectively. Moreover, even after retrieval, the performance of Llama2-7B-chat-4k still lags behind the other two models. The results suggest that the retrieval technique can only serve as a performance compensation for models that cannot well model long context, and is not a shortcut to solving long context understanding tasks.
We also study the effect of using modelgenerated summary as a context compression technique. Specifically, we first utilize the model to generate a brief summary for each text chunk, and concatenate the summaries together as the compressed context. We experiment on the summarization tasks in LongBench, and the results are as shown in Table 5. We find that this compression method improves the performance of the models only on the VCSUM task (3-4), since the data in VCSUM are longer than in the other three datasets.

heading: 4.3 Context Understanding or Memorization?
text: Since the model may have potentially encountered the long context during pre-training, it may rely on memorization rather than context understanding to answer the question. We conduct an experiment to assess the degree to which these tasks rely on memorization rather than long context understanding capability. Specifically, we withhold the long
context from the model, posing only questions to it, and evaluate its performance. Table 6 shows the results for GPT-3.5-Turbo-16k, Llama2-7B-chat4k and ChatGLM2-6B-32k on the Single-Doc QA and Multi-Doc QA datasets in LongBench.
We observe that the memorization performance (w/o context) on HotpotQA, 2WikiMultihopQA, and MusiQue is relatively high. This is likely because these datasets are derived from Wikipedia, a common source for training prevalent LLMs. The ∆ score (original score minus the score when context is absent) addresses the memorization phenomenon (Yu et al., 2024), and could also serve as an important indicator for a model’s long context understanding ability.

heading: 5 Conclusion
text: In this paper, we introduce LongBench, a multitask bilingual benchmark tailored for gauging long context understanding abilities of LLMs. LongBench covers six key categories and a total of 21 tasks, with data lengths extending from thousands of tokens up to tens of thousands of tokens. We also develop LongBench-E which features a more evenly data length distribution. We conduct extensive experiments on LongBench and LongBench-E, yielding insightful conclusions about the capabilities of current LLMs on long context understanding. Moreover, our analysis suggests that LongBench and LongBench-E serve as ideal testbeds for future research in long context modeling.

heading: 6 Limitations
text: Despite LongBench offers a more comprehensive testbed for long context understanding, it still has its shortcomings, as we summarized below. (1) Potentially unreliable automatic metrics: As previous studies suggest (Bai et al., 2023), the automatic evaluation metrics (ROUGE-L, F1) may not accurately reflect the quality of the response. Particularly, the results on these metrics may be underestimated for models that are used to generating longer responses. Although using LLM as
examiner may reduce this problem (Bai et al., 2023; An et al., 2023), the runtime overhead for evaluation may be high, and LLM also has bias when used as an evaluation metric (Zheng et al., 2023a). (2) Coupling with instruction-following capabilities: Our primary objective is to assess the models’ proficiency in long-context modeling regardless of their instruction-following capabilities. However, as the tasks in LongBench are closer to real-world applications, mastering them inevitably demands a certain level of instruction-following capability. Consequently, the performance on LongBench is coupled with the models’ instruction-following capabilities.

heading: Acknowledgement
text: This work is supported by a grant from the Institute for Guo Qiang, Tsinghua University (2019GQB0003), Zhipu AI and Tsinghua University Initiative Scientific Research Program. We would also like to thank our anonymous reviewers for their suggestions.

heading: A Dataset Details
text: Table 7 lists the instantiation of (I, C,A) for each dataset in LongBench. Table 8 reports the number of data on each task that falls in the length range of 0-4k, 4k-8k, and 8k+ in LongBench-E.
Details of MultiFieldQA document sources. The source of the documents in MultiFieldQA include:
• Arxiv (for academic papers): open-accessed and can be downloaded freely by anyone.
• C4 Dataset: publicly available dataset with ODC-BY license.
• WuDaoCorpora: open-accessed dataset.
• Chinese Judgements Online (for Chinese legal documents): open Chinese judgements download website.
• Wikipedia (for encyclopedias): grant free access and licensed under CC BY-SA.
• Chinese Government Website (for Chinese government report): open Chinese government report download website.
Annotation guidelines for MultiFieldQA. Here we provide the annotation guidelines: “Please propose one question and a groundtruth answer for each of the following documents. Requirements: 1. Questions should be as clear as possible and have definitive and relatively short answers. 2. The distribution of the evidence paragraphs should be as random as possible throughout the article. 3. Ensure that the questions have varied types, including, but not limited to, information-extracting questions (e.g., the time of an
event, the date of birth of a person, etc.), summarizing questions (e.g., which people do the article mainly describes), and multi-hop reasoning questions.”
The average time taken to annotate each data sample in MultiFieldQA is around 5 minutes. The annotators involved in this process are Ph.D. students with extensive research experience in the field of NLP, positioning them as expert annotators. Cross-validation among annotators shows a 100% accuracy rate of the annotated answers.

heading: B Evaluation Setups
text: Evaluation Prompts. In this section, we present a collection of customized prompt templates designed for each dataset within LongBench, utilized during our evaluation. Recall that each data instance is accompanied by an input I as well as a context C. We place the instruction both at the beginning and end of the prompt, ensuring the models fully grasp what to do.
NarrativeQA: You are given a story, which can be either a novel or a movie script, and a question. Answer the question as concisely as you can, using a single phrase if possible. Do not provide any explanation. Story: {context} Now, answer the question based on the story as concisely as you can, using a single phrase if possible. Do not provide any explanation. Question: {input} Answer:
Qasper: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write “unanswerable”. If the question is a yes/no question, answer “yes”,
“no”, or “unanswerable”. Do not provide any explanation. Article: {context} Answer the question based on the above article as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write “unanswerable”. If the question is a yes/no question, answer “yes”, “no”, or “unanswerable”. Do not provide any explanation. Question: {input} Answer:
MultiField-en: Read the following text and answer briefly. {context} Now, answer the following question based on the above text, only give me the answer and do not output any other words. Question: {input} Answer:
MultiField-zh: 阅读以下文字并用中文简短回答： {context} 现在请基于上面的文章回答下面的问题，只告诉我答案，不要输出任何其他字词。 问题：{input} 回答：
HotpotQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: {input} Answer:
2WikiMultihopQA: Answer the question based on the given passages. Only give me the answer and do not output any other words. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: {input} Answer:
MuSiQue: Answer the question based on the given passages. Only give me the answer and do not output any other words. The following are given passages. {context} Answer the question based on the given passages. Only give me the answer and do not output any other words. Question: {input} Answer:
DuReader: 请基于给定的文章回答下述问题。 文章：{context} 请基于上述文章回答下面的问题。
问题：{input} 回答：
GovReport: You are given a report by a government agency. Write a one-page summary of the report. Report: {context} Now, write a one-page summary of the report. Summary:
QMSum: You are given a meeting transcript and a query containing a question or instruction. Answer the query in one or more sentences. Transcript: {context} Now, answer the query based on the above meeting transcript in one or more sentences. Query: {input} Answer:
MultiNews: You are given several news passages. Write a one-page summary of all news. News: {context} Now, write a one-page summary of all the news. Summary:
VCSUM: 下面有一段会议记录，请你阅读后，写一段总结，总结会议的内容。 会议记录： {context} 会议总结：
TREC: Please determine the type of the question below. Here are some examples of questions. {context} {input}
TriviaQA: Answer the question based on the given passage. Only give me the answer and do not output any other words. The following are some examples. {context} {input}
SAMSum: Summarize the dialogue into a few short sentences. The following are some examples. {context} {input}
LSHT: 请判断给定新闻的类别，下面是一些例子。 {context} {input}
PassageCount: There are some paragraphs below sourced from Wikipedia. Some of them may be duplicates. Please carefully read these paragraphs and determine how many unique paragraphs there are after removing duplicates. In other words, how many non-repeating paragraphs are there in total? {context} Please enter the final count of unique paragraphs after removing duplicates. The output format should
only contain the number, such as 1, 2, 3, and so on. The final answer is:
PassageRetrieval-en: Here are 30 paragraphs from Wikipedia, along with an abstract. Please determine which paragraph the abstract is from. {context} The following is an abstract. {input} Please enter the number of the paragraph that the abstract is from. The answer format must be like “Paragraph 1”, “Paragraph 2”, etc. The answer is:
PassageRetrieval-zh: 以下是若干段落文字，以及其中一个段落的摘要。请确定给定的摘要 出自哪一段。 {context} 下面是一个摘要 {input} 请输入摘要所属段落的编号。答案格式必须是“段落1”，“段落2”等格式 答案是：
LCC: Please complete the code given below. {context}Next line of code:
RepoBench-P: Please complete the code given below. {context}{input}Next line of code:
Maximum Output Length. We set a maximum output length on each dataset during evaluation to prevent the models from non-stop generation.
1-1 1-2 1-3 1-4 2-1 2-2 2-3 2-4 3-1 3-2 3-3 3-4 4-1 4-2 4-3 4-4 5-1 5-2 5-3 6-1 6-2
128 128 64 64 32 32 32 128 512 512 512 512 64 32 128 64 32 32 32 64 64

heading: C Radar Plot and Analysis
text: Among the 6 major tasks, summarization and code completion tend not to be sufficiently discerning. This may be due to the fact that the similarity-based metrics (ROUGE-L, Edit Sim) on these tasks are
not sensitive enough to well distinguish between the strong and weak models. Meanwhile, we find that synthetic tasks tend to offer a higher level of discernment, where models either achieve a high score or display a near-zero performance. These findings lead us to believe that it may not be a good idea to simply obtain an average over all tasks as the sign of models’ long context capability, as used in previous benchmark (Shaham et al., 2023) — since the performance on the more discerning tasks, such as the synthetic tasks in our benchmark, may dominate the final rank. This necessitates an evaluation strategy like we use in LongBench that separately assesses each task category, potentially leading to more meaningful benchmarking results.

heading: D Analysis on the Inter-task Correlation on LongBench
text: We analyze the multi-task property of LongBench by the inter-task correlation among and across each category of tasks, as shown in Figure 5. We observe that most tasks within the same task category have a high correlation, except for PassageCount (5-1), which exhibits low correlation with almost all tasks since models perform poorly (almost random) on this challenging task. Meanwhile, we notice that the correlations between Qasper (1-2), RepoBench-P (6-2) and the other tasks are also lower, which implies that these tasks potentially require a different attention pattern than the other tasks. Notably, tasks in the same language have a higher correlation with each other, e.g., a high correlation between Chinese tasks (1-4, 2-4, 3-4, 4-4, 5-3). These observations suggest that LongBench provides a more comprehensive evaluation result by integrating various types of tasks and languages.

heading: E Full results on LongBench-E
text: We show the full results on LongBench-E in Table 9.

heading: Future Work
text: Moreover, our analysis suggests that LongBench and LongBench-E serve as ideal testbeds for future research in long context modeling