L-Eval: Instituting Standardized Evaluation for Long Context Language Models (2024)

Abstract: Recently, there has been growing interest in long-context scaling of large language models (LLMs). To facilitate research in this field, we propose L-Eval to institute a more standardized evaluation for Long-Context Language Models (LCLMs) addressing two key aspects: dataset construction and evaluation metrics. On the one hand, we build a new evaluation suite containing 20 sub-tasks, 508 long documents, and more than 2,000 human-labeled query-response pairs including diverse task types, domains, and input length (3k∼200k tokens). On the other hand, we investigate the effectiveness of evaluation metrics for LCLMs and we show that Length-instruction-enhanced (LIE) evaluation and LLM judges can better correlate with human judgments. We conducted a comprehensive study of 4 popular commercial LLMs and 12 open-source counterparts using the L-Eval benchmark. Our empirical findings offer useful insights into the study of LCLMs and lay the groundwork for the development of a more principled evaluation of these models.1

heading: 1 Introduction
text: Currently, extending the context length of large language models has emerged as a significant direction for the evolution of LLMs, with a considerable amount of effort being dedicated to this pursuit (Peng et al., 2023b; Chen et al., 2023b; Song et al., 2023; Jin et al., 2024). As a consequence, these widely used benchmarks that mainly consist of short prompts (Chen et al., 2021a; Hendrycks et al., 2021c,a), and whose majority of test samples contain fewer than 4k tokens, are becoming increasingly inadequate for evaluating these LongContext Language Models (LCLMs). Specifically designed evaluation suites, such as Scrolls and ZeroScrolls (Shaham et al., 2022, 2023) have been proposed. However, they exhibit a limited range
1We release our new evaluation suite, code, and all generation results at https://github.com/OpenLMLab/LEval.
of task types and input lengths; tasks like summarization and abstractive QA are overwhelmingly predominant and usually have an average input length of less than 16k tokens. Furthermore, these benchmarks mostly employ n-gram metrics such as ROUGE (Lin, 2004) for automatic evaluation, while recent work underscored the limitations of such metrics in accurately evaluating the performance of LLMs (Sellam et al., 2020; Zheng et al., 2023; Zhang et al., 2023b; Li et al., 2023d).
To address these issues, we propose L-Eval, offering a more standardized long-context evaluation for LLMs through data construction and the investigation of evaluation metrics. On data construction, L-Eval divides the tasks into two groups. Tasks in the closed-ended group usually have clear answers and can be quickly and objectively scored. The open-ended group on the other hand, comprises tasks that allow for a wide range of acceptable responses, such as summarization, posing challenges for automatic evaluation (Nguyen, 2021; Chang et al., 2023). Overall, L-Eval consists of 20 subtasks, 4 of which with 378 query-response pairs are manually annotated from scratch to expand the coverage of domains, input lengths, and task types (Table 1). Given the scarcity of closed-ended datasets with long prompts (Pang et al., 2022), we design 75% of our new tasks to be closed-ended. Of the remaining 16 sub-tasks in our suite, 5 are re-annotated from public datasets with updated instructions, and the remaining 11 are derived from existing datasets. In L-Eval, we prioritize quality over quantity by manually checking all the queryresponse pairs and removing mistaken test samples after data construction.
L-Eval adopts diverse evaluation approaches including n-gram metrics, LLM judges, and human evaluation, to better evaluate open-ended generation results. Our experiments suggest that the dominant n-gram metrics in long-context tasks often cannot correlate well with human evalua-
14388
tion in the zero-shot setting. A crucial reason is that LLMs often struggle to generate responses of a similar length to the reference answers without being trained on domain-specific data (see Table 2). Therefore, we propose the LengthInstruction-Enhanced (LIE) evaluation in which LLMs are guided toward answers of a desired length. The empirical results demonstrate a substantial improvement for all reference-based metrics in the Kendall-Tau correlation coefficient (τ ) with human judgments (Figure 2). We also validate LLM-as-a-judge, which is proposed as a costeffective alternative to human evaluation for openended tasks (Zheng et al., 2023; Chiang and Lee, 2023; Zhang et al., 2023b), in long-context evaluation scenarios. This yields more accurate results compared to the n-gram metrics.
We conducted a comprehensive study with 16 different LLMs in L-Eval. Some of our key findings are summarized below: (1) There is still a remarkable gap between open-source and commercial models, for both closed-ended tasks and open-ended tasks, while the performance gap is not accurately reflected by n-gram metrics. (2) Although current extension approaches for opensource models significantly improve performance on closed-ended tasks, they fall short on openended tasks. This is largely due to the models’ inadequate capacity to understand instructions in longer inputs. (3) Experiments on gpt-3.5-turbo with both dense and sparse retrievers show that endto-end long-context models outperform traditional retrieval based systems (4) Training-free scaled positional embeddings can enhance the retrieval capability of LLMs over longer input, while it can have a negative impact on reasoning ability.
More findings are shown in §5.2 and §A.3. A detailed discussion with concurrent work can be found in §2.2. We hope L-Eval and our findings contribute to a deeper understanding of current LCLMs as well as their evaluation metrics.

heading: 2 Related Work

heading: 2.1 Long Context Language Models
text: Feeding long context leads to bottlenecks in language model training and inference due to computational resources. Some community efforts focus on developing efficient attention mechanisms to build efficient language models (Sun et al., 2023; Ding et al., 2023; Li et al., 2023c; Fu et al., 2023; Peng et al., 2023a). In addition to optimizing the atten-
tion mechanism, some works (Bulatov et al., 2023; Dai et al., 2019; Mohtashami and Jaggi, 2023) focus on chunking the input to model both the current text in the chunk and the previous context states, effectively extending the length of context processing. Besides the efficiency challenge, the scalability of positional embedding is also crucial. ALiBi (Press et al., 2022), and XPOS (Sun et al., 2022) emphasize the significance of local context to enhance the language model’s ability to perform extrapolation. Moreover, position interpolation (PI) (Chen et al., 2023a) and NTK-aware (LocalLLaMA, 2023b,a) are the most popular approaches based on RoPE (Su et al., 2022) to effectively extend the context length.

heading: 2.2 Long Sequences Benchmarks
text: Tay et al. (2020) introduce the Long Range Arena (LRA), a benchmark encompassing five distinct classification tasks. CAB (Zhang et al., 2023a) is another benchmark for different efficient attention designs by comparing both efficiency and accuracy. In the language domain, previous work on LCLMs tends to report PPL to evaluate language models (Su et al., 2022; Peng et al., 2023b) on lengthy context. However, PPL may not usually correlate with the actual performance (Sun et al., 2021). L-Eval differs from concurrent work on long context evaluation (Dong et al., 2023; Li et al., 2023b; Kwan et al., 2023; He et al., 2023) such as S3Eval (Lei et al., 2023) and LongBench (Bai et al., 2023) in 3 aspects: (a) L-Eval moves beyond solely relying on n-gram metrics. We adopt diverse evaluation approaches for open-ended tasks and suggest mitigating length bias when using reference-based metrics. (b) L-Eval offers a better diversity of task types and domain coverage by not only utilizing existing datasets but also by manually annotating new tasks. (c) L-Eval has revealed new findings that enrich the understanding provided by other work, through a comprehensive analysis of 16 recently released models across a range of open-ended and closed-ended tasks.

heading: 3 Data Construction
text: In this section, we detail the essential processes involved in constructing L-Eval. Specifically, we describe the pipeline for annotation, re-annotation, and post-processing, as well as provide statistics of L-Eval. Detailed explanation of tasks in L-Eval is in Appendix §B.

heading: 3.1 Data Annotation from Scratch
text: There are four datasets annotated from scratch in L-Eval: Coursera, SFiction, CodeU, and LongFQA. Generally, the new datasets aim to extend the coverage of domains and task types in L-Eval. While previous tasks are usually open-ended, we annotate more closed-ended tasks, which are usually less susceptible to biases in metrics.
Coursera This dataset originates from the Coursera website.2 We use this task to test the reasoning ability of LLMs on lengthy courses that are difficult to comprehend. This sub-task comprises lectures on big data and machine learning, requiring learners to possess a strong foundation in computer science. The lengthy input consists of video subtitles filled with domain-specific terms. Questions, options, and the ground truth answers are labeled by the authors. Coursera is a closed-ended task and we set multiple correct options in this task. The model must discern the subtle differences among the options and understand the contexts that make each one valid. We find that GPT-4-32k (OpenAI, 2023) outperforms other models by the largest margin on this task. An example is in Appendix §B.1.1.
SFcition We annotate this sub-task to assess the adherence of the LCLM to the input context. We argue that contextual knowledge (stored in lengthy inputs) is more vital than parametric knowledge (gained during pretraining) for LCLMs (Neeman et al., 2022). Therefore, LLMs should prioritize alignment with the input context when processing long documents that contain information contradictory to their parametric knowledge. To simulate this scenario, we annotate a closed-ended dataset sourced from science fiction, consisting of True or False questions. Most of the answers to these questions contradict real-world principles and do not comply with actual physical laws. We find that even powerful proprietary models like gpt-3.5-turbo also face difficulties with this task as they tend to heavily rely on their knowledge. An example can be found in Appendix §B.1.2.
CodeU Previous closed-ended code datasets, like HumanEval (Chen et al., 2021a), are limited to short-context samples. In L-Eval, we create a longcontext dataset to test the code understanding capability. CodeU demands LLMs to deduce the final output of long Python programs involving multiple function calls. To solve this task, LLMs should first
2https://coursera.org/
identify which functions have been invoked, and understand the action of each function. These functions are from Numpy and a string-processing codebase built by us. To prevent language models from inferring the behavior of the invoked functions based solely on their names, we replace the original descriptive function names with anonymized placeholders such as Op1(), Op2(), ...OpN() throughout the code snippet. An example is provided in Appendix §B.1.3.
LongFQA Finance is also a crucial application scenario for LCLMs. However, existing finance datasets, such as FinQA (Chen et al., 2021b), are all short-context. To overcome this limitation, we curate a dataset of long input documents by collecting public earnings call transcripts from the Investor Relations sections of six company websites. An example is provided in Appendix §B.1.4.

heading: 3.2 Data Re-annotation from Public Datasets
text: Five datasets in L-Eval are re-annotated from public datasets with new instructions. We present examples and more details in Appendix B.2.
GSM(16-shot) is derived from 100 grade school math problems in the GSM8k dataset (Cobbe et al., 2021) to test the few-shot learning ability. We construct 16 in-context examples with lengthy Chainof-Thought where 8 examples come from chainof-thought-hub3. The remaining 8 examples are constructed by us which improve the accuracy of gpt-3.5-16k from 79% (8-shot) to 84% (16-shot). We inject some new synthesis instructions to test global context modeling into QuALITY (Pang et al., 2022). The Openreview dataset contains papers collected from openreview.net. Our new instructions include: writing an abstract, summarizing the related work, and giving feedback. SPACE (Angelidis et al., 2021) is a review summarization dataset, and we add diverse user queries from different aspects for this task. As for synthetic tasks, previous work (Li et al., 2023a; Xiong et al., 2023) has used a first topic/sentence retrieval task to test the ability of modeling long-range dependency. However, we observe that retrieving the first topic is too easy to distinguish the ability of different models (Figure 1) while retrieving the second and the third topics presents a significantly higher level of challenge. L-Eval enhances the task with second/third topic retrieval, denoted as TopicRet
3https://github.com/FranxYao/ chain-of-thought-hub
in Table 1.

heading: 3.3 Data Filtering and Correction
text: The remaining 11 tasks are derived from existing datasets. L-Eval includes an additional postprocessing procedure in which errors that cannot be automatically verified are subjected to manual review. We initially employ Claude-100k (Anthropic, 2023) to filter all samples. Concretely, we prompt Claude to generate both the answer and the supporting evidence from the document. If Claude’s response significantly deviates from the ground truth, or if it suggests that the answer cannot be inferred from the provided context, we will review this QA pair and then opt to either discard or correct it. We provide examples in Appendix B.3.

heading: 3.4 Statistics
text: The statistics of L-Eval are shown in Table 1. The L-Eval contains various task types such as multiple-choice QA (TOEFL (Tseng et al., 2016), QuALITY), true or false QA (SFiction), Few-shot learning (GSM), Code understanding (CodeU), goal-oriented dialogues (MultiDoc2Dial (Feng et al., 2021)), extractive QA (CUAD (Hendrycks et al., 2021b)), abstractive QA (LongFQA, NarrativeQA (Kočiský et al., 2017), Qasper (Dasigi et al., 2021), NQ (Kwiatkowski et al., 2019)), singledocument summarization (GovReport (Huang et al., 2021), BigPatent (Sharma et al., 2019), SummScreen (Chen et al., 2022), QMSum (Zhong et al., 2021)), multi-document summarization (Multi-News (Fabbri et al., 2019)), and so on. The long documents in L-Eval across many domains such as law, finance, academic papers, lectures, conversations, news, famous Python codebase, long-form novels, and meetings. The average input length in L-Eval ranges from 4k to 60k. The maximum sample in L-Eval contains nearly 200k tokens. This diversity represents real-world scenarios where different tasks may require different lengths of context and instructions.

heading: 4 Evaluation Metrics
text: In this section, we present various evaluation metrics for text generation, including exam evaluation for close-ended tasks and different levels of openended evaluation, most of which are referencebased metrics. We also conduct experiments to study the correlation between automated metrics and human scoring.
Exam evaluation This is designed for closedended tasks, i.e., multiple-choice questions. The evaluation metric used for these tasks follows the exact match format (accuracy %), similar to grading exam papers. Each question’s score is calculated as 100 divided by the number of questions.
Human evaluation Currently, automatic metrics for open-ended generations still cannot replace human evaluation (Chiang and Lee, 2023). Our human evaluation procedure is in Appendix A.2. We engage human evaluators to score the outputs on a scale of 1 to 5, which signifies from poor output to excellent output. To reduce the cost of human evaluation, we select 12 long documents with 85 open-ended questions. Human evaluation for longcontext tasks is extremely time-consuming. Our human evaluation of 85 questions across 7 systems takes around 30 hours (4 days) for each annotator.
LLM judges In short-context settings, LLM acting as a judge is commonly employed for openended tasks evaluation (Zheng et al., 2023; Zeng et al., 2023). In L-Eval, we also incorporate LLM judges and utilize a pairwise battle format following Li et al. (2023d); Dubois et al. (2023). We report the win rate versus gpt-3.5-turbo-16k. Unlike short-context settings, long-context evaluation does not typically allow for feeding entire lengthy prompts into the judge. Consequently, the results of the evaluation depend primarily on the reference answers and the instructions provided. LLM evaluators have been reported to favor more detailed and lengthy responses (Zheng et al., 2023). This bias becomes more pronounced in long context settings because the incomplete input makes it difficult to accurately determine the correctness of specific details and information. Therefore, the judgment model must remember that details not corroborated by the reference answers should not be considered beneficial. We enhance the judgment prompt with Additional details or information that are not mentioned in the reference answer cannot
be considered as advantages and do not let them sway your judgment. In the experiment section, we mainly use GPT-4 as the judge model. We preliminarily evaluate four different models on four popular datasets with GPT-4 judge, as shown in Table 2. This incurs a cost of $125. Due to budget constraints, it is not feasible to adopt GPT-4 judge for all open-ended questions in L-Eval. Therefore, we carefully choose 96 open-ended questions (1-2 documents per task) to evaluate all the baselines.
N-gram metrics Assessing all tasks is still expensive for human/LLM evaluators. L-Eval also takes into account n-gram metrics like ROUGE and F-1, which have been widely used in previous text generation benchmarks. Notice that n-gram metrics are susceptible to length bias. Performing lexical matching between ground truth and predictions with significant length differences often can not yield accurate results.

heading: 4.1 Length Instruction Enhanced Evaluation
text: In preliminary experiments, we find that LLMs tend to generate lengthy responses with detailed explanations, which usually lead to predictions and ground truth being at different levels of granularity (see ∆L Table 2). This length bias results in a
significant influence on the n-gram metrics. Compared with N-gram metrics, LLM judges are more accurate and robust to output length. For instance, Claude-100k only achieves a 9.84 F-1 score due to undesired output length. In L-Eval, we argue that long context language models should further focus on more accurate content rather than accurate length. Practically, issues about undesired generation length can be easily solved by prompting. We first adopt Length-Instruction-Enhanced (LIE) evaluation in LLMs evaluation benchmarks which is simple but effective in overcoming the length bias, i.e., the number of words of ground truth is directly exposed to LCLMs. LIE evaluation in this work is implemented by injecting the model with the desired length into the original instruction (e.g., [Original Instruction]: Please summarize the opinions of the professor. [Length Instruction]: We need a 50-word summary, assuming that 50 is the number of words in the reference answer). The results of Claude-100k in Table 2 demonstrate a substantial improvement in terms of the F-1 score: there is a near 50-point gap depending on whether or not the model generates with the expected length.
To validate the LIE evaluation, we then conduct a human evaluation (Appendix §A.2) on the 85
open-ended questions. We have 3 annotators to verify 7 models and calculate the Kendall-Tau correlation coefficient (τ ) between these metrics and the average human score. The main results are shown in Figure 2 (Blue bar) and experimental settings are in §A.2. Results indicate that all these automatic metrics (except GPT-4) fail to correlate with human evaluation without length control. As we can see from Figure 2, the improvements brought by length instruction are marked with yellow, and after adding the length instructions, τ has been improved from 0.5 to 0.8 for ROUGE-L and the τ of GPT-4 evaluator has reached to 1.0. In Figure 3, we rank 6 models evaluated by 6 different evaluation systems. Figure 3 (a) shows the results given by metrics without length instruction. These hexagons are often distorted because these metrics usually cannot achieve good correlation. When comparing the models that are tested with length instructions (Figure 3 (b)), we observe that the hexagons become more regular, indicating improved correlation among these metrics.

heading: 5 Benchmarking LLMs with L-Eval
text: In this section, we list our 16 baseline models and the results on both open-ended and closed-ended tasks. Generally, there are considerable gaps between open-source models and commercial models. A detailed description of baseline models can be
found in §A.1. The prompt templates for each task are available in §B. We run all the experiments using FlashAttention (Dao et al., 2022) on a single NVIDIA A800 GPU. The document input is truncated from the right.

heading: 5.1 Baselines
text: Commercial Models (1) Claude-100k developed by Anthropic, (2) GPT-4-32k, OpenAI’s most powerful long context model, (3) Turbo-4k-0613 and (4) Turbo-16k-0613 is the snapshot of GPT-3.5 from June 13th 2023 which can handle up to 4k/16k input tokens.
Open-source Models (5) Llama1 (Touvron et al., 2023a), a widely used open-source model developed by Meta AI with a 2k pre-training length, (6) Vicuna1.3 (Chiang et al., 2023), tuned on shareGPT based on Llama1, (7) Longchat-16k, the long context version of Vicuna1.3 using PI, (8) Llama2 with 4k pre-training context, (9) Llama2chat (10) Llama2-NTK, extending the context length of Llama2-chat with NTK-aware RoPE, (11) Vicuna1.5-16k (Zheng et al., 2023), the long context version of Llama2 using PI & ShareGPT (12) Longchat1.5-32k, the 32k context version of Llama2 using PI & ShareGPT. (13) Chatglm28k (Du et al., 2022), (14) Chatglm2-32k, the 32k context length version, (15) XGen-8k-inst (Nijkamp et al., 2023), an 8k context model developed by salesforce (16) MPT-7B-StoryWriter-65k, based on MPT-7B and ALiBi with a context length of 65k tokens on a subset of Books3 dataset.
Retriever We implement the dense retriever with the OpenAI AdaEmbedding as the dense retriever and BM25 as the sparse retriever to extract 4 pieces of most related 1k-chunked documents, which are further provided as the context to answer questions.

heading: 5.2 Main Results
text: The performance of LCLMs on closed-ended tasks is shown in Table 3. As for open-ended tasks, we test the 96-question subset (Table 4) with GPT-4 evaluation. Results from n-gram metrics on all test sets and the rankings of LLMs are listed in §A.3.
From the main results, we have the following observations. GPT-4-32k clearly outperforms all other models by a very significant margin, establishing SOTA in L-Eval closed-ended tasks. There is still a near 20-points gap between the best opensource 16k models and Turbo-16k. As for openended tasks, since the input texts are generally longer and a global understanding of the context is required, Claude-100k, with the longest context length, surpasses all baseline models including GPT-4-32k. Although results of n-gram metrics indicate that open-source LCLMs have achieved performance close to Turbo-16k-0613 on open-ended tasks, the evaluation outcomes from both LLM (Table 4) and human judges (Table 5) reveal that there is still a significant gap between them. Moreover, retrieval-based methods based on Turbo-4k fall short in comparison to encoding the entire con-
text (Turbo-16k), as certain tasks are difficult to address through simple retrieval.
Fine-tuning longer benefits closed-ended tasks but falls short in open-ended tasks In Table 3, for open-source models using scaled positional embedding, Longchat and Vicuna1.5-16k obviously outperform Vicuna1.3 and Llama2-chat. The results suggest that further tuning on longer input does benefit long-context tasks. However, according to Table 4, unlike results on closed-ended tasks, the best model Vicuna1.5-13b only wins Turbo-16k by 34%, 8 points lower than its short version Llama2-13b. Llama2-13b-chat (Touvron et al., 2023a) is still the strongest open-source baseline, indicating that current LCLMs simply based on scaled position embedding may not be enough for these challenging open-ended generation tasks. Based on our human evaluation, we find that although scaled position embedding techniques such as NTK (LocalLLaMA, 2023b) or PI (Sun et al., 2022) effectively extend models’ context length, the models tend to get lost when facing lengthy input tokens and are unable to follow the instruction. We classify these outputs as ‘invalid outputs’. To investigate model performance on different context lengths, we split the test set into 2 parts: PART-A only contains samples with a short length, and PART-B contains sequences longer than 4k. We compare the number of invalid outputs from Llama2-4k/Vicuna1.5-16k and Turbo-4k/Turbo-16k in Figure 4. Results show that the number of invalid outputs from Turbo-16k remains a very small amount on both PART-A and B while the invalid outputs from Vicuna1.5-16k dramatically increase when facing longer input. However, the performance improvement on closedended tasks is impressive. A possible reason is that the training corpus is highly likely to contain many training samples with similar question styles. This strongly enhances their instruction-following ability on closed-ended tasks.
Performance on retrieval tasks contradicts reasoning tasks NTK-aware scaled RoPE is the most popular extrapolation method for Llama2 (LocalLLaMA, 2023a) and only requires increasing the base frequency in the vanilla RoPE. However, we find that the performance on topic retrieval tasks does not match the reasoning capability over lengthy context. The RoPE base frequency is 10k for Llama-based models. As can be seen from Figure 5, when we increase the base from 20k
to 160k, there is a continuous improvement on topic retrieval accuracy. However, performance on math problems with lengthy in-context examples exhibits a completely opposite trend, indicating that it is challenging for the model to maintain its reasoning abilities when increasing the base. In contrast, the performance on retrieval tasks seems to remain unaffected after the base reaches 60k.

heading: 6 Conclusion
text: In conclusion, the much-needed rigorous benchmark L-Eval introduced in this work provides a comprehensive suite of tasks and evaluation met-
rics to assess the capabilities of LCLMs. Our analysis using L-Eval offers valuable insights into the current state and limitations of LCLMs. We believe that with its focus on practical, long-form documents across domains, L-Eval can serve as a challenging testbed to drive advances in modeling longer contexts.

heading: Limitations
text: One significant limitation of L-Eval is the potential for data contamination which is also a common but challenging issue for LLMs benchmarks. The risk of training on data that includes test sets could lead to inflated performance metrics that do not accurately reflect a model’s true generalization capabilities. The main solution currently is to introduce newer data that postdates the training period. However, there is no guarantee that this new data will not be used for training later on. The only solution would be to continuously annotate new long-context test data, but this is still expensive at present. It is essential for future users of this benchmark to strive for fair comparisons by ensuring that models are evaluated on similar training data and do not intentionally introduce overlap with the test data.

heading: Ethics Statement
text: We re-annotate Openreview dataset to examine LCLMs’ performance on reading and understanding long academic papers. Noted that we discourage reviewers from using large models for reviews. Our goal is to assist authors in further improving their own papers.

heading: Acknowledgements
text: We thank Hang Yan, Zhiyong Wu, Fei Huang and Siyu Ren for their helpful comments. This research was supported in part by the joint research scheme of the National Natural Science Foundation
of China (NSFC) and the Research Grants Council (RGC) under grant number N_HKU714/21.

heading: A Appendix

heading: A.1 Baseline Models in L-Eval

heading: Commercial Models
text: • Claude-100k is developed by Anthropic4 and targets understanding extremely long documents and answering related questions. It has the longest context length among all the LLMs.
• GPT-4-32k is developed by OpenAI5. It is the long context version of GPT-4 maintaining very strong reasoning ability over 32k context length but also the most expensive model.
• Turbo-4k-0613 is the snapshot of GPT-3.56
from June 13th 2023 which can handle up to 4k input tokens. Turbo-16k-0613 is the released long context version of Turbo-4k-0613.

heading: Open-source Models
text: • Llama1 (Touvron et al., 2023a)7 is a widely used open-source model developed by Meta AI with a 2k pre-training context length. The first version of Llama did not release a chatbotbased model.
• Vicuna1.3 (Chiang et al., 2023) 8 is a chatbot fine-tuned from Llama1 on shareGPT.
• Longchat-16k (Li et al., 2023a) 9 is the long context version of Vicuna. It uses positional interpolation to adapt 16k context. Concretely, they further fine-tune Llama1 on lengthy dialogues (16k tokens) from shareGPT.
• Llama2 (Touvron et al., 2023b) is the second version of Llama recently released by Meta AI. The updated version has 4k pretraining context with more powerful long context understanding capabilities.
• Llama2-chat (Touvron et al., 2023b) is a chatbot based on Llama2 released together with Llama2. Please notice that if we do not follow the pre-defined input format, i.e., ignore
4https://www.anthropic.com/index/ 100k-context-windows
5https://platform.openai.com/docs/models/ gpt-4
6https://platform.openai.com/docs/models/ gpt-3-5
7https://github.com/facebookresearch/llama 8https://github.com/lm-sys/FastChat.git 9https://github.com/DachengLi1/LongChat
the special tokens, there will be a significant degradation in performance.
• Llama2-NTK-chat (LocalLLaMA, 2023b) is the long context version of Llama2-chat. It uses NTK-aware positional embedding. If we want to extend the model context window to t (we call t as a scale-up factor) times its original pretraining size, we just need to increase the original base=10,000 of RoPE (θn = 10000−2n/d) to 10,000 ×t d d−2 where d
is the head dimension in Transformer. In our experiments, this theory does not hold in practical tasks (see section §A.3), which means the model still tends to generate random tokens when setting t = 4 on 16k context length. We set t = 8 in experiments.
• Llama2-NTK-chat (Dyn) (LocalLLaMA, 2023a) is the dynamic version of Llama2NTK-chat. The only difference is that the scale-up factor t in dynamic NTK depends on the current input length L and the pretraining length l, i.e., t = Ll
• Vicuna1.5-16k uses Llama2 as the base model and performs further finetuning on concatenated 16k tokens lengthy dialogues from shareGPT. This model is based on positional interpolation which helps the training process converge fast.
• LongChat1.5-32k is the 32k version of Vicuna1.5-16k.
• Chatglm2-8k (Du et al., 2022)10 is the second version of the open-source bilingual chat model Chatglm. The context length of the base model is further pretrained with 32k context window and finetuned on dialogue data with 8k context window.
• Chatglm2-32k is the long context version of Chatglm2 using positional interpolation.
• XGen-8k-inst11 developed by salesforce follows a multi-stage pretraining procedure. They first train the model with 2k context length and progressively increase the pretraining length to 4k, finally reaching 8k.
10https://github.com/THUDM/ChatGLM2-6B 11https://github.com/salesforce/xgen
• MPT-7B-StoryWriter-65k12 is designed to handle super-long context lengths. It was tuned on MPT-7B with a context length of 65k tokens on a subset of Books3 dataset.

heading: A.2 Human Evaluation
text: Evaluating long-sequence, open-ended tasks remains a challenge. Consequently, human evaluation is still needed to assess these models. In this section, we detail the human evaluation procedure conducted on seven baseline models using 85 openended questions from L-Eval. We aim to assess the correlation between human judgment and automatic metrics by manually scoring outputs from various models.
Experimental setup We evaluate seven models, comprising three commercial and four open-source models: (1) Claude-100k, (2) Turbo-16k-0613, (3) Turbo-4k-0613, (4) Vicuna1.5-7b-16k (Llama2), (5) Longchat-7b-16k (Llama1), (6) Llama2-7bchat, and (7) Llama2-13b-chat. These models are tested on 85 open-ended questions from L-Eval open-ended tasks. Each sample is evaluated by three well-educated annotators who are Ph.D. students specializing in long-context models and are familiar with these tasks.
We launched a pre-evaluation stage in which annotators were asked to evaluate one document from NarrativeQA (Kočiský et al., 2017), which had been carefully scored by the authors for all questions. Based on the pre-evaluation results, we instructed the annotators by providing detailed feedback. The model outputs are ranked on a fivelevel scale:
• Level-1 (worst): The response is totally unhelpful to answer the question.
• Level-2: The output generally deviates from the original question, but some information is useful to solve the problem.
• Level-3: The response is partially correct, but the generated answer may contain some errors or omit key information.
• Level-4: Most of the response is correct, but there may be minor issues such as being overly long (which cannot be considered a flaw if it is a reasonable explanation), or it might omit some information, but this does not affect the overall meaning.
12https://www.mosaicml.com/blog/mpt-7b
• Level-5 (best): The output is close-to-human or even better.
We calculate the average score to obtain the final human evaluation results. To determine if the ranking produced by these automatic metrics correlates with the ranking provided by the annotators, we use the Kendall-Tau correlation coefficient. We allow each model to generate outputs twice: first in the original mode without any length instruction, and then with the given length instructions. To minimize variance, we use greedy search as the decoding algorithm.
We create a web page UI to improve annotators’ experience during the annotation process. The model’s name remains anonymous to annotators, and a reference answer is provided. The suggested workflow for annotators includes 30 minutes for reading the paper and 15 minutes for grading the 7 outputs in total. Actually, the evaluation process takes each annotator about 30 hours, equivalent to 4 working days. This indicates that widely relying on human evaluation for long-context models is still impractical due to the high cost.
Human evaluation results The results of our human evaluation are presented in Table 5. We employ Fleiss’s kappa and the system-level KendallTau correlation coefficient to assess the interannotator agreement among human evaluators. The results indicate that Fleiss’s kappa is 0.66, and the Kendall-Tau correlation coefficient is 1.0. As can be seen, despite being fine-tuned on longer contexts, open-source models still struggle with very long input sequences during inference. When fed with numerous input tokens, the number of Level1 outputs from open-source LCLMs significantly increases, while the LLMs with only a 4k context length can maintain their generation quality at a partially correct level, albeit without achieving high scores. It’s also observable that the n-gram metrics F-1 and ROUGE generally do not correlate with the human evaluation results. Given the impracticality of testing a large number of samples using LLMs due to high costs and inefficiency, we also urge for more advanced metrics.

heading: A.3 Analysis
text: Results from n-gram metrics Testing all cases in open-ended tasks in L-Eval with GPT-4 is affordable. To give an overview of all the models on open-ended tasks, we test all models with ngram metrics. As can be seen from the win rate
from LLM judges (Table 4) and human evaluation (Table 5), there is still a significant margin between commercial LLMs and open-source LLMs. However, the margin is not clear enough based on n-gram metrics. Based on n-gram metrics, the open-source LCLMs also fail to beat their origin short-context model on truncated context. Overall, current open-source LCLMs generally excel more in conventional summarization tasks that involve instructions like “Summarize this document”
compared with query-based summarization and QA tasks. As for query-based tasks that pose questions from a specific perspective, performance can be significantly degraded if the instruction isn’t fully understood. As we mentioned before, the increased input length can also lower the model’s ability to comprehend lengthy instructions, thereby inhibiting its capability to generate answers that closely match the length of the ground truth. This phenomenon is less likely to be observed with more sophisticated LLMs (i.e. Turbo-16k). A naive solution is adding the instruction at both the beginning and end of the long input but there is still room to improve the ability of instruction understanding for LCLMs.
Retrieve-based models vs long context models We compare a representative LCLM baseline Turbo-16k-0613 with its short version Turbo-4k0613 but enhanced with retrieval in Table 3(closedended tasks) and Table 4(open-ended tasks). We use a sparse retrieval retriever bm25 and a strong dense retriever text-embedding-ada-002. Retrievalbased approaches generally yield better outcomes for tasks that have readily retrievable answers. For example, in the understanding of long lectures, where the document often contains definitions and explanations of academic terms, retrieval-based approaches yield better results. However, retrieval is not a general solution as its performance is strongly related to instruction and document style. For example, they would never answer questions like how many sentences are there in a document. Our re-
Table 9: Performance of various models on long document summarization tasks in terms of ROUGE.
Model Tokens GovReport Multi-News BigPatent SummScreen
AvgR-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L
Turbo-16k-0613 16k 45.9 15.6 23.6 35.3 8.1 16.1 46.0 20.3 29.3 32.0 5.4 16.9 24.5 AdaEmb-Turbo-0613 4k 45.0 14.3 20.8 35.7 7.7 15.4 45.6 15.9 27.6 30.0 3.3 15.2 23.0 bm25-Turbo-0613 4k 44.6 14.2 21.5 38.4 9.1 16.8 43.3 15.5 27.1 31.0 4.6 15.4 23.4
Truncating input tokens to the pretraining context length
llama2-7b-chat 4k 43.7 15.3 22.2 33.2 6.4 15.5 49.2 22.9 31.6 29.4 4.8 15.6 24.1 llama2-13b-chat 4k 46.3 16.1 24.0 34.9 8.1 16.3 48.4 20.9 30.3 32.6 6.6 17.2 25.1 vicuna1.3-7b 2k 44.6 16.4 23.2 32.9 6.9 14.8 44.7 20.4 28.8 28.7 3.6 14.8 23.3 longchat-7b-16k 2k 43.6 16.2 23.7 28.1 4.8 13.0 47.0 22.2 30.9 27.2 3.0 14.4 22.8 chatglm2-6b-8k 2k 45.2 18.3 24.6 32.1 6.9 15.0 44.6 22.1 30.0 26.4 2.6 13.8 23.4 xgen-7b-8k 2k 45.1 17.2 22.9 35.0 7.5 15.5 49.6 25.2 34.6 28.8 3.6 15.4 23.9
Truncating input tokens to the further finetuning context length
chatglm2-6b-32k 32k 38.1 16.1 21.0 24.2 5.8 12.8 46.5 24.1 32.5 23.4 4.2 13.8 21.8 longchat1.5-7b-32k 32k 45.7 17.7 24.0 36.8 8.7 15.7 42.0 18.2 27.2 21.5 2.7 13.0 22.7 longchat-7b-16k 16k 47.2 18.9 23.9 27.7 5.4 13.4 46.2 20.9 30.1 26.2 3.3 14.7 23.1 vicuna1.5-7b-16k 16k 47.2 18.9 25.0 32.3 6.8 15.5 48.1 25.1 32.4 26.0 3.6 14.8 24.6 longchat-13b-16k 16k 46.2 18.2 24.1 35.2 7.6 15.8 45.3 22.6 29.8 31.9 6.0 17.3 24.0 vicuna1.5-13b-16k 16k 45.2 17.9 24.2 31.6 6.8 15.2 46.1 21.8 30.0 28.3 3.7 16.3 23.9 llama2-13b-NTK 16k 33.0 11.0 17.7 26.0 6.4 13.5 37.9 13.5 22.9 25.6 5.3 14.0 18.9 llama2-13b-NTK(Dyn) 16k 42.0 14.9 22.4 34.0 7.8 15.9 45.3 19.1 28.5 25.5 3.9 13.9 22.7 longchat-13b-16k 8k 49.3 19.5 25.1 34.9 7.4 15.5 43.5 20.1 28.0 31.0 4.5 15.7 24.5 chatglm2-6b 8k 40.6 14.3 21.5 32.9 7.2 15.1 46.3 22.3 31.4 27.5 2.6 14.5 23.0 xgen-7b-8k 8k 40.2 13.8 21.1 31.9 6.0 15.3 45.9 21.4 29.2 28.2 3.3 15.2 22.6 mpt-7b-65k 8k 33.3 10.7 19.3 13.6 1.5 9.2 25.5 12.2 20.2 11.0 1.3 6.4 13.6
sults show that CodeU and GSM(16-shot) in LEval can not be solved by retrieval. Retrieval-based methods also face difficulties in automatically identifying the query from user inputs. Retrieval methods demonstrate comparatively less satisfactory performance in tasks where the answer cannot be retrieved, such as topic retrieval or tasks that demand models with long-range reasoning abilities like financial QA. Retrieve-based models produce similar or even superior results for summarization tasks. This may be because some paragraphs resembling summaries can be retrieved. Besides, we also noticed that the main reason why regular Turbo-0613 outperforms Turbo-16k is its superior ability to accurately follow instructions. However, even for these tasks, there are instances where the predicted answer might be “I don’t know" or “not mentioned" due to the limitation of the retrieval process. When evaluating retrievers, bm25 often matches the performance of the dense retriever, ada-embedding, in closed-ended tasks. However, in the open-ended tasks, the dense retriever adaembedding outperforms BM25 by more than two points. This superior performance can be attributed to the dense retriever’s ability to leverage not only term matching but also semantic matching.
Figure 6: Peformance of different NTK-based methods when tackling input length at multiple scales.
Dynamic NTK scaling rules do not hold in practical tasks Dynamic NTK-aware positional embedding (LocalLLaMA, 2023a) is becoming more and more popular for extrapolation without further training. Based on dynamic NTK, given an input sequence with length L and the model pretraining length l, we can set the original base 10,000 in RoPE to 10, 000× Ll d
d−2 where d is the head dimension, if we want to adapt the model to the longer context length L. We find that the scaling rule does not hold in practical tasks when the number of input tokens changes. The performance can be further enhanced by using some variants of NTK. We study 2 simple modifications on the original dynamic NTK: (1) NTK-bias which means we use
the base 10, 000× (Ll + 1) d
d−2 where 1 is the bias (2) NTK-weighted which means we use the base 10, 000×(Ll ∗2) d d−2 . Results are shown in Figure 6 where Llama2-PI-sharegpt is a fine-tuned baseline using position interpolation. We test the results of 4 models by truncating the input length of test cases in Coursera. We can observe that employing which variants of NTK are strongly affected by the maximum tokens of the dataset. When the input length is between 4k and 8k, NTK+bias gets the best results and NTK-weighted baseline is more robust on 16k input tokens.

heading: B Data Collection and Annotation for L-Eval
text: In our pursuit of diverse, comprehensive, and relevant data, we sourced datasets from a wide array of platforms and sources. These datasets represent various facets of everyday life and specialized fields and present different challenges for LCLMs. We leveraged resources from previous open-source datasets, Coursera subtitles, earning call transcripts from corporate websites, GitHub, etc. The instruction styles in L-Eval include multiple-choice questions, school math with many examples, key topics retrieval from lengthy dialogues, text summarization, and abstractive question answering, encompassing a wide range of tasks. The construction of each dataset and our effort to make it more challenging are as follows.
For the annotation process of dataset created in this work, we engaged three PhD students as annotators. To measure the label agreement, we employed Fleiss’s kappa. The label agreement of Coursera was 0.712 and the label agreement for SFiction was 0.775.For the CodeU dataset, our efforts were concentrated on building the long code database and selecting functions for use. The ground truth answers are provided by executing the code.

heading: B.1 Data Annotated from Scratch

heading: B.1.1 Coursera (Advanced lectures)
text: This dataset originates from the Coursera website13. We selected and completed 4 courses:
1. Ask Questions to Make Data-Driven Decisions,
2. Data Scientist’s Toolbox,
3. Process data from dirty to clean,
4. Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization.
Example 1
Input: <A long lecture>\n\n
Question: When working with a new team, which of the
following actions can help you to adapt to different
communication expectations? Select all that apply.
A. Ask questions when you are unsure of something
B. Learn the team’s preferred communication style
C. Observe how teammates communicate with each other
13https://coursera.org/
D. Ignore the team’s communication preferences and use
your own style
\n\n Answer:
Ground truth: ABC
The input long document is the subtitles of the videos and we merge courses in one week into one single long lecture. Questions and the ground truth answers are labeled by the authors. The instruction style of Coursera takes the format of multiple choice. In order to increase the difficulty of the task, we have set multiple correct options. Failure to select all correct choices will result in receiving only a quarter of the total points for that question.

heading: B.1.2 SFcition (Scientific fictions)
text: We annotate this sub-task to test the loyalty of the LCLM to the input context. LLMs have acquired a significant amount of commonsense in their pretraining corpus known as parametric knowledge (Wang et al., 2023). However, we argue that in LCLMs, contextual knowledge is more crucial than parametric knowledge. In real-world applications, many long documents are private and can never be seen during pretraining. It may contain new knowledge or describe a new world which may be opposite to the pretraining knowledge. The language model should follow contextual knowledge instead of parametric knowledge. To simulate this scenario, we annotate a science fiction dataset consisting of True or False questions. The original works are sourced from SFGram14. We manually select documents that fit our experimental conditions and annotate them with questions and corresponding answers. Most of the answers to these questions contradict real-world principles and do not comply with actual physical laws, such as the statement: Humans have invented the time machine. As a result, open-source models have very serious hallucination problems which in turn help them acquire a high score on this dataset. So we also give the answer based on real-world knowledge, and the final accuracy is calculated by the average of loyalty and factuality.
Example 2
Input: <A scientific fiction>\n\n
Question: We cannot get to the centre of the Earth,
True or False? Answer this question based on the world
described in the document.
14https://github.com/nschaetti/SFGram-dataset
Ground truth: False
Question: We cannot get to the centre of the
Earth, True or False? Answer this question based on
the real-world knowledge and facts up until your last
training.
Ground truth: True

heading: B.1.3 CodeU (Python)
text: This dataset is used to test the capability of understanding long code. Given a lengthy code base, we will call some functions defined in the codebase and the model should infer the final output of the program. We mainly use source code from Numpy15. We also write a string processing codebase containing more than 100 functions that take a string as input such as extracting the email address from an input string. To prevent LLMs from answering the question based on their parametric knowledge, we replace the original function name defined in Numpy with Op1, Op2..., OpN. The Language Model (LLM) should first identify where the function is called and determine which functions are invoked, ultimately ascertaining the results of the operations. CodeU represents the most challenging task within L-Eval. Even the most potent model, GPT-4-32k, achieves an accuracy of only 25.55%.
Example 3
Input: <The beginning of a lengthy Python program>
def Op1(): ...
def Op2(): ...
args = [4,5,6]
output = Op1(args)
print(output)
<The rest of the program>\n\n
Instruction: What is the output of this program? Please
carefully read through these code snippets and comments.
You should first identify where the functions are
defined and then figure out what they do.
\n\n let’s think step by step:
Ground truth: [1,2,3]

heading: B.1.4 LongFQA (Finance)
text: We find that there is a lack of long open-ended QA datasets in finance. The long context finance dataset is derived from earnings call transcripts obtained from the Investor Relations section of the company websites. We annotate 6 transcripts
15https://github.com/numpy/numpy
from 6 different incorporations, including Lumentum Oclaro16, Theragenics17, FS KKR Capital Corp18, LaSalle Incorporated19, Renewable Energy Group20 with 54 questions based on these transcripts.
Example 4
Input: <A long document>\n\n
Instruction: You are asked to act as a member of
the Financial Results Conference Call and answer the
question: What major actions has Greg Dougherty, the
CEO of Oclaro, highlighted as being undertaken by the
company for its restructuring plan? \n Answer this
question with xx words.
Ground truth: Oclaro has been implementing a
significant restructuring plan, which includes closing
our second major...

heading: B.2 Data Re-Annotated Based on Existing Datasets
text: B.2.1 GSM(16-shot)(Grade school math)
This dataset is derived from 100-grade school math problems in the GSM8k dataset (Cobbe et al., 2021). Increasing the number of high-quality and complex examples usually has a positive effect on solving math problems. We construct 16 incontext examples with lengthy Chain-of-thought for this task where 8 examples come from chainof-thought-hub21 using the hardest prompt and the remaining 8 examples are constructed by us. We selected 8 questions from GSM8k based on their difficulty and annotated the solving process. Models with 2k or 4k context length face difficulties while encoding the 16 examples. We experiment with the newly constructed examples and it perform better than only encoding 8 examples. Concretely, the accuracy rises from 79 (8-shot) to 84 (16-shot) when using turbo-16k-0613 as the base model.
16https://investor.lumentum.com/overview/ default.aspx
17https://www.sec.gov/Archives/edgar/data/ 18https://www.fskkradvisor.com/ investor-relations/ 19https://ir.jll.com/overview/default.aspx 20https://www.regi.com/resources/ press-releases 21https://github.com/FranxYao/ chain-of-thought-hub/blob/main/gsm8k/lib_prompt/ prompt_hardest.txt
Example 5
Input: <example 1> \n\n <example 2> \n\n ... <example
n> \n\n
Question: Janet’s ducks lay 16 eggs per day. She eats
three for breakfast every morning and bakes muffins
for her friends every day with four. She sells the
remainder at the farmers’ market daily for $2 per fresh
duck egg. How much in dollars does she make every day
at the farmers’ market? \n\n
Let’s think step by step
Ground truth: 18

heading: B.2.2 QuALITY (Gutenberg)
text: This dataset is sourced from the multiple choice QA dataset QuALITY (Pang et al., 2022) which contains multiple-choice questions derived from the literature on Gutenberg. We filter 20 long stories and 202 questions and correct/delete questions with annotation errors. We found that most questions in QuALITY can be solved by extracting paragraphs from long texts. We further enhance some synthesis questions that need a global understanding of the document. Examples of the annotated synthesis questions are as follows:
1. What can we infer from the longest sentence in the story?
2. The longest dialogue is spoken by whom?
3. Extract names mentioned in the longest sentence in the story.
4. How many words are there in the story?

heading: 5. How many sentences are there in the story?
text: Example 6
Input: <A long story>\n\n
Instruction: Why did Syme accept the mission with Tate?
(A) He needed a way back to Earth
(B) He felt he would collect a reward along the way
(C) He respected Tate
(D) He had no plan for his life, so he jumped on the
adventure
Ground truth: (B) He felt he would collect a reward
along the way
The reference source sentences are automatically located and the ground truth answers are manually annotated by us. An example of the original question in QuALITY is listed above.

heading: B.2.3 TopicRet (Lengthy conversation)
text: This dataset comes from the LongChat repository (Li et al., 2023a)22, and its task style focuses on retrieving topics from extensive chat histories. Recent studies show that language models are good at retrieving information from the very beginning or end of its input context but are usually lost in the middle (Liu et al., 2023).
Example 7
Input: <A long conversation > \n\n
Question: What is the second topic we discussed? Only
give me the topic name. Do not summarize yourself.
Ground truth: The future of space tourism
To make the task more challenging, we enhance the original task by asking the model to extract the second and the third topic.

heading: B.2.4 Openreview (Papers)
text: This task aims to help researchers working on scientific papers by dealing with tasks like correcting grammar errors or typos and writing some sections. We include 3 tasks in the paper writing assistant task of L-Eval: 1) writing an Abstract section, (2) writing a Related Work section, and (3) finally giving a review of this paper including valuable suggestions and questions. Notably, we discourage reviewers from using LLMs for reviews. Our aim is to assist authors in further improving their papers. Therefore, we ask the model to give some valuable suggestions and raise some questions for authors. We filter 20 papers with well-written reviews for L-Eval. We use the processed PDF files from Yuan et al. (2021).
Example 8
Input: <A long paper>\n\n
1. Instruction: Please generate the Abstract section
for this paper. \n Answer this question with xx words.
2. Instruction: Please summarize related work and you
should include the following works [a list of papers].
\n Answer this question with xx words.

heading: 3. Instruction: Please write a review for this paper
text: and you should provide some suggestions and raise some
questions in your review. \n Answer this question with
xx words.
Ground truth: Conventional out-of-distribution (OOD)
detection schemes based on variational autoencoder or
Random Network Distillation (RND) have been observed
22https://github.com/DachengLi1/LongChat
to assign ...

heading: B.2.5 SPACE (Reviews)
text: The review (opinion) summarization aims to summarize the reviews from customer reviews on a restaurant or hotel. We obtain 20 samples from the validation and test set of SPACE (Angelidis et al., 2021) where human-written abstractive summaries are created for 50 hotels based on 100 input reviews each. SPACE consists of customer reviews of hotels from TripAdvisor, with 1.1 million training reviews for 11,000 hotels. The original task asks the model to summarize hotels from multiple aspects: food, location, cleanliness, etc. We construct the instructions for review summarization with GPT-4 and some examples.
Example 9
Input: <Multiple reviews>\n\n
Instruction: Give a broad summary of guest impressions
about Doubletree by Hilton Seattle Airport. \n Answer
this question with xx words.
Ground truth: The staff are friendly and exceptional.
Every room (lobby included) was very clean. They are
spacious, very quiet, and come with a coffee maker...

heading: B.3 Data Cleaned from Existing Datasets

heading: B.3.1 TOFEL (English tests)
text: This dataset is sourced from the TOEFL Practice Online and we collect the data from TOEFLQA (Tseng et al., 2016; Chung et al., 2018) and all lectures from a single TPO have been consolidated into one lengthy lecture. After the consolidation, we select the top 15 longest lectures.
Example 10
Input: <Multiple long lectures> \n\n
Question: why did Frantzen go to the sales barn
A. to study human form and movement
B. to earn money by painting portraits
C. to paint farm animals in an outdoor setting
D. to meet people who could model for her painting
\n\n Answer:
Ground truth: A
B.3.2 CUAD (Law) Questions on the Legal domain are drawn from the CUAD (Contract Understanding Atticus Dataset) dataset (Hendrycks et al., 2021b) designed for supporting NLP research for automating legal contract
review. We manually filter 20 documents with annotated QA pairs from CUAD.
Example 11
Input: <Legal contracts> \n\n
Instruction: Highlight the parts (if any) of this
contract related to Ëxpiration Dateẗhat should be
reviewed by a lawyer. Details: On what date will the
contract’s initial term expire? \n Answer this question
with xx words.
Ground truth: The term of this Agreement shall commence
on the Effective Date and shall continue in full force
and effect for an initial period of five (5) years.
B.3.3 MultiDoc2Dial (Dialogues over multi-documents)
This dataset is sampled from the MultiDoc2Dial dataset (Feng et al., 2021) which aims to model goal-oriented dialogues grounded in multiple documents. It contains dialogues from 4 different domains: Finance, Travel, Entertainment, and Shopping. Each dialogue in the dataset is grounded in 2-5 relevant documents covering different topics within the domain.
Example 12
Input: <Multiple long documents> \n\n
Instruction: How long will Driver’s Ed courses be valid
for? \n Answer this question with xx words.
Ground truth: For roughly 1 one year. Maybe longer
depending on the course.

heading: B.3.4 Natural Questions (Wikipedia)
text: We filter 20 Wikipedia long documents from Natural Question (Kwiatkowski et al., 2019) on Google Research datasets. Questions that can be answered with the same documents are merged, and duplicate questions are removed.
Example 13
Input: <Documents from Wiki>\n\n
Instruction: when did season 2 of handmaid’s tale start?
\n Answer this question with xx words.
Ground truth: April 25, 2018

heading: B.3.5 NarrativeQA (Narratives)
text: This dataset is collected from NarrativeQA (Kočiský et al., 2017) which has the longest document length in L-Eval. The original question-answering dataset was created using entire books from Project Gutenberg23 and movie
23https://www.gutenberg.org
scripts from various websites. Summaries of the books and scripts were taken from Wikipedia and given to annotators. Our work focuses on correcting the annotation errors. For example, some questions reference a main character who does not appear in the input document.
Example 14
Input: <A long novel>\n\n

heading: Instruction: Why did Mary pay off the debt for Ann’s
text: family? \n Answer this question with xx words.
Ground truth: Mary was in love with Ann.

heading: B.3.6 Qasper (Papers)
text: This dataset is filtered from the Qasper dataset (Dasigi et al., 2021), which is a questionanswering resource focused on NLP papers. The dataset was constructed using NLP papers that were extracted from the Semantic Scholar Open Research Corpus (S2ORC). After filtering, we remove the unanswerable questions and the extractive version answers. We also discovered instances where identical questions yielded contradictory answers. We addressed this issue by meticulously reviewing the paper and rectifying the incorrect responses.
Example 15
Input: <A long paper>\n\n
Instruction: How did they obtain the dataset? \n Answer
this question with xx words.
Ground truth: public resources where suspicious Twitter
accounts were annotated, list with another 32 Twitter
accounts from BIBREF19 that are considered trustworthy.

heading: B.3.7 GovReport (Government Reports)
text: This dataset is filtered from the government report summarization dataset (Huang et al., 2021), the dataset consists of long reports written by U.S. government research agencies such as the Congressional Research Service and Government Accountability Office. The documents and summaries in this dataset are longer compared to other long document summarization datasets. We manually filter 13 documents with human-written summaries from the original dataset.
Example 16
Input: <A government report>\n\n
Instruction: Please help me summarize this government
report. \n Answer this question with xx words.
Ground truth: The President of the United States has
available certain powers that may be exercised in the
event that the nation is threatened by crisis, exigency,
or emergency circumstances...

heading: B.3.8 QMSum (Meetings)
text: This dataset is sourced from the QMSum (Zhong et al., 2021). It contains query-based meeting summarizations, which aims to summarize the document given a specific aspect. We selected 20 meeting transcripts accompanied by queries, specifically choosing those that could not be easily addressed through retrieval methods.
Example 17
Input: <Meeting trancripts>\n\n
Instruction: What was agreed upon on sample
transcripts? \n Answer this question with xx words.
Ground truth: To save time, speaker mn005 will only
mark the sample of transcribed data for regions of
overlapping speech, as opposed to marking all acoustic
events...

heading: B.3.9 Multi-News (News)
text: This dataset is sourced from Multi-News (Fabbri et al., 2019), which contains news articles and human-written summaries compiled from newser.com. Each summary in the original MultiNews dataset is derived from multiple short news articles. We selected 10 articles for inclusion in the L-Eval benchmark.
Example 18
Input: <News articles>\n\n
Instruction: Please summarize these news articles. \n
Answer this question with xx words.
Ground turth: Why did Microsoft buy Nokia’s phone
business? We now know Microsoft’s answer: The computing
giant released a 30-slide presentation today arguing
that the move will improve Microsoft...

heading: B.3.10 BigPatent (Patents)
text: This dataset is derived from the BigPatent (Sharma et al., 2019) project, which consists of 1.3 million records of U.S. patent documents along with human-written abstractive summaries, we select 13 patents from the original dataset.
Example 19
Input: <A long patent>\n\n
Instruction: You are a patent examiner. Please write a
summary of this patent. \n Answer this question with
xx words.
Ground truth: The invention provides a method and system
for cleaning pet paws by providing a bounded container
containing...

heading: B.3.11 SummScreen (TV show)
text: This dataset originates from the SummScreen (Chen et al., 2022). The original dataset is an abstractive summarization dataset combining TV series transcripts and episode recaps, constructed from fan-contributed websites. We use 13 of these transcripts in L-Eval.
Example 20
Input: <TV series transcripts> \n\n
Instruction: Write a summary of the scene. \n Answer
this question with xx words.
Ground turth: Feeling guilty over Phoebe missing out
on London, the gang plans a weekend trip to Atlantic
City, but just as they are about to leave...

heading: Future Work
text: Ground truth: The future of space tourism
To make the task more challenging, we enhance the original task by asking the model to extract the second and the third topic