MLE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models (2024)

Abstract: Wai-Chung Kwan*, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li,Yuxin Jiang Lifeng Shang, Qun Liu, Kam-Fai Wong The Chinese University of Hong Kong Huawei Noah’s Ark Lab The Hong Kong University of Science and Technology MoE Key Laboratory of High Confidence Software Technologies {wckwan,kfwong}@se.cuhk.edu.hk {zeng.xingshan,wangyufei44,sun.yusen1,liliangyou,Shang.Lifeng,qun.liu}@huawei.com yjiangcm@connect.ust.hk Abstract

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15568–15592 August 11-16, 2024 ©2024 Association for Computational Linguistics
Our evaluation of 11 prominent LLMs reveals that 1) Current LLMs struggle to understand long context, particularly when tasks require multiple-span attention. 2) Semantic retrieval is more difficult for competent LLMs. 3) Models fine-tuned on longer text with position interpolation have comparable performance to those using Neural Tangent Kernel (NTK) aware scaling methods without fine-tuning. We make our benchmark publicly available to encourage future research in this challenging area 2.

heading: 1 Introduction
text: Large language models (LLMs) are gaining traction in addressing diverse NLP challenges. LLMs, mostly transformer-based models (Vaswani et al.,
*Work done during an internship at Huawei Noah’s Ark Lab.
1The released benchmark would contain samples up to 32k words. Even longer samples and other types of tasks can be constructed using our method.
2Code and data are available at https://github.com/ KwanWaiChung/M4LE.
2017), are trained on a large amount of data with numerous parameters (Ouyang et al., 2022; Touvron et al., 2023b). These models have demonstrated impressive capabilities across a wide range of tasks (Brown et al., 2020; Schick et al., 2023; Shen et al., 2023; Bang et al., 2023). As LLMs continue to evolve, their ability to handle long-sequence tasks, such as extracting specific information from or summarizing lengthy documents, has become an important and competitive feature (Du et al., 2022; Chiang et al., 2023; Li et al., 2023). Therefore, a comprehensive, fair, and objective benchmark to evaluate the long-sequence capabilities of models is necessary for the progress of LLMs.
Despite numerous efforts to develop benchmarks for assessing the knowledge or reasoning ability of LLMs (Hendrycks et al., 2021; Suzgun et al., 2022; Huang et al., 2023), comprehensive evaluation of their long-context understanding ability has received limited attention. Recent concurrent works, such as L-Eval (An et al., 2023) and LongBench (Bai et al., 2023), primarily rely on existing long-sequence NLP datasets which usually limit the task diversity and flexibility in conducting length-control experiments. They lack an objective and comprehensive understanding of the model’s capability across different dimensions of long sequences.
In this study, we aim to maximize the diversity of constructed tasks and analyze the long-context capabilities of LLMs from a user’s practical perspective. We discovered that when processing instructions based on long sequences, the essential components for task completion can be classified as single-span, multiple-span, or global, based on relevance. Building on this and considering how to locate the relevant information, we categorize long-context understanding into five distinct abilities and introduce an automated method to transform short-sequence tasks into a comprehensive long-sequence scenario encompassing all these ca-
15568
pabilities. As a result, M4LE is proposed, a multiability, multi-range, multi-task, and multi-domain long-context evaluation benchmark for evaluating LLMs’ ability to handle long inputs (Figure 1).
• Multi-ability: M4LE includes tasks with five different types of understanding abilities, determined by whether single or multiple parts of the ongoing context are relevant to the current tasks and whether explicit or semantic hints are used in the question.
• Multi-range: Each task in M4LE consists of samples with variable lengths, from 1K to 8K words, divided evenly into five buckets to measure the effect of length on model performance.
• Multi-task: M4LE encompasses 36 datasets covering 11 task types, including original tasks such as classification and summarization, and their combination for more complex scenarios.
• Multi-domain: M4LE spans a wide variety of domains, including Wikipedia, academic, news, E-Commerce, etc., prompting diversity and comprehensiveness.
Table 1 compares our benchmark with the existing similar benchmarks. M4LE targets comprehensively evaluating LLMs’ long-context understanding capabilities across different abilities and
length ranges, rather than simply assessing naturally long input tasks. Therefore, the tasks in M4LE are constructed from both existing long-context datasets and short-context datasets widely used in the NLP community, where short instances can be aggregated into long-context ones with designed procedures covering different abilities with varied instructions. Our approach is able to extend existing datasets to arbitrary context lengths. While the generated instances may not perfectly mimic natural long-form texts like lengthy reports, we believe that evaluating these instances effectively test model performance across the five defined abilities, thereby adequately reflects the model’s longcontext understanding capabilities. Moreover, this construction method can effectively prevent data leakage issues since the models are unlikely to have been trained on similarly constructed datasets.
We conducted a systematic evaluation over 11 well-known LLMs, especially those claimed to support long inputs, with M4LE. This involves evaluating their long-context understanding ability across different length ranges and their performance in our proposed five different abilities. We also delve into the factors influencing long-context understanding capability, including LLMs performance under different languages and the positioning of relevant information (Liu et al., 2023). We find that current LLMs still struggle to understand long-context inputs, especially when multiple-span attention is required. While semantic retrieval is considered more complex than explicit, the consistent perfor-
mance drop in this scope can only be observed on competent models. A more effective fine-tuning approach deserves exploration, as current methods show no significant improvement over simple Neural Tangent Kernel (NTK) aware scaling methods. We also observe that language differences and the positioning of relevant information impact the long-context understanding capabilities.

heading: 2 Related Work

heading: 2.1 Long-Context Modelling for LLMs
text: To address length extrapolation challenges in LLMs beyond the training context window, several methodologies have emerged. Position embeddings such as Alibi (Press et al., 2022) and XPos (Sun et al., 2023) have been developed. Alibi employs an exponential decay on the attention matrix to mitigate out-of-distribution positions’ influence, while XPos introduces a block-wise causal attention mask. While these techniques require integration during training, alternative approaches enhance existing RoPE-based LLMs (Su et al., 2021), notably LLaMA (Touvron et al., 2023a), LLaMA 2 (Touvron et al., 2023b), and PaLM (Chowdhery et al., 2022). Concurrently, kaiokendev (2023) and Chen et al. (2023) propose extending context length by modifying RoPE through Position Interpolation and subsequent limited data finetuning. Another line of research introduces fine-tuning free approaches (bloc97, 2023; emozilla, 2023; Peng et al., 2023), including NTK-aware and dynamic NTK interpolations.

heading: 2.2 Existing Evaluation Benchmarks for LLMs
text: As LLMs have demonstrated superior performance in a wide range of NLP tasks, comprehensively and effectively evaluating their ability becomes increasingly critical. Many of the research efforts focus on developing benchmarks for specific knowledge types (Hendrycks et al., 2021; Zhong et al., 2023) and specific task families (Chen et al., 2021;
Cobbe et al., 2021). For more details, we refer readers to the recent LLMs evaluation survey (Chang et al., 2023; Wang et al., 2023). Several preliminary studies have begun to assess the model capability on long context input. Long Range Areana (Tay et al., 2020) verifies the capability of transformerbased models to handle various long sequence inputs, such as languages, vision tokens, and symbols. SCROLLS (Shaham et al., 2022) simply collects a set of naturally long NLP benchmarks covering multiple tasks and domains. Recently, ZeroSCROLLS (Shaham et al., 2023), L-Eval (An et al., 2023) and LongBench (Bai et al., 2023) are proposed to evaluate long text modelling capability of LLMs. However, these benchmarks are mainly compiled from a set of existing long NLP benchmarks, thereby suffering from data diversity (i.e., limited evaluation patterns) and data leakage (i.e., LLMs potentially already using these benchmarks for pre-training or alignment). In contrast, M4LE not only constructs evaluation instances from various tasks, domains, and length ranges but also covers three types of attention spans, offering a comprehensive evaluation of LLMs’ long text capability.
3 M4LE
This section outlines the M4LE benchmark’s rationale, design principles, data sources, and task construction methodologies. M4LE is designed to comprehensively evaluate large language models’ (LLMs) abilities in understanding long contexts. It covers a wide range of tasks, domains, and context lengths, ensuring a thorough assessment of LLMs’ competencies in this crucial area.

heading: 3.1 Design Principle
text: Each sample in M4LE is a tuple of ⟨Task description, Context, Instruction, Response⟩. To follow the instructions, LLMs must identify relevant information within a lengthy context. This information can be a single text segment (single-span), multiple
text segments (multiple-span), or the entire context (global). The models locate these segments either through direct hints (explicit) or inferred meaning (semantic) in the instructions. We categorize the understanding ability into five distinct types: explicit single-span, semantic single-span, explicit multiple-span, semantic multiple-span, and global context understanding (Figure 1). This classification helps in assessing the models’ comprehension capabilities.
To ensure a comprehensive evaluation, we prioritize task diversity in two aspects:
• Data Source: We select widely-used Chinese and English datasets in NLP which cover a variety of representative task types (e.g., QA, Summarization) and domains (e.g., News, Wiki, Web). In addition, we introduce tasks that integrate multiple task types, like Classification and Retrieval. These newly integrated tasks help measure LLMs’ ability to solve more complex tasks.
• Length Range: It is important to reveal how LLMs perform on various lengths of contexts. In our benchmark, we evenly divide samples into buckets according to their context lengths. In addition, in order to alleviate the effects of the location of relevant parts in context (Liu et al., 2023), we intentionally construct instances with the relevant paragraphs uniformly distributed in the input context.
By focusing on these five core abilities and maximizing task diversity, M4LE offers a comprehensive assessment of LLMs’ long-context understanding capabilities.

heading: 3.2 Data Collection
text: We collect established datasets, both in English and Chinese, to cover a broad range of tasks and domains. We not only select datasets featuring long inputs, but also include datasets with shorter inputs for our customized construction, and at the same time, enriching the domain variety. The shortcontext datasets can be adapted to longer contexts using our designed process, which will be introduced in the next subsection. Below we describe the datasets selected in the benchmark briefly.
Question-Answering (QA): We include TriviaQA (Joshi et al., 2017), a single-document QA dataset based on web snippets and Wikipedia, with documents extended to 12k words. Additionally, NQOpen (Lee et al., 2019), HotpotQA (Yang et al.,
2018), and DRCD (Shao et al., 2019) are included, all of which are based on Wikipedia articles. We further collect NewsQA (Trischler et al., 2017) and DuoRC (Saha et al., 2018), both in English and constructed from news articles and movie plots. We also add C3 (Sun et al., 2021), a Chinese dataset comprising textbook questions.
Classification: We incorporate BIGPATENT (Sharma et al., 2019) which includes long patent documents, and MNDS News (Petukhova and Fachada, 2023) in English and THUCNews (Hu et al., 2019) in Chinese which would be further processed for different abilities. We also utilize a sentiment classification dataset collected from e-commerce platforms (SophonPlus, 2013).
Summarization: For English, we include Arxiv, Pubmed (Cohan et al., 2018), BIGPATENT (Sharma et al., 2019), and Booksum (Kryscinski et al., 2022), where the corresponding domains span across academic, medical, patent documents and books. We also introduce shorter summarization datasets enabling extension, such as CNNNews (See et al., 2017) and MNDS News, featuring news articles, and Wikihow (Koupaee and Wang, 2018). For Chinese, we incorporate CNewsum (Wang et al., 2021), CLTS+ (Liu et al., 2022), and News2016 (Xu, 2019), all constructed from long news articles. The LCSTS (Hu et al., 2015) dataset contains shorter news articles, while CEPSUM (Li et al., 2020) comprises product descriptions from e-commerce platforms. We also use NCLS (Zhu et al., 2019) to establish a bilingual task that generates a Chinese summary for a specific English news article.
Natural Language Inference (NLI): We construct two tasks using English and Chinese Wikipedia articles from WikiText-103 (Merity et al., 2016) and Wiki2019zh (Xu, 2019), respectively.
Translation: Three translation datasets are included, depending on sentence-level translation alignments to form long contexts, including Tedtalks (Qi et al., 2018), OpenSubtitles (Lison and Tiedemann, 2016), and News commentary (Tiedemann, 2012).
Retrieval: Lastly, we construct two retrieval tasks from the same datasets used for the NLI task for both languages. Since M4LE comprises numerous tasks combined with retrieval capability, we do not construct additional standalone retrieval datasets.
Each instance comprises a tuple containing the task description, context, instruction, and response. 1 : The process begins by estimating the number of samples needed to achieve the desired target length. This is accomplished by dividing the median length of the context in the dataset by the target length. Subsequently, N instances are sampled from the source dataset. 2 : The context of each sample is then marked with an explicit identifier and combined. 3 : For single-span tasks, we uniformly sample one context to construct the query. For multi-span tasks, multiple contexts are sampled. We incorporate the explicit identifiers for explicit tasks and semantic hints for semantic tasks in the instruction. 4 : If the total length exceeds the target length, the process returns to step one. Otherwise, the constructed sample is added to the test dataset.

heading: 3.3 Task Construction
text: This subsection details the dataset construction process of the evaluation benchmark. We construct test instances with diverse length ranges by transforming instances from collected datasets.
Figure 2 illustrates the construction process. To construct a test instance for a specific task with a target length range K, we first sample N instances from a single source dataset. These original instances contain context, such as an article, a talk transcript, or several text segments. We then concatenate their context paragraphs into a single sequence as “Context“, marking each paragraph with an explicit identifier at the beginning for indexing. The value of N is determined by dividing K by the dataset’s median context length. For each task, we manually craft a description and make sure
LLaMA2-7B-Chat can understand it through preliminary testing with a few examples. We further provide instructions to guide the model to locate relevant information within the context using paragraph identifiers for explicit tasks and semantic hints for semantic tasks. This approach extends existing datasets with short contexts to accommodate arbitrary context lengths. Table 2 provides an overview of the constructed datasets in M4LE. Appendix A provides the detailed statistics of the datasets used. In the following sections, we elaborate on the instruction construction process for each of the five abilities.
Explicit Single-Span Understanding. Instructions for tasks within this scope should direct models to complete the task based on a specific paragraph, with explicit hints to be located. For in-
stance, in a question-answering task, the model might be asked to answer a question based on paragraph II. This approach has been used to construct ten unique datasets covering a wide range of task types and domains for the ability. Consequently, the task types are a fusion of retrieval and their original task, such as classification, which is labeled as “CLS + RET”.
Semantic Single-Span Understanding. Analogous to explicit single-span understanding, the instructions for the tasks long to this ability instruct models to complete tasks based on a designated paragraph. Rather than using explicit identifiers, we provide hints about the paragraph, and models are tasked with retrieving it based on semantic information. For example, in a translation task, the model might be prompted to translate a paragraph associated with sports. Tasks within this ability are designed to introduce increased complexity and challenges since semantic-level retrieval necessitates the model to understand all paragraphs to pinpoint the right one. We have constructed nine distinct datasets aligned with this ability.
Explicit Multiple-Span Understanding. We add further complexities to the tasks within this ability. Specifically, models are tasked with handling assignments related to multiple, disjoint paragraphs within the lengthy input context. This could necessitate addressing several original instances, for example, summarizing the first and the third paragraphs. Despite these complexities, the instructions for this ability continue to utilize explicit hints. We have constructed four distinct datasets to align with this ability.
Semantic Multiple-Span Understanding. We replace the explicit hints in explicit multiple-span understanding with semantic ones, resulting in the instructions for tasks in this scope. We’ve developed three distinct datasets of high complexity in line with this. Within this ability, we’ve incorporated counting tasks (labeled as “CNT”), which demand the counting of relevant paragraphs. Such tasks pose a challenge since counting is not an innate function of language models.
Global Context Understanding. Finally, we present tasks in global context understanding, which is a special case within our construction process. When the original instances have sufficiently extensive context, such that the target length range K can be attained with N = 1, we directly employ
them for the associated tasks, indicating that the entire context is relevant to the task completion, and global understanding is required. Within this category, we have included ten different datasets.

heading: 3.4 Models
text: We introduce the five families of LLMs evaluated in this study, comprising a total of 11 models.
LLaMA 2: It is a family of LLMs that support a maximum 4k input length (Touvron et al., 2023b). These models use rotary positional embeddings (RoPE) (Su et al., 2021). LLaMA 2 has 7B, 13B and 70B variant. We focus on its 7B and 13B models in this section. We also include their aligned versions: LLaMA2-7B-Chat and LLaMA2-13BChat.
Vicuna: We employ Vicuna-7B-v1.5-16K and Vicuna-13B-v1.5-16K (Chiang et al., 2023), finetuned based on the LLaMA2 models with 125k conversational data, collected from ShareGPT with context length up to 16K tokens using linear positional interpolation (Chen et al., 2023).
LongChat: We leverage LongChat-7B-v1.532K and LongChat-13B-16K (Li et al., 2023), finetuned on 80K and 18K conversations respectively, with context lengths up to 32K and 16K tokens, respectively. They utilize linear positional interpolation.
ChatGLM2: ChatGLM2-6B and ChatGLM26B-32K are based on the GLM (Du et al., 2022) models. Similar to LLaMA2, ChatGLM2 leverage RoPE. Both models are further refined on 8K and 32K input data, respectively, using linear positional interpolation.
GPT-3.5-Turbo: It is a closed-source language model developed based on InstructGPT (Ouyang et al., 2022). Analogous to LLaMA 2, it is finetuned with instruction data and refined by RLHF. We use the GPT-3.5-Turbo-16K variant 3, which supports a 16K context length.

heading: 3.5 Inference Details
text: Apart from the tuples introduced in Section 3.1, we also employ a concise and short in-context example, from the same dataset, to demonstrate the desired output format. Several full examples used in this work can be found in Appendix E. The main goal of M4LE is to evaluate the performance variations across different context length buckets and abilities. We did not perform extensive prompt engineering
3We use the GPT-3.5-Turbo-16K-0613 api from https://cuhk-api-dev1-apim1.developer.azure-api.net.
for each task to obtain the optimal performance. Instead, we focus on analyzing performance changes of particular LLMs with longer input contexts.
Since LLaMA 2 models were trained on data within 4k tokens, we used dynamic NTK-aware RoPE scaling (emozilla, 2023; Peng et al., 2023) for context longer than 4k. We used 16 floating points precision during inference. To facilitate fair comparisons across various tasks with different metrics, we normalized the raw performance score r(M, l) (i.e., the performance of LLM M at context length l) as follows:
r̂(M, l) = r(M, l)
r(GPT-3.5-Turbo-16K, 1000) + r(M, l)
r̂(M, l) provides a measure of how other models perform relative to GPT-3.5-Turbo-16K in the length range bucket of 0-1000 tokens, and how their performance deteriorates with longer input.

heading: 3.6 Results
text: Figure 3 illustrates the changes in normalized average scores for various evaluated models as context lengths extend, and Figure 4 depicts their ability in the context length range of 0-1000, 1000-4000, and 4000-8000 (the full results for each task can be found in Appendix C). Based on the figures, several key observations emerge:
The performance of all models significantly deteriorates with increasing context lengths. This trend is expected, given that a longer context might necessitate more sophisticated modelling capabilities. It suggests that these LLMs struggle with understanding extensive context. The performance gap between ChatGPT and most open-source models widens as context length increases. This is largely because open-source models tend to exhibit a steeper decline, particularly when the context length exceeds 4k. For example, Vicuna-13B-v1.516K achieves competitive performance, compared to GPT-3.5-Turbo-16K, in the 0-4K length range, but its performance drops significantly after that. A notable exception is ChatGLM2-6B-32k which achieves similar performance when testing on 6K and 8K instances and is only surpassed by GPT3.5-Turbo-16K on 8K instances.
Fine-tuning with additional long context data does not offer a significant advantage over simply NTK scaling for understanding long contexts. Both Vicuna and LongChat models are
claimed to support long context as they are directly fine-tuned with longer context data. However, their performance still drops quickly when context length exceeds 4k, with no additional advantage compared to LLaMA2 models, which are trained only on 4k data and merely equipped with NTK scaling method when context length exceeds 4k. This suggests that existing long-context finetuning methods contribute minimally to improving long context understanding and a more efficient and effective way to enhance long context understanding ability is needed.

heading: Multiple-span understanding is more difficult,
text: and semantic retrieval is even harder for competent models. There is a significant drop in performance on tasks requiring multiple-span attention as context lengthens. This is expected since attending to multiple positions is naturally harder than a single position, and it might require additional ability to distinguish and determine compared to global understanding. Surprisingly, semantic retrieval is only more challenging for GPT-3.5-Turbo-16K, the most competent model in the experiment. We hypothesize that this is because explicit retrieval, looking for relative information by an identifier, is an unnatural task for less competent and generalized LLM. On the contrary, semantic retrieval is more similar to tasks like QA that these models experienced during instruction fine-tuning.

heading: 3.7 Ablation Study
text: We perform further analysis to understand how models behave in different languages and locations of the supporting document.
Impact of language differences on long-context understanding. Tasks in different languages may have distinct ability requirements due to the nature of languages and the effects of tokenization. While most models presented in this study are primarily trained on English data, we aim to assess the influence of language differences on the results. In Figure 5, we compare the performance of the top-performing models (namely ChatGPT, ChatGLM2, Vicuna, and LongChat) in both Chinese and English tasks to determine if their long-context understanding abilities differ across languages.
We observe a comparable decline in performance for both GPT-3.5-Turbo-16K and ChatGLM2-6B32K across the two languages. However, the Vicuna and LongChat models exhibit a more pronounced performance drop in Chinese. This sug-
gests that the degradation of understanding ability when the context length increases is not unique to English. Furthermore, the diversity of data employed during fine-tuning, as highlighted by ChatGLM2’s emphasis on its bilingual (Chinese and English) proficiency during its tuning process, appears to be a successful strategy in handling bilingual long context input.
Lost-in-the-middle exists in other NLP long sequence tasks. Recently, Liu et al. (2023) find that LLMs tend to ignore the information in the middle of long input context for the task of questionanswering and retrieval. In this section, following the setup in Liu et al. (2023), we conduct a comprehensive experiment to study the impact of positions of the supporting paragraphs within the context based on our proposed M4LE benchmark. Specifically, we generate additional instances from
the tasks in M4LE, each containing an identical input but with the supporting paragraph placed at different locations. We employ four datasets for question-answering and summarization, and two datasets for retrieval tasks. The setup details are in Appendix B.
The average score for each relative position of the supporting document across the three tasks is presented in Figure 6, demonstrating that models typically perform better when the supporting document is positioned either at the beginning or the end of the context, a finding consistent with Liu et al. (2023). Consequently, this suggests that the tendency for LLM to ignore information in the middle of the context is ubiquitous across various languages, models, and tasks. This also shows the potential of M4LE in discovering interesting and unique LLMs behavior in the long context scenario.

heading: 4 Conclusion
text: In this paper, we propose M4LE for LLMs assessing their capability of long-context understanding. To establish a benchmark with diverse NLP tasks, rather than just those that are inherently lengthy, we propose a systematic method to convert short NLP task instances into long context inputs, encompassing five distinct abilities. We collect and construct in total of 36 tasks from different sources and domains covering multiple length ranges to maximize the diversity of the tasks in benchmark, with our customized construction methods which enable flexibility to extend arbitrary context lengths. We evaluate 11 well-known LLMs with our benchmark and find that current models struggle to understand long-context inputs and the corresponding performance related to ability types, data used when finetuning, and positions of the relevant information.

heading: Limitations
text: Due to computational constraints, our experiments are restricted to smaller open-source models and
lengths of up to 8K. Nevertheless, our method can create instances of arbitrary length (the released benchmark will include instances up to 32,000 words) and the analyses in this paper reveal meaningful observations concerning long-context understanding capabilties. Additionally, our study focuses on English and Chinese, the two most commonly used languages. We suggest future research to apply our methodology to construct long instances in additional languages.

heading: Acknowledgements
text: This research work is partially supported by CUHK direct grant No. 4055209 and CUHK Knowledge Transfer Project Fund No. KPF23GWP20.

heading: A Datasets
text: This section describes the datasets used in M4LE. Table 2 provides an overview of the constructed datasets.

heading: A.1 MNDS News
text: MNDS News (Petukhova and Fachada, 2023) is an English hierarchical news category classification dataset comprising 10,917 news articles from 260 sources. We only use the 17 first-level categories as the labels for this study. For multiple retrieval tasks, we randomly sample a class label that appears in the instance.

heading: A.2 THUCNews
text: THUCNews (Hu et al., 2019) is a Chinese classification dataset containing 74 million news articles from Sina, with each article belonging to one of the ten categories. We filter out the articles with the number of words less than 20. The multiple retrieval task is built similarly to MNDS News.

heading: A.3 MARC
text: MARC (Keung et al., 2020) is a dataset for the bilingual (English and Chinese) setting. It contains multilingual Amazon reviews with star ratings from 1 to 5, where 5 is the best. We use 1-star and 5-star reviews for negative and positive reviews respectively, and ask models to return all positive reviews.

heading: A.4 Online Shopping
text: Online Shopping (SophonPlus, 2013) is a Chinese sentiment dataset containing 60K product reviews from Chinese e-commerce platforms. Each review is marked as positive or negative.

heading: A.5 BIGPATENT
text: BIGPATENT (Sharma et al., 2019) consists of 1.3 million records of U.S. BIGPATENT documents across nine technological areas. The abstract of the document is used as the golden document summary.

heading: A.6 CEPSUM
text: CEPSUM (Li et al., 2020) is a dataset containing product descriptions and summary pairs collected from a popular Chinese e-commerce platform. We removed instances with less than 60 words in the product description.

heading: A.7 CNNNews
text: CNNNews (See et al., 2017) contains English online news articles from CNN, where each of it is paired with a multi-sentence summary. We preprocess the data using the script from See et al. (2017) and select the instances with at least 30 words in the article.

heading: A.8 LCSTS
text: LCSTS (Hu et al., 2015) is a Chinese summarization dataset consisting of over 2 million posts and short summary pairs collected from the Chinese microblogging website Sina Weibo. We use part two of the data, which consists of 10,666 (text, summary) pairs with a human-labeled score to indicate the relevance between the post and the summary. The score ranges from 1 to 5, where 5 indicates the most relevant. We select only the samples with a score of 5 in the relevance score.

heading: A.9 NCLS
text: NCLS (Zhu et al., 2019) is a cross-lingual summarization dataset consisting of pairs of articles in one language and summaries in another language (Chinese or English), constructed from the CNNNews and LCSTS datasets.

heading: A.10 WikiHow
text: WikiHow (Koupaee and Wang, 2018) comprises 230,000 English articles that describe a procedural task along with corresponding summaries. Each article has a title that starts with “How to”. The procedures described in the article are separated into multiple steps, where each step corresponds to a paragraph. Each paragraph has a short summary. These summaries are concatenated to form the summary of the article. We remove instances with articles that have less than 60 words.
A.11 News2016
News2016 (Xu, 2019), encompassing over 2 million Chinese news articles. Each article contains a title and keywords. The title is used as the golden summary of the news article. We remove instances with the number of words less than 200 and more than 800.

heading: A.12 Arxiv
text: Arxiv (Cohan et al., 2018) consists of 215K academic papers from arXiv.org. The abstracts of the papers are used as the golden summary.

heading: A.13 Booksum
text: Booksum (Kryscinski et al., 2022), which includes 405 English books including plays, short stories, and novels with human-written summaries for each chapter. We combine the consecutive chapters and the corresponding summaries to construct instances for any context length range.

heading: A.14 CNewsum
text: CNewsum (Wang et al., 2021) contains 304,307 Chinese news articles from different press publishers with human-written summaries.
A.15 CLTS+
CLTS+ (Liu et al., 2022) is an improved Chinese new articles summarization dataset based on CLTS (Liu et al., 2020). CLTS contains more than 180,000 Chinese long articles with human-written summaries. CLTS+ utilizes back translation to enhance the abstractiveness of the summaries.

heading: A.16 NewsQA
text: NewsQA (Trischler et al., 2017) is an English QA dataset based on 12,744 news articles from CNN. Crowdsourced workers are recruited to generate 119,633 questions and answers.
A.17 C3
C3 (Sun et al., 2021) is a Chinese textbook-based machine comprehension dataset. The questions are multiple-choice questions collected from exams for second-language Chinese learners.

heading: A.18 DuoRC
text: DuoRC (Saha et al., 2018) is an English questionanswer dataset based on 7680 movie plots collected from IMDb and Wikipedia. Crowdsourced workers are hired to create 186,089 unique question-answer pairs.

heading: A.19 NQ-Open
text: NaturalQuestions-Open (NQ-Open) (Lee et al., 2019) is an open-domain question-answering dataset based on Wikipedia documents. The questions are collected from Google Search queries. We directly use the processed version from Liu et al. (2023).

heading: A.20 DuReader
text: DuReader (He et al., 2018) is an open-domain Chinese machine reading comprehension dataset, con-
sisting of 200K questions collected from Baidu Search.

heading: B Experiment Details for Lost-In-The-Middle
text: For the experiment in Figure 6, which explores the effects of the positions of the relevant paragraphs, we additionally construct the following instances:
In the QA task, 100 instances, each comprising 20 paragraphs, are generated from NQ-Open and DuoRC for English, and from DRCD and C3 for Chinese. Similarly, for the summarization task, we generate 100 instances each from WikiHow and CNNNews for English and News2016, and LCSTS for Chinese. For the retrieval task, we formulate 200 instances each using WoW for English and DRCD for Chinese. The supporting paragraph will be evenly placed at different locations.

heading: C Main Results
text: We report the results used for plotting Figure 3.

heading: D Task Results
text: We show the results of each task in Table 7 to 45

heading: E Prompts
text: In this section, we describe the prompts used in M4LE. The prompt begins with the task definition, followed by the in-context example and the testing instance. Below we show the prompt examples used for each of the five abilities. Other tasks’ prompts are constructed similarly.

heading: Future Work
text: We suggest future research to apply our methodology to construct long instances in additional languages