Marathon: A Race Through the Realm of Long Context with Large Language Models (2024)

Abstract: With the advancement of large language models (LLMs) and the expansion of their context windows, existing long-context benchmarks fall short in effectively evaluating the models’ comprehension and reasoning abilities in extended texts. Moreover, conventional benchmarks relying on F1 metrics often inaccurately score responses: they may undervalue correct answers that differ from the reference responses and overvalue incorrect ones that resemble the reference texts. In response to these limitations, we introduce Marathon, a novel evaluation benchmark that adopts a multiple-choice question format. It is specifically designed to overcome the constraints of previous benchmarks and provide a rapid, precise, and unbiased appraisal of the long-context comprehension skills of large language models. We conducted comprehensive evaluations on the Marathon benchmark with a range of state-ofthe-art LLMs and assessed the effectiveness of various optimization strategies tailored for long-context generation. We anticipate that the Marathon benchmark and its associated leaderboard will enable a more precise and equitable evaluation of LLMs’ capabilities in understanding and reasoning over extended contexts. Marathon is available at https: //github.com/Hambaobao/Marathon.1

heading: 1 Introduction
text: In the rapidly evolving landscape of artificial intelligence technologies, the emergence of large language models (LLMs), as exemplified by ChatGPT (OpenAI et al., 2024), showcases notable capabilities. The influence of these models extends beyond the well-established ChatGPT, gaining increasing prominence across diverse sectors. Existing LLMs are typically built upon Transformer
†Min Yang is the corresponding author. 1We also provide an online evaluation website: https:
//openbenchmark.online/marathon
architectures, which demand memory and computational resources that grow quadratically with sequence length. Consequently, Transformer language models have historically been trained with relatively modest predetermined context windows. For instance, LLaMA (Touvron et al., 2023a) employs a context size of 2048 tokens, while Llama2 (Touvron et al., 2023b) utilizes a context size of 4096 tokens. However, the pre-defined size imposes constraints on LLMs in various applications, such as summarizing extensive documents or addressing lengthy questions.
Significant research efforts have been devoted to extending the context length of LLMs. Due to the prohibitive expense of training LLMs with extended context lengths from scratch, the predominant studies have endeavored to enhance the capabilities of LLMs to comprehend long contexts through fine-tuning. These methods encompass extending the context window (Chen et al., 2023b), incorporating recurrent memory (Bulatov et al., 2024), employing sparse attention mechanisms (Xiao et al., 2023a), and augmenting with external memory (Wang et al., 2023). Concurrently, an increasing multitude of benchmarks have been introduced to assess the long-context understanding capabilities of LLMs. LongBench (Bai et al., 2023b) stands out as the first bilingual, multi-task benchmark specifically designed for the assessment of long-context understanding. This dataset continues to depend on the F1 score, which evaluates the responses of LLMs against a predefined set of possible answers. LooGLE (Li et al., 2023b) encompasses intricate long dependency tasks, including event timeline reordering, comprehension/reasoning, and computation. Nevertheless, the diverse nature of model-generated content introduces a challenge, as these predefined answers may not encompass all valid responses, thereby diminishing the precision of assessing model performance. There is a growing demand for high-quality
5201
benchmarks characterized by significantly longer text lengths and more challenging tasks, ensuring comprehensive evaluations.
In this study, we introduce a novel benchmark named Marathon, designed for long-context understanding and reasoning. In particular, this benchmark is constructed upon the foundations established by LooGLE (Li et al., 2023b) and LongBench (Bai et al., 2023b). The contextual lengths within this benchmark span from 2K to over 260K characters. For each extensive context provided, an associated question is paired with four meticulously crafted response options. These options have been carefully reviewed by humans and contain only one correct answer, with the remaining options designed to be highly misleading. This design makes the Marathon benchmark a particularly challenging one. The task for the large language model is to discern the accurate response option based on the extensive context provided.
The main contributions of this work are threefold:
• We introduce a novel multiple-choice long context benchmark that comprehensively evaluates the long context understanding and reasoning capabilities across 10 leading opensource large language models, as well as ChatGPT and GPT-4, covering six diverse types of tasks.
• We compare two prevalent methods for long context optimization (Prompt Compression and Retrieval Augmented Generation) along with two leading embedding models, assessing their impact on enhancing the long context
reasoning abilities of large language models.
• Our findings reveal a general tendency among current open-source large language models to generate lengthier responses, accompanied by a notable deficiency in following instructions accurately.

heading: 2 Related Work

heading: 2.1 Prompt Compression
text: Although larger context windows enable large language models to handle longer contextual information, processing long-context information requires a significant amount of computing resources and places high demands on hardware. It also necessitates longer computational time, even in the inference stage. Therefore, some methods like LLMLingua (Jiang et al., 2023c) and LongLLMLingua (Jiang et al., 2023b) have been proposed to compress long contexts.

heading: 2.2 Retrieval Augmented Generation
text: Retrieval Augmented Generation (RAG) was originally proposed and applied to NLP tasks in (Lewis et al., 2020), and it has now become a mainstream method for improving the generation capability of large language models. RAG can extract the most relevant data from external knowledge bases and hand it over to the large language model for processing. This can alleviate the hallucination problem of large language models and enable people to trace the source of the content generated by large language models, ensuring the reliability of the generated content. Additionally, RAG can
also be used to extract information from long documents that is most relevant to the user’s query. This ensures that key information required to provide correct answers to questions is not lost while reducing the length of the context. Many projects such as Longchain 2 and LlamaIndex 3 have achieved significant progress in combining RAG with large language models, greatly facilitating related research in this direction.

heading: 2.3 Long Context Models
text: The ability of large language models for handling long contexts has become increasingly important. ChatGPT 4 supports a window size of 16k, while GPT-4 supports a window size of 128k, and Claude2.1 supports a window size of 200k5. Many opensource large language models have started to expand the size of their context window. Longchat (Li et al., 2023a) and MPT (Team, 2023b) have achieved a window size of 16k, while Mistral (Jiang et al., 2023a) and Zephyr (Tunstall et al., 2023) have achieved a window size of 32k. By utilizing an adapted Rotary Embedding (Su et al., 2022) and sliding window (Beltagy et al., 2020) during finetuning, MistralLite, based on Mistral, has achieved a window size of 128k, enabling large language models to handle even longer contextual information.

heading: 2.4 Long Context Benchmarks
text: There have been many recent benchmarks (Tay et al., 2020; Pang et al., 2022; Shaham et al., 2022) used to assess the long context processing ability of large language models, such as LooGLE (Li et al., 2023b) and LongBench (Bai et al., 2023b). Liu et al. (2023b) on the other hand, noticed that the position of key information in long contexts greatly affects the capability of large language models to correctly understand and process text. Therefore, they used the NaturalQA (Kwiatkowski et al., 2019) dataset to construct a new benchmark to test the impact of different positions of key information in long context on the text processing capability of large language models.
Pang et al. (2022) proposed a multiple-choice long context benchmark, but its average text length is only 5k, which is significantly below the context window of mainstream large language mod-
2https://github.com/langchain-ai/langchain 3https://github.com/jerryjliu/llama_index 4https://chat.openai.com 5https://www.anthropic.com/index/claude-2-1
els. Although SCROLLS (Shaham et al., 2022), LooGLE (Li et al., 2023b), and LongBench (Bai et al., 2023b) have constructed a relatively comprehensive set of evaluation tasks, the evaluation metrics used are still F1-score, Bleu or Rouge, which cannot accurately evaluate the ability of large language models to handle and understand long contexts.

heading: 3 Marathon
text: Present benchmarks for evaluating large language models primarily use a multiple-choice format, highlighted by studies such as MMLU (Hendrycks et al., 2021) and C-Eval (Huang et al., 2023). This multiple-choice approach helps prevent situations where large language models produce correct answers but are scored low due to missing corresponding elements in the reference answers or when they produce incorrect answers that are scored high because they resemble the reference answers closely. Therefore, influenced by LooGLE (Li et al., 2023b) and LongBench (Bai et al., 2023b), we developed a multiple-choice, long-context benchmark to more accurately evaluate the ability of large language models to understand extended contexts.

heading: 3.1 Overview
text: The Marathon benchmark includes six tasks: Comprehension and Reasoning, Multiple Information Retrieval, Timeline Reorder, Computation, Passage Retrieval, and Short Dependency QA. These tasks are grouped into four categories based on the type of questions they involve: Question Answering, Timeline Reordering, Computation, and Passage Retrieval. Table 1 provides the number of test samples for each task. Figure 2 presents example questions for each category.

heading: 3.2 Construction
text: All the test samples in the benchmark are in the form of multiple-choice questions, with each question containing one correct answer option and several distractor options. We use GPT-4 to generate the distractor options for each question. For each question, we divide the long context into multiple fragments of length 12,000 tokens and randomly select one fragment. We require GPT-4 to generate three distractor options based on the given context fragment, question, and correct answer. The purpose of this approach is to avoid using excessively long context that exceeds GPT-4’s context window, which may affect the accuracy of the generated results. By using shorter contexts, we can obtain distractor options that are more relevant to these shorter contexts. The distractor options for all test samples were generated using the OpenAI GPT-4-0613 API, with a total expenditure of approximately $900.
Finally, to ensure the effectiveness and accuracy of these distractor options, we manually verify the options of each test sample.

heading: 3.3 Question Answering
text: Comprehension and Reasoning, Multiple Information Retrieval and Short Dependency QA are all types of traditional question-answer formats. The difference lies in the fact that Comprehension and
Reasoning, Multiple Information Retrieval are selected from the Long Dependency QA dataset in LooGLE (Li et al., 2023b), while Short Dependency QA is selected from the Short Dependency QA dataset in LooGLE (Li et al., 2023b). In the question-answering tasks, each question is accompanied by a corresponding long context, and the large language model is required to infer the correct answer according to the long context. For the Short Dependency QA task, the relevant content for the correct answer is relatively concentrated within the long context. For Comprehension and Reasoning and Multiple Information Retrieval tasks, the content relevant to the correct answer is more scattered throughout the long context. Therefore, the large language model needs to possess strong long context understanding capability in order to solve the question correctly.
In the upper left of Figure 2, an example of a Question Answering task is provided. The question asks the large language model to answer a related question based on the content in the long context.

heading: 3.4 Timeline Reorder
text: Timeline Reorder task is a relatively novel questionanswering task. Unlike traditional questionanswering tasks, in the Timeline Reorder task, the question format requires large language models to sort a series of events described in a long context according to their chronological order. This
task aims to examine the large language models’ understanding of temporal relationships. Due to the dispersed distribution of events that need to be sorted by chronological order in the long context, large language models not only need to possess a correct understanding of temporal order but also require strong long context processing capabilities to answer correctly, which makes it a challenging task.
In the upper right of Figure 2, an example of the Timeline Reorder task is provided. The question requires the large language model to sort three events mentioned in the long context according to their chronological order.

heading: 3.5 Computation
text: Computation task is also different from traditional question-answering tasks. Its question format involves providing a question related to numerical computation and requires the large language model to perform numerical calculations based on relevant content in the long context. For example, it may require calculating the number of children a certain character has at a specific time point, considering that the long context describes the character’s life events, including the death of a child due to illness, which may affect the number of the character’s offspring at subsequent time points. Therefore, to answer this question correctly, the large language model not only needs to be able to perform ordinary
numerical calculations but also needs to capture all the key information related to the question. Compared to traditional computation and question answering tasks, this task is more challenging and can better reflect the large language model’s capability to comprehend long context.
In the bottom left of Figure 2, an example of a Computation task is provided. The question requires the large language model to complete a numerical calculation question based on the content in the long context.

heading: 3.6 Passage Retrieval
text: Passage Retrieval task is one form of task in the LongBench (Bai et al., 2023b). In order to enhance the diversity of our benchmark tasks, we have sampled 300 test data from the Passage Retrieval task in LongBench (Bai et al., 2023b), and reformed them into multiple-choice format using the method mentioned above. We have incorporated this task into our benchmark. The Passage Retrieval task requires large language models to locate the paragraph in a long context that corresponds to the given description in the question. Since the test data of the Passage Retrieval task is sampled from LongBench (Bai et al., 2023b), there are some limitations in terms of context length and timeliness. However, it remains a highly valuable task format. In future work, we will update its content to make it more suitable for the current needs of evaluating
large language models. A sample of the Passage Retrieval task is provided in the bottom right of Figure 2. The task requires large language models to locate the paragraph in a long context that corresponds to the given description in the question.

heading: 4 Experiments

heading: 4.1 Experimental Setup

heading: 4.1.1 Models
text: In this analysis, we incorporated a diverse array of models, distinguished by their parameter sizes, which span from 7B to 70B, and their context window capacities, extending from 8K to 200K. Additionally, the evaluation encompassed models constructed on a state-space architectural framework. The models scrutinized in this investigation comprises ChatGLM3-6B-32K (Zeng et al., 2022; Du et al., 2022), Mistral-7B-Instructv0.1 (Jiang et al., 2023a), Zephyr-7B-β (Tunstall et al., 2023), StripedHyena-Nous-7B (Poli et al., 2023), Longchat-13B-16K (Li et al., 2023a), Qwen14B-Chat (Bai et al., 2023a), Yi-34B (AI et al., 2024), Alfred-40B-1023 (Hallström et al., 2023), StableBeluga-2-70B (Mahan et al., 2023), Tulu-2DPO-70B (Ivison et al., 2023), ChatGPT-1106 6, and GPT-4-1106-preview (OpenAI et al., 2024).

heading: 4.1.2 Prompt Template
text: Figure 4 illustrates the prompt used in our model evaluation process. The system prompt is denoted in green, the provided long context in cyan, the question related to the long context in yellow, the quartet of options in blue, and the orange segment delineates the response format for the model, accompanied by a concrete example. The instructions and response templates may vary across different models. To ensure consistency, we adjust the prompts during our evaluations to match the templates used during the models’ training phases.

heading: 4.1.3 Optimization Methods
text: In this evaluation, we first assessed the inherent ability of various models to comprehend long contexts. Then, we evaluated the current mainstream methods for handling long contexts: Compression and RAG. Specifically, for the compression method, we assessed LongLLMLingua (Jiang et al., 2023b), while for the RAG method, we evaluated two retrieval approaches, one based on OpenAI Embed-
6https://chat.openai.com
ding and the other on Jina Embedding (Günther et al., 2023).

heading: 4.1.4 Evaluation Metric
text: To assess the long-context comprehension abilities of large language models, we employed a multiplechoice format for the questions, each designed with several options, among which only one is the correct answer.
Accuracy = No. of Correct Answers No. of Total Answers

heading: 4.1.5 Post Processing
text: As illustrated in the prompt template of Figure 4, we require the model to output responses in JSON format. However, current large language models are not capable of following user instructions 100% of the time. Therefore, after the model generates a response, we first parse the output using JSON format. For outputs that do not comply with the JSON format, we use regular expressions to extract the model’s responses.

heading: 4.2 Implementation Details

heading: 4.3 Vanilla
text: For models evaluated using the Vanilla method, we adopt the same strategy as Longbench, controlling the length of the input sequence within the context window range of the model to be evaluated by deleting the content in the middle.

heading: 4.3.1 LongLLMLingua
text: For LongLLMLingua, we set the compression rate to 0.5, the dynamic context compression ratio to 0.4, the condition in question to "after", and the condition compare to True. We also sort the compressed contexts based on their importance.

heading: 4.3.2 Embedding RAG
text: For Embedding RAG, we utilize the ServiceContext and VectorStoreIndex of the Llama-Index 7. We employ various models as LLMs (Language Models), testing the OpenAI Embedding model and the Jina Embedding model as Embedding Models, respectively. The default parameter settings are retained, with a chunk size of 1024 and a top-k value of 2. As for Jina Embedding, we set the pooling method to "mean" to align with Jina’s encode implementation.
7https://github.com/jerryjliu/llama_index

heading: 4.3.3 Hardware
text: All experiments in this evaluation were conducted on a server with 4 * A100 80GB GPUs. We set the batch size to 1. For models with a scale of 7B and 13B, we use one GPU; for models around 30B, we use two GPUs; and for models at the 70B scale, we use four GPUs.

heading: 4.4 Results

heading: 4.4.1 Main results
text: The overall accuracy of various models on the Marathon benchmark is depicted in Figure 1. Detailed performance metrics of these models, utilizing distinct optimization techniques across a range of tasks, are presented in Table 4 within the appendix. To facilitate a more comprehensive comparative analysis of the outcomes, Figures 6, 7, 8, 9, 10, and 11 are provided in the appendix. The analysis indicates that the OpenAI Embedding Retrieval and Jina Embedding Retrieval models exhibit superior performance relative to the LongLLMLingua compression.
Moreover, all examined models exhibit diminished accuracy on both the Timeline Reorder and Computation tasks relative to their performance on alternative tasks. The implementation of the LongLLMLingua failed to yield any notable enhancements, and the advancements facilitated by the RAG were similarly constrained.

heading: 4.4.2 Vanilla
text: Within the subset of the Vanilla method, the Yi-34B model, characterized by its 34 billion parameters, attains the highest accuracy, registering at 55.91%. This is closely followed by the ChatGLM3-6B32K, which, despite its more modest parameter count of 6B, achieves an accuracy of 55.05%. Subsequently, the Beluga-70B model, notable for its context window limitation of 4K tokens, records an accuracy of 49.51%. The average accuracy observed across the remaining models shows minimal variance, with none exceeding the 40% threshold.

heading: 4.4.3 LongLLMLingua
text: In contrast to the Vanilla approach, the implementation of LongLLMLingua yielded marginal improvements in accuracy for certain models: Qwen witnessed an enhancement of 4.85%, Alfred experienced a 1.51% increase, Beluga saw a 3.08% uplift, and Tulu2 benefited from an 8.64% augmentation. Conversely, this methodology had a detrimental effect on the performance of other models: ChatGLM3 encountered a 7.14% decrement in accuracy, Mistral suffered a 2.8% reduction, Zephyr experienced a significant 7.74% decrease, StripedHyena and Longchat showed a marginal decline of 0.10% and 0.26% respectively, and Yi’s accuracy diminished by 7.25%.

heading: 4.4.4 OpenAI Embedding RAG
text: When juxtaposed with the baseline Vanilla methodology, the incorporation of OpenAI Embedding Retrieval notably enhances accuracy for several
models: Mistral’s accuracy improved by 10.37%, Zephyr’s by 11.66%, StripedHyena’s by 16.26%, Qwen’s by 14.19%, Yi’s by 7.65%, Alfred’s by 14.05%, and Tulu2’s by an impressive 24.05%. Conversely, this approach has been observed to negatively impact the accuracy of certain models, with ChatGLM3 experiencing a 4.06% reduction, Longchat a 5.92% decrease, and Beluga a slight decline of 1.27%.

heading: 4.4.5 Jina Embedding RAG
text: Relative to the foundational Vanilla approach, the adoption of Jina Embedding Retrieval has led to accuracy enhancements across a majority of the evaluated models. Notably, Mistral’s accuracy experienced a 12.23% increase, Zephyr’s accuracy rose by 15.82%, StripedHyena’s accuracy increased 17.37%, Longchat saw a 1.91% improvement, Qwen’s accuracy was augmented by 18.15%, Yi’s accuracy escalated by 7.9%, Alfred’s accuracy advanced by 13.93%, Beluga’s accuracy grew by 6.21%, and Tulu2’s accuracy surged by 23.60%.

heading: 4.5 Instruction Following Capability
text: In our evaluation, numerous models exhibited limited ability to follow instructions accurately. We explicitly requested responses in JSON format, as exemplified by a sample provided. Nonetheless, models occasionally responded in alternate formats or attempted JSON responses that were either incomplete or incorrect. Our statistical analysis, summarized in Table 3, categorizes responses as "JSON" for correct JSON format, "JSON-like" for flawed attempts at JSON due to errors like truncation or formatting issues, and "Plain Text" for responses in other formats. For a clearer compari-
son of models’ ability to follow instructions after applying various optimizations, we focused on the rate of correct JSON responses as a measure of this capability, as depicted in Figure 5.
While Yi exhibited high accuracy in question answering, its compliance with instructions was notably lower, at 38.95%. In contrast, Beluga’s adherence rate to instructions was even lower, at 22.48%, despite its capabilities. On the other hand, Longchat, despite its modest accuracy in answering questions, showcased a remarkable proficiency in following instructions, achieving a 92.29% compliance rate, closely trailing behind ChatGPT’s 99.61%. The three distinct optimization techniques in our assessment demonstrated efficacy in diminishing the context length. However, it is noteworthy that none of these strategies consistently enhance the models’ ability to follow instructions.

heading: 5 Discussion

heading: 5.1 Tendency of Long Responses
text: During our analysis, we observed that open-source large language models often generate lengthy responses, even with clear instructions for concise JSON-formatted answers. This tendency results in the generation of extraneous content, necessitating post-processing to isolate the needed information. Table 3 presents statistics on the models’ output formats, highlighting their instruction-following capabilities. This issue likely stems from the models’ training on predominantly long responses, making it challenging for them to comply with requests for brevity.

heading: 5.2 State Space Models
text: Recent studies, such as Mamba (Gu and Dao, 2023), highlight the advantages of state space models (SSMs) for long context reasoning tasks. StripedHyena (Poli et al., 2023) innovatively merges SSMs with transformer structures, indicating a new direction in large language models. Despite these advancements, our analysis reveals that StripedHyena underperforms in detailed long context question answering compared to traditional transformers and does not reduce memory usage effectively, even with advanced attention mechanisms like Flash Attention 2 (Dao, 2023). These findings suggest the need for further optimization in State Space Models.

heading: 5.3 JSON Format
text: During the recent OpenAI Developer Day8, significant advancements in the capabilities of GPT-4 (OpenAI et al., 2024) were unveiled by OpenAI, notably the introduction of parallel function invocation and the specification of response formats in JSON. The parallel function invocation allows for the concurrent execution of multiple utility functions by large language models, thereby facilitating the efficient completion of complex user tasks. Moreover, the integration of JSON format for responses is instrumental in ensuring the seamless transmission of parameters and retrieval of results during function invocation, which is critical for the interoperability and functionality of AGI systems.

heading: 6 Future Work

heading: 6.1 Document as Context
text: Following the enhancements introduced at OpenAI Developer Day, GPT-4 (OpenAI et al., 2024) has been equipped with a Knowledge Retrieval feature. This allows the model to utilize user-uploaded documents for answering queries, marking a significant development in Retrieval-Augmented Generation (RAG) applications. This trend suggests that future large language models will likely adopt similar functionalities, impacting the evaluation methodologies for long-context question answering. Instead of embedding lengthy contexts into prompts, future benchmarks should focus on the models’ ability to extract and utilize information from user-provided documents to respond to queries. This approach necessitates a reevaluation
8https://devday.openai.com
of current benchmarks to align with these emerging capabilities.

heading: 6.2 Multi-modal Long Context
text: Models such as GPT4V (OpenAI et al., 2024) and Gemini (Team, 2023a) have exhibited robust capabilities in facilitating interactions that span both visual and linguistic modalities. Likewise, opensource counterparts, including LLaVA (Liu et al., 2023a) and MiniGPT-4 (Zhu et al., 2023), have demonstrated commendable performance in assessments tailored to multimodal contexts. The utility of such models extends to various real-world applications that necessitate the processing of multimodal, extensive contexts, exemplified by the comprehensive analysis and synthesis of corporate annual reports. These applications demand not only the capacity of large language models to comprehend and infer within long textual contexts but also necessitate the integration of visual understanding abilities. Presently, the open-source community is lack of benchmarks specifically designed to evaluate the proficiency of models in handling extended, multimodal contexts. Therefore, establishing a comprehensive benchmark for multimodal, long-context capabilities is of significant importance.

heading: 7 Conclusion
text: In this paper, we compared 10 open-source large language models, including variations in their parameter sizes and context windows, along with OpenAI’s ChatGPT and GPT-4. We assessed two prevalent optimization techniques, such as LongLLMLingua and RAG. The experimental results indicate that RAG-based optimization enhances the performance of large language models within longcontext scenarios for QA-type tasks. However, the improvement is limited for tasks involving Timeline Reorder and Computation. Despite high accuracy in question-answering, these models show limited ability to follow instructions.

heading: Acknowledgments
text: Min Yang was supported by National Key Research and Development Program of China (2022YFF0902100), National Natural Science Foundation of China (62376262), the Natural Science Foundation of Guangdong Province of China (2024A1515030166), Shenzhen Science and Technology Innovation Program
(KQTD20190929172835662), Shenzhen Basic Research Foundation (JCYJ20210324115614039).
Limitations
Context Length Distribution. As depicted in Figure 3, the distribution of context lengths within the Marathon benchmark exhibits a lack of uniformity. The test instances corresponding to the tasks of Comprehension and Reasoning, Multiple Information Retrieval, Computation, Short Dependency QA, and Timeline Reorder predominantly feature context lengths that are concentrated at, or below, 130K characters. Conversely, test instances with context lengths surpassing 200K characters are notably scarce.
The test instances for the Passage Retrieval task derive from the LongBench (Bai et al., 2023b) dataset, which accounts for the markedly shorter context lengths in comparison to those associated with the remaining five tasks. This discrepancy underlies the superior performance metrics achieved by all models on the Passage Retrieval task. It is our intention to revise the test instances for Passage Retrieval to ensure consistency in context lengths with the other tasks. Furthermore, our ongoing efforts are directed towards augmenting the test instances for the remaining tasks, with the objective of achieving a uniform distribution of context lengths ranging from 60K to 260K characters across all tasks.
Evaluation. This paper presents a preliminary evaluation of optimization techniques for long contexts, which is not all-encompassing. In terms of optimization strategies, our evaluation of the Retrieval-Augmented Generation (RAG) method was limited to the employment of the OpenAI and Jina Embedding systems, exemplifying leading commercial and open-source embedding models, respectively. However, constraints related to time and financial resources precluded the examination of several advanced embedding systems, such as Voyage (Voyage.AI, 2023), Cohere (Cohere.Team, 2023), and BGE Embeddings (Xiao et al., 2023b). In the case of the Prompt Compression approach, aside from LongLLMLingua, there are other techniques like MemWalker (Chen et al., 2023a) that merit future exploration to fully assess the advantages and drawbacks of each embedding model and optimization method.
Moreover, in scenarios involving long context, while model accuracy and adherence are crucial,
the speed of inference and memory demand are also vital factors to consider. This area features a variety of sophisticated optimization methods, including H2O (Zhang et al., 2023) and StreamingLLM (Xiao et al., 2023a). Subsequent research will focus on evaluating the performance of these inference optimization methods in scenarios with extensive textual content, with an emphasis on their speed of inference, memory consumption, QA precision, and instruction following capability.
Ethical Considerations
Data Source and Use. The benchmark leverages datasets that are publicly available and designated for research purposes. We have ensured that the use of these datasets adheres to their respective licenses and terms of use, emphasizing that our utilization is strictly confined to academic and research contexts.
Content Sensitivity and Bias. Our benchmark has been meticulously curated to exclude any content that could be deemed sensitive, such as violence, discriminatory language, or adult material.
Transparency and Reproducibility. In the spirit of fostering an open and fair research community, we will make the questions, contexts, and options of our benchmark’s test cases publicly available. However, to maintain the integrity of the evaluation process, the correct answers to the test cases will not be disclosed. Instead, we will provide an online evaluation platform where researchers can submit their models’ responses for assessment. This system is designed to ensure fairness and objectivity in the benchmarking process, allowing for an equitable comparison of different models’ capabilities.

heading: A Dataset Construction
text: A.1 Data Collection
The data in the Marathon benchmark primarily originates from LongBench (Bai et al., 2023b) and LooGLE (Li et al., 2023b). Table 2 provides specific details regarding the sources of the data and the licensing information for the benchmark.
A.2 Data Processing
For Comprehension and Reasoning, Multiple Information Retrieval, Short Dependency QA, Timeline Reorder and Computation tasks, we use data from LooGLE (Li et al., 2023b) to construct the test samples. For each question, we divide its corresponding long context into text segments of 12,000 tokens each and randomly select one segment to generate a distractor option, each segment is used no more than once. We send the selected long context segment Ci, the question Q, and the correct answer A in a specific format to GPT-4, requesting it to provide a distractor option Oi based on Ci, Q, and A.
For Passage Retrieval task, we construct test samples with Python program. We sampled data from Longbench (Bai et al., 2023b) and increased the number of passages to extend the context length.
To ensure the effectiveness and accuracy of these distractor options, we manually verify the options of each test sample.

heading: B Model Description
text: ChatGLM. The latest open-source model in the ChatGLM series, ChatGLM3-6B, as outlined by (Zeng et al., 2022; Du et al., 2022), maintains several outstanding features of its predecessors, such as smooth dialogue capabilities and a low deployment threshold.
Mistral. The Mistral-7B-Instruct-v0.1 Large Language Model (LLM), referenced by (Jiang et al., 2023a), is an instructionally fine-tuned variant of the Mistral-7B-v0.1 generative text model. It has been enhanced using a diverse range of publicly accessible conversational datasets.
Zephyr. The Zephyr-7B-β (Tunstall et al., 2023) represents the second installment in the Zephyr series. It is a refined version of the Mistral-7B-v0.1 (Jiang et al., 2023a), specifically enhanced through training on a combination of publicly available and synthetic datasets utilizing Direct Preference Optimization (DPO).
StripedHyena. StripedHyena (Poli et al., 2023) is the inaugural alternative model that rivals the top open-source Transformers of comparable size in both short and long-context evaluations.
Longchat. Longchat-13b-16k (Li et al., 2023a) is an open-source chatbot developed through fine-tuning the llama-13b model (Touvron et al., 2023a). It utilizes conversations shared by users on ShareGPT, applying the condensing rotary embedding technique as discussed in the blog9.
Qwen. Qwen-14B-chat (Bai et al., 2023a) represents the 14-billion parameter iteration of the Qwen large language model series, abbreviated as Tongyi Qianwen, developed by Alibaba Cloud.
Yi. The Yi-34B (AI et al., 2024) model represents the latest generation of open-source large language models, independently trained from scratch by 01.AI.
9https://lmsys.org/blog/2023-06-29-longchat/
Alfred. Alfred-40B-1023 (Hallström et al., 2023) is an enhanced version of the Falcon-40B model, featuring an expanded context length capacity of 8192 tokens.
Beluga. StableBeluga-2-70B (Mahan et al., 2023) is a variant of the Llama2 70B model (Touvron et al., 2023a), specifically fine-tuned using a dataset styled after Orca.
Tulu2. Tulu2-DPO-70B (Ivison et al., 2023) is a refined iteration of the Llama2 model (Touvron et al., 2023a), trained using a combination of publicly available, synthetic, and human datasets through the application of Direct Preference Optimization (DPO).

heading: C Detailed Evaluation Results
text: Table 4 presents the detailed performance metrics of various models, utilizing distinct optimization techniques across a range of tasks. C&R refers to Comprehension and Reasoning task; MIR refers to Multiple Information Retrieval task; TR refers to Timeline Reorder task; Com. refers to Computation task; PR refers to Passage Retrieval task; SDQA refers to Short Dependency Question Answering task; Avg. denotes the average accuracy across all tasks. To provide a more intuitive comparison of the effects of different optimization approaches on the long-context comprehension and reasoning capabilities of various models across different tasks, we also illustrated Figures 6, 7, 8, 9, 10, and 11.

heading: D Detailed Instruction Following Capability
text: As shown in Figure 4, we asked the models to produce results in JSON format to assess how well they follow instructions based on their output format. Table 3 summarizes the performance of 10 open-source large language models in this regard. "JSON" means the output was exactly in JSON format. "JSON-like" refers to outputs that tried to be in JSON format but included mistakes or extra text. "Plain Text" covers outputs in other formats. Since ChatGPT and GPT-4 can always gave results in JSON format, they’re not included in Table 3.

heading: Future Work
text: , 2023a) that merit future exploration to fully assess the advantages and drawbacks of each embedding model and optimization method. Moreover, in scenarios involving long context, while model accuracy and adherence are crucial,
the speed of inference and memory demand are also vital factors to consider. This area features a variety of sophisticated optimization methods, including H2O (Zhang et al. , 2023) and StreamingLLM (Xiao et al. , 2023a). Subsequent research will focus on evaluating the performance of these inference optimization methods in scenarios with extensive textual content, with an emphasis on their speed of inference, memory consumption, QA precision, and instruction following capability. Ethical Considerations
Data Source and Use. The benchmark leverages datasets that are publicly available and designated for research purposes. We have ensured that the use of these datasets adheres to their respective licenses and terms of use, emphasizing that our utilization is strictly confined to academic and research contexts. Content Sensitivity and Bias. Our benchmark has been meticulously curated to exclude any content that could be deemed sensitive, such as violence, discriminatory language, or adult material. Transparency and Reproducibility. In the spirit of fostering an open and fair research community, we will make the questions, contexts, and options of our benchmark’s test cases publicly available. However, to maintain the integrity of the evaluation process, the correct answers to the test cases will not be disclosed. Instead, we will provide an online evaluation platform where researchers can submit their models’ responses for assessment. This system is designed to ensure fairness and objectivity in the benchmarking process, allowing for an equitable comparison of different models’ capabilities