LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios
Authors: Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu, Wei Zhang
Accepted at: acl25

LIFBENCH: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios

Xiaodong Wu†  Minhao Wang†  Yichen Liu †  Xiaoming Shi †
He Yan §  Xiangju Lu §  Junmin Zhu §  Wei Zhang ‡‡*
†East China Normal University   §iQIYI Inc   ‡Shanghai Innovation Institute
†{51255901079,51275901104,51275901148}@stu.ecnu.edu.cn  xmshi@ir.hit.edu.cn
§{yanhe, luxiangju, zhujunmin}@qiyi.com   *zhangwei.thu2011@gmail.com

Abstract

As Large Language Models (LLMs) evolve in natural language processing (NLP), their ability to stably follow instructions in long-context inputs has become critical for real-world applications. However, existing benchmarks seldom focus on instruction-following in long-context scenarios or stability on different inputs. To bridge this gap, we introduce LIFBENCH, a scalable dataset designed to evaluate LLMs’ instruction-following capabilities and stability across long contexts. LIFBENCH comprises three long-context scenarios and eleven diverse tasks, featuring 2,766 instructions generated through an automated expansion method across three dimensions: length, expression, and variables. For evaluation, we propose LIFEVAL, a rubric-based assessment method that enables precise, automated scoring of complex LLM responses without reliance on LLM-assisted assessments or human judgment. This method allows for a comprehensive analysis of model performance and stability from multiple perspectives. We conduct detailed experiments on 20 prominent LLMs across six length intervals. Our work contributes LIFBENCH and LIFEVAL as robust tools for assessing LLM performance in complex and long-context settings, offering valuable insights to guide future advancements in LLM development.1

<table>
  <tr>
    <th>Benchmark</th>
    <th>Long.</th>
    <th>Inst.</th>
    <th>Stab.</th>
    <th>Unlim.</th>
  </tr>
  <tr>
    <td>ZeroSCROLLS (2023)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>BAMBOO (2024)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>Longbench (2023)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>∞ Bench (2024c)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>RULER (2024)</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>IFEval (2023a)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>FollowBench (2023)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>InfoBench (2024)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <td>CELLO (2024)</td>
    <td>✗</td>
    <td>✓</td>
    <td>✗</td>
    <td>✗</td>
  </tr>
  <tr>
    <th>LIFBENCH (Ours)</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
  </tr>
</table>

Table 1: A comparison of our LIFBENCH with some relevant datasets. We summarize their focus, including Long-context scenarios, Instruction-following, and model Stability. ‘Unlim.’ denotes whether the data length can be Unlimited.

2024; Chen et al., 2025), and problem-solving (Lu et al., 2024; Li et al., 2024b). Despite these achievements, significant challenges remain. On the one hand, LLMs often struggle to accurately and consistently follow human instructions, such as restating input content precisely or adhering stably to specific formatting constraints (He et al., 2024). On the other hand, studies show that as input length increases, the LLMs’ performance in tasks such as reasoning (Levy et al., 2024), retrieval (Li et al., 2024a), and general NLP (Bai et al., 2023) deteriorates. These challenges pose substantial barriers to their effectiveness in real-world applications.

Numerous evaluation benchmarks, as summarized in Table 1, have been proposed to guide the development of LLMs. However, they each exhibit notable limitations when it comes to evaluating instruction-following capabilities and stability in long-context scenarios. Some benchmarks focus either on long-context scenarios (Bai et al., 2023; Zhang et al., 2024c) or complex instruction-following abilities (Zhou et al., 2023a; Jiang et al., 2023). However, none of these works evaluate instruction-following abilities in long-context scenarios, and their reliance on fixed data lengths fails to accommodate the state-of-the-art LLMs’
Figure 1: The framework of LIFBENCH, where the task Single-ID in the List scenario is used as an example. Bold denotes the scenario description \( D \); normal denotes the context \( X \); *italics* denotes instruction \( I \), with red indicate the instruction variables \( var \), and the remaining black parts correspond to the instruction template \( tpl \).

ever-expanding context length. Other efforts (Li et al., 2024a; Hsieh et al., 2024) attempt to extend evaluation to longer contexts by constructing synthetic datasets, but their tasks are hard to provide a comprehensive and in-depth assessment of LLMs’ instruction-following abilities. In terms of evaluation outcomes, most existing benchmarks focus exclusively on task completion performance, often neglecting stability—a critical factor in ensuring reliable real-world performance.

To address these limitations, we introduce the Long-context Instruction Following Benchmark (LIFBENCH), a scalable benchmark for evaluating LLMs’ instruction-following capability and stability in long-context scenarios. The framework of our benchmark is shown in Figure 1. Considering real-world scenarios, we construct three long-context scenarios based on the granularity of information to be processed. On this basis, eleven delicate tasks are designed, which can illustrate various dimensions of instruction-following capabilities. We manually craft templates for all tasks, and introduce an automated instruction expansion method from three dimensions (length, expression, and variables), enabling LIFBENCH to expand significantly in both the quantity and length of instructions. As an example, we construct a dataset of 2,766 instructions spanning six length intervals, reaching up to 128k tokens.

For evaluation, traditional metrics for downstream tasks are often unsuitable for complex instruction-following scenarios (Honovich et al., 2023). Moreover, while many studies rely on GPT-4 for automated and open-ended assessment, these approaches encounter limitations due to notable gaps between GPT-4 and human experts (Qin et al., 2024; Jiang et al., 2023), as well as potential bias problems (Wang et al., 2023). To address these challenges, we propose LIFEVAL, a systematic and efficient evaluation method for complex LLM responses, without relying on LLMs or human evaluators. Specifically, by designing task-specific scoring rubrics, we decompose evaluations into fine-grained and quantifiable scoring points, each of which can be assessed automatically. In addition, through the score-capability map and a novel metric—IFS, LIFEVAL provides insights into models’ fundamental capabilities and stability from various perspectives.

Overall, our contributions are as follows:
• We introduce LIFBENCH, a benchmark designed to evaluate instruction-following capabilities in long-context scenarios, containing 11 tasks across three scenarios.
• We develop methods for dataset expansion across three perspectives, enabling high scalability in both the quantity and length of instructions.
• We propose LIFEVAL, an automatic evaluation method for accurately and comprehensively assessing the quality and stability of LLMs’ complex responses.
• We conduct extensive experiments across six length intervals, which evaluate and analyze the
instruction-following capabilities and stability of 20 well-known LLMs, encompassing both open-source and closed-source models.

2 Related Work

Several studies focus on LLMs’ performance in long contexts. These benchmarks collect data from traditional NLP tasks (e.g., summarization and question answering (QA)) to form comprehensive datasets containing long data (Shaham et al., 2022, 2023; Dong et al., 2024; Li et al., 2024c; Gavin et al., 2024). Additionally, due to the excellent performance of LLMs on open-ended tasks, some benchmarks have also designed synthetic tasks to better probe the models’ ability in math, reasoning, and logic (Kwan et al., 2023; Zhang et al., 2024c; Wang et al., 2024; Li et al., 2024a; Chen et al., 2024b). For evaluation, some works adopt regular metrics (e.g., Acc, ROUGE, and BLEU), which can be obtained by automated calculations. However, some open-ended tasks cannot be effectively evaluated using these metrics, hence powerful LLMs, such as GPT-4, are used as alternative evaluators. For example, the studies (An et al., 2023; Li et al., 2023) feed the model predictions and ground-truth answers to the GPT-4 evaluator, which is tasked with conducting a direct scoring or comparison to evaluate baselines’ performance on partial tasks. Unlike LIFBENCH, these benchmarks only assess problem-solving capabilities in long-context scenarios, overlooking challenges of complex instruction following that arise in real-world applications.

Given the complexities of evaluating instruction-following abilities, several studies introduce meaningful innovations in their evaluation methodologies to better tackle this multifaceted challenge. The studies (Cook et al., 2024; Wen et al., 2024; Qin et al., 2024; Zhang et al., 2024a) decompose instructions into checklists composed of YES/NO questions or PASS/FAIL criteria, which answers should meet. CELLO (He et al., 2024) defines four criteria that should be assessed as they can encompass common errors made by LLMs, and develops automated evaluation metrics to reflect models’ ability in complex instruction-following scenarios. However, these studies fail to concern evaluation in long-context instruction-following scenarios and largely overlook stability in the instruction-following process. Additionally, while the study (Sakai et al., 2024) explores the impact of various prompt templates and languages on LLM, it focuses solely on NLU tasks.

In summary, existing benchmarks either emphasize long-context scenarios or instruction-following capabilities, whereas LIFBENCH uniquely targets both simultaneously.

3 LIFBENCH

3.1 Problem Definition

As shown in Figure 1, we model the instruction-following task in long-context scenarios as follows: Given a prompt consisting of a scenario description (\( D \)), context (\( X \)), and instruction (\( I \)), the model is expected to output an answer (\( A \)). This process can be represented as:

\[
(D, X_{len}, I_{tpl,var}) \rightarrow A .
\]

In this setup, the scenario description \( D \) provides task background at the beginning of the prompt, and all tasks within a scenario share the same \( D \). The context \( X \), as the main body of the prompt, provides essential information and varies by scenario. For example, in the List scenario (three scenarios are constructed in LIFBENCH, see Section 3.2), \( X \) is a long list, while in the OneDoc scenario, \( X \) represents a lengthy processed document. The parameter \( len \) represents the number of tokens in \( X \). The instruction \( I \) is placed at the end of the prompt, which consists of two components: (1) the instruction template (\( tpl \)), outlining the task requirements, and (2) the instruction variable (\( var \)), representing variable part within the template. Generally, in LIFBENCH, \( D \) and \( I \) tend to be short, while \( X \) is a long text with thousands of tokens.

3.2 Dataset Construction

To simulate real-world LLM applications in long-text processing, we construct 3 scenarios and 11 tasks (see Table 2) based on the following principles: (1) Task Diversity: Tasks should encompass varied constraints (e.g., format, quantity) to evaluate different instruction-following abilities; (2) Performance Distinguishability: Tasks must balance simplicity and complexity to distinguish model performance; (3) Input Scalability: Tasks should support extended input lengths to assess long-context capabilities; (4) Automated Evaluation: Task constraints can be assessed through an automated program.

List The List scenario tests how well LLMs can handle structured lists, such as retrieving specific items and processing structured data. The input \( X \)
is an ordered list with UUIDs and natural language instructions.

The scenario includes six tasks: Single-ID and Multi-ID focus on retrieving specific elements from an ordered list using provided IDs. Building on these basic tasks, the Offset-ID, Offset-element, Blur-ID, and Blur-element tasks introduce spatial constraints, adding complexity to the retrieval process. "Offset" tasks require precise index-based retrieval to test fine-grained spatial awareness, while "Blur" tasks involve broader spatial ranges, allowing more flexibility. "ID" and "Element" refer to the type of reference in the retrieval process, representing either the position number in the ordered list or the list element itself.

MultiDoc The MultiDoc scenario evaluates how well LLMs process multiple documents, such as summarization and retrieval. Models need to compare documents, find differences, and handle batch operations.

The input X consists of multi-document collections from diverse sources. Each document has six fields: "text", "id", "ID2", "title", "date", and "source", with a length of 300–500 tokens. Tasks include Find-dup-doc, which finds duplicate documents, and Batch-label, which assigns labels to documents based on given attributes.

OneDoc The OneDoc scenario tests how well LLMs process a single long document, such as extracting key information or answering questions.

A long document is created by combining essays from the Paul Graham Essays dataset2. Some sentences are marked as key information. Tasks include Repeat, where models repeat a given amount of key information; Extract, where models extract specific key details; and QA, where models check if a sentence is labeled as key information.

We manually write scenario descriptions and instruction templates for all tasks, with detailed examples provided in Appendix H. To further ensure the tasks and scenarios are challenging and discriminative, special efforts are made during data collection, as elaborated in Appendix A.

3.3 Data Extension

In this section, we expand manual templates in three dimensions (length, expression, and variable) to form a sizeable test dataset.

Length A number of works (Bai et al., 2023; Ni et al., 2024; Li et al., 2024c; Levy et al., 2024) have found that the length of input text has an impact on the ability of the LLMs. In LIFBENCH, different lengths of prompts allow us to explore the impact of context length on the instruction-following capabilities of LLMs, making it essential to introduce variations in prompt length.

In all three scenarios, we adjust the length of the prompt by controlling the context token count \( l \). Specifically, we modify the number of elements in the List or the number of documents in MultiDoc and OneDoc to achieve the desired length. Ample corpus are pre-constructed for each scenario, supporting expansions up to 2M tokens in one prompt.

Expression In real-world contexts, due to personality and individual differences, individuals often provide significantly varied descriptions of the same subject, which undoubtedly challenges the stability of large models in following instructions. To assess LLM robustness in this regard, we diversify instruction templates to create multiple expressions style with differing wording and syntax.

Our approach follows a four-step process, namely "Rewriting-Encoding-Clustering-Sampling (RECS)". First, to ensure diversity and mitigate biases from any single model, we use GPT-4 (Achiam et al., 2023) and Claude (Anthropic, 2024) to generate 40+ rewrites of each original instruction template, complemented by a subsequent manual review to further validate and refine the outputs. Next, the rewritten templates are encoded into vector representations for clustering, with the number of clusters set to the target number of rewritten instructions for each task. For instance, if a task needs five rewritten instructions, we will create five clusters. Finally, from each cluster, we select the usable template nearest to the center as the final diversified expression. Further details and effectiveness evidence are provided in Appendix B.

<table>
  <tr>
    <th>Scenario</th>
    <th>Task</th>
    <th>ID</th>
    <th>#Exp.</th>
    <th>#Var.</th>
    <th>#Data</th>
  </tr>
  <tr>
    <td rowspan="6">List</td>
    <td>Single-ID</td>
    <td>LSI</td>
    <td>5</td>
    <td>6</td>
    <td>180</td>
  </tr>
  <tr>
    <td>Multi-ID</td>
    <td>LMI</td>
    <td>5</td>
    <td>5</td>
    <td>150</td>
  </tr>
  <tr>
    <td>Offset-ID</td>
    <td>LOI</td>
    <td>11</td>
    <td>6</td>
    <td>396</td>
  </tr>
  <tr>
    <td>Offset-Element</td>
    <td>LOE</td>
    <td>12</td>
    <td>6</td>
    <td>432</td>
  </tr>
  <tr>
    <td>Blur-ID</td>
    <td>LBI</td>
    <td>11</td>
    <td>6</td>
    <td>396</td>
  </tr>
  <tr>
    <td>Blur-Element</td>
    <td>LBE</td>
    <td>12</td>
    <td>6</td>
    <td>432</td>
  </tr>
  <tr>
    <td rowspan="2">MultiDoc</td>
    <td>Batch-label</td>
    <td>MB</td>
    <td>5</td>
    <td>5</td>
    <td>150</td>
  </tr>
  <tr>
    <td>Find-dup-doc</td>
    <td>MF</td>
    <td>5</td>
    <td>5</td>
    <td>150</td>
  </tr>
  <tr>
    <td rowspan="3">OneDoc</td>
    <td>Repeat</td>
    <td>OR</td>
    <td>5</td>
    <td>5</td>
    <td>150</td>
  </tr>
  <tr>
    <td>QA</td>
    <td>OQ</td>
    <td>5</td>
    <td>6</td>
    <td>180</td>
  </tr>
  <tr>
    <td>Extract</td>
    <td>OE</td>
    <td>5</td>
    <td>5</td>
    <td>150</td>
  </tr>
</table>

Table 2: Statistics of LIFBENCH. #Exp. and #Var. represent the count in the data extension of Expression and Variable.

2https://huggingface.co/datasets/sgoel9/paul_graham_essays
Figure 2: Rubric score distribution across different scenarios and capabilities. Scores across different scenarios and capabilities tend to be evenly distributed.

Variable Some placeholders are preset in the instruction templates to indicate variable parts of instructions, i.e., instruction variables. These variables encompass elements such as query keys (for retrieval tasks), categorization criteria (for classification tasks), and format requirements. For position-related or numerical variables, we maintain an even distribution, and manually adjust other variables during task iterations to precisely control the task difficulty (see examples in Appendix C). By analyzing LLM performance across varying instruction variables, we can evaluate the models’ understanding and consistency in executing instructions. Ideally, models with strong instruction-following abilities execute instructions stably across variable conditions, while weaker models may exhibit inconsistency.

Table 2 details the task counts for each scenario. Compared to the other two scenarios, tasks in the List scenario are simpler, so they carry less weight (see Section 4.1) but are more numerous.

4 LIFEVAL

In this section, we introduce LIFEVAL, an automatic method that provides accurate assessments and detailed insights into LLMs’ long-context instruction-following capabilities and stability.

4.1 Automated Rubric-based Scoring

To evaluate the output quality of LLMs on LIFBENCH tasks, we introduce Automated Rubric-based Scoring (ARS), an accurate and efficient programmatic evaluation method. As shown in Table 6 in Appendix D, we manually craft scoring rubrics \( \mathcal{R} \) for each task. Each rubric consists of several scoring points \( s \) and is assigned a weight \( \bar{s} \) according to its complexity and difficulty. In other words, a larger \( \bar{s} \) means greater complexity, requiring more steps for evaluation. All of these points can be assessed automatically through a program. The scoring process on task \( t \) can be shown as follows:

\[
ARS_t(\mathcal{A}_t) = \frac{1}{\tilde{R}_t} \sum_{s \in \mathcal{R}_t} f_s(\mathcal{A}_t), \tilde{R}_t = \sum_{s \in \mathcal{R}_t} \bar{s}.
\]

In this equation, \( \mathcal{A}_t \) represents the model’s responses on task \( t \), and \( \tilde{R}_t \) is the sum of weights across all scoring points. The function \( f_s(\cdot) \in [0, \bar{s}] \) represents the average score of all outputs for scoring point \( s \). Its implementation relies on programmatic evaluation pipelines tailored to each scoring point. For example, in structural verification (e.g., JSON Dict format, see Program 1), we use a multi-stage detection process that includes symbol check, Parsing check, and KV Check. These stages progressively impose stricter requirements on the model’s output, with final scores based on performance at each stage. Overall, while the core logic is manually engineered according to the rubric, the execution is fully automated through programs. We provide detailed considerations regarding the scoring rubric, as well as additional examples of evaluation programs, in Appendix D.

Naturally, defining \( \mathcal{T} = \{t_i\}_{i=1}^{N_t} \) as the set of all tasks, the overall test result for any model on LIFBENCH is the weighted average of scores across \( \mathcal{T} \):

\[
ARS_{overall} = \frac{\sum_{t \in \mathcal{T}} \tilde{R}_t \cdot ARS_t}{\sum_{t \in \mathcal{T}} \tilde{R}_t}.
\]

Although a well-designed scoring rubric enables LIFEVAL to provide a more comprehensive and rigorous evaluation, the relatively high human cost inevitably limits its generalizability. To address this, when extending LIFBENCH with new samples or adapting LIFEVAL to other datasets, users may adopt a simplified approach by replacing weighted averaging in Equation 2 and 3 with a simple mean. This streamlined method bypasses the rubric design process while still satisfying the requirements of many practical applications.

4.2 Score-Capability Mapping

To further offer insights into the model’s strengths and weaknesses across various dimensions of instruction following, we introduce Score-Capability Mapping, which maps the scoring point \( s \) to six
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">OneDoc</th>
    <th colspan="3">List</th>
    <th colspan="3">MultiDoc</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>OR</th>
    <th>OQ</th>
    <th>OE</th>
    <th>LSI</th>
    <th>LMI</th>
    <th>LOI</th>
    <th>LOE</th>
    <th>LBI</th>
    <th>TBE</th>
    <th>MB</th>
    <th>MF</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.797</td>
    <td>0.862</td>
    <td>0.815</td>
    <td>0.818</td>
    <td>0.836</td>
    <td>0.746</td>
    <td>0.825</td>
    <td>0.747</td>
    <td>0.825</td>
    <td>0.747</td>
    <td>0.838</td>
    <td>0.785</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.707</td>
    <td>0.820</td>
    <td>0.736</td>
    <td>0.893</td>
    <td>0.735</td>
    <td>0.704</td>
    <td>0.781</td>
    <td>0.750</td>
    <td>0.832</td>
    <td>0.600</td>
    <td>0.777</td>
    <td>0.738</td>
  </tr>
  <tr>
    <td colspan="13"><i>Models Larger Than 20B Parameters</i></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst.</td>
    <td>0.759</td>
    <td>0.774</td>
    <td>0.817</td>
    <td>0.867</td>
    <td>0.674</td>
    <td>0.680</td>
    <td>0.681</td>
    <td>0.818</td>
    <td>0.806</td>
    <td>0.669</td>
    <td>0.584</td>
    <td>0.706</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Inst.</td>
    <td>0.730</td>
    <td>0.860</td>
    <td>0.711</td>
    <td>0.805</td>
    <td>0.798</td>
    <td>0.651</td>
    <td>0.798</td>
    <td>0.693</td>
    <td>0.788</td>
    <td>0.657</td>
    <td>0.531</td>
    <td>0.694</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst.</td>
    <td>0.702</td>
    <td>0.792</td>
    <td>0.850</td>
    <td>0.662</td>
    <td>0.612</td>
    <td>0.542</td>
    <td>0.517</td>
    <td>0.763</td>
    <td>0.761</td>
    <td>0.608</td>
    <td>0.476</td>
    <td>0.650</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-08-2024 (32B)<sup>†</sup></td>
    <td>0.529</td>
    <td>0.838</td>
    <td>0.535</td>
    <td>0.692</td>
    <td>0.619</td>
    <td>0.494</td>
    <td>0.530</td>
    <td>0.667</td>
    <td>0.615</td>
    <td>0.729</td>
    <td>0.660</td>
    <td>0.626</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-v01 (35B)<sup>†</sup></td>
    <td>0.495</td>
    <td>0.818</td>
    <td>0.420</td>
    <td>0.641</td>
    <td>0.579</td>
    <td>0.489</td>
    <td>0.506</td>
    <td>0.694</td>
    <td>0.643</td>
    <td>0.721</td>
    <td>0.646</td>
    <td>0.595</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B<sup>†</sup></td>
    <td>0.402</td>
    <td>0.694</td>
    <td>0.428</td>
    <td>0.604</td>
    <td>0.579</td>
    <td>0.453</td>
    <td>0.497</td>
    <td>0.642</td>
    <td>0.666</td>
    <td>0.726</td>
    <td>0.522</td>
    <td>0.548</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>0.337</td>
    <td>0.273</td>
    <td>0.118</td>
    <td>0.668</td>
    <td>0.394</td>
    <td>0.525</td>
    <td>0.581</td>
    <td>0.695</td>
    <td>0.681</td>
    <td>0.708</td>
    <td>0.308</td>
    <td>0.422</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B<sup>†</sup></td>
    <td>0.347</td>
    <td>0.529</td>
    <td>0.280</td>
    <td>0.263</td>
    <td>0.334</td>
    <td>0.232</td>
    <td>0.306</td>
    <td>0.338</td>
    <td>0.297</td>
    <td>0.732</td>
    <td>0.380</td>
    <td>0.394</td>
  </tr>
  <tr>
    <td colspan="13"><i>Models With 7-20B Parameters</i></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst.</td>
    <td>0.393</td>
    <td>0.768</td>
    <td>0.637</td>
    <td>0.525</td>
    <td>0.601</td>
    <td>0.457</td>
    <td>0.385</td>
    <td>0.700</td>
    <td>0.570</td>
    <td>0.391</td>
    <td>0.349</td>
    <td>0.347</td>
  </tr>
  <tr>
    <td>InternLM2.5-7b-chat-1m</td>
    <td>0.446</td>
    <td>0.828</td>
    <td>0.378</td>
    <td>0.609</td>
    <td>0.438</td>
    <td>0.543</td>
    <td>0.631</td>
    <td>0.713</td>
    <td>0.764</td>
    <td>0.619</td>
    <td>0.428</td>
    <td>0.523</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Inst.</td>
    <td>0.507</td>
    <td>0.812</td>
    <td>0.447</td>
    <td>0.626</td>
    <td>0.568</td>
    <td>0.436</td>
    <td>0.531</td>
    <td>0.684</td>
    <td>0.701</td>
    <td>0.445</td>
    <td>0.436</td>
    <td>0.519</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Inst.</td>
    <td>0.537</td>
    <td>0.681</td>
    <td>0.413</td>
    <td>0.705</td>
    <td>0.353</td>
    <td>0.397</td>
    <td>0.537</td>
    <td>0.668</td>
    <td>0.637</td>
    <td>0.522</td>
    <td>0.308</td>
    <td>0.491</td>
  </tr>
  <tr>
    <td>GLM-4.9B-chat-1m</td>
    <td>0.484</td>
    <td>0.813</td>
    <td>0.267</td>
    <td>0.705</td>
    <td>0.354</td>
    <td>0.371</td>
    <td>0.514</td>
    <td>0.648</td>
    <td>0.666</td>
    <td>0.688</td>
    <td>0.300</td>
    <td>0.490</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.273</td>
    <td>0.530</td>
    <td>0.237</td>
    <td>0.339</td>
    <td>0.226</td>
    <td>0.194</td>
    <td>0.202</td>
    <td>0.309</td>
    <td>0.290</td>
    <td>0.699</td>
    <td>0.397</td>
    <td>0.384</td>
  </tr>
  <tr>
    <td>Llama-3.1-Chat-1M (7B)</td>
    <td>0.413</td>
    <td>0.577</td>
    <td>0.275</td>
    <td>0.633</td>
    <td>0.209</td>
    <td>0.165</td>
    <td>0.190</td>
    <td>0.280</td>
    <td>0.250</td>
    <td>0.520</td>
    <td>0.281</td>
    <td>0.375</td>
  </tr>
  <tr>
    <td>Llama-3.1-7B</td>
    <td>0.347</td>
    <td>0.287</td>
    <td>0.040</td>
    <td>0.600</td>
    <td>0.207</td>
    <td>0.242</td>
    <td>0.554</td>
    <td>0.625</td>
    <td>0.622</td>
    <td>0.471</td>
    <td>0.455</td>
    <td>0.375</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B<sup>†</sup></td>
    <td>0.268</td>
    <td>0.491</td>
    <td>0.233</td>
    <td>0.106</td>
    <td>0.113</td>
    <td>0.068</td>
    <td>0.102</td>
    <td>0.119</td>
    <td>0.149</td>
    <td>0.244</td>
    <td>0.233</td>
    <td>0.213</td>
  </tr>
  <tr>
    <td>LWM-Text-1M (7B)</td>
    <td>0.307</td>
    <td>0.252</td>
    <td>0.049</td>
    <td>0.164</td>
    <td>0.136</td>
    <td>0.110</td>
    <td>0.306</td>
    <td>0.420</td>
    <td>0.452</td>
    <td>0.112</td>
    <td>0.220</td>
    <td>0.204</td>
  </tr>
</table>

Table 3: The ARS scores of models on different tasks. The abbreviations of the tasks can be found in Table 2. The overall score is calculated by Eq. 3. The best performing score is highlighted in bold and second-best is underlined. † indicates that the context X on the longest interval is right-truncated.

fundamental capabilities. With reference to previous studies (He et al., 2024; Zhou et al., 2023b), the six capabilities are defined as follows:
Ori (Original Content): Abilities to reproduce the original input accurately.
Num (Numerical Ability): Abilities in handling numerical data, such as recognition, counting, and basic arithmetic.
Spat (Spatial Awareness): Abilities in understanding spatial relationships and sequences.
Fmt (Format): Abilities in modifying and structuring content according to format rules.
Logic (Logic Execution): Abilities to follow logical conditions and decision branches.
Recog (Recognition Ability): Abilities to differentiate and focus on key elements of the input.

Building on the ARS score, we further compute the Instruction Following Performance (IFP) for each capability. The IFP for a specific capability c is defined as:

\[
IFP_c = \frac{\sum_{t \in T} \sum_{s \in \mathcal{R}_t} \mathbb{I}(s, c) \cdot f_s(\mathcal{A})}{\sum_{t \in T} \sum_{s \in \mathcal{R}_t} \mathbb{I}(s, c) \cdot s}
\]

where \( \mathbb{I}(s, c) \) is the indicator function that identifies whether a scoring point s is related to the capability c. We manually crafted this mapping so that when a scoring point s is relevant to the capability c, the indicator \( \mathbb{I}(s, c) \) equals 1, and 0 otherwise. Table 6 provides more detailed information on this. Importantly, as shown in Figure 2, scores across different scenarios and capabilities are carefully balanced during data collection to ensure less bias in LIFEVAL.

4.3 Instruction Following Stability

To measure whether the model can consistently follow instructions, we introduce Instruction Following Stability (IFS). Specifically, we define the observation perspective of stability, p, which refers to a specific feature of the model input. In LIFEBENCH, there are three perspectives: prompt length, expression (i.e., the template of instruction I), and instruction variables. Based on the selected perspective, we partition model’s responses \( \mathcal{A}_t \) into \( N_p \) groups, denoted as \( \mathcal{A}_t^p = \{ \mathcal{A}_t^j \}_{j=0}^{N_p} \). For example, a set of responses \( \mathcal{A}_t \) may originate from inputs spanning five different length intervals (such as 4k, 8k, etc.). In this case, we group the responses into five distinct groups based on the input lengths. Subsequently, ARS scores for each group of answers will be computed, resulting in a set of performance values \( \mathcal{Y}_t^p \), which can be expressed as follows:

\[
\mathcal{Y}_t^p = \{ARS_t(\mathcal{A}_t^j)\}_{j=0}^{N_p}.
\]

Finally, the Instruction Following Stability (IFS) on task t is calculated as the standard deviation (denoted by \( \sigma(\cdot) \)) of the performance values Y divided by their mean, formally expressed as:

\[
IFS_t^p = \frac{\sigma(\mathcal{Y}_t^p)}{\overline{\mathcal{Y}}_t^p} \in [0, +\infty)
\]

A lower IFS indicates greater stability in instruction-following under perspective p, whereas a higher IFS signifies reduced stability.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">IFS</th>
    <th colspan="4">Overall</th>
  </tr>
  <tr>
    <th>Expression</th>
    <th>Variable</th>
    <th>Length</th>
    <th>IFS AVG(7)</th>
    <th>ARS</th>
    <th></th>
  </tr>
  <tr>
    <td colspan="8"><b>API Models</b></td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.067</td>
    <td>0.063</td>
    <td>0.065</td>
    <td>0.066</td>
    <td>0.079</td>
    <td>0.078</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4†</td>
    <td>0.062</td>
    <td>0.101</td>
    <td>0.155</td>
    <td>0.107</td>
    <td>0.728</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="8"><b>Models Larger Than 70B Parameters</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst†</td>
    <td>0.166</td>
    <td>0.165</td>
    <td>0.165</td>
    <td>0.165</td>
    <td>0.165</td>
    <td>0.165</td>
    <td></td>
  </tr>
  <tr>
    <td>C4AI-cmd-v01†</td>
    <td>0.135</td>
    <td>0.082</td>
    <td>0.143</td>
    <td>0.120</td>
    <td>0.596</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst†</td>
    <td>0.103</td>
    <td>0.087</td>
    <td>0.182</td>
    <td>0.124</td>
    <td>0.651</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Inst</td>
    <td>0.101</td>
    <td>0.076</td>
    <td>0.263</td>
    <td>0.147</td>
    <td>0.694</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-08-2024</td>
    <td>0.145</td>
    <td>0.077</td>
    <td>0.232</td>
    <td>0.157</td>
    <td>0.626</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-72B†</td>
    <td>0.145</td>
    <td>0.117</td>
    <td>0.685</td>
    <td>0.332</td>
    <td>0.396</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-32B†</td>
    <td>0.196</td>
    <td>0.117</td>
    <td>0.685</td>
    <td>0.332</td>
    <td>0.396</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>0.225</td>
    <td>0.165</td>
    <td>0.380</td>
    <td>0.257</td>
    <td>0.433</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="8"><b>Models With 7-20B Parameters</b></td>
  </tr>
  <tr>
    <td>InternLM-2.5-7b-chat-1m†</td>
    <td>0.125</td>
    <td>0.195</td>
    <td>0.175</td>
    <td>0.355</td>
    <td>0.400</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Inst†</td>
    <td>0.142</td>
    <td>0.124</td>
    <td>0.206</td>
    <td>0.168</td>
    <td>0.520</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst†</td>
    <td>0.133</td>
    <td>0.124</td>
    <td>0.206</td>
    <td>0.154</td>
    <td>0.548</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Inst</td>
    <td>0.154</td>
    <td>0.120</td>
    <td>0.199</td>
    <td>0.158</td>
    <td>0.491</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GLM-4.9B-chat-1m</td>
    <td>0.151</td>
    <td>0.138</td>
    <td>0.175</td>
    <td>0.155</td>
    <td>0.490</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Llama-3.1-1B-Inst</td>
    <td>0.171</td>
    <td>0.121</td>
    <td>0.515</td>
    <td>0.269</td>
    <td>0.386</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-14B†</td>
    <td>0.171</td>
    <td>0.121</td>
    <td>0.515</td>
    <td>0.269</td>
    <td>0.386</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LWM-Text-Chat-1M</td>
    <td>0.204</td>
    <td>0.130</td>
    <td>0.282</td>
    <td>0.206</td>
    <td>0.411</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>LWM-Text-1M</td>
    <td>0.237</td>
    <td>0.179</td>
    <td>0.280</td>
    <td>0.232</td>
    <td>0.205</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B†</td>
    <td>0.268</td>
    <td>0.123</td>
    <td>0.785</td>
    <td>0.391</td>
    <td>0.213</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Instruction Following Stability (IFS) from three perspectives, with rankings in parentheses (smaller is better). The overall order is based on average IFS rankings. † denotes truncation of context \( X \) on the longest interval.

5 Experiments

5.1 Experiment Setup

We evaluate 20 popular LLMs with long-context capabilities, including models from GPT (Achiam et al., 2023), Llama (Dubey et al., 2024), Qwen (Yang et al., 2024), C4AI (Cohere For AI, 2024), LWM (Liu et al., 2024a), InternLM (Cai et al., 2024), and GLM (GLM et al., 2024) series, all claiming to support context lengths exceeding 128k tokens. Notably, the Qwen2.5-Inst. model extends its 32k context length to 128k with YaRN (Peng et al., 2023). GPT-4 and GPT-4o are accessed via its official API, while open-source models are deployed using vLLM (Kwon et al., 2023).

The experiments were conducted across six context length intervals, ranging from 4k to 128k tokens, with task-specific output limits to ensure sufficient space for model generation. Token counts are calculated using GPT-4’s tokenizer\footnote{https://github.com/openai/tiktoken}, and truncation is applied to adjust context \( X \) for models unable to process the longest contexts. In data extension, 5–6 template variants are created for each original instruction, and each instruction variable has 5–10 candidates for sampling. Further details can be found in Appendix E.

5.2 Results on LIFBENCH

Task-categorized Performance As shown in Table 3, two closed-source models achieve the highest scores, with the top score reaching only 0.758. This highlights substantial room for improvement in instruction-following capabilities. Generally, larger parameter sizes are associated with better performance. However, models with supervised instruction tuning often outperform their base counterparts, even when smaller in size. For instance, InternLM2.5-7b-chat-1m scores 0.101 higher than Llama-3.1-70B, demonstrating that fine-tuning on instruction or conversational data significantly enhances performance. While closed-source models dominate most tasks, open-source models excel in only a few (e.g., Blur-ID and Batch-label).

Moreover, compared to LSI, most of models perform worse on LOI and LBI tasks, likely due to LSI’s closer alignment with their training data, highlighting the need to enhance their ability to handle complex instructions. Similarly, the model performance on LBE (LOE) tasks is better than that on LBI (LOI), indicating that referencing specific elements is easier than using non-semantic IDs.

Capability-categorized Performance As shown in Figure 3, closed-source models lead in all dimensions. Within each model series, the solid lines consistently enclose their corresponding dashed lines, highlighting the significant impact of instruction fine-tuning on overall performance. Notably, the Format dimension shows the tightest clustering, suggesting that formatting requirements are well-covered across training corpora for most models. In contrast, the Recog dimension shows marked differences, likely reflecting varying levels of effort in incorporating numerical cognition data during training.

Stability Table 4 presents the IFS scores and rank-

![A radar chart showing instruction following performance in six capabilities for various models](page_1012_120_377_377.png)

Figure 3: The Instruction Following Performance in six capabilities. Lines of the same color in the chart represent models from the same series. Dashed lines represent base models, while solid lines represent their officially fine-tuned variants.
ings from three perspectives, revealing discrepancies between model stability and task completion ability. For instance, while Qwen2.5-72B-Inst. underwent truncation, potentially affecting its stability in "Length", it still outperformed GPT-4, which had a higher ARS score. Furthermore, models exhibited distinct strengths and weaknesses across perspectives: GPT-4o showed less stability in "Expression", while GPT-4 struggled with instruction variables where Llama-3.1-70B-Inst. excelled. Instruction fine-tuning generally improved stability, but larger parameter size did not guarantee better performance, as Qwen2.5-72B-Inst. surpassed all closed-source models in "Expression".

5.3 Effects of longer inputs

Overall Performance and Stability As shown in Figure 4(a), the performance of most models declines significantly as input length increases, particularly beyond 16k or 32k tokens. However, the rates of decline vary across models. For instance, GPT-4o maintains relatively high scores in long-context scenarios, whereas Llama-3-70B-Inst. experiences a sharp drop, indicating its limited ability to handle extended inputs.

The negative impact of input length is evident in stability metrics. As illustrated in Figure 4(b), most models demonstrate poorer stability in long-context scenarios, with the lowest stability observed at the longest input lengths. Interestingly, some models, such as C4AI-cmd-r-08-2024 and GLM-4-9b-chat-1m, exhibit the least stability at mid-range input lengths (e.g., 64k tokens), diverging from their overall performance trends. Additionally, the sensitivity to input length also varies across different perspectives. For instance, models like C4AI-cmd-r-08-2024 and Qwen2.5-7B-Inst. show more significant upward trend in "Expression" compared to "Variable". This highlights potential areas for improvement in enhancing instruction-following stability under long input contexts.

Performance in six core capabilities Figure 5 demonstrates the declining trends of six core capabilities across varying model sizes as input length increases. Notably, "Format" performance remains relatively stable across all input lengths in most models, suggesting that tasks related to formatting are less sensitive to longer contexts. Conversely, "Recog" experiences the steepest decline, highlighting the challenges models face in maintaining recognition ability as input length grows.

![Overall ARS score (a) and instruction following stability (b) under different input length.](page_120_120_1208_480.png)

Figure 4: Overall ARS score (a) and instruction following stability (b) under different input length.

![Instruction following performance in six core capabilities under different input length.](page_120_600_1208_1024.png)

Figure 5: Instruction following performance in six core capabilities under different input length.
Figure 6: The impact of increasing context length on ARS scores in tasks with different instruction complexities.

Model size also plays a crucial role in long-context instruction-following. Larger models generally are better at capabilities like "Ori," "Num," and "Spat," indicating that handling original content and spatial reasoning demands more model parameters. In contrast, format-related abilities are effectively managed even by smaller models, suggesting they are less dependent on model size.

5.4 Impact of Instruction Complexity

To analyze how context length and instruction complexity simultaneously affect model performance, we divide tasks into two groups based on scoring weight: easy (\( \leq 10 \)) and hard (\( >10 \)). As noted in Section 4.1, higher weights indicate greater challenges, with 10 chosen to balance the number of tasks in each group.

As shown in Figure 6, the performance of different models varies significantly. Llama-3.1-70B-Inst. struggles with long contexts across all tasks, highlighting its difficulty in handling long inputs. For GPT-4 and Qwen2.5-72B-Inst., the performance on both groups is similar in short contexts, but has sharper declines on hard tasks with longer contexts, revealing their limitations in handling complex tasks in long-context scenarios. Other models exhibit larger performance gaps even in short input (i.e., 4k tokens), indicating greater task difficulty impact for them. Generally, hard group degrades more significantly as context length increases, suggesting a compounded negative effect of task complexity and context length on LLM performance.

6 Conclusion

In this work, we systematically evaluate LLMs’ instruction-following capabilities and stability in long-context scenarios. We develop LIBBENCH, a benchmark that encompasses three long-context scenarios and a diverse set of 11 tasks, complemented by a method for instruction expansion across three distinct perspectives. For evaluation, we introduce LIFEVAL, an automated rubric-based scoring method, enabling fast and accurate assessments of model task performance and stability. Based on the benchmark and scoring method, we conduct extensive experiments on 20 prominent LLMs, revealing significant room for improvement in instruction-following capabilities and stability, especially in long-context scenarios.

Limitation

Our study has several limitations. Firstly, due to constraints in programmatic validation, our task scenarios lack comprehensive support for semantic
constraints, necessitating future improvements for better validation. Secondly, the inference process for very long inputs requires significant computational resources and time, limiting the scale of our dataset and potentially affecting the reproducibility of our work. As a result, our benchmark example comprises fewer than 3,000 samples, tested across only three perspectives. In fact, many unexplored aspects of stability remain, such as LLMs’ consistency in handling input formatting and context domain shifts. Additionally, larger datasets can enable more statistically significant conclusions. We encourage the community to use our proposed protocol to expand the evaluation set and conduct more extensive analyzes.

Finally, while LIFEVAL enables efficient and automated evaluations, the reliability of its results depends heavily on the design of the scoring rubric and the implementation of the evaluation programs. These components require significant time and effort to prepare before conducting evaluations. We leave the automation or reduction of these manual efforts for future work.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China (No. 92270119) and Key Laboratory of Advanced Theory and Application in Statistics and Data Science, Ministry of Education.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.

AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxia Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Charles E Brown. 1998. Coefficient of variation. In Applied multivariate statistics in geohydrology and related sciences, pages 155–157. Springer.

Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

Zheng Cai, Maosong Cao, Haojing Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.

Kedi Chen, Qin Chen, Jie Zhou, Yishen He, and Liang He. 2024b. Dialahu: A dialogue-level hallucination evaluation benchmark for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 9057–9079. Association for Computational Linguistics.

Kedi Chen, Zhikai Lei, Fan Zhang, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Qipeng Guo, Kai Chen, and Wei Zhang. 2025. Code-driven inductive synthesis: Enhancing reasoning abilities of large language models with sequences. CoRR, abs/2503.13109.

Aakanksha Chowdhery, Sharar Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Cohere For AI. 2024. c4ai-command-r-08-2024.

Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. 2024. Ticking all the boxes: Generated checklists improve llm evaluation and generation. arXiv preprint arXiv:2410.03608.

Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2086–2099.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Shawn Gavin, Tuney Zheng, Jiaheng Liu, Quehry Que, Noah Wang, Jian Yang, Chenchen Zhang, Wenhao Huang, Wenhui Chen, and Ge Zhang. 2024. Longins: A challenging long-context instruction-based exam for llms. ArXiv, abs/2406.17588.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen-hui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Han-lin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793.
Greg Kamradt. 2023. LLMTest: Needle in a Haystack. https://github.com/gkamradt/LLMTest_NeedleInAHaystack.

Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024. Can large language models understand real-world complex instructions? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18188–18196.

Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.

Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. arXiv preprint arXiv:2104.02112.

Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2023. Followbench: A multi-level fine-grained constraints following benchmark for large language models. arXiv preprint arXiv:2310.20410.

Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. 2023. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310.19240.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hau Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.

Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023. Loogle: Can long-context language models understand long contexts? arXiv preprint arXiv:2311.04939.

Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024a. Needlebench: Can llms do retrieval and reasoning in 1 million context window? arXiv preprint arXiv:2407.11963.

Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. 2024b. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402.19255.

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhui Chen. 2024c. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024a. World model on million-length video and language with blockwise ringattentio. ArXiv, abs/2402.08268.

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR).

J Macqueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press.

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.

Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, and Piji Li. 2024. XI^2 bench: A benchmark for extremely long context understanding with long-range dependencies. arXiv preprint arXiv:2404.05446.

Joe O’Connor and Jacob Andreas. 2021. What context features can transformer language models use? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 851–864.

Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13679–13707.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071.

Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. Infobench: Evaluating instruction following ability in large language models. arXiv preprint arXiv:2401.03601.

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191.

Yusuke Sakai, Adam Nohejl, Jiangnan Hang, Hide-taka Kamigaito, and Taro Watanabe. 2024. Toward the evaluation of large language models considering score variance across instruction templates. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 499–529.

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989.

Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. Scrolls: Standardized comparison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12007–12021.

Orit Shechtman. 2013. The coefficient of variation as an index of measurement reliability. In Methods of clinical epidemiology, pages 39–49. Springer.

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, et al. 2024. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca.

Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, et al. 2024. Leave no document behind: Benchmarking long-context llms with extended multi-doc qa. arXiv preprint arXiv:2406.17419.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926.

Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, et al. 2024. Benchmarking complex instruction-following with multiple constraints composition. arXiv preprint arXiv:2407.03978.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, et al. 2024a. Cfbench: A comprehensive constraints-following benchmark for llms. arXiv preprint arXiv:2408.01122.

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen Mckeown, and Tatsunori B Hashimoto. 2024b. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 11:39–57.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024c. ∞ bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15262–15277.

Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023a. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.
A  Details in Data Collection

A.1 List
To ensure the quality of tasks, the context \( X \) for the List scenario is constructed as a ordered list, consisting of two types of elements: randomly generated UUIDs and natural language instruction texts. (Liu et al., 2024b) used UUIDs to build key-value pairs and explained how large models utilize input context. Inspired by this work, we generated a set of unique 128-bit UUIDs as the first part of the list. However, real-world scenarios often involve some level of semantic noise, and transformer-based language models may exhibit varying sensitivities to different linguistic features in their input (O’Connor and Andreas, 2021). Therefore, to enhance complexity and realism, we selected a subset of instruction texts from the Alpaca-52k dataset (Taori et al., 2023), which serve as the second type of elements in the list. We found that when instruction texts are mixed into the retrieval list, the model’s attention tends to be drawn to the embedded instructions, leading to a prioritization of following them rather than focusing on the originally assigned task. As a result, we chose the “Instructions” part of the Alpaca-52k dataset as list elements. In addition, to ensure the appropriateness of text length, all selected instructions were limited to 5~40 tokens.

A.2 MultiDoc
To construct contexts \( X \) in this scenario, we selected documents from four datasets: GovReport (Huang et al., 2021), XSum (Narayan et al., 2018), QMSum (Zhong et al., 2021), and the Paul Graham Essays\footnote{https://huggingface.co/datasets/sgoel9/paul_graham_essays}. GovReport is a long document summary dataset consisting of reports written by government research agencies; XSum and QMSum are respectively a news summary dataset and a meeting summary dataset , both of which cover a wide range of domains; and the Paul Graham Essays dataset collects articles written by Paul Graham in a variety of fields. These selective datasets span multiple domains and various forms of text, offering a high degree of diversity.

For each dataset, we extract the main text (excluding summaries) to construct multi-document task contexts \( X \), limiting each text to 300–500 tokens through filtering and truncation. As shown in Table 12, each document contains six fields: "text", "id", "iD2", "title", "date", and "source". The "text" field holds the processed content, while the other five are attributes designed for the tasks, partially sourced from the dataset and partially constructed manually. Each document has unique "id" and "iD2" fields, though some may lack "title" or "source". To reduce LLMs’ reliance on parameter knowledge, we randomly annotate the "source" field, breaking its correlation with the "text". This ensures the model cannot infer the source using pre-trained knowledge, creating additional challenges for instruction adherence. Lastly, we introduced duplicates by reusing some "text" fields in different documents, maintaining a duplication rate of 25%.

A.3 OneDoc
To create the context \( X \) for OneDoc, we synthesized an extra-long document by concatenating entries from the Paul Graham Essays dataset, following the approach in (Greg Kamradt, 2023). Additionally, some sentences are randomly tagged as key information, with each tag specifying a type (e.g., Topic, Evidence, Concession) and a unique identifier to aid LLMs in identifying and categorizing critical content.

B  Effectiveness of Expression Extension

Implementation details  To implement RECS for expression extension, we chose the BGE-M3 text embedding model (Chen et al., 2024a) for encoding and applied K-means clustering (Macqueen, 1967). Additionally, since LLMs can introduce inaccuracies in rewrites due to misinterpretations, we manually filtered out unsuitable rewrites during the sampling phase, resulting in 5~6 instruction templates for each task.

Validity Experiment  To validate the effectiveness of the RECS method proposed in Section 3.3, we compared it with a random sampling approach. Specifically, after the Rewriting phase, we constructed a dataset by randomly sampling (rather than clustering) the same number of instructions and calculated the IFS values for sixteen models on this dataset.

As shown in 7, compared to random sampling, RECS led to higher expression IFS scores in 75% of the models, outperforming the IFS improvement rates observed for the Variable and Length perspectives, at 37.5% and 62.5% respectively. Additionally, when comparing the mean IFS values across all models, we observed that RECS im-
Figure 7: IFS Gap Across Three Perspectives with RECS vs. Random Sampling. Positive values indicate an IFS increase with RECS compared to Random Sampling, while negative values indicate a decrease. "MEAN" on the x-axis represents the average IFS gap.

<table>
  <tr>
    <th>Prompt for Rewriting in RECS</th>
  </tr>
  <tr>
    <td>Please rewrite the given prompt according to the following requirements:<br>1. The rewritten prompt must retain the same meaning as the original, without altering the intent.<br>2. Try your best to use different vocabulary or sentence structures.<br>3. Ensure that the rewritten prompt is clear and accurate, avoiding any expressions that could lead to ambiguity.<br>4. Please keep the placeholders in the prompt (i.e., "{ }" and the contents therein) exactly as they are during rewriting.<br>5. Please keep the example in the prompt, but you can make some small changes while keeping the original meaning.<br>6. Output the result in Json List format, without anything else.<br>7. Please generate 20 different rewrites at once.<br>prompt: [Prompt to be rewritten]</td>
  </tr>
</table>

Figure 8: Prompt template for rewriting task prompt.

proved expression IFS while slightly lowering variable IFS and length IFS. This suggests that the RECS method make it more challenging for models to maintain stability when handling varied expressions, which shows the effectiveness in expression extension as well.

C Sampling Space for Instruction Variables

As shown in Figure 9, we reserve identical placeholders for each instruction template and set the sampling space accordingly. Depending on the task requirements, the types of instruction variables include numerical values, lists, phrases, long sentences, and format indicators, etc. We carefully consider variable distribution to avoid biases from the sampling mechanism. For example, in position-related variables within the List scenario, inputs were divided into three sections (beginning, middle, and end), with the middle section receiving the largest portion. We then randomly sample 2–3 elements from each section to ensure balance. Additionally, we introduce unconventional causal relationships in certain rule-based variables to increase task difficulty. For instance, in the QA task, models are required to use "False" for correct answers and "True" for incorrect ones, making it more challenging to follow instructions.

D Rubric Design and Evaluation Program

Rubric Design  For the weight \( \tilde{s} \) assignment of scoring points, we primarily considered three factors:

(1) **Evaluation complexity**: Scoring points with higher evaluation complexity require more assessment steps and are therefore assigned greater weight. For example, in assessing format correctness, the Batch-Label (MB) task is given higher scoring weights compared to the QA (OQ) task due to its more complex formatting requirements.

(2) **Task difficulty for models**: Some scoring points may not be as complex to evaluate but are particularly challenging for the model to fulfill. In such cases, we allocate greater weight to these points as well.

(3) **Balance across capabilities and scenarios**: Based on the considerations in (1) and (2), we made
List: Single-ID

Instruction templates:
1. Retrieve the entry at position {pos} in the list. Display it immediately.
2. What is the {pos} element in the list? Output it directly.
3. ...

Sampling Space
pos: [1st, 3rd, 8th, ...]

Instruction after sampling
1. Retrieve the entry at position 8th in the list. Display it immediately.
2. ...

OneDoc: QA

Instruction templates:
1. Is the phrase "{evidence}" the KEY SENTENCE? Reply with "{options[0]}" for yes and "{options[1]}" for no.
2. Is "{evidence}" the KEY SENTENCE of the text? Indicate "{options[0]}" for yes, "{options[1]}" for no.
3. ...

Sampling Space
evidence: [a sentence sampled from the input document.]
options: [(True, False), (Yes, No), (apple, banana), ...]

Instruction after sampling
1. Is "With the result that ..." the KEY SENTENCE of the text? Indicate "True" for yes, "False" for no.
2. ...

Figure 9: Examples of the sampling space. Red indicates the instruction variables var, and {} indicates a placeholder in the instruction.

fine adjustments to ensure that the weights assigned to different instruction-following capabilities and scenarios are as balanced as possible. This balance is shown in Figure 2.

On this basis, we present the rubric of LIFEVAL in Table 6. Additionally, the pseudocode examples reflect our considerations above. Program 1 show the correspondence between evaluation steps and their respective weights. In Program 2, we award an additional point for responses that fully meet the requirements, highlighting the difficulty of that particular scoring point.

Evaluation Program As described in Section 4.1, we design an automated evaluation program based on the scoring rubric to evaluate the quality of LLM responses. Inspired by (Zhou et al., 2023a), during task design, we aim to ensure that the correct answers could be captured by the program. However, our tasks incorporate more complex formatting and logical constraints, making it challenging to directly obtain accurate evaluation results through simple automated methods. To address this, we iteratively decomposed and refined each scoring point s into sub-evaluation criteria that could be directly assessed by the program, enabling a more detailed and discriminative evaluation.

Our program evaluation adheres to two principles: (1) correct answers should achieve full scores, and (2) partially correct answers should be distinguishable in terms of scores. In general, we first extract structured information from the outputs based on the format constraints of each task, and then perform evaluations on other dimensions using this information. However, LLM responses are not always well-structured and may contain minor errors that hinder the extraction of structured information. To mitigate this, we carefully designed task-specific validation programs that employ techniques such as regex matching and substring retrieval to maximize the extraction of valid part from LLM responses (see Program 1). Finally, we manually provided reference answers for all samples in LIFBENCH. We verified that the automated evaluation programs in LIFEVAL consistently achieve full scores on all reference answers, thereby fulfilling the first principle outlined above.

E Experiments Setup

Baselines For closed-source models, we specifically use gpt-4o-2024-08-06 and gpt-4-0125-preview to represent the performance of GPT-4o and GPT-4, respectively. For open-source models, all implementations were sourced from Hugging Face5 (as detailed in Table 5), and we test the performance of both base models and fine-tuned models. Models with the suffix "-Inst." or "-chat" indicate that they have been fine-tuned on instruction or dialogue data.

Inference During the inference process, we complete the deployment of all open-source models with the vLLM (Kwon et al., 2023). The temperature is set to 0 to ensure deterministic outputs. For the SFT versions of the models, we used the official chat template. For the base versions, we add a suffix "Output: " to prompt the models to generate answers according to the instructions. Token counts were calculated using GPT-4's tokenizer6, and truncation was applied to adjust context for models

5https://huggingface.co/
6https://github.com/openai/tiktoken
<table>
  <tr>
    <th>Model</th>
    <th>Hugging Face ID</th>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Inst.</td>
    <td>meta-llama/Llama-3.1-70B-Inst</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Inst.</td>
    <td>meta-llama/Llama-3.1-8B-Inst</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>meta-llama/Llama-3.1-70B</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>meta-llama/Llama-3.1-8B</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst.</td>
    <td>Qwen/Qwen2.5-72B-Inst</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst.</td>
    <td>Qwen/Qwen2.5-32B-Inst</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst.</td>
    <td>Qwen/Qwen2.5-14B-Inst</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Inst.</td>
    <td>Qwen/Qwen2.5-7B-Inst</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>Qwen/Qwen2.5-7B</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>Qwen/Qwen2.5-32B</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>Qwen/Qwen2.5-14B</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>Qwen/Qwen2.5-7B</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-08-2024</td>
    <td>CohereLabs/c4ai-command-r-08-2024</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-v01</td>
    <td>CohereLabs/c4ai-command-r-v01</td>
  </tr>
  <tr>
    <td>LWM-Text-Chat-1M</td>
    <td>LargeWorldModel/LWM-Text-Chat-1M</td>
  </tr>
  <tr>
    <td>LWM-Text-1M</td>
    <td>LargeWorldModel/LWM-Text-1M</td>
  </tr>
  <tr>
    <td>InternLM2.5-7b-chat-1m</td>
    <td>internlm/internlm2.5-7b-chat-1m</td>
  </tr>
  <tr>
    <td>GILM-49b-chat-1m</td>
    <td>TTHUD/Gilm-49b-chat-1m</td>
  </tr>
</table>

Table 5: A mapping between the Hugging Face model IDs and the aliases used in the paper.

unable to process the longest contexts. To ensure diversity and accuracy in instruction phrasing, we generated 5–6 template variants for each original instruction and manually filtered them. Each template included placeholders for variables (e.g., numerical values, lists, phrases, long sentences, and format indicators), dynamically sampled from curated sets of 5–10 candidates to create task-specific prompts. In order to avoid unnecessary time consumption resulting from endless repetitions in the output, we set maximum generation lengths for different tasks: List scenario tasks that output a single element were limited to 100 tokens, tasks in the MultiDoc scenario are set to 4096 tokens, and the rest of the tasks are limited to 512 tokens.

Due to variations in encoding efficiency across models’ tokenizer, some models can not accommodate the longest inputs. To address this, we apply right truncation to the context \( X \), ensuring the completeness of scenario description \( D \) and instruction \( I \). The experiments were conducted across six context length intervals, ranging from 4k to 128k tokens, with task-specific output limits to ensure sufficient space for model generation.

F Inspiration for IFS

IFS is derived from the coefficient of variation (CV) (Brown, 1998), a statistical measure widely used in fields such as risk assessment and quality control (Shechtman, 2013). CV is calculated as the ratio of the standard deviation to the mean, providing a normalized measure of variability.

Using standard deviation alone to measure instruction-following stability can lead to misleading conclusions. For instance, if a model’s performance across all intervals is zero, its standard deviation would also be zero, suggesting perfect stability despite the model’s complete inability to perform. IFS avoids this issue by normalizing variability relative to the mean, ensuring that stability is assessed in a scale-independent and interpretable manner.

The application of CV to instruction-following tasks aligns well with its traditional use in evaluating consistency and reliability across varying scenarios. By capturing fluctuations in model performance relative to its average capability, IFS offers a robust and fair metric for comparing the stability of different LLMs.

Program 1 Format Validation (JSON Dict)

Target: Verify if the output meets the JSON Dict format.
Input: LLM answer \( a \), full score \( \hat{s} = 4 \)
Output: Validation score \( s_a = f_s(a) \in [0, \hat{s}] \)

Step 1: Symbol Check (Max 1 point)
1: \( s_a \leftarrow 0 \)
2: if count(\{\}) \( \geq 2 \) \& count(") \( \geq 4 \) \& count(:) \( \geq 1 \) then
3: \( s_a \leftarrow s_a + 1 \)
4: end if

Step 2: Parsing Check (Max 2 points)
5: \( a' \leftarrow \text{regex\_extract}(a, '\/\{[^{}]*\}+}') \)
6: if json.loads(a) succeeds then
7: \( s_a \leftarrow s_a + 2 \)
8: else if json.loads(a') succeeds then
9: \( s_a \leftarrow s_a + 1 \)
10: end if

Step 3: KV Check (Max 1 point)
11: if check_key_value_format(a/a') is correct then
12: \( s_a \leftarrow s_a + 1 \)
13: end if
14: return \( s_a \)

Note: The function check_key_value_format() serves as an example for additional format checks, with specifics depending on the task requirements.

Program 2 Quantity Verification (LMI)

Target: Verify if the number of output elements is correct.
Input: LLM answer \( a \), target number \( n_{gold} \), full score \( \hat{s} = 3 \)
Output: Validation score \( s_a = f_s(a) \in [0, \hat{s}] \)

1: \( n_{pred} \leftarrow \text{extract\_quantity}(a) \)
   ▷ Extract the number of elements in the output
2: if \( n_{pred} = n_{gold} \) then
3: \( s_a \leftarrow \hat{s} \)
   ▷ Full score if the quantity matches
4: else
5: \( s_a \leftarrow \max \left(0, \left(1.0 - \frac{|n_{pred} - n_{gold}|}{n_{gold}}\right)\right) \times 2 \)
   ▷ Penalty based on relative deviation
6: end if
7: return \( s_a \)

Note: The function \( n_{pred} = \text{extract\_quantity}(a) \) can be executed in various ways. If the LLM answer \( a \) passes format checks and is parsable, the target quantity is directly obtained. Otherwise, string processing techniques are employed to extract it with feedback.
<table>
  <tr>
    <th>Tasks</th>
    <th>Score Point \( s \)</th>
    <th>Weight \( \bar{s} \)</th>
    <th>Related Capability</th>
  </tr>
  <tr>
    <td rowspan="4">LSI, LOI, LOE</td>
    <td>Format correctness.</td>
    <td>1</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Answer is from the input list.</td>
    <td>2</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td>Answer correctness.</td>
    <td>1</td>
    <td><i>Recog</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>4</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">LMI</td>
    <td>Format correctness.</td>
    <td>2</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Order correctness.</td>
    <td>2</td>
    <td><i>Spat</i></td>
  </tr>
  <tr>
    <td>The number of output elements is correct.</td>
    <td>3</td>
    <td><i>Num</i></td>
  </tr>
  <tr>
    <td>Answers correctness.</td>
    <td>3</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>10</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">LBI, LBE</td>
    <td>Format correctness.</td>
    <td>1</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Answer is from the input list.</td>
    <td>1</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td>The output element conform to the position constraint.</td>
    <td>3</td>
    <td><i>Spat</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>5</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">MB</td>
    <td>Answer correctness (conforms to label logic).</td>
    <td>3</td>
    <td><i>Logit</i></td>
  </tr>
  <tr>
    <td>Output labels are from the candidate set.</td>
    <td>3</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td>The number of output labels matches the number of input documents.</td>
    <td>3</td>
    <td><i>Num, Recog</i></td>
  </tr>
  <tr>
    <td>Format correctness.</td>
    <td>5</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>14</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">MF</td>
    <td>Answer correctness.</td>
    <td>4</td>
    <td><i>Logit, Recog</i></td>
  </tr>
  <tr>
    <td>Find the correct number of duplicate documents.</td>
    <td>5</td>
    <td><i>Num, Logit</i></td>
  </tr>
  <tr>
    <td>Format correctness.</td>
    <td>5</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Document properties in the output are in the input</td>
    <td>6</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>20</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="5">OR</td>
    <td>Answer correctness.</td>
    <td>3</td>
    <td><i>Logit</i></td>
  </tr>
  <tr>
    <td>Output sentences are from the input document.</td>
    <td>2</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td>Format correctness.</td>
    <td>3</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Output sentences are the key sentence.</td>
    <td>2</td>
    <td><i>Recog</i></td>
  </tr>
  <tr>
    <td>The number of output key sentences is correct.</td>
    <td>4</td>
    <td><i>Num</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>14</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="2">OQ</td>
    <td>Answer correctness.</td>
    <td>3</td>
    <td><i>Logit</i></td>
  </tr>
  <tr>
    <td>Format correctness.</td>
    <td>2</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>5</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">OE</td>
    <td>Output sentences are from the input document.</td>
    <td>2</td>
    <td><i>Ori</i></td>
  </tr>
  <tr>
    <td>Format correctness.</td>
    <td>4</td>
    <td><i>Fmt</i></td>
  </tr>
  <tr>
    <td>Output sentences are target key sentences.</td>
    <td>4</td>
    <td><i>Recog</i></td>
  </tr>
  <tr>
    <td>The output order matches the ids sort.</td>
    <td>4</td>
    <td><i>Spat</i></td>
  </tr>
  <tr>
    <td><i>Total Weight (\( \bar{R} \))</i></td>
    <td>14</td>
    <td>-</td>
  </tr>
</table>

Table 6: The scoring rubric and Score-Capability Map in LIFEVAL..
G  Full Results on LIFBENCH

G.1 ARP Score

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Length OR</th>
    <th colspan="7">OneDoc</th>
    <th colspan="5">List</th>
    <th colspan="3">MultiDoc</th>
    <th>Overall</th>
  </tr>
  <tr>
    <th>OQ</th>
    <th>OE</th>
    <th>LSI</th>
    <th>LMI</th>
    <th>LOE</th>
    <th>LBL</th>
    <th>LBE</th>
    <th>MB</th>
    <th>MF</th>
    <th>NFR</th>
    <th>DRC</th>
    <th>CR</th>
  </tr>
  <tr>
    <td rowspan="6">GPT-4o</td>
    <td>4k</td><td>0.981</td><td>0.783</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>0.787</td>
  </tr>
  <tr>
    <td>8k</td><td>0.959</td><td>0.949</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>0.837</td>
  </tr>
  <tr>
    <td>16k</td><td>0.830</td><td>0.863</td><td>0.325</td><td>0.835</td><td>0.578</td><td>0.489</td><td>0.556</td><td>0.863</td><td>0.843</td><td></td><td></td><td></td><td></td><td></td><td>0.890</td>
  </tr>
  <tr>
    <td>32k</td><td>0.780</td><td>0.837</td><td>0.241</td><td>0.884</td><td>0.709</td><td>0.672</td><td>0.740</td><td>0.886</td><td>0.737</td><td></td><td></td><td></td><td></td><td></td><td>0.885</td>
  </tr>
  <tr>
    <td>64k</td><td>0.757</td><td>0.801</td><td>0.216</td><td>0.851</td><td>0.655</td><td>0.620</td><td>0.645</td><td>0.896</td><td>0.621</td><td></td><td></td><td></td><td></td><td></td><td>0.862</td>
  </tr>
  <tr>
    <td>128k</td><td>0.766</td><td>0.839</td><td>0.329</td><td>0.856</td><td>0.574</td><td>0.607</td><td>0.665</td><td>0.794</td><td>0.570</td><td></td><td></td><td></td><td></td><td></td><td>0.845</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Length</th>
    <th colspan="7">OneDoc</th>
    <th colspan="5">List</th>
    <th colspan="3">MultiDoc</th>
    <th>Overall</th>
  </tr>
  <tr>
    <th>OQ</th>
    <th>OE</th>
    <th>LSI</th>
    <th>LMI</th>
    <th>LOE</th>
    <th>LBL</th>
    <th>LBE</th>
    <th>MB</th>
    <th>MF</th>
    <th>NFR</th>
    <th>DRC</th>
    <th>CR</th>
  </tr>
  <tr>
    <td rowspan="6">GPT-4</td>
    <td>4k</td><td>0.939</td><td>0.937</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>0.873</td>
  </tr>
  <tr>
    <td>8k</td><td>0.953</td><td>0.927</td><td>0.641</td><td>0.869</td><td>0.721</td><td>0.807</td><td>0.819</td><td>0.848</td><td>0.783</td><td></td><td></td><td></td><td></td><td></td><td>0.853</td>
  </tr>
  <tr>
    <td>16k</td><td>0.849</td><td>0.782</td><td>0.612</td><td>0.827</td><td>0.677</td><td>0.781</td><td>0.734</td><td>0.844</td><td>0.661</td><td></td><td></td><td></td><td></td><td></td><td>0.818</td>
  </tr>
  <tr>
    <td>32k</td><td>0.787</td><td>0.750</td><td>0.591</td><td>0.923</td><td>0.380</td><td>0.816</td><td>0.506</td><td>0.863</td><td>0.764</td><td></td><td></td><td></td><td></td><td></td><td>0.853</td>
  </tr>
  <tr>
    <td>64k</td><td>0.867</td><td>0.819</td><td>0.673</td><td>0.874</td><td>0.802</td><td>0.875</td><td>0.673</td><td>0.884</td><td>0.808</td><td></td><td></td><td></td><td></td><td></td><td>0.887</td>
  </tr>
  <tr>
    <td>128k</td><td>0.524</td><td>0.825</td><td>0.570</td><td>0.834</td><td>0.457</td><td>0.708</td><td>0.645</td><td>0.761</td><td>0.714</td><td></td><td></td><td></td><td></td><td></td><td>0.629</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Length</th>
    <th>OneDoc</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>MB</th>
    <th>MF</th>
    <th></th>
    <th></th>
    <th>Overall</th>
  </tr>
  <tr>
    <th>OQ</th>
    <th>OE</th>
    <th>LSI</th>
    <th>LMI</th>
    <th>LOE</th>
    <th>LBL</th>
    <th>LBE</th>
    <th>NFR</th>
    <th>DRC</th>
    <th>CR</th>
  </tr>
  <tr>
    <td rowspan="6">Qwen2.5-72B-Inst.</td>
    <td>4k</td><td>0.995</td><td>0.907</td><td>0.994</td><td>0.850</td><td>0.957</td><td>0.821</td><td>0.834</td><td>0.867</td><td>0.768</td><td>0.586</td><td>0.822</td>
  </tr>
  <tr>
    <td>8k</td><td>0.849</td><td>0.840</td><td>0.921</td><td>0.826</td><td>0.378</td><td>0.472</td><td>0.652</td><td>0.851</td><td>0.599</td><td>0.587</td><td>0.659</td>
  </tr>
  <tr>
    <td>16k</td><td>0.772</td><td>0.782</td><td>0.800</td><td>0.963</td><td>0.823</td><td>0.578</td><td>0.798</td><td>0.877</td><td>0.800</td><td>0.555</td><td>0.662</td>
  </tr>
  <tr>
    <td>32k</td><td>0.749</td><td>0.773</td><td>0.802</td><td>0.853</td><td>0.844</td><td>0.525</td><td>0.820</td><td>0.877</td><td>0.807</td><td>0.500</td><td>0.655</td>
  </tr>
  <tr>
    <td>64k</td><td>0.811</td><td>0.877</td><td>0.856</td><td>0.969</td><td>0.398</td><td>0.876</td><td>0.788</td><td>0.931</td><td>0.805</td><td>0.256</td><td>0.748</td>
  </tr>
  <tr>
    <td>128k</td><td>0.799</td><td>0.820</td><td>0.980</td><td>0.737</td><td>0.574</td><td>0.540</td><td>0.763</td><td>0.689</td><td>0.570</td><td>0.156</td><td>0.736</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Length</th>
    <th>OneDoc</th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>MB</th>
    <th>MF</th>
    <th></th>
    <th></th>
    <th>Overall</th>
  </tr>
  <tr>
    <th>OQ</th>
    <th>OE</th>
    <th>LSI</th>
    <th>LMI</th>
    <th>LOE</th>
    <th>LBL</th>
    <th>LBE</th>
    <th>NFR</th>
    <th>DRC</th>
    <th>CR</th>
  </tr>
  <tr>
    <td rowspan="6">Llama-3.1-70B-Inst.</td>
    <td>4k</td><td>0.873</td><td>0.943</td><td>0.979</td><td>0.933</td><td>0.849</td><td>0.910</td><td>0.910</td><td>0.856</td><td>0.920</td><td>0.769</td><td>0.818</td>
  </tr>
  <tr>
    <td>8k</td><td>0.888</td><td>0.886</td><td>0.951</td><td>0.948</td><td>0.808</td><td>0.814</td><td>0.768</td><td>0.834</td><td>0.778</td><td>0.616</td><td>0.840</td>
  </tr>
  <tr>
    <td>16k</td><td>0.910</td><td>0.856</td><td>0.892</td><td>0.892</td><td>0.848</td><td>0.866</td><td>0.821</td><td>0.818</td><td>0.648</td><td>0.660</td><td>0.783</td>
  </tr>
  <tr>
    <td>32k</td><td>0.835</td><td>0.812</td><td>0.886</td><td>0.857</td><td>0.840</td><td>0.760</td><td>0.805</td><td>0.597</td><td>0.630</td><td>0.536</td><td>0.709</td>
  </tr>
  <tr>
    <td>64k</td><td>0.884</td><td>0.827</td><td>0.878</td><td>0.867</td><td>0.783</td><td>0.819</td><td>0.769</td><td>0.680</td><td>0.555</td><td>0.398</td><td>0.696</td>
  </tr>
  <tr>
    <td>128k</td><td>0.457</td><td>0.842</td><td>0.056</td><td>0.554</td><td>0.233</td><td>0.620</td><td>0.544</td><td>0.621</td><td>0.185</td><td>0.181</td><td>0.479</td>
  </tr>
</table>

 ... (remaining tables cut off)

(Continued on next page)
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Length</th>
    <th colspan="2">OneDoc</th>
    <th colspan="2">List</th>
    <th colspan="2">MultiDoc</th>
    <th rowspan="2">Overall</th>
  </tr>
  <tr>
    <th>OR</th><th>OE</th>
    <th>LSI</th><th>LMI</th>
    <th>LOI</th><th>LOE</th>
    <th>LBI</th><th>LBE</th>
    <th>MB</th><th>MF</th>
  </tr>
  <tr><td rowspan="8">Llama-3.1-8B-Inst.</td><td>4k</td><td>0.713</td><td>0.60</td><td>0.532</td><td>0.721</td><td>0.680</td><td>0.437</td><td>0.607</td><td>0.739</td><td>0.540</td><td>0.456</td><td>0.580</td></tr>
  <tr><td>8k</td><td>0.547</td><td>0.734</td><td>0.353</td><td>0.754</td><td>0.668</td><td>0.392</td><td>0.545</td><td>0.639</td><td>0.599</td><td>0.630</td><td>0.372</td><td>0.527</td></tr>
  <tr><td>16k</td><td>0.624</td><td>0.679</td><td>0.517</td><td>0.783</td><td>0.536</td><td>0.467</td><td>0.613</td><td>0.731</td><td>0.731</td><td>0.638</td><td>0.292</td><td>0.548</td></tr>
  <tr><td>32k</td><td>0.426</td><td>0.648</td><td>0.167</td><td>0.642</td><td>0.530</td><td>0.399</td><td>0.557</td><td>0.646</td><td>0.655</td><td>0.623</td><td>0.340</td><td>0.457</td></tr>
  <tr><td>64k</td><td>0.464</td><td>0.543</td><td>0.552</td><td>0.725</td><td>0.428</td><td>0.403</td><td>0.530</td><td>0.703</td><td>0.688</td><td>0.356</td><td>0.219</td><td>0.446</td></tr>
  <tr><td>128k</td><td>0.449</td><td>0.874</td><td>0.355</td><td>0.604</td><td>0.366</td><td>0.287</td><td>0.373</td><td>0.548</td><td>0.612</td><td>0.317</td><td>0.191</td><td>0.388</td></tr>
  <tr><td>4k</td><td>0.578</td><td>0.837</td><td>0.388</td><td>0.738</td><td>0.692</td><td>0.425</td><td>0.633</td><td>0.702</td><td>0.749</td><td>0.743</td><td>0.441</td><td>0.589</td></tr>
  <tr><td>8k</td><td>0.450</td><td>0.877</td><td>0.262</td><td>0.788</td><td>0.616</td><td>0.415</td><td>0.511</td><td>0.671</td><td>0.747</td><td>0.742</td><td>0.431</td><td>0.549</td></tr>
  <tr><td rowspan="8">GLM-4-9b-chat-1m</td><td>16k</td><td>0.496</td><td>0.829</td><td>0.282</td><td>0.820</td><td>0.428</td><td>0.308</td><td>0.463</td><td>0.613</td><td>0.795</td><td>0.299</td><td>0.309</td><td>0.409</td></tr>
  <tr><td>32k</td><td>0.503</td><td>0.751</td><td>0.212</td><td>0.717</td><td>0.468</td><td>0.255</td><td>0.555</td><td>0.670</td><td>0.661</td><td>0.352</td><td>0.282</td><td>0.382</td></tr>
  <tr><td>64k</td><td>0.423</td><td>0.830</td><td>0.166</td><td>0.688</td><td>0.484</td><td>0.314</td><td>0.467</td><td>0.646</td><td>0.604</td><td>0.713</td><td>0.196</td><td>0.437</td></tr>
  <tr><td>128k</td><td>0.452</td><td>0.837</td><td>0.254</td><td>0.475</td><td>0.443</td><td>0.272</td><td>0.458</td><td>0.574</td><td>0.626</td><td>0.354</td><td>0.183</td><td>0.383</td></tr>
  <tr><td>4k</td><td>0.301</td><td>0.719</td><td>0.489</td><td>0.708</td><td>0.602</td><td>0.467</td><td>0.348</td><td>0.503</td><td>0.401</td><td>0.783</td><td>0.455</td><td>0.519</td></tr>
  <tr><td>8k</td><td>0.260</td><td>0.673</td><td>0.472</td><td>0.548</td><td>0.527</td><td>0.360</td><td>0.459</td><td>0.516</td><td>0.197</td><td>0.730</td><td>0.383</td><td>0.462</td></tr>
  <tr><td>16k</td><td>0.248</td><td>0.613</td><td>0.476</td><td>0.525</td><td>0.356</td><td>0.409</td><td>0.423</td><td>0.545</td><td>0.389</td><td>0.801</td><td>0.660</td><td>0.518</td></tr>
  <tr><td>32k</td><td>0.202</td><td>0.397</td><td>0.036</td><td>0.131</td><td>0.195</td><td>0.207</td><td>0.224</td><td>0.252</td><td>0.407</td><td>0.851</td><td>0.366</td><td>0.324</td></tr>
  <tr><td rowspan="8">Qwen2.5-14B†</td><td>64k</td><td>0.405</td><td>0.790</td><td>0.044</td><td>0.629</td><td>0.231</td><td>0.292</td><td>0.656</td><td>0.591</td><td>0.639</td><td>0.087</td><td>0.502</td><td>0.366</td></tr>
  <tr><td>128k</td><td>0.322</td><td>0.447</td><td>0.036</td><td>0.020</td><td>0.104</td><td>0.065</td><td>0.060</td><td>0.054</td><td>0.071</td><td>0.189</td><td>0.163</td><td>0.156</td></tr>
  <tr><td>4k</td><td>0.373</td><td>0.644</td><td>0.036</td><td>0.654</td><td>0.331</td><td>0.478</td><td>0.631</td><td>0.722</td><td>0.615</td><td>0.252</td><td>0.519</td><td>0.403</td></tr>
  <tr><td>8k</td><td>0.399</td><td>0.718</td><td>0.036</td><td>0.692</td><td>0.318</td><td>0.181</td><td>0.528</td><td>0.650</td><td>0.560</td><td>0.069</td><td>0.530</td><td>0.364</td></tr>
  <tr><td>16k</td><td>0.426</td><td>0.721</td><td>0.261</td><td>0.638</td><td>0.264</td><td>0.141</td><td>0.595</td><td>0.432</td><td>0.571</td><td>0.129</td><td>0.548</td><td>0.395</td></tr>
  <tr><td>32k</td><td>0.442</td><td>0.740</td><td>0.036</td><td>0.671</td><td>0.361</td><td>0.562</td><td>0.643</td><td>0.717</td><td>0.597</td><td>0.142</td><td>0.520</td><td>0.409</td></tr>
  <tr><td>64k</td><td>0.405</td><td>0.790</td><td>0.044</td><td>0.629</td><td>0.231</td><td>0.292</td><td>0.656</td><td>0.591</td><td>0.639</td><td>0.087</td><td>0.502</td><td>0.366</td></tr>
  <tr><td>128k</td><td>0.435</td><td>0.770</td><td>0.036</td><td>0.517</td><td>0.243</td><td>0.198</td><td>0.576</td><td>0.426</td><td>0.558</td><td>0.087</td><td>0.503</td><td>0.346</td></tr>
  <tr><td rowspan="8">LWM-Text-Chat-1M</td><td>4k</td><td>0.473</td><td>0.251</td><td>0.014</td><td>0.735</td><td>0.361</td><td>0.681</td><td>0.747</td><td>0.773</td><td>0.722</td><td>0.788</td><td>0.635</td><td>0.521</td></tr>
  <tr><td>8k</td><td>0.473</td><td>0.251</td><td>0.014</td><td>0.735</td><td>0.361</td><td>0.681</td><td>0.747</td><td>0.773</td><td>0.722</td><td>0.788</td><td>0.635</td><td>0.521</td></tr>
  <tr><td>16k</td><td>0.415</td><td>0.287</td><td>0.017</td><td>0.753</td><td>0.151</td><td>0.447</td><td>0.455</td><td>0.660</td><td>0.607</td><td>0.705</td><td>0.404</td><td>0.403</td></tr>
  <tr><td>32k</td><td>0.357</td><td>0.267</td><td>0.057</td><td>0.529</td><td>0.142</td><td>0.344</td><td>0.562</td><td>0.636</td><td>0.652</td><td>0.166</td><td>0.388</td><td>0.311</td></tr>
  <tr><td>64k</td><td>0.276</td><td>0.340</td><td>0.056</td><td>0.513</td><td>0.114</td><td>0.413</td><td>0.532</td><td>0.588</td><td>0.600</td><td>0.188</td><td>0.381</td><td>0.298</td></tr>
  <tr><td>128k</td><td>0.290</td><td>0.307</td><td>0.030</td><td>0.340</td><td>0.243</td><td>0.042</td><td>0.414</td><td>0.414</td><td>0.541</td><td>0.200</td><td>0.415</td><td>0.278</td></tr>
  <tr><td>4k</td><td>0.192</td><td>0.624</td><td>0.450</td><td>0.033</td><td>0.100</td><td>0.000</td><td>0.026</td><td>0.000</td><td>0.000</td><td>0.363</td><td>0.388</td><td>0.264</td></tr>
  <tr><td>8k</td><td>0.214</td><td>0.665</td><td>0.353</td><td>0.084</td><td>0.103</td><td>0.022</td><td>0.103</td><td>0.020</td><td>0.073</td><td>0.633</td><td>0.233</td><td>0.274</td></tr>
  <tr><td rowspan="8">Qwen2.5-7B†</td><td>16k</td><td>0.245</td><td>0.301</td><td>0.438</td><td>0.316</td><td>0.113</td><td>0.226</td><td>0.311</td><td>0.393</td><td>0.299</td><td>0.000</td><td>0.183</td><td>0.229</td></tr>
  <tr><td>32k</td><td>0.208</td><td>0.565</td><td>0.125</td><td>0.136</td><td>0.118</td><td>0.125</td><td>0.292</td><td>0.334</td><td>0.175</td><td>0.009</td><td>0.193</td><td>0.193</td></tr>
  <tr><td>64k</td><td>0.298</td><td>0.565</td><td>0.125</td><td>0.136</td><td>0.118</td><td>0.125</td><td>0.292</td><td>0.334</td><td>0.175</td><td>0.009</td><td>0.193</td><td>0.193</td></tr>
  <tr><td>128k</td><td>0.336</td><td>0.320</td><td>0.036</td><td>0.083</td><td>0.116</td><td>0.031</td><td>0.035</td><td>0.055</td><td>0.179</td><td>0.021</td><td>0.164</td><td>0.134</td></tr>
  <tr><td>4k</td><td>0.334</td><td>0.293</td><td>0.036</td><td>0.259</td><td>0.126</td><td>0.233</td><td>0.311</td><td>0.565</td><td>0.428</td><td>0.151</td><td>0.230</td><td>0.230</td></tr>
  <tr><td>8k</td><td>0.315</td><td>0.187</td><td>0.069</td><td>0.240</td><td>0.109</td><td>0.081</td><td>0.359</td><td>0.466</td><td>0.471</td><td>0.151</td><td>0.241</td><td>0.220</td></tr>
  <tr><td>16k</td><td>0.279</td><td>0.240</td><td>0.050</td><td>0.116</td><td>0.125</td><td>0.081</td><td>0.335</td><td>0.307</td><td>0.450</td><td>0.113</td><td>0.194</td><td>0.186</td></tr>
  <tr><td>32k</td><td>0.325</td><td>0.261</td><td>0.036</td><td>0.148</td><td>0.139</td><td>0.149</td><td>0.280</td><td>0.482</td><td>0.414</td><td>0.111</td><td>0.266</td><td>0.216</td></tr>
  <tr><td rowspan="8">LWM-Text-1M</td><td>64k</td><td>0.294</td><td>0.267</td><td>0.059</td><td>0.178</td><td>0.113</td><td>0.118</td><td>0.304</td><td>0.512</td><td>0.519</td><td>0.128</td><td>0.190</td><td>0.208</td></tr>
  <tr><td>128k</td><td>0.293</td><td>0.267</td><td>0.044</td><td>0.042</td><td>0.205</td><td>0.000</td><td>0.250</td><td>0.189</td><td>0.429</td><td>0.016</td><td>0.197</td><td>0.167</td></tr>
</table>

Table 7: ARS scores in different input lengths \( l \). † indicates that the context \( X \) on the longest interval is right-truncated.

G.2 Instruction Following Performance in Six Capabilities

<table>
  <tr>
    <th>Models</th>
    <th>Format</th>
    <th>Logit</th>
    <th>Num</th>
    <th>Ori</th>
    <th>Recog</th>
    <th>Spat</th>
  </tr>
  <tr><td>GPT-4o</td><td>0.712</td><td>0.582</td><td>0.842</td><td>0.836</td><td>0.756</td><td>0.806</td></tr>
  <tr><td>GPT-4</td><td>0.789</td><td>0.509</td><td>0.765</td><td>0.921</td><td>0.615</td><td>0.729</td></tr>
  <tr><td>Qwen2.5-72B-Inst.†</td><td>0.727</td><td>0.468</td><td>0.703</td><td>0.840</td><td>0.640</td><td>0.718</td></tr>
  <tr><td>Llama-3.1-70B-Inst.</td><td>0.766</td><td>0.509</td><td>0.684</td><td>0.747</td><td>0.626</td><td>0.699</td></tr>
  <tr><td>Qwen2.5-32B-Inst.†</td><td>0.760</td><td>0.400</td><td>0.671</td><td>0.752</td><td>0.492</td><td>0.647</td></tr>
  <tr><td>C4AI-cmd-r-08-2024 (32B)†</td><td>0.721</td><td>0.480</td><td>0.678</td><td>0.833</td><td>0.453</td><td>0.508</td></tr>
  <tr><td>C4AI-cmd-r-v01 (35B)†</td><td>0.767</td><td>0.400</td><td>0.596</td><td>0.811</td><td>0.379</td><td>0.470</td></tr>
  <tr><td>Qwen2.5-72B†</td><td>0.615</td><td>0.344</td><td>0.526</td><td>0.770</td><td>0.408</td><td>0.524</td></tr>
  <tr><td>Llama-3.1-70B</td><td>0.492</td><td>0.253</td><td>0.437</td><td>0.594</td><td>0.290</td><td>0.410</td></tr>
  <tr><td>Qwen2.5-32B†</td><td>0.432</td><td>0.342</td><td>0.483</td><td>0.537</td><td>0.303</td><td>0.332</td></tr>
  <tr><td>Qwen2.5-14B-Inst.†</td><td>0.659</td><td>0.400</td><td>0.612</td><td>0.591</td><td>0.410</td><td>0.523</td></tr>
  <tr><td>InternLM2.5-7b-chat-1m</td><td>0.727</td><td>0.401</td><td>0.576</td><td>0.644</td><td>0.275</td><td>0.438</td></tr>
  <tr><td>Qwen2.5-7B-Inst.†</td><td>0.707</td><td>0.359</td><td>0.612</td><td>0.595</td><td>0.317</td><td>0.420</td></tr>
  <tr><td>Llama-3.1-8B-Inst.</td><td>0.465</td><td>0.349</td><td>0.539</td><td>0.627</td><td>0.393</td><td>0.553</td></tr>
  <tr><td>GLM-4-9b-chat-1m</td><td>0.672</td><td>0.363</td><td>0.604</td><td>0.548</td><td>0.329</td><td>0.364</td></tr>
  <tr><td>Qwen2.5-14B†</td><td>0.420</td><td>0.318</td><td>0.401</td><td>0.574</td><td>0.307</td><td>0.318</td></tr>
  <tr><td>LWM-Text-Chat-1M</td><td>0.505</td><td>0.265</td><td>0.370</td><td>0.578</td><td>0.186</td><td>0.312</td></tr>
  <tr><td>Llama-3.1-8B</td><td>0.586</td><td>0.328</td><td>0.331</td><td>0.521</td><td>0.088</td><td>0.303</td></tr>
  <tr><td>Qwen2.5-7B†</td><td>0.274</td><td>0.254</td><td>0.229</td><td>0.274</td><td>0.117</td><td>0.206</td></tr>
  <tr><td>LWM-Text-1M</td><td>0.293</td><td>0.105</td><td>0.103</td><td>0.374</td><td>0.044</td><td>0.208</td></tr>
</table>

Table 8: Instruction following performance in six core capabilities. † indicates that the context \( X \) on the longest interval is right-truncated.
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Length</th>
    <th colspan="6">Formal</th>
    <th colspan="6">Logit</th>
    <th colspan="6">Ori</th>
    <th colspan="6">Recog</th>
    <th colspan="6">Spat</th>
  </tr>
  <tr>
    <th>Logit</th>
    <th>Logit</th>
    <th>Ori</th>
    <th>Spat</th>
    <th>Logit</th>
    <th>Logit</th>
    <th>Ori</th>
    <th>Spat</th>
    <th>Logit</th>
    <th>Logit</th>
    <th>Ori</th>
    <th>Spat</th>
    <th>Logit</th>
    <th>Logit</th>
    <th>Ori</th>
    <th>Spat</th>
    <th>Logit</th>
    <th>Logit</th>
    <th>Ori</th>
    <th>Spat</th>
  </tr>
  <tr>
    <td>GPT-<i>lo</i></td>
    <td>4k</td><td>0.683</td><td>0.519</td><td>0.833</td><td>0.851</td><td>0.896</td><td>0.853</td><td>0.878</td><td>0.701</td><td>0.630</td><td>0.859</td><td>0.852</td><td>0.909</td><td>0.826</td>
  </tr>
  <tr>
    <td></td>
    <td>8k</td><td>0.712</td><td>0.569</td><td>0.862</td><td>0.866</td><td>0.927</td><td>0.862</td><td>0.847</td><td>0.715</td><td>0.619</td><td>0.873</td><td>0.906</td><td>0.836</td><td>0.846</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.715</td><td>0.619</td><td>0.873</td><td>0.906</td><td>0.836</td><td>0.846</td><td>0.732</td><td>0.633</td><td>0.887</td><td>0.866</td><td>0.741</td><td>0.834</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.732</td><td>0.633</td><td>0.887</td><td>0.866</td><td>0.741</td><td>0.834</td><td>0.737</td><td>0.592</td><td>0.783</td><td>0.792</td><td>0.630</td><td>0.773</td>
  </tr>
  <tr>
    <td></td>
    <td>64k</td><td>0.737</td><td>0.592</td><td>0.783</td><td>0.792</td><td>0.630</td><td>0.773</td><td>0.693</td><td>0.502</td><td>0.816</td><td>0.751</td><td>0.520</td><td>0.706</td>
  </tr>
  <tr>
    <td></td>
    <td>128k</td><td>0.768</td><td>0.564</td><td>0.817</td><td>0.944</td><td>0.853</td><td>0.837</td><td>0.786</td><td>0.564</td><td>0.817</td><td>0.944</td><td>0.853</td><td>0.837</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst.<sup>†</sup></td>
    <td>4k</td><td>0.768</td><td>0.581</td><td>0.853</td><td>0.893</td><td>0.860</td><td>0.818</td><td>0.790</td><td>0.545</td><td>0.761</td><td>0.953</td><td>0.806</td><td>0.818</td>
  </tr>
  <tr>
    <td></td>
    <td>8k</td><td>0.771</td><td>0.424</td><td>0.708</td><td>0.962</td><td>0.682</td><td>0.720</td><td>0.771</td><td>0.424</td><td>0.708</td><td>0.962</td><td>0.682</td><td>0.720</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.741</td><td>0.437</td><td>0.724</td><td>0.892</td><td>0.584</td><td>0.734</td><td>0.667</td><td>0.562</td><td>0.653</td><td>0.741</td><td>0.517</td><td>0.618</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.599</td><td>0.276</td><td>0.555</td><td>0.503</td><td>0.369</td><td>0.581</td><td>0.893</td><td>0.548</td><td>0.793</td><td>0.977</td><td>0.710</td><td>0.715</td>
  </tr>
  <tr>
    <td></td>
    <td>64k</td><td>0.864</td><td>0.402</td><td>0.740</td><td>0.898</td><td>0.643</td><td>0.704</td><td>0.793</td><td>0.409</td><td>0.693</td><td>0.842</td><td>0.507</td><td>0.652</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst.<sup>†</sup></td>
    <td>4k</td><td>0.793</td><td>0.409</td><td>0.693</td><td>0.842</td><td>0.507</td><td>0.652</td><td>0.643</td><td>0.375</td><td>0.599</td><td>0.628</td><td>0.425</td><td>0.642</td>
  </tr>
  <tr>
    <td></td>
    <td>8k</td><td>0.706</td><td>0.406</td><td>0.615</td><td>0.673</td><td>0.399</td><td>0.648</td><td>0.662</td><td>0.261</td><td>0.583</td><td>0.495</td><td>0.268</td><td>0.520</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.759</td><td>0.625</td><td>0.767</td><td>0.944</td><td>0.606</td><td>0.555</td><td>0.754</td><td>0.505</td><td>0.370</td><td>0.891</td><td>0.510</td><td>0.515</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.742</td><td>0.376</td><td>0.583</td><td>0.825</td><td>0.358</td><td>0.462</td><td>0.597</td><td>0.353</td><td>0.562</td><td>0.866</td><td>0.319</td><td>0.469</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-v01<sup>†</sup></td>
    <td>4k</td><td>0.779</td><td>0.402</td><td>0.714</td><td>0.922</td><td>0.723</td><td>0.464</td><td>0.772</td><td>0.323</td><td>0.547</td><td>0.860</td><td>0.193</td><td>0.379</td>
  </tr>
  <tr>
    <td></td>
    <td>8k</td><td>0.457</td><td>0.353</td><td>0.558</td><td>0.589</td><td>0.430</td><td>0.502</td><td>0.447</td><td>0.258</td><td>0.452</td><td>0.618</td><td>0.342</td><td>0.363</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>16k</td><td>0.549</td><td>0.250</td><td>0.512</td><td>0.644</td><td>0.340</td><td>0.305</td><td>0.538</td><td>0.268</td><td>0.452</td><td>0.643</td><td>0.308</td><td>0.512</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.489</td><td>0.203</td><td>0.385</td><td>0.570</td><td>0.251</td><td>0.394</td><td>0.483</td><td>0.216</td><td>0.369</td><td>0.570</td><td>0.190</td><td>0.343</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst.<sup>†</sup></td>
    <td>4k</td><td>0.733</td><td>0.628</td><td>0.767</td><td>0.729</td><td>0.633</td><td>0.622</td><td>0.813</td><td>0.507</td><td>0.767</td><td>0.759</td><td>0.399</td><td>0.503</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.676</td><td>0.415</td><td>0.595</td><td>0.647</td><td>0.462</td><td>0.527</td><td>0.649</td><td>0.293</td><td>0.554</td><td>0.501</td><td>0.339</td><td>0.458</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.653</td><td>0.296</td><td>0.565</td><td>0.504</td><td>0.291</td><td>0.492</td><td>0.527</td><td>0.240</td><td>0.513</td><td>0.404</td><td>0.180</td><td>0.377</td>
  </tr>
  <tr>
    <td>GLM-4-9b-chat-1m</td>
    <td>4k</td><td>0.841</td><td>0.487</td><td>0.692</td><td>0.810</td><td>0.470</td><td>0.536</td><td>0.813</td><td>0.465</td><td>0.759</td><td>0.791</td><td>0.392</td><td>0.465</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.749</td><td>0.381</td><td>0.628</td><td>0.728</td><td>0.401</td><td>0.456</td><td>0.644</td><td>0.307</td><td>0.506</td><td>0.426</td><td>0.247</td><td>0.387</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.652</td><td>0.270</td><td>0.600</td><td>0.486</td><td>0.245</td><td>0.321</td><td>0.545</td><td>0.242</td><td>0.496</td><td>0.361</td><td>0.141</td><td>0.317</td>
  </tr>
  <tr>
    <td>LWM-Text-Chat-1M</td>
    <td>4k</td><td>0.726</td><td>0.482</td><td>0.727</td><td>0.629</td><td>0.461</td><td>0.488</td><td>0.716</td><td>0.458</td><td>0.682</td><td>0.579</td><td>0.423</td><td>0.374</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.713</td><td>0.358</td><td>0.624</td><td>0.605</td><td>0.341</td><td>0.332</td><td>0.672</td><td>0.323</td><td>0.591</td><td>0.584</td><td>0.286</td><td>0.350</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.626</td><td>0.312</td><td>0.522</td><td>0.498</td><td>0.261</td><td>0.321</td><td>0.626</td><td>0.312</td><td>0.522</td><td>0.498</td><td>0.261</td><td>0.321</td>
  </tr>
  <tr>
    <td></td>
    <td>64k</td><td>0.578</td><td>0.246</td><td>0.477</td><td>0.391</td><td>0.199</td><td>0.319</td><td>0.640</td><td>0.313</td><td>0.370</td><td>0.534</td><td>0.107</td><td>0.334</td>
  </tr>
  <tr>
    <td></td>
    <td>128k</td><td>0.557</td><td>0.329</td><td>0.345</td><td>0.506</td><td>0.081</td><td>0.251</td><td>0.570</td><td>0.365</td><td>0.351</td><td>0.548</td><td>0.087</td><td>0.361</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B<sup>†</sup></td>
    <td>4k</td><td>0.807</td><td>0.543</td><td>0.759</td><td>0.781</td><td>0.694</td><td>0.648</td><td>0.959</td><td>0.488</td><td>0.390</td><td>0.582</td><td>0.667</td><td>0.432</td>
  </tr>
  <tr>
    <td></td>
    <td>8k</td><td>0.807</td><td>0.538</td><td>0.801</td><td>0.732</td><td>0.711</td><td>0.673</td><td>0.923</td><td>0.384</td><td>0.590</td><td>0.685</td><td>0.392</td><td>0.460</td>
  </tr>
  <tr>
    <td></td>
    <td>16k</td><td>0.783</td><td>0.586</td><td>0.739</td><td>0.833</td><td>0.722</td><td>0.728</td><td>0.904</td><td>0.347</td><td>0.563</td><td>0.647</td><td>0.325</td><td>0.350</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.703</td><td>0.458</td><td>0.682</td><td>0.722</td><td>0.864</td><td>0.565</td><td>0.720</td><td>0.384</td><td>0.580</td><td>0.685</td><td>0.392</td><td>0.544</td>
  </tr>
  <tr>
    <td>Llama-1-3.1-8B</td>
    <td>16k</td><td>0.949</td><td>0.363</td><td>0.458</td><td>0.645</td><td>0.253</td><td>0.278</td><td>0.477</td><td>0.178</td><td>0.250</td><td>0.464</td><td>0.065</td><td>0.317</td>
  </tr>
  <tr>
    <td></td>
    <td>32k</td><td>0.467</td><td>0.167</td><td>0.176</td><td>0.416</td><td>0.098</td><td>0.107</td><td>0.310</td><td>0.110</td><td>0.161</td><td>0.205</td><td>0.044</td><td>0.155</td>
  </tr>
  <tr>
    <td>Llama-2-1-8B-Prof</td>
    <td>16k</td><td>0.280</td><td>0.280</td><td>0.235</td><td>0.099</td><td>0.095</td><td>0.325</td><td>0.204</td><td>0.095</td><td>0.109</td><td>0.095</td><td>0.095</td><td>0.105</td>
  </tr>
</table>

Table 9: Instruction following performance in different input lengths l. † indicates that the context X on the longest interval is right-truncated.
G.3 Instruction Following Stability

<table>
  <tr>
    <th>Models</th>
    <th>4k</th>
    <th>8k</th>
    <th>16k</th>
    <th>32k</th>
    <th>64k</th>
    <th>128k</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.102</td>
    <td>0.124</td>
    <td>0.100</td>
    <td>0.124</td>
    <td>0.140</td>
    <td>0.155</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.073</td>
    <td>0.075</td>
    <td>0.086</td>
    <td>0.116</td>
    <td>0.121</td>
    <td>0.188</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst.<sup>†</sup></td>
    <td>0.100</td>
    <td>0.103</td>
    <td>0.092</td>
    <td>0.094</td>
    <td>0.120</td>
    <td>0.136</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Inst.</td>
    <td>0.127</td>
    <td>0.124</td>
    <td>0.118</td>
    <td>0.154</td>
    <td>0.150</td>
    <td>0.261</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst.<sup>†</sup></td>
    <td>0.109</td>
    <td>0.137</td>
    <td>0.152</td>
    <td>0.164</td>
    <td>0.152</td>
    <td>0.169</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-08-2024<sup>†</sup></td>
    <td>0.121</td>
    <td>0.133</td>
    <td>0.177</td>
    <td>0.180</td>
    <td>0.229</td>
    <td>0.226</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-v01<sup>†</sup></td>
    <td>0.171</td>
    <td>0.191</td>
    <td>0.233</td>
    <td>0.185</td>
    <td>0.191</td>
    <td>0.189</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B<sup>†</sup></td>
    <td>0.193</td>
    <td>0.227</td>
    <td>0.207</td>
    <td>0.214</td>
    <td>0.319</td>
    <td>0.243</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>0.485</td>
    <td>0.510</td>
    <td>0.301</td>
    <td>0.311</td>
    <td>0.301</td>
    <td>0.235</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B<sup>†</sup></td>
    <td>0.218</td>
    <td>0.224</td>
    <td>0.749</td>
    <td>1.250</td>
    <td>0.426</td>
    <td>0.521</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst.<sup>†</sup></td>
    <td>0.150</td>
    <td>0.184</td>
    <td>0.181</td>
    <td>0.207</td>
    <td>0.187</td>
    <td>0.196</td>
  </tr>
  <tr>
    <td>InternLM2.5-7b-chat-1m</td>
    <td>0.145</td>
    <td>0.157</td>
    <td>0.152</td>
    <td>0.173</td>
    <td>0.202</td>
    <td>0.265</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Inst.<sup>†</sup></td>
    <td>0.153</td>
    <td>0.146</td>
    <td>0.181</td>
    <td>0.217</td>
    <td>0.259</td>
    <td>0.296</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Inst.</td>
    <td>0.221</td>
    <td>0.244</td>
    <td>0.203</td>
    <td>0.271</td>
    <td>0.228</td>
    <td>0.225</td>
  </tr>
  <tr>
    <td>GLM-4-9b-chat-1m</td>
    <td>0.165</td>
    <td>0.163</td>
    <td>0.195</td>
    <td>0.214</td>
    <td>0.254</td>
    <td>0.229</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B<sup>†</sup></td>
    <td>0.249</td>
    <td>0.271</td>
    <td>0.230</td>
    <td>0.521</td>
    <td>0.515</td>
    <td>0.806</td>
  </tr>
  <tr>
    <td>LWM-Text-Chat-1M</td>
    <td>0.164</td>
    <td>0.341</td>
    <td>0.334</td>
    <td>0.196</td>
    <td>0.373</td>
    <td>0.379</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.356</td>
    <td>0.349</td>
    <td>0.276</td>
    <td>0.330</td>
    <td>0.322</td>
    <td>0.406</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B<sup>†</sup></td>
    <td>0.740</td>
    <td>0.877</td>
    <td>0.319</td>
    <td>1.418</td>
    <td>0.568</td>
    <td>0.866</td>
  </tr>
  <tr>
    <td>LWM-Text-1M</td>
    <td>0.200</td>
    <td>0.366</td>
    <td>0.469</td>
    <td>0.333</td>
    <td>0.349</td>
    <td>0.608</td>
  </tr>
</table>

Table 10: Instruction following stability (Expression) in different input lengths \( l \). † indicates that the context \( X \) on the longest interval is right-truncated.

<table>
  <tr>
    <th>Models</th>
    <th>4k</th>
    <th>8k</th>
    <th>16k</th>
    <th>32k</th>
    <th>64k</th>
    <th>128k</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.093</td>
    <td>0.085</td>
    <td>0.101</td>
    <td>0.105</td>
    <td>0.117</td>
    <td>0.164</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>0.089</td>
    <td>0.097</td>
    <td>0.119</td>
    <td>0.145</td>
    <td>0.183</td>
    <td>0.187</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B-Inst.<sup>†</sup></td>
    <td>0.074</td>
    <td>0.108</td>
    <td>0.132</td>
    <td>0.141</td>
    <td>0.179</td>
    <td>0.142</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B-Inst.</td>
    <td>0.113</td>
    <td>0.075</td>
    <td>0.125</td>
    <td>0.122</td>
    <td>0.198</td>
    <td>0.273</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B-Inst.<sup>†</sup></td>
    <td>0.136</td>
    <td>0.132</td>
    <td>0.177</td>
    <td>0.165</td>
    <td>0.152</td>
    <td>0.184</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-08-2024<sup>†</sup></td>
    <td>0.140</td>
    <td>0.156</td>
    <td>0.148</td>
    <td>0.156</td>
    <td>0.205</td>
    <td>0.147</td>
  </tr>
  <tr>
    <td>C4AI-cmd-r-v01<sup>†</sup></td>
    <td>0.167</td>
    <td>0.146</td>
    <td>0.151</td>
    <td>0.184</td>
    <td>0.175</td>
    <td>0.196</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B<sup>†</sup></td>
    <td>0.143</td>
    <td>0.120</td>
    <td>0.168</td>
    <td>0.223</td>
    <td>0.271</td>
    <td>0.211</td>
  </tr>
  <tr>
    <td>Llama-3.1-70B</td>
    <td>0.405</td>
    <td>0.500</td>
    <td>0.272</td>
    <td>0.183</td>
    <td>0.172</td>
    <td>0.220</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B<sup>†</sup></td>
    <td>0.203</td>
    <td>0.146</td>
    <td>0.302</td>
    <td>0.600</td>
    <td>0.254</td>
    <td>0.483</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Inst.<sup>†</sup></td>
    <td>0.162</td>
    <td>0.186</td>
    <td>0.193</td>
    <td>0.161</td>
    <td>0.192</td>
    <td>0.145</td>
  </tr>
  <tr>
    <td>InternLM2.5-7b-chat-1m</td>
    <td>0.191</td>
    <td>0.155</td>
    <td>0.181</td>
    <td>0.217</td>
    <td>0.336</td>
    <td>0.240</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Inst.<sup>†</sup></td>
    <td>0.156</td>
    <td>0.169</td>
    <td>0.204</td>
    <td>0.216</td>
    <td>0.252</td>
    <td>0.187</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Inst.</td>
    <td>0.149</td>
    <td>0.167</td>
    <td>0.183</td>
    <td>0.320</td>
    <td>0.212</td>
    <td>0.249</td>
  </tr>
  <tr>
    <td>GLM-4-9b-chat-1m</td>
    <td>0.205</td>
    <td>0.195</td>
    <td>0.187</td>
    <td>0.227</td>
    <td>0.230</td>
    <td>0.239</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B<sup>†</sup></td>
    <td>0.200</td>
    <td>0.214</td>
    <td>0.182</td>
    <td>0.233</td>
    <td>0.308</td>
    <td>0.572</td>
  </tr>
  <tr>
    <td>LWM-Text-Chat-1M</td>
    <td>0.158</td>
    <td>0.159</td>
    <td>0.256</td>
    <td>0.195</td>
    <td>0.234</td>
    <td>0.206</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B</td>
    <td>0.384</td>
    <td>0.233</td>
    <td>0.268</td>
    <td>0.211</td>
    <td>0.235</td>
    <td>0.427</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B<sup>†</sup></td>
    <td>0.456</td>
    <td>0.420</td>
    <td>0.145</td>
    <td>0.816</td>
    <td>0.364</td>
    <td>0.499</td>
  </tr>
  <tr>
    <td>LWM-Text-1M</td>
    <td>0.158</td>
    <td>0.379</td>
    <td>0.331</td>
    <td>0.281</td>
    <td>0.311</td>
    <td>0.533</td>
  </tr>
</table>

Table 11: Instruction following stability (Variable) in different input lengths \( l \). † indicates that the context \( X \) on the longest interval is right-truncated.
<table>
  <tr>
    <th>Task Name</th>
    <th>Test Example</th>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Single-ID</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i> Retrieve the entry at position <span style="color:red;">8th</span> in the list. Display it immediately.</td>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Multi-ID</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i> Identify the items at the corresponding places in list [1, 5, 7] and deliver the result in JSON list form.</td>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Offset-ID</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i> Please identify <span style="color:red;">the next</span> item to <span style="color:red;">8th</span> in the list described above.</td>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Offset-Element</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i> "Considering the arrangement of the list, what is <span style="color:red;">the next element after '4f78df6f47f5111ef8b42581122bf941e'?"</span></td>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Blur-ID</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i>Randomly select an item from the list above, after the 8th element.</td>
  </tr>
  <tr>
    <td>Scenario: <i>List</i><br>Task: <i>Blur-Element</i></td>
    <td>You're a searcher. You need to output the corresponding list elements based on the instructions and the list below. Please follow the instructions directly without anything else.<br>List to be retrieved:<br>1. 4f63efbe7f5111ef8b42581122bf941e<br>2.Summarize the Rome Statute of the International Criminal Court.<br>...<br>153. 4f7ea64c7f5111ef8b42581122bf941e<br><i>Instruction:</i> Retrieve the entry at position <span style="color:red;">8th</span> in the list. Display it immediately.</td>
  </tr>
</table>
<table>
  <tr>
    <th>Task Name</th>
    <th>Test Example</th>
  </tr>
  <tr>
    <td>Scenario: <i>MultiDoc</i><br>Task: <i>Find-dup-doc</i></td>
    <td>You are a document manager. Here is a collection of documents. Each document includes information such as title, date, source, id, iD2 and specific article content (text). You need to read the documents and follow the instructions to give some information directly, without something else. Also note:
      <ol>
        <li>Some documents may be missing information such as title or source, which may affect the final output.</li>
        <li>Some article content (i.e. values corresponding to the text keyword) may be duplicated.</li>
      </ol>
      Documents:
      <pre>
      *************** doc-1 ***************
      text: Following discussions with the PSNI, ...
      source: meeting
      iD2: bec60ab1-35bc-417d-b378-0b2e86dfcd23
      id: xWTp7xEyQrqGmGj5p6CdVQ
      title: 39272756
      date: 2016-06-20
      *************** doc-2 ***************
      id: SOI-GT2FSyKFYJzBhkYzxw
      text: What did the participants think about using CD’s for backup? ...
      source: news
      date: 2001-01-15
      iD2: 12a0f93d-0f57-46bc-bf28-9ec856fc974a
      title: tr-sq-167
      *************** doc-3 ***************
      ....
      </pre>
      <i>Instruction: Within the supplied documents, certain documents contain replicated content in their 'text' field, although other fields (such as id, iD2, date, title, source) may be different. Additionally, there could be N sets of documents, each set comprising any number of replicates. Please identify these replicated documents and present iD2 in sequence. The output should have N lines, each line symbolizing a set of replicated documents. Format the output for each document set as depicted in the example. Avoid providing explanations. If a document lacks information in a specific field, use 'None' instead.</i>
      <i>output example:</i>
      <pre>
      ["iD2_1", "iD2_2", "iD2_3"]
      ["iD2_4", "iD2_5"]
      </pre>
  </td>
  </tr>
  <tr>
    <td>Scenario: <i>MultiDoc</i><br>Task: <i>Batch-label</i></td>
    <td>You are a document manager. Here is a collection of documents. Each document includes information such as title, date, source, id, iD2 and specific article content (text). You need to read the documents and follow the instructions to give some information directly, without something else. Also note:
      <ol>
        <li>Some documents may be missing information such as title or source, which may affect the final output.</li>
        <li>Some article content (i.e. values corresponding to the text keyword) may be duplicated.</li>
      </ol>
      Documents:
      <pre>
      *************** doc-1 ***************
      text: Following discussions with the PSNI, ...
      source: meeting
      iD2: bec60ab1-35bc-417d-b378-0b2e86dfcd23
      id: xWTp7xEyQrqGmGj5p6CdVQ
      title: 39272756
      date: 2016-06-20
      *************** doc-2 ***************
      id: SOI-GT2FSyKFYJzBhkYzxw
      text: What did the participants think about using CD’s for backup? ...
      source: news
      date: 2001-01-15
      iD2: 12a0f93d-0f57-46bc-bf28-9ec856fc974a
      title: tr-sq-167
      *************** doc-3 ***************
      ....
      </pre>
      <i>Instruction: Assign labels to documents in order using the provided list of ['11311', '22422', '33233', '44444']. The labeling rules to follow are:</i>
      <ol>
        <li>If the document contains both title and source information, mark it as "11311".</li>
        <li>If the document is missing the source information but not the title, mark it as "22422".</li>
        <li>If the document is missing title information but not source, mark it as "33233".</li>
        <li>If the document is missing both title and source information, mark it as "44444".</li>
      </ol>
      <i>The tags should be output in JSON dictionary format, for example:</i>
      <pre>
      {"doc1":"11311","doc2":"22422"}
      </pre>
  </td>
  </tr>
</table>
<table>
  <tr>
    <th>Task Name</th>
    <th>Test Example</th>
  </tr>
  <tr>
    <td>Scenario: <i>OneDoc</i><br>Task: <i>Repeat</i></td>
    <td>There are several different types of KEY SENTENCE in the input text, which are marked by special tags. These special tags a total of six kinds, respectively is <i>&lt;#Topic#&gt;</i>, <i>&lt;@argument@&gt;</i>, "<i>&lt;Transition&gt;"</i>", "<i>&lt;Summary&gt;"</i>", "<i>&lt;Evidence&gt;"</i>", "<i>&lt;Concession&gt;"</i>". Different tags represent different types of key sentence. If a sentence in the text is KEY SENTENCE, we will add a special tag with the same attribute to the beginning and end of the sentence. The head tag also contains id order information in the format <type-id>. For example, the head tag with type "#Topic#" and id 1 is <i>&lt;#Topic#-1&gt;</i>. Also note that when the head tag and tail tag attributes are inconsistent, this means that the sentence is a fake KEY SENTENCE. Please read the input text carefully and give the answer directly according to the instruction requirements.<br>Input text: Remember the essays you had to write in high school? Topic sentence, introductory paragraph, supporting paragraphs, conclusion. <i>&lt;#Topic#-2&gt;</i>With the result that writing is made to seem boring and pointless.<i>&lt;#Topic#</i> Who cares about symbolism in Dickens? ...<br><i>Instruction:</i> Provide <b>4</b> KEY SENTENCE and their categories directly. Display the output on <b>4</b> individual lines with each line containing a KEY SENTENCE and its category, separated by <b>|||</b>.<br><i>Output example:</i><br>[KEY_SENTENCE_1] ||| #Topic#<br>[KEY_SENTENCE_2] ||| *Evidence*</td>
  </tr>
  <tr>
    <td>Scenario: <i>OneDoc</i><br>Task: <i>Extract</i></td>
    <td>There are several different types of KEY SENTENCE in the input text, which are marked by special tags. These special tags a total of six kinds, respectively is <i>&lt;#Topic#&gt;</i>, <i>&lt;@argument@&gt;</i>, "<i>&lt;Transition&gt;"</i>", "<i>&lt;Summary&gt;"</i>", "<i>&lt;Evidence&gt;"</i>", "<i>&lt;Concession&gt;"</i>". Different tags represent different types of key sentence. If a sentence in the text is KEY SENTENCE, we will add a special tag with the same attribute to the beginning and end of the sentence. The head tag also contains id order information in the format <type-id>. For example, the head tag with type "#Topic#" and id 1 is <i>&lt;#Topic#-1&gt;</i>. Also note that when the head tag and tail tag attributes are inconsistent, this means that the sentence is a fake KEY SENTENCE. Please read the input text carefully and give the answer directly according to the instruction requirements.<br>Input text: Remember the essays you had to write in high school? Topic sentence, introductory paragraph, supporting paragraphs, conclusion. <i>&lt;#Topic#-2&gt;</i>With the result that writing is made to seem boring and pointless.<i>&lt;#Topic#</i> Who cares about symbolism in Dickens? ...<br><i>Instruction:</i> Gather every instance of KEY SENTENCE classified as <b>#Topic#</b>. The output should be a Json list arranged by ids. If none are found, provide an empty array.<br><i>Output Example 1:</i> [KEY_SENTENCE1, KEY_SENTENCE2, KEY_SENTENCE3,...]<br><i>Output Example 2:</i> []</td>
  </tr>
  <tr>
    <td>Scenario: <i>OneDoc</i><br>Task: <i>QA</i></td>
    <td>There are several different types of KEY SENTENCE in the input text, which are marked by special tags. These special tags a total of six kinds, respectively is <i>&lt;#Topic#&gt;</i>, <i>&lt;@argument@&gt;</i>, "<i>&lt;Transition&gt;"</i>", "<i>&lt;Summary&gt;"</i>", "<i>&lt;Evidence&gt;"</i>", "<i>&lt;Concession&gt;"</i>". Different tags represent different types of key sentence. If a sentence in the text is KEY SENTENCE, we will add a special tag with the same attribute to the beginning and end of the sentence. The head tag also contains id order information in the format <type-id>. For example, the head tag with type "#Topic#" and id 1 is <i>&lt;#Topic#-1&gt;</i>. Also note that when the head tag and tail tag attributes are inconsistent, this means that the sentence is a fake KEY SENTENCE. Please read the input text carefully and give the answer directly according to the instruction requirements.<br>Input text: Remember the essays you had to write in high school? Topic sentence, introductory paragraph, supporting paragraphs, conclusion. <i>&lt;#Topic#-2&gt;</i>With the result that writing is made to seem boring and pointless.<i>&lt;#Topic#</i> Who cares about symbolism in Dickens? ...<br><i>Instruction:</i> Is <i>"With the result that writing is made to seem boring and pointless." the KEY SENTENCE of the text?</i> Indicate <b>"True"</b> for yes, <b>"False"</b> for no.</td>
  </tr>
</table>

Table 12: Examples of all tasks in LIFBench. <b>Bold</b> denotes the scenario description \( D \); normal denotes the context \( X \); <i>italics</i> denotes instruction \( I \), with <span style="color:red">red</span> indicate the instruction variables \( var \), and the remaining black parts correspond to the instruction template \( tpl \).

Reviewer paper summaries: This paper proposes LIFBench, a dataset designed to evaluate instruction-following capabilities in long-context scenarios, along with its evaluation framework, LIFEval. LIFBench includes tasks classified into three main categories, such as List, MultiDoc, and OneDoc, which are further divided into 11 subcategories. The evaluation framework, LIFEval, incorporates Automated Rubric-based Scoring (ARS) for task performance, Instruction Following Stability (IFS) for stability, and Instruction Following Performance (IFP) as a supplementary metric to assess instruction-following ability. The paper conducts a comprehensive evaluation using 20 different LLMs.

The paper introduces LIFBENCH, a novel benchmark designed to evaluate the instruction-following performance and stability of large language models (LLMs) in long-context scenarios. LIFBENCH consists of three long-context scenarios and eleven diverse tasks, covering over 2,700 instructions expanded across three dimensions: length, expression, and variables. It incorporates LIFEVAL, a rubric-based automated scoring system for assessing complex model outputs without human judgment or reliance on other LLMs. Detailed experiments were conducted on 20 prominent LLMs, revealing insights into their long-context performance and stability.
Summary of strengths: - Proposes LIFBench, a benchmark designed to evaluate instruction-following capabilities in long-context scenarios.
- Introduces LIFEval, an evaluation framework for assessing both quality and stability.
- Provides an analysis of performance and stability in long-context environments using 20 LLMs.

1. This paper proposes a new benchmark especially focusing on long-context instruction following.
2. This paper proposes a rational method to expanse the dataset. The inclusion of diverse tasks spanning multiple dimensions ensures broad applicability.
3. This paper proposes an automatic evaluation method to assess the instruction following capacity of LLM in long-context scenario.
Summary of weaknesses: - **LIFBench**
    - **Motivation**: The paper mentions the motivation stated in L207: “to accurately simulate real-world application scenarios for LLMs in long-text processing.” However, it is unclear why evaluating the selected 11 tasks fulfills this objective. You should elaborate on this point and explain the rationale behind selecting these 11 tasks. Why are 11 tasks necessary, and why would 10 not suffice?
        - In my opinion, tasks like searching for a UUID in a long text seem disconnected from real-world applications. It is likely that simplifications were made for simulation purposes, but even so, you should clarify which real-world scenarios they aimed to simulate and how they designed these tasks accordingly.
        - Once again, you should justify why these tasks are reasonable choices. It is not enough to merely gather tasks; you should describe the criteria you followed to collect and design them.
    - **Presentation**: It is difficult to understand the nature of each task in LIFBench by reading Section 3.2 (Dataset Construction) alone. I could discern the tasks’ details by examining the prompts in the appendix. The main text should provide more concrete descriptions of the tasks.
        - The dataset creation appears to be the main contribution of this paper. However, the details about how it was created and what the tasks entail are hardly understandable from the main text. For instance, there is no illustration for MultiDoc and OneDoc tasks. The main text should be self-contained without relying on the appendix.
        - You should write not just task overviews but also ensure minimum reproducibility in the main content by clearly detailing how the dataset was constructed, why these tasks were necessary, and what these tasks involve.
    - **Others**:
        - In L267, the paper mentions, “We manually write scenario descriptions and instruction templates for all tasks.” However, starting from L293, it describes using GPT-4 for extensions. This cannot be considered fully “manual” as stated. Biases inherent in LLMs, such as those from structure or terminology, could implicitly influence the dataset. You should address whether such biases were accounted for.
        - Regarding the List Scenario, there is no information about whether the placeholder focuses on the beginning, middle, or end of the document. Additionally, details such as the distribution of correct answers should be included. The placeholder values used in the experiment are also unclear, making it difficult to interpret the results. If the dataset is designed for scalability, the experimental settings should be described explicitly.
- **LIFEval**
    - The scoring weight is unclear. The paper mentions that “We carefully designed” the scoring, but it does not provide details on what constitutes “carefully.” You should explain the reasoning behind the chosen score weights and justify why they are appropriate.
    - Section 4.2 (Score-Capability Mapping) becomes clear only after checking Table 5 in the appendix, where it is shown that not all six aspects are scored for every task. If the paper proposes an original evaluation scale, it should describe it in detail.
    - It is still unclear how the six core capabilities were measured. What criteria were used to determine the difficulty of a score? You need to provide objective evidence to show that the scoring is fair and not arbitrary.
- **General**
    - There seems to be a lack of mentions of related work. For instance, stability has already been addressed in works like [1], and instruction-following in long contexts has been explored in works like [2]. You should clarify how LIFBench is superior to these existing benchmarks. Additionally, citing relevant papers on MQM-based evaluation scales could strengthen the paper. Furthermore, while this dataset appears to be manually created, it seems to include synthetic data. You should clearly explain how it differs from the tasks mentioned in the related work of the paper, particularly from benchmarks like M4LE.
    - The experimental setup is unclear. For example, in Section 3.3 (Data Extension), apart from mentioning variations in length (4k-128k), there is no information about the number of expressions or variables used. This makes it difficult to interpret the experimental results.
    - In Table 3 of Section 5.2 (Results on LIFBench), since the paper’s main contribution is the proposed tasks, the focus should not be on model differences but on how the tasks compare. It should investigate trends and differences among tasks, such as LSI v.s. LOI or LSI v.s. LOI. rather than focusing on model performance.

---

[1] Sakai et al., "Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates",  arXiv:2408.12263.

[2] Gavin et al., "LongIns: A Challenging Long-context Instruction-based Exam for LLMs", arXiv:2406.17588.

1. Overall, long-context understanding and complex instruction following seem like two orthogonal tasks. Except for a few position-related instructions, many complex instruction followings of long texts seem to involve first understanding the long text and then answering complex instructions related to its content (in Table 11).
2. How can we ensure the representativeness of the metric? Figure 2 shows some defined metrics, but they are all hard constraint metrics and cannot fully reflect the model's ability to follow long-text-related instructions, such as some soft constraints related to semantics and reasoning.
3. When models perform poorly on LIFBENCH, what proportion is due to long-text understanding issues, and what proportion is due to complex instruction understanding issues? There seems to be a lack of relevant ablation in this aspect.
Comments, suggestions and typos: - If existing datasets are used in Dataset Construction, a statement regarding the licenses should be included. Furthermore, if LIFBench is to be distributed, the paper should explicitly mention that there are no license violations related to the dataset.

1. It would be better to outline the differences in the performance of various models on other long-text-related/complex instruction-related benchmarks compared to this benchmark. There is a need to highlight new findings and contributions discovered by this benchmark.
2. Definitions should be provided before being used. In Section 3.1, the author introduces the problem setting of the paper and describes various scenarios (e.g., "List," "OneDoc"). However, these terms are ambiguous, as their explanations are deferred to the following section.
Overall assessment scores: 3

3