LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios
Authors: Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Lu Xiangju, Junmin Zhu, Wei Zhang
Accepted at: acl25


Reviewer paper summaries: This paper proposes LIFBench, a dataset designed to evaluate instruction-following capabilities in long-context scenarios, along with its evaluation framework, LIFEval. LIFBench includes tasks classified into three main categories, such as List, MultiDoc, and OneDoc, which are further divided into 11 subcategories. The evaluation framework, LIFEval, incorporates Automated Rubric-based Scoring (ARS) for task performance, Instruction Following Stability (IFS) for stability, and Instruction Following Performance (IFP) as a supplementary metric to assess instruction-following ability. The paper conducts a comprehensive evaluation using 20 different LLMs.

The paper introduces LIFBENCH, a novel benchmark designed to evaluate the instruction-following performance and stability of large language models (LLMs) in long-context scenarios. LIFBENCH includes three long-context scenarios and eleven diverse tasks, covering over 2,700 instructions expanded across three dimensions: length, expression, and variables. It also features LIFEVAL, a rubric-based automated scoring system that assesses complex model outputs without requiring human judgment or reliance on other LLMs. Comprehensive experiments on 20 prominent LLMs provide valuable insights into their long-context performance and stability.

This paper presents LIFBench, an instruction-following benchmark for evaluating LLM performance in long-context scenarios, and LIFEval, a rubric-based evaluation method that eliminates the need for LLM-based or human judgment. The benchmark aims to assess instruction-following capabilities and stability across different NLP tasks such as retrieval and QA, addressing gaps in existing evaluation datasets.
Summary of strengths: - Proposes LIFBench, a benchmark designed to evaluate instruction-following capabilities in long-context scenarios.
- Introduces LIFEval, an evaluation framework for assessing both quality and stability.
- Provides an analysis of performance and stability in long-context environments using 20 LLMs.
- Addressed my comments from the previous submission.

1. This paper introduces a new benchmark specifically designed for long-context instruction following.
2. This paper presents a systematic method for expanding the dataset, incorporating diverse tasks across multiple dimensions to ensure broad applicability.
3. This paper proposes an automated evaluation method to assess the instruction-following ability of LLMs in long-context scenarios.

- The dataset focuses on long-context instruction following, which is crucial for real-world LLM applications but often overlooked in existing benchmarks.
- The proposed rubric-based evaluation (LIFEval) ensures that assessment does not rely on subjective LLM-based scoring or human annotators, improving consistency.
- The benchmark covers a variety of NLP tasks, including retrieval, QA, and document processing, making it useful for evaluating LLM capabilities in extended contexts.
- I feel that, this dataset is highly relevant to the research community, particularly for improving LLMs' performance in long-context instruction following.
Summary of weaknesses: I was Reviewer GfLk in the previous submission. I have carefully reviewed the revised manuscript and confirm that my previous comments have been addressed. Thank you for responding to them so promptly.

The following are minor comments and suggestions:

- L282: ample corpus → typo or capitalize?
- L340: “Table 5 in Appendix D”: While some figures or tables may be moved into the main content in the final version, in general, it is helpful to explicitly indicate whether a figure or table is in the appendix.
- I suggest including a mapping between the HuggingFace model IDs and the aliases used in the paper in Appendix E (Experiment Setup) to ensure reproducibility.
- It would be helpful to mention in Section 4.3 (Instruction Following Stability) that IFS is based on the coefficient of variation. Adding a reference to Appendix F would also improve clarity. Additionally, Sakai et al. (2024) propose a similar metric called the “Sharpe Score,” which focuses on the score itself, while the paper focuses on variance. If possible, it would be beneficial to clarify this distinction.
- LIFEBENCH and LIFEVAL: Appendix D has greatly improved the clarity of the paper. However, to make it more appealing to a wider audience, several clarifications would be helpful.
    - Is it necessary to use LIFEVAL when using LIFEBENCH?
    - When using LIFEBENCH or evaluating with LIFEVAL, is the use of the Rubric score required? If someone wishes to extend the dataset, the need to apply Rubric score weighting may become a constraint. This could present challenges when expanding the dataset or adapting it to a different target, so I recommend addressing this as a potential limitation.
    - As one of many benchmark datasets, it seems that the cost of using LIFEBENCH is relatively high. To promote broader use, it may help to state explicitly that LIFEBENCH can be used independently.
    - In other words, the paper should make it clear that LIFEBENCH is usable even without LIFEVAL. For example, if there is a high correlation between the simple average and the weighted score, then the average alone may be sufficient for many use cases. ARS could then be positioned as a stricter way for more rigorous evaluation. Such clarification could help make the dataset more attractive to a wider audience.
    - Overall, this paper gives the impression that LIFEBENCH is highly dependent on LIFEVAL. It would be better to explicitly state that LIFEBENCH is a standalone dataset that can be used independently.

1. The coverage of long-text scenarios is limited, only addressing List and Document cases. More long-text scenarios should be considered, such as extended conversations. The structured data examples are also limited to lists. Consider including more structured data types like tables, JSON, and markdown.
2. The OneDoc domain is too narrow, sourcing only from Paul Graham Essays. Consider incorporating articles from more diverse sources and domains.
3. Section 3's dataset description lacks critical details, such as statistical descriptions of difficulty control and comparative analysis of data before and after extending in length, expression, and variables. Better characterization of dataset diversity and quality is needed.
4. The significance of the benchmark needs to be better outlined through experimental results. Do models that perform well on long-text benchmarks (like LongBench) also perform well on your benchmark? Consider adding a radar chart to highlight performance differences across long-text understanding, instruction-following benchmarks, and LIFBench.
5. Some descriptions in the paper are confusing, such as Table 1's statement that "data length can be unlimited." What does this mean?

- The authors need to clarify the definitions of model stability vs. scalability to context length. For example, ∞Bench explicitly evaluates performance degradation at extreme lengths, yet Table 1 suggests it does not handle stability. A clearer distinction between stability and scalability is required.
- While LIFBench is valuable, the paper does not clearly differentiate it from existing benchmarks. Would it be more effective to showcase how LLM performance on prior datasets fails to capture long-context stability, while LIFBench addresses this gap?
- While Table 3 provides valuable results, it does not explicitly convey how LIFBench differs from prior datasets. A more direct comparison would strengthen the motivation for this benchmark.
- The paper does not explain how the weights for each rubric metric are determined. Since these weights influence evaluation, transparency in their selection is crucial.
- Can LIFEval's evaluation method be generalized to other datasets? If so, how should the S̃ tilda weights be chosen to ensure fair assessment across different benchmarks?
Comments, suggestions and typos: Please see the minor comments listed in the Weaknesses section.

1. Are you testing reasoning models?
2. Does a single prompt contain only one constraint?
3. Is the primary bottleneck in model performance related to long-text understanding or instruction following?

See Weakness secton
Overall assessment scores: 3.5

3

3