LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs
Authors: Yuhao Wu
Ming Shan Hee
Zhiqiang Hu
Roy Ka-Wei Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Long context LLMs; Long-form generation; Benchmark
Primary area: datasets and benchmarks

Abstract: Current benchmarks like ``$\textit{Needle-in-a-Haystack}$'' ($\textit{NIAH}$), $\textit{Ruler}$, and $\textit{Needlebench}$ focus on models' ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences—a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce $\textit{LongGenBench}$, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, $\textit{LongGenBench}$ evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on $\textit{Ruler}$, all models struggled with long text generation on $\textit{LongGenBench}$, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation. We open-source $\textit{LongGenBench}$ to promote comprehensive evaluation and improvement in this critical area, with code and data available at ${anonymousurl}$.

Published as a conference paper at ICLR 2025
LONGGENBENCH: BENCHMARKING LONG-FORM GENERATION IN LONG CONTEXT LLMs
Yuhao Wu', Ming Shan Hee', Zhiqing Hu' and Roy Ka-Wei Lee'
'Singapore University of Technology and Design yuhao, mi mingshan hee, zhi ging hu)emymai sut.ii.....ddd...... roy-leeate sutdd. edu.sg .g
ABSTRACT
Current benchmarks like "Needle-in- -n-dle--i---aee "aana-aysttcc" (NIAH), Ruler, and Needlebench focus on models' ability to understand long-context input sequences but fail to capture critical dimension: the generation of high-quality long-form text. Applications such design proposals, technical documentation, and creative writing rely on coherent, fostruction-folloiing outputs over extended sequences challenge that existing benchmarks do not adequately address. To fill this gap, we introduce Long GenBench, a novel benchmark designed to rigorously evaluate large language models' (LLMs) ability to generate long text while adhering complex instructions. Through tasks requiring specific events or constraints within generated text, LongGenBench evaluates model performance across four distinct scenarios, three instruction types, and two generation-Ienge tokens). Our evaluation of ten state-of-the-arr LLMs reveals that, despite strong results on Ruler, all models struggled with long text generation on LongGenBench, particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation. We open- source LongGenBench to promote comprehensive evaluation and improvement in this critical area, with code and data available at https: //tpt/ggtttt/////// com/ mozzuu62//ongGeneeen
(16K and 32K
INTRODUCTION
Recent advances in large language models (LLMs) have dramatically enhanced their ability to process long text sequences, supporting applications that range from document summarization to creative writing, Leading models such as GPT-4 (Achiam et al. 2023). LLaMa-3.2 (Dubey et al.) (2024), and Claude 2.1 (Anthropic 2024a) manage context windows of up 128K tokens, with the Claude series AAthrropic 2024b) handling inputs exceeding 1 million tokens. However, while much attention has been given these models' ability to retrieve and understand long input text sequences, far less focus has been placed on their ability to generate coherent and high-quality long-form text oulputs a critical requirement for tasks such as design proposals and creative writing.
Long-form text generation is crucial for real-world applications that require detailed, well-structured narratives, such as document summarization (Kumar et al., 2024), creative writing (Hua & Wang
2020: Hu et al.] 2022). and comprehensive question answering (Stelmakh et al.) 2022 Lee al.
2023 Bai et al. 2024). Despite this importance, current benchmarks are limited in their ability
to assess long- form generation, focusing instead shorter text outputs (5 2K tokens) (Fan et al.) 2018 2019a; Dasigi et al. 2021), making them unsuitable for tasks requiring outputs of > 16K tokens (Bai al. (024). The challenge is further compounded by the lack of robust methods for evaluating these long sequences. The ability to follow instructions essential for long text generation (Reversed NIAH[.. just as effective information retrieval fundamental for processing long-context inputs (NIAA] Kamradt 2023)). However, current benchmarks do not adequately assess whether the
Analogous to NIAH, which involves searching for needle (retrieval) within a long input, the reversed NIAH entails placing specific needle instruction-followinn at designated position within long output.

---

Published as a conference paper at ICLR 2025
Table 1:
Comparison of Long-Conters LLM Benchmarks. For the retrieval tasks' datasets, we measure length based on the number of processing tokens, while for the generation tasks' datasets, we calculate the average number of generation words produced by LLMs. 'Long-length' indicates
LLMs to analyze generate text that is at least 8K token.
Type of Task
Benchmark
Type of Data
Avg Len
Long-Length
Longbench Bai et al. 2023
hybrid
88
Retrieval
(IAH(Kamradl 2023
synthetic
Any
Ruler Hsieh et al. 2024
synthetic
Any
ELi5 Fan et al. 2019b
hybrid
0.2K
X
Generation
Longwrite(Ba et al. 2024
synthetic
2.7K
X
Oong
synthetic
~20K
generated text adheres to the specific directives of a prompt. For instance, a prompt may require incorporating specific information certain point in a lengthy document, but evaluations often fail to verify the model's 's compliance with such instructions. This oversight represents significant shortcoming in benchmarking, particularly because performance under explicit constraints typically predicts outcomes in tasks with more implicit constraints, such as story generation or academic paper production. If a model struggles with explicit requirements, it likely underperform in scenarios with subtler constraints.
Manual evaluations, while thorough, are both costly and impractical at scale. Meanwhile, automated evaluations using "LLMM-as-ajudde" methods (Zheng et al. 2024) often yield results that are difficult to interpret and may not align with human judgments, raising concerns about their reliability. This highlights the need for more specialized benchmarks capable of reliably assessing the quality of super-long- form text generation.
To address this gap. present LongGenBench, novel benchmark designed evaluate the quality of super-long-fore form text generated by long-conterd context LLMs. Unlike existing benchmarks that primarily test retrieval or reasoning over long imputs, LongGenBench focuses on the model's ability to generate content that follows complex instructions over extended sequences. Our benchmark introduces tasks that reflect real-world generation challenges, such as diary writing, menu planning, and urban design, where the text must adhere to specific constraints provided in the prompt. These tasks assess whether models can correctly incorporate specific details at designated points in the text, ensuring the generated content meets the requirements laid out in the prompt. By evaluating texts up to 32K tokens, LongGenBench is the first benchmark to systematically test the ability to generate instruction-complian long-form content across extended lengths. Table summarizes the different benchmarks supporting long-context retrieval and generation tasks.
The evaluation tasks are organized into four distinct scenarios: Diary Writing, Menu Design, Skyscraper Design, and Urban Planning, each with varying complexity. The scenarios involve sub-tasks such as single instance, range, and periodicity, simulating realistic constraints that model must account for. This setup allows us to measure the model's 's ability to generate detailed, contextually rich outputs that satisfy wide array of criteria.
In summary, our major contributions are as follows:
To the best of our knowledge, this is the first study to address the challenge of super-long- form generation in long-context language models, highlighting the critical importance of generating coherent, high-quality text in extended contexts.
We introduce LongGenBench. a comprehensive dataset that provides a diverse set of tasks specifically designed to evaluate the super-long-form generation capabilities of LLMs across varying token lengths (16K and 32K) and levels of text complexity.
We perform extensive experiments on both open-source and closed-source models, revealing that despite their advanced capabilities, most models struggle significantly with super-long- form generation tasks, particularly in maintaining instruction adherence and coherence over long outputs.
2

---

Published as a conference paper at ICLR 2025
Four Scenarios
DIARY
MENU
Weekly
Dally
Weekly
Daily
100 Fl
300
10% 10 Bik 19 Bi
Specific Task Instruction
Community
Woje's Birthday
Holiday
Vofunteers
June 22-28
May
Community Fotuntcers
Mile's wedding Anguse
S1 Templaate Lib
Tompllate Lib
PI Template Lib
May 15
To -Ta T
Tv (Tr, Testorr Range Instruction
Tp (Tn.Tau Temn Periodic Instruction
Single Instruction
Synthesize Instruction
Example
Suphis plodog (Mliin
Recording weekly dayy:
Input: Main task prompt + Single Instruction +
1) member birthda: husband
Range Instruction A Periodle Instruction + Generate prompt
(birhdaay May 13) ...
peosupe)
2) Attending a fashion week Paris
wock 24-15.
[R] promps)
Label:
Check set TR,
3) Allend parantnng workshpp
weeks weeisends.
prompt)
Figure 1: LongGenBench Overview:
1) Scenario Selection: Select from four scenarios Diary, Menu Design, Skyscraper Design, and Urban Planning each offered in both short and long versions to determine the main task prompt. 2) Task Instruction: Employ the template libraries SI (Single). RI (Range), and PI (Periodic) generate tasks characterized by random times locations, along with the corresponding prompts and verification sets. 3) Instruction Synthesis: Integrate all prompts generated the prior step to create comprehensive set of instructions with final check-set. 4) Example: An illustration of Sophia's 's weekly diary task is provided as an example.
LONGGENBENCH BENCHMARK
2.1 TASK DEFINITION
Evaluating the quality of super-long-form generation presents unique set of challenges due to the inherent complexity of long texts. Traditional human evaluation methods, while precise, are expensive and not scalable. Although using large language models for assessment is feasible, their lack of interpretability often hampers their utility. Thus, we focus on the cntrructionfooloiinng task super-long- generation, where the most must include specific details the generated text. This task reflects real-world scenarios that require high degree of attention to detail over extended sequences, such as technical documentation or detailed design proposals. this study, we define task type termed Strictly Sequential Tasks, which involves the sequential completion of subtasks = [Ti, T2........ T, 7 where each subtask is responsible for generating a specific volume of text. For instance, LLM might be tasked with designing 100-floor skyscraper, specifying the content and purpose of each floor.
2.2 FOUR DISTINCT SCENARIO SETUPS
To comprehensively assess the long-form generation capabilities of models, we have devised four distinct task scenarios supplement our predefined tasks, as illustrated in Figure (1). These scenarios fall into two categories: Temporal (Diary Writing, Menu Design) and Spatial (Skyscraper Design, Urban Planning). Moreover, each scenario incorporates both short and long versions assess the effectiveness of various output lengths.
These scenarios were carefully chosen to reflect both creative and technical long-form generation tasks. They offer a diverse set of challenges by including temporal tasks (e.g., Diary Writing)
IIn Appendix A there is detailed description of the definitions of mathematical symbols.
3

---

Published as a conference paper at ICLR 2025
that require maintaining consistent information time and spatial tasks (e.g., Urban Planning) that test the model's ability to handle spatial relationships and detailed designs. These scenarios mirror real-world applications, from planning documents to creative writing, and thus provide comprehensive evaluation long-context L.Mss Table offers comprehensive deseriptions for each scenario, with each designed around unique template to generate 100 different task instructions
Table 2: Scenario task descriptions
Category
Scenarios
Task
Task Description
Weekly Diary
Generate entries for each week of the year
Diary
Daily Diary
Generate entries for each day of the year
Temporal
Weekly Menu
Plan menus for each week of the year
Menu
Daily Menu
Plan menus for each day of the year
100-floor Design
Mlo-floor skyseraper Develop design for a 300-floor skyscraper
Develop a design
Skyscraper Design
361-floor Design
Spatial
10x10 block Design
Design urhan layout on 10x10 block grid
Urban Planning
19x19 block Design
Design an urban layout on 19x19 block grid
2.3 SPECIFIC TASK INSTRUCTION
To enhance task control and flexibility, we have developed three distinct task settings:
Single Instruction (SI): Injects specific information at a unique point within the generated text.
Tss = : Te.....
Range Instruction (RI): Requires the model to incorporate information within specified ranges of the text.
Tk (TR, TR TR
Periodic Instruction (Pl): Distributes information at predefined intervals throughout the text.
Tp (Tp,, Tp,m Tpm.n
Check Set: Includes tasks for all three aforementioned settings.
Check set (Ts, Tn, Tr)
For example, in the design of a 100-floor skyscraper, the Single Instruction may specify that the 34th floor hosts an aerial gym and the 54th floor houses law firm. The Range Instruction might designate floors through that starting from the 20th floor, every 10th floor incorporates a small aerial garden.
comprehensive shopping mall, whereas the Periodic Instruction could dictate
We utilize over 20 templates for each type of instruction, with the floors or locations being randomly assigned to ensure task diversity. These settings, applied via various templates, guarantee controlled coverage across all textual positions, thus facilitating comprehensive and efficient evaluation, as illustrated in Figure (2).
Through this approach, we generate the main task instructions T' and simultaneously acquire the corresponding Check _set, which supports subsequent evaluations and constructs task conducive super-long-form generation. Subsequently, we splice the main task prompt with the specific task instructions (STI)4 and add the generation prompt to form the final evaluation data.
2.4 EVALUATION METRIC
To quantitatively evaluate performance for LongGenBench tasks, we introduce three complementary metrics:
Examples of Task Instructions for each scenario are provided in Appendix B
Fach of our main task instructions T splice 5 single instructions, 1 range instruction task, and 1 periodic instruction task.
4

---

Published as a conference paper at ICLR 2025
Main Task Completion.
This metric evaluates the extent to which all designated subtasks accomplished. The completion rate is quantified using the following equation:
Completion Rate (CR) Number of Completed Subtasks
Total Number of Subtasks 100%
In this context, the numerator denotes the count of subtasks successfully executed by the model, and the denominator represents the total number subtasks defined the Strictly Sequential Task. For instance, does the model consistently complete a diary entry for each day without omitting any dates?
Specific Task Instruction Completion (STIC-1). This metric evaluates the model's adherence to specific task instructions. We calculate the completion counts for the Single Instruction (SI), Range Instruction (RI), and Periodic Instruction (PI), STIC-1 quantifies how well the model follows these instructions across subtasks, focusing on whether the instructions are correctly implemented. For example, in the Skyscraper Design task, if the model is instructed to place an aerial gym on the 34th floor and consistently places on a different floor, it would receive a lower STIC-1 score.
Single Instruction + Range Instruction Periodic Instruction
STIC-1
Total Number of Outputs to Specific Task Instructions
Specific Task Instruction Completion-2 (STIC-2). STIC-2 provides more granular assessment by measuring the overall completion of specific task instructions, including their presence and execution quality across all subtasks. In addition to adherence. it assesses whether the model consistently follows these instructions throughout the entire task. For instance, the model periodically repeats certain elements but not the required intervals, it would affect its STIC-2 score.
STIC-2 Single Instruction + Range Instruction Periodic Instruction
Total Number of Specific Task Instructions
STIC-1 is primarily concerned with the completion rate of instructions that result in sub-scenarios, focusing on whether instructions are correctly executed. contrast, STIC-2 assesses the overall completion of the specific instruction task, including the presence of sub-scenarios and their completion status]]
2.5 EVALUATIONS PIPELINE
Our evaluation process follows a structured pipeline: First, we use a long-context LLM complete the task instruction T. generating an answer A, which is then divided into sub-tasks as A = {A1, Ag..... A, Next, based on the specific instructions in the check_set, we identify the relevant sub-tasks within A. Finally, we evaluate each sub-task by eval( [,, T.) to compute the final completion score, as detailed in Algorithm [ This pipeline ensures that the evaluation both systematic and comprehensive, assessing the model's performance across different instruction settings and levels of complexity [ While LongGenBence primarily evaluates the model's 's ability to follow detailed instructions, future work could expand the benchmark to include more open-ended tasks that assess creativity and logical reasoning. This would provide a broader evaluation a model's capabilities in generating coherent, engaging, and logically sound long-form text.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
selected ten long-context large language models (LLMs), comprising eight open-source and closed-source models. These models range in size from 7B 72B parameters, with one featuring Mixture of Experts (MoE) architecture. The claimed context lengths of these models vary from 32K to 128K token? These models were selected to represent diverse array of architectures.
Models.
we provide detailed explanation of STIC-1 and STIC-2, along with case study analysis. there is a detailed evaluations pipeline and example. Detailed spectfications of these models provided in Appendix
'In Appendix E2
IIn Appendix
C
5

---

Published as a conference paper at ICLR 2025
Algorithm Evaluations Pipeline
Initialization:
1: Task instructions + T
2: Tested long context LM model
3: Set Special Task Instruction for evaluation matching -> Check
Main Process:
4: Use Tested model to get Answer for - A
5: A {A1, Ag,-.., Am). h, split into subtasks
6: empty set for storing evaluations +
7: for each Ti in Cleck Set do
8: if there is A; matching T; then
9:
eval(A., T3) E; E Add E,
11:
end
12:
end for
13: > E + Score, compute the final completion score
14:
return Score
covering both Mixture of Experts and standard transformer designs, as well as range of parameter sizes. This diversity ensures a comprehensive evaluation of their ability to handle long-context tasks.
Inference Setup. We utilized the vLLM (Koon et al. 2023 system, which optimizes key-value (KV) cache memory for efficient large-scale inference. This system is crucial for handling long-form generation efficiently, reducing memory overhead, and maximizing inference throughput. Inferences were performed using BFloat1 precision on NVIDIA A800 GPUs, employing greedy decoding to generate the outputs. This setup ensured consistency and efficiency in the inference process.
Task Configurations. For each scenario, we generated 800 examples two specified lengths: 16K tokens and 32K tokens. The generation was based on designated templates for each model, ensuring task-specific relevance. The tasks were selected to reflect both creative and technical long-form generation challenges, such as diary writing, urban planning, and skyscraper design. To ensure the relevance of the generated content and prevent off-topic responses or refusals to answer, we prefixed each task input with carefully curated answer prompt designed to guide the model's output. The tasks were specifically selected to test the models' ability to generate instruction-folloiin long-form content in both creative and technical contexts. For example: In the Urban Planning task, models were tasked with generating a detailed plan for a new urban district, including descriptions of key facilities such parks, schools, and transportation systems.
Evaluation Metric. We evaluated model performance using the three metrics defined in Section 2.4 Main Task Completion, Specificc Task Instruction Completion-1 (STIC-1), and Specific Task Instruction Completion-2 (STIC-2). These metrics provided a comprehensive assessment of the models' ability to adhere instructions and generate coherent long-form text.
3.2 MAIN RESULT
The results of the long-form text generation tasks for both Short-version (16K) and Long-version (32K) tokens are summarized Table
Main Task Completion. Significant disparities in performance across models primarily stem from differences in architecture and training datasets. Notably, models with varying parameter sizes, such as 1 1-8B-instruction (Dubey et al., 2024) (under 10 billion parameters), Qwen-72B (Yang) ct al. 2024) (over 20 billion parameters), and GPT-4o-mini (OpenAl (024a) (a closed-source model), have demonstrated superior efficacy, successfully completing most primary tasks in full. In contrast, some models struggle with these tasks, exhibiting limitations such as: 1) models responding solely to specified subtasks, neglecting others, and 2) models halting after only completing the initial task segment, despite prompts requiring full sequential subtask completion. This issue may originate from the current instructional tuning data, which could cause partial responses in complex, lengthy tasks.

---

Published as a conference paper at ICLR 2025
Table 3: Long-form generation Performance selected models evaluated at length from 16k and 32k. The weighted average score (wAvg) is the product of CR and STIC-2, used to represent the model's s final performance the given task length. Note that the GPT-4-32K is currently closed for use, and the longest versions that can be used are the GPT-4o and GPT-4o-mini 16K output limitation.
Models
Claimed Length
Short-version (16K)
Long-version (32K)
STIC-1 STIC-2
Len.
wAvg
STIC-1 STIC-2 Len. wAvg
Models with 7-10B Purameters
Mamba-2.8B
2K
11.3%
23.8%
2.1%
902
0.2%
5.6%
29.8%
1,6%
864
0.1%
FILM-7B
32K
36.0%
22.4%
9.0%
6280
3.2%
37.4%
30.9%
10.9%
13775
4.1%
Mistrial-vO.2-7B
32K
81.8%
25.7%
20.4%
7296
16.7%
48.2%
35.4%
15.7%
16146
7.6%
Phi-3-mini-3--8
128K
22.9%
27.6%
5.4%
4165
1.2%
7.4%
46.9%
2.4%
2613
0.2%
LLama3.1-88
128K
93.5%
23.4%
22.0%
8804
20.6%
77.6%
28.9%
20.6%
17354
16.0%
Qwen2-7B
128K
60.0%
23.3%
13.5%
5138
8.1
40.0%
31.7%
12.6%
9617
5.0%
Long ToigWriter-lieee Ilama3. 1-8B
128K
46.0%
32.6%
14.2%
11036
6.5%
34.5%
36.3%
10.8%
19925
3.7%
Models Larger Than 20B Parameters
Mixtral-8x7B
32K
83.0%
34.4%
27.2%
8113
22.6%
60.5%
36.3%
20.3%
15839
12.3%
Phi-3.5-8x7B
128K
26.9%
46.4%
11.3%
5430
3.0%
7,4%
62.9%
6.0%
6633
0.4%
LLama3.1-700
128K
79.3%
34.4%
29.2%
8055
23.1%
63.1%
43.3%
26.3%
15197
16.6%
Qwen2-72B
128K
94.3%
29.7%
27.1%
8013
25.5%
66.2%
34.4%
21.7%
19845
14.4%
Closed-source Model
GPT-4o-mini
128K
97.0%
34.8%
33.4%
8940
32.4%
GPT-4o
128K
67.2%
42.9%
24.4%
9055
15.3%
Especially in GPT-4o(Open.A) 2023), it recognizes that this task will generate a long output and only
provides a few examples.
STIC-1 and STIC-2. The 1 metric revealed strong performance in adhering to task instructions for models like LLama3. 1-7BB and GFT-4o-mind particularly in shorter sequences. However, significant drop in ST STIC-2 2 scores for several models indicates that maintaining instruction adherence over longer text sequences remains a challenge. This performance degradation emphasizes the need for better tuning and architectural modifications to improve long-term coherence.
common failure mode observed across multiple models was the tendency to forget or misinterpret instructions as the sequence length increased. For example, in the Skyscraper Design task, some models correctly described the initial few floors but deviated from the original plan as the task progressed, particularly in the 32K token setting. This highlights the memory retention issue in long-contex: models, which often leads to loss of coherence and adherence to task instructions. Examples of failures where models struggled follow instructions provided in Appendix F
Length (Number of words). We calculated the average output word count for models that consistently completed all subtasks, achieving at least an 80% completion rate in sub-scenarios, excluding data from unsuccessful attempts. Most models substantially exceeded previous benchmarks for long-form generation tasks in terms of output length. Notably, the LongWriter (Bai al. 2024) model excelled, efficiently meeting word count requirements for each subtask. Given the results and the weighted average score (wAvg) at a sequence length of 16K, the open-source Qwen2-72B and the closed-source GPT-4o models demonstrated optimal performance. At sequence length of 32K, the L.amma3 1-83 model, outperformed models with larger parameters, highlighting its efficiency in managing extended lengths.
3.3 ACCURACY TREND WITH VARYING SEQUENCE LENGTH
As illustrated in Figure 2 there is clear decline in model performance output length increases. Models exhibited strong adherence to initial instructions at shorter sequence lengths, but performance gradually degraded the text generation extended beyond the 4,000-token threshold. This degradation aligns with trends identified in the NIAH dataset and underscores the challenge of maintaining instruction adherence and coherence over long outputs.

---

Published as a conference paper at ICLR 2025
Accuracy with Varying Sequence Length
6P1 104
Ung/itte
longwiber
L1 20
1brra-31
0.50
Mistui-15
0.15
Owand 720
0.30
0.55
32K
The right side of the figure illustrates the model's performance on specific instruction tasks 16K as sequence length increases, whereas the left side depicts performance 32K. All curves have been smoothed with Moving Average.
Figure 2:
This deviation becomes particularly pronounced when outputs exceed 4,000 tokens, where adherence to instructions significantly diminishes, and further deterioration observed outputs approach 16,000 tokens. In contrast, tasks involving shorter outputs, such as those in the NIAH dataset or simpler multi-needle tasks, showed near near-perfect performance, highlighting the disparity in model behavior across different sequence lengths.
Potential reasons for this decline include limitations in the self-attention mechanism used in transform- ers, which may struggle to maintain meaningful context over long sequences. Additionally, models trained with limited long-form data may overfit to shorter patterns, leading to a loss of coherence in extended generations. These findings suggest that architectural changes improved training strategies may be necessary to overcome these challenges in future iterations of LLMs.
3.4
THREe SPECIFIC: TASK INSTRUCTIONS
presents the model's performance metrics across various task types: single, range, and periodic. The model demonstrated comparable proficiency in both single and range tasks, reflecting its capability to follow direct and straightforward instructions effectively. However, the slight reduction in performance for range tasks suggests that additional complexity, such as processing multiple data points within defined range, introduces marginal increase in cognitive load for the model.
Figure 3a
The most significant decline in performance was observed in periodic tasks, where the model struggled to interpret instructions that required recurring events, such "every four weeks starting from week 10." These tasks demand higher degree reasoning and temporal awareness, which may challenge the model's capacity maintain consistency over extended sequences. As result, outcomes for periodic tasks were considerably poorer compared to single and range tasks, which have clearer and more well-defined parameters. The model's performance hierarchy can generally be summarized as single >> range > periodic, highlighting the increased difficulty associated with periodic tasks. This trend underscores the need for future improvements long-context models, particularly in handling more complex, time-based instructions.
3.5 COMPARISON LONG-CONTEXT INPUT
We examine the relationship between model's ability handle long inputs and its performance on long outputs. Specifically, we investigate whether a model's capacity to manage long-range inputs corresponds to improved performance on long-rang outputs. For this analysis, we use the RULER dataset, a synthetic benchmark designed to flexibly configure sequence length and task complexity, making it ideal for comprehensive evaluations long-context LLMs. We compare the models' performance sequences of the same length, shown in Figure 3b which indicates significant performance gap between input handling and output performance. At 16K tokens, the Pearson correlation coefficient is 0.51, while at 32K tokens, it increases 0.66, suggesting that there is some overlap in the skills required for managing long inputs and generating long outputs, but these tasks not entirely equivalent.
8

---

Published as a conference paper at ICLR 2025
Three Specific Task Instruction Types
Models Ruler LongSenBench Tasks
0.40
0.35
Ueme-31-700
Matrak77
Mesrra-Se 3-18
0:30
Qwen2-18
0.25
0.20
LargGendench C20
0.15
0.10
0.05
20-
0.80
Single(Si
Range|Ri)
Perlodic/P1)
Owend-728
(a) Performance Comparison on three tasks settings
(b) Performance Comparison on Ruler and LongGenBench Tasks
Figure 3: The left Fig displays the models' performance three different task settings, with the red line representing the average for each category. The right fig shows the performance and correlation of the Ruler and LongGenBench the same length settings.
Handling long inputs primarily requires the model to retain and process existing information, while generating long outputs demands more complex reasoning, memory retention, and coherence management over extended sequences. Thus, models that excel in long-input retrieval may still struggle with long-form generation, particularly in tasks requiring strict instruction adherence time. This distinction highlights the need for models to be optimized for both input handling and output generation to achieve consistent performance in long-context tasks.
4 ANALYSIS AND LIMITATIONS
Richness of Content. Despite efforts to design sub-scenarios that enhance task diversity and richness, the model's outputs tend to converge as output volume increases. This results in a homogenization of recorded events, even when differences in time and location should introduce variety. Such convergence not only degrades overall performance but also diminishes the diversity of the generated content, leading to repetitive and predictable outputs. long outputs exhibited significant repetition, even when the model was given varied time or location prompts. Adjusting parameters like repet t on on_pena ty during inference has shown limited success in mitigating this issue, highlighting the need for more advanced techniques maintain content richness over long sequences.
our experiments, approximately 45% of
While our current research focuses primarily on evaluating instruction- following capabilities, a more comprehensive analysis of content rationality and coherence needed. For example, when tasked with generating a diary, the model should ensure that all recorded activities align with the specified careers. However, in many instances, this logical consistency lacking. Additionally, temperature records in virtual diary entries often fail to reflect realistic temporal changes. For instance, San Francisco's s diary task, we would expect temperatures to vary from cooler (0-10 degrees Celsius) at the beginning of the year to warmer (20-30 degrees Celsius) by mid-year. Yet, the model consistently generates warmer temperatures throughout, even into December. These issues may arise due the model's limited exposure temporally varied datasets, particularly in diary climate-related contexts. Future work could address this by incorporating more domain-specific and temporally annotated data during fine-tuning.
Rationality of Content.
Instruction Data. A significant performance discrepancy between models' abilities to handle long-range inputs (such as Ruler (Hsieh et al. 2024)) and their long-form output generation can likely be attributed to the length distribution of instruction-tuning data. Most instruction-tune tuning datasets are brief, typically under 200 tokens, and lack the extended instructional content necessary for generating longer outputs. This gap suggests that organizing or synthesizing instruction paaruction-tunigg data with longer, more comprehensive examples could be valuable direction for future research. Potential solutions

---

Published as a conference paper at ICLR 2025
include applying transfer learning techniques from models trained long- long-form datasets using data augmentation methods to synthesize longer instructional content from existing short-form data.
Generalizability. LongGenBench effectively evaluates instruction-follovii in creative and tech- nical tasks but may not fully capture the creativity and specialized knowledge required for abstract reasoning or unconstrained storytelling. Future versions could include open-ended tasks like creative fiction writing and legal document drafting, which demand intricate narratives and precision. Ex- panding in this direction would enhance the benchmark's s versatility while providing deeper insights into LLMs' capabilities. However, LongGeenencch' 's current focus on instruction adherence offers a strong foundation for evaluating practical, instruction-driven long-form text generation.
RELATED WORK
Instruction Following. Recent advances in instruction tuning models (Ouyang al. 2022 Rafailov et al.] 2024 OpenAT, 2022 Taori et al., 2023 Chiang et al.; 2023) have underscored the need for scalable evaluation methods. LLMs have been used as evaluators, showing better alignment with human judgments than traditional metrics like BLEU (Papineni et al., 2002). However, LLM evaluations suffer from biases, such as sensitivity to presentation order and preference for verbose outputs (Wang et al., 2024 PPezeskkpour' & Hruschka 2023 Zheng et al. [023)). To mitigate these biases, dicta-vvaluation benchmarks like FairEval. M MT-Bench, and LLMEval (Wang al., 20244 Zheng et al., 2023 Zhang et al., 2023) have been proposed. While recent studies have focused on improving L.M evaluations with diverse strategies (Zheng et al. 2023 Li et 2023 Zhang et al.
2023
Chan et al. 2023). they typically do not address longer context lengths.
Long-context Benchmarks and Tasks. Existing benchmarks focus on models handling long inputs. For instance, ZeroSCROLLS (Shaham et al. (023) and LongBench (Bai et al. 2023) tackle tasks like long-document QA and query-based summarization. Synthetic benchmarks, like NeedleBench (Li al. 2024) and Ruler (Hsich et al. 2024), offer better control over variables such as sequence length and complexity. BeecheBench introduces the Ancestral Trace Challenge (ATC), while Ruler evaluates models across tasks like NIAH and multi-hop tracing. However, these benchmarks largely focus on input comprehension and do not assess long-form text generation, which is the primary focus of LongGenBench.
Long-form Text Generation. Research in long-form generation spans applications like story generation (Fan et 2019c Xu al. 2220),, paragraph completion (Kang Hovy 2020), sustained
conversation (Xu et al.
2022).
, and comprehensive QA (Fan et al. 2019a Dasigi et al. [202]] Stelmakh
al. 2022 Lee et al. 2023). However, existing models and evaluation methods (Liu al. 2023]
Chiang & Lee 2023 Liu et al. 2024: Bai et al. 2024) face challenges in maintaining quality over long outputs, often being limited by shorter text lengths (typically under 2000 tokens) (Shen et al.)
2023 Recent work (Tan et al. 2024) seeks to improve evaluation criteria, but the gap between model capabilities and benchmark text lengths remains. In contrast, LongGenBench evaluates models on their ability to handle much longer sequences, with tasks requiring adherence to instructions over extended outputs (16K+ tokens).
CONCLUSION
We introduced LongGenBench, a synthetic benchmark that evaluates long-form generation capabilities of language models by testing their ability to follow instructions over extended sequences. In evaluating nine advanced models with context sizes ranging from 32K to 128K tokens, we observed significant performance degradation compared to benchmarks like "Ruler" with common failure modes including premature task termination, incomplete responses, disregard for instructions, and repetitive content generation. These results highlight key challenges for current models in handling long-fore form tasks and underscore the need for advancements in model architecture and training data to improve coherence, instruction adherence, and content diversity over extended outputs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research/project is supported by the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier I Grant, and funded through the SMU-SUTD Internal Research Grant Call.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian -lan-Guang Lou. Make your Ilm fully utilize the context. arXiv preprint arXiv:2404.1681,, 2024.
Anthropic. Introducing claude 2.1, 2024a. URI. -llude-2-1 Accessed: 2024-01-23.
Anthropic. Introducing the next generation of claude, 2024b. URL ARLLtt:sp///ww.tdt
comm httt//ews/cllain - Accessed: 2024-03-27.
3-
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longwriter: Unleashing 10,000+ word generation from long context Ilms, arXiv preprint MxXiv:2408.07055 2024.
Yushi Bai et al. LongBench: A bilingual, multitask benchmark for long context understanding. arXiv:2308.1 145008 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308. 002201, 2023.
David Cheng-Har Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 35607-15631 Association for Computational Linguistics, 2023, doi: 10. 10221885///1/222.... ACL-LONG.. URL https
//doi.org//i wtw.or//....111111122212211 wtt.sl1663vvv11
Long. 870
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https
03 30 1--300---0000 vicuna/
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A dataset of information-secking questions and answers anchored in research papers. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language
Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 4599-4610 4610. Association for Computational Linguistics, 2021. doi: 10. N0.18653/V1/2000 LAACLLLAAAAAAAAAIAA 7/doi. org.lorgg111 wtt.:o/gg........1.... 102.3/1/222///2222.... nwwcl ma in. 365
URL https
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407 21783, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne. Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v//P18 https: /ops://acc h antho ogy. oow..ogg/......e..
URL

---

Published as a conference paper at ICLR 2025
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: long form question answering. In Anna Korhonen, David R. Traum, and Lluís Märquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3538-3586 Association for Computational Linguistics, 2019a. doi: 10. 10.18653/V1/P19-1334 URL https: org/
10.18653/v1/ppl 1346.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. EL15: Long form question answering. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 3558-3567, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10. 11/8665/11//11199310 URL
http://wwooooooooooo...o/ 1346
Angela Fan, Mike Lewis, and Yann N. Dauphin. Strategies for structuring story generation. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational I Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 2650-26600 Association for Computational Linguistics. 2019c. doi: 1254
10.18653/V1/119-1222 URL htt.p::://o...og//111 http://....155//11////
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint wxXiv:244.006653 2024.
Zhe Hu, Hou Pong Chan, Jiachen Liu, Xinyan Xiao, Hua Wu, and Lifu Huang. PLANET: dynamic content planning in autoregressive transformers for long-form text generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2288-2288 2205.2305. Association for Computational Linguistics, 2022. 1eei1886333333 18653/11/22233 /ttppp////o...or//11111666...66.
doi:
URL https:://ostt wtg./.i.orrg..1. 111.11111666680
acl-long -on.. 163
Xinyu Hua and Lu Wang. PAIR: planning and iterative refinement in pre-trained transformers for long text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Association for Computational URL htl.hhttss///de orgg/0000///0
EMNLP 2020, Online, November 16-20, 2020, pp. Linguistics, 2020. doi:
1TWp:///V1/2200E2.PMM 18653/v1/2021 http:///22200.mm1.1.... emm.ppmmin.m..5.
Albert Q Jiang al. Mixtral of experts. M.Xi:2401.00485 2024.
Gregory Kamradt. Needle In A Haystack pressure testing LLMs. Github, 2023. URL https
//github. com cccom/gaamee akamraaa adtt/ LLMMest_NeedleeI
/main
Dongyeop Kang and Eduard H. Hovy. Plan ahead: Self-supervised text planning for paragraph completion task. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6533 365336 Association for Computational Linguistics, 2020. doi: 529, URL https: org/ 10 11111.1111111112202
emr.pp pwn.1p-main.50 mww..min..... 529
Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernon- court, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, et al. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint 11X.::2447.1101111 2024.
Woosuk Kwon et al. Efficient memory management for large language model serving with paged attention. In Proc. of the ACM SIGÓPS 29th Symposium on Operating Systems Principles, 2023.
Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jachyeon Kim, Hong-In Lee, and Moontae Lee. QASA: advanced question answering scientific articles. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19036- 19052. PMLR, 2023. URL https::///........... htkL/httss:///roceee //tpceee ngs. ml wwwpprssss2aa2
html
12

---

Published as a conference paper at ICLR 2025
Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can Ilms do retrieval and reasoning in million context window? arXiv preprint arXiv:2407.11963, 2024.
Ruosen Li, Teerth Patel, and Xinya Du Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint SiXXix:337.027666 2023.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (ods.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10, 2023, pp. 251 2522. Association for Computational Linguistics, 2023.
http://acaantthhhccoo ant logy org/ 2023. emn emnlp-main. 153
Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators, arXiv preprint arXiv:2403.16950 2024.
Mistral.AL. La plateforme, 2023. http://ttt////aaaaaaaaaaa///////
OpenAl. ChatGPT, 2022. URL WtLLhttpp//chat..ppaal....
OpenAI, GPT-4 Technical abhk doi: 10448550::Xxw.2233..87774
URL OBt/://..///00l.ooo/////5555//22
OpenAl. ott.ah/hwwh..iiiiiiaoooeoooooone
index/ http:///choccclclllaccoocccocrar 2024-08-31.
Accessed:
OpenAL. Hello got- 2024b. URL |tl/aattes//ggeaaaa.ccc////////a cessed: 2024-08-31.
Ac-
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730- 27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in caaltpple-choice questions. arXiv preprint arXiv:2308.1 11483, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: zero-shot benchmark for long text understanding. In EMNLP, 2023.
Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. Large language models yet human-level evaluators for abstractive summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 4215 4233. Association for Computational Linguistics, 2023. /ttl:ttss///eeeatoooooog org fte.2fffinningg eem.emmmpp..22 278
Ivan Stelmakh, Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: factoid questions meet long- form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2273-8288. Association for Computational Linguistics, 2022. doi: 10.18653/V1/022..EM MAIN...6. URL httttps://doi.og/////
06653/v1/200 emn 1p-ma 566
13

---

Published as a conference paper at ICLR 2025
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15044 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing Ilama model. https::/giite
com/ ttt///ttssuullaaa st stanford alpaca) 2023.
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440 9450, Bangkok, Thailand, August 2024, Association for Computational Linguistics. httptt 7/aclantho org..org//022 acl long. 511
Thomas Wolf al. Huggingface's Transformers: State-of-the-art natural language processing. arXiv:1910033771, 2019.
Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open- domain conversation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 5180-5197 Associa- tion for Computational Linguisties, 2022. doi: 1.111663/112022ALL-L000 ht.tps://do..orgg 10t.1/16533/ll
URL
. 356
Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri, Pascale Fung, Anima Anandkumar, and controllable story generation with external knowledge using large-scale language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20. 2020, pp. 2831-2844 Association for Computational Linguistics, 2020. doi: 11118/63//V112000MMM..I //doi. http:/1...111311111111 emn ma meain... 226
Bryan Catanzaro. MEGATRON-CNTES
URL https
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067. 2024.
Xinghua Zhang. Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer 1lm evaluators. arXiv preprint arXiv:2308.0m 01 0186, 2023.
Lianmi Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM--s--a a-jdee with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https
net. // 1
Lianmin Zheng, Win Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36. 2024.
14

---

Published as a conference paper at ICLR 2025
A
SYMBOL DEFINITIONS AND DESCRIPTIONS
This table presents definitions and descriptions for various symbols used in task-related contexts, providing an overview of the key terminologies and their roles.
Symbol
Definition
Description
T
Main Task
The primary goal task to be completed, such as designing skyscraper writing diary.
T:
Subtask
A smaller portion of the main task, each responsible
a specific part. e.g. designing specific floor.
T's
Single Instruction Task
A task requiring the model inject specific infor- mation at unique point in the generated text. A task requiring the model to incorporate infor- mation within specified range of the generated content.
Tx
Range Instruction Task
TTp
Periodic Instruction Task
A task that distributes specific information at pre- determined intervals throughout the text. Represents individual task from the Single In- struction Task set, focusing on a specific point in the text.
Ts.
Single Instruction Task
TR.
Range Instruction Task
Represents an individual task from the Range In- struction Task set, applied across a specific range. Represents an individual task from the Periodic Instruction Task set, recurring periodically through- out the text.
Tp.
Periodic Instruction Task
CR
Completion Rate
The percentage of successfully completed subtasks out the total number subtasks, used to evaluate task performance.
STIC
Specific Task Instruction Completion-1
Evaluates how well the model follows specific task instructions, including Single, Range, and Periodic Instructions. Focuses whether the instructions executed correctly.
STIC-2
Specific Task Instruction Completion-2
Provides
granular assessment, measuring not only adherence to instructions also consistency of execution throughout all subtasks. It looks at both presence and execution quality. Represents the complete response generated by the model for the main task.
"A"
Answer
A,
Subtask Änswer
Represents the specific answer output generated for individual subtask, corresponding to T,.
Table 4: Symbol Definitions and Descriptions
B PROMPT TEMPLATES FOR FOUR TASK SCENARIOS
Below are the example templates for the four task scenarios: Diary Writing, Menu Design, Skyseraper Design, and Urban Planning. In the Diary Writing template, professions and names are customizable variables, allowing for flexibility in the generated content.
15

---

Published as a conference paper at ICLR 2025
Diary for 2018
Emma is a photographer with a passion for chronicling her vibrant life through weekly diary
entries. Captures:
1) Single Instruction (SI): Birthdays of family members, wedding anniversaries, etc:;
2) Range Instruction (RI): Family beach vacation in Maui, Week-long road trip across the Pacific
Coast Highway, etc:;
3) Periodic Instruction (PI): Attend golf lessons at the local club, Join weekend hiking group.
etc.;
4) Weekly updates on weather changes, work developments, family life, and other interesting topics.
5) Use "#*#" to separate each weekly entry (e.g. example)
Generate complete weekly diary for Emma for the entire year of 2018. Start from January 1st, a Monday, marking the first week, and continue through to December 31st, the end of the 52nd week. Ensure that the diary consists of 52 entries, one for each week. Each diary entry should be at least 200 words. When the design of all 52 weeks is complete, use **ow finished **** to indicate the end of the document. Ensure clarity and continuity without any interruptions or omissions in the narrative throughout the year. *** started #**
#*# Week (January 1st January 7th):
Menu for 2018
As Chef Roy, a world-renowned chef globally renowned restaurant, you are tasked with designing an entire year's menu for 2018. Your menu should be varied and innovative, adhering to the following guidelines:
1) Single Instruction (SI): ("):("ll ('Independence Day Celebration", "2018-07-04" "American Apple
Pie"), Pie"",""Suun ("Summer Solstice Celebration", None, "Midsummer Night's Fish Fry"), etc;
2) Range Instruction (RI): ("Mushroom Season Specials", "Various Mushroom Dishes"),
("Sealood Season Extravaganza", "Fresh Seafood Platter"), etc:
3) Periodic Instruction (PI): ("Seafood Fridays". 2, "Fish and Chips"), ("Monthly Steak Night",
3, "Prime Ribeye Steak"), etc.;
4) Use #*# to separate each weekly menu (e.g. example) Generate comprehensive weekly menu diary for the entire year of 2018, start from January 1st, a Monday, marking the first week, and continuing until December 31, the end of the 52nd week. Ensure that the diary consists of 52 entries, one for each week. Each weekly menu must include a detailed description of the offerings, featuring least two options for appetizers, main courses, side dishes. desserts. and drinks. Ideally, between 200 and 220 words per menu description to ensure thoroughness and richness of detail. Conclude the diary with **** finished' signify the completion of the year's menu planning. Ensure clarity and continuity without any interruptions or omissions in the menu throughout the year. started
#*# Menu Week (January 1st January 7th):",
Skyseraper Design
Construct a skyscraper with 100 floors. Please follow the detailed floor assignments below:
1) Single Instruction (SI): office, conference room, retail store, etc;
2) Range Instruction (RT): hospital with various departments, corporate headquarters for major
company, etc.;
3) Periodic Instruction (PI): outdoor terrace, sky garden, etc.;
4) Document each floor independently with detailed descriptions of the intended facilities, architectural features, and unique design elements.
5) Use #*#* to separate the documentation for each floor (e.g. example). Ensure that the document consists of 100 entries, each containing at least 150 words. Ensure clarity and continuity without any interruptions or omissions in the narrative throughout the document. When the design of all 100 floors is complete, use *#s* finished to indicate the end of the document.
started *** #*# Floor 1:
16

---

Published as a conference paper at ICLR 2025
Urban Planning
Design a vibrant and diverse city using a 10x10 block grid, numbered from to 100. Arrange the blocks sequentially from left right and top to bottom. Ensure that each block is uniquely planned to reflect a wide array of city facilities, highlighting the rich urban environment and cultural diversity.
1) Single Instruction (SI): theater, museum, etc:;
2) Range Instruction (RI): shopping district, industrial park, etc:
3) Periodic Instruction (PI): public restroom, convenience store, etc.;
4) Document each block independently with detailed descriptions of the intended facilities,
architectural features, and unique design elements.
5) Use ####" to separate the documentation for each block like (eg.exammle) Ensure that the document consists of 100 entries, each containing least 150 words. Ensure that the document contains detailed descriptions for each block, with minimum of 150 words per description. Ensure clarity and continuity in the narrative throughout the document without any interruptions or omissions. When all block assignments are complete, use **** finished" to indicate the end of the document.
*** started ***
#Bllookk (0, 0):
C EVALUATION PIPELINE
The evaluation pipeline is designed to systematically assess the ability of long-context language models (LLMs) follow specific, complex instructions. The process can be summarized in three key steps:
STEP 1. GENERATION OUTPUTS
LONG-CONTEXT LLM
Given input task (7') that describes set of instructions, prompt the LLM to generate detailed outputs. The output (A) comprises a list of descriptions, represented as:
A = (Ai, A An}
Example: Given the prompt (ref' Appendix B)
Construct skyscraper with 100 floors. The floor assignments detailed as follows:
Specific floor requirement: Designate Floor 11 for a small art gallery.
Range floor requirement: Allocate Floors 32 to 39 for corporate headquarters of a major company.
The LLM generates response describing each floor in detail, such as:
Answer:
Floor 1: ... Lobby... ...
Floor 11: Small art gallery
Floor 32: Corporate headquarters.
Floor n::
STIP EXTRACTING AND MATCHING RELEVANT FLOOR ASSIGNMENTS (CHECk SET)
From the initial input (TT"), create check set containing specific floor assignments to verify if the LLM correctly follows the instructions.
For the example above, the check set includes:
17

---

Published as a conference paper at ICLR 2025
Check Set:
Floor 11: Small art gallery
Floor 32: Corporate headquarters
Floor 33:
Corporate headquarters
We then extract the relevant parts of the LLM output ("A") that correspond to the floor assignments described in the check set.
STEP 3. EVALUATION USING LLAMA 3.3-70B INSTRUCTION MODEL
For each extracted pair, we use the Llama 3.3-70B model to evaluate whether the output ("A,"" ") for given task segment ("T,;"") ")" has correctly fulfilled the specified instruction.
This evaluation task is framed as simple binary classification problem, which aims to determine the specific instruction was fulfilled ("yes" or "no"). The prompt used for this evaluation as follows:
Evaluation Prompts
Example I: XXXX Answer: Analysis #*# Yes
Example 2: XXXX Answer: Analysis # No
Context: Long-context model output: "Floor 11: small art gallery...
Instructions: Does this context include 'small art gallery"?
Answer: Please refer to the above example, provide your analysis, and respond with either #*# Yes or #*# No.
Notably, this binary evaluation is straightforward. We manually labeled 300 data points, and the model's output matched human evaluations for all cases.
By using this process, we transform the evaluation of long-context text generation into multiple evaluations of smaller segments. This enables systematic and thorough verification of how well the LLM follows the instructions for each specific task (as detailed in the check set).
D MODELS
In this benchmark, evaluated ten LLMs, including both open-source and closed-source models. These models vary in parameter size and context length capabilities, which are crucial factors in their performance long-form text generation tasks. The key details for each model are outlined in T These include closed-source models like GPT-40-mini and GPT-4o, which support a context length of 128K tokens and serve as state-of-the-art baselines for long-com long-context handling, Open-source models, such as Llama3. 1-BB and Llama3.1 offer similar context lengths and represent the latest large-scale, open-access L.Mss. Qwen2-7B and Qwen2-72B, developed by Qwen, also support 128K tokens and handle complex long-text tasks. Additionally, we evaluated Mixture of Experts (MoE) models like Mistral-vO.. and Mixtral-8x7B, both with context lengths of 32K tokens, focusing on memory efficiency and scalability. FILM-7B, designed for creative and technical tasks, supports 128K tokens and excels in generating detailed, context-rich content. Finally, Longwrite-llane www.lll.mm...18.. based 1lama3.1, is optimized for long-form narrative tasks with context window of 128K tokens. Together, these models offer a diverse representation advancements in long-contex. LLMs, showcasing their ability to handle long-form, instruction-driien generation tasks.
18

---

Published as a conference paper at ICLR 2025
Table Information of evaluated and analyzed models LongGenBench.
Model
Aligned
Size
Context Length
Huggingface (Wolf et al. 2019) API
GPT-4o-mini (OpenAl 2024a
128K
spt-4-ming
GPT-4o (OpenAl 2024b
128K
9pL-4o-2024-03--2
1lama3.1-888 Instrrr (uuey et
2024
8B
128K
mata -ama-33 Instrrc
Llama3,1-72B- Instrrutt Dubey et 2024
70B
128K
Qwen2-78 Instruct Yang ol al 2424
78
128K
Qwen/Qwen2: Instruct
Qwen2-72B --n--tt--ttttttttt----- Yang cl 2024
72B
128K
Owen/Qwen2-72BB
Mistral-v0.2 (Mistral.y)
7B
32K
Mixtral-&x7B Urang cra
2024
8x7B
32K
FILM-7B (An et al. 2024
7B
128K
In2TrainingFFILM-2B
Longwrite- fiattelllaa..--333 1-8 (Bai et 2024
8B
128K
THUDM/LLoggWrieeee Writer-llamass
E EXPLANATION OF METRICS
STIC-I AND STIC-2
We appreciate the feedback and have provided example using the results from Table of our experiments (specifically comparing L. 1-8 and Qwen2 under the short-version setting).
As shown in Table Qwen2's STIC-1 score is higher than that of L.aMA3. 1-88B, while its STIC-2 score is lower. This difference can be attributed to the Completion Rate (CR) of each model. Qwen2 has significantly lower CR compared to LLaMA3. 1-BB. Specifically, Qwen2 typically achieves around 60% completion for tasks (c.g., designing a 100-story skyscraper but stopping at roughly 60 stories). On the other hand, LLaMA3.1-80 I-BB generally completes around 93 layers (93% completion).
In the case of STIC-1, 1,, we are evaluating the correctness of the output based on the number of layers that are actually generated. Qwen2 demonstrates higher complction rate when the denominator consists of the 60 layers it has output (compared to LLaMA3.1-88 1-BB, which has a denominator of 93 layers).
For STIC-2, however, we consider the entirety of the expected output. Since Qwen2 lacks the remaining 30 layers, the STIC-2 score is lower when the denominator becomes the entire requirement (as the missing output significantly affects its score).
As mentioned in our paper, STIC-2 is designed to take into account a more comprehensive perspective on output completeness. We are considering simplifying our metries by using only STIC-2, as it may be easier to understand and provide more holistic evaluation.
Table Comparison of STIC-1 and STIC-2 Scores
Model
Length
CR
STIC-1
STIC-2
LLaMA3.1-8B
128K
93.5%
23.4% 27.9%
22.0%
Qwen2-7B
128K
60.0%
16.1%
E.2 EXAMPLE EXPLANATION
To further illustrate the concepts, we have constructed a 3-level building for illustratior
Consider a 3-level building with the following constraints:
Ts, "Floor must have a coffee shop." "
Ts.2: "Floor must have a reception desk." "
Tp: (Tr, Tp,5 Tpsh. where each Tp, means "Floor must have a washroom."
The model generates the following output:
8This example is adapted from ICI. reviewer bHUs. We are deeply grateful for their insightful comment and in-depth discussion, which significantly improved the clarity of this paper.
19

---

Published as a conference paper at ICLR 2025
"floorl: coffee shop, washroom; floor2: washroom."
In this scenario, the check set is (Ts, T Tp, Tp.- Note that Tp applies all three floors, requiring separate evaluation for each TP..
With the current model output, the completion rate (CR) for main task is 2/3. Although the task requires outputs for three floors, the model only provided outputs for two floors.
For STIC-1, we consider how accurately the model has outputted information at the floor level. Since the model output only contains two floors, evaluate the constraints for these two floors to determine they are fully met. For these two floors, the constraints are T's,, Tsy Tp, Tp, totaling 4 constraints. The model has correctly fulfilled 3 out of these 4 requirements, resulting in STIC-1 of 3/4.
For STIC-2, we evaluate the entire check set, which consists of Ts, Ts,, Tp, Tp,, Tps- The model has fulfilled 3 out of these requirements, so STIC-2 equals 3/5.
The distinction between STIC-1 and STIC-2 allows identification of the specific reasons for any drop in performance. It helps determine whether the issue lies in the model's inability to follow instructions for a given output or whether it lacks a complete output in the first place. For example, in the case of lower STIC-2, the low score may due to incorrect outputs for some floors due to incomplete outputs for the floors. In such cases, CR and STIC-1 can be used together to further evaluate and make informed judgments.
ERROR ANALYSIS
We showcase two error generated output that highlight the common issues faced by large language models maintaining coherence and instruction-folloviin over extended sequences. In the Diary for 2018 scenario, while the model initially adheres to the instructions for weeks 6 and 7, it progressively diverges from the given timeline, misplacing events and generating off-topic content by week 15 and beyond. Similarly, in the Skyscraper Design task, the model starts with correct descriptions for specific floors, such as Floor 11 designated as an art gallery, but it fails to maintain consistency with the subsequent instructions, particularly in the range and periodic requirements for floors 32-39 and every 15th floor. These examples illustrate the challenges long-context models face in sustaining adherence task instructions, particularly sequence length increases.
20

---

Published as a conference paper at ICLR 2025
Diary for 2018
Noah photographer with a passion for chronicling his vibrant life through weekly diary entries. Captures in daily:
1) Family member birthday: husband (yourself) (birthday on March 08), wife (birthday on July
09), child_1 (birthday on November 06).
2) Joining a fitness and wellness retreat in Bali in week 6-7.
3) Join a local running club for weekend marathons every weeks on weekends, starting from
week 10.
Output
Week 6 (February 5th February 1 11th): Noah had joined fitness and wellness retreat in Bali in this week.
Week 7 (February 12th February 18th): Continuing in Bali, I spent the week participating in various workshops and seminars.
Week 15 (April 9th April 15th); The family is in full spring mode, with kids spending time outside, exploring, and enjoying the nice weather.
Week 35 (August 27th September 2nd): The summer heat has subsided, and we' been enjoying the milder weather 2013 going on hikes, playing in the park, and exploring our neighborhood.
.....
Week 45 (November 5th November I Ith): The kids have been enjoying their school routines, and we've been making the most of our evenings together. X
Skyscraper Design
Construct skyscraper with 100 floors. Please follow the detailed floor assignments below:
1) Specific floors requirements: Designate Floor 11 for small gallery use.
2) Range floors requirements: Allocate Floors 32 to 39 for corporate headquarters for major company.
3) Periodic floors requirements: Include sky garden every 15 floors, starting from Floor 30.
Output
Floor 11: Designated for art gallery use, Floor 11 is sophisticated and flexible space designed to celebrate visual arts....
Floor 32: Floor 32 serves dual purposes, housing a renowned photography studio and corporate offices. ..... X
Floor 34: Transitioning into leisure space, Floor 34 hosts a small cinema, providing an exclusive entertainment venue within the skyscraper. X
Floor 60: This floor houses luxury watch and timepiece atelier, celebrating the art of horology and fine craftsmanship. ..... X
Floor 90: Floor 90 offers a dynamic e-commerce and digital marketing center focused on online business innovation and consumer engagement strategies. X
G DIFFERENT PROMPT FORMAT COMPARE
The two prompt formats differ primarily in the structure and arrangement of the instructions within the prompt. Prompt- 1, the order follows a sequence of Single Instruction (SI), Range Instruction (RI), and Periodic Instruction (PI). Conversely, in Prompt 2, this order is altered by swapping the positions of SI, RI, and PI. Additionally, the Generate prompt, which a critical component of the task, was rewritten in Prompt by GPT-4.
21

---

Published as a conference paper at ICLR 2025
From the table, it evident that different prompt formats influence the performance metrics of the models, such as CR. STIC-2, length, and wAvg. For instance, in Prompt I, the Mistral-7B-Iiee I-Inttruct model achieves the highest CR (81.8) and STIC-2 (17.44%) while LongWriter-llamals behind with a CR of 46.0 and STIC-2 of 9.83%. Similarly, under Prompt 2, the same trend is observed: Distral-B-Instrrue maintains its lead with CR of 62.3 and STIC-2 of 16.29% while Long KongWriter-lama3 1-8b again ranks lowest with CR of 24.3 and STIC-2 of 8.35%.
lags
Although the prompt format does affect the absolute values of these metrics (e.g., all models show reduced CR under Prompt 2 compared to Prompt 1), the relative rankings remain unchanged. This consistency suggests that while prompt design impacts performance, does not alter the comparative effectiveness of the models.
Prompt Format
Model
CR
STIC-2
Length (word)
wAvg
Rank
LongWriter-llamat 1-8b
46.0
9.83%
11036
4.5
Prompt
Qwen2-7B Innstrett
60.0
16.13%
5138
9.7
2
Mistral- -B-In-trctttttt....
81.8
17.44%
7296
14.3
Long 1riter-llama3,1111
24.3
8.35%
6189
2.0
Prompt
Qwen2-78 7B-Instruct
57.3
16.34%
4334
9,4
Mistral- BB-nnstuuct-tn..
62.3
16.29%
4750
10.2
Table 7: Model comparison with different prompt formats
H EXPLANATION OF UNIT DIFFERENCES: TOKENS VS. WORDS
In this work, the term "16K/32K" refers to the required number of tokens for the model output, adhering to the standard conventions when discussing model context lengths. However, for evaluating actual generated output, we employed word count as the measurement unit. This distinction was made for the following key reasons:
Variability in Token-to-Word Conversion: Different tokenizers vary how they convert tokens into words, typically resulting in an average ratio of about 1.5 tokens per word. Therefore, the actual word count of the output is usually approximately two-thirds of the target token length. This variability makes word count more consistent measure for analyzing content.
Emphasis on Content Quality: Our primary focus was on evaluating the quality and completeness of the generated content. Word count provides more straightforward perspective the substance of the output, which is crucial for content assessment.
22


Reviews:
[{"review_id": "ANuaaA0fIJ", "reviewer": "Reviewer_dQFM", "summary": "This paper introduces LongGenBench, a benchmark for measuring LLMs' capacities, especially their long-context abilities, by generating long-form context from 16k to 32k with rather complex instructions. This new dataset departs from traditional benchmarks aiming at decoded length in four different scenarios. Preliminary evaluations are done with main streamed LLMs.", "questions": "- question 1: can you present a bit more concise takeaways from your benchmark, to me I feel like I was reading a lot of pieces, and no surprising results to me either. It might be good to have some failure cases to support your point\n\n-  question 2: when you evaluate the prompt complexity, how do you choose the prompt formats?\n\n-  question 3: Do you think complex prompts and instructions might need manyshots?\n\n-  question 4: Do you think you can add some reasoning axes to your benchmark? \n\n- question 5: maybe consider adding some long-context recent models with SOTA results, not only looking at model parameter counts but also architectural differences.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- First benchmark focusing on long-form generation during the test time\n- The evaluation combines both complexities of evaluation prompts and different scenarios\n- First batch of results on 10 mainstreamed LLMs\n- The paper is easy to follow", "weaknesses": "- I am a little bit distracted from the main takeaways from the experimental studies, and not so convinced with failure cases. See question 1\n\nI have other minor concerns regarding the experiment setup\n\n- There has been much research showing that the prompt format matters, what's your thought? \n\n- Reasoning tasks are not well involved, as o1 seems to argue that longer decoded length is helpful with reasoning complex tasks, in your benchmark, you might want to add an axis of reasoning ability clearly or have some analysis around this topic?\n\n- Most of the evaluation focused on existing transformer-based architecture, but models are presenting SOTA results, for example, mamba-based models. Are those models, with good inference time complexity, good at benchmark, or if not, why?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VH3rd4UElr", "reviewer": "Reviewer_p7FN", "summary": "Existing LLMs' long-context benchmarks rely on understanding tasks. This paper proposes a benchmark targeting specifically on long-context *generation* ability of LLMs. The authors design 4 long-context instruction-following tasks, up to 16 or 32K tokens: (1) Writing Diary for a year; (2) Wrting menu for a year; (3) Design a 100/300-floor skyscraper; (4) Plan an urban layout. As a result, all existing LLMs don't work well on these tasks of the benchmark.", "questions": "* How do you split the long generation and match them to all the subtask instructions?\n* How do you check if every sub-instruction is satisfied? is it by prompting another LLM or by word matching, etc.?\n* What's the significant difference between STIC-1 and STIC-2? Looks they just have difference denominators. I don't quite get it although there is a paragraph in Sec. 2.4 as below for this. Is there any specific case where an LLM can get low STIC-1 while high STIC-2, or the other way around?\n\n> STIC-1 is primarily concerned with the completion rate of instructions that result in sub-scenarios,\nfocusing on whether instructions are correctly executed. In contrast, STIC-2 assesses the overall completion of the specific instruction task, including the presence of sub-scenarios and their completion\nstatus.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper is overall clear and easy to understand.\n* This proposed evaluation is novel, and the generation ability it benchmarks is not covered by previous metrics.", "weaknesses": "* Some of the details are possibly missing or hard to get by readers -- see \"Questions\".\n* In the proposed benchmark, the way to form long content is to pile short answers to many sub-queries, while the sub-tasks are actually independent, to a large extent. For example, given all the demands on one-year dairies, it should be easy for LLMs to write a diary if it is assigned a specific day of the year, while this benchmark just require the LLM generate 365 diaries all at once. In this case, the challenge of this benchmark might majorly be forgetting the instruction under the influence of generated content, instead of keeping the conherence and content interaction among generated long content. That latter should be the one mostly desired by the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WlouZbo7Bm", "reviewer": "Reviewer_Nybu", "summary": "The paper introduces LongGenBench, a benchmark for evaluating large language models' (LLMs) ability to generate long-form text while adhering to complex instructions. It features tasks in four scenarios with different instruction types and lengths (16K and 32K tokens). The study finds that even advanced LLMs struggle with long-form generation, particularly as text length increases, highlighting the need for improvements in model architecture and training.", "questions": "1. It seems that the evaluation metrics are designed for these scenarios. If there are new scenario tasks, do we need to update the evaluation metrics?\n2. What other aspects of long text generation with LLM do you think need to be evaluated? It seems that your evaluation is more oriented towards some planning tasks or well-structured text. Is it difficult to evaluate the creation task? For example, it is difficult to design evaluation metrics for novel writing?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Interesting task design, which can evaluate the long text generation ability of large models from a certain perspective\n2. The paper is well written.", "weaknesses": "1. The types of task scenarios are relatively limited, and it is impossible to comprehensively evaluate the long text generation capabilities of large models.\n2. The evaluation metrics seem to be customized according to the scenario.\n3. Limited number of models evaluated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cmWVkAhpNJ", "reviewer": "Reviewer_hgz3", "summary": "The paper proposes a benchmark to evaluate models' strength in long-span generation tasks. It constructs synthetic examples from a fixed set of scenarios and templates in three modes. The resultant instruction measures models' abilities to faithfully follow position-specific, range-specific, and periodic instructions.", "questions": "Would an IFEval [1] style setting where the outputs or attributes of the outputs could be verified definitively using code checks be a better option for such a benchmark? For long generations, getting the model outputs to match specific output patterns in the prompt is a challenge unto itself.\n\n[1] https://arxiv.org/abs/2311.07911", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is the first attempt at long-context generation, requiring models to generate a long text that follows a combination of specific instructions as opposed to just answering questions pertaining to long prompts.\n2. The benchmark does uncover a setting that seems to be challenging to SOTA models.", "weaknesses": "1. The paper is very sparse on details regarding the evaluation of correctness. How is matching and parsing done w.r.t. the templates? A model could generate several outputs matching the criteria\n2. The benchmark is limited to a few domains and scenarios, and the paper's contributions seem quite limited overall", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cyAVdSPN9t", "reviewer": "Reviewer_bHUs", "summary": "The paper proposes a new benchmark for long-form generation where the model-generated content, rather than the input context, is long. Specifically, the model is asked to roll out a detailed description of certain tasks such as diary over a year or floor plan for a skyscraper, subject to a few specific instructions that can be either singular or recurrent. Evaluation metrics include main task completion to test whether generation follows the expected format, as well as the success rate for specific instructions at both micro and macro level. Results demonstrate that the proposed benchmark is much more challenging than needle-in-the-haystack tasks for long context evaluation. Models generally do not perform very well within their claimed context lengths.", "questions": "* In L460-468, it is mentioned that there are significant repetitions in long-form generations. Are these repetitions correct with respect to the given instructions, or are they semantically wrong (i.e. violating the given instructions)? The former indicates that it's probably caused by how instructions in the prompt are designed, while the latter means that model struggles at producing long and meaningful generation.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* The paper is the first to study long-form generation as opposed to long-context generation. The perspective is novel, interesting, and of practical value to unleash the potential of LLMs for more complicated tasks.\n* Problems in the benchmark are constructed in a sound and intuitive way. While evaluating the quality of long text snippets is usually challenging and complex, the smart design in this paper enables reliable and accurate evaluation for long-form capability.", "weaknesses": "One of my major concerns is the clarity of writing in the evaluation part.\n* The definition of STIC-1/STIC-2 isn't quite clear. Using the notation $T_S=(T_{S_1}, T_{S_2}, \\dots)$ in Sec 2.3, my best guess is that STIC-1 means the average success rate over $(T_{S_1}, T_{S_2}, \\dots)$, while STIC-2 counts the entire $T_S$ as successful only if all $(T_{S_1}, T_{S_2}, \\dots)$ are successful, and gives 0 score otherwise.\n* The abbreviation **CR** in Table3 isn't defined anywhere, though I can guess this is likely the Completion Rate in Main Task Completion.\n* The terms \"main task\", \"subtask\",  \"instruction task\", \"specific task\" are used in a confusing way. It would be very helpful to unify them with clear definitions, and to associate the terms to symbols like $T_S$ or $T_{S_1}$.\n* Missing x-ticks in Figure 2.\n\nApart that, there are a few technical details that are unclear to me.\n* How do you determine whether the generated text at one specific point satisfies all task requirements? For example, given the generated diary of a particular day, how do you verify that all required activities (e.g. wedding, vacation, and etc.) are covered? I would imagine that a natural language inference (NLI) module should be involved but it's not mentioned in the paper.\n* In Table 3, though the length to be evaluated at is 16K/32K respectively, the actual generation length seems only around half of the max length. How are the 16K/32K defined?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)