LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

## Abstract

Abstract Current benchmarks like “ Needle-in-a-Haystack ” ( NIAH ), Ruler , and Needlebench focus on models’ ability to understand long-context input sequences but fail to capture a critical dimension: the generation of high-quality long-form text. Applications such as design proposals, technical documentation, and creative writing rely on coherent, instruction-following outputs over extended sequences—a challenge that existing benchmarks do not adequately address. To fill this gap, we introduce LongGenBench , a novel benchmark designed to rigorously evaluate large language models’ (LLMs) ability to generate long text while adhering to complex instructions. Through tasks requiring specific events or constraints within generated text, LongGenBench evaluates model performance across four distinct scenarios, three instruction types, and two generation-lengths (16K and 32K tokens). Our evaluation of ten state-of-the-art LLMs reveals that, despite strong results on Ruler , all models struggled with long text generation on LongGenBench , particularly as text length increased. This suggests that current LLMs are not yet equipped to meet the demands of real-world, long-form text generation. We open-source LongGenBench to promote comprehensive evaluation and improvement in this critical area, with code and data available at https://github.com/mozhu621/LongGenBench .

## 1 Introduction

Recent advances in large language models (LLMs) have dramatically enhanced their ability to process long text sequences, supporting applications that range from document summarization to creative writing. Leading models such as GPT-4 (Achiam et al., ), LLaMa-3.2 (Dubey et al., ), and Claude 2.1 (Anthropic, ) manage context windows of up to 128K tokens, with the Claude 3 series (Anthropic, ) handling inputs exceeding 1 million tokens. However, while much attention has been given to these models’ ability to retrieve and understand long input text sequences, far less focus has been placed on their ability to generate coherent and high-quality long-form text outputs—a critical requirement for tasks such as design proposals and creative writing.

Long-form text generation is crucial for real-world applications that require detailed, well-structured narratives, such as document summarization (Kumar et al., ), creative writing (Hua & Wang, ; Hu et al., ), and comprehensive question answering (Stelmakh et al., ; Lee et al., ; Bai et al., ). Despite this importance, current benchmarks are limited in their ability to assess long-form generation, focusing instead on shorter text outputs ($\leq$ 2K tokens) (Fan et al., ; ; Dasigi et al., ), making them unsuitable for tasks requiring outputs of $\geq$16K tokens (Bai et al., ). The challenge is further compounded by the lack of robust methods for evaluating these long sequences. The ability to follow instructions is essential for long text generation (Reversed NIAH(^1^11Analogous to NIAH, which involves searching for a needle (retrieval) within a long input, the reversed NIAH entails placing a specific needle (instruction-following) at a designated position within a long output.)), just as effective information retrieval is fundamental for processing long-context inputs (NIAH(Kamradt, )). However, current benchmarks do not adequately assess whether the generated text adheres to the specific directives of a prompt. For instance, a prompt may require incorporating specific information at a certain point in a lengthy document, but evaluations often fail to verify the model’s compliance with such instructions. This oversight represents a significant shortcoming in benchmarking, particularly because performance under explicit constraints typically predicts outcomes in tasks with more implicit constraints, such as story generation or academic paper production. If a model struggles with explicit requirements, it is likely to underperform in scenarios with subtler constraints.

Manual evaluations, while thorough, are both costly and impractical at scale. Meanwhile, automated evaluations using ”LLM-as-a-judge” methods (Zheng et al., ) often yield results that are difficult to interpret and may not align with human judgments, raising concerns about their reliability. This highlights the need for more specialized benchmarks capable of reliably assessing the quality of super-long-form text generation.

To address this gap, we present LongGenBench, a novel benchmark designed to evaluate the quality of super-long-form text generated by long-context LLMs. Unlike existing benchmarks that primarily test retrieval or reasoning over long inputs, LongGenBench  focuses on the model’s ability to generate content that follows complex instructions over extended sequences. Our benchmark introduces tasks that reflect real-world generation challenges, such as diary writing, menu planning, and urban design, where the text must adhere to specific constraints provided in the prompt. These tasks assess whether models can correctly incorporate specific details at designated points in the text, ensuring the generated content meets the requirements laid out in the prompt. By evaluating texts up to 32K tokens, LongGenBench  is the first benchmark to systematically test the ability to generate instruction-compliant long-form content across extended lengths. Table 1 summarizes the different benchmarks supporting long-context retrieval and generation tasks.

The evaluation tasks are organized into four distinct scenarios: Diary Writing, Menu Design, Skyscraper Design, and Urban Planning, each with varying complexity. The scenarios involve sub-tasks such as single instance, range, and periodicity, simulating realistic constraints that a model must account for. This setup allows us to measure the model’s ability to generate detailed, contextually rich outputs that satisfy a wide array of criteria.

In summary, our major contributions are as follows:

- •
To the best of our knowledge, this is the first study to address the challenge of super-long-form generation in long-context language models, highlighting the critical importance of generating coherent, high-quality text in extended contexts.
- •
We introduce LongGenBench , a comprehensive dataset that provides a diverse set of tasks specifically designed to evaluate the super-long-form generation capabilities of LLMs across varying token lengths (16K and 32K) and levels of text complexity.
- •
We perform extensive experiments on both open-source and closed-source models, revealing that despite their advanced capabilities, most models struggle significantly with super-long-form generation tasks, particularly in maintaining instruction adherence and coherence over long outputs.

**Table 1: Comparison of Long-Context LLM Benchmarks. For the retrieval tasks’ datasets, we measure length based on the number of processing tokens, while for the generation tasks’ datasets, we calculate the average number of generation words produced by LLMs. ‘Long-length’ indicates if LLMs to analyze or generate text that is at least 8K token.**
| Type of Task | Benchmark | Type of Data | Avg Len | Long-Length |
| --- | --- | --- | --- | --- |
| Retrieval | Longbench(Bai et al., ) | hybrid | $\sim$8k | ✓ |
| NIAH(Kamradt, ) | synthetic | Any | ✓ |  |
| Ruler(Hsieh et al., ) | synthetic | Any | ✓ |  |
| Generation | ELi5(Fan et al., ) | hybrid | $\sim$0.2K | ✗ |
| Longwrite(Bai et al., ) | synthetic | $\sim$2.7K | ✗ |  |
| LongGenBench(Ours) | synthetic | $\sim$20K | ✓ |  |

## 2 LongGenBench Benchmark

Figure: Figure 1: LongGenBench Overview: 1) Scenario Selection: Select from four scenarios—Diary, Menu Design, Skyscraper Design, and Urban Planning—each offered in both short and long versions to determine the main task prompt. 2) Task Instruction: Employ the template libraries SI (Single), RI (Range), and PI (Periodic) to generate tasks characterized by random times or locations, along with the corresponding prompts and verification sets. 3) Instruction Synthesis: Integrate all prompts generated in the prior step to create a comprehensive set of instructions with a final check-set. 4) Example: An illustration of Sophia’s weekly diary task is provided as an example.
Refer to caption: x1.png

### 2.1 Task Definition

Evaluating the quality of super-long-form generation  presents a unique set of challenges due to the inherent complexity of long texts. Traditional human evaluation methods, while precise, are expensive and not scalable. Although using large language models for assessment is feasible, their lack of interpretability often hampers their utility. Thus, we focus on the ”instruction-following” task in super-long-form generation, where the most must include specific details in the generated text. This task reflects real-world scenarios that require a high degree of attention to detail over extended sequences, such as technical documentation or detailed design proposals. In this study, we define a task type termed Strictly Sequential Tasks, which involves the sequential completion of subtasks $\mathbf{T}=\{T_{1},T_{2},T_{3},\ldots,T_{n}\}$, where each subtask is responsible for generating a specific volume of text. For instance, an LLM might be tasked with designing a 100-floor skyscraper, specifying the content and purpose of each floor.

### 2.2 Four Distinct Scenario Setups

To comprehensively assess the long-form generation capabilities of models, we have devised four distinct task scenarios to supplement our predefined tasks, as illustrated in Figure 1 (1). These scenarios fall into two categories: Temporal (Diary Writing, Menu Design) and Spatial (Skyscraper Design, Urban Planning). Moreover, each scenario incorporates both short and long versions to assess the effectiveness of various output lengths.

These scenarios were carefully chosen to reflect both creative and technical long-form generation tasks. They offer a diverse set of challenges by including temporal tasks (e.g., Diary Writing) that require maintaining consistent information over time and spatial tasks (e.g., Urban Planning) that test the model’s ability to handle spatial relationships and detailed designs. These scenarios mirror real-world applications, from planning documents to creative writing, and thus provide a comprehensive evaluation of long-context LLMs. Table 2 offers comprehensive descriptions for each scenario, with each designed around a unique template to generate 100 different task instructions(^2^22Examples of Task Instructions for each scenario are provided in Appendix A.).

**Table 2: Scenario task descriptions**
| Category | Scenarios | Task | Task Description |
| --- | --- | --- | --- |
| Temporal | Diary | Weekly Diary | Generate entries for each week of the year |
| Daily Diary | Generate entries for each day of the year |  |  |
| Menu | Weekly Menu | Plan menus for each week of the year |  |
| Daily Menu | Plan menus for each day of the year |  |  |
| Spatial | Skyscraper Design | 100-floor Design | Develop a design for a 100-floor skyscraper |
| 361-floor Design | Develop a design for a 300-floor skyscraper |  |  |
| Urban Planning | 10x10 block Design | Design an urban layout on a 10x10 block grid |  |
| 19x19 block Design | Design an urban layout on a 19x19 block grid |  |  |

### 2.3 Specific Task Instruction

To enhance task control and flexibility, we have developed three distinct task settings:

- •
Single Instruction (SI): Injects specific information at a unique point within the generated text.
$T_{S}=\{T_{s_{1}},T_{s_{2}},\ldots\}$
- •
Range Instruction (RI): Requires the model to incorporate information within specified ranges of the text.
$T_{R}=\{T_{i},T_{i+1},\ldots,T_{i+j}\}$
- •
Periodic Instruction (PI): Distributes information at predefined intervals throughout the text.
$T_{P}=\{T_{n},T_{2n},\ldots,T_{m*n}\}$
- •
Check Set: Includes tasks for all three aforementioned settings.
$Check_set=\{T_{S},T_{R},T_{P}\}$

For example, in the design of a 100-floor skyscraper, the Single Instruction may specify that the 34th floor hosts an aerial gym and the 54th floor houses a law firm. The Range Instruction might designate floors 1 through 9 as a comprehensive shopping mall, whereas the Periodic Instruction could dictate that starting from the 20th floor, every 10th floor incorporates a small aerial garden.

We utilize over 20 templates for each type of instruction, with the floors or locations being randomly assigned to ensure task diversity. These settings, applied via various templates, guarantee controlled coverage across all textual positions, thus facilitating a comprehensive and efficient evaluation, as illustrated in Figure 1 (2).

Through this approach, we generate the main task instructions $T$ and simultaneously acquire the corresponding $Check_set$, which supports subsequent evaluations and constructs a task conducive to super-long-form generation. Subsequently, we splice the main task prompt with the specific task instructions (STI)(^3^33Each of our main task instructions $T$ splice 5 single instructions, 1 range instruction task, and 1 periodic instruction task.) and add the generation prompt to form the final evaluation data.

### 2.4 evaluation Metric

To quantitatively evaluate performance for LongGenBench tasks, we introduce three complementary metrics:

#### Main Task Completion.

This metric evaluates the extent to which all designated subtasks are accomplished. The completion rate is quantified using the following equation:

$$ $\text{Completion Rate}=\frac{\text{Number of Completed Subtasks}}{\text{Total Number of Subtasks}}\times 100\%$ $$

In this context, the numerator denotes the count of subtasks successfully executed by the model, and the denominator represents the total number of subtasks defined in the Strictly Sequential Task. For instance, does the model consistently complete a diary entry for each day without omitting any dates?

#### Specific Task Instruction Completion (STIC-1).

This metric evaluates the model’s adherence to specific task instructions. We calculate the completion counts for the Single Instruction (SI), Range Instruction (RI), and Periodic Instruction (PI). STIC-1 quantifies how well the model follows these instructions across subtasks, focusing on whether the instructions are correctly implemented. For example, in the Skyscraper Design task, if the model is instructed to place an aerial gym on the 34th floor and consistently places it on a different floor, it would receive a lower STIC-1 score.

$$ ${\text{STIC-1}=\frac{\text{Single Instruction}+\text{Range Instruction}+\text{ Periodic Instruction}}{\text{Total Number of {Outputs} to Specific Task Instructions}}}$ $$

#### Specific Task Instruction Completion-2 (STIC-2).

STIC-2 provides a more granular assessment by measuring the overall completion of specific task instructions, including their presence and execution quality across all subtasks. In addition to adherence, it assesses whether the model consistently follows these instructions throughout the entire task. For instance, if the model periodically repeats certain elements but not at the required intervals, it would affect its STIC-2 score.

$$ ${\text{STIC-2}=\frac{\text{Single Instruction}+\text{Range Instruction}+\text{ Periodic Instruction}}{\text{Total Number of Specific Task Instructions}}}$ $$

STIC-1 is primarily concerned with the completion rate of instructions that result in sub-scenarios, focusing on whether instructions are correctly executed. In contrast, STIC-2 assesses the overall completion of the specific instruction task, including the presence of sub-scenarios and their completion status.

### 2.5 evaluations Pipeline

Our evaluation process follows a structured pipeline: First, we use a long-context LLM to complete the task instruction $T$, generating an answer $A$, which is then divided into sub-tasks as $A=\{A_{1},A_{2},\ldots,A_{n}\}$. Next, based on the specific instructions in the check_set, we identify the relevant sub-tasks within $A$. Finally, we evaluate each sub-task by $\text{eval}(A_{i},T_{i})$ to compute the final completion score, as detailed in Algorithm 1. This pipeline ensures that the evaluation is both systematic and comprehensive, assessing the model’s performance across different instruction settings and levels of complexity. While LongGenBench primarily evaluates the model’s ability to follow detailed instructions, future work could expand the benchmark to include more open-ended tasks that assess creativity and logical reasoning. This would provide a broader evaluation of a model’s capabilities in generating coherent, engaging, and logically sound long-form text.

Figure: Algorithm 1 Evaluations Pipeline

## 3 Experiments

### 3.1 Experimental Setup

#### Models.

We selected ten long-context large language models (LLMs), comprising eight open-source and two closed-source models. These models range in size from 7B to 72B parameters, with one featuring a Mixture of Experts (MoE) architecture. The claimed context lengths of these models vary from 32K to 128K tokens(^4^44Detailed specifications of these models are provided in Appendix C.). These models were selected to represent a diverse array of architectures, covering both Mixture of Experts and standard transformer designs, as well as a range of parameter sizes. This diversity ensures a comprehensive evaluation of their ability to handle long-context tasks.

#### Inference Setup.

We utilized the vLLM (Kwon et al., ) system, which optimizes key-value (KV) cache memory for efficient large-scale inference. This system is crucial for handling long-form generation efficiently, reducing memory overhead, and maximizing inference throughput. Inferences were performed using BFloat16 precision on $8\times$ NVIDIA A800 GPUs, employing greedy decoding to generate the outputs. This setup ensured consistency and efficiency in the inference process.

#### Task Configurations.

For each scenario, we generated 800 examples at two specified lengths: 16K tokens and 32K tokens. The generation was based on designated templates for each model, ensuring task-specific relevance. The tasks were selected to reflect both creative and technical long-form generation challenges, such as diary writing, urban planning, and skyscraper design. To ensure the relevance of the generated content and prevent off-topic responses or refusals to answer, we prefixed each task input with a carefully curated answer prompt designed to guide the model’s output. The tasks were specifically selected to test the models’ ability to generate instruction-following long-form content in both creative and technical contexts. For example: In the Urban Planning task, models were tasked with generating a detailed plan for a new urban district, including descriptions of key facilities such as parks, schools, and transportation systems.

#### Evaluation Metric.

We evaluated model performance using the three metrics defined in Section 2.4: Main Task Completion, Specific Task Instruction Completion-1 (STIC-1), and Specific Task Instruction Completion-2 (STIC-2). These metrics provided a comprehensive assessment of the models’ ability to adhere to instructions and generate coherent long-form text.

### 3.2 Main Result

**Table 3: Long-form generation Performance (%) of selected models evaluated at length from 16k and 32k. The weighted average score (wAvg) is the product of CR and STIC-2, used to represent the model’s final performance at the given task length. Note that the GPT-4-32K is currently closed for use, and the longest versions that can be used are the GPT-4o and GPT-4o-mini 16K output limitation.**
| Models | Claimed | Short-version (16K) | Long-version (32K) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Length | CR | STIC-1 | STIC-2 | Len. | wAvg | CR | STIC-1 | STIC-2 | Len. | wAvg |
| Models with 7-10B Parameters |  |  |  |  |  |  |  |  |  |  |  |
| Mistrial-v0.2-7B | 32K | 81.8% | 22.0% | 17.4% | 7296 | 14.3% | 48.2% | 37.0% | 16.3% | 16146 | 7.9% |
| LLama3.1-8B | 128K | 93.5% | 23.4% | 22.0% | 8804 | 20.6% | 77.6% | 26.5% | 18.9% | 17354 | 14.6% |
| Qwen2-7B | 128K | 60.0% | 27.9% | 16.1% | 5138 | 9.7% | 40.0% | 31.7% | 12.6% | 9617 | 5.0% |
| FILM-7B | 128K | 36.0% | 9.9% | 3.9% | 6280 | 1.4% | 37.4% | 30.9% | 10.9% | 13775 | 4.1% |
| LongWriter-llama3.1-8B | 128K | 46.0% | 22.6% | 9.8% | 11036 | 4.5% | 34.5% | 33.6% | 10.0% | 19925 | 3.5% |
| Models Larger Than 20B Parameters |  |  |  |  |  |  |  |  |  |  |  |
| Mixtral-8x7B | 32K | 83.0% | 35.4% | 28.0% | 8113 | 23.3% | 60.5% | 39.9% | 22.3% | 15839 | 13.5% |
| LLama3.1-70B | 128K | 79.3% | 24.9% | 21.1% | 8055 | 16.7% | 63.1% | 35.8% | 21.7% | 15197 | 13.7% |
| Qwen2-72B | 128K | 94.3% | 25.5% | 24.0% | 8013 | 22.7% | 66.2% | 27.5% | 17.4% | 19845 | 11.5% |
| Closed-source Model |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o-mini | 128K | 97.0% | 29.0% | 27.9% | 8940 | 26.9% | – | – | – | – | – |
| GPT-4o | 128K | 67.2% | 34.9% | 19.9% | 9055 | 12.5% | – | – | – | – | – |

#### Results.

The results of the long-form text generation tasks for both 16K and 32K tokens are summarized in Table 3. The models exhibited significant performance variations, highlighting differences in architecture, parameter size, and training data.

Main Task Completion. Significant disparities in performance across models primarily stem from differences in architecture and training datasets. Notably, models with varying parameter sizes, such as Llama3.1-8B-instruction (Dubey et al., ) (under 10 billion parameters), Qwen-72B (Yang et al., ) (over 20 billion parameters), and GPT-4o-mini (OpenAI, ) (a closed-source model), have demonstrated superior efficacy, successfully completing most primary tasks in full. In contrast, some models struggle with these tasks, exhibiting limitations such as: 1) models responding solely to specified subtasks, neglecting others, and 2) models halting after only completing the initial task segment, despite prompts requiring full sequential subtask completion. This issue may originate from the current instructional tuning data, which could cause partial responses in complex, lengthy tasks. Especially in GPT-4o(OpenAI, ), it recognizes that this task will generate a long output and only provides a few examples.

STIC-1 and STIC-2. The STIC-1 metric revealed strong performance in adhering to task instructions for models like Mixtral-8x7B and GPT-4o-mini, particularly in shorter sequences. However, a significant drop in STIC-2 scores for several models indicates that maintaining instruction adherence over longer text sequences remains a challenge. This performance degradation emphasizes the need for better tuning and architectural modifications to improve long-term coherence. The MoE model, Mistral-8x7B generally outperformed dense counterparts in instruction-following over extended sequences, but both model types struggled with STIC-2 in longer generations.

A common failure mode observed across multiple models was the tendency to forget or misinterpret instructions as the sequence length increased. For example, in the Skyscraper Design task, some models correctly described the initial few floors but deviated from the original plan as the task progressed, particularly in the 32K token setting. This highlights the memory retention issue in long-context models, which often leads to a loss of coherence and adherence to task instructions. Examples of these failures, where models struggled to follow prompt instructions over extended outputs, are provided in Appendix D.

#### Length (Number of words).

We calculated the average output word count for models that consistently completed all subtasks, achieving at least an 80% completion rate in sub-scenarios, excluding data from unsuccessful attempts. Most models substantially exceeded previous benchmarks for long-form generation tasks in terms of output length. Notably, the LongWriter (Bai et al., ) model excelled, efficiently meeting word count requirements for each subtask. Given the results and the weighted average score (wAvg) at a sequence length of 16K, the open-source Qwen2-72B and the closed-source GPT-4o models demonstrated optimal performance. At a sequence length of 32K, the Llama3.1-8B model, outperformed models with larger parameters, highlighting its efficiency in managing extended lengths.

### 3.3 Accuracy Trend with Varying Sequence Length

As illustrated in Figure 2, there is a clear decline in model performance as output length increases. Models exhibited strong adherence to initial instructions at shorter sequence lengths, but performance gradually degraded as the text generation extended beyond the 4,000-token threshold. This degradation aligns with trends identified in the NIAH dataset and underscores the challenge of maintaining instruction adherence and coherence over long outputs.

This deviation becomes particularly pronounced when outputs exceed 4,000 tokens, where adherence to instructions significantly diminishes, and further deterioration is observed as outputs approach 16,000 tokens. In contrast, tasks involving shorter outputs, such as those in the NIAH dataset or simpler multi-needle tasks, showed near-perfect performance, highlighting the disparity in model behavior across different sequence lengths.

Potential reasons for this decline include limitations in the self-attention mechanism used in transformers, which may struggle to maintain meaningful context over long sequences. Additionally, models trained with limited long-form data may overfit to shorter patterns, leading to a loss of coherence in extended generations. These findings suggest that architectural changes or improved training strategies may be necessary to overcome these challenges in future iterations of LLMs.

Figure: Figure 2: The right side of the figure illustrates the model’s performance on specific instruction tasks at 16K as sequence length increases, whereas the left side depicts performance at 32K. All curves have been smoothed with a Moving Average.
Refer to caption: x2.png

### 3.4 Three Specific Task Instructions

Figure 3 presents the model’s performance metrics across various task types: single, range, and periodic. The model demonstrated comparable proficiency in both single and range tasks, reflecting its capability to follow direct and straightforward instructions effectively. However, the slight reduction in performance for range tasks suggests that additional complexity, such as processing multiple data points within a defined range, introduces a marginal increase in cognitive load for the model.

The most significant decline in performance was observed in periodic tasks, where the model struggled to interpret instructions that required recurring events, such as ”every four weeks starting from week 10.” These tasks demand a higher degree of reasoning and temporal awareness, which may challenge the model’s capacity to maintain consistency over extended sequences. As a result, outcomes for periodic tasks were considerably poorer compared to single and range tasks, which have clearer and more well-defined parameters. The model’s performance hierarchy can generally be summarized as $single>range>periodic$, highlighting the increased difficulty associated with periodic tasks. This trend underscores the need for future improvements in long-context models, particularly in handling more complex, time-based instructions.

### 3.5 Comparison with Long-Context Input

We examine the relationship between a model’s ability to handle long inputs and its performance on long outputs. Specifically, we investigate whether a model’s capacity to manage long-range inputs corresponds to improved performance on long-range outputs. For this analysis, we use the RULER dataset, a synthetic benchmark designed to flexibly configure sequence length and task complexity, making it ideal for comprehensive evaluations of long-context LLMs. We compare the models’ performance on sequences of the same length, as shown in Figure 3, which indicates a significant performance gap between input handling and output performance. At 16K tokens, the Pearson correlation coefficient is 0.51, while at 32K tokens, it increases to 0.66, suggesting that there is some overlap in the skills required for managing long inputs and generating long outputs, but these tasks are not entirely equivalent.

Handling long inputs primarily requires the model to retain and process existing information, while generating long outputs demands more complex reasoning, memory retention, and coherence management over extended sequences. Thus, models that excel in long-input retrieval may still struggle with long-form generation, particularly in tasks requiring strict instruction adherence over time. This distinction highlights the need for models to be optimized for both input handling and output generation to achieve consistent performance in long-context tasks.

Figure: (a) Performance Comparison on three tasks settings
Refer to caption: extracted/5900542/Threading_the_needle/fig/Three.png

## 4 analysis and Limitations

#### Richness of Content.

Despite efforts to design sub-scenarios that enhance task diversity and richness, the model’s outputs tend to converge as output volume increases. This results in a homogenization of recorded events, even when differences in time and location should introduce variety. Such convergence not only degrades overall performance but also diminishes the diversity of the generated content, leading to repetitive and predictable outputs. In our experiments, approximately 45% of long outputs exhibited significant repetition, even when the model was given varied time or location prompts. Adjusting parameters like repetition_penalty during inference has shown limited success in mitigating this issue, highlighting the need for more advanced techniques to maintain content richness over long sequences.

#### Rationality of Content.

While our current research focuses primarily on evaluating instruction-following capabilities, a more comprehensive analysis of content rationality and coherence is needed. For example, when tasked with generating a diary, the model should ensure that all recorded activities align with the specified careers. However, in many instances, this logical consistency is lacking. Additionally, temperature records in virtual diary entries often fail to reflect realistic temporal changes. For instance, in a San Francisco’s diary task, we would expect temperatures to vary from cooler (0-10 degrees Celsius) at the beginning of the year to warmer (20-30 degrees Celsius) by mid-year. Yet, the model consistently generates warmer temperatures throughout, even into December. These issues may arise due to the model’s limited exposure to temporally varied datasets, particularly in diary or climate-related contexts. Future work could address this by incorporating more domain-specific and temporally annotated data during fine-tuning.

#### Instruction Data.

A significant performance discrepancy between models’ abilities to handle long-range inputs (such as Ruler (Hsieh et al., )) and their long-form output generation can likely be attributed to the length distribution of instruction-tuning data. Most instruction-tuning datasets are brief, typically under 200 tokens, and lack the extended instructional content necessary for generating longer outputs. This gap suggests that organizing or synthesizing instruction-tuning data with longer, more comprehensive examples could be a valuable direction for future research. Potential solutions include applying transfer learning techniques from models trained on long-form datasets or using data augmentation methods to synthesize longer instructional content from existing short-form data.

#### Generalizability.

LongGenBench effectively evaluates instruction-following in creative and technical tasks but may not fully capture the creativity and specialized knowledge required for abstract reasoning or unconstrained storytelling. Future versions could include open-ended tasks like creative fiction writing and legal document drafting, which demand intricate narratives and precision. Expanding in this direction would enhance the benchmark’s versatility while providing deeper insights into LLMs’ capabilities. However, LongGenBench’s current focus on instruction adherence offers a strong foundation for evaluating practical, instruction-driven long-form text generation.

## 5 Related Work

#### Instruction Following.

Recent advances in instruction tuning models (Ouyang et al., ; Rafailov et al., ; OpenAI, ; Taori et al., ; Chiang et al., ) have underscored the need for scalable evaluation methods. LLMs have been used as evaluators, showing better alignment with human judgments than traditional metrics like BLEU (Papineni et al., ). However, LLM evaluations suffer from biases, such as sensitivity to presentation order and preference for verbose outputs (Wang et al., ; Pezeshkpour & Hruschka, ; Zheng et al., ). To mitigate these biases, meta-evaluation benchmarks like FairEval, MT-Bench, and LLMEval^2 (Wang et al., ; Zheng et al., ; Zhang et al., ) have been proposed. While recent studies have focused on improving LLM evaluations with diverse strategies (Zheng et al., ; Li et al., ; Zhang et al., ; Chan et al., ), they typically do not address longer context lengths.

#### Long-context Benchmarks and Tasks.

Existing benchmarks focus on models handling long inputs. For instance, ZeroSCROLLS (Shaham et al., ) and LongBench (Bai et al., ) tackle tasks like long-document QA and query-based summarization. Synthetic benchmarks, like NeedleBench (Li et al., ) and Ruler (Hsieh et al., ), offer better control over variables such as sequence length and complexity. NeedleBench introduces the Ancestral Trace Challenge (ATC), while Ruler evaluates models across tasks like NIAH and multi-hop tracing. However, these benchmarks largely focus on input comprehension and do not assess long-form text generation, which is the primary focus of LongGenBench.

#### Long-form Text Generation.

Research in long-form generation spans applications like story generation (Fan et al., ; Xu et al., ), paragraph completion (Kang & Hovy, ), sustained conversation (Xu et al., ), and comprehensive QA (Fan et al., ; Dasigi et al., ; Stelmakh et al., ; Lee et al., ). However, existing models and evaluation methods (Liu et al., ; Chiang & Lee, ; Liu et al., ; Bai et al., ) face challenges in maintaining quality over long outputs, often being limited by shorter text lengths (typically under 2000 tokens) (Shen et al., ). Recent work (Tan et al., ) seeks to improve evaluation criteria, but the gap between model capabilities and benchmark text lengths remains. In contrast, LongGenBench evaluates models on their ability to handle much longer sequences, with tasks requiring adherence to instructions over extended outputs (16K+ tokens).

## 6 Conclusion

We introduced LongGenBench, a synthetic benchmark that evaluates long-form generation capabilities of language models by testing their ability to follow instructions over extended sequences. In evaluating nine advanced models with context sizes ranging from 32K to 128K tokens, we observed significant performance degradation compared to benchmarks like ”Ruler” with common failure modes including premature task termination, incomplete responses, disregard for instructions, and repetitive content generation. These results highlight key challenges for current models in handling long-form tasks and underscore the need for advancements in model architecture and training data to improve coherence, instruction adherence, and content diversity over extended outputs.

## Appendix A PROMPT TEMPLATES FOR FOUR TASK SCENARIOS

Below are the example templates for the four task scenarios: Diary Writing, Menu Design, Skyscraper Design, and Urban Planning. In the Diary Writing template, professions and names are customizable variables, allowing for flexibility in the generated content.

## Appendix B evaluation prompts

We provide the prompts used for evaluation and the function $\text{eval}(A_{i},T_{i})$ in Algorithm 1, where Corresponds to the output part of Specific Task Instruction represents $A_{i}$, and Specific Task Instruction represents $T_{i}$. We present both a ”yes” example and a ”no” example for the few-shot evaluation.

## Appendix C Models

In this benchmark, we evaluated ten LLMs, including both open-source and closed-source models. These models vary in parameter size and context length capabilities, which are crucial factors in their performance on long-form text generation tasks. The key details for each model are outlined in Table 4. These include closed-source models like GPT-4o-mini and GPT-4o, which support a context length of 128K tokens and serve as state-of-the-art baselines for long-context handling. Open-source models, such as Llama3.1-8B and Llama3.1-70B, offer similar context lengths and represent the latest in large-scale, open-access LLMs. Qwen2-7B and Qwen2-72B, developed by Qwen, also support 128K tokens and handle complex long-text tasks. Additionally, we evaluated Mixture of Experts (MoE) models like Mistral-v0.2 and Mixtral-8x7B, both with context lengths of 32K tokens, focusing on memory efficiency and scalability. FILM-7B, designed for creative and technical tasks, supports 128K tokens and excels in generating detailed, context-rich content. Finally, Longwrite-llama3.1-8B, based on Llama3.1, is optimized for long-form narrative tasks with a context window of 128K tokens. Together, these models offer a diverse representation of advancements in long-context LLMs, showcasing their ability to handle long-form, instruction-driven generation tasks.

**Table 4: Information of evaluated and analyzed models in LongGenBench.**
| Model | Aligned | Size | Context Length | Huggingface (Wolf et al., ) / API |
| --- | --- | --- | --- | --- |
| GPT-4o-mini (OpenAI, ) | ✓ | - | 128K | gpt-4-mini |
| GPT-4o (OpenAI, ) | ✓ | - | 128K | gpt-4o-2024-08-06 |
| Llama3.1-8B-Instruct (Dubey et al., ) | ✓ | 8B | 128K | meta-llama/Meta-Llama-3.1-8B-Instruct |
| Llama3.1-72B-Instruct (Dubey et al., ) | ✓ | 70B | 128K | meta-llama/Meta-Llama-3.1-70B-Instruct |
| Qwen2-7B-Instruct (Yang et al., ) | ✓ | 7B | 128K | Qwen/Qwen2-7B-Instruct |
| Qwen2-72B-Instruct (Yang et al., ) | ✓ | 72B | 128K | Qwen/Qwen2-72B-Instruct |
| Mistral-v0.2 (Mistral.AI, ) | ✓ | 7B | 32K | mistralai/Mistral-7B-Instruct-v0.2 |
| Mixtral-8x7B (Jiang et al., ) | ✓ | 8x7B | 32K | mistralai/Mixtral-8x22B-Instruct-v0.1 |
| FILM-7B (An et al., ) | ✓ | 7B | 128K | In2Training/FILM-7B |
| Longwrite-llama3.1-8B (Bai et al., ) | ✓ | 8B | 128K | THUDM/LongWriter-llama3.1-8b |

## Appendix D Error Analysis

We showcase two error generated output that highlight the common issues faced by large language models in maintaining coherence and instruction-following over extended sequences. In the Diary for 2018 scenario, while the model initially adheres to the instructions for weeks 6 and 7, it progressively diverges from the given timeline, misplacing events and generating off-topic content by week 15 and beyond. Similarly, in the Skyscraper Design task, the model starts with correct descriptions for specific floors, such as Floor 11 designated as an art gallery, but it fails to maintain consistency with the subsequent instructions, particularly in the range and periodic requirements for floors 32-39 and every 15th floor. These examples illustrate the challenges long-context models face in sustaining adherence to task instructions, particularly as sequence length increases.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - question 1: can you present a bit more concise takeaways from your benchmark, to me I feel like I was reading a lot of pieces, and no surprising results to me either. It might be good to have some failure cases to support your point

-  question 2: when you evaluate the prompt complexity, how do you choose the prompt formats?

-  question 3: Do you think complex prompts and instructions might need manyshots?

-  question 4: Do you think you can add some reasoning axes to your benchmark? 

- question 5: maybe consider adding some long-context recent models with SOTA results, not only looking at model parameter counts but also architectural differences.
soundness: 3
strengths: - First benchmark focusing on long-form generation during the test time
- The evaluation combines both complexities of evaluation prompts and different scenarios
- First batch of results on 10 mainstreamed LLMs
- The paper is easy to follow
summary: This paper introduces LongGenBench, a benchmark for measuring LLMs' capacities, especially their long-context abilities, by generating long-form context from 16k to 32k with rather complex instructions. This new dataset departs from traditional benchmarks aiming at decoded length in four different scenarios. Preliminary evaluations are done with main streamed LLMs.
weaknesses: - I am a little bit distracted from the main takeaways from the experimental studies, and not so convinced with failure cases. See question 1

I have other minor concerns regarding the experiment setup

- There has been much research showing that the prompt format matters, what's your thought? 

- Reasoning tasks are not well involved, as o1 seems to argue that longer decoded length is helpful with reasoning complex tasks, in your benchmark, you might want to add an axis of reasoning ability clearly or have some analysis around this topic?

- Most of the evaluation focused on existing transformer-based architecture, but models are presenting SOTA results, for example, mamba-based models. Are those models, with good inference time complexity, good at benchmark, or if not, why?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * How do you split the long generation and match them to all the subtask instructions?
* How do you check if every sub-instruction is satisfied? is it by prompting another LLM or by word matching, etc.?
* What's the significant difference between STIC-1 and STIC-2? Looks they just have difference denominators. I don't quite get it although there is a paragraph in Sec. 2.4 as below for this. Is there any specific case where an LLM can get low STIC-1 while high STIC-2, or the other way around?

> STIC-1 is primarily concerned with the completion rate of instructions that result in sub-scenarios,
focusing on whether instructions are correctly executed. In contrast, STIC-2 assesses the overall completion of the specific instruction task, including the presence of sub-scenarios and their completion
status.
soundness: 3
strengths: * This paper is overall clear and easy to understand.
* This proposed evaluation is novel, and the generation ability it benchmarks is not covered by previous metrics.
summary: Existing LLMs' long-context benchmarks rely on understanding tasks. This paper proposes a benchmark targeting specifically on long-context *generation* ability of LLMs. The authors design 4 long-context instruction-following tasks, up to 16 or 32K tokens: (1) Writing Diary for a year; (2) Wrting menu for a year; (3) Design a 100/300-floor skyscraper; (4) Plan an urban layout. As a result, all existing LLMs don't work well on these tasks of the benchmark.
weaknesses: * Some of the details are possibly missing or hard to get by readers -- see "Questions".
* In the proposed benchmark, the way to form long content is to pile short answers to many sub-queries, while the sub-tasks are actually independent, to a large extent. For example, given all the demands on one-year dairies, it should be easy for LLMs to write a diary if it is assigned a specific day of the year, while this benchmark just require the LLM generate 365 diaries all at once. In this case, the challenge of this benchmark might majorly be forgetting the instruction under the influence of generated content, instead of keeping the conherence and content interaction among generated long content. That latter should be the one mostly desired by the community.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. It seems that the evaluation metrics are designed for these scenarios. If there are new scenario tasks, do we need to update the evaluation metrics?
2. What other aspects of long text generation with LLM do you think need to be evaluated? It seems that your evaluation is more oriented towards some planning tasks or well-structured text. Is it difficult to evaluate the creation task? For example, it is difficult to design evaluation metrics for novel writing?
soundness: 3
strengths: 1. Interesting task design, which can evaluate the long text generation ability of large models from a certain perspective
2. The paper is well written.
summary: The paper introduces LongGenBench, a benchmark for evaluating large language models' (LLMs) ability to generate long-form text while adhering to complex instructions. It features tasks in four scenarios with different instruction types and lengths (16K and 32K tokens). The study finds that even advanced LLMs struggle with long-form generation, particularly as text length increases, highlighting the need for improvements in model architecture and training.
weaknesses: 1. The types of task scenarios are relatively limited, and it is impossible to comprehensively evaluate the long text generation capabilities of large models.
2. The evaluation metrics seem to be customized according to the scenario.
3. Limited number of models evaluated

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Would an IFEval [1] style setting where the outputs or attributes of the outputs could be verified definitively using code checks be a better option for such a benchmark? For long generations, getting the model outputs to match specific output patterns in the prompt is a challenge unto itself.

[1] https://arxiv.org/abs/2311.07911
soundness: 2
strengths: 1. The paper is the first attempt at long-context generation, requiring models to generate a long text that follows a combination of specific instructions as opposed to just answering questions pertaining to long prompts.
2. The benchmark does uncover a setting that seems to be challenging to SOTA models.
summary: The paper proposes a benchmark to evaluate models' strength in long-span generation tasks. It constructs synthetic examples from a fixed set of scenarios and templates in three modes. The resultant instruction measures models' abilities to faithfully follow position-specific, range-specific, and periodic instructions.
weaknesses: 1. The paper is very sparse on details regarding the evaluation of correctness. How is matching and parsing done w.r.t. the templates? A model could generate several outputs matching the criteria
2. The benchmark is limited to a few domains and scenarios, and the paper's contributions seem quite limited overall

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 2
questions: * In L460-468, it is mentioned that there are significant repetitions in long-form generations. Are these repetitions correct with respect to the given instructions, or are they semantically wrong (i.e. violating the given instructions)? The former indicates that it's probably caused by how instructions in the prompt are designed, while the latter means that model struggles at producing long and meaningful generation.
soundness: 3
strengths: * The paper is the first to study long-form generation as opposed to long-context generation. The perspective is novel, interesting, and of practical value to unleash the potential of LLMs for more complicated tasks.
* Problems in the benchmark are constructed in a sound and intuitive way. While evaluating the quality of long text snippets is usually challenging and complex, the smart design in this paper enables reliable and accurate evaluation for long-form capability.
summary: The paper proposes a new benchmark for long-form generation where the model-generated content, rather than the input context, is long. Specifically, the model is asked to roll out a detailed description of certain tasks such as diary over a year or floor plan for a skyscraper, subject to a few specific instructions that can be either singular or recurrent. Evaluation metrics include main task completion to test whether generation follows the expected format, as well as the success rate for specific instructions at both micro and macro level. Results demonstrate that the proposed benchmark is much more challenging than needle-in-the-haystack tasks for long context evaluation. Models generally do not perform very well within their claimed context lengths.
weaknesses: One of my major concerns is the clarity of writing in the evaluation part.
* The definition of STIC-1/STIC-2 isn't quite clear. Using the notation $T_S=(T_{S_1}, T_{S_2}, \dots)$ in Sec 2.3, my best guess is that STIC-1 means the average success rate over $(T_{S_1}, T_{S_2}, \dots)$, while STIC-2 counts the entire $T_S$ as successful only if all $(T_{S_1}, T_{S_2}, \dots)$ are successful, and gives 0 score otherwise.
* The abbreviation **CR** in Table3 isn't defined anywhere, though I can guess this is likely the Completion Rate in Main Task Completion.
* The terms "main task", "subtask",  "instruction task", "specific task" are used in a confusing way. It would be very helpful to unify them with clear definitions, and to associate the terms to symbols like $T_S$ or $T_{S_1}$.
* Missing x-ticks in Figure 2.

Apart that, there are a few technical details that are unclear to me.
* How do you determine whether the generated text at one specific point satisfies all task requirements? For example, given the generated diary of a particular day, how do you verify that all required activities (e.g. wedding, vacation, and etc.) are covered? I would imagine that a natural language inference (NLI) module should be involved but it's not mentioned in the paper.
* In Table 3, though the length to be evaluated at is 16K/32K respectively, the actual generation length seems only around half of the max length. How are the 16K/32K defined?

Meta-review: This paper proposes LongGenBench, a novel benchmark to evaluate the skills of LLMs in generating texts with an average of 20K. These generation tasks are synthetically constructed for two domains, diary/menu writing (e.g., plan menus for each week of the year) and building design/urban planning (design a 100-floor skyscraper). Evaluation is performed using LLMs-as-judge, where a long generated text is decomposed into individual components (e.g., design plan for each floor of the skyscraper), and each component is rated using an LLMs for correctness. Evaluation revealed that most state-of-the-art LLMs struggle with generating coherent texts beyond 10K+ tokens, despite registering strong results on existing long context benchmarks, such as Needle-in-a-Haystack and Ruler.

Most reviewers recognized the novelty and significance of this paper as the first benchmark on measuring LLM performance in generating extremely long texts (dQFM, p7FN, hgz3, bHUs). The benchmark is also designed in a clever way to facilitate automated evaluation and “enable reliable and accurate evaluation for long-form capability” (bHUs). The benchmark “uncovers a setting that seems to be challenging to SOTA models” (hgz3). The paper is also “well written” (Nybu) and “easy to follow” (dQFM, p7FN).

The major weaknesses are raised by Reviewer hgz3 and Nybu, namely the limited scope of synthetic domains and scenarios (hgz3, Nybu), as well as the comprehensiveness of the evaluation metrics (Nybu), such as being too task-specific and unable to capture more interesting aspects such as creativity. There are also other issues raised in the review, such as potential correctness issues using LLM-as-judge, but they are addressed during the rebuttal phase with additional experiment results. 

While there are still open questions around the limitations of using synthetic domains and improving evaluation methodologies using deterministic metrics besides LLMs-as-judge, as the research direction of long text generation / CoT is receiving more traction, we believe that this paper serves a timely and significant contribution to the community by offering the first benchmark in this area. The evaluation method is also already a step forward compared to classical Needle-in-the-Haystack approaches. Therefore, the decision is Acceptance.

Decision: Accept (Poster)