Integrating Planning into Single-Turn Long-Form Text Generation

## Abstract

Abstract Generating high-quality, in-depth textual documents, such as academic papers, news articles, Wikipedia entries, and books, remains a significant challenge for Large Language Models (LLMs).
In this paper, we propose to use planning to generate long form content. To achieve our goal, we generate intermediate steps via an auxiliary task that teaches the LLM to plan, reason and structure before generating the final text. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning.
To overcome the scarcity of training data for these intermediate steps, we leverage LLMs to generate synthetic intermediate writing data such as outlines, key information and summaries from existing full articles.
Our experiments demonstrate on two datasets from different domains, namely the scientific news dataset SciNews and Wikipedia datasets in KILT-Wiki and FreshWiki , that LLMs fine-tuned with the auxiliary task generate higher quality documents. We observed +2.5% improvement in ROUGE-Lsum, and a strong 3.60 overall win/loss ratio via human SxS evaluation, with clear wins in organization, relevance, and verifiability.

## 1 Introduction

Large language models (LLMs) have achieved remarkable progress in various text generation tasks, ranging from creative writing to summarization and dialogue generation [, ].
However, generating high-quality, coherent, and substantive long-form documents, such as academic papers, news articles, and books, remains a significant challenge [, ].
Existing work mostly tackles this challenge by sequentially prompting LLMs to write a small segment of the document at each call and integrating the outputs into the final long-form document [, , ].
Such systems usually require additional components to ensure the coherence or consistency of the entire document [, ].

In this paper, we present a novel approach that directly fine-tunes LLMs to generate long-form documents in one single call.
This approach enables us to fully leverage the token-level attention mechanism in the decoding process, thereby ensuring the coherence and consistency of the generated document.
It also significantly reduces the complexity of the system during serving and deployment to write long-form documents.

Writing high-quality long-form documents often involves a pre-writing stage [] where authors outline the structure, develop key arguments and plan for the overall flow of the document.
This additional stage enables the writers to simplify the tasks into manageable sub-tasks, similar to the idea of Chain-of-Thought (CoT) in multi-step reasoning [].
In fact, this stage is often included in the design of multi-stage long-form document writing systems [, , ] through multiple rounds of prompting.

Motivated by this idea, we propose to integrate this pre-writing stage into our development of the single-turn LLM writer.
Specifically, we introduce a series of auxiliary training tasks to endow LLMs with the skills to plan and structure long-form documents before generating the final full article.
For example, one auxiliary task could involve providing the LLM with the writing context as input and expecting it to produce an outline with key insights as the output. Another auxiliary task can present the LLM with the writing context and an outline, with the goal of generating the complete article.
We argue that fine-tuning the LLM writer with a mixture of writing tasks, coupled with guidance at varying levels of granularity, can enhance the model’s ability to produce long-form documents that are inherently well-structured and coherent.

While it is relatively easy to obtain sufficiently large corpora of full articles for supervised fine-tuning, obtaining intermediate writings such as article outline and key points directly from human writers is considerably more challenging as these are typically not well documented and made public.
To address this, we leverage the few-shot capabilities of LLMs to generate synthetic intermediate writings from full articles, along with the original document structure when available.
Note that, generating a concise summary, excerpt, or outline from a full-length, detailed article is much easier than doing the reverse.
Therefore, it becomes rather manageable to create abundant intermediate-writing data for the purpose of fine-tuning LLMs towards learning to plan for writing full articles.

Our experiments on multiple datasets demonstrate that LLM writers trained with the auxiliary tasks generate higher quality long-form documents in a single pass, even when the final inference task does not prompt the model to produce intermediate planning steps.

Our main contributions are summarized as follows:

- •
We propose a novel approach that directly fine-tunes LLMs to generate the entire long-form document in a single pass, simplifying the generation process and enhancing coherence.
- •
Inspired by human writing practices, the proposed framework incorporates the pre-writing stages by introducing auxiliary training tasks that teach the LLMs to plan and structure documents before generating the final text.
- •
To overcome the challenge of limited training data for intermediate writing steps, we leverage LLMs’ ability to generate synthetic summaries, outlines, and key information from existing full articles. This innovative approach unlocks a vast new source of training data for LLM planning.
- •
Our extensive experimental results demonstrated the effectiveness of the proposed approach on multiple datasets, showing that LLMs fine-tuned with the auxiliary tasks produce higher quality, more coherent long-form documents in a single pass.

## 2 Related work

#### Planning.

Our work contributes to the field of planning in long-form text generation. Humans typically simplify complex tasks into manageable subtasks, a method mirrored in recent approaches employing large language models (LLMs) for planning.
Techniques such as Chain of Thought (CoT) [], Zero-shot-CoT [], Self-consistent CoT (CoT-SC) [] guide LLMs through sequential reasoning by utilizing intermediate reasoning steps.
More advanced methods, like Tree of Thoughts (ToT) [], GoT [] enhance these strategies with tree-like and graph-based reasoning structures, respectively.
Additional methods, including RePrompting [] and ReWOO [] ensure planning accuracy by integrating observational data and using corrective prompting
HuggingGPT [] further decomposes tasks into sub-goals solved through iterative LLM interactions, contrasting the one-shot approach of CoT and Zero-shotCoT.
Despite these innovations, specialized zero-shot plan generation for specific domains remains to be challenging, addressed by models like LLM+P [] , LLM-DP [], and CO-LLM [].
Built upon previous works, our approach trains models specifically to enhance planning within the domain of long-form text generation.

#### Long-form text generation.

Long-form text generation and question answering (LFQA), which require maintaining coherence over extended texts, remain highly challenging for large language models (LLMs), as evidenced by numerous studies [, , , , ]. [] introduced a progressive generation technique utilizing pretrained language models to enhance the creation of extended narratives. [] explored discourse-aware neural extractive text summarization, essential for maintaining logical flow and thematic consistency in long documents. There have also been efforts to generate Wikipedia articles, as documented by [, , ], along with recent advancements by []. [] developed the Imitate Retrieve-Paraphrase framework to enhance expository writing at the paragraph level, particularly focusing on the integration of information from diverse sources. Our research introduces a novel method that leverages organic long documents to create intermediate text generation plans. This approach trains models to enhance their abilities not only in generating text but also in generating and adhering to these structured plans, thereby distinguishing our method from previous work. The use of the Retrieval-Augmented Generation (RAG) technique is outside the scope of our current discussion, although our method is compatible with RAG applications.

## 3 Problem Setup

Figure: Figure 1: Input/output of training and inference tasks.
Refer to caption: extracted/5908544/figures/training_and_inference.png

#### Long-form text generation.

Given an input context $x_{i}$, which can either be the source academic paper for SciNews generation or a sentence prompt (e.g., “generate wiki page about {topic}” for Wikipedia page generation), the objective is to fine-tune an LLM to generate a long-form article $y_{i}$ from $x_{i}$.

The dataset $D$ used for fine-tuning is structured to align with this objective. It consists of pairs of input contexts and their corresponding final documents. Formally, the dataset $D$ is defined as:

$$ $D=\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n})\}$ $$

where each tuple $(x_{i},y_{i})$ represents an input context $x_{i}$ and the final document $y_{i}$.

#### Intermediate steps.

To reduce the cognitive burden on LLM when generating a full article directly from the input context, intermediate steps are often introduced, gradually leading from the input context to the final full article.
In particular, for each input context $x_{i}$ and its corresponding article $y_{i}$, the intermediate steps ${\mathbf{z}}_{i}$ can either be a set of distinct pieces of information (e.g., summary, outline, key information) or a sequence of information where each element $(j+1)$-th is dependent on the preceding element $j$. We denote ${\mathbf{z}}_{i}$ as:

$$ ${\mathbf{z}}_{i}=\{z_{i1},z_{i2},\ldots,z_{ik}\}$ $$

The overall idea is that the auxiliary information provided at these intermediate steps ${\mathbf{z}}_{i}$ can effectively guide the LLM to gradually approach the final target article $y_{i}$ from the potentially abstract or noisy input context $x_{i}$. There are multiple ways to instantiate the concrete content of ${\mathbf{z}}_{i}$. For example:

- 1.
Linear: One can decompose the full article writing task into writing one section at a time. In this approach, the first intermediate step $z_{i1}$ would be the first section; the $j$-th intermediate step $z_{ij}$ would be concatenating one more section to the $(j-1)$-th draft.
- 2.
Top-down: Another common strategy is to use intermediate steps from the most abstract outline to gradually more detailed content. For example, the first intermediate step $z_{i1}$ can be an abstract outline only with the title of each section of the article.
Each subsequent intermediate step would gradually elaborate on the content within each section. This kind of intermediate steps are not usually readily available, and may need to be constructed from the full article, which we will describe later in this paper.

Note that many existing works aim to developing multi-turn LLM writers, which involves a series of tasks such as generating $z_{i1}$ from $x_{i}$, then $z_{ij}$ from $z_{i(j-1)}$. However, this is not the focus of our paper.
In this paper, as shown in Figure 1, we introduced intermediate steps to create different training tasks for fine-tuning LLMs.
Our final objective remains to generate the full article $y_{i}$ from the input context $x_{i}$ in a single turn.
During inference, the input is always only the input context $x_{i}$. The model can either generate only the article $y_{i}$ or both the intermediate steps ${\mathbf{z}}_{i}$ and the article $y_{i}$, as long as the article can be easily extracted from the complete model output through post-processing.

## 4 Methodology

Our proposed framework
addresses the challenge of generating long-form text with LLMs by utilizing the inherent structure of documents.
Specifically, we construct intermediate steps ${\mathbf{z}}_{i}$ based on this structure to guide the generation process.
The key insight is that the inherent structure, such as an article’s outline, provides crucial guidance for organizing the generated article $y_{i}$.

Within this framework, we construct the intermediate steps ${\mathbf{z}}_{i}$ by extracting the implicit structural information inherent in a formulated document and then constructing a hierarchical writing plan accordingly.
These intermediate steps then serve as a foundation for fine-tuning the LLM, focusing on tasks that emphasize structural understanding and plan interpretation. At inference time, the fine-tuned LLM, equipped with enhanced structural and plan interpretation capabilities, generates the final long-form text $y_{i}$ in a single pass. The specific components of this framework are discussed in the following sections.

### 4.1 Writing Plan as Intermediate Steps

Writing high-quality long-form documents typically begins with a pre-writing phase, where authors establish the structural framework, develop key arguments, and formulate the overall trajectory of the narrative. In this work, we define the pre-writing stage as a series of intermediate steps essential for generating long-form documents. Specifically, without loss of generality, we concentrate on three distinct types of intermediate steps: document summary, document outline, and document key information.

#### Document Summary

A document summary is a concise representation that captures the core message of a document, summarizing key points, themes, or arguments while omitting peripheral details. It provides a quick overview, enabling readers to grasp the essential content without reading the entire document.

#### Document Outline

A document outline represents the hierarchical structure of a document. It reveals the organization and flow of ideas, the relationships between sections, and the key points within the document.

#### Document Key Information

Document key information includes the most crucial facts, findings, or insights within a document. It typically represents the core knowledge that the document aims to convey or support.

### 4.2 Structured Intermediate Steps Construction

Figure: Figure 2: Constructing intermediate steps.
Refer to caption: extracted/5908544/figures/intermediate_construction.png

The intermediate steps ${\mathbf{z}}_{i}$ involved in the creating long-form documents provide valuable structural guidance for the generation process. However, this structured information is not always explicitly available, nor is the alignment between the structure and the final document. As discussed above, authors typically establish these intermediate steps when writing long-form documents. Therefore, we posit that such implicit structural information is already embedded within the final document itself.

Building on this observation, we propose a method to extract the intermediate steps from established documents $y_{i}$, such as news articles and Wikipedia entries. Leveraging recent advances in LLMs, particularly their impressive few-shot learning capabilities, we aim to synthetically generate these intermediate steps directly from the documents.
This approach captures the implicit structural information without relying on explicitly provided intermediate steps. Furthermore, it can generate multiple intermediate steps for a single document, thus enabling exploration of various organizational strategies. Lastly, our method provides a consistent and scalable solution for generating intermediate steps, making it well-suited for large-scale document processing. Figure 2 summarizes the process of our proposed process to construct intermediate steps from the article.

To extract the intermediate steps from an article $y_{i}$, we prompt pre-trained LLMs, i.e., Gemini Ultra, to generate $K$ intermediate step candidates $\{\mathbf{z}_{i}^{k}\}_{k=1}^{K}$, where ${\mathbf{z}}_{i}$ can be any kinds of structure information including document summary, document outline, and document key information.
Despite LLM’s impressive summarization capabilities, they can sometimes produce hallucination information. To mitigate this, we select the best intermediate step from $K$ candidates using a designed scoring function that evaluate coherence and completeness with respect to the document $y_{i}$.
The candidate that demonstrates the highest overall quality is chosen.

The construction of the intermediate steps is detailed in Algorithm 1. Take the document summary as an example, we first normalize the word and paragraph counts relative to the input text. We then apply a sinusoidal scoring function, $\mathbf{g}(\cdot)$ in the Algorithm 1, derived from empirical observations, to assess the adherence to desirable length and structure. Additionally, we incorporate an entailment score using a hallucination model, measuring the bi-directional logical coherence between the summary and the original news article. In our experiment, we use the hallucination model introduced in []. This entailment score assessed both the extent to which the summary accurately reflected the news (no hallucination) and the degree to which the summary covered the key information in the news. These individual scores (word/paragraph count, entailment) were combined multiplicatively to produce a final score for each summary, allowing us to select the highest-scoring option as the optimal summary.

Figure: Algorithm 1 Constructing intermediate steps data from organic long-form text data

### 4.3 Fine-tuning

To empower the LLM with the ability to utilize structural information during long-form text generation, we propose a fine-tuning approach that goes beyond the conventional input-to-output (e.g., $x_{i}\rightarrow y_{i}$) paradigm. Our method leverages intermediates steps ${\mathbf{z}}$, which encapsulate the structural essence of the desired output.

Specifically, we introduce two auxiliary fine-tuning tasks in addition to the standard $x_{i}\rightarrow y_{i}$ task.

#### Input to output with intermediate steps ( x i → 𝐳 i ⊕ y i → subscript 𝑥 𝑖 direct-sum subscript 𝐳 𝑖 subscript 𝑦 𝑖 x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → bold_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⊕ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )

: This task trains the LLM to generate both the final long-form text $y_{i}$ and the corresponding intermediate steps ${\mathbf{z}}_{i}$. It encourages the model to internalize the relationship between content and structure, fostering a deeper understanding of how different elements of a document contribute to its overall organization.
For this task, the training dataset is,

$$ $D=\{(x_{1},{\mathbf{z}}_{1}\oplus y_{1}),(x_{2},{\mathbf{z}}_{2}\oplus y_{2}), \ldots,(x_{n},{\mathbf{z}}_{n}\oplus y_{n})\}$ $$

#### Input and intermediate steps to output ( x i ⊕ 𝐳 i → y i → direct-sum subscript 𝑥 𝑖 subscript 𝐳 𝑖 subscript 𝑦 𝑖 x_{i}\oplus{\mathbf{z}}_{i}\rightarrow y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⊕ bold_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )

: In this task, the LLM is provided with both the input $x$ and the synthetically generated intermediate steps ${\mathbf{z}}_{i}$ as context. The model is then trained to generate the final long-form text $y_{i}$, conditioned on the structural guidance provided by ${\mathbf{z}}$. Here, ${\mathbf{z}}$ acts as a blueprint for generating the long-form text, helping the model maintain focus and generate more coherent and structured output. In this task, the training dataset is:

$$ $D=\{(x_{1}\oplus{\mathbf{z}}_{1},y_{1}),(x_{2}\oplus{\mathbf{z}}_{2},y_{2}), \ldots,(x_{n}\oplus{\mathbf{z}}_{n},\oplus y_{n})\}$ $$

The two tasks complement each other by focusing on different aspects of the
planned generation process. The $(x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i})$ task teaches the model to generate structure alongside content, emphasizing the inherent relationship between the two. Meanwhile, the $(x_{i}\oplus{\mathbf{z}}_{i}\rightarrow y_{i})$ task trains the model to effectively utilize provided structural information, improving its ability to follow a given plan.
By incorporating these auxiliary tasks, our fine-tuning process aims to improve the LLM’s capability of generating long-form text that is not only informative but also well-structured.

## 5 Experiments

We conduct extensive experiments on multiple datasets with different setups to validate the effectiveness of our proposed methods.

### 5.1 Benchmark Datasets

To validate the effectiveness of our planned generation method, we conducted experiments using the SciNews dataset and Wikipedia dataset.

#### SciNews Dataset.

SciNews dataset [] is developed to facilitate the task of compiling academic publications into scientific news reports.
This dataset provides a parallel collection of academic publications and their corresponding scientific news reports across 9 disciplines, including Medicine, Biology, Physics, and Chemistry.
It comprises 41,872 samples, with each academic publication averaging 7,760.90 tokens and each news report averaging 694 tokens.
In this dataset, we use each academic publication as the input $x_{i}$ and its corresponding news report as $y_{i}$.
We randomly sample 33,497 items as the training split, 1,000 items for validation and 200 items as the evaluation split.

#### Wikipedia Dataset.

We use the Wikipedia corpus from the KILT benchmark dataset [] to construct the training and validation splits.
This benchmark dataset is sourced from a snapshot of Wikipedia taken on August 1, 2019, and contains 5.9 million articles.
The evaluation split is constructed from the FreshWiki corpus [], which consists of 100 high-quality Wikipedia page with most edits for each month from February 2022 to September 2023.
Both dataset includes the full text of Wikipedia pages, along with descriptions and titles.
We filtered both corpus by removing articles with less than 1,000 words and those lacking any structured sections.

For each Wikipedia article $y_{i}$, we formulated the input context $x_{i}$ as a one sentence prompt:

> *Generate a comprehensive Wikipedia page about the specified topic.* *topic: [entity]*

We randomly sampled 1,900 items from the KILT corpus as the training split, 232 items for the validation split, and used all the 98 filtered items from the FreshWiki corpus as the evaluation split.

### 5.2 Evaluation Metrics

#### Offline metrics.

To assess the effectiveness of the proposed method,
we utilized the F1 scores of ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL), and ROUGE-Lsum (RLsum) metrics for this evaluation.

#### Human and auto SxS.

In addition to these ROUGE metrics, and in line with the concept of critical evaluation [], we also conducted SxS evaluations, both by human expert raters and by an LLM evaluator. For the LLM evaluator, we employed a more capable LLM (Gemini Ultra) as an automatic SxS rater to compare the generated articles from two methods. For both human and auto SxS, raters are asked to rate which one of two generated articles is better.
The base and the test sides are randomly flipped to avoid potential bias towards one side. We calculate and report the win rate and W/L ratio of our proposed method compared against the baseline.

### 5.3 Experiment setup

We use Gemini Pro model for both our baselines and all of our fine-tuning experiments.
For experiments on the SciNews dataset, we limit the input sequence length to the model to be no more than around 16k tokens, and the output sequence length to be no more than 4k tokens.
For experiments on the Wikipedia dataset, we limit the input sequence length to be no more than 1k tokens as the input context in this data set is much shorter. The output sequence length limit is set to 6k tokens.

All the fine-tuning experiments are conducted on TPU.
The batch size is set to 16 and 32 respectively for SciNews and Wikipedia datasets.
The maximum learning rate for all experiments is set as $10^{-5}$.

### 5.4 Results

#### Overall comparison.

Table 1 shows the results of the following models and prompt settings:

- 1.
Zero-shot (ZS): Directly using the LLM without fine-tuning. The input context $x_{i}$ to the LLM is the same as the our proposed methods.
- 2.
Fine-tuning without intermediate steps (FT w/o I): Only fine-tune the LLM with the input context $x_{i}$ as input and the output article $y_{i}$ as output without using any intermediate steps ${\mathbf{z}}_{i}$.
- 3.
Fine-tuning with intermediate steps (FT w/ I): Fine-tune the LLM with the a mixture of two training tasks: generating the full article $y_{i}$ from the input context $x_{i}$ ($x_{i}\rightarrow y_{i}$), and generating all the intermediate steps ${\mathbf{z}}_{i}$ and the full article $y_{i}$ from the input context $x_{i}$ ($x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$). The instruction prompt from the two tasks are different to ensure the LLM can differentiate these tasks. See Section 4.3 for more details. During inference the LLM is prompted with input context $x_{i}$ to only output the full article $y_{i}$ ($x_{i}\rightarrow y_{i}$).
- 4.
Fine-tuning and inference with intermediate steps (FT w/ I, Output w/ I): Similar to the fine-tuning with intermediate steps method above, except that during inference the LLM is prompted with input context $x_{i}$ to generate both intermediate steps ${\mathbf{z}}_{i}$ and the full article $y_{i}$ ($x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$). Only the full article is extracted from the generated output for evaluation.

Table: Table 1: Performance of different methods with various training and prompt setups

Compared with the zero-shot baseline ZS, fine-tuning LLM, even without the intermediate steps, significantly improved the ROUGE scores.
Moreover, when LLM is fine-tuned with the mixture training data which incorporated the task of generating intermediate steps, the performance is further improved on both datasets.
Specifically, on both datasets, the R1 score improved by more than +2.7-4.4% when compared to the FT w/o I baseline, the R2 score by more than +1.3-2.0%, and the RLsum score by more than +2.5-4.4%.

These results demonstrate that the inclusion of the generating intermediate steps task during fine-tuning significantly enhances long-form text generation by LLMs. The improvements across various metrics and both datasets highlight the robustness of the proposed methods.

We also observe that prompting the LLM to output the intermediate steps ${\mathbf{z}}_{i}$ during inference (FT w/I, Output W/I) achieves better performance when prompting the LLM to only output the article (FT w/I).
This is expected as explicitly writing down the planning process during the inference can help the model to generate more structured article, similar to the usage of chain-of-thought prompting for reasoning tasks [].

#### Comparison of different training and inference recipes.

We also conduct a comparison to understand whether different combinations of training data mixtures and inference tasks lead to different effectiveness.
Each mixture contains one or multiple training tasks: directly generating the article from input context $x_{i}\rightarrow y_{i}$; generating both the intermediate steps and the article from the input context $x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$; and generating the article from the input context and the intermediate steps: $x_{i}\oplus{\mathbf{z}}_{i}\rightarrow y_{i}$.
We also test different inference tasks, including only prompting the LLM to output the entire article ($x_{i}\rightarrow y_{i}$) or prompting the LLM to output both the intermediate steps and the article ($x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$).

Table: Table 2: Performance of models with different training mixtures during fine-tuning.

Table 2 illustrates the comparison results.
The results show that models fine-tuned on the training mixture with intermediate steps consistently outperform models only fine-tuned to generate the article directly.
On both datasets, the best performance is achieved when the model is fine-tuned on the mixture of $x_{i}\rightarrow y_{i}$ task and the $x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$ task, and prompted to output both the intermediate steps and the article ($x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$) during inference.

We also experiment with the mixture of three tasks on the Wikipedia data set: the vanilla $x_{i}\rightarrow y_{i}$ task, the $x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$ task which generates the intermediate steps, and the $x_{i}\oplus{\mathbf{z}}_{i}\rightarrow y_{i}$ task which generates the article from both the input context and the intermediate steps. However, further adding the third task does not seem to bring extra performance improvement.
This might suggest that writing an article given the outlines is a relatively trivial task for the underlying LLM in our experiments. Further fine-tuning the LLM on this task does not bring additional insight for the model to write better article.

Table: Table 3: Single-turn vs. multi-turn inference on SciNews.

#### Single-turn vs. multi-turn.

In addition to identifying the best recipe of training mixture and single-turn inference, we also compare the most competitive training recipe in Table 2 against multi-turn inference on SciNews. Results in Table 3 show that single-turn inference notably outperforms multi-turn inference.

Table: Table 4: Human SxS results comparing the proposed FT w/ I method to FT w/o I baseline

#### Human and auto SxS results.

To further validate the effectiveness of our methods, we conducted two sets of side-by-side ratings, one by expert human raters and one by LLM autorater, to compare the outputs of our methods against those of the baseline.

For the SciNews dataset, we select the FT w/I method using the training mixture of $x_{i}\rightarrow y_{i}$ and $x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$ tasks as it demonstrated the best performance.
For the Wikipedia dataset, we select the FT w/I method using the training mixture of three tasks: $x_{i}\rightarrow y_{i}$, $x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i}$ and $x_{i}\oplus{\mathbf{z}}_{i}\rightarrow y_{i}$, as it shows comparable performance to the method using only two mixtures.
In both cases, the baseline is the vanilla FT w/o I method without using intermediate steps.

For human SxS, we asked expert human raters to rate 50 items of each data set. They were asked to compare the two outputs for each input item (“paper body” for SciNews, “topic” for Wikipedia) in three of the criteria defined in [] that are most relevant for our task, namely coherence and organization, relevance and focus, and verifiability, and to provide an overall assessment. The results are presented in Table 4. Our methods show strong win against the baseline on SciNews, both in the overall quality and in each of the three criteria. On Wikipedia, we also observed a positive result in all criteria and overall.

Besides Human SxS evaluation, we also employed the automated LLM SxS as additional validation, on 200 samples from SciNews and all 100 samples of FreshWiki.
The results show that the FT w/I method achieves W/L ratio of 2.85 and 1.20 on SciNews and Wikipedia datasets, respectively, both larger than 1.

Table: Table 5: Performance of different strategies on SciNews with Gemini 1.5 Flash as the base model.

#### Alternative base model: Gemini 1.5 Flash.

To show that our method are applicable to the latest generation of LLMs, we also conducted smaller scale experiments (due to cost limitations) on Gemini 1.5 Flash as the base model. Table 5 show the results on SciNews, which corroborate the findings on Gemini 1.0 Pro shown in Table 2.

## 6 Conclusions

In this paper, we explore to fine-tune LLM to write long-form articles in one single turn.
We propose to include intermediate planning steps, such as starting with a concise summary, writing down the outlines and collecting some key information into the mixture of the training data.
Noting that such intermediate steps are not available in most existing data sets, we propose to construct synthetic intermediate steps from existing full-length articles. We prompt an LLM to extract, shorten and summarize the article into a tree structure, where each level corresponds to an intermediate step.
We fine-tune the LLM writer with different mixtures before prompt the LLM writer to write the full article.
Our experiments on two data sets from different domains: SciNews and Wikipedia, verify that our proposed method can substantially boost the quality of the generated articles.

Our exploration creates a new paradigm in long-form text generation: instead of applying LLM iteratively to generate intermediate steps until the full article is obtained, we include the intermediate steps into a mixture of training data and directly fine-tune an LLM to generate all of them in one turn.
While we only experiment with one specific way of constructing the intermediate steps (as tree-structured planning outlines), there are many other different possible ways to create the auxiliary training mixture.
As modern LLMs support longer and longer input and output sequence lengths, we believe it is possible to build even more more sophisticated long-form LLM writers with embedded capabilities other than planning. For example, one can also fine-tune the LLM to perform fact-checking reasoning on-the-fly to improve the factuality of the generated article.

## 7 Limitations

In this work we focus on writing long-form articles in a totally closed-book setup, where the only external information is provided in the input context.
However, a more realistic setting would be retrieval-augmented generation (RAG), which equips the LLMs with a retriever to external knowledge.
Since we do not adopt a RAG setting, we also do not compare to other existing methods that adopt RAG (e.g., STORM [])
While we do not adopt a RAG setting, we believe our work can be easily adapted to incorporate RAG to achieve better performance.

## 8 Ethics Statement

Our approach to incorporating planning steps into LLM training aims to significantly improve the quality of long-form text generation. This advancement has the potential to benefit various sectors by providing more coherent, accurate, and contextually rich content. We believe that the benefits of our work, including the enhancement of educational resources and the facilitation of better information dissemination, outweigh the minimal risks involved.

## Appendix A Prompts

In this section we provide the prompts we used for different tasks.

### A.1 SciNews

#### Only generating the full article ( x i → y i → subscript 𝑥 𝑖 subscript 𝑦 𝑖 x_{i}\rightarrow y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ).

Below we provide the prompt for

#### Generating the intermediate steps and the full article ( x i → 𝐳 i ⊕ y i → subscript 𝑥 𝑖 direct-sum subscript 𝐳 𝑖 subscript 𝑦 𝑖 x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → bold_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⊕ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ).

Below we provide the prompt for

### A.2 Wikipedia

#### Only generating the full article ( x i → y i → subscript 𝑥 𝑖 subscript 𝑦 𝑖 x_{i}\rightarrow y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ).

Below we provide the prompt for

#### Generating the intermediate steps and the full article ( x i → 𝐳 i ⊕ y i → subscript 𝑥 𝑖 direct-sum subscript 𝐳 𝑖 subscript 𝑦 𝑖 x_{i}\rightarrow{\mathbf{z}}_{i}\oplus y_{i} italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → bold_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⊕ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ).

Below we provide the prompt for

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. I am curious about the impact of this training method on the model's performance in generating short texts.
2. If we directly guide the model to first generate intermediate steps and then the full article, how does its effectiveness compare to the FT w/ I, Output w/ I method? Is the reasoning time roughly the same for both? For broader application, I believe the former might be better; however, considering efficiency, FT w/ I could be more suitable.
3. What is the difference between the loss calculated for the second task mentioned in section 4.3 and the first task in terms of the $y_i$ part?
soundness: 2
strengths: The authors have validated the effectiveness of their method through both automated metrics and human evaluation, showcasing the method's practical utility.
summary: This paper presents an approach to integrate planning into single-turn long-form text generation using Large Language Models (LLMs). The authors propose generating intermediate steps through an auxiliary task that teaches LLMs to plan, reason, and structure content before final text generation. To address the scarcity of training data for intermediate steps, the authors leverage LLMs to create synthetic intermediate writing data from existing full articles. Experiments on the SciNews and Wikipedia datasets demonstrate that fine-tuning LLMs with this auxiliary task results in higher quality documents, with improvements in ROUGE scores and human evaluation metrics.
weaknesses: 1. There is significant room for improvement in the writing style and structure of the paper. For example, Sections 4.1, 4.2, and 5.1 are overly verbose and could benefit from a more concise presentation, with implementation details potentially moved to the appendix. The organization of the paper is somewhat disjointed, failing to highlight key points effectively. For instance, the second task introduced in section 4.3 is mentioned as ineffective in the experiments (Table 2); thus, its inclusion in the methodology section seems unnecessary.
2. The methodology lacks novelty. The experiments are limited in scope and content, especially considering the paper could benefit from a more streamlined language.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: - Although it might not impact the baseline comparison because there are no comparison against groundtruth data, for Wikipedia dataset, is there any possible data contamination?
- What does multi-turn inference mean in line 432? Is it prompt LLM to generate each intermediate first then generate the full document?
- Why is the coherence & organization for Wikipedia models win ratio bad? Is it because more hallucinations appears? Does this mean that the hallucination mitigation is less effective on Wikipedia data?
soundness: 3
strengths: - Very well-written paper! The proposed method is explained in a clear way, with the difference of other objectives (generating in multiple turns) distinguished.
- The proposed method’s intermediate steps are all in scope, and all intuitively should help with full document generation. They also try to mitigate hallucinations by using a score function of coherence and completeness to choose the highest overall quality. As confirmed by both automatic metrics as well as human/LLM as judge metric, the method constantly performs better than the baseline.
- The ablation is relatively thorough. The authors compare against different evaluation/training paradigms, different mixture of training data, and single-turn / multi-turn inference.
summary: This paper proposes to fine-tune LLMs to generate long-form content directly in a single pass through a pre-writing auxiliary task that teaches the LLM to plan, reason and structure. They leverage Gemini Ultra to generate the training data for this auxiliary task, namely generating synthetic summaries, outlines and key informations from full articles. Three training paradigms are used to fine-tune the model. To mitigate the possible hallucinations when generating synthetic data, they have a sinusoidal scoring function to assess the adherence to length and structure, and a hallucination model for coherence between the synthetic data and original article. Improvements are shown in Rouge-LSUM, and 3.6 win/loss ratio in organization, relevance and verifiability.
weaknesses: - For evaluation metrics, although human and auto SxS are used, only 50 articles are rated by human. Furthermore, the LLM that is used to generate the synthetic data is used as the rater. As suggests by some previous work (Panickssery et al. 2024), for example, LLM raters might be able to recognize and favor their own generations. Therefore, there are chances that the model that is trained on additional Gemini Ultra generated data might be favored more than zero-shot.
- Length impact — it seems that for SciNews, the news report is 694 tokens, but the LLM is allowed to generate no more than 4k tokens, and for Wikipedia dataset, FreshWiki’s length is less than 3k words, and the LLM is allowed to generate no more than 6k tokens. Although there is no comparison against the groundtruth documents, is it possible that some method generates more/less tokens and thus is favored by either Rouge scores or SxS raters?
- The paper only tested the Gemini family models.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In line 38-40, the author mentioned that the proposed approach enables us to fully leverage the token-level attention mechanism in the decoding process. How does the token-level attention mechanism is fully used in the decoding process? How does it ensure the coherence and consistency of the generated document? These are not clear to me.
2. Compared to the previous work involving multi-turn generation passes, does the proposed approach have any advantages in terms of the length of the generated text? Can the proposed approach generate longer and better text than the previous work?
soundness: 2
strengths: Pros:
* The paper is well-written and easy to follow.
* The construction of the training data is automatic, leveraging off-the-shelf LLMs to generate synthetic intermediate writing data based on the full articles, which saves the expensive cost of manual annotation. 
* With the auxiliary task at training time, the model can learn to generate more coherent and structured long-form text in a single pass.
summary: In this paper, the authors propose integrating planning into single-turn long-form text generation using Large Language Models (LLMs). The approach involves an auxiliary task, generating intermediate steps, to enhance coherence and structure in the finally generated text. The authors construct training data by leveraging LLMs to generate synthetic intermediate writing data based on the full articles. Experiments demonstrate improved quality and coherence in long-form documents generated by LLMs fine-tuned with this method.
weaknesses: Cons:
* The paper lacks a comparison with previous work like ProGen (Tan et al., 2021), Ex$^3$ (Huang et al., 2024), etc. The invloved baseline training and prompt setups are too simple to demonstrate the superiority of the proposed approach. 
* Besides the cohenrence and structure, the approach in this paper appears to be more efficient and faster than the previous work. The paper lacks metrics to measure the efficiency and speed of the proposed approach compared to previous work.
* The offline metrics used in the paper are not sufficient to evaluate the *cohenrence* and *structure* of the generated text. 

References:
[1] Tan, Bowen, et al. "Progressive Generation of Long Text with Pretrained Language Models." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
[2] Lei, Huang, et al. "Ex3: Automatic Novel Writing by Extracting, Excelsior and Expanding." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Is there an analysis of output length across different generation methods?
- Has the quality of the generated outlines been evaluated?
- For human evaluations, what is the inter-annotator agreement score?
soundness: 3
strengths: - This paper studies an important task of long-form text generation. Incorporating auxiliary tasks to improve long-form writing performance is interesting;
- Both automatic and human evaluations on two datasets prove the effectiveness of the method;
- The analysis, particularly the findings in Table 2, is interesting and may offer insights to future research.
summary: This paper proposes a training approach that enhances long-form writing capabilities in LLMs by incorporating pre-writing stages through auxiliary training tasks. In addition to the conventional end-to-end training for direct output generation, the approach includes two intermediate tasks: generating an outline from the input and creating a complete article based on both the input and outline. The model is trained using a mixture of these tasks. During inference, the model can either generate a full article directly or follow the intermediate steps before producing the final output. The method is evaluated on the SciNews and Wikipedia datasets, with both automated metrics (ROUGE and LLM evaluations) and human assessments demonstrating that LLMs fine-tuned with these auxiliary tasks deliver improved outputs.

[After rebuttal] After reading the author responses and other reviewers' comments, I tend to keep my score unchanged.
weaknesses: - The proposed approach is similar to [1], which also employs multitask training for long-form generation. Clarifying the distinctions between the two methods would enhance the paper;
- Only one LLM (Gemini) is used in the experiments. Including recent open-source LLMs, such as LLAMA3, would strengthen the results' validity, especially as fine-tuning smaller LLMs is often more feasible in practical applications;
- Evaluation is another concern. ROUGE primarily measures word overlap; including more advanced metrics, such as BERTScore, would provide a deeper assessment. Additionally, LLM-based evaluations currently focus on overall quality, lacking fine-grained aspects. Faithfulness and factuality are also absent;
- Sample outputs and error analysis are not provided, which limits understanding of common issues and qualitative insights.



[1] MOCHA: A Multi-Task Training Approach for Coherent Text Generation from Cognitive Perspective, EMNLP'22

Meta-review: The paper addresses long-form text generation by planning intermediate steps. Reviewers gave inconsistent scores. Compliments include being well-written and thorough analysis. Concerns, however, are also raised, regarding the lack of novelty and lack of comparison with previous work.

The authors mention that their work differs from previous work in multiple ways, such as CoT reasoning and single-turn vs. multi-turn. However, I do not feel such difference is substantial enough to warrant a significant contribution or the waiver of experimental comparison.

Decision: Reject