Venue: NAACL 2024

# Assisting in Writing Wikipedia-like Articles From Scratch

with Large Language Models

Yijia Shao Yucheng Jiang Theodore A. Kanell Peter Xu Omar Khattab Monica S. Lam Stanford University {shaoyj, yuchengj, tkanell, peterxu, okhattab}@stanford.edu lam@cs.stanford.edu

###### Abstract

We study how to apply large language models to write grounded and organized long-form articles from scratch, with comparable breadth and depth to Wikipedia pages. This underexplored problem poses new challenges at the _pre-writing_ stage, including how to research the topic and prepare an outline prior to writing. We propose **STORM**, a writing system for the Synthesis of Topic Outlines through **R**etrieval and **M**ulti-perspective Question Asking. STORM models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. For evaluation, we curate FreshWiki, a dataset of recent high-quality Wikipedia articles, and formulate outline assessments to evaluate the pre-writing stage. We further gather feedback from experienced Wikipedia editors. Compared to articles generated by an outline-driven retrieval-augmented baseline, more of STORM's articles are deemed to be organized (by a 25% absolute increase) and broad in coverage (by 10%). The expert feedback also helps identify new challenges for generating grounded long articles, such as source bias transfer and over-association of unrelated facts.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ppas6252–6278 June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs) have demonstrated impressive writing capabilities Yang et al. (2023); Pavlik (2023); Wenzlaff and Spaeth (2022); Fitria (2023), but it is unclear how we can use them to write grounded, long-form articles, like full-length Wikipedia pages. Such expository writing, which seeks to inform the reader on a topic in an organized manner Weaver III and Kintsch (1991); Balepur et al. (2023), requires thorough research and planning in the _pre-writing_ stage Rohman (1965), even before the actual writing process can start. However, prior work on generating Wikipedia articles Banerjee and Mitra (2015); Minguillon et al. (2017); Liu et al. (2018); Fan and Gardent (2022) has generally bypassed the pre-writing stage: for instance, Liu et al. (2018) presume reference documents are provided in advance, while Fan and Gardent (2022) assume an article outline is available and focus on expanding each section. These assumptions do not hold in general, as collecting references and crafting outlines demand advanced information literacy skills Doyle (1994) to iden

Figure 1: We explore writing Wikipedia-like articles from scratch, which demands a pre-writing stage before producing the article. In this stage, simpler approaches like Direct Prompting have limited planning capacity. In contrast, STORM researches the topic via perspective-guided question asking in simulated conversations.



tify, evaluate, and organize external sources - a task that is challenging even for experienced writers. Automating this process can facilitate individuals in initiating in-depth learning about a topic and greatly reduce the expensive expert hours necessary for their expository writing.

We explore these challenges by focusing on how to generate Wikipedia-like articles _from scratch_. We decompose this problem into two tasks. The first is to conduct research to generate an outline, _i.e._, a list of multi-level sections, and collect a set of reference documents. The second uses the outline and the references to produce the full-length article. Such a task decomposition mirrors the human writing process which usually includes phases of pre-writing, drafting, and revising Rohman (1965); Munoz-Luna (2015).

As pre-trained language models inherently possess a wealth of knowledge, a direct approach is to rely on their parametric knowledge for generating outlines or even entire articles (_Direct Gen_). However, this approach is limited by a lack of details and hallucinations Xu et al. (2023), particularly in addressing long-tail topics Kandpal et al. (2023). This underscores the importance of leveraging external sources, and current strategies often involve retrieval-augmented generation (_RAG_), which circles back to the problem of researching the topic in the pre-writing stage, as much information cannot be surfaced through simple topic searches.

Human learning theories Tawfik et al. (2020); Booth et al. (2003) highlight _asking effective questions_ in information acquisition. Although instruction-tuned models Ouyang et al. (2022) can be prompted directly to generate questions, we find that they typically produce basic "What", "When", and "Where" questions (Figure 1 (A)) which often only address surface-level facts about the topic. To endow LLMs with the capacity to conduct better research, we propose the **STORM** paradigm for the **S**ynthesis of **T**opic **O**utlines through **R**etrieval and **M**ulti-perspective Question Asking.

The design of STORM is based on two hypotheses: (1) diverse perspectives lead to varied questions; (2) formulating in-depth questions requires iterative research. Building upon these hypotheses, STORM employs a novel multi-stage approach. It first discovers diverse perspectives by retrieving and analyzing Wikipedia articles from similar topics and then personifies the LLM with specific perspectives for question asking (Figure 1 (B)). Next, to elicit follow-up questions for iterative research (Figure 1 (C)), STORM simulates multi-turn conversations where the answers to the generated questions are grounded on the Internet. Finally, based on the LLM's internal knowledge and the collected information, STORM creates an outline that can be expanded section by section to develop a full-length Wikipedia-like article.

We evaluate STORM using our **FreshWiki** dataset (SS2.1) which curates recent, high-quality Wikipedia articles to avoid data leakage during pretraining.1 To facilitate the study of the pre-writing stage, we define metrics for evaluating the outline quality against human-written articles.

Footnote 1: Our resources and code are released at [https://github.com/stanford-oval/storm](https://github.com/stanford-oval/storm).

We further invited a group of experienced Wikipedia editors for expert evaluation. The editors found STORM outperforms an outline-driven RAG baseline, especially regarding the breadth and organization of the articles. They also identified challenges for future research, including addressing cases where: (1) the bias on the Internet affects the generated articles; (2) LLMs fabricate connections between unrelated facts. These challenges present new frontiers to grounded writing systems.

Our main contributions include:

* To evaluate the capacity of LLM systems at generating long-form grounded articles from scratch, and the pre-writing challenge in particular, we curate the FreshWiki dataset and establish evaluation criteria for both outline and final article quality.
* We propose STORM, a novel system that automates the pre-writing stage. STORM researches the topic and creates an outline by using LLMs to ask incisive questions and retrieving trusted information from the Internet.
* Both automatic and human evaluation demonstrate the effectiveness of our approach. Expert feedback further reveals new challenges in generating grounded long-form articles.

## 2 FreshWiki

We study generating Wikipedia-like articles from scratch, placing emphasis on the _pre-writing_ stage Rohman (1965), which involves the demanding sub-tasks of gathering and curating relevant information ("research"). This models the human 

writing approach which has prompted some educators to view Wikipedia article writing as an educational exercise for academic training (Tardy, 2010).

Table 1 compares our work against prior benchmarks for Wikipedia generation. Existing work has generally focused on evaluating the generation of shorter snippets (_e.g._, one paragraph), within a narrower scope (_e.g._, a specific domain or two), or when an explicit outline or reference documents are supplied. A notable example is WikiSum (Liu et al., 2018), which treats generating Wikipedia articles as a multi-document summarization problem, with respect to the reference documents.

Our setup emphasizes the capability of long-form grounded writing systems to research and curate content. Specifically, given a topic \(t\), the task is to find a set of references \({\cal R}\) and generate a full-length article \({\cal S}=s_{1}s_{2}...s_{n}\), where each sentence \(s_{i}\) cites a list of documents in \({\cal R}\).2

Footnote 2: In practice, \({\cal S}\) also includes organizational elements such as section and subsection titles, which do not require citations.

### The FreshWiki Dataset

Creating a new Wikipedia-like article demands not only fluent writing but also good research skills. As modern LLMs are generally trained on Wikipedia text, we mitigate data leakage by explicitly seeking out _recent_ Wikipedia articles that were created (or very heavily edited) after the training cutoff of the LLMs we test. Our process can be repeated at future dates when new LLMs emerge.

To apply our date criteria, we focus on the top 100 most-edited pages, based on edit counts, for each month from February 2022 to September 20233. To ensure high-quality references, we filter these articles to keep only those having B-class quality or above assessed by ORES4. We also exclude list articles5 and articles that have no subsections. While high-quality Wikipedia articles usually contain structured data (_e.g._, tables) and are multi-modal, we only consider the plain text component in constructing the dataset to simplify our task. More details of the dataset are in Appendix A.

Footnote 3: Obtained from [https://wikimedia.org/api/rest_v1metrics/edited-pages/top-by-edits/en.wikipedia/all-editor-types/content/](https://wikimedia.org/api/rest_v1metrics/edited-pages/top-by-edits/en.wikipedia/all-editor-types/content/)[year]/[month]/all-days

Footnote 4: [https://www.mediawiki.org/wiki/ORES](https://www.mediawiki.org/wiki/ORES)

### Outline Creation and Evaluation

A full-length article is hard to generate or evaluate (Xu et al., 2023; Krishna et al., 2023). When human educators teach students academic writing, they sometimes supervise students at the outline stage Eriksson and Makitalo, 2015) because an extensive outline indicates a comprehensive understanding of the topic and provides a solid foundation for writing the full-length article Dietz and Foley, 2019). Inspired by this, we decompose the generation of \({\cal S}\) into two stages. In the pre-writing stage, we require the system to create an outline \({\cal O}\), which is defined as a list of multi-level section headings6. In the writing stage, the system uses the topic \(t\), the references \({\cal R}\), and an outline \({\cal O}\) to produce the full-length article \({\cal S}\).

Footnote 6: Since language models process and produce sequences, we can linearize \({\cal O}\) by adding “#” to indicate section titles, “##’ to indicate subsection titles, etc.

To evaluate the outline coverage, we introduce two metrics: _heading soft recall_ and _heading entity recall_. These metrics compare the multi-level section headings of the human-written article, considered as ground truth, and those in \({\cal O}\). Recognizing that an exact match between elements in these two sets of headings is unnecessary, we calculate the _heading soft recall_(Franti and Mariescu-Istodor, 2023) using cosine similarity derived from Sentence-BERT (Reimers and Gurevych, 2019) embeddings of the headings (details in C.1). We also compute the _heading entity recall_ which is quantified as the percentage of named entities in human-written article headings covered by \({\cal O}\). We extract entities with FLAIR named entity recognition (NER) (Akbik et al., 2019).

## 3 Method

We present STORM to automate the pre-writing stage by researching a given topic via effective question asking (SS3.1, SS3.2) and creating an outline (SS3.3). The outline will be extended to a full-length article grounded on the collected references

\begin{table}
\begin{tabular}{l l l l l} \hline  & **Domain** & \multicolumn{1}{c}{**Scope**} & \multicolumn{1}{c}{**Given**} & \multicolumn{1}{c}{**Given**} \\  & & & **Outline?** & **Refs?** \\ \hline Balepur et al. (2023) & One & One para. & / & Yes \\ Qian et al. (2023) & All & One para. & / & No \\ Fan and Gardent (2022) & One & Full article & Yes & No \\ Liu et al. (2018) & All & One para. & / & Yes \\ Sauper and Barzilay (2009) & Two & Full article & No & No \\ \hline Ours & All & Full article & No & No \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of different Wikipedia generation setups in existing literature. Generating one paragraph does not need an article outline.

 

(SS3.4). Figure 2 gives an overview of STORM and we include the pseudo code in Appendix B.

### Perspective-Guided Question Asking

Rohman (1965) defines pre-writing as the stage of discovery in the writing process. In parallel with stakeholder theory in business Freeman et al. (2010), where diverse stakeholders prioritize varying facets of a company, individuals with distinct perspectives may concentrate on different aspects when researching the same topic and discover multifaceted information. Further, the specific perspectives can serve as prior knowledge, guiding individuals to ask more in-depth questions. For example, an event planner might ask about the "transportation arrangements" and "budget" for "the 2022 Winter Olympics opening ceremony", whereas a layperson might ask more general questions about the event's basic information (Figure 1 (A)).

Given the input topic \(t\), STORM discovers different perspectives by surveying existing articles from similar topics and uses these perspectives to control the question asking process. Specifically, STORM prompts an LLM to generate a list of related topics and subsequently extracts the tables of contents from their corresponding Wikipedia articles, if such articles can be obtained through Wikipedia API7 (Figure 2 1). These tables of contents are concatenated to create a context to prompt the LLM to identify \(N\) perspectives \({\cal P}=\{p_{1},...,p_{N}\}\) that can collectively contribute to a comprehensive article on \(t\) (Figure 2 2). To ensure that the basic information about \(t\) is also covered, we add \(p_{0}\) as "basic fact writer focusing on broadly covering the basic facts about the topic" into \({\cal P}\). Each perspective \(p\in{\cal P}\) will be utilized to guide the LLM in the process of question asking in parallel.

Footnote 7: [https://yppi.org/project/Wikipedia-API/](https://yppi.org/project/Wikipedia-API/)

### Simulating Conversations

The theory of questions and question asking Ram (1991) highlights that while answers to existing questions contribute to a more comprehensive understanding of a topic, they often simultaneously give rise to new questions. To kick off this dynamic process, STORM simulates a conversation between a Wikipedia writer and a topic expert. In the \(i\)-th round of the conversation, the LLM-powered Wikipedia writer generates a single question \(q_{i}\) based on the topic \(t\), its assigned perspective \(p\in{\cal P}\), and the conversation history \(\{q_{1},a_{1},...,q_{i-1},a_{i-1}\}\) where \(a_{j}\) denotes the simulated expert's answer. The conversation history enables the LLM to update its understanding of the topic and ask follow-up questions. In practice, we limit the conversation to at most \(M\) rounds.

To ensure that the conversation history provides factual information, we use trusted sources from the Internet to ground the answer \(a_{i}\) to each query \(q_{i}\). Since \(q_{i}\) can be complicated, we first prompt the LLM to break down \(q_{i}\) into a set of search queries (Figure 2 4) and the searched results will be evaluated using a rule-based filter according to

Figure 2: The overview of STORM that automates the pre-writing stage. Starting with a given topic, STORM identifies various perspectives on covering the topic by surveying related Wikipedia articles (1�⃝2). It then simulates conversations between a Wikipedia writer who asks questions guided by the given perspective and an expert grounded on trustworthy online sources (3⃝3⃝6). The final outline is curated based on the LLM’s intrinsic knowledge and the gathered conversations from different perspectives (7⃝8).

 

the Wikipedia guideline8 to exclude untrustworthy sources (Figure 2)5). Finally, the LLM synthesizes the trustworthy sources to generate the answer \(a_{i}\), and these sources will also be added to \(\mathcal{R}\) for full article generation (SS3.4).

Footnote 7: [https://en.wikipedia.org/wiki/Wikipedia](https://en.wikipedia.org/wiki/Wikipedia): Reliable_sources

### Creating the Article Outline

After thoroughly researching the topic through \(N+1\) simulated conversations, denoted as \(\{\mathcal{C}_{0},\mathcal{C}_{1},...,\mathcal{C}_{N}\}\), STORM creates an outline before the actual writing starts. To fully leverage the internal knowledge of LLMs, we first prompt the model to generate a draft outline \(\mathcal{O}_{\text{D}}\) given only the topic \(t\) (Figure 2)7). \(\mathcal{O}_{\text{D}}\) typically provides a general but organized framework. Subsequently, the LLM is prompted with the topic \(t\), the draft outline \(\mathcal{O}_{\text{D}}\), and the simulated conversations \(\{\mathcal{C}_{0},\mathcal{C}_{1},...,\mathcal{C}_{N}\}\) to refine the outline (Figure 2)8). This results in an improved outline \(\mathcal{O}\) which will be used for producing the full-length article.

Footnote 8: [https://en.wikipedia.org/wiki/Wikipedia](https://en.wikipedia.org/wiki/Wikipedia): Reliable_sources

### Writing the Full-Length Article

Building upon the references \(\mathcal{R}\) collected and the outline \(\mathcal{O}\) developed during the pre-writing stage, the full-length article can be composed section by section. Since it is usually impossible to fit the entire \(\mathcal{R}\) within the context window of the LLM, we use the section title and headings of its all-level subsections to retrieve relevant documents from \(\mathcal{R}\) based on semantic similarity calculated from Sentence-BERT embeddings. With the relevant information at hand, the LLM is then prompted to generate the section with citations. Once all sections are generated, they are concatenated to form the full-length article. Since the sections are generated in parallel, we prompt the LLM with the concatenated article to delete repeated information to improve coherence. Furthermore, in alignment with Wikipedia's stylistic norms, the LLM is also utilized to synthesize a summary of the entire article, forming the lead section at the beginning.

## 4 Experiments

### Article Selection

STORM is capable of researching complicated topics and writing long articles from detailed outlines. However, in this controlled experiment, we limit the final output to at most 4000 tokens (roughly 3000 words). For a meaningful comparison, we randomly select 100 samples from the FreshWiki dataset (see SS2.1) that have human-written articles not exceeding 3000 words.

### Automatic Metrics

As discussed in SS2.2, we evaluate the outline quality to assess the pre-writing stage by calculating the _heading soft recall_ and _heading entity recall_. A higher recall score signifies a more comprehensive outline relative to the human-written article.

To assess the full-length article quality, we adopt _ROUGE scores_Lin (2004) and compute the _entity recall_ in the article level based on FLAIR NER results. Moreover, based on Wikipedia criteria9, we evaluate the article from the aspects of (1) _Interest Level_, (2) _Coherence and Organization_, (3) _Relevance and Focus_, (4) _Coverage_, and (5) _Verifiability_. For aspects (1)-(4), we use Prometheus Kim et al. (2023), a 13B evaluator LLM to score the article based on a 5-point rubric collaboratively developed with two experienced Wikipedia editors (see Appendix C.2). For verifiability, we calculate the _citation recall_ and _citation precision_ based on the definition in Gao et al. (2023). We use Mistral 7B-Instruct Jiang et al. (2023) to examine whether the cited passages entail the generated sentence.

Footnote 9: [https://en.wikipedia.org/wiki/Wikipedia](https://en.wikipedia.org/wiki/Wikipedia): Good_article_criteria

### Baselines

As prior works use different setups and do not use LLMs, they are hard to compare directly. Instead, we use the following three LLM-based baselines.

1. [leftmargin=*]
2. _Direct Gen_, a baseline that directly prompts the LLM to generate an outline, which is then used to generate the full-length article.
3. _MAG_, a retrieval-augmented generation baseline that searches with the topic and uses the searched results together with the topic \(t\) to generate an outline or the entire article.
4. _Outline-driven RAG (oRAG)_, which is identical to _MAG_ in outline creation, but further searches additional information with section titles to generate the article section by section.

### Storm Implementation

We build STORM with zero-shot prompting using the DSPy framework Khattab et al. (2023). Appendix B includes the pseudo code and corresponding prompts. The hyperparameters \(N\) and \(M\) 

in STORM are both set as 5. We use the chat model gpt-3.5-turbo for question asking and use gpt-3.5-turbo-instruct for other parts of STORM. We also experiment with using gpt-4 for drafting and refining the outline (Figure 2) 7\(\_\)8). For reported results, the simulated topic expert in STORM is grounded on the You.com search API10, although the proposed pipeline is compatible with other search engines. The ground truth Wikipedia article is excluded from the search results.

Footnote 10: [https://documentation.you.com/api-reference/search](https://documentation.you.com/api-reference/search)

For final article generation, we only report the results using gpt-4 as gpt-3.5 is not faithful to sources when generating text with citations Gao et al. (2023). We set temperature as 1.0 and top_p as 0.9 for all experiments.

## 5 Results and Analysis

### Main Results

We use outline coverage as a proxy to assess the pre-writing stage (see SS2.2). Table 3 shows the heading soft recall and entity recall. Outlines directly generated by LLMs (_Direct Gen_) already demonstrate high heading soft recall, indicating LLMs' ability to grasp high-level aspects of a topic through their rich parametric knowledge. However, STORM, by asking effective questions to research the topic, can create higher recall outlines that cover more topic-specific aspects. Notably, although _RAG_ leverages additional information, presenting unorganized information in the context window makes outline generation more challenging for the weaker model, _i.e._, GPT-3.5, leading to worse performance. To test the limit of the _RAG_ baseline, we further expand the retrieved sources by starting with the outline produced by _RAG_, using its section titles as search queries to collect more sources, and inputting the newly collected sources together with the initial outline to LLM to generate a polished outline. This modified approach is referred to as "_MAG-expand_" in Table 3. The experiment results indicate that even though having an additional round of search and refinement can improve the outline produced by _RAG_, our proposed STORM still surpasses its performance.

We further evaluate the full-length article quality. As shown in Table 2, _oRAG_ significantly outperforms _RAG_, highlighting the effectiveness of using outlines for structuring full-length article generation. Despite this method's advantages in leveraging retrieval and outlining, our approach still outperforms it. The effective question asking mechanism enhances the articles with greater entity recall. The evaluator LLM also rates these articles with significantly higher scores in the aspects of "Interest Level", "Relevance and Focus", and "Coverage". Nonetheless, we acknowledge the possibility of the evaluator LLM overrating machine-generated text. Our careful human evaluation (SS6) reveals that STORM still has much room for improvement.

Although this work primarily focuses on the pre-writing stage and does not optimize generating text with citations, we still examine the citation quality of articles produced by our approach. As reported

\begin{table}
\begin{tabular}{l|c c c|c c c c} \hline  & \multicolumn{3}{c|}{**Demersion with Human-written Articles**} & \multicolumn{3}{c}{**Rubric Grading**} \\  & ROUGE-1 & ROUGE-L & Entity Recall & Interest Level & Organization & Relevance & Coverage \\ \hline Direct Gen & 25.62 & 12.63 & 5.08 & 2.87 & 4.60 & 3.10 & 4.16 \\ RAG & 28.52 & 13.18 & 7.57 & 3.14 & 4.22 & 3.05 & 4.08 \\ oRAG & 44.26 & 16.51 & 12.57 & 3.90 & 4.79 & 4.09 & 4.70 \\ \hline
**STORM** & **45.82** & **16.70** & **14.10†** & **3.99 

in Table 4, Mistral 7B-Instruct judges 84.83% of the sentences are supported by their citations. Appendix C.3 investigates the unsupported sentences and reveals that the primary issues stem from drawing improper inferences and inaccurate paraphrasing, rather than hallucinating non-existent contents.

### Ablation Studies

As introduced in SS3, STORM prompts LLMs to ask effective questions by discovering specific perspectives and simulating multi-turn conversations. We conduct the ablation study on outline creation by comparing STORM with two variants: (1) "STORM w/o Perspective", which omits perspective in the question generation prompt; (2) "STORM w/o Conversation", which prompts LLMs to generate a set number of questions altogether. To ensure a fair comparison, we control an equal total number of generated questions across all variants. Table 3 shows the ablation results and full STORM pipeline produces outlines with the highest recall. Also, "STORM w/o Conversation" gives much worse results, indicating reading relevant information is crucial to generating effective questions. We further examine how many unique sources are collected in \({\cal R}\) via different variants. As shown in Table 5, the full pipeline discovers more different sources and the trend is in accord with the automatic metrics for outline quality.

We also verify whether having an outline stage is necessary with STORM. In Table 2, "STORM w/o Outline Stage" denotes the results of generating the entire article given the topic and the simulated conversations. Removing the outline stage significantly deteriorates the performance across all metrics.

## 6 Human Evaluation

To better understand the strengths and weaknesses of STORM, we conduct human evaluation by collaborating with 10 experienced Wikipedia editors who have made at least 500 edits on Wikipedia and have more than 1 year of experience. We randomly sample 20 topics from our dataset and evaluate the articles generated by our method and _oRAG_, the best baseline according to the automatic evaluation. Each pair of articles is assigned to 2 editors.

We request editors to judge each article from the same five aspects defined in SS4.2, but using a 1 to 7 scale for more fine-grained evaluation. While our automatic evaluation uses citation quality as a proxy to evaluate _Verifiability_, we stick to the Wikipedia standard of "verifiable with no original research" in human evaluation. Besides rating the articles, editors are asked to provide open-ended feedback and pairwise preference. After the evaluation finishes, they are further requested to compare an article produced by our method, which they have just reviewed, with its human-written counterpart, and report their perceived usefulness of STORM using a 1-5 Likert scale. More human evaluation details are included in Appendix D. Table 6 presents the rating and pairwise comparison results.11

Footnote 11: For the 1-7 scale rating results on each criterion, we calculate the Kripendroff’s Alpha to measure the inter annotator agreement (IAA), and the results are as follows: _Interest Level_ (0.349), _Organization_ (0.221), _Relevance_ (0.256), _Coverage_ (0.346), _Verifiability_ (0.388).

**Articles produced by STORM exhibit greater breadth and depth than oRAG outputs.** In accord with the finding in SS5.1, editors judge articles produced by STORM as more interesting, organized, and having broader coverage compared to _oRAG_ outputs. Specifically, 25% more articles produced by STORM are considered organized (_Organization_ rating \(\geq 4\)), and 10% more are deemed to have good coverage (_Coverage_ rating \(\geq 4\)). Even in comparison with human-written articles, one editor praises our result as providing "a bit more

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{oRAG} & \multicolumn{2}{c}{STORM} \\  & Avg. & \(\geq 4\) Rates & Av.g. & \(\geq 4\) Rates & \(p\)-value \\ \hline Interest Level & 3.63 & 57.5\% & **4.03** & **70.0\%** & 0.077 \\ Organization & 3.25 & 45.0\% & **4.00** & **70.0\%** & 0.005 \\ Relevance & 3.93 & 62.5\% & **4.15** & **65.0\%** & 0.347 \\ Coverage & 3.58 & 57.5\% & **4.00** & **67.5\%** & 0.084 \\ Verifiability & **3.85** & 67.5\% & 3.80 & 67.5\% & 0.843 \\ \hline \#Preferred & \multicolumn{2}{c}{14} & \multicolumn{2}{c}{**26**} \\ \hline \hline \end{tabular}
\end{table}
Table 6: Human evaluation results on 20 pairs of articles generated by STORM and _oRAG_. Each pair of articles is evaluated by two Wikipedia editors. The ratings are given on a scale between 1 and 7, with values \(\geq 4\) indicating good quality (see Table 10). We conduct paired \(t\)-test and report the \(p\)-value.

\begin{table}
\begin{tabular}{l c c} \hline \hline  & **Citation Recall** & **Citation Precision** \\ \hline STORM & 84.83 & 85.18 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Citation quality judged by Mistral 7B-Instrect.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline  & \multicolumn{2}{c}{oRAG} & \multicolumn{2}{c}{STORM} \\  & Avg. & \(\geq 4\) Rates & Av.g. & \(\geq 4\) Rates \\ \hline Interest Level & 3.63 & 57.5\% & **4.03** & **70.0\%** & 0.077 \\ Organization & 3.25 & 45.0\% & **4.00** & **70.0\%** & 0.005 \\ Relevance & 3.93 & 62.5\% & **4.15** & **65.0\%** & 0.347 \\ Coverage & 3.58 & 57.5\% & **4.00** & **67.5\%** & 0.084 \\ Verifiability & **3.85** & 67.5\% & 3.80 & 67.5\% & 0.843 \\ \hline \#Preferred & \multicolumn{2}{c}{14} & \multicolumn{2}{c}{**26**} \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average number of unique references (\(|{\cal R}|\)) collected using different methods.

 

background information" and another notes that "I found that the AI articles had more depth compared to the Wikipedia articles". STORM also outperforms the best baseline in pairwise comparison.

**More information in \(|{\cal R}|\) poses challenges beyond factual hallucination.** We examine 14 pairwise comparison responses where editors prefer oRAG outputs over STORM. Excluding 3 cases where pairwise preferences do not align with their ratings, editors assign lower _Verifiability_ scores to articles from our approach in over 50% of the cases. Through analyzing the articles and editors' free-form feedback, we discover that _low Verifiability scores stem from red herring fallacy or overspeculation issues_. These arise when the generated articles introduce unverifiable connections between different pieces of information in \(|{\cal R}|\) or between the information and the topic (examples included in Table 11). Compared to the widely discussed factual hallucination Shuster et al. (2021); Huang et al. (2023), addressing such verifiability issues is more nuanced, surpassing basic fact-checking Min et al. (2023).

**Generated articles trail behind well-revised human works.** While STORM outperforms the oRAG baseline, editors comment that the generated articles are _less informative than actual Wikipedia pages_. Another major issue identified is _the transfer of bias and tone from Internet sources to the generated article_, with 7 out of 10 editors mentioning that the STORM-generated articles sound "emotional" or "unneutral". More analysis is discussed in Appendix E. This feedback suggests that reducing the retrieval bias in the pre-writing stage is a worthwhile direction for future work.

**Generated articles are a good starting point.** As shown in Figure 3, editors are unanimous in agreeing that STORM can aid them in their pre-writing stage. It is gratifying to know that the tool is helpful to experienced editors. 80% of the editors think that STORM can help them edit a Wikipedia article for a new topic. More reservation is expressed to the usefulness of STORM for the Wikipedia community at large; nonetheless, 70% of the editors think it is useful, with only 10% disagreeing.

## 7 Related Works

Retrieval-Augmented Generation (RAG)Augmenting language models (LMs) with retrieval at inference time is a typical way to leverage external knowledge stores Ram et al. (2023); Izacard et al. (2023). While some works use retrieval to construct demonstrations for in-context learning Li et al. (2023); Liu et al. (2022); Agrawal et al. (2023); Poesia et al. (2022); Shi et al. (2022); Khattab et al. (2022), another line of works uses retrieval to provide additional information for LMs to ground on. Lewis et al. (2020) study RAG on knowledge-intensive NLP tasks and find it improves diversity and factuality. Semnani et al. (2023) designs a RAG-based chatbot grounded on English Wikipedia to stop LLM-based chatbots from hallucination. Besides, RAG can be used to generate text with citations Menick et al. (2022); Gao et al. (2023) and build attributed question answering systems Bohnet et al. (2023). While RAG is widely studied in question answering, how to use it for long-form article generation is less investigated.

As a general framework, RAG is flexible in both the retrieval source and time. The retrieval sources can vary from domain databases Zakka et al. (2023), code documentation Zhou et al. (2023), to the whole Internet Nakano et al. (2022); Romeili et al. (2022). Regarding the time, besides a one-time retrieval before generation, the system can be designed to self-decide when to retrieve across the course of the generation Jiang et al. (2023); Parisi et al. (2022); Shuster et al. (2022); Yao et al. (2023).

Automatic Expository WritingDifferent from other types of long-form generation Yang et al. (2022); Feng et al. (2018), automatic expository writing requires grounding on external documents and leveraging the interplay between reading and writing. Balepur et al. (2023) propose the Imitate-Retrieve-Paraphrase framework for expository writing at the paragraph level to address the challenges in synthesizing information from multiple sources. Beyond summarizing sources, Shen et al. (2023) highlight that expository writing requires the author's sensemaking process over source documents and good outline planning. We tackle these challenges by focusing on the pre-writing stage.

Figure 3: Survey results of the perceived usefulness of STORM (\(n=10\)).

 

Question Asking in NLPQuestion asking capabilities in NLP systems have expanded across several fronts, including generating clarification questions to understand user intents Aliannejadi et al. (2019); Rahmani et al. (2023), and breaking large questions into smaller ones to improve compositional reasoning Press et al. (2023). While humans usually ask questions to learn new knowledge Tawfik et al. (2020); Booth et al. (2003), how to optimize question informativeness and specificity in information-seeking conversations remains less explored. The closest work is Qi et al. (2020) which defines the question informativeness using the unigram precision function and uses reinforcement learning to increase the question informativeness.

## 8 Conclusion

We propose STORM, an LLM-based writing system that automates the pre-writing stage for creating Wikipedia-like articles from scratch. We curate the FreshWiki dataset and establish evaluation criteria to study the generation of grounded long-form articles. Experimental results demonstrate that the question asking mechanism in STORM improves both the outline and article quality. With the improved breadth and depth, STORM helps surface new challenges for grounded writing systems through expert evaluation. The experienced Wikipedia editors in our study unanimously agree that STORM is helpful for their pre-writing stage.

### Limitations

In this work, we explore generating Wikipedia-like articles from scratch as a way to push the frontier of automatic expository writing and long-form article generation. While our approach significantly outperforms baseline methods in both automatic and human evaluations, the quality of machine-written articles still lags behind well-revised human-authored articles, specifically in aspects of neutrality and verifiability. Although STORM discovers different perspectives in researching the given topic, the collected information may still be biased towards dominant sources on the Internet and may contain promotional content. Moreover, the verifiability issues identified in this work go beyond factual hallucination, which highlights new challenges to grounded writing systems.

Another limitation of this work is that although we focus on the task of generating Wikipedia-like articles from scratch, our task setup is still simplified to only consider the generation of free-form text. Human-authored high-quality Wikipedia articles usually contain structured data and multimodal information. We leave the exploration of generating multi-modal grounded articles for future work.

## Acknowledgements

We thank You.com for generously providing the search API that supported our experiments. We also thank Sina J. Semnani, Shicheng Liu, Eric Zelikman for providing helpful feedback and the ACL ARR reviewers for their valuable comments. This work is supported in part by the Verdant Foundation and Microsoft Azure AI credits. Yijia Shao is supported by a Stanford School of Engineering Fellowship.

### Ethics Statement

Different from the creative generation, grounded article generation may impact how people learn about topics or consume source information. All the studies and the evaluation in this work are designed to prevent the dissemination of misinformation by not publishing generated content online and implementing strict accuracy checks. We avoid any disruption to Wikipedia or related communities, as our system does not interact with live pages. Also, although we try to generate grounded articles, we believe there is no privacy issue related to this work as we only use information publicly available on the Internet.

The primary risk of our work is that the Wikipedia articles written by our system are grounded on information on the Internet which contains some biased or discriminative content on its own. Currently, our system relies on the search engine to retrieve information but does not include any post-processing module. We believe improving the retrieval module to have good coverage of different viewpoints and adding a content sifting module to the current system will be a critical next step to achieve better neutrality and balance in the generated articles.

Another limitation we see from an ethical point of view is that we only consider writing English Wikipedia articles in this work. Extending the current system to a multilingual setup is a meaningful direction for future work as more topics do not have Wikipedia pages in non-English languages 



## References

* Agrawal et al. (2023) Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. Incontext examples selection for machine translation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 8857-8873, Toronto, Canada. Association for Computational Linguistics.
* Akbik et al. (2019) Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 54-59, Minneapolis, Minnesota. Association for Computational Linguistics.
* Aliannejadi et al. (2019) Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In _Proceedings of the 42nd international acm sigir conference on research and development in information retrieval_, pages 475-484.
* Balepur et al. (2023) Nishant Balepur, Jie Huang, and Kevin Chang. 2023. Expository text generation: Imitate, retrieve, paraphrase. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 11896-11919, Singapore. Association for Computational Linguistics.
* Banerjee and Mitra (2015) Siddhartha Banerjee and Prasenjit Mitra. 2015. WikiKreator: Improving Wikipedia stubs automatically. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 867-877, Beijing, China. Association for Computational Linguistics.
* Bohnet et al. (2023) Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Liemi Seostrain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2023. Attributed question answering: Evaluation and modeling for attributed large language models.
* Booth et al. (2003) Wayne C Booth, Gregory G Colomb, and Joseph M Williams. 2003. _The craft of research_. University of Chicago press.
* Dietz and Foley (2019) Laura Dietz and John Foley. 2019. Trec car y3: Complex answer retrieval overview. In _Proceedings of Text REtrieval Conference (TREC)_.
* Doyle (1994) Christina S Doyle. 1994. _Information literacy in an information society: A concept for the information age_. Diane Publishing.
* Eriksson and Makitalo (2015) Ann-Marie Eriksson and Asia Makitalo. 2015. Supervision at the outline stage: Introducing and encountering issues of sustainable development through academic writing assignments. _Text & Talk_, 35(2):123-153.
* Fan and Gardent (2022) Angela Fan and Claire Gardent. 2022. Generating biographies on Wikipedia: The impact of gender bias on the retrieval-based generation of women biographies. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8561-8576, Dublin, Ireland. Association for Computational Linguistics.
* Feng et al. (2018) Xiaocheng Feng, Ming Liu, Jiahao Liu, Bing Qin, Yibo Sun, and Ting Liu. 2018. Topic-to-essay generation with neural networks. In _IJCAI_, pages 4078-4084.
* Fitria (2023) Tira Nur Fitria. 2023. Artificial intelligence (ai) technology in openai chatgpt application: A review of chatgpt in writing english essay. In _ELT Forum: Journal of English Language Teaching_, volume 12, pages 44-58.
* Franti and Mariescu-Istodor (2023) Pasi Franti and Radu Mariescu-Istodor. 2023. Soft precision and recall. _Pattern Recognition Letters_, 167:115-121.
* Freeman et al. (2010) R Edward Freeman, Jeffrey S Harrison, Andrew C Wicks, Bidhan L Parmar, and Simone De Colle. 2010. Stakeholder theory: The state of the art.
* Gao et al. (2023) Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6465-6488, Singapore. Association for Computational Linguistics.
* Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiangling Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.
* Izacard et al. (2023) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. _Journal of Machine Learning Research_, 24(251):1-43.
* Jiang et al. (2023a) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral ?b. _arXiv preprint arXiv:2310.06825_.
* Jiang et al. (2023b) Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. Active retrieval augmented generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7969-7992, Singapore. Association for Computational Linguistics

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In _International Conference on Machine Learning_, pages 15696-15707. PMLR.
* Khattab et al. (2022) Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. _arXiv preprint arXiv:2212.14024_.
* Khattab et al. (2023) Omar Khattab, Arnav Singhvi, Pardihi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. Dspy: Compiling declarative language model calls into self-improving pipelines. _arXiv preprint arXiv:2310.03714_.
* Kim et al. (2023) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Huaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. _arXiv preprint arXiv:2310.08491_.
* Romeili et al. (2022) Mojtaba Romeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmented dialogue generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8460-8478, Dublin, Ireland. Association for Computational Linguistics.
* Krishna et al. (2023) Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 1650-1669, Dubrovnik, Croatia. Association for Computational Linguistics.
* Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Pitkus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kutler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Advances in Neural Information Processing Systems_, 33:9459-9474.
* Li et al. (2023) Xiaanan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023. Unified demonstration retriever for inconstext learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4644-4668, Toronto, Canada. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2022) Jiachao Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114, Dublin, Ireland and Online. Association for Computational Linguistics.
* Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In _International Conference on Learning Representations_.
* Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes.
* Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 12076-12100, Singapore. Association for Computational Linguistics.
* Minguillon et al. (2017) Julia Minguillon, Maura Lerga, Eduard Ajbar, Josep Llados-Masllorens, and Antoni Meseguer-Artola. 2017. Semi-automatic generation of a corpus of wikipedia articles on science and technology. _Professional de la Informacion_, 26(5):995-1005.
* Munoz-Luna (2015) Rosa Munoz-Luna. 2015. Main ingredients for success in l2 academic writing: Outlining, drafting and proofreading. _PloS one_, 10(6):e0128309.
* Nakano et al. (2022) Reitichiro Nakano, Jacob Hilton, Suchr Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Kruger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assisted question-answering with human feedback.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models.
* Pavlik (2023) John V Pavlik. 2023. Collaborating with chatgpt: Considering the implications of generative artificial intelligence for journalism and media education. _Journalism & Mass Communication Educator_, 78(1):84-93 

Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In _International Conference on Learning Representations_.
* Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 5687-5711, Singapore. Association for Computational Linguistics.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, and Christopher D. Manning. 2020. Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 25-40, Online. Association for Computational Linguistics.
* Qian et al. (2023) Hongjing Qian, Yutao Zhu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Webbrain: Learning to generate factually correct articles for queries by grounding on large web corpus.
* Rahmani et al. (2023) Hossein A. Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz, and Aldo Lipani. 2023. A survey on asking clarification questions datasets in conversational systems. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2698-2716, Toronto, Canada. Association for Computational Linguistics.
* Ram (1991) Ashwin Ram. 1991. A theory of questions and question asking. _Journal of the Learning Sciences_, 1(3-4):273-318.
* Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlagy, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. _Transactions of the Association for Computational Linguistics_.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Rohman (1965) D Gordon Rohman. 1965. Pre-writing the stage of discovery in the writing process. _College composition and communication_, 16(2):106-112.
* Sauper and Barzilay (2009) Christina Sauper and Regina Barzilay. 2009. Automatically generating Wikipedia articles: A structure-aware approach. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_, pages 208-216, Suntec, Singapore. Association for Computational Linguistics.
* Semennai et al. (2023) Sina Semennai, Violet Yao, Heidi Zhang, and Monica Lam. 2023. WikiChat: Stopping the hallucination of large language model chatbots by few-shot grounding on Wikipedia. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2387-2413, Singapore. Association for Computational Linguistics.
* Shen et al. (2023) Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo, Jonathan Bragg, Jeff Hammerbacher, Doug Downey, Joseph Chee Chang, and David Sontag. 2023. Beyond summarization: Designing ai support for real-world expository writing tasks.
* Shi et al. (2022) Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022. Nearest neighbor zero-shot inference. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3254-3265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Shuster et al. (2022) Kurt Shuster, Mojtaba Komelii, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, pages 373-393, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Tardy (2010) Christine M Tardy. 2010. Writing for the world: Wikipedia as an introduction to academic writing. In _English teaching forum_, volume 48, page 12. ERIC.
* Tawfik et al. (2020) Andrew A Tawfik, Arthur Graesser, Jessica Gatewood, and Jaclyn Gishbaugher. 2020. Role of questions in inquiry-based instruction: towards a design taxonomy for question-asking and implications for design. _Educational Technology Research and Development_, 68:653-678.
* Waver III and Kintsch (1991) Charles A Weaver III and Walter Kintsch. 1991. Expository text.
* Wenzlaff and Spaeth (2022) Karsten Wenzlaff and Sebastian Spaeth. 2022. Smarter than humans? validating how openai's chatgpt model explains crowdfundling, alternative finance and community finance. _Validating how OpenAI's ChatGPT model explains Crowdfundling, Alternative Finance and Community Finance.(December 22, 2022)_.
* Xu et al. (2023) Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3225-3245, Toronto, Canada. Association for Computational Linguistics 

Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023. DOC: Improving long story coherence with detailed outline control. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3378-3465, Toronto, Canada. Association for Computational Linguistics.
* Yang et al. (2022) Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive re prompting and revision. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 4393-4479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Yao et al. (2023) Shunyau Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
* Zakka et al. (2023) Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R Dalal, Jennifer L Kim, Michael Moor, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, et al. 2023. Almanac: Retrieval-augmented language models for clinical medicine. _Research Square_.
* Zhou et al. (2023) Shuyan Zhou, Uri Alon, Frank F. Xu, Zhengbao Jiang, and Graham Neubig. 2023. Docprompting: Generating code by retrieving the docs. In _The Eleventh International Conference on Learning Representations_ 



## Appendix A Dataset Details

As discussed in SS2.1, we curate the FreshWiki dataset by collecting _recent_ and _high-quality_ English Wikipedia articles. We select the most-edited pages over a specific period rather than using creation dates as a cutoff because most of Wikipedia articles are "stubs" or are of low quality when they were created. For quality, we consider articles predicted to be of B-class quality or above. According to Wikipedia statistics12, only around 3% of existing Wikipedia pages meet this quality standard. As LLMs can generate reasonably good outputs, we think it is important to use high-quality human-written articles as references for further research.

Footnote 12: [https://en.wikipedia.org/wiki/Wikipedia:Content_assessment](https://en.wikipedia.org/wiki/Wikipedia:Content_assessment)

For experiments in this work, we randomly select 100 samples with human-written articles under 3000 words to have a meaningful comparison. Table 7 gives the data statistics. Notably, human-authored articles have a large number of references but they require numerous edits to achieve this. Figure 4 illustrates the evolution of the reference count in the article edit process and Figure 5 gives the distribution of edit counts for human-authored articles used in our experiments.

## Appendix B Pseudo Code of STORM

In SS3, we introduce STORM, a framework that automates the pre-writing stage by discovering different perspectives, simulating information-seeking conversations, and creating a comprehensive outline. Algorithm 1 displays the skeleton of STORM.

We implement STORM with zero-shot prompting using the DSPy framework (Khattab et al., 2023). Listing 1 and 2 show the prompts used in our implementation. We highlight that STORM offers a general framework designed to assist the creation of grounded, long-form articles, without depending extensively on prompt engineering for a single domain.

## Appendix C Automatic Evaluation Details

### Soft Heading Recall

We calculate the soft heading recall between the multi-level headings in the generated outline, considered as the prediction \(P\), and those in the human-written article, considered as the ground truth \(G\). The calculation is based on the soft recall definition in Franti and Mariescu-Istodor (2023). Given a set \(A=\{Ai\}_{i=1}^{K}\), _soft count_ of an item is defined as the inverse of the sum of its similarity to other items in the set:

\[\begin{split}&\mathrm{count}\left(A_{i}\right)=\frac{1}{\sum_{j=1}^{ K}\operatorname*{Sim}\left(A_{i},A_{j}\right)}\\ &\operatorname*{Sim}\left(A_{i},A_{j}\right)=\cos\left(\mathrm{ embed}(A_{i}),\mathrm{embed}(A_{j})\right),\end{split} \tag{1}\]

where \(\mathrm{embed}(\cdot)\) in Equation (1) is parameterized by paraphrase-MiniLM-L6-v2 provided in the Sentence-Transformers library13. The cardinality

Figure 4: Evolution of reference count in the Wikipedia article editing process.

\begin{table}
\begin{tabular}{l c} \hline \hline Average Number of Sections & 8.4 \\ Average Number of All-level Headings & 15.8 \\ \hline Average Length of a Section & 327.8 \\ Average Length of Total Article & 2159.1 \\ \hline Average Number of References & 90.1 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Statistics of the dataset used in our experiments.

Figure 5: Distribution of edit counts for Wikipedia articles in our experiments (\(n=100\)).



classGenRelatedTopicsPrompt(dspy.Signature):
2"""
3I'mwritingaWikipediapageforatopicmentionedbelow.PleaseidentifydrecommendsomeWikipediapagesoncloselyrelatedsubjects.I'mlookingforexamplesthatprovidinginsightsintinterestingaspectscommonlyassociatedwiththistopic,orexamplesthathelpmeunderstandthetypicalcontentandstructureincludedinWikipediagessforsimilartopics.4Pleaselistetheurlsinseparatelines.5"""
6
7topic=dspy.InputField(prefix="Topicofinterest:",format=str)related_topics=dspy.OutputField()
9classGenPerspectivesPrompt(dspy.Signature):
11"""
12Youneedtoselectagraphgroupoffikipedieditorswhowillworktogethertocreateacomprehensivearticleonthetopic.Eachofthemrepresentsadifferentperspective,role,oraffiliationrelatedtothistopic.YourcanuseotherWikipediapagesofrelatedtopicsforinspiration.Foreacheditor,adddescriptionofwhattheywillfocuson.
13Giveyouranswerinthefollowingformat:1.shortsummaryofeditor1:description\n2.shortsummaryofeditor2:description\n...
14__##
15
16topic=dspy.InputField(prefix='Topicofinterest:',format=str)examples=dspy.InputField(prefix='Wiki pageoutlinesofrelatedtopicsforinspiration:\n',format=str) perspectives=dspy.OutputField()
17
18classGen0nprompt(dspy.Signature):
19"""
20YouareanexperiencedWikipediawriterandwanttoeditapecificepage.Besidesyouridentityasawikipediawriter,youhaveascpecificfocuswhenresearchingetopic.Now,yoarechatingwithanevertotgetinformation.Askgoodquestionstotgetermusefulinformation.Whenyohavecommorequestionotsask,say"Thankyoosomuchforyourhelp!"toendtheconversation.
21Pleaseonlyaskonequestionatatimemanddon'taskwhatyowhavesakedbefore.Yourquestionsshouldberelatedtothemoticjobewnanttowrite.
22"""
23topic=dspy.InputField(prefix='Topicyow wanttowrite:',format=str)personna<dspy.InputField(prefix='Yourspecificperspective:',format=str)conv=dspy.InputField(prefix='Conversationhistory:\n',format=str)question=dspy.OutputField()
24
25classGenQueriesPrompt(dspy.Signature):
26"""
27YouwanttoanswerthequestionusingGooglesearch.Whatdoyoutypeinthesearchbox?Writethequeriesyouwillusedinthefollowingformat:-query{}$n-query{}$ 

classGenAnswerPrompt(dspy.Signature):
2"""
3Youareanerwhocanuseinformationeffectively.YouarechattingwithaWikipediawriterwhowantstowriteaWikipediapageontopicyouknow.Youhavegatheredtherelatedinformationandwillnowusetheinformationtoformaresponse.4Makeyourresponseasinformativeasspossibleandmakesureeverysentenceissupportedbythegatheredinformation.5"""
6
7topic=dspy.InputField(prefix='Topicyouarediscussingabout:',format=strr)conv=dspy.InputField(prefix='Question:\n'',format=str)info=dspy.InputField(prefix='Gatheredinformation:\n'',format=str)answer=dspy.OutputField(prefix='Nowgiveyourresponse:\n'')
8classDirectGenOutlinePrompt(dspy.Signature):
9"""
10WritenoutlineforaWikipediapage.Hereistheformatofyourwriting:
11Use"#Title"toindicatesecentitle,"##Tile"toindicatesubsectiontitle,andsoon.
12Doncontincludeotherinformation.
13"""
14topic=dspy.InputField(prefix="Topicyouwantotwrite:",format=str)outline=dspy.OutputField(prefix="WritetheWikipediapageoutline:\n'')
15
16classRefineOutlinePrompt(dspy.Signature):
17"""
18ImproveanumoutlineforaWikipediapage.Youalreadyhaveadraftoutlinethatcoversthengeneralinformation.Nowyouswanttoimproveitbasedonthemformationlearnedfromanimformation-seekingconversationtomakeitmorecomprehensive.
19Hereistheformatofyourwriting:
201Use"#"Title"toindicatesecentile,"##"Title"toindicatesubsectiontitle,"###"Title"toindicatesubsubsectiontitle,andsoon.
21Donotincludethoderinformation.
22"""
23topic=dspy.InputField(prefix="Topicyouwanttowrite:",format=strr) conv=dspy.InputField(prefix='Conversationhistory:\n'',format=str) old_outline=dspy.OutputField(prefix='Currentoutline:\n'',format=str) outline=dspy.OutputField( prefix='Writerbigeoutline:\n'') ```

Listing 2: Prompts used in STORM (continue), corresponding to Line 24, 31, 32 in Algorithm 1.

 

```
Input : Topic \(t\), maximum perspective \(N\), maximum conversation round \(M\) Output : Outline \({\cal O}\), references \({\cal R}\)
1P0 = "basic fact writer ... "// Constant.
2\({\cal R}\leftarrow[~{}]\)
3// Discover perspectives \({\cal P}\).
4related_topics \(\leftarrow\) gen_related_topics(\(t\)) tocs \(\leftarrow[~{}]\)foreach related_t in related_topics do
5article \(\leftarrow\) get_wiki_article(related_t) if article then
6tocs.append(extract_toc(article))
7 end for
8
9end
10 end while
11\({\cal P}\leftarrow\) gen_perspectives(\(t\), tocs) \({\cal P}\leftarrow[\text{P0}]+{\cal P}[:N]\)
12// Simulate conversations. convos \(\leftarrow[~{}]\) foreach\(p\)in \({\cal P}\)do
13convo_history \(\leftarrow[~{}]\)for\(i\) = 1 to \(M\)do
14// Question asking.
15\(q\leftarrow\) gen_on(\(t\), \(p\), dlg_history) convo_history.append(\(q\)) // Question answering. queries \(\leftarrow\) gen_queries(\(t\), q) sources \(\leftarrow\) search_and_sit(queries) \(a\leftarrow\) gen_ans(\(t\), q, sources) convo_history.append(\(a\)) \({\cal R}\).append(sources)
16 end for convos.append(convo_history)
17
18 end for
19// Create the outline.
20\({\cal D}_{\rm D}\leftarrow\) direct_gen_outline(\(t\)) \({\cal O}\leftarrow\) refine_outline(\(t\), \({\cal O}_{\rm D}\), convos) return\({\cal O}\), \({\cal R}\)
```

**Algorithm 1****STORM**

The following procedure is based on the following two steps:

* The first step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The second step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The third step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The fourth step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The fifth step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The sixth step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.
* The seventh step is to find a set of \(n\)-dimensional vectors from each row and compute the corresponding vector.

 



\begin{table}
\begin{tabular}{l l p{284.5pt}} \hline Criteria Description & **Interest Level:** How engaging and thought-provoking is the article? \\ Score 1 Description & Not engaging at all; no attempt to capture the reader’s attention. \\ Score 2 Description & Fairly engaging with a basic narrative but lacking depth. \\ Score 3 Description & Moderately engaging with several interesting points. \\ Score 4 Description & Quite engaging with a well-structured narrative and noteworthy points that frequently capture and retain attention. \\ Score 5 Description & Exceptionally engaging throughout, with a compelling narrative that consistently stimulates interest. \\ \hline Criteria Description & **Coherence and Organization:** Is the article well-organized and logically structured? \\ Score 1 Description & Disorganized; lacks logical structure and coherence. \\ Score 2 Description & Fairly organized; a basic structure is present but not consistently followed. \\ Score 3 Description & Organized; a clear structure is mostly followed with some lapless in coherence. \\ Score 4 Description & Good organization; a clear structure with minor lapless in coherence. \\ Score 5 Description & Excellently organized; the article is logically structured with seamless transitions and a clear argument. \\ \hline Criteria Description & **Relevance and Focus**: Does the article stay on topic and maintain a clear focus? \\ Score 1 Description & Off-topic; the content does not align with the headline or core subject. \\ Score 2 Description & Somewhat on topic but with several digressions; the core subject is evident but not consistently adhered to. \\ Score 3 Description & Generally on topic, despite a few unrelated details. \\ Score 4 Description & Mostly on topic and focused; the narrative has a consistent relevance to the core subject with infrequent digressions. \\ Score 5 Description & Exceptionally focused and entirely on topic; the article is tightly centered on the subject, with every piece of information contributing to a comprehensive understanding of the topic. \\ \hline Criteria Description & **Broad Coverage:** Does the article provide an in-depth exploration of the topic and have good coverage? \\ Score 1 Description & Severely lacking; offers little to no coverage of the topic’s primary aspects, resulting in a very narrow perspective. \\ Score 2 Description & Partial coverage; includes some of the topic’s main aspects but misses others, resulting in an incomplete portrayal. \\ Score 3 Description & Acceptable breadth; covers most main aspects, though it may stray into minor unnecessary details or overlook some relevant points. \\ Score 4 Description & Good coverage; achieves broad coverage of the topic, hitting on all major points with minimal extraneous information. \\ Score 5 Description & Exemplary in breadth; delivers outstanding coverage, thoroughly detailing all crucial aspects of the topic without including irrelevant information. \\ \hline \end{tabular}
\end{table}
Table 8: Scoring rubrics on a 1-5 scale for the evaluator LLM.

\begin{table}
\begin{tabular}{p{76.9pt} p{108.4pt} p{108.4pt} p{108.4pt}} \hline \hline
**Error Type** & **Topic** & **Unsupported Sentence** & **Source** \\ \hline Improper Inferential Linking & Lhahina, Hawaii & Throughout its history, religion has remained the paramount aspect of Hawaiian life in Lhahina, _(The source diseases religious as part of Hawaiian life under new mention Landau)_. \\  & & Operating everyday daily activity and significant event[5]. \\ \hline Inaccurate Paraphrasing & 2022 Crimean Bridge explosion & Completed in June 2020, the bridge serves as a major supply route for Russian forces in the region and is significant to Russia’s claim over the disrupted territory[21][11]. \\ Citing Irrelevant Sources & LR-99 & For example, comparisons have been drawn between the performance of IL-9 and the dynamic resolution capabilities of video games such as Battlefield 2042[22]. & [22] Battlefield 2042 PC performance guide: The best settings for high frame rate* (_The source is irrelevant to LE_892)_ \\ \hline \hline \end{tabular}
\end{table}
Table 9: Examples of different error types of unsupported sentences.



We use Mistral 7B-Instruct15(Jiang et al., 2023a) to examine whether the cited passages entail the generated sentence. Table 4 reports the citation quality of articles produced by our approach, showing that around 15% sentences in generated articles are unsupported by citations. We further investigate the failure cases by randomly sampling 10 articles and an author manually examines all the unsupported sentences in these articles. Besides sentences that are incorrectly split16, lack citations, or are deemed supported by the author's judgment, our analysis identifies three main error categories (examples are given in Table 9): _improper inferential linking_, _inaccurate paraphrasing_, and _citing irrelevant sources_.

Footnote 15: [https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)

Footnote 16: Following Gao et al. (2023), we check citation quality in the sentence level and split articles into sentences using NLTK sent_tokenize. sent_tokenize sometimes fails to split sentences correctly when the article contains special words like “No.12847”, “Bhatia et al.”, _etc_.

We show the error distribution in Figure 6. Notably, the most common errors stem from the tendency of LLMs to form improper inferential links between different pieces of information presented in the context window. Our analysis of citation quality suggests that, in addition to avoiding hallucinations, future research in grounded text generation should also focus on preventing LLMs from making overly inferential leaps based on the provided information.

## Appendix D Human Evaluation Details

We recruited 10 experienced Wikipedia editors to participate in our study by creating a research page on Meta-Wiki17 and reaching out to active editors who have recently approved articles for Wikipedia.18 Our participation group includes 3 editors with 1-5 years of experience, 4 with 6-10 years, and 3 with over 15 years of contribution. The study was approved by the Institutional Review Board of our institution and the participants signed the consent form through Qualtrics questionnaires before the study started.

Footnote 17: [https://meta.wikimedia.org](https://meta.wikimedia.org)

To streamline the evaluation of grounded articles, we developed a web application, which features a side-by-side display of the article and its citation snippets, to gather ratings and open-ended feedback for each article. Figure 7 shows the screenshot of our web application and the full article produced by STORM is included in Table 12. For human evaluation, we use a 1 to 7 scale for more fine-grained evaluation. The grading rubric is included in Table 10.

We collected the pairwise preferences and the perceived usefulness of STORM via an online questionnaire. Specifically, for the perceived usefulness, we request editors to rate their agreement with statements "I think it can be specifically helpful for my pre-writing stage (_e.g._, collecting relevant sources, outlining, drafting).", "I think it will help me edit a Wikipedia article for a new topic", "I think it can be a potentially useful tool for the Wikipedia community" on a Likert scale of 1-5, corresponding to _Strongly disagree_, _Somewhat disagree_, _Neither agree nor disagree_, _Somewhat agree_, _Strongly agree_.

## Appendix E Error Analysis

While articles produced by STORM are preferred by both automatic metrics and human evaluation, experienced editors still identified multiple problems with the machine-generated articles. We analyze the free-form comments and summarize the major issues in Table 11.

The primary issue raised is that the generated articles often contain emotional language and lack neutrality, primarily due to the source material. STORM currently retrieves grounding sources from the Internet which is not neutral and contains considerable promotional content on its own. Addressing this bias in the pre-writing stage represents a valuable direction for future research. Another major issue is the red herring fallacy or the over-association of unrelated facts. Addressing this challenge calls for high-level sensemaking rather than mere fact-level verification.

 



**Reflevance and Focus**

1: Off-topic; the content does not align with the headline or core subject.
2: Mostly off-topic with some relevant points.
3: Somewhat on topic but with several digressions; the core subject is evident but not consistently adhered to.
4: Generally on topic, despite a few unrelated details.
5: Mostly on topic and focused; the narrative has a consistent relevance to the core subject with infrequent digesressions.
6: Highly relevant with a focused narrative and purpose.
7: Exceptionally focused and entirely on topic; the article is tightly centered on the subject, with every piece of information contributing to a comprehensive understanding of the topic.

\begin{table} \end{table}
Table 10: Scoring rubrics on a 1-7 scale for human evaluation.





\begin{table}
\begin{tabular}{c c l} \hline
**Issue** & **Mentioned Time** & **Example Comments** \\ \hline Use of emotional words, unneutral & 12 & The word “significant” is used 17 times in this article. Vague and unsupported claims are made about broader political importance and “pivotal role[s]”, and is unencyclopedic. (comment on article _Lahaina, Hawaii_) \\ \cline{3-4}  & & [...] but they still have not fixed the issue of neutral point of view. It is also evident in this article that the writer’s standpoint is biased towards Taylor Swift. Other than that, it did a good job at summarizing key points and putting depth into this. (comment on article _Speak Now (Taylor’s Version)_) \\ \cline{3-4}  & & “The film was also featured in an art and film festival hosted by The California Endowment, highlighting the power of stories in reshaping narratives about communities.” Yes, technically the source says that, but it’s a stretch to say in Wikipedia voice and just sounds like non-neutral, promotional prosec. (comment on article _Gehraviyan_) \\ \hline Red herring fallacy, associating unrelated sources & 11 & Polling from America shouldn’t be included and links to climate change shouldn’t be made unless explicitly connected by the source. (comment on article _Typhoon Hinammor_) \\ \cline{3-4}  & & Sourcing seems mostly fine, though some aren’t directly related (Ex. 39,40). \\ \cline{3-4}  & & (moment on article _Gehravian_) \\ \cline{3-4}  & & Here is a lengthy digression about KISS, not necessary because the article on the band should be linked to. (comment _on article 2022 AFL Grand Final_) \\ \hline Missing important information & 6 & “One study, conducted by Sinead Griffin, a physicist at the Lawrence Berkeley National Laboratory, provided some analysis of LK-99’s abilities using supercomputer simulations[20].” This is not enough information about the analysis, which would have been very useful in the article. (common on article _LAF-99_) \\ \cline{3-4}  & & Although the earthquake’s immediate aftermath and response are adequately covered, there could be more about the long-term socioeconomic impact and recovery processes. (comment on article _2022 West Java earthquake_) \\ \hline Improper handling of time-sensitive information & 5 & Words like “now” should be avoided in Wikipedia articles to prevent them from becoming dated and phrases such as, “as of December 2023” should be used instead. (comment on article _Cyclone Batsirai_) \\ \cline{3-4}  & & “as of December 13” doesn’t specify a year, and is old information (comment on article _2022 West Java earthquake_) \\ \hline Section organization problem & 5 & too many subsections in the “Recovery and Rehabilitation” section (comment on article _2022 West Java earthquake_) \\ \cline{3-4}  & & I do not like how the article is organized, with too many headers cluttertering the article, making it not as readable. Other than that, the AI did great work on the piece. (comment on article _2022 Crimean Bridge explosion_) \\ \hline \end{tabular}
\end{table}
Table 11: Summary of major issues found on articles produced by STORM.



* [16] M. C. Bader, J. A. Krizhevsky, and D. V. Kingma, "Adam: a method for stochastic optimization with applications to machine learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pp. 235-248, 2017.
* [17] S. G. Hinton, R. H. Friedman, and I. Goodfellow, "Learning from tiny data sets using deep neural networks," _arXiv preprint arXiv:1902.03092_, 2019.
* [18] T. L. Nguyen, P. N. Torr, and Y. Bengio, "Deep residual learning for image recognition," _IEEE Transactions on Image Processing_, vol. 12, no. 1, pp. 1-10, Jan 2016.
* [19] T. L. Nguyen, P. N. Torr, and Y. Bengio, "A fast approach to minimizing deep belief nets," _Journal of Machine Learning Research_, vol. 12, no. 1, pp. 1-10, Jan 2016.

[MISSING_PAGE_POST]

 

Taylor Hawkins

Oliver Taylor Hawkins (1972-2022) was an American musician, best known as the drummer for the rock band Foo Fighters[1]. Born in Fort Walton, Texas, Hawkins' love for music was ignited at a young age, particularly after watching a Queen concert in 1982[2]I[3]I[5]. He kick-started his professional career as the drummer for Alanis Morissette's band during her world tour for the hit album 'Jagged Little Piill'[8]S[9]. His talents were recognized by Dave Grohl, who invited him to join the Foo Fighters in 1997, marking the beginning of his impactful tenure with the band[7]I[8].

Hawkins was celebrated for his versatile drumming style, drawing influence from renowned drummers like Roger Taylor, Neil Peart, Phil Collins, Alex Van Halen, and Stewart Copeland[14]. His performances, marked by a unique energy and aggressive style of play, earned him recognition as one of the top rock drummers of his era[15]. Apart from his role in the Foo Fighters, Hawkins' passion for music saw him involved in numerous side projects and collaborations, cementing his place in the world of rock music[10].

Outside of his professional life, Hawkins was known for his resilience and dedication to his family. Despite personal struggles, including a near-fatal drug overdose in 2001, Hawkins remained committed to his musical career[4]I[9]. His legacy continues to inspire musicians and fans alike, as his contributions to rock music, coupled with his indomitable spirit, made him an unforgettable icon in the music industry[13].

Hawkins' sudden death in 2022 while on tour in Bogota, Colombia, sent shockwaves through the music worId[34]. Tributes poured in from around the globe, reflecting the respect and admiration Hawkins had garnered during his lifetime[21]I[31]. His life and career were honored at a star-studded tribute concert in Los Angeles, attesting to the enduring impact of his music[22].
 

Oliver Taylor Hawkins, known as Taylor Hawkins, was born and raised in Fort Walton, Texas[3]. His family moved to Laguna Beach, California when he was four years old[3]. He has two younger siblings, a brother named Jason, and a sister named Heather[3]. As a child, Hawkins was particularly influenced by his paternal grandmother, Josie Hawkins, who had grown up during the Great Depression and lived in Jackson, Mississippi[1].

During his high school days at Laguna Beach High School, from where he graduated in 1990, he became friends with Jon Davison, who later became the lead vocalist of the band Yes[2][3]. His interest in music was nurtured from an early age, particularly after watching a Queen concert in 1982 which inspired him to learn to play the drums[2][5]. He noted that music was a constant presence in his family home[5].

Despite facing certain hardships during his upbringing, including his mother's struggles with "demons", Hawkins pursued his musical ambitions[4]. He credits his older sister Heather for taking care of the family during difficult times[4].

His first major musical experience came from playing drums for Alanis Morissette's album, Jagged Little Pill, and accompanying her on the subsequent tour[3]. This marked the beginning of his professional career in the music industry.

# Career
Taylor Hawkins began his professional music career playing in Alanis Morissette's band during her 18-month world tour in support of the hit album 'Jagged Little Pill' from 1995 to 1997[8][9]. His performances not only in the tour but also in the music videos for "You Oughta Know", "All I Really Want" and "You Learn" introduced him to the world of rock music and ultimately led to his meeting with Dave Grohl[8]. Throughout this time, Hawkins contributed significantly to the band's sound and performance, transforming the songs from their original drum loop format to a rock-band vibe that resonated with audiences[1][7].

In 1997, Hankins was asked by Grohl to join the Foo Fighters, an invitation that he readily accepted[7][8]. At the time, Grohl thought it was a long shot to recruit Hawkins given that Morissette was at the height of her career, but Hawkins' desire to be a part of a rock band compelled him to make the move[7]. This marked the beginning of Hawkins' tenure as the drummer of the Foo Fighters, a role that he would play until his passing[6][9].

Apart from his work with Morissette and the Foo Fighters, Hawkins had an array of other musical experiences[10]. He drummed for Sass Jordan before joining Morrissette's touring band[10]. He was part of an ad hoc drum supergroup called SOS Allstars and filled the void for Coheed and Cambria's 2007 album after their drummer Josh Eppard left the group[10]. In addition, Hawkins formed his own side project, the Coattail Riders, in 2005, through which he recorded his own music and took the project on the road, performing in small clubs despite the Foo Fighters' arena-status[7]. His son, Shane Hawkins, has since taken on his father's legacy, joining the Foo Fighters for a performance during the Boston Calling Music Festival in 2023[6].

# Musical Style and Influences

Taylor Hawkins was a profound drummer, with his musical style and influences spreading across a wide array of rock genres[11]. Known for his passionate random of groups that came before him, Hawkins regularly expressed his admiration for bands like Rush, Genesis, and the Police, all of which featured some of the greatest drummers in rock history like Neil Peart, Phil Collins, and Stewart Copeland[11].

He was heavily influenced by his love for classic rock, as evidenced by his performances, where he covered songs from bands like Van Halen[11].

 

Hawkins drew influences from a variety of drumming styles, developing a signature style inspired by greats like Roger Taylor, Neil Peart, Phil Collins, Alex Van Halen, and Stewart Copeland[14]. This distinctive style and influence extended to his drum kit, which incorporated elements like rototoms and concert toms[14].

Beyond his influences, Hawkins had a unique energy that made him stand out as a drummer. His performances were recognized for their power, and he was known for his enthusiastic and aggressive style of play[15]. This earned him recognition as one of the top rock drummers of his time, with his passion for music living on through his performances[14].

Through his career, Hawkins left an indelible mark on rock music, through his distinct style, passion, and contributions to the music industry[13]. His love for music and dedication to his craft made him an unforgettable icon in the world of rock music[13].

Personal Life Taylor Hawking married Alison Hawkins, an American celebrity and entrepreneur, in 2005[18]. The couple had three children, Oliver, Annabelle, and Everleigh[19]. Hawkins' commitment to his family was evident; in fact, he even wrote a song for his middle child, Annabelle[9].

In his personal life, Hawkins had also struggled with drug use, which nearly claimed his life in a 2001 overdose[9][7][4]. However, he managed to overcome this challenge, and later expressed gratitude for the experience as a lesson that allowed him to realize the destructive path he was on[7].

Outside of his main role in the Foo Fighters, Hawkins also pursued various side projects including the Birds of Satan, NHC, and Chevy Metal. His motivation for such ventures was a constant drive to create and his love for music[7]. Hawkins was also known for his unabashed fanboy nature, often vocalizing his admiration for fellow musicians and his heroes[7].

Legacy and Impact Taylor Hawkins was known for his raw and authentic drumming style, described as "courageous, damaged and unflinchingly authentic"[20]. His work with the Foo Fighters, as well as his various collaborations and side projects, made him a celebrated figure in rock 'n' roll[10].

Hawkins' death in 2022 was met with heartfelt tributes from colleagues and fans around the world. Notable tritbates came from rock legends like Roger Taylor of Queen, who considered Hawkins as a kind, brilliant man and an inspirational mentor, likening his death to "losing a younger favourite brother"[21]. Similarly, Led Zeppelin's Jimmy Page admired his technique, energy and spirited enthusiasm[21].

An LA tribute concert held in his honor included guest drummers like Lars Ulrich of Metallica, Travis Barker of blink-182, and Brad Wilk of Rage Against the Machine. Singers like Miley Cyrus and Alanis Morissette also performed at the concert[22].

Apart from his music, Taylor Hawkins also contributed to charities Music Support and MusiCares, both of which were chosen by the Hawkins family[23]. He had received numerous accolades throughout his career, including 27 Grammy nominations, of which he won 14[2]. In 2021, the Foo Fighters were inducted into the Rock and Roll Hall of Fame[9].

Discography Taylor Hawkins also led a notable music career through his own side projects and collaborations[10]. Aside from his work with the Foo Fighters, Hawkins formed and fronted the band Taylor Hawkins & The Coattail riders, a project which originated from jamming sessions with his friend Drew Hester[0].
 

Taylor Hawkins & The Coattail Riders, a band formed in 2004, have released three albums and their music spans genres including Hard Rock, Art Rock, and Alternative Rock[24][25][26]. The band grew from an initial casual jamming session, gradually evolving into a more formal arrangement that led to the production of record albums. Notably, these albums featured guest appearances by renowned musicians such as Dave Grohl, Queen's Brian May and Roger Taylor, The Cars' Elliot Easton, Perry Farrell, and Jon Davison, who is a school friend of Hawkins'[10].

### Red Light Fever

Red Light Fever, released on April 19, 2010, was the band's first album[29][38]. Prior to its release, Hawkins revealed in an interview that the album had completed the recording and production stages, but its title and release date were yet to be determined[29]. Red Light Fever was recorded at the Foo Fighters' Studio 606 in California and featured guest musicians such as Brian May and Roger Taylor of Queen, Dave Grohl of Foo Fighters, and Elliot Easton of The Cars[29][38].

## Get the Money

Get the Money, the third album from Taylor Hawkins & The Coattail Riders, was released on November 8, 2019[29]. The album's first single, "Crossed the Line", released on October 15, 2019, featured Dave Grohl and Jon Davison, the frontman of Yes[29]. The music video for the single "I Really Blew It" also featured appearances from Grohl and Perry Farrell[29].

# Collaborations and Guest Appearances
Throughout his career, Taylor Hawkins collaborated with various prominent artists and bands. The Coattail Riders' albums notably featured appearances from luminaries such as Brian May and Roger Taylor of Queen, Chrisise Hynde, Nancy Wilson of Heart, Sex Pistol Steve Jones and James Gang's Joe Walsh[28]. Hawkins also fronted another group, The Birds of Satan, which evolved from his heavy rock covers band, Chevy Metal[28].

Despite his diverse musical engagements, Hawkins always maintained a close allegiance with the Foo Fighters, which remained the center of his music life[7][28].

## Tragic Passing

Taylor Hawkins, the esteemed drummer of the alt-rock band Foo Fighters, passed away suddenly on March 25, 2022, while on tour with his band in Bogota, Colombia[34]. The official cause of death was cardiac arrest, though inquiries were raised concerning the presence of drugs in his system and their potential contribution to his death[33][34]. On the night of his passing, paramectics were called to the Four Seasons hotel in Bogota due to reports of chest pain from an unnamed guest, later revealed to be Hawkins[34]. Unfortunately, resuscitation efforts were unsuccessful, and Hawkins was declared dead at the scene[34].

The news of Hawkins' sudden demise was announced on the morning of March 25th, 2022, which left the music world in shock[32]. The band confirmed the news with a short statement, expressing their devastation at the loss of Hawkins, whose "musical spirit and infectious laughter" would live on forever[32].

As a result of Hawkins' untimely passing, the band canceled their ongoing South American tour[33]. The festival stage at the Estereo Picnic Festival, where the Foo Fighters were scheduled to perform that night, was transformed into a candellight vigil in memory of Hawkins[33].
 

In the wake of Hawkins' death, tributes from fans and colleagues alike poured in from around the world[21][31]. Among the many paying their respects were legendary rock and roll musicians like Roger Taylor, the drummer of Queen, who Hawkins credited with inspiring his own career behind the drum set[21]. In heartfelt social media posts, Taylor described Hawkins as an "inspirational mentor" and a "kind brilliant man"[21], while Led Zeppelin's Jimmy Page reminisiced about sharing the stage with Hawkins and praised his "technique, energy and spirited enthusiasm"[21].

There were also numerous onstage tributes to Hawkins. Notably, Miley Cyrus expressed her grief and sent peaceful wishes to the Foo Fighters and the Hawkins family during a performance at Lollapalooza[31]. Similarly, Liam Gallagher of Oasis dedicated one of the band's biggest hits to Hawkins during a concert at the Royal Albert Hall in London[31].

Fans gathered outside the hotel where Hawkins died, lighting candles, leaving flowers, and singing the band's songs in his honor[31].

Hawkins' life and career were celebrated in a star-studded tribute concert in Los Angeles, which saw performances from over 50 musicians, including his former bands and colleagues from Def Leppard, Queen, and Foo Fighters[22].

\begin{table} \end{table}
Table 12: STORM’s generated article for “Taylor Hawking”. “#”. “##” indicate the section title and subsection title respectively. Numbers in brackets indicate the cited references.